I wanted to read a file in Spark Scala having name: monthlyPurchaseFile{202205}-May.TXT
I am using below code:
val df = spark.read.text("handel_special_ch/monthlyPurchaseFile{202205}-May.TXT"

But I am getting below exception:

org.apache.spark.sql.AnalysisException: Path does not exist: file:/home/hdp_batch_datalake_dev/handel_special_ch/monthlyPurchaseFile{202205}-May.TXT
  at org.apache.spark.sql.execution.datasources.DataSource$.$anonfun$checkAndGlobPathIfNecessary$3(DataSource.scala:792)
  at org.apache.spark.util.ThreadUtils$.$anonfun$parmap$2(ThreadUtils.scala:372)
  at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
  at scala.util.Success.$anonfun$map$1(Try.scala:255)
  at scala.util.Success.map(Try.scala:213)
  at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
  at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
  at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
  at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
  at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
  at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
  at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
  at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
  at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)

Please suggest, how I can read that file having character {, } in its name.

CodePudding user response：

The path you are passing to the spark.read.text method is treated as a regular expression. Since { and } are special characters, Spark tries to match a path against that expression. You can use the ? character to match any character, so the following should work:

val df = spark.read.text("handel_special_ch/monthlyPurchaseFile?202205?-May.TXT"