I'm trying to read all the csv under a HDFS directory to a dataframe, but got an error that says its "not a valid DFS filename" Could someone help to point out what I did wrong? I tried without the hdfs:// part as well but it says path could not be found. Many thanks.

val filelist = "hdfs://path/to/file/file1.csv,hdfs://path/to/file/file2.csv "
val df = spark.read.csv(filelist)

CodePudding user response：

val df = spark.read.csv(filelist:_*)