I'm trying to learn Scala. I ask for help to understand loop foreach I have a function, it reads only last csv from path. But it works when I point only one way:
val path = "src/main/resources/historical_novel/"
def getLastFile(path: String, spark: SparkSession): String = {
val hdfs = ...}
but how can I apply this function to the list such as
val paths: List[String] = List(
"src/main/resources/historical_novel/",
"src/main/resources/detective/",
"src/main/resources/adventure/",
"src/main/resources/horror/")
I want to get such result:
src/main/resources/historical_novel/20221027.csv
src/main/resources/detective/20221026.csv
src/main/resources/adventure/20221026.csv
src/main/resources/horror/20221027.csv
I create df with column (path), then apply function through WithColumn and it is work, but I want to do it with foreach, understand it.
CodePudding user response:
let's say your function is like this
def f(s: String): Unit = {}
you can simply do this
paths.foreach(p => f(p))
CodePudding user response:
foreach
applies a function you define or provide on each element in a Collection
.
The simplest example is to print each path to the console
paths.foreach(path => println(path))
To apply a series of functions as you describe you can use {}
in the foreach
body and call multiple functions.
paths.foreach(path => {
val file = loadFile(path)
writeToDataBase(file)
})