Spark Scala - Create an object with element from my dataframe-CodePudding

For a Scala Project I am working on, I had to create a class Table as below :

class Table(PbddName : String , PTableName: String, POutputFilename: String="", PFilteringFlag: Boolean=true) {
  var bddName : String = PbddName
  var TableName: String = PTableName
  var OutputFilename: String = POutputFilename
  var FilteringFlag: Boolean =PFilteringFlag
}

To create a Table, I use this code in a Scala object :

        val randomName = new Table(bddName, blablaTableName, blablaOutputFilename,blablaFilteringFlag)
        list_tables =randomName

I also add it to list_Tables that obviously list all the tables.

But as I have a lot of "Tables" to create, I made a csv where each line contains the needed values to create a Table (matching the values of the class). I loaded my CSV file in a DataFrame called empDF that matches this :

|bdd_name|    table_name|     file_name|filtering_flag|
 -------- -------------- -------------- -------------- 
|    bdd1|name1tablename| name1Filename|          true|
|    bdd2|name2tablename| name2Filename|          true|
|    bdd3|name3tablename| name3Filename|         false|
|    bdd4|name4tablename| name4Filename|          true|
 -------- -------------- -------------- --------------

I would like to know how to automate the creation of my "Tables" elements ?(and add the created table to my list_Tables). I am not sure, but I think I might have to browse my dataframe somehow to create the element at each line that is readden.

Also, each val name (here randomName) must be different, but can be random.

If you have any idea, or any clue on how I could do this, that would help me a lot.

Thank's for your help.

CodePudding user response：

I found a solution to do what I wanted :

for (row <- empDF.rdd.collect) {
  var bddname = row.mkString(",").split(",")(0)
  var tablename = row.mkString(",").split(",")(1)
  var Outputfilename = row.mkString(",").split(",")(2)
  var filteringflag = row.mkString(",").split(",")(3)



  val nomtable =new Table(bddname, tablename, Outpufilename, filteringflag, tablename)
  list_tables  =nomtable
}

I actually don't need a different name for all the tables so this solution works for me at this point. The only thing is, I counldn't process the same method for both String and Boolean so I had to change all my elements from my Table class to String.