Transform list of map to dataframe-CodePudding

I have the following data:

d = Some(List(Map(id -> 1, n -> Hi), Map(id -> 2, n -> Hello)))

I would like to transform it into a dataframe like the following:

 -------- 
|id|n    |
 -------- 
|1 |Hi   |
 -------- 
|2 |Hello|
 --------

I tried the following:

import spark.implicits._

val df = d
  .map( m => (m.get("id"),m.get("n")))
  .toDF("id", "n")

But im getting:

error: value get is not a member of Any
  .map( m => (m.get("id"),m.get("n")))

CodePudding user response：

Your top level here is Option and i think thats the reason why you cant handle it with single map. I managed to do it with something like this:

import spark.implicits._

val d = Some(List(Map("id" -> "1", "n" -> "Hi"), Map("id" -> "2", "n" -> "Hello")))

val data = d.fold (List.empty [(Option [String], Option [String])]) (_.map (m => (m.get ("id"), m.get ("n"))))

val df = data.toDF("id", "n").show()

Output:

 --- ----- 
| id|    n|
 --- ----- 
|  1|   Hi|
|  2|Hello|
 --- -----