How to create Spark dataframe in Scala from Lists-CodePudding

I have below lists,

val col1 = List(1,2,3,4,5)

val col2 = List("a", "b", "c", "d", "e")

val col3 = List(6,7,8)

Requirement is to create a dataframe as below in Scala,

--------------------
| col1 | col2| col3|
--------------------
|  1   |  a  |  6  |
|  2   |  b  |  7  |
|  3   |  c  |  8  |
|  4   |  d  | null|
|  5   |  e  | null|
--------------------

Thank you.

CodePudding user response：

If you're using Scala 2.12 or older, zipAll function may be helpful. Also using Option for nullability.

val data = col1.map(Option.apply)
  .zipAll(col2.map(Option.apply), None, None)
  .zipAll(col3.map(Option.apply), (None, None), None)
  .map { case ((c1, c2), c3) => (c1, c2, c3) }

val df = data.toDF("col1", "col2", "col3")

CodePudding user response：

Build rows iterating over indexes to get

import sparkSession.implicits._

val df = List((1,’a’),(2,’b’)).toDF()