Dynamic Query on Scala Spark-CodePudding

I'm basically trying to do something like this but spark doesn’t recognizes it.

val colsToLower: Array[String] = Array("col0", "col1", "col2")

    val selectQry: String = colsToLower.map((x: String) => s"""lower(col(\"${x}\")).as(\"${x}\"), """).mkString.dropRight(2)

    df
      .select(selectQry)
      .show(5)

Is there a way to do something like this in spark/scala?

CodePudding user response：

If you need to lowercase the name of your columns there is a simple way of doing it. Here is one example:

df.columns.foreach(c => {
  val newColumnName = c.toLowerCase
  df = df.withColumnRenamed(c, newColumnName)
})

This will allow you to lowercase the column names, and update it in the spark dataframe.

CodePudding user response：

I believe I found a way to build it:

    def lowerTextColumns(cols: Array[String])(df: DataFrame): DataFrame = {
     val remainingCols: String = (df.columns diff cols).mkString(", ")
     val lowerCols: String = cols.map((x: String) => s"""lower(${x}) as ${x}, """).mkString.dropRight(2)
  
     val selectQry: String =
       if (colsToSelect.nonEmpty) lowerCols   ", "   remainingCols
       else lowerCols
  
     df
      .selectExpr(selectQry.split(","):_*)
  }