Scala from_json function is throwing error when i use options-CodePudding

Below is my code:

val opts = Map("allowUnquotedFieldNames" -> "true")
    
val df_withSchema = df.withColumn("Data", from_json(col("Item.Data.S"),schema, opts))

Below is the error:

(<console>:198: error: overloaded method value from_json with alternatives:
  (e: org.apache.spark.sql.Column,schema: org.apache.spark.sql.Column,options: java.util.Map[String,String])org.apache.spark.sql.Column <and>
  (e: org.apache.spark.sql.Column,schema: String,options: scala.collection.immutable.Map[String,String])org.apache.spark.sql.Column <and>
  (e: org.apache.spark.sql.Column,schema: String,options: java.util.Map[String,String])org.apache.spark.sql.Column <and>
  (e: org.apache.spark.sql.Column,schema: org.apache.spark.sql.types.DataType,options: java.util.Map[String,String])org.apache.spark.sql.Column <and>
  (e: org.apache.spark.sql.Column,schema: org.apache.spark.sql.types.StructType,options: java.util.Map[String,String])org.apache.spark.sql.Column <and>
  (e: org.apache.spark.sql.Column,schema: org.apache.spark.sql.types.DataType,options: scala.collection.immutable.Map[String,String])org.apache.spark.sql.Column <and>
  (e: org.apache.spark.sql.Column,schema: org.apache.spark.sql.types.StructType,options: scala.collection.immutable.Map[String,String])org.apache.spark.sql.Column
 cannot be applied to (org.apache.spark.sql.Column, org.apache.spark.sql.types.StructType, scala.collection.Map[String,String])
           val df_withSchema = df.withColumn("Data", from_json(col("Item.Data.S"),schema, opts))
                                                     ^
,0,1388)

Can someone please help me understand why this code is throwing error and how to fix this?

Background:

Data element is a Nested JSON stored as string. So i have defined a schema for Data element and converting it as JSON using from_JSON. Also, the fields names inside Data are under quotes so i am trying to use options here(Map("allowUnquotedFieldNames" -> "true"))

CodePudding user response：

It wants scala.collection.immutable.Map, and you have scala.collection.Map ...

I know, one would naively expect to be able to use just any kind of a map with the code that promises not to mutate it, but it does't work like that. It is actually the other way around: immutable.Map is a subclass of Map.

Try doing opts.toMap, that should fix it.