How to get the input from local and store it to local in a dataframe-CodePudding

    import org.apache.spark.sql.SparkSession
    import org.apache.spark.sql.functions.{min, max, avg}
    val spark = SparkSession.builder().getOrCreate()
    
    val data = spark.read.option("header","true").format("csv").load("cpid_data.csv")
    
   val equi = data.withColumn("value",col("value").cast("double")).groupBy("id").agg(avg("value"), max("value"), min("value")).show()

In the above code when I try to write the output in a csv file like,

equi.write.option("header",true).csv("cpido.csv")

It is throwing error like write is not a member of unit.

could anyone help me with this? How to load the output to a csv file?

CodePudding user response：

The error message is giving you a strong indication: write is not a member of Unit means there's no method called write on the type Unit.

Said differently: equi is of type Unit, which is probably not what you desired.

Just remove the call to .show() and it'll work fine:

val equi = data
  .withColumn("value",col("value").cast("double"))
  .groupBy("id")
  .agg(avg("value"), max("value"), min("value"))

equi.write.option("header",true).csv("cpido.csv")