How to add empty map<string,string> type column to DataFrame in PySpark?-CodePudding

I tried below code but its not working:

df=df.withColumn("cars", typedLit(Map.empty[String, String]))

Gives the error: NameError: name 'typedLit' is not defined

CodePudding user response：

Perhaps you can use pyspark.sql.functions.expr:

>>> from pyspark.sql.functions import *
>>> df.withColumn("cars",expr("map()")).printSchema()                                                                                                       
root
 |-- col1: string (nullable = true)
 |-- cars: map (nullable = false)
 |    |-- key: string
 |    |-- value: string (valueContainsNull = false)

CodePudding user response：

Create an empty column and cast it to the type you need.

from pyspark.sql import functions as F, types as T

df = df.withColumn("cars", F.lit(None).cast(T.MapType(T.StringType(), T.StringType())))

df.select("cars").printSchema()
root
 |-- cars: map (nullable = true)
 |    |-- key: string
 |    |-- value: string (valueContainsNull = true)