Spark dataframe from dictionary-CodePudding

I'm trying to create a spark dataframe from a dictionary which has data in the format {'33_45677': 0, '45_3233': 25, '56_4599': 43524} .. etc.

dict_pairs={'33_45677': 0, '45_3233': 25, '56_4599': 43524} df=spark.createDataFrame(data=dict_pairs). It throws TypeError

TypeError: can not infer schema for type: <class 'str'>

Is it because of the underscore in the keys of the dictionary.

CodePudding user response：

Enclose dict using square braces '[]'. Its not because of _ in your keys.

dict_pairs={'33_45677': 0, '45_3233': 25, '56_4599': 43524}
df=spark.createDataFrame(data=[dict_pairs])
df.show()

dict_pairs=[{'33_45677': 0, '45_3233': 25, '56_4599': 43524}]
df=spark.createDataFrame(data=dict_pairs)
df.show()