My input is

In pandas dataframe I used literal_eval to convert 'hashtags' column from str to list, but I don't know how to do that in Spark Dataframe. Any solution for my case?

Tks all supports for my case!

CodePudding user response：

You can use:

df1.withColumn("list_hashtags", expr("from_json(hashtags, 'array<string>')"))

Therefore, list_hashtags[0] will give you #ucl.

Hope this is what you want!