'DataFrame' object does not support item assignment-CodePudding

I imported a df into Databricks as a pyspark.sql.dataframe.DataFrame. Within this df I have 3 columns (which I have verified to be strings) that I wish to concatenate. I have tried to use a simple " " function first, eg.

df["fullname"] = df["firstname"]   df["middlename"]   df["lastname"]

But I keep receiving the error "'DataFrame' object does not support item assignment". So I tried to add .astype(str) after every column with no avail. Finally I tried to simply add another column full of the number 5:

df['new_col'] = 5

and received the same error. So now Im thinking maybe this dataframe is immutable. But I even tried to make a copy of the original df hoping I could modify it

df2 = df.select('*')

But once again I could not concatenate or modify the new dataframe.

Any help is greatly appreciated!

CodePudding user response：

It sounds like you're trying to modify the dataframe in place, which is not allowed in PySpark. Instead, you can use the withColumn() method to create a new dataframe with the concatenated column. Here's an example:

from pyspark.sql.functions import concat, lit

# Concatenate the columns and create a new column called "fullname"
df = df.withColumn("fullname", concat(df["firstname"], df["middlename"], df["lastname"]))

# You can also add a new column with a constant value using the same method
df = df.withColumn("new_col", lit(5))

The concat() function is used to concatenate the columns together, and the lit() function is used to create a column with a constant value.

Hope this helps!

CodePudding user response：

The error message you are getting suggests that the DataFrame object you are trying to modify is immutable, which means that it cannot be changed. To solve this problem, you will need to create a new DataFrame object that contains the concatenated column. You can do this using the withColumn method, which creates a new DataFrame by adding a new column to the existing DataFrame. Here is an example of how you can use withColumn to concatenate the three columns in your DataFrame and create a new DataFrame:

from pyspark.sql.functions import concat

# Concatenate the columns and create a new DataFrame
df2 = df.withColumn("fullname", concat(df["firstname"], df["middlename"], df["lastname"]))

This will create a new DataFrame called df2 that contains the concatenated column. You can then use this new DataFrame for any further operations you need to perform on your data.

Alternatively, if you don't need to keep the original DataFrame and want to modify the existing DataFrame, you can use the withColumn method in place of the assignment operator (=) to add the new column to the existing DataFrame:

from pyspark.sql.functions import concat

# Concatenate the columns and add the new column to the existing DataFrame
df = df.withColumn("fullname", concat(df["firstname"], df["middlename"], df["lastname"]))

This will add the new column to the existing DataFrame and you can then use the updated DataFrame for any further operations you need to perform.