Add dataframe to list of dataframes -pyspark-CodePudding

I have created a list of dataframes and now want to add another dataframe in order to union this list afterwards.

df_list = [spark.createDataFrame([("3", "4")], db_tables_schema), spark.createDataFrame([("1", "2")], db_tables_schema), spark.createDataFrame([("7", "4")], db_tables_schema)]

df_list = df_list.append([spark.createDataFrame([("7", "4")], db_tables_schema)])

#Reduce to one df
df_complete = reduce(DataFrame.union, df_list)

I thought that with .append one could easily add the element, however if I do so a "NoneType" Element is being created - so basically my list isn´t existing anymore. What´s the matter here. Thanks for helping!

CodePudding user response：

Careful, you are using append function and it adds an item to a list inplace. This function returns None, so if you assign to df_list the return of df_list.append it is the same as : df_list = None. Just do it like this:

df_list.append(my_element)

Secondly, you are trying to append a list of item(s), if you do it with append your list will look like this:

my_list = [1, 2, 3]
my_list.append([4])
print(my_list)  # [1, 2, 3, [4]]
my_list.append([5, 6])
print(my_list)  # [1, 2, 3, [4], [5, 6]]

You should use extend instead:

my_list = [1, 2, 3]
my_list.extend([4])
print(my_list)  # [1, 2, 3, 4]
my_list.extend([5, 6])
print(my_list)  # [1, 2, 3, 4, 5, 6]