Create a pyspark dataframe-CodePudding

My code is

pdf = pd.DataFrame(
{
    "Year": [x for x in range(2013, 2051)],
    "CSIRO Adjusted Sea Level": 0.0,
}
) 
pdf.head()

df_pyspark = spark.createDataFrame(pdf)
df_pyspark.show()

The above results in this error:

An error occurred while calling o406.showString.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 11.0 failed 1 times, most recent failure: Lost task 0.0 in stage 11.0 (TID 11) (192.168.1.66 executor driver): 
  org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "C:\spark-3.2.1-bin-hadoop3.2\python\lib\pyspark.zip\pyspark\worker.py", line 601, in main
  File "C:\spark-3.2.1-bin-hadoop3.2\python\lib\pyspark.zip\pyspark\worker.py", line 71, in read_command
  File "C:\spark-3.2.1-bin-hadoop3.2\python\lib\pyspark.zip\pyspark\serializers.py", line 160, in _read_with_length
    return self.loads(obj)
  File "C:\spark-3.2.1-bin-hadoop3.2\python\lib\pyspark.zip\pyspark\serializers.py", line 430, in loads
    return pickle.loads(obj, encoding=encoding)
AttributeError: Can't get attribute '_fill_function' on <module 'pyspark.cloudpickle' from 'C:\\spark-3.2.1-bin-hadoop3.2\\python\\lib\\pyspark.zip\\pyspark\\cloudpickle\\__init__.py'>

And there's a lot more text. What did I do wrong?

I also tried

lista =[(i, 0.0) for i in range(2013, 2051)]
df = spark.createDataFrame(
[
    lista
],  
"Year, Sea Level",  
 )

...and got this error:

ValueError: Length of object (38) does not match with length of fields (2)

CodePudding user response：

Its simple if all you need is a spark dataframe:

>>> from pyspark.sql.functions import *
>>> df = spark.range(2013,2051).withColumnRenamed("id","Year").withColumn("Sea Level",lit(0.0))
>>> df.show(10)
 ---- ---------                                                                 
|Year|Sea Level|
 ---- --------- 
|2013|      0.0|
|2014|      0.0|
|2015|      0.0|
|2016|      0.0|
|2017|      0.0|
|2018|      0.0|
|2019|      0.0|
|2020|      0.0|
|2021|      0.0|
|2022|      0.0|
 ---- --------- 
only showing top 10 rows