Tuple of Tuple as data in DataFrame results in AttributeError: 'tuple' object has no attri-CodePudding

Say I've got sample data:

sdata = [(1,(10,20,30)),
         (2,(100,20)),
         (3,(100,200,300))]

columns = [('Sn','Products')]
df1 = spark.createDataFrame(([x[0],*x[1]] for x in sdata), schema=columns)

Getting error:

AttributeError: 'tuple' object has no attribute 'encode'

How to load this variable length data ?

CodePudding user response：

You can represent tuples as StructType; but it has fixed fields. I am not sure about "variable length" tuples; but if your requirement is to support variable number of elements in a collection type, then you can either define an explicit schema:

sdata = [(1,(10,20,30)),
         (2,(100,20)),
         (3,(100,200,300))]

schema = StructType([
  StructField('Sn', LongType()),
  StructField('Products', ArrayType(LongType())),
])

df1 = spark.createDataFrame(sdata, schema=schema)

[Out]:
 --- --------------- 
| Sn|       Products|
 --- --------------- 
|  1|   [10, 20, 30]|
|  2|      [100, 20]|
|  3|[100, 200, 300]|
 --- ---------------

or use field directly as an array:

sdata = [(1,[10,20,30]),
         (2,[100,20]),
         (3,[100,200,300])]

columns = ['Sn','Products']

df1 = spark.createDataFrame(sdata, schema=columns)

[Out]:
 --- --------------- 
| Sn|       Products|
 --- --------------- 
|  1|   [10, 20, 30]|
|  2|      [100, 20]|
|  3|[100, 200, 300]|
 --- ---------------

CodePudding user response：

To load variable-length data into a PySpark DataFrame, you can use the ArrayType() function from the pyspark.sql.types module to define the schema of your DataFrame. The ArrayType() function allows you to specify the data type of the elements in an array, which can be used to define a column in a DataFrame that contains a variable number of elements.

Here is an example of how to use the ArrayType() function to define the schema of a DataFrame that contains variable-length data:

# Import the ArrayType() function
from pyspark.sql.types import ArrayType

# Define the sample data
sdata = [(1,(10,20,30)),
            (2,(100,20)),
            (3,(100,200,300))]

# Use the ArrayType() function to define the schema of the DataFrame
columns = [('Sn', IntegerType()),
            ('Products', ArrayType(IntegerType()))]

# Create the DataFrame with the defined schema
df1 = spark.createDataFrame(([x[0],*x[1]] for x in sdata), schema=columns)

# Print the schema of the DataFrame
df1.printSchema()