Change the datatype of any fields of Arraytype column in Pyspark-CodePudding

I want to change the datatype of the field "value", which is inside the arraytype column "readings". The column "reading" has two fields, "key" nd "value".

root
 |-- name: string (nullable = true)
 |-- languagesAtSchool: array (nullable = true)
 |    |-- element: string (containsNull = true)
 |-- languagesAtSchool1: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- _id: integer (nullable = true)
 |-- languagesAtWork: array (nullable = true)
 |    |-- element: string (containsNull = true)
 |-- currentState: string (nullable = true)
 |-- previousState: double (nullable = true)
 |-- readings: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- value: integer (nullable = true)
 |    |    |-- key: string (nullable = true)

Expected Schema is

   root
     |-- name: string (nullable = true)
     |-- languagesAtSchool: array (nullable = true)
     |    |-- element: string (containsNull = true)
     |-- languagesAtSchool1: array (nullable = true)
     |    |-- element: struct (containsNull = true)
     |    |    |-- _id: integer (nullable = true)
     |-- languagesAtWork: array (nullable = true)
     |    |-- element: string (containsNull = true)
     |-- currentState: string (nullable = true)
     |-- previousState: double (nullable = true)
     |-- readings: array (nullable = true)
     |    |-- element: struct (containsNull = true)
     |    |    |-- value: string (nullable = true)
     |    |    |-- key: string (nullable = true)

CodePudding user response：

Transform using higher order function

df1=df.withColumn('readings', expr('transform(readings, x-> struct(cast(x.value as integer) value,x.key))'))

df.withColumn('readings', F.transform('readings', lambda x: x.withField('value', x['value'].cast('int'))))


root
 |-- name: string (nullable = true)
 |-- languagesAtSchool: array (nullable = true)
 |    |-- element: string (containsNull = true)
 |-- languagesAtSchool1: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- id: integer (nullable = true)
 |-- languagesAtWork: array (nullable = true)
 |    |-- element: string (containsNull = true)
 |-- currentState: string (nullable = true)
 |-- previousState: double (nullable = true)
 |-- readings: array (nullable = true)
 |    |-- element: struct (containsNull = false)
 |    |    |-- value: integer (nullable = true)
 |    |    |-- key: string (nullable = true)