select from a column made of string array pyspark or python high order function multiple values-CodePudding

i have a table like this and i want to create a new column based on what is listed on the column example

df.withcolumn('good',.when('java' or 'php' isin ['booksIntereste']).lit(1).otherwise(0))

desired output when it contain java or php get 1 else 0

CodePudding user response：

You can directly use a Higher Order Function - array_contains for this , additionally you can browse through this article to understand more

Data Preparation

d = {
    'name':['James','Washington','Robert','Micheal'],
    'booksInterested':[['Java','C#','Python'],[],['PHP','Java'],['Java']]
}

sparkDF = sql.createDataFrame(pd.DataFrame(d))

sparkDF.show()

 ---------- ------------------ 
|      name|   booksInterested|
 ---------- ------------------ 
|     James|[Java, C#, Python]|
|Washington|                []|
|    Robert|       [PHP, Java]|
|   Micheal|            [Java]|
 ---------- ------------------

Array Contains

sparkDF = sparkDF.withColumn('good',F.array_contains(F.col('booksInterested'), 'Java'))

 ---------- ------------------ ----- 
|      name|   booksInterested| good|
 ---------- ------------------ ----- 
|     James|[Java, C#, Python]| true|
|Washington|                []|false|
|    Robert|       [PHP, Java]| true|
|   Micheal|            [Java]| true|
 ---------- ------------------ -----

ForAll Array Contains - Multiple

sparkDF = sparkDF.withColumn('good_multiple',F.forall(F.col('booksInterested'), lambda x: x.isin(['Java','Python','PHP'])))

sparkDF.show()

 ---------- ------------------ ----- ------------- 
|      name|   booksInterested| good|good_multiple|
 ---------- ------------------ ----- ------------- 
|     James|[Java, C#, Python]| true|        false|
|Washington|                []|false|         true|
|    Robert|       [PHP, Java]| true|         true|
|   Micheal|            [Java]| true|         true|
 ---------- ------------------ ----- -------------