how to replace ilike in pyspark sql-CodePudding

I have a query

select * from 
table 
 where (primary_product NOT IN ('No Technical Enforcement') or group_name ilike ('%stove%'))

I want to convert the same query into Pyspark SQL but I am not able to do the same as I don't know the substitute of ILIKE Please suggest a substitute for the same.

CodePudding user response：

ILIKE does exist in Spark SQL. Here you can find the documentation.

Example:

SELECT ilike('Spark', '_Park');

Returns true.

CodePudding user response：

You can use like this

Import col from sql functions in pyspark

from pyspark.sql.functions import col

like filter condition

df.filter(col("group_name").like("%stove%")).show()

OR

spark.sql("select * from table where group_name like '%stove%'").show()