I have a problem statement all the column type is string

Column A pattern if it matches in column C then update Good Stuff with 1 ,else (-)

Any Query in pyspark, sparksql

Thanks Anuj Gupta

CodePudding user response：

If I understood your problem correctly, this should work for you:

from pyspark.sql.functions import col, when

df.withColumn('Good Stuff', when(col('RouteTypeRankFinal').startsWith(col('Text(RouteRank)')), lit(1)).otherwise(lit('-')))