Drop columnns from dataframe where last three characters equal-CodePudding

I wish to only select particular columns from a dataframe, however the columns I don't want all end with "Nav".

How can I accomplish this?

I've tried something similar to the below

jsonDF2.select([c for c in jsonDF2.columns if c not in {'%Nav'}])

Any advice would be appreciated.

CodePudding user response：

Try this:

from pyspark.sql import functions as F

jsonDF2.select(*[F.col(c) for c in jsonDF2.columns if not c.endswith("Nav")])

CodePudding user response：

This should do it

[c for c in jsonDF2.columns if c[-3:] != 'Nav']

CodePudding user response：

You can use colRegex which is available in spark >= 2.3

df.select(df.colRegex('`.*(?<!nav)`'))