Home > Software design >  PySpark type hinting for returning a Column (expression)?
PySpark type hinting for returning a Column (expression)?

Time:10-07

How do you type hint that a function will return a PySpark Column type?

import pyspark.sql.functions as F
def get_some_filter_expression(col_string_name) -> TODO:
    return F.col(col_string_name)

I included a dummy example above, and I am trying to figure out what TODO should be.

EDIT: I guess the reason I had this question was because I was assuming that PySpark "column types" under spark.sql.types could be used for type-hinting. This is not the case. In case it is a common confusion, I will leave this question as is. See the answer of LiamFiddler below.

CodePudding user response:

You can do the following:

from pyspark.sql import Column, Functions as F
def get_some_filter_expression(col_string_name) -> Column:
     return F.col(col_string_name)
  • Related