I am getting this error on Google Colab. Is it because I'm using the wrong version?
ImportError: cannot import name 'VarcharType' from 'pyspark.sql.types' (/content/spark-3.1.2-bin-hadoop2.7/python/pyspark/sql/types.py)
CodePudding user response:
That is because VarcharType does not exists yet in spark 3.1.2.
The source code of pyspark.sql.types
in __all__
declares the only available types to import:
__all__ = [
"DataType", "NullType", "StringType", "BinaryType", "BooleanType", "DateType",
"TimestampType", "DecimalType", "DoubleType", "FloatType", "ByteType", "IntegerType",
"LongType", "ShortType", "ArrayType", "MapType", "StructField", "StructType"
]
Therefore, StringType
is the alternative:
from pyspark.sql.types import StringType