PYSpark data Frame schema is showing String for every column-CodePudding

I am reading CSV file from below code snippet

df_pyspark = spark.read.csv("sample_data.csv") df_pyspark

and when i try to print data Frame its output is like shown as below:

DataFrame[_c0: string, _c1: string, _c2: string, _c3: string, _c4: string, _c5: string]

For each column dataType is showing 'String' even though column contains different dataType's as below:

df_pyspark.show()

|_c0|       _c1|      _c2|                 _c3|        _c4|       _c5|
 --- ---------- --------- -------------------- ----------- ---------- 
| id|first_name|last_name|               email|     gender|     phone|
|  1|    Bidget| Mirfield|bmirfield0@scient...|     Female|5628618353|
|  2|   Gonzalo|    Vango|    [email protected]|       Male|9556535457|
|  3|      Rock| Pampling|rpampling2@guardi...|   Bigender|4472741337|
|  4|   Dorella|  Edelman|dedelman3@histats...|     Female|4303062344|
|  5|     Faber|  Thwaite|fthwaite4@google....|Genderqueer|1348658809|
|  6|     Debee| Philcott|dphilcott5@cafepr...|     Female|7906881842|`

I want to print the exact DataType of every column?

CodePudding user response：

Use inferSchema parameter during read of CSV file it'll Show the exact/correct datatype according to the values in columns

    df_pyspark = spark.read.csv("sample_data.csv", header=True, inferSchema=True)

     --- ---------- --------- -------------------- ----------- ---------- 
    | id|first_name|last_name|               email|     gender|     phone|
     --- ---------- --------- -------------------- ----------- ---------- 
    |  1|    Bidget| Mirfield|bmirfield0@scient...|     Female|5628618353|
    |  2|   Gonzalo|    Vango|    [email protected]|       Male|9556535457|
    |  3|      Rock| Pampling|rpampling2@guardi...|   Bigender|4472741337|
    |  4|   Dorella|  Edelman|dedelman3@histats...|     Female|4303062344|
    |  5|     Faber|  Thwaite|fthwaite4@google....|Genderqueer|1348658809|
     --- ---------- --------- -------------------- ----------- ---------- 
    only showing top 5 rows

    df_pyspark.printSchema()

    root
     |-- id: integer (nullable = true)
     |-- first_name: string (nullable = true)
     |-- last_name: string (nullable = true)
     |-- email: string (nullable = true)
     |-- gender: string (nullable = true)
     |-- phone: long (nullable = true)