Home > Software design >  Spark Datasets available in Python?
Spark Datasets available in Python?

Time:09-27

Here, it is stated:

..you can create Datasets within a Scala or Python..

while here, the following is stated:

Python does not have the support for the Dataset API

Are datasets available in python?

CodePudding user response:

Perhaps the question is about Typed Spark Datasets.

If so, then the answer is no.

Mentioned spark datasets are only available in Scala and Java.

In Python implementation of Spark (or PySpark) you have to choose between DataFrames as the preferred choice and RDD.

Reference: RDD vs. DataFrame vs. Dataset

Update 2022-09-26: Clarification regarding typed spark datasets

  • Related