The following code will convert an Apache Spark DataFrame to a Great_Expectations DataFrame. For if I wanted to convert the Spark DataFrame, spkDF to a Great_Expectations DataFrame I would do the following:

ge_df = SparkDFDataset(spkDF)

Can someone let me know how convert a Great_Expectation dataframe to a Spark DataFrame.

So what would I need to do to convert the new Great_Expectations dataframe ge_df back to Spark DataFrame?

CodePudding user response：

According to the official documentation, the class SparkDFDataset holds the original pyspark dataframe:

This class holds an attribute spark_df which is a spark.sql.DataFrame.

So you should be able to access it with :

ge_df.spark_df