If we have a big table. I created a DataFrame from it. In another spark session, I need same table. Is there any way we can use the same table to create dataframes in two different spark sessions?

CodePudding user response：

Take a look at createGlobalTempView it will be available across Spark Sessions.

If you want to have a temporary view that is shared among all sessions and keep alive until the Spark application terminates, you can create a global temporary view.

More information: https://spark.apache.org/docs/latest/sql-getting-started.html#global-temporary-view

CodePudding user response：

Creating a hive table or writing the parquet/Orc files to a shared folder?

It is more permanent but it might make sense so one spark session ending can't kill the others data.