Azure synapse equivalent commands-CodePudding

Looking for below mentioned equivalent command in Azure synapse analytics notebook. Below is from databricks.

''''1.spark.conf.set("spark.databricks.io.cache.enabled", "true")   
''''2.spark.conf.set("spark.databricks.delta.optimizeWrite.enabled","true")   
''''3.spark.conf.set("spark.databricks.delta.autoCompact.enabled","true")

CodePudding user response：

These are some Delta cache and auto optimization features which are applicable to Databricks only. Databricks is code oriented while Synapse allows UI based analytics.

Synapse is a separate data Analytics service provided by Azure which has different feature sets. It's not mandatory that Databricks and Synapse will share same features.

Synapse Apache Spark allows you to optimize components like data serialization, joins, shuffle, job execution, etc. These components are different from Databricks but the objective are quite similar.

You can learn more in details about Optimize Apache Spark jobs in Azure Synapse Analytics.

CodePudding user response：

Got the solution

spark.conf.set("spark.sql.autoBroadcastJoinThreshold", 1 * 1024 * 1024 * 1024)