where do I create a spark configuration file and set.driver.memory to 2gb?-CodePudding

I am new to spark and relatively new to Linux in general. I am running Spark on local Ubuntu in client mode. I have RAM of 16 GB. I installed apache spark following this

CodePudding user response：

You can change the default value for all sessions in

$SPARK_HOME/spark-defaults.conf

If you do not find spark-defaults.conf you should have a file spark-defaults.conf.template, just cp spark-defaults.conf.template spark-defaults.conf and edit it uncommenting the line:

# spark.driver.memory              5g

Alternatively, you can set the value just for the current session using .config in the session builder:

spark = SparkSession.builder \
       .master("local[*]") \
       .appName("myApp") \
       .config("spark.driver.memory", "5g") \
       .getOrCreate()

(perhaps you might also want to increase spark.executor.memory)

See also my other answer to a similar question.