How to connect airflow with local spark-CodePudding

I try to execute local spark job through airflow task:

spark = (SparkSession
            .builder
            .master("spark://172.22.102.229:7077")
            .appName("Test")
            .getOrCreate())

But get a error:

py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.
: java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem

When i replace .master("spark://172.22.102.229:7077") to .master("local") it is working

My spark deployed and i can get web ui at http://172.22.102.229:4040/ address

CodePudding user response：

If you can access Spark WebUI go to the WebUI and you will see something like this:

Use the URL to create your SparkSession. For example:

spark = SparkSession
            .builder
            .master("spark://10.0.6.187:7077")
            .appName("Test")
            .getOrCreate()

Hope this can help you! Best Regards