Home > other >  Multiple users share the spark
Multiple users share the spark

Time:10-29

Consult, I set up on a Linux server Spark Standalone mode, but found that different users to create instances of Spark in python is different,
Such as user 1, perform the following statement:
 the from pyspark. SQL import SparkSession 
The from pyspark import SparkConf
Master="spark://localhost: 7077
The conf=SparkConf (.) setAppName (" Spark_Standalone "). SetMaster (master)
The spark=SparkSession. Builder. Config (conf=conf.) getOrCreate ()
Sc=spark. SparkContext

Tips on port 4040 launched SparkUI

Then use the user login, 2 also execute the statement:
System does suggest 4040 port is not binding, in 4041 started the SparkUI,

Just starting the Spark, and online search for information about multiple users at the same time use the Spark, but did not find the answer,
1. The above two users use getOrCreate () is created two different instances of the Spark?

2. Users 1 and 2 can't execute a statement at the same time (if the user did not take the initiative to stop the spark) 1, how can you make many users submit tasks at the same time?
User 1: perform the following sample program
 num_samples=500000000 
Def inside (p) :
X, y=random. The random (), random, random ()
Return x * x + y * y & lt; 1
Count=sc. Parallelize (range (0, num_samples)). The filter (inside). The count ()
PI=4 * count/num_samples


complete, whether the statement of user 1 2 users perform the same statement, will be stuck not perform, see the user 2 SparkUI, said waiting for (should be wait for the user resource release 1), if the user 1 carried out spark. After the stop (), 2 began to execute the statement of the user, how can a multi-user execution at the same time? Because the server is not a single person with
  • Related