Single query mysql to query on the spark, the logic how to implement?-CodePudding

Now want to implement such a logic:

Single, generate a random number of mysql query, if the query, further to the spark for more information on the cluster,

The program written, submitted to the cluster can run, but think if wrong:

Is not submitted to the cluster, each node will be distributed to generate its own random number, and then each node each query mysql?

I want to implement the logic of how to do?

CodePudding user response:

You know what code execution on the Driver, what on the Executor implementation will understand,

CodePudding user response:

reference 1st floor link0007 response:

you understand what code execution on the Driver, what on the Executor implementation will understand,

+ 1, there seems to be wise,

But if the entire process to make cycle, it will switch back and forth between driver and cluster?

CodePudding user response:

refer to the second floor sizheng0320 response:

Quote: refer to 1st floor link0007 response:

You know what code execution on the Driver, what on the Executor implementation will understand,

+ 1, there seems to be wise,

But if the entire process to make cycle, it will switch back and forth between driver and cluster?

Your Spark program is a Driver, you all sorts of RDD operator and Spark SQL is the executor implementation,
For example:

 
List List=XXXDao. GetXXX ();//driver execution 
JavaRDD RDD=sparkContext. Parallelize (list);//driver to send data to the executor, constructs the RDD 
RDD. The map (STR - & gt; STR. The trim ());//implementation of executor 

Another example: 
For (String SQL: list) {//such as this list is a number of SQL 
SqlContext. SQL (SQL);//loop control in driver execution, and the implementation of SQL is on the executor; Driver submitted such operation, can complete and feedback synchronization waiting executor, 
} 
Multithreading: 
For (final String SQL: list) {
New Thread () {
Public void the run () {
SqlContext. SQL (SQL);//as long as the cluster resources abundant, multiple SQL tasks concurrently 
} 
}.start();//sqlContext is thread-safe, 
}

CodePudding user response:

~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ pass by!! 1