The spark concept on yarn problem-CodePudding

Spark related things, recently doing some concept problems consult you

1, the spark on this architecture, yarn is suitable for real-time calculation? Or only for off-line calculation??
2, if I need to implement a web call spark, how should do?? I need to get some scene spark after the operation, the results of that what should I do??
3, I saw a lot of examples, the Internet can be used in Java sparkContext to invoke the spark, but at the time of setting the master, website points out that only can fill the master Cluster URL to connect to (e.g. Mesos://host: port, spark://host: port, local [4]), there is no yarn Cluster and client mode
4, some code directly write setMaster (" yarn - cluster "), the question, set up such a string, how to know what is the address of the spark program, how do you know what is the address of the yarn? My program and spark cluster is not on one machine, I think it must be have a problem of

This a few problems troubled me for a long time, the great god show ~ ~ ~

CodePudding user response:

Nobody answer agghh ah ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~

CodePudding user response:

1, the spark on yarn, is only the spark program running on the yarn,
2, the message-driven
3 + 4, if you need to make sure the Spark where Driver machine, Spark - submit to start the application of time specified by the client mode, namely, the deploy - mode client, Driver will run on the machines you commit, or, you can let the Driver start, get its own IP, and then write a place such as database, told to call the Spark program to access the IP,

CodePudding user response:

refer to the second floor link0007 response:

1, the spark on yarn, is only the spark program running on the yarn,
2, the message-driven
3 + 4, if you need to make sure the Spark where Driver machine, Spark - submit to start the application of time specified by the client mode, namely, the deploy - mode client, Driver will run on the machines you commit, or, you can let the Driver start, get their own IP, and then write a place such as database, told to call the Spark program to access the IP,

Message-driven mean?? What kind of process?? There are examples??

I know with the spark - how to carry out the submit, I want to know when you use the code to invoke the spark, how to call the spark on yarn

CodePudding user response:

reference sky402101 reply: 3/f

Quote: refer to the second floor link0007 response:

1, the spark on yarn, is only the spark program running on the yarn,
2, the message-driven
3 + 4, if you need to make sure the Spark where Driver machine, Spark - submit to start the application of time specified by the client mode, namely, the deploy - mode client, Driver will run on the machines you commit, or, you can let the Driver start, get their own IP, and then write a place such as database, told to call the Spark program to access the IP,

Message-driven mean?? What kind of process?? There are examples??

I know with the spark - how to carry out the submit, I want to know when you use the code to invoke the spark, how to call the spark on yarn

Server process request some time longer, general is to use the message driven asynchronous mode to deal with, for example, the front-end request to the big data side do wordcount, web services, to press the wordcount request into the queue and given a serial number, large data terminal receives the request, after processing is completed, the serial number and result in a cache or database, such as redis front need to query whether the serial number to complete, and after the completion of the outcome, don't need long time to take up servlet link

Spark Standalone and yarn, is your web application deployed in tomcat, or weblogic a meaning,,, there's not much difference between program to run

CodePudding user response:

reference 4 floor link0007 response:

Quote: reference sky402101 reply: 3/f

Quote: refer to the second floor link0007 response:

1, the spark on yarn, is only the spark program running on the yarn,
2, the message-driven
3 + 4, if you need to make sure the Spark where Driver machine, Spark - submit to start the application of time specified by the client mode, namely, the deploy - mode client, Driver will run on the machines you commit, or, you can let the Driver start, get their own IP, and then write a place such as database, told to call the Spark program to access the IP,

Message-driven mean?? What kind of process?? There are examples??

I know with the spark - how to carry out the submit, I want to know when you use the code to invoke the spark, how to call the spark on yarn

Server process request some time longer, general is to use the message driven asynchronous mode to deal with, for example, the front-end request to the big data side do wordcount, web services, to press the wordcount request into the queue and given a serial number, large data terminal receives the request, after processing is completed, the serial number and result in a cache or database, such as redis front need to query whether the serial number to complete, and after the completion of the outcome, don't need long time to take up servlet link

Spark Standalone and yarn, is your web application deployed in tomcat, or weblogic a meaning, and there's not much difference, the program to run

The first question, what do you mean with asynchronous way, I see; But if there is need to scene synchronous way?? The spark for this way??

Second question, I ask is not a Standalone and the difference between the yarn, the difference between them: I know, I asked the program calls the way how to write??????? Not to sparkcontext object, and then set the master? Standalone mode directly write the spark://XXXX, XXX. Xx, but yarn pattern did not know how to write

CodePudding user response:

reference 5 floor sky402101 reply:

Quote: refer to 4th floor link0007 response:

Quote: refer to the third floor sky402101 response:

Quote: refer to the second floor link0007 response:

1, the spark on yarn, is only the spark program running on the yarn,
2, the message-driven
3 + 4, if you need to make sure the Spark where Driver machine, Spark - submit to start the application of time specified by the client mode, namely, the deploy - mode client, Driver will run on the machines you commit, or, you can let the Driver start, get their own IP, and then write a place such as database, told to call the Spark program to access the IP,

Message-driven mean?? What kind of process?? There are examples??

I know with the spark - how to carry out the submit, I want to know when you use the code to invoke the spark, how to call the spark on yarn

Server process request some time longer, general is to use the message driven asynchronous mode to deal with, for example, the front-end request to the big data side do wordcount, web services, to press the wordcount request into the queue and given a serial number, large data terminal receives the request, after processing is completed, the serial number and result in a cache or database, such as redis front need to query whether the serial number to complete, and after the completion of the outcome, don't need long time to take up servlet link

Spark Standalone and yarn, is your web application deployed in tomcat, or weblogic a meaning, and there's not much difference, the program to run

The first question, what do you mean with asynchronous way, I see; But if there is need to scene synchronous way?? The spark for this way??

Second question, I ask is not a Standalone and the difference between the yarn, the difference between them: I know, I asked the program calls the way how to write??????? Not to sparkcontext object, and then set the master? Standalone mode directly write the spark://XXXX, XXX. Xx, but yarn model don't know how to write the

Synchronous proposal with other real-time flow framework

The yarn yarn - site. The XML copy to spark the conf directory, spark when submit master the yarn is ok, the official written documents have
http://spark.apache.org/docs/latest/running-on-yarn.html