import org. Apache. Spark. SparkConf; 
The import org. Apache. Spark. API. Java. JavaRDD; 
The import org. Apache. Spark. API. Java. JavaSparkContext; 
The import org. Apache. Spark. API. Java. The function. The function; 
Import org. Apache. Spark. API. Java function. Function2; 

import java.util.ArrayList; 
import java.util.List; 

/* * 
* Computes an approximation to PI 
* Usage: JavaSparkPi [slices] 
*/
Public final class RemoteDebug {

Public static void main (String [] args) throws the Exception {
SparkConf SparkConf=new SparkConf (). SetAppName (" JavaSparkPi ") 
SetMaster (" spark://10.3.9.135:7077 ") 
SetJars (new String [] {" D: \ \ * * * * \ \ workspace \ \ SparkRemoteDebug \ \ out \ \ artifacts \ \ SparkRemoteDebug_jar \ \ SparkRemoteDebug jar "}); 
JavaSparkContext JSC=new JavaSparkContext (sparkConf); 
//JSC. AddJar (" D: \ \ * * * \ \ workspace \ \ SparkRemoteDebug \ \ out \ \ artifacts \ \ SparkRemoteDebug_jar \ \ SparkRemoteDebug jar "); 

Int slices=(args. The length==1)? Integer. ParseInt (args [0]) : 2; 
Int n=100000 * slices; 
List L=new ArrayList (n); 
for (int i=0; i L.A. dd (I); 
} 

JavaRDD The dataSet=JSC. Parallelize (l, slices); 

Int count=dataSet. The map ((Function) The integer - & gt; {
Double x=Math. The random () * 2-1; 
Double y=Math. The random () * 2-1; 
Return (x * x + y * y & lt; 1)? 1:0; 
}). Reduce ((Function2 & lt; Integer, Integer, Integer>) (integer, integer2) - & gt; The integer + integer2); 

System. Out. Println (" Pi is roughly "+ 4.0 * count/n); 

JSC. Stop (); 
} 
} 

Error:

 
55:15 WARN Utils: Service 'SparkUI could not bind on port 4040. Attempting the port 4041. 
16/08/24 12:55:15 INFO Utils: Successfully started service 'SparkUI on port 4041. 
16/08/24 12:55:15 INFO SparkUI: Started SparkUI at http://10.3.9.155:4041 
16/08/24 12:55:15 INFO HttpFileServer: HTTP File server directory is C: \ Users \ lw_co \ AppData \ Local \ Temp \ spark - a3786942 - b296 afc6a575b6 \ HTTPD ab - 4123-99-68-6 b1db50c dbe - 6184-4-91 c5-477710823 b53 
16/08/24 12:55:15 INFO HttpServer: Starting the HTTP Server 
16/08/24 12:55:15 INFO Utils: Successfully started service 'HTTP file server on port 2115. 
16/08/24 12:55:15 INFO SparkContext: Added the JAR D: \ jiuzhouwork \ workspace \ SparkRemoteDebug \ out \ artifacts \ SparkRemoteDebug_jar \ SparkRemoteDebug JAR at http://10.3.9.155:2115/jars/SparkRemoteDebug.jar with the timestamp 1472014515970 
16/08/24 12:55:16 INFO AppClient $ClientEndpoint: Connecting to master the spark://10.3.9.135:7077... 
16/08/24 12:55:16 ERROR TransportClient: Failed to send 6246311634782781203 to/RPC 10.3.9.135:7077: Java. Lang. AbstractMethodError: org.apache.spark.net work. Protocol. MessageWithHeader. Touch (Ljava/lang/Object) Compared/netty/util/ReferenceCounted; 
Java. Lang. AbstractMethodError: org.apache.spark.net work. Protocol. MessageWithHeader. Touch (Ljava/lang/Object) Compared/netty/util/ReferenceCounted; 
At io.net, ty. Util. ReferenceCountUtil. Touch (ReferenceCountUtil. Java: 73) 
At io.net, ty. Channel. DefaultChannelPipeline. Touch (DefaultChannelPipeline. Java: 107) 
At io.net, ty. Channel. AbstractChannelHandlerContext. Write (AbstractChannelHandlerContext. Java: 796) 
At io.net, ty. Channel. AbstractChannelHandlerContext. Write (AbstractChannelHandlerContext. Java: 709) 
At io.net, ty. Handler. Codec. MessageToMessageEncoder. Write (MessageToMessageEncoder. Java: 111) 
At io.net, ty. Channel. AbstractChannelHandlerContext. InvokeWrite0 (AbstractChannelHandlerContext. Java: 724) 

Which is suitable for the local IP 10.3.9.155 is not set on a cluster of IP, for what, nan Ming set up

CodePudding user response:

Solve the building Lord, I remote server debugging, encountered the same problem

CodePudding user response:

reference 1st floor changxiangyinyue response:

solved the building Lord, I met the same problem to a remote server to debug,

No problem, recently is directly write programs, not debugging

CodePudding user response:

The building Lord solved?

CodePudding user response:

Maven such as can, pom. XML

 
<? The XML version="1.0" encoding="utf-8"?> 
XMLNS: xsi="http://www.w3.org/2001/XMLSchema-instance" 
Xsi: schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" & gt; 
 4.0.0 & lt;/modelVersion> 

 Cn. Arvidlw. Jz. Keywords 
 Keywords - news 
 1.0 - SNAPSHOT 
 
 
 Com. Hankcs 
 Hanlp 
 Portable - 1.3.1 & lt;/version> 
 
 
 Org. Apache. Hadoop 
 Hadoop - common
          

          

            
Page link：https//www.codepudding.com/other/22001.html
          
          

               
Prev:Single query mysql to query on the spark, the logic how to implement?
               
Next:AWS S3 USES the jar