#! The/usr/bin/env bash
Export SPARK_WORKER_MEMORY=3000 m
# the native IP
SPARK_LOCAL_IP=10.1.5.172
JAVA_HOME=/JAVA/jdk1.8.0 _181
SPARK_MASTER_HOST=10.1.5.172
To establish the development environment on Windows, use is the IDEA, configure the development environment, writing code, the build works, content as follows:
The import org. Apache. Spark. {SparkConf, SparkContext}
The object Main {
Def main (args: Array [String]) : Unit={
Var masterUrl="spark://10.1.5.172:7077
Var inputPath="D: \ \ \ \ SparkWork \ \ helloSpark TXT"
Val conf=new SparkConf ()
. SetAppName (" myProject1 ")
. SetMaster (masterUrl)
Set (" spark. Cores. Max ", "2")
Set (" spark. Executor. The memory ", "512 m")
Val sc=new SparkContext (conf)
Val textFile=sc. TextFile (inputPath)
Println (textFile. The count ())
}
}
Submitted to the server computing:
In the IDEA of the following, consistent failure to calculate complete
The 2018-11-07 16:40:46 INFO StandaloneAppClient $ClientEndpoint: 54 - Executor updated: app - 20181107164005-0001/23 is now RUNNING
The 2018-11-07 16:40:46 INFO CoarseGrainedSchedulerBackend $DriverEndpoint: 54 - Asked to remove non - existent executor 22
The 2018-11-07 16:40:48 INFO StandaloneAppClient $ClientEndpoint: 54 - Executor updated: app - 20181107164005-0001/23 is now EXITED (Command EXITED with code 1)
The 2018-11-07 16:40:48 INFO StandaloneSchedulerBackend: 0001/23 - Executor app - 20181107164005-54 removed: Command exited with code 1
The 2018-11-07 16:40:48 INFO StandaloneAppClient $ClientEndpoint: 54 - Executor added: app on worker - 20181107155456-20181107164005-0001/24-10.1.5.172-33496 (33496) 10.1.5.172: with 2 core (s)
The 2018-11-07 16:40:48 INFO BlockManagerMaster: 54 - Removal of executor 23 requested
The 2018-11-07 16:40:48 INFO CoarseGrainedSchedulerBackend $DriverEndpoint: 54 - Asked to remove non - existent executor 23
The 2018-11-07 16:40:48 INFO StandaloneSchedulerBackend: 54 - Granted executor ID app - 20181107164005-0001/24 on hostPort 10.1.5.172:33496 with 2 core (s), 512.0 MB RAM
The 2018-11-07 16:40:48 INFO BlockManagerMasterEndpoint: 54 - Trying to remove executor of 23 from BlockManagerMaster.
The 2018-11-07 16:40:48 INFO StandaloneAppClient $ClientEndpoint: 54 - Executor updated: app - 20181107164005-0001/24 is now RUNNING
Infinite loop,
On the spark UI
Running Applications (1)
Application ID Name Cores Memory per Executor Submitted Time User State Duration
App - 20181107164005-0001 (kill) myProject1 2 512.0 MB 2018/11/07 16:40:05 user RUNNING 1.2 min
The task is as follows:
Executor Summary (79)
ExecutorID Worker Cores State Memory Logs
Worker - 20181107155456-10.1.5.172-33496 2 512 78 RUNNING stdout stderr
14 the worker - 20181107155456-10.1.5.172-33496 2 512 EXITED stdout stderr
The worker - 20181107155456-47 10.1.5.172-33496-2, 512 EXITED stdout stderr
The worker - 20181107155456-25 10.1.5.172-33496-2, 512 EXITED stdout stderr
Stderr error is as follows:
The Exception in the thread "main" Java. Lang. Reflect. UndeclaredThrowableException
At org, apache hadoop. Security. UserGroupInformation. DoAs (UserGroupInformation. Java: 1713)
The at org. Apache. Spark. Deploy. SparkHadoopUtil. RunAsSparkUser (SparkHadoopUtil. Scala: 63)
At org. Apache. Spark. Executor. CoarseGrainedExecutorBackend $. The run (188). CoarseGrainedExecutorBackend scala:
The at org. Apache. Spark. Executor. CoarseGrainedExecutorBackend $. The main (CoarseGrainedExecutorBackend. Scala: 293)
The at org. Apache. Spark. Executor. CoarseGrainedExecutorBackend. Main (CoarseGrainedExecutorBackend. Scala)
Under Caused by: org. Apache. Spark. SparkException: Exception thrown in awaitResult:
The at org. Apache. Spark. Util. ThreadUtils $. AwaitResult (ThreadUtils. Scala: 205)
The at org. Apache. Spark. RPC. RpcTimeout. AwaitResult (RpcTimeout. Scala: 75)
The at org. Apache. Spark. RPC. RpcEnv. SetupEndpointRefByURI (RpcEnv. Scala: 101)
The at org. Apache. Spark. Executor. CoarseGrainedExecutorBackend $$anonfun $run $1. Apply $MCV $sp (CoarseGrainedExecutorBackend. Scala: 201)
At org. Apache. Spark. Deploy. SparkHadoopUtil $$$2. -anon run (64). SparkHadoopUtil scala:
At org. Apache. Spark. Deploy. SparkHadoopUtil $$$2. -anon run (63). SparkHadoopUtil scala:
The at Java. Security. The AccessController. DoPrivileged (Native Method)
At javax.mail. Security. Auth. Subject. The doAs (422) Subject. Java:
At org, apache hadoop. Security. UserGroupInformation. DoAs (UserGroupInformation. Java: 1698)
. 4 more
Under Caused by: Java. IO. IOException: Failed to connect to az -/192.168.30.59 TDR - 1:61237
At org.apache.spark.net work. Client. TransportClientFactory. CreateClient (TransportClientFactory. Java: 245)
At org.apache.spark.net work. Client. TransportClientFactory. CreateClient (TransportClientFactory. Java: 187)
The at org.apache.spark.rpc.netty.Net tyRpcEnv. CreateClient (NettyRpcEnv. Scala: 198)
At org.apache.spark.rpc.net. Ty Outbox $$$1. -anon call (194) Outbox. Scala:
At org.apache.spark.rpc.net. Ty Outbox $$$1. -anon call (190) Outbox. Scala:
The at Java. Util. Concurrent. FutureTask. Run (FutureTask. Java: 266)
nullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnull