Everyone a great god is good, recently tried to use the spark on yarn model access another enabled kerberos hadoop cluster on the data, the program executes on a cluster is a user of negotiable instruments, the local mode executable program is to be able to access, but specify the -- master yarn, whether the client mode or cluster mode has the following error, bitter fruit, found on the Internet have to come for help:
WARN the scheduler. TaskSetManager: Lost task in stage 0.0 0.0 (dar 0, stbdd001, executor (1) : Java. IO. IOException: Failed on local exception: Java. IO. IOException: org.. Apache hadoop. Security. AccessControlException: Client always authenticate via: [TOKEN, KERBEROS]; The Host Details: the local Host is: "stbddxx1/XXX. Xx. XXX. XXX". The destination host is: "dmbxx2" : 8020;
At org.apache.hadoop.net.NetUtils.wrapException NetUtils. Java: (772)
At org, apache hadoop. Ipc. Client. Call (1476). The Client Java:
At org, apache hadoop. Ipc. Client. Call (1409). The Client Java:
At org, apache hadoop. Ipc. ProtobufRpcEngine $Invoker. Invoke (ProtobufRpcEngine. Java: 230)
At com. Sun. Proxy. $Proxy23. GetBlockLocations (Unknown Source)
At org, apache hadoop. HDFS. ProtocolPB. ClientNamenodeProtocolTranslatorPB. GetBlockLocations (ClientNamenodeProtocolTranslatorPB. Java: 256)
At sun. Reflect. NativeMethodAccessorImpl. Invoke0 (Native Method)
At sun. Reflect. NativeMethodAccessorImpl. Invoke (NativeMethodAccessorImpl. Java: 57)
At sun. Reflect. DelegatingMethodAccessorImpl. Invoke (43) DelegatingMethodAccessorImpl. Java:
The at Java. Lang. Reflect. Method. Invoke (606) Method. The Java:
At org, apache hadoop. IO. Retry. RetryInvocationHandler. InvokeMethod (RetryInvocationHandler. Java: 256)
At org, apache hadoop. IO. Retry. RetryInvocationHandler. Invoke (RetryInvocationHandler. Java: 104)
At com. Sun. Proxy. $Proxy24. GetBlockLocations (Unknown Source)
At org, apache hadoop. HDFS. DFSClient. CallGetBlockLocations (DFSClient. Java: 1279)
At org, apache hadoop. HDFS. DFSClient. GetLocatedBlocks (DFSClient. Java: 1266)
At org, apache hadoop. HDFS. DFSClient. GetLocatedBlocks (DFSClient. Java: 1254)
At org, apache hadoop. HDFS. DFSInputStream. FetchLocatedBlocksAndGetLastBlockLength (DFSInputStream. Java: 305)
At org, apache hadoop. HDFS. DFSInputStream. OpenInfo (DFSInputStream. Java: 271)
At org, apache hadoop. HDFS. DFSInputStream. & lt; init> (DFSInputStream. Java: 263)
At org, apache hadoop. HDFS. DFSClient. Open (1585) DFSClient. Java:
At org, apache hadoop. HDFS. DistributedFileSystem $4. DoCall (DistributedFileSystem. Java: 326)
At org, apache hadoop. HDFS. DistributedFileSystem $4. DoCall (DistributedFileSystem. Java: 322)
At org, apache hadoop. Fs. FileSystemLinkResolver. Resolve (FileSystemLinkResolver. Java: 81)
At org, apache hadoop. HDFS. DistributedFileSystem. Open (322) DistributedFileSystem. Java:
At org, apache hadoop. Fs. FileSystem. Open (783) FileSystem. Java:
At org, apache hadoop. Mapred. LineRecordReader. & lt; init> (LineRecordReader. Java: 109)
At org, apache hadoop. Mapred. TextInputFormat. GetRecordReader (TextInputFormat. Java: 67)
At org. Apache. Spark. RDD. HadoopRDD $$$1. -anon & lt; init> (HadoopRDD. Scala: 240)
At org.apache.spark.rdd.HadoopRDD.com pute (HadoopRDD. Scala: 211)
At org.apache.spark.rdd.HadoopRDD.com pute (HadoopRDD. Scala: 101)
At org.apache.spark.rdd.RDD.com puteOrReadCheckpoint (RDD. Scala: 306)
The at org. Apache. Spark. RDD. RDD. Iterator (RDD. Scala: 270)
At org.apache.spark.rdd.MapPartitionsRDD.com pute (MapPartitionsRDD. Scala: 38)
At org.apache.spark.rdd.RDD.com puteOrReadCheckpoint (RDD. Scala: 306)
The at org. Apache. Spark. RDD. RDD. Iterator (RDD. Scala: 270)
The at org. Apache. Spark. The scheduler. ResultTask. RunTask (ResultTask. Scala: 66)
At org. Apache. Spark. The scheduler. Task. Run (89) Task. Scala:
The at org. Apache. Spark. Executor. $TaskRunner executor. Run (executor. Scala: 242)
The at Java. Util. Concurrent. ThreadPoolExecutor. RunWorker (ThreadPoolExecutor. Java: 1145)
The at Java. Util. Concurrent. ThreadPoolExecutor $Worker. The run (ThreadPoolExecutor. Java: 615)
The at Java. Lang. Thread. The run (Thread. Java: 745)
Under Caused by: Java. IO. IOException: org.. Apache hadoop. Security. AccessControlException: Client always authenticate via: [TOKEN, KERBEROS]
At org, apache hadoop. Ipc. Client $Connection $1. The run (688). The Client Java:
The at Java. Security. The AccessController. DoPrivileged (Native Method)
At javax.mail. Security. Auth. Subject. The doAs (415) Subject. Java:
At org, apache hadoop. Security. UserGroupInformation. DoAs (UserGroupInformation. Java: 1920)
At org, apache hadoop. Ipc. Client $Connection. HandleSaslConnectionFailure (651). The Client Java:
At org, apache hadoop. Ipc. Client $Connection. SetupIOstreams (739). The Client Java:
At org, apache hadoop. Ipc. Client $Connection. The access of $2900 (376). The Client Java:
At org, apache hadoop. Ipc. Client. GetConnection (1525). The Client Java:
At org, apache hadoop. Ipc. Client. Call (1448). The Client Java:
. 39 more

CodePudding user response:

@ zhong930

CodePudding user response:

Can help top,,, don't use KERBEROS

CodePudding user response:

Version 1.6 adds sparkconf. Set (" spark. Yarn. Access. Namenodes ", "HDFS://the namenode")
2. Add sparkconf x version. The set (" spark. Yarn. Access. HadoopFileSystems ", "HDFS://the namenode")nullnullnullnullnullnullnull