Home > other >  Spark when read HDFS IO exception
Spark when read HDFS IO exception

Time:09-18

Now, I only deployed a node to test, but found that cannot read files in HDFS, throw the IO exception, every time for the great god save
Nero01 for host name, IP address 192.168.189.101 for,


Scala> TextFile. Count
Java. IO. IOException: Failed on local exception: com. Google. Protobuf. InvalidProtocolBufferException: Message missing required fields: callId, status; The Host Details: the local Host is: "nero01/192.168.189.101"; The destination host is: "nero01" : 9000;
At org.apache.hadoop.net.NetUtils.wrapException NetUtils. Java: (760)
At org, apache hadoop. Ipc. Client. Call (1229). The Client Java:
At org, apache hadoop. Ipc. ProtobufRpcEngine $Invoker. Invoke (ProtobufRpcEngine. Java: 202)
At sun. Proxy. $Proxy16. GetFileInfo (Unknown Source)
At sun. Reflect. NativeMethodAccessorImpl. Invoke0 (Native Method)
At sun. Reflect. NativeMethodAccessorImpl. Invoke (NativeMethodAccessorImpl. Java: 39)
At sun. Reflect. DelegatingMethodAccessorImpl. Invoke (DelegatingMethodAccessorImpl. Java: 25)
The at Java. Lang. Reflect. Method. Invoke (597) Method. The Java:
At org, apache hadoop. IO. Retry. RetryInvocationHandler. InvokeMethod (RetryInvocationHandler. Java: 164)
At org, apache hadoop. IO. Retry. RetryInvocationHandler. Invoke (RetryInvocationHandler. Java: 83)
At sun. Proxy. $Proxy16. GetFileInfo (Unknown Source)
At org, apache hadoop. HDFS. ProtocolPB. ClientNamenodeProtocolTranslatorPB. GetFileInfo (ClientNamenodeProtocolTranslatorPB. Java: 628)
At org, apache hadoop. HDFS. DFSClient. GetFileInfo (DFSClient. Java: 1532)
At org, apache hadoop. HDFS. DistributedFileSystem. GetFileStatus (DistributedFileSystem. Java: 803)
At org, apache hadoop. Fs. FileSystem. GlobStatusInternal (1635) FileSystem. Java:
At org, apache hadoop. Fs. FileSystem. GlobStatus (1581) FileSystem. Java:
At org, apache hadoop. Mapred. FileInputFormat. ListStatus (FileInputFormat. Java: 174)
At org, apache hadoop. Mapred. FileInputFormat. GetSplits (FileInputFormat. Java: 205)
The at org. Apache. Spark. RDD. HadoopRDD. GetPartitions (HadoopRDD. Scala: 201)
The at org. Apache. Spark. RDD. RDD $$$partitions anonfun $2. Apply (RDD. Scala: 205)
The at org. Apache. Spark. RDD. RDD $$$partitions anonfun $2. Apply (RDD. Scala: 203)
At the scala. Option. GetOrElse (120) Option. The scala:
The at org. Apache. Spark. RDD. RDD. Partitions (RDD. Scala: 203)
The at org. Apache. Spark. RDD. MappedRDD. GetPartitions (28) MappedRDD. Scala:
The at org. Apache. Spark. RDD. RDD $$$partitions anonfun $2. Apply (RDD. Scala: 205)
The at org. Apache. Spark. RDD. RDD $$$partitions anonfun $2. Apply (RDD. Scala: 203)
At the scala. Option. GetOrElse (120) Option. The scala:
The at org. Apache. Spark. RDD. RDD. Partitions (RDD. Scala: 203)
The at org. Apache. Spark. SparkContext. RunJob (SparkContext. Scala: 1328)
The at org. Apache. Spark. RDD. RDD. Count (RDD. Scala: 910)
The at $iwC $$iwC $$iwC $$iwC. & lt; init> (& lt; Console> :
15)The at $iwC $$iwC $$iwC. & lt; init> (& lt; Console> : 20)
The at $iwC $$iwC. & lt; init> (& lt; Console> : 22)
The at $iwC. & lt; init> (& lt; Console> : 24)
The at & lt; init> (& lt; Console> : 26)
Palawan & lt; init> (& lt; Console> : 30)
Palawan & lt; Clinit> (& lt; Console>)
Palawan & lt; init> (& lt; Console> : 7)
Palawan & lt; Clinit> (& lt; Console>)
The at $print (& lt; Console>)
At sun. Reflect. NativeMethodAccessorImpl. Invoke0 (Native Method)
At sun. Reflect. NativeMethodAccessorImpl. Invoke (NativeMethodAccessorImpl. Java: 39)
At sun. Reflect. DelegatingMethodAccessorImpl. Invoke (DelegatingMethodAccessorImpl. Java: 25)
The at Java. Lang. Reflect. Method. Invoke (597) Method. The Java:
The at org. Apache. Spark. Repl. SparkIMain $ReadEvalPrint. Call (852). SparkIMain scala:
The at org. Apache. Spark. Repl. SparkIMain $Request. LoadAndRun (SparkIMain. Scala: 1125)
The at org. Apache. Spark. Repl. SparkIMain. LoadAndRunReq $1 (674). SparkIMain scala:
The at org. Apache. Spark. Repl. SparkIMain. Interpret (SparkIMain. Scala: 705)
The at org. Apache. Spark. Repl. SparkIMain. Interpret (SparkIMain. Scala: 669)
The at org. Apache. Spark. Repl. SparkILoop. ReallyInterpret $1 (828). SparkILoop scala:
The at org. Apache. Spark. Repl. SparkILoop. InterpretStartingWith (SparkILoop. Scala: 873)
Mand at org.apache.spark.repl.SparkILoop.com (SparkILoop. Scala: 785)
The at org. Apache. Spark. Repl. SparkILoop. ProcessLine $1 (628). SparkILoop scala:
The at org. Apache. Spark. Repl. SparkILoop. InnerLoop $1 (636). SparkILoop scala:
At org. Apache. Spark. Repl. SparkILoop. Loop (641). SparkILoop scala:
At org. Apache. Spark. Repl. SparkILoop $$anonfun $process $1. Apply $MCZ $sp (SparkILoop. Scala: 968)
The at org. Apache. Spark. Repl. SparkILoop $$anonfun $process $1. Apply (SparkILoop. Scala: 916)
The at org. Apache. Spark. Repl. SparkILoop $$anonfun $process $1. Apply (SparkILoop. Scala: 916)
At the scala. Tools. NSC. Util. ScalaClassLoader $. SavingContextLoader (ScalaClassLoader. Scala: 135)
The at org. Apache. Spark. Repl. SparkILoop. Process (SparkILoop. Scala: 916)
The at org. Apache. Spark. Repl. SparkILoop. Process (SparkILoop. Scala: 1011)
The at org. Apache. Spark. Repl. Main $. The Main (31). The Main scala:
The at org. Apache. Spark. Repl. Main. The Main (Main. Scala)
At sun. Reflect. NativeMethodAccessorImpl. Invoke0 (Native Method)
nullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnull
  • Related