Home > other >  Hadoop dynamic concurrent appear to create a custom file because this file is already being created
Hadoop dynamic concurrent appear to create a custom file because this file is already being created

Time:09-25

The Configuration of the conf=context. GetConfiguration ();
FileSystem fs=FileSystem. Get (conf);
Try {
Path outputDir=new Path (/hisdata "/" + + srcTime srcTime + "/");
if(! Fs. The exists (outputDir)) {
Fs. CreateNewFile (outputDir);
}
OS=fs. Append (outputDir);
The content=content + "\ n";
Byte [] buff=content. getBytes ();
OS. Write (buff, 0, buff. Length);

} the catch (Exception e) {
e.printStackTrace();
os.close();
} the finally {
os.close();
}

Above is the code involved, considering on the map is executed in the following exception (running on the cluster) single node will not appear, if set to 1 when the specified in the reduce processing efficiency performance took a very long time, don't know how to solve, a great god taught,

The Log Type: stderr
Log the Upload Time: Wed Aug 17 02:49:16 + 0800 2016
The Log Length: 3990
Org, apache hadoop. Ipc. RemoteException (. Org. Apache hadoop. HDFS. Protocol. AlreadyBeingCreatedException) : Failed to create the file/hisdata/20150123/20150123 for [DFSClient_attempt_1470123720013_0172_m_002669_0_1142726201_1] for client [192.168.186.156], because this file is already being created by [DFSClient_attempt_1470123720013_0172_m_002671_0_1755405648_1] on [192.168.186.156]
At org, apache hadoop. HDFS. Server. The namenode. FSNamesystem. RecoverLeaseInternal (FSNamesystem. Java: 3122)
At org, apache hadoop. HDFS. Server. The namenode. FSNamesystem. AppendFileInternal (FSNamesystem. Java: 2905)
At org, apache hadoop. HDFS. Server. The namenode. FSNamesystem. AppendFileInt (FSNamesystem. Java: 3186)
At org, apache hadoop. HDFS. Server. The namenode. FSNamesystem. AppendFile (FSNamesystem. Java: 3149)
At org, apache hadoop. HDFS. Server. The namenode. NameNodeRpcServer. Append (NameNodeRpcServer. Java: 611)
At org, apache hadoop. HDFS. Server. The namenode. AuthorizationProviderProxyClientProtocol. Append (AuthorizationProviderProxyClientProtocol. Java: 124)
At org, apache hadoop. HDFS. ProtocolPB. ClientNamenodeProtocolServerSideTranslatorPB. Append (ClientNamenodeProtocolServerSideTranslatorPB. Java: 416)
At org, apache hadoop. HDFS. Protocol. The proto. ClientNamenodeProtocolProtos $ClientNamenodeProtocol $2. CallBlockingMethod (ClientNamenodeProtocolProtos. Java)
At org, apache hadoop. Ipc. ProtobufRpcEngine $$ProtoBufRpcInvoker Server. Call (617) ProtobufRpcEngine. Java:
At org, apache hadoop. Ipc. RPC $Server. The call (RPC) Java: 1060)
At org, apache hadoop. Ipc. Server $Handler $1. The run (2086) Server. Java:
At org, apache hadoop. Ipc. Server $Handler $1. The run (2082) Server. Java:
The at Java. Security. The AccessController. DoPrivileged (Native Method)
At javax.mail. Security. Auth. Subject. The doAs (422) Subject. Java:
At org, apache hadoop. Security. UserGroupInformation. DoAs (UserGroupInformation. Java: 1671)
At org, apache hadoop. Ipc. Server $Handler. The run (2080) Server. Java:

At org, apache hadoop. Ipc. Client. Call (1472). The Client Java:
At org, apache hadoop. Ipc. Client. Call (1403). The Client Java:
At org, apache hadoop. Ipc. ProtobufRpcEngine $Invoker. Invoke (ProtobufRpcEngine. Java: 230)
At com. Sun. Proxy. $Proxy13. Append (Unknown Source)
At org, apache hadoop. HDFS. ProtocolPB. ClientNamenodeProtocolTranslatorPB. Append (ClientNamenodeProtocolTranslatorPB. Java: 313)
At sun. Reflect. NativeMethodAccessorImpl. Invoke0 (Native Method)
At sun. Reflect. NativeMethodAccessorImpl. Invoke (NativeMethodAccessorImpl. Java: 62)
At sun. Reflect. DelegatingMethodAccessorImpl. Invoke (43) DelegatingMethodAccessorImpl. Java:
The at Java. Lang. Reflect. Method. Invoke (497) Method. The Java:
At org, apache hadoop. IO. Retry. RetryInvocationHandler. InvokeMethod (RetryInvocationHandler. Java: 252)
At org, apache hadoop. IO. Retry. RetryInvocationHandler. Invoke (RetryInvocationHandler. Java: 104)
At com. Sun. Proxy. $Proxy14. Append (Unknown Source)
At org, apache hadoop. HDFS. DFSClient. CallAppend (DFSClient. Java: 1842)
At org, apache hadoop. HDFS. DFSClient. Append (DFSClient. Java: 1878)
At org, apache hadoop. HDFS. DFSClient. Append (DFSClient. Java: 1871)
At org, apache hadoop. HDFS. DistributedFileSystem $4. DoCall (DistributedFileSystem. Java: 329)
At org, apache hadoop. HDFS. DistributedFileSystem $4. DoCall (DistributedFileSystem. Java: 325)
At org, apache hadoop. Fs. FileSystemLinkResolver. Resolve (FileSystemLinkResolver. Java: 81)
At org, apache hadoop. HDFS. DistributedFileSystem. Append (DistributedFileSystem. Java: 325)
At org, apache hadoop. Fs. FileSystem. Append (1172) FileSystem. Java:
At com. Ultrapower. Secsight. Etl. HisLogToJson $LogMapper. The map (119) HisLogToJson. Java:
At org, apache hadoop. Graphs. Mapper. The run (145) Mapper. Java:
At org, apache hadoop. Mapred. MapTask. RunNewMapper (MapTask. Java: 787)
At org, apache hadoop. Mapred. MapTask. Run (341) MapTask. Java:
At org, apache hadoop. Mapred. YarnChild $2. The run (163) YarnChild. Java:
The at Java. Security. The AccessController. DoPrivileged (Native Method)
At javax.mail. Security. Auth. Subject. The doAs (422) Subject. Java:
At org, apache hadoop. Security. UserGroupInformation. DoAs (UserGroupInformation. Java: 1671)
At org, apache hadoop. Mapred. YarnChild. Main (YarnChild. Java: 158)
  • Related