Hello
I use s3distcp to EMR graphs of the output file from HDFS to S3, I in my separate EMR cluster running in the MR and s3distcp as two steps,
When I use the s3distcp from HDFS copy dat file to S3, I got the following error:
Container pid=8994, containerID=container_1395749857988_0002_01_000009 is running beyond virtual memory limits. The Current usage is: 735.9 MB of 1.5 GB of physical memory, informs; 7.5 GB of 7.5 GB virtual memory 2. This time the container.
The Dump of the process - tree for container_1395749857988_0002_01_000009:
My configuration is as follows:
The master - m1. Large
The core - m1. Large
Task - m1. Medium
The wrong details, please refer to the attachment,
Thank you for your help

CodePudding user response:

Hello, you use what version of Hadoop? What is the AMI ID?
Can you tell us your cluster ID, so that we can check your log files, and see where the problem is,
This problem may be caused by the heap size, usually can through the following ways to solve:
http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/HadoopMemoryIntensive.html
Please let us know
Thank you

CodePudding user response:

Thanks for your reply,
The following is my configuration details
Hadoop: Amazon 2.2.0
AMI version: 3.0.3
Cluster ID: j - 20 y1jrrwxzwlw
Please tell me which need to be amended?
Thank you,

CodePudding user response:

Hello
I read your error log, may be the problem of memory,
Container pid=985, containerID=container_1396077109868_0002_01_000009 over the virtual memory limit the size of the current usage is: 1.5 GB of physical memory used 757.7 MB; 7.5 GB of virtual memory used 7.5 GB, as a result, the process of the container were killed,
You can take a look at this log:
http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-manage-view-web-log-files.html
According to the previous advice, please follow the following connection methods to configure your work heap size, in this connection the next page of a specific method,
http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/HadoopMemoryIntensive.html