java - 容器应要求被杀死。退出代码是 143 java.io.IOException: Job failed

Question

$HADOOP_HOME/bin/hadoop jar ProductSalePerCountry.jar /inputMapReduce /mapreduce_output_sales

19/06/11 20:47:01 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
19/06/11 20:47:02 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
19/06/11 20:47:02 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
19/06/11 20:47:02 INFO mapred.FileInputFormat: Total input files to process : 1
19/06/11 20:47:02 WARN hdfs.DataStreamer: Caught exception
java.lang.InterruptedException
    at java.lang.Object.wait(Native Method)
    at java.lang.Thread.join(Thread.java:1252)
    at java.lang.Thread.join(Thread.java:1326)
    at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:973)
    at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:624)
    at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:801)
19/06/11 20:47:02 INFO mapreduce.JobSubmitter: number of splits:2
19/06/11 20:47:03 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1560285969214_0001
19/06/11 20:47:03 INFO impl.YarnClientImpl: Submitted application application_1560285969214_0001
19/06/11 20:47:03 INFO mapreduce.Job: The url to track the job: http://hadoop1:8088/proxy/application_1560285969214_0001/
19/06/11 20:47:03 INFO mapreduce.Job: Running job: job_1560285969214_0001
19/06/11 20:47:18 INFO mapreduce.Job: Job job_1560285969214_0001 running in uber mode : false
19/06/11 20:47:18 INFO mapreduce.Job:  map 0% reduce 0%
19/06/11 20:47:18 INFO mapreduce.Job: Job job_1560285969214_0001 failed with state FAILED due to: Application application_1560285969214_0001 failed 2 times due to AM Container for appattempt_1560285969214_0001_000002 exited with  exitCode: -103
Failing this attempt.Diagnostics: Container [pid=19902,containerID=container_1560285969214_0001_02_000001] is running beyond virtual memory limits. Current usage: 116.7 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1560285969214_0001_02_000001 :
    |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
    |- 19910 19902 19902 19902 (java) 439 19 2793091072 29136 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Djava.io.tmpdir=/app/hadoop/tmp/nm-local-dir/usercache/hduser/appcache/application_1560285969214_0001/container_1560285969214_0001_02_000001/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/hadoop/logs/userlogs/application_1560285969214_0001/container_1560285969214_0001_02_000001 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog -Xmx1024m org.apache.hadoop.mapreduce.v2.app.MRAppMaster 
    |- 19902 19900 19902 19902 (bash) 0 0 11538432 745 /bin/bash -c /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Djava.io.tmpdir=/app/hadoop/tmp/nm-local-dir/usercache/hduser/appcache/application_1560285969214_0001/container_1560285969214_0001_02_000001/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/usr/local/hadoop/logs/userlogs/application_1560285969214_0001/container_1560285969214_0001_02_000001 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog  -Xmx1024m org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1>/usr/local/hadoop/logs/userlogs/application_1560285969214_0001/container_1560285969214_0001_02_000001/stdout 2>/usr/local/hadoop/logs/userlogs/application_1560285969214_0001/container_1560285969214_0001_02_000001/stderr  

Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
For more detailed output, check the application tracking page: http://hadoop1:8088/cluster/app/application_1560285969214_0001 Then click on links to logs of each attempt.
. Failing the application.
19/06/11 20:47:18 INFO mapreduce.Job: Counters: 0
java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:873)
    at SalesCountry.SalesCountryDriver.main(SalesCountryDriver.java:38)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:234)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:148)

纱线站点.xml

<?xml version="1.0"?>
<configuration>
<property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
</property>

<property>
    <name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
    <value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>

hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
    <name>dfs.replication</name>
    <value>4</value>
    <description>Duplicate Data on slave nodes.</description>
</property>
</configuration>

mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
    <name>mapred.jobtracker.address</name>
    <value>node-master:54311</value>
    <description>The host and port that the MapReduce job tracker runs at. If "$
</property>
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>
<property>
    <name>yarn.app.mapreduce.am.resource.mb</name>
    <value>512</value>
</property>
<property>
    <name>mapreduce.map.memory.mb</name>
    <value>256</value>
</property>
<property>
    <name>mapreduce.reduce.memory.mb</name>
    <value>256</value>
</property>
</configuration>

核心站点.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
    <name>hadoop.tmp.dir</name>
    <value>/app/hadoop/tmp</value>
    <description>Parent dir for tmp dirs.</description>
</property>
<property>
    <name>fs.defaultFS </name>
    <value>hdfs://hadoop1:54310</value>
    <description>The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem.</description>
</property>
</configuration>

奴隶

hadoop1
hadoop2
hadoop3
hadoop4

/etc/hosts

127.0.0.1       localhost
192.52.33.99    hadoop4
192.52.33.82    hadoop3
192.52.34.114   hadoop2
192.52.34.131   hadoop1

~/.bashrc

export HADOOP_HOME=/usr/local/hadoop
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL

示例程序 https://www.guru99.com/create-your-first-hadoop-program.html

对不起，我不知道是什么导致了错误。也许有多个错误。

配置有什么问题。任务是在教程中使用一主三从（集群）代替教程，而不是仅使用一个节点。

标签： javahadoop