python - 无法启动 MapReduce 作业

Question

当我尝试开始简单的 wordcount mapreduce 示例时，我得到了这个

[cloudera-scm@ivana-namenode2 /opt/MapReduce/wordcount]$ pydoop script    wc.py /user/cloudera-scm/Mapduce/wordcount/data/text /user/cloudera-scm/MapReduce/wordcount/output
18/08/14 14:11:49 INFO Configuration.deprecation: mapred.cache.files is deprecated. Instead, use mapreduce.job.cache.files
18/08/14 14:11:49 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
18/08/14 14:11:49 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
18/08/14 14:11:49 INFO Configuration.deprecation: hadoop.pipes.java.recordreader is deprecated. Instead, use mapreduce.pipes.isjavarecordreader
18/08/14 14:11:49 INFO Configuration.deprecation: mapred.cache.archives is deprecated. Instead, use mapreduce.job.cache.archives
18/08/14 14:11:49 INFO Configuration.deprecation: hadoop.pipes.java.recordwriter is deprecated. Instead, use mapreduce.pipes.isjavarecordwriter
18/08/14 14:11:49 INFO Configuration.deprecation: mapred.textoutputformat.separator is deprecated. Instead, use mapreduce.output.textoutputformat.separator
18/08/14 14:11:49 INFO Configuration.deprecation: mapred.compress.map.output is deprecated. Instead, use mapreduce.map.output.compress
18/08/14 14:11:50 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm59
18/08/14 14:11:51 WARN mapreduce.JobResourceUploader: No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
18/08/14 14:11:51 INFO input.FileInputFormat: Total input paths to process : 1
18/08/14 14:11:52 INFO mapreduce.JobSubmitter: number of splits:1
18/08/14 14:11:52 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1534248543348_0007
18/08/14 14:11:52 INFO mapred.YARNRunner: Job jar is not present. Not adding any jar to the list of resources.
18/08/14 14:11:52 INFO impl.YarnClientImpl: Submitted application application_1534248543348_0007
18/08/14 14:11:52 INFO mapreduce.Job: The url to track the job: http://ivana-namenode2:8088/proxy/application_1534248543348_0007/
18/08/14 14:11:52 INFO mapreduce.Job: Running job: job_1534248543348_0007
18/08/14 14:15:07 INFO mapreduce.Job: Job job_1534248543348_0007 running in uber mode : false
18/08/14 14:15:07 INFO mapreduce.Job:  map 0% reduce 0%
18/08/14 14:15:07 INFO mapreduce.Job: Job job_1534248543348_0007 failed with state KILLED due to: Application killed by user.
18/08/14 14:15:07 INFO mapreduce.Job: Counters: 0
18/08/14 14:15:07 INFO util.ExitUtil: Exiting with status 1
Traceback (most recent call last):
File "/usr/local/bin/pydoop", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/site-packages/pydoop/app/main.py", line 81, in main
args.func(args, unknown)
File "/usr/local/lib/python2.7/site-packages/pydoop/app/script.py", line 130, in run
scripter.run()
File "/usr/local/lib/python2.7/site-packages/pydoop/app/script.py", line 119, in run
submitter.run()
File "/usr/local/lib/python2.7/site-packages/pydoop/app/submit.py", line 376, in run
logger=self.logger, keep_streams=False)
File "/usr/local/lib/python2.7/site-packages/pydoop/hadut.py", line 302, in run_class
keep_streams=keep_streams)
File "/usr/local/lib/python2.7/site-packages/pydoop/hadut.py", line 167, in run_cmd
raise RunCmdError(p.returncode, ' '.join(_args), error)
pydoop.hadut.RunCmdError: command exited with 1 status

此外，当我尝试启动 jar 示例时，我遇到了同样的问题，所以这与代码无关。

下面，您有指向我的 main.py、script.py 和 submit.py 的链接。我不能把它们放在这里，因为堆栈溢出限制最多 30000 个字符

https://www.dropbox.com/sh/1mabmgfywa38xib/AAB946SUK8-A8VfWfieQG_4ha?dl=0

标签： pythonhadoopmapreducecloudera