java - Hadoop API 在尝试初始化集群以使用 DistCp 时抛出错误
问题描述
我正在尝试使用 Hadoop API 使用 DistCp 类进行分布式复制,但在尝试连接到集群时会引发错误。我曾尝试更改 hadoop 和 hdfs 的配置文件,但似乎不起作用。我正在最新的 Cloudera 快速入门上测试应用程序
我运行这个命令来执行这个类。java -cp myjar com.keedio.hadoop.Mover
包 com.keedio.hadoop;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.tools.DistCp;
import org.apache.hadoop.tools.DistCpOptionSwitch;
import org.apache.hadoop.tools.DistCpOptions;
import org.apache.hadoop.util.ToolRunner;
import java.util.Collections;
public class Mover {
public static void main(String args []) {
try {
FileSystem fs = FileSystem.getLocal(new Configuration());
FileSystem fs2= FileSystem.get(java.net.URI.create("file:///"),new
Configuration());
DistCpOptions distCpOptions=new DistCpOptions(new Path("file:/Users/jvasquez/Desktop/ficheros1"),new Path("file:/Users/jvasquez/Desktop/ficheros2"));
String argumentos [] = {"file:////Users/jvasquez/Desktop/ficheros1","file:///Users/jvasquez/Desktop/ficheros2"};
Configuration conf=new Configuration();
conf.addResource(new org.apache.hadoop.fs.Path("/etc/hadoop/conf/yarn-site.xml"));
conf.addResource(new org.apache.hadoop.fs.Path("/etc/hadoop/conf/core-site.xml"));
conf.addResource(new org.apache.hadoop.fs.Path("/etc/hadoop/conf/mapred-site.xml"));
conf.addResource(new Path("/etc/hadoop/conf/hdfs-site.xml"));
DistCp distCp= new DistCp(conf,distCpOptions);
ToolRunner.run(distCp,argumentos);
}
catch (Exception e) {
e.printStackTrace();
}
}
}
这是错误
log4j:WARN No appenders could be found for logger
(org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:143)
at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:108)
at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:101)
at org.apache.hadoop.tools.DistCp.createMetaFolderPath(DistCp.java:471)
at org.apache.hadoop.tools.DistCp.<init>(DistCp.java:107)
at com.keedio.hadoop.Mover.main(Mover.java:48)
解决方案
最后我解决了它,结果是我只需要使用命令 hadoop jar 将应用程序作为 mapreduce 应用程序启动。
推荐阅读
- laravel - 在 laravel 中管理多选搜索显示
- android - 在简单 Flutter 应用程序中使用 GridViews 的问题
- intern - 在使用 Intern 运行浏览器测试之前,如何在 Node 中运行一些代码?
- wordpress - My website button works in Chrome but not in Firefox
- python - Is there an efficient way of solving sparse linear equations in Tensorflow that is compatible with gradient tape?
- javascript - 如何通过单击按钮更改添加元素的顺序?
- maximo - Launch automation script prior to WO being closed?
- python - ctypes.cdll.LoadLibrary on a dll in a zipped directory
- linux - 如何在 shell 脚本中获取 python 版本?
- java - 为什么这两种获取布尔值的方法会有不同的结果?