时间戳

首页 > 解决方案 > 无法在 HDFS“datanode”中写入数据 - 已在排除列表中添加节点

hadoop - 无法在 HDFS“datanode”中写入数据 - 已在排除列表中添加节点

问题描述

我在同一个 jvm 中运行“namenode”和“datanode”，当我尝试写入数据时出现以下异常

org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy$NotEnoughReplicasException：在 org.apache.hadoop.hdfs.server 的 org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:836)。 blockmanagement.BlockPlacementPolicyDefault.chooseRandom(BlockPlacementPolicyDefault.java:724) 在 org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseLocalRack(BlockPlacementPolicyDefault.java:631) 在 org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault。在 org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault 的 org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTargetInOrder(BlockPlacementPolicyDefault.java:490) 选择LocalStorage(BlockPlacementPolicyDefault.java:591)。在 org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:297) 处选择目标(BlockPlacementPolicyDefault.java:421) java:148) at org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicyDefault.chooseTarget(BlockPlacementPolicyDefault.java:164) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:2127)在 org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:294) 在 org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2771) 在 org.apache .hadoop.hdfs.server.namenode.NameNodeRpcServer。addBlock(NameNodeRpcServer.java:876) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:567) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos .java) 在 org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025) 在 org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524) 在 org. apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) 在 org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822) 在 java.base/java.security.AccessController。 org.apache.hadoop.security.UserGroupInformation 的 java.base/javax.security.auth.Subject.doAs(Subject.java:423) 的 doPrivileged(Native Method)。doAs(UserGroupInformation.java:1730) 在 org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)

        final File file = new File("C:\\ManageEngine\\test\\data\\namenode");
        final File file1 = new File("C:\\ManageEngine\\test\\data\\datanode1");
        BasicConfigurator.configure();
        final HdfsConfiguration nameNodeConfiguration = new HdfsConfiguration();
        FileSystem.setDefaultUri(nameNodeConfiguration, "hdfs://localhost:5555");
        nameNodeConfiguration.set(DFSConfigKeys.DFS_NAMENODE_NAME_DIR_KEY, file.toURI().toString());
        nameNodeConfiguration.set(DFSConfigKeys.DFS_REPLICATION_KEY, "1" );
        final NameNode nameNode = new NameNode(nameNodeConfiguration);



        final HdfsConfiguration dataNodeConfiguration1 = new HdfsConfiguration();
        dataNodeConfiguration1.set(DFSConfigKeys.DFS_DATANODE_DATA_DIR_KEY, file1.toURI().toString());
        dataNodeConfiguration1.set(DFSConfigKeys.DFS_DATANODE_ADDRESS_KEY, "localhost:5556" );
        dataNodeConfiguration1.set(DFSConfigKeys.DFS_REPLICATION_KEY, "1" );
        FileSystem.setDefaultUri(dataNodeConfiguration1, "hdfs://localhost:5555");
        final DataNode dataNode1 = DataNode.instantiateDataNode(new String[]{}, dataNodeConfiguration1);



        final FileSystem fs = FileSystem.get(dataNodeConfiguration1);

        Path hdfswritepath = new Path(fileName);
        if(!fs.exists(hdfswritepath)) {
            fs.create(hdfswritepath);
            System.out.println("Path "+hdfswritepath+" created.");
        }
        System.out.println("Begin Write file into hdfs");

        FSDataOutputStream outputStream=fs.create(hdfswritepath);
        //Cassical output stream usage
        outputStream.writeBytes(fileContent);
        outputStream.close();
        System.out.println("End Write file into hdfs");

请求数据 - 图片

标签： hadoophdfs

解决方案

副本数不能高于数据节点数。

如果要在单个节点上运行，dfs.replication请在 hdfs-site.xml 中设置为 1。

推荐阅读