hadoop - 如何减少 Hadoop 在 datanode 上的 inode 使用量？

Question

我们在 Kubernetes 上使用 Hadoop。在 datanode pod 上，我们收到如下错误：

2021-08-31 10:22:16,154 WARN org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to place enough replicas, still in need of 1 to reach 1 (unavailableStorages=[], storagePolicy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}, newBlock=true) For more information, please enable DEBUG log level on org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy and org.apache.hadoop.net.NetworkTopology
2021-08-31 13:22:16  
2021-08-31 10:22:16,004 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1232821494_159082059, replicas=10.244.4.172:9866 for /tmp/temporary-d8a826fa-131f-437e-bf71-eb0f961ec76e/.metadata.456528e1-7374-4f0d-9139-119bdcc0469f.tmp
2021-08-31 13:22:16  
2021-08-31 10:22:15,684 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 231071 Total time for transactions(ms): 1511 Number of transactions batched in Syncs: 68080 Number of syncs: 162991 SyncTimes(ms): 262896 
2021-08-31 13:22:16  
  at java.lang.Thread.run(Thread.java:745)
2021-08-31 13:22:16  
  at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:292)
2021-08-31 13:22:16  
  at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:107)
2021-08-31 13:22:16  
  at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:173)
2021-08-31 13:22:16  
  at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:763)
2021-08-31 13:22:16  
  at org.apache.hadoop.hdfs.server.datanode.DataXceiver.getBlockReceiver(DataXceiver.java:1312)
2021-08-31 13:22:16  
  at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:216)
2021-08-31 13:22:16  
  at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:1423)
2021-08-31 13:22:16  
  at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbw(FsVolumeImpl.java:1225)
2021-08-31 13:22:16  
  at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsVolumeImpl.createRbwFile(FsVolumeImpl.java:945)
2021-08-31 13:22:16  
  at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice.createRbwFile(BlockPoolSlice.java:350)
2021-08-31 13:22:16  
  at org.apache.hadoop.hdfs.server.datanode.DatanodeUtil.createFileWithExistsCheck(DatanodeUtil.java:69)
2021-08-31 13:22:16  
  at org.apache.hadoop.hdfs.server.datanode.FileIoProvider.createFile(FileIoProvider.java:302)
2021-08-31 13:22:16  
  at java.io.File.createNewFile(File.java:1012)
2021-08-31 13:22:16  
  at java.io.UnixFileSystem.createFileExclusively(Native Method)
2021-08-31 13:22:16  
java.io.IOException: No space left on device

当我们检查可用空间时，在所述卷上就足够了，链接到该位置没有成功持久化文件，但是，可用 inode 计数已经消失。

不知道，如何增加特定 pod 和卷的 inode 数量，但是有没有办法减少 Hadoop 打开的文件数量以减少 inode 使用作为修复的替代方法？

标签： hadoopkubernetesfilesystemskubernetes-podinode