memory-leaks - 使用kafka流时Java堆外的内存泄漏

Question

我使用 kafka 流加入流，但堆外内存失控。

我使用 jemalloc 来查找原因。首先，rocksdb 使用堆外内存的比例很高

1806344172  67.4%  67.4% 1806344172  67.4% rocksdb::BlockFetcher::ReadBlockContents
588400270  22.0%  89.4% 588400270  22.0% os::malloc@921040
132120590   4.9%  94.3% 132120590   4.9% rocksdb::Arena::AllocateNewBlock
50331648   1.9%  96.2% 50331648   1.9% init
17587683   0.7%  96.8% 17981107   0.7% rocksdb::VersionSet::ProcessManifestWrites
15688131   0.6%  97.4% 15688131   0.6% rocksdb::WritableFileWriter::WritableFileWriter
12943699   0.5%  97.9% 12943699   0.5% rocksdb::port::cacheline_aligned_alloc
11800800   0.4%  98.4% 12588000   0.5% rocksdb::LRUCacheShard::Insert
 8784504   0.3%  98.7% 1811954485  67.6% rocksdb::BlockBasedTable::PartitionedIndexIteratorState::NewSecondaryIterator
 7606272   0.3%  99.0%  7606272   0.3% rocksdb::LRUHandleTable::Resize

时间长了就变了

Total: 4502654593 B
3379447055  75.1%  75.1% 3379447055  75.1% os::malloc@921040
620666890  13.8%  88.8% 620666890  13.8% rocksdb::BlockFetcher::ReadBlockContents
142606352   3.2%  92.0% 142606352   3.2% rocksdb::Arena::AllocateNewBlock
129603986   2.9%  94.9% 129603986   2.9% rocksdb::port::cacheline_aligned_alloc
67797317   1.5%  96.4% 67797317   1.5% rocksdb::LRUHandleTable::Resize
50331648   1.1%  97.5% 50331648   1.1% init
32501412   0.7%  98.2% 230760042   5.1% Java_org_rocksdb_Options_newOptions__
18600150   0.4%  98.6% 19255895   0.4% rocksdb::VersionSet::ProcessManifestWrites
16393216   0.4%  99.0% 16393216   0.4% rocksdb::WritableFileWriter::WritableFileWriter
 5629242   0.1%  99.1%  5629242   0.1% updatewindow

os::malloc@921040 消耗了大部分内存，并且一直在增长

所以任何人都可以提供一些帮助吗？

标签： memory-leaksapache-kafkaapache-kafka-streamsrocksdb