apache-kafka - Kafka Streams 应用程序内存不足
问题描述
我们构建了一个具有相当复杂拓扑的 Kafka-Streams 应用程序：
Sub-topology: 0
Source: KSTREAM-SOURCE-0000000000 (topics: [sourceTopic1])
--> KSTREAM-MAPVALUES-0000000011, KSTREAM-PEEK-0000000002
Processor: KSTREAM-MAPVALUES-0000000011 (stores: [])
--> KSTREAM-KEY-SELECT-0000000012
<-- KSTREAM-SOURCE-0000000000
Processor: KSTREAM-KEY-SELECT-0000000012 (stores: [])
--> KSTREAM-FILTER-0000000015
<-- KSTREAM-MAPVALUES-0000000011
Processor: KSTREAM-FILTER-0000000015 (stores: [])
--> KSTREAM-SINK-0000000014
<-- KSTREAM-KEY-SELECT-0000000012
Processor: KSTREAM-PEEK-0000000002 (stores: [])
--> none
<-- KSTREAM-SOURCE-0000000000
Sink: KSTREAM-SINK-0000000014 (topic: pop-to-pop-metrics-store-repartition)
<-- KSTREAM-FILTER-0000000015
Sub-topology: 1
Source: KSTREAM-SOURCE-0000000001 (topics: [sourceTopic2])
--> KSTREAM-FILTER-0000000003
Processor: KSTREAM-FILTER-0000000003 (stores: [])
--> KSTREAM-MAPVALUES-0000000005, KSTREAM-PEEK-0000000004
<-- KSTREAM-SOURCE-0000000001
Processor: KSTREAM-MAPVALUES-0000000005 (stores: [])
--> KSTREAM-KEY-SELECT-0000000006
<-- KSTREAM-FILTER-0000000003
Processor: KSTREAM-KEY-SELECT-0000000006 (stores: [])
--> KSTREAM-FILTER-0000000009
<-- KSTREAM-MAPVALUES-0000000005
Processor: KSTREAM-FILTER-0000000009 (stores: [])
--> KSTREAM-SINK-0000000008
<-- KSTREAM-KEY-SELECT-0000000006
Processor: KSTREAM-PEEK-0000000004 (stores: [])
--> none
<-- KSTREAM-FILTER-0000000003
Sink: KSTREAM-SINK-0000000008 (topic: pop-to-site-metrics-store-repartition)
<-- KSTREAM-FILTER-0000000009
Sub-topology: 2
Source: KSTREAM-SOURCE-0000000010 (topics: [pop-to-site-metrics-store-repartition])
--> KSTREAM-AGGREGATE-0000000007
Processor: KSTREAM-AGGREGATE-0000000007 (stores: [pop-to-site-metrics-store])
--> KTABLE-TOSTREAM-0000000017
<-- KSTREAM-SOURCE-0000000010
Processor: KTABLE-TOSTREAM-0000000017 (stores: [])
--> KSTREAM-MAP-0000000018
<-- KSTREAM-AGGREGATE-0000000007
Processor: KSTREAM-MAP-0000000018 (stores: [])
--> KSTREAM-FILTER-0000000022, KSTREAM-FILTER-0000000033
<-- KTABLE-TOSTREAM-0000000017
Processor: KSTREAM-FILTER-0000000033 (stores: [])
--> KSTREAM-FILTER-0000000035
<-- KSTREAM-MAP-0000000018
Processor: KSTREAM-FILTER-0000000022 (stores: [])
--> KSTREAM-SINK-0000000021
<-- KSTREAM-MAP-0000000018
Processor: KSTREAM-FILTER-0000000035 (stores: [])
--> KSTREAM-SINK-0000000034
<-- KSTREAM-FILTER-0000000033
Sink: KSTREAM-SINK-0000000021 (topic: KSTREAM-MAP-0000000018-repartition)
<-- KSTREAM-FILTER-0000000022
Sink: KSTREAM-SINK-0000000034 (topic: KSTREAM-FILTER-0000000033-repartition)
<-- KSTREAM-FILTER-0000000035
Sub-topology: 3
Source: KSTREAM-SOURCE-0000000016 (topics: [pop-to-pop-metrics-store-repartition])
--> KSTREAM-AGGREGATE-0000000013
Processor: KSTREAM-AGGREGATE-0000000013 (stores: [pop-to-pop-metrics-store])
--> KTABLE-TOSTREAM-0000000019
<-- KSTREAM-SOURCE-0000000016
Processor: KTABLE-TOSTREAM-0000000019 (stores: [])
--> KSTREAM-MAP-0000000020
<-- KSTREAM-AGGREGATE-0000000013
Processor: KSTREAM-MAP-0000000020 (stores: [])
--> KSTREAM-FILTER-0000000025
<-- KTABLE-TOSTREAM-0000000019
Processor: KSTREAM-FILTER-0000000025 (stores: [])
--> KSTREAM-SINK-0000000024
<-- KSTREAM-MAP-0000000020
Sink: KSTREAM-SINK-0000000024 (topic: KSTREAM-MAP-0000000020-repartition)
<-- KSTREAM-FILTER-0000000025
Sub-topology: 4
Source: KSTREAM-SOURCE-0000000023 (topics: [KSTREAM-MAP-0000000018-repartition])
--> KSTREAM-WINDOWED-0000000027
Source: KSTREAM-SOURCE-0000000026 (topics: [KSTREAM-MAP-0000000020-repartition])
--> KSTREAM-WINDOWED-0000000028
Processor: KSTREAM-WINDOWED-0000000027 (stores: [KSTREAM-JOINTHIS-0000000029-store])
--> KSTREAM-JOINTHIS-0000000029
<-- KSTREAM-SOURCE-0000000023
Processor: KSTREAM-WINDOWED-0000000028 (stores: [KSTREAM-JOINOTHER-0000000030-store])
--> KSTREAM-JOINOTHER-0000000030
<-- KSTREAM-SOURCE-0000000026
Processor: KSTREAM-JOINOTHER-0000000030 (stores: [KSTREAM-JOINTHIS-0000000029-store])
--> KSTREAM-MERGE-0000000031
<-- KSTREAM-WINDOWED-0000000028
Processor: KSTREAM-JOINTHIS-0000000029 (stores: [KSTREAM-JOINOTHER-0000000030-store])
--> KSTREAM-MERGE-0000000031
<-- KSTREAM-WINDOWED-0000000027
Processor: KSTREAM-MERGE-0000000031 (stores: [])
--> KSTREAM-KEY-SELECT-0000000032
<-- KSTREAM-JOINTHIS-0000000029, KSTREAM-JOINOTHER-0000000030
Processor: KSTREAM-KEY-SELECT-0000000032 (stores: [])
--> KSTREAM-FILTER-0000000038
<-- KSTREAM-MERGE-0000000031
Processor: KSTREAM-FILTER-0000000038 (stores: [])
--> KSTREAM-SINK-0000000037
<-- KSTREAM-KEY-SELECT-0000000032
Sink: KSTREAM-SINK-0000000037 (topic: KSTREAM-KEY-SELECT-0000000032-repartition)
<-- KSTREAM-FILTER-0000000038
Sub-topology: 5
Source: KSTREAM-SOURCE-0000000036 (topics: [KSTREAM-FILTER-0000000033-repartition])
--> KSTREAM-WINDOWED-0000000040
Source: KSTREAM-SOURCE-0000000039 (topics: [KSTREAM-KEY-SELECT-0000000032-repartition])
--> KSTREAM-WINDOWED-0000000041
Processor: KSTREAM-WINDOWED-0000000040 (stores: [KSTREAM-JOINTHIS-0000000042-store])
--> KSTREAM-JOINTHIS-0000000042
<-- KSTREAM-SOURCE-0000000036
Processor: KSTREAM-WINDOWED-0000000041 (stores: [KSTREAM-JOINOTHER-0000000043-store])
--> KSTREAM-JOINOTHER-0000000043
<-- KSTREAM-SOURCE-0000000039
Processor: KSTREAM-JOINOTHER-0000000043 (stores: [KSTREAM-JOINTHIS-0000000042-store])
--> KSTREAM-MERGE-0000000044
<-- KSTREAM-WINDOWED-0000000041
Processor: KSTREAM-JOINTHIS-0000000042 (stores: [KSTREAM-JOINOTHER-0000000043-store])
--> KSTREAM-MERGE-0000000044
<-- KSTREAM-WINDOWED-0000000040
Processor: KSTREAM-MERGE-0000000044 (stores: [])
--> KSTREAM-FILTER-0000000045
<-- KSTREAM-JOINTHIS-0000000042, KSTREAM-JOINOTHER-0000000043
Processor: KSTREAM-FILTER-0000000045 (stores: [])
--> KSTREAM-KEY-SELECT-0000000046
<-- KSTREAM-MERGE-0000000044
Processor: KSTREAM-KEY-SELECT-0000000046 (stores: [])
--> KSTREAM-SINK-0000000047
<-- KSTREAM-FILTER-0000000045
Sink: KSTREAM-SINK-0000000047 (topic: targetTopic1)
<-- KSTREAM-KEY-SELECT-0000000046
Sub-topology: 6
Source: KSTREAM-SOURCE-0000000048 (topics: [targetTopic1])
--> KSTREAM-KEY-SELECT-0000000060
Processor: KSTREAM-KEY-SELECT-0000000060 (stores: [])
--> KSTREAM-FILTER-0000000062
<-- KSTREAM-SOURCE-0000000048
Processor: KSTREAM-FILTER-0000000062 (stores: [])
--> KSTREAM-SINK-0000000061
<-- KSTREAM-KEY-SELECT-0000000060
Sink: KSTREAM-SINK-0000000061 (topic: KSTREAM-KEY-SELECT-0000000060-repartition)
<-- KSTREAM-FILTER-0000000062
Sub-topology: 7
Source: KSTREAM-SOURCE-0000000049 (topics: [sourceTopic3])
--> KSTREAM-MAPVALUES-0000000051, KSTREAM-PEEK-0000000050
Processor: KSTREAM-MAPVALUES-0000000051 (stores: [])
--> KSTREAM-FILTER-0000000052
<-- KSTREAM-SOURCE-0000000049
Processor: KSTREAM-FILTER-0000000052 (stores: [])
--> KSTREAM-KEY-SELECT-0000000053
<-- KSTREAM-MAPVALUES-0000000051
Processor: KSTREAM-KEY-SELECT-0000000053 (stores: [])
--> KSTREAM-FILTER-0000000056
<-- KSTREAM-FILTER-0000000052
Processor: KSTREAM-FILTER-0000000056 (stores: [])
--> KSTREAM-SINK-0000000055
<-- KSTREAM-KEY-SELECT-0000000053
Processor: KSTREAM-PEEK-0000000050 (stores: [])
--> none
<-- KSTREAM-SOURCE-0000000049
Sink: KSTREAM-SINK-0000000055 (topic: effective-link-selector-store-repartition)
<-- KSTREAM-FILTER-0000000056
Sub-topology: 8
Source: KSTREAM-SOURCE-0000000057 (topics: [effective-link-selector-store-repartition])
--> KSTREAM-AGGREGATE-0000000054
Processor: KSTREAM-AGGREGATE-0000000054 (stores: [effective-link-selector-store])
--> KTABLE-TOSTREAM-0000000058
<-- KSTREAM-SOURCE-0000000057
Processor: KTABLE-TOSTREAM-0000000058 (stores: [])
--> KSTREAM-MAP-0000000059
<-- KSTREAM-AGGREGATE-0000000054
Processor: KSTREAM-MAP-0000000059 (stores: [])
--> KSTREAM-FILTER-0000000065, KSTREAM-FILTER-0000000073
<-- KTABLE-TOSTREAM-0000000058
Processor: KSTREAM-FILTER-0000000073 (stores: [])
--> KSTREAM-FILTER-0000000078
<-- KSTREAM-MAP-0000000059
Processor: KSTREAM-FILTER-0000000065 (stores: [])
--> KSTREAM-SINK-0000000064
<-- KSTREAM-MAP-0000000059
Processor: KSTREAM-FILTER-0000000078 (stores: [])
--> KSTREAM-SINK-0000000077
<-- KSTREAM-FILTER-0000000073
Sink: KSTREAM-SINK-0000000064 (topic: KSTREAM-MAP-0000000059-repartition)
<-- KSTREAM-FILTER-0000000065
Sink: KSTREAM-SINK-0000000077 (topic: KSTREAM-FILTER-0000000073-repartition)
<-- KSTREAM-FILTER-0000000078
Sub-topology: 9
Source: KSTREAM-SOURCE-0000000063 (topics: [KSTREAM-KEY-SELECT-0000000060-repartition])
--> KSTREAM-WINDOWED-0000000067
Source: KSTREAM-SOURCE-0000000066 (topics: [KSTREAM-MAP-0000000059-repartition])
--> KSTREAM-WINDOWED-0000000068
Processor: KSTREAM-WINDOWED-0000000067 (stores: [KSTREAM-JOINTHIS-0000000069-store])
--> KSTREAM-JOINTHIS-0000000069
<-- KSTREAM-SOURCE-0000000063
Processor: KSTREAM-WINDOWED-0000000068 (stores: [KSTREAM-JOINOTHER-0000000070-store])
--> KSTREAM-JOINOTHER-0000000070
<-- KSTREAM-SOURCE-0000000066
Processor: KSTREAM-JOINOTHER-0000000070 (stores: [KSTREAM-JOINTHIS-0000000069-store])
--> KSTREAM-MERGE-0000000071
<-- KSTREAM-WINDOWED-0000000068
Processor: KSTREAM-JOINTHIS-0000000069 (stores: [KSTREAM-JOINOTHER-0000000070-store])
--> KSTREAM-MERGE-0000000071
<-- KSTREAM-WINDOWED-0000000067
Processor: KSTREAM-MERGE-0000000071 (stores: [])
--> KSTREAM-KEY-SELECT-0000000072
<-- KSTREAM-JOINTHIS-0000000069, KSTREAM-JOINOTHER-0000000070
Processor: KSTREAM-KEY-SELECT-0000000072 (stores: [])
--> KSTREAM-FILTER-0000000075
<-- KSTREAM-MERGE-0000000071
Processor: KSTREAM-FILTER-0000000075 (stores: [])
--> KSTREAM-SINK-0000000074
<-- KSTREAM-KEY-SELECT-0000000072
Sink: KSTREAM-SINK-0000000074 (topic: KSTREAM-KEY-SELECT-0000000072-repartition)
<-- KSTREAM-FILTER-0000000075
Sub-topology: 10
Source: KSTREAM-SOURCE-0000000076 (topics: [KSTREAM-KEY-SELECT-0000000072-repartition])
--> KSTREAM-WINDOWED-0000000080
Source: KSTREAM-SOURCE-0000000079 (topics: [KSTREAM-FILTER-0000000073-repartition])
--> KSTREAM-WINDOWED-0000000081
Processor: KSTREAM-WINDOWED-0000000080 (stores: [KSTREAM-JOINTHIS-0000000082-store])
--> KSTREAM-JOINTHIS-0000000082
<-- KSTREAM-SOURCE-0000000076
Processor: KSTREAM-WINDOWED-0000000081 (stores: [KSTREAM-JOINOTHER-0000000083-store])
--> KSTREAM-JOINOTHER-0000000083
<-- KSTREAM-SOURCE-0000000079
Processor: KSTREAM-JOINOTHER-0000000083 (stores: [KSTREAM-JOINTHIS-0000000082-store])
--> KSTREAM-MERGE-0000000084
<-- KSTREAM-WINDOWED-0000000081
Processor: KSTREAM-JOINTHIS-0000000082 (stores: [KSTREAM-JOINOTHER-0000000083-store])
--> KSTREAM-MERGE-0000000084
<-- KSTREAM-WINDOWED-0000000080
Processor: KSTREAM-MERGE-0000000084 (stores: [])
--> KSTREAM-MAP-0000000085
<-- KSTREAM-JOINTHIS-0000000082, KSTREAM-JOINOTHER-0000000083
Processor: KSTREAM-MAP-0000000085 (stores: [])
--> KSTREAM-BRANCH-0000000086
<-- KSTREAM-MERGE-0000000084
Processor: KSTREAM-BRANCH-0000000086 (stores: [])
--> KSTREAM-BRANCHCHILD-0000000087, KSTREAM-BRANCHCHILD-0000000088
<-- KSTREAM-MAP-0000000085
Processor: KSTREAM-BRANCHCHILD-0000000088 (stores: [])
--> KSTREAM-KEY-SELECT-0000000089
<-- KSTREAM-BRANCH-0000000086
Processor: KSTREAM-BRANCHCHILD-0000000087 (stores: [])
--> KSTREAM-FILTER-0000000091
<-- KSTREAM-BRANCH-0000000086
Processor: KSTREAM-KEY-SELECT-0000000089 (stores: [])
--> KSTREAM-FILTER-0000000094
<-- KSTREAM-BRANCHCHILD-0000000088
Processor: KSTREAM-FILTER-0000000091 (stores: [])
--> KSTREAM-SINK-0000000090
<-- KSTREAM-BRANCHCHILD-0000000087
Processor: KSTREAM-FILTER-0000000094 (stores: [])
--> KSTREAM-SINK-0000000093
<-- KSTREAM-KEY-SELECT-0000000089
Sink: KSTREAM-SINK-0000000090 (topic: KSTREAM-BRANCHCHILD-0000000087-repartition)
<-- KSTREAM-FILTER-0000000091
Sink: KSTREAM-SINK-0000000093 (topic: KSTREAM-KEY-SELECT-0000000089-repartition)
<-- KSTREAM-FILTER-0000000094
Sub-topology: 11
Source: KSTREAM-SOURCE-0000000092 (topics: [KSTREAM-BRANCHCHILD-0000000087-repartition])
--> KSTREAM-WINDOWED-0000000096
Source: KSTREAM-SOURCE-0000000095 (topics: [KSTREAM-KEY-SELECT-0000000089-repartition])
--> KSTREAM-WINDOWED-0000000097
Processor: KSTREAM-WINDOWED-0000000096 (stores: [KSTREAM-OUTERTHIS-0000000098-store])
--> KSTREAM-OUTERTHIS-0000000098
<-- KSTREAM-SOURCE-0000000092
Processor: KSTREAM-WINDOWED-0000000097 (stores: [KSTREAM-OUTEROTHER-0000000099-store])
--> KSTREAM-OUTEROTHER-0000000099
<-- KSTREAM-SOURCE-0000000095
Processor: KSTREAM-OUTEROTHER-0000000099 (stores: [KSTREAM-OUTERTHIS-0000000098-store])
--> KSTREAM-MERGE-0000000100
<-- KSTREAM-WINDOWED-0000000097
Processor: KSTREAM-OUTERTHIS-0000000098 (stores: [KSTREAM-OUTEROTHER-0000000099-store])
--> KSTREAM-MERGE-0000000100
<-- KSTREAM-WINDOWED-0000000096
Processor: KSTREAM-MERGE-0000000100 (stores: [])
--> KSTREAM-FLATMAP-0000000101
<-- KSTREAM-OUTERTHIS-0000000098, KSTREAM-OUTEROTHER-0000000099
Processor: KSTREAM-FLATMAP-0000000101 (stores: [])
--> KSTREAM-MAPVALUES-0000000102
<-- KSTREAM-MERGE-0000000100
Processor: KSTREAM-MAPVALUES-0000000102 (stores: [])
--> KSTREAM-SINK-0000000103
<-- KSTREAM-FLATMAP-0000000101
Sink: KSTREAM-SINK-0000000103 (topic: linkStatus)
<-- KSTREAM-MAPVALUES-0000000102
Sub-topology: 12
Source: KSTREAM-SOURCE-0000000104 (topics: [linkStatus])
--> KSTREAM-KEY-SELECT-0000000105
Processor: KSTREAM-KEY-SELECT-0000000105 (stores: [])
--> KSTREAM-FILTER-0000000108
<-- KSTREAM-SOURCE-0000000104
Processor: KSTREAM-FILTER-0000000108 (stores: [])
--> KSTREAM-SINK-0000000107
<-- KSTREAM-KEY-SELECT-0000000105
Sink: KSTREAM-SINK-0000000107 (topic: my-app-agg-store-repartition)
<-- KSTREAM-FILTER-0000000108
Sub-topology: 13
Source: KSTREAM-SOURCE-0000000109 (topics: [my-app-agg-store-repartition])
--> KSTREAM-AGGREGATE-0000000106
Processor: KSTREAM-AGGREGATE-0000000106 (stores: [my-app-agg-store])
--> KTABLE-TOSTREAM-0000000110
<-- KSTREAM-SOURCE-0000000109
Processor: KTABLE-TOSTREAM-0000000110 (stores: [])
--> KSTREAM-MAP-0000000111
<-- KSTREAM-AGGREGATE-0000000106
Processor: KSTREAM-MAP-0000000111 (stores: [])
--> KSTREAM-MAPVALUES-0000000112
<-- KTABLE-TOSTREAM-0000000110
Processor: KSTREAM-MAPVALUES-0000000112 (stores: [])
--> KSTREAM-SINK-0000000113
<-- KSTREAM-MAP-0000000111
Sink: KSTREAM-SINK-0000000113 (topic: finalTargetTopic)
<-- KSTREAM-MAPVALUES-0000000112
在负载下运行服务时，服务被 OOM 杀死。
在执行测试期间，我们注意到线程膨胀（超过 400 个）。我们使用 gdb 将它们跟踪到 RocksDb：
427 Thread 0x7f97a4722700 (LWP 18049) "java" 0x00007f97d9f6b9f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 426 Thread 0x7f9772fe5700 (LWP 18080) " rocksdb:dump_st " 0x00007f97d9f6bda2 in pthread_cond_timedwait@@GLIBC_2.3.2 ( ) 来自 /lib64/libpthread.so.0 425 线程 0x7f97737e6700 (LWP 18081) "java" 0x00007f97d9f6b9f5 in pthread_cond_wait@@GLIBC_2.3.2 () 来自 /lib64/libpthread.so.0 424 线程 0x7f977112) " rocksdb0:dump_st881 “ pthread_cond_timedwait@@GLIBC_2.3.2 () 中的 0x00007f97d9f6bda2 来自 /lib64/libpthread.so.0 423 线程 0x7f97697d2700 (LWP 18228) “java” 0x00007f97d9f6b9f5 in pthread_cond_wait () 来自 /@GLIBC_2.3.3.so.0@@GLIBC_2.3.3.线程 0x7f9767fcf700 (LWP 18244) "岩石数据库：dump_st“ pthread_cond_timedwait@@GLIBC_2.3.2 () 中的 0x00007f97d9f6bda2 来自 /lib64/libpthread.so.0 421 线程 0x7f97687d0700 (LWP 18245) “java” 0x00007f97d9f6b9f5 in pthread_cond_wait () 来自 /lib64/libpthread.so.0 421来自 /lib64/libpthread.so.0 的 pthread_cond_timedwait@@GLIBC_2.3.2 () 中的线程 0x7f976cfd9700 (LWP 18276) “ rocksdb:dump_st ” 0x00007f97d9f6bda2
当停止到服务的数据输入流时，进程高内存消耗保持稳定（它不会按预期下降）。
我们试图禁用 RockDb 的块缓存以限制其内存使用（如此处所述），但无济于事。
我们在 CentOS Linux 版本 7.3.1611 (Core) 上运行。我们的机器有 2 个 vCPU 和 4 GB 内存。我们的 JVM 配置为使用 500MB。我们的 KStream 版本是 2.3.1
有任何想法吗？
解决方案
推荐阅读
- powershell - 在对 WSL 的 Powershell 调用中使用变量
- c# - SignalR GetHubContext 按名称
- json - 如何解析熊猫数据框中的 JSON 列并将新数据框连接到原始数据框？
- azure - Azure B2C 密码字段文本和占位符
- sql - 生成动态列值 SQL
- sql-server - 是否有可能是一个具有共同值但只有一列具有不同数据的表以使其像单行中的枢轴一样？
- git - 如何在 Media Temple Centos 上通过 ssh 连接到 git 远程？
- javascript - 如何使用 Tone.JS 激活淡出效果？
- python - Psycopg2 update 不更新表，也不会抛出任何错误
- android - 如何制作清单
可打包？