java - Netty tcnative - 本机代码崩溃：[libc.so.6+0x81a8c] cfree+0x1c
问题描述
我们在客户场所部署了我们的应用程序。在多个安装中，一个实例在执行标题中提到的本机代码时不断崩溃。
以非常不均匀的间隔观察到崩溃，从 4 小时到甚至超过 5 天不等。所有的出现都表明崩溃是在执行 Java 框架时发生的：io.netty.internal.tcnative.SSL.readFromSSL以及本地库和[libc.so.6+0x81a8c]. 在大多数情况下，Java 框架看起来很相似。
依赖项：
netty-all-4.1.50.Final.jar
netty-tcnative-2.0.30.Final-linux-x86_64.jar
libc:glibc 2.28 NPTL 2.28
OpenSSL: OpenSSL 1.0.2v-fips 5 May 2020
提供故障转储文件的以下相关部分。任何输入都会有很大帮助。
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007fdb64c26a8c, pid=8969, tid=9065
#
# JRE version: OpenJDK Runtime Environment (Zulu11.39+16-SA) (11.0.7+10) (build 11.0.7+10-LTS)
# Java VM: OpenJDK 64-Bit Server VM (11.0.7+10-LTS, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C [libc.so.6+0x81a8c] cfree+0x1c
#
ost: Intel(R) Xeon(R) Gold 6142 CPU @ 2.60GHz, 4 cores, 15G, VMware Photon OS 3.0
--------------- T H R E A D ---------------
Current thread (0x00007fdaf8018000): JavaThread "nioEventLoopGroup-10-4" [_thread_in_native, id=9065, stack(0x00007fdab94f1000,0x00007fdab95f2000)]
Stack: [0x00007fdab94f1000,0x00007fdab95f2000], sp=0x00007fdab95efde8, free space=1019k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libc.so.6+0x81a8c] cfree+0x1c
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 3701 io.netty.internal.tcnative.SSL.readFromSSL(JJI)I (0 bytes) @ 0x00007fdb4fbaa177 [0x00007fdb4fbaa120+0x0000000000000057]
J 10590 c2 io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap([Ljava/nio/ByteBuffer;II[Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult; (1438 bytes) @ 0x00007fdb5032c00c [0x00007fdb5032ba40+0x00000000000005cc]
J 6066 c2 io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(Lio/netty/handler/ssl/SslHandler;Lio/netty/buffer/ByteBuf;IILio/netty/buffer/ByteBuf;)Ljavax/net/ssl/SSLEngineResult; (134 bytes) @ 0x00007fdb4fd49d90 [0x00007fdb4fd499e0+0x00000000000003b0]
J 10485 c2 io.netty.handler.ssl.SslHandler.unwrap(Lio/netty/channel/ChannelHandlerContext;Lio/netty/buffer/ByteBuf;II)I (629 bytes) @ 0x00007fdb5034d364 [0x00007fdb5034d080+0x00000000000002e4]
J 10202 c2 io.netty.handler.ssl.SslHandler.decode(Lio/netty/channel/ChannelHandlerContext;Lio/netty/buffer/ByteBuf;Ljava/util/List;)V (31 bytes) @ 0x00007fdb4fee5bc0 [0x00007fdb4fee58a0+0x0000000000000320]
J 10198 c2 io.netty.handler.codec.ByteToMessageDecoder.callDecode(Lio/netty/channel/ChannelHandlerContext;Lio/netty/buffer/ByteBuf;Ljava/util/List;)V (172 bytes) @ 0x00007fdb4fee92c8 [0x00007fdb4fee91e0+0x00000000000000e8]
J 6825 c2 io.netty.handler.codec.ByteToMessageDecoder.channelRead(Lio/netty/channel/ChannelHandlerContext;Ljava/lang/Object;)V (328 bytes) @ 0x00007fdb4fe2a200 [0x00007fdb4fe29d80+0x0000000000000480]
J 8672 c2 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(Ljava/lang/Object;)Lio/netty/channel/ChannelHandlerContext; (12 bytes) @ 0x00007fdb500e9f28 [0x00007fdb500e9cc0+0x0000000000000268]
J 7050 c2 io.netty.handler.timeout.IdleStateHandler.channelRead(Lio/netty/channel/ChannelHandlerContext;Ljava/lang/Object;)V (42 bytes) @ 0x00007fdb4fe5eafc [0x00007fdb4fe5eaa0+0x000000000000005c]
J 8672 c2 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(Ljava/lang/Object;)Lio/netty/channel/ChannelHandlerContext; (12 bytes) @ 0x00007fdb500e9f28 [0x00007fdb500e9cc0+0x0000000000000268]
J 6504 c2 io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(Lio/netty/channel/ChannelHandlerContext;Ljava/lang/Object;)V (9 bytes) @ 0x00007fdb4fdd6964 [0x00007fdb4fdd6920+0x0000000000000044]
J 9935 c2 io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read()V (298 bytes) @ 0x00007fdb4fea0678 [0x00007fdb4fea0180+0x00000000000004f8]
J 9969 c2 io.netty.channel.nio.NioEventLoop.processSelectedKey(Ljava/nio/channels/SelectionKey;Lio/netty/channel/nio/AbstractNioChannel;)V (136 bytes) @ 0x00007fdb4fea9300 [0x00007fdb4fea9220+0x00000000000000e0]
J 10373% c2 io.netty.channel.nio.NioEventLoop.run()V (419 bytes) @ 0x00007fdb5033cef4 [0x00007fdb5033ca20+0x00000000000004d4]
j io.netty.util.concurrent.SingleThreadEventExecutor$4.run()V+44
j io.netty.util.internal.ThreadExecutorMap$2.run()V+11
j io.netty.util.concurrent.FastThreadLocalRunnable.run()V+4
j java.lang.Thread.run()V+11 java.base@11.0.7
v ~StubRoutines::call_stub
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000288
Top of Stack: (sp=0x00007fdab95efde8)
0x00007fdab95efde8: 00007fdaea98389a 00007fdb142ab520
0x00007fdab95efdf8: 00007fdaea7ed9fc 0000000000000001
0x00007fdab95efe08: 00007fdb5cf82540 0000000000000100
0x00007fdab95efe18: 00007fdab95effd0 00007fdb5cf82540
Instructions: (pc=0x00007fdb64c26a8c)
0x00007fdb64c2698c: 85 c0 48 89 c2 75 2c 4d 85 e4 74 27 0f 1f 84 00
0x00007fdb64c2699c: 00 00 00 00 4c 89 e7 48 89 ee e8 b5 c5 ff ff 48
0x00007fdb64c269ac: 89 ea 48 89 de 48 89 c7 49 89 c4 e8 24 ed ff ff
0x00007fdb64c269bc: 48 89 c2 4d 85 e4 0f 84 25 ff ff ff e9 f1 fe ff
0x00007fdb64c269cc: ff 0f 1f 00 48 8b 05 a1 04 13 00 31 d2 64 c7 00
0x00007fdb64c269dc: 16 00 00 00 e9 e9 fd ff ff 48 8d 0d 6c bd 0f 00
0x00007fdb64c269ec: 48 8d 35 13 7a 0f 00 48 8d 3d 4e bb 0f 00 ba e9
0x00007fdb64c269fc: 0c 00 00 e8 ec b4 ff ff 48 89 fb e9 6a fd ff ff
0x00007fdb64c26a0c: 0f 1f 40 00 48 83 ec 18 48 8b 05 cd 04 13 00 48
0x00007fdb64c26a1c: c7 00 00 00 00 00 8b 05 3c 08 13 00 85 c0 78 14
0x00007fdb64c26a2c: 48 8b 54 24 18 48 83 c4 18 e9 06 fd ff ff 66 0f
0x00007fdb64c26a3c: 1f 44 00 00 48 89 74 24 08 48 89 3c 24 e8 c2 bb
0x00007fdb64c26a4c: ff ff 48 8b 74 24 08 48 8b 3c 24 48 8b 54 24 18
0x00007fdb64c26a5c: 48 83 c4 18 e9 db fc ff ff 66 66 2e 0f 1f 84 00
0x00007fdb64c26a6c: 00 00 00 00 48 8b 05 89 04 13 00 48 8b 00 48 85
0x00007fdb64c26a7c: c0 0f 85 dd 00 00 00 48 85 ff 0f 84 ac 00 00 00
0x00007fdb64c26a8c: 48 8b 47 f8 48 8d 77 f0 a8 02 74 48 8b 15 1e 08
0x00007fdb64c26a9c: 13 00 85 d2 75 33 48 3b 05 e7 07 13 00 76 2a 48
0x00007fdb64c26aac: 3d 00 00 00 02 77 22 48 3b 35 7e 2e 13 00 0f 83
0x00007fdb64c26abc: b0 00 00 00 48 83 e0 f8 48 89 05 c5 07 13 00 48
0x00007fdb64c26acc: 01 c0 48 89 05 ab 07 13 00 48 89 f7 e9 43 b9 ff
0x00007fdb64c26adc: ff 0f 1f 00 48 89 c2 83 e2 04 48 83 3d ba 07 13
0x00007fdb64c26aec: 00 00 74 2b 48 8b 0d 91 02 13 00 48 85 d2 64 4c
0x00007fdb64c26afc: 8b 01 74 40 48 89 f7 48 81 e7 00 00 00 fc 48 8b
0x00007fdb64c26b0c: 3f 49 39 f8 75 1a 48 83 e0 f8 64 48 29 41 08 48
0x00007fdb64c26b1c: 85 d2 74 2c 48 89 f0 48 25 00 00 00 fc 48 8b 38
0x00007fdb64c26b2c: 31 d2 e9 0d cb ff ff 0f 1f 44 00 00 f3 c3 66 0f
0x00007fdb64c26b3c: 1f 44 00 00 48 8d 3d f9 10 13 00 49 39 f8 74 c6
0x00007fdb64c26b4c: 48 8d 3d ed 10 13 00 31 d2 e9 e6 ca ff ff 66 0f
0x00007fdb64c26b5c: 1f 44 00 00 48 8b 34 24 ff e0 66 2e 0f 1f 84 00
0x00007fdb64c26b6c: 00 00 00 00 48 3b 35 b9 2d 13 00 0f 83 43 ff ff
0x00007fdb64c26b7c: ff e9 53 ff ff ff 66 66 2e 0f 1f 84 00 00 00 00
Stack slot to memory mapping:
stack at sp + 0 slots: 0x00007fdaea98389a: SSL_SESSION_free+0x000000000000012a in /lib/libssl.so.1.1 at 0x00007fdaea942000
stack at sp + 1 slots: 0x00007fdb142ab520 points into unknown readable memory: 70 f3 3f 14 db 7f 00 00
stack at sp + 2 slots: 0x00007fdaea7ed9fc: OPENSSL_LH_doall_arg+0x000000000000004c in /lib/libcrypto.so.1.1 at 0x00007fdaea67c000
stack at sp + 3 slots: 0x0000000000000001 is an unknown value
stack at sp + 4 slots: 0x00007fdb5cf82540 points into unknown readable memory: c0 cc 9c ea da 7f 00 00
stack at sp + 5 slots: 0x0000000000000100 is an unknown value
stack at sp + 6 slots: 0x00007fdab95effd0 is pointing into the stack for thread: 0x00007fdaf8018000
stack at sp + 7 slots: 0x00007fdb5cf82540 points into unknown readable memory: c0 cc 9c ea da 7f 00 00
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: None
Heap address: 0x0000000705400000, size: 4012 MB, Compressed Oops mode: Zero based, Oop shift amount: 3
Narrow klass base: 0x0000000800000000, Narrow klass shift: 0
Compressed class space size: 1073741824 Address: 0x0000000800000000
Heap:
garbage-first heap total 311296K, used 166454K [0x0000000705400000, 0x0000000800000000)
region size 1024K, 132 young (135168K), 2 survivors (2048K)
Metaspace used 47712K, capacity 48362K, committed 48668K, reserved 1091584K
class space used 5432K, capacity 5678K, committed 5760K, reserved 1048576K
Metaspace:
Usage:
Non-class: 41.68 MB capacity, 41.29 MB (>99%) used, 317.92 KB ( <1%) free+waste, 86.06 KB ( <1%) overhead.
Class: 5.54 MB capacity, 5.31 MB ( 96%) used, 209.49 KB ( 4%) free+waste, 35.69 KB ( <1%) overhead.
Both: 47.23 MB capacity, 46.59 MB ( 99%) used, 527.41 KB ( 1%) free+waste, 121.75 KB ( <1%) overhead.
Virtual space:
Non-class space: 42.00 MB reserved, 41.90 MB (>99%) committed
Class space: 1.00 GB reserved, 5.62 MB ( <1%) committed
Both: 1.04 GB reserved, 47.53 MB ( 4%) committed
Chunk freelists:
Non-Class: 140.00 KB
Class: 0 bytes
Both: 140.00 KB
MaxMetaspaceSize: unlimited
CompressedClassSpaceSize: 1.00 GB
CodeHeap 'non-profiled nmethods': size=120032Kb used=10300Kb max_used=10300Kb free=109731Kb
bounds [0x00007fdb4fa32000, 0x00007fdb50452000, 0x00007fdb56f6a000]
CodeHeap 'profiled nmethods': size=120032Kb used=12721Kb max_used=19635Kb free=107310Kb
bounds [0x00007fdb484fa000, 0x00007fdb4985a000, 0x00007fdb4fa32000]
CodeHeap 'non-nmethods': size=5696Kb used=1238Kb max_used=1298Kb free=4457Kb
bounds [0x00007fdb47f6a000, 0x00007fdb481da000, 0x00007fdb484fa000]
total_blobs=17519 nmethods=8176 adapters=533
compilation: enabled
stopped_count=0, restarted_count=0
full_count=0
VM Arguments:
jvm_args: -Dlog4j.configuration=file:/............/conf/log4j.properties
java_command: /......./application-version.jar
java_class_path (initial): /......./application-version.jar
Launcher Type: SUN_STANDARD
[Global flags]
intx CICompilerCount = 3 {product} {ergonomic}
uint ConcGCThreads = 1 {product} {ergonomic}
uint G1ConcRefinementThreads = 4 {product} {ergonomic}
size_t G1HeapRegionSize = 1048576 {product} {ergonomic}
uintx GCDrainStackTargetSize = 64 {product} {ergonomic}
size_t InitialHeapSize = 264241152 {product} {ergonomic}
size_t MarkStackSize = 4194304 {product} {ergonomic}
size_t MaxHeapSize = 4206886912 {product} {ergonomic}
size_t MaxNewSize = 2523922432 {product} {ergonomic}
size_t MinHeapDeltaBytes = 1048576 {product} {ergonomic}
uintx NonNMethodCodeHeapSize = 5830732 {pd product} {ergonomic}
uintx NonProfiledCodeHeapSize = 122913754 {pd product} {ergonomic}
uintx ProfiledCodeHeapSize = 122913754 {pd product} {ergonomic}
uintx ReservedCodeCacheSize = 251658240 {pd product} {ergonomic}
bool SegmentedCodeCache = true {product} {ergonomic}
bool UseCompressedClassPointers = true {lp64_product} {ergonomic}
bool UseCompressedOops = true {lp64_product} {ergonomic}
bool UseG1GC = true {product} {ergonomic}
Logging:
Log output configuration:
#0: stdout all=warning uptime,level,tags
#1: stderr all=off uptime,level,tags
Environment Variables:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
libc:glibc 2.28 NPTL 2.28
解决方案
推荐阅读
- r - r- dummyVars 中的重复行
- php - PHP如何通过while循环进行Whileloop
- python-3.x - Python - For 循环（两个并行迭代）
- java - 获取数组中单词的第一个字母以大写
- c# - C#如何仅从没有扩展名的文件夹中获取.pdf文件名？
- html - 如何在 Angular 项目的 html 组件之间传递字符串变量作为参数？
- unity3d - ..的统一变量尚未分配
- c - Linux 套接字：第二次 read() 在 localhost 上失败
- matlab - 细节特征提取算法
- python - 如何从控制台为模型创建对象，类似于 Django 如何创建 createsuperuser 命令