out-of-memory - 当有 9GB 可用内存时,oom-killer 杀死了我的应用程序
问题描述
我们的一台生产服务器在 2020-12-26 20:05:30 遇到了 oom-killer 问题,但我无法弄清楚此事件的原因。zabbix 页面显示,当 oom-killer 杀死我的 java 进程时,可用内存约为 9GB。请帮我弄清楚为什么会这样,谢谢。
这是/var/log/message中的日志(在此之前有很多“内核:TCP:太多孤立的套接字”)
Dec 26 20:05:30 myhost kernel: systemd invoked oom-killer: gfp_mask=0x3000d0, order=2, oom_score_adj=0
Dec 26 20:05:30 myhost kernel: systemd cpuset=/ mems_allowed=0
Dec 26 20:05:30 myhost kernel: CPU: 4 PID: 1 Comm: systemd Not tainted 3.10.0-1062.9.1.el7.x86_64 #1
Dec 26 20:05:30 myhost kernel: Hardware name: Alibaba Cloud Alibaba Cloud ECS, BIOS 8c24b4c 04/01/2014
Dec 26 20:05:30 myhost kernel: Call Trace:
Dec 26 20:05:30 myhost kernel: [<ffffffffae37ac23>] dump_stack+0x19/0x1b
Dec 26 20:05:30 myhost kernel: [<ffffffffae375ce9>] dump_header+0x90/0x229
Dec 26 20:05:30 myhost kernel: [<ffffffffadd06142>] ? ktime_get_ts64+0x52/0xf0
Dec 26 20:05:30 myhost kernel: [<ffffffffaddc1714>] oom_kill_process+0x254/0x3e0
Dec 26 20:05:30 myhost kernel: [<ffffffffadd32e71>] ? cpuset_mems_allowed_intersects+0x21/0x30
Dec 26 20:05:30 myhost kernel: [<ffffffffaddc11bd>] ? oom_unkillable_task+0xcd/0x120
Dec 26 20:05:30 myhost kernel: [<ffffffffaddc1266>] ? find_lock_task_mm+0x56/0xc0
Dec 26 20:05:30 myhost kernel: [<ffffffffaddc1f66>] out_of_memory+0x4b6/0x4f0
Dec 26 20:05:30 myhost kernel: [<ffffffffaddc8a6f>] __alloc_pages_nodemask+0xacf/0xbe0
Dec 26 20:05:30 myhost kernel: [<ffffffffadc98f6d>] copy_process+0x1dd/0x1a50
Dec 26 20:05:30 myhost kernel: [<ffffffffadc9a991>] do_fork+0x91/0x330
Dec 26 20:05:30 myhost kernel: [<ffffffffae388a26>] ? trace_do_page_fault+0x56/0x150
Dec 26 20:05:30 myhost kernel: [<ffffffffadc9acb6>] SyS_clone+0x16/0x20
Dec 26 20:05:30 myhost kernel: [<ffffffffae38e2b4>] stub_clone+0x44/0x70
Dec 26 20:05:30 myhost kernel: [<ffffffffae38dede>] ? system_call_fastpath+0x25/0x2a
Dec 26 20:05:30 myhost kernel: Mem-Info:
Dec 26 20:05:30 myhost kernel: active_anon:13196525 inactive_anon:589 isolated_anon:35#012 active_file:950 inactive_file:1072 isolated_file:93#012 unevictable:0 dirty:0 writeback:30 unstable:0#012 slab_reclaimable:209382 slab_unreclaimable:237991#012 mapped:612 shmem:724 pagetables:36600 bounce:0#012 free:1721704 free_pcp:226 free_cma:0
Dec 26 20:05:30 myhost kernel: Node 0 DMA free:15904kB min:16kB low:20kB high:24kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Dec 26 20:05:30 myhost kernel: lowmem_reserve[]: 0 2810 63223 63223
Dec 26 20:05:30 myhost kernel: Node 0 DMA32 free:242744kB min:3000kB low:3748kB high:4500kB active_anon:1444208kB inactive_anon:24kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3111608kB managed:2878104kB mlocked:0kB dirty:0kB writeback:0kB mapped:20kB shmem:40kB slab_reclaimable:571840kB slab_unreclaimable:419836kB kernel_stack:65344kB pagetables:2092kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Dec 26 20:05:30 myhost kernel: lowmem_reserve[]: 0 0 60412 60412
Dec 26 20:05:30 myhost kernel: Node 0 Normal free:6628168kB min:64560kB low:80700kB high:96840kB active_anon:51341892kB inactive_anon:2332kB active_file:3800kB inactive_file:4288kB unevictable:0kB isolated(anon):140kB isolated(file):372kB present:62914560kB managed:61862736kB mlocked:0kB dirty:0kB writeback:120kB mapped:2428kB shmem:2856kB slab_reclaimable:265688kB slab_unreclaimable:532128kB kernel_stack:167552kB pagetables:144308kB unstable:0kB bounce:0kB free_pcp:904kB local_pcp:104kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Dec 26 20:05:30 myhost kernel: lowmem_reserve[]: 0 0 0 0
Dec 26 20:05:30 myhost kernel: Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15904kB
Dec 26 20:05:30 myhost kernel: Node 0 DMA32: 8570*4kB (UEM) 6220*8kB (UEM) 1155*16kB (UEM) 528*32kB (UEM) 647*64kB (UEM) 268*128kB (UEM) 76*256kB (UEM) 31*512kB (EM) 12*1024kB (UEM) 0*2048kB 0*4096kB = 242744kB
Dec 26 20:05:30 myhost kernel: Node 0 Normal: 1280224*4kB (EM) 188481*8kB (EM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6628744kB
Dec 26 20:05:30 myhost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Dec 26 20:05:30 myhost kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Dec 26 20:05:30 myhost kernel: 3295 total pagecache pages
Dec 26 20:05:30 myhost kernel: 0 pages in swap cache
Dec 26 20:05:30 myhost kernel: Swap cache stats: add 0, delete 0, find 0/0
Dec 26 20:05:30 myhost kernel: Free swap = 0kB
Dec 26 20:05:30 myhost kernel: Total swap = 0kB
Dec 26 20:05:30 myhost kernel: 16510540 pages RAM
Dec 26 20:05:30 myhost kernel: 0 pages HighMem/MovableOnly
Dec 26 20:05:30 myhost kernel: 321353 pages reserved
Dec 26 20:05:30 myhost kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Dec 26 20:05:30 myhost kernel: [ 532] 0 532 24185 100 53 0 0 systemd-journal
Dec 26 20:05:30 myhost kernel: [ 555] 0 555 11277 235 22 0 -1000 systemd-udevd
Dec 26 20:05:30 myhost kernel: [ 788] 0 788 13882 114 27 0 -1000 auditd
Dec 26 20:05:30 myhost kernel: [ 820] 0 820 58999 1724 26 0 0 CmsGoAgent.linu
Dec 26 20:05:30 myhost kernel: [ 836] 999 836 153093 1667 59 0 0 polkitd
Dec 26 20:05:30 myhost kernel: [ 837] 81 837 14597 203 32 0 -900 dbus-daemon
Dec 26 20:05:30 myhost kernel: [ 885] 0 885 6703 199 18 0 0 systemd-logind
Dec 26 20:05:30 myhost kernel: [ 892] 0 892 6477 53 18 0 0 atd
Dec 26 20:05:30 myhost kernel: [ 894] 0 894 31593 202 20 0 0 crond
Dec 26 20:05:30 myhost kernel: [ 896] 0 896 27527 34 10 0 0 agetty
Dec 26 20:05:30 myhost kernel: [ 898] 0 898 27527 34 11 0 0 agetty
Dec 26 20:05:30 myhost kernel: [ 948] 0 948 832809 11158 156 0 0 exe
Dec 26 20:05:30 myhost kernel: [ 1512] 0 1512 25724 515 48 0 0 dhclient
Dec 26 20:05:30 myhost kernel: [ 1587] 0 1587 73789 4313 97 0 0 salt-minion
Dec 26 20:05:30 myhost kernel: [ 1588] 0 1588 143550 3328 95 0 0 tuned
Dec 26 20:05:30 myhost kernel: [ 1592] 0 1592 29699 3019 20 0 0 node_exporter
Dec 26 20:05:30 myhost kernel: [ 1595] 0 1595 259453 450 342 0 0 rsyslogd
Dec 26 20:05:30 myhost kernel: [ 1756] 0 1756 28230 261 56 0 -1000 sshd
Dec 26 20:05:30 myhost kernel: [ 1825] 0 1825 285329 12268 162 0 0 salt-minion
Dec 26 20:05:30 myhost kernel: [ 1827] 0 1827 95073 5869 100 0 0 salt-minion
Dec 26 20:05:30 myhost kernel: [ 2149] 1001 2149 1572340 335768 806 0 0 java
Dec 26 20:05:30 myhost kernel: [11492] 997 11492 20486 213 40 0 0 zabbix_agentd
Dec 26 20:05:30 myhost kernel: [11493] 997 11493 20512 534 40 0 0 zabbix_agentd
Dec 26 20:05:30 myhost kernel: [11494] 997 11494 20542 261 41 0 0 zabbix_agentd
Dec 26 20:05:30 myhost kernel: [11495] 997 11495 20542 266 41 0 0 zabbix_agentd
Dec 26 20:05:30 myhost kernel: [11496] 997 11496 20542 261 41 0 0 zabbix_agentd
Dec 26 20:05:30 myhost kernel: [11497] 997 11497 20542 261 41 0 0 zabbix_agentd
Dec 26 20:05:30 myhost kernel: [11498] 997 11498 20542 279 41 0 0 zabbix_agentd
Dec 26 20:05:30 myhost kernel: [11499] 997 11499 20542 261 41 0 0 zabbix_agentd
Dec 26 20:05:30 myhost kernel: [11500] 997 11500 20542 261 41 0 0 zabbix_agentd
Dec 26 20:05:30 myhost kernel: [11501] 997 11501 20542 261 41 0 0 zabbix_agentd
Dec 26 20:05:30 myhost kernel: [11502] 997 11502 20542 262 41 0 0 zabbix_agentd
Dec 26 20:05:30 myhost kernel: [11503] 997 11503 20542 261 41 0 0 zabbix_agentd
Dec 26 20:05:30 myhost kernel: [11504] 997 11504 20520 241 41 0 0 zabbix_agentd
Dec 26 20:05:30 myhost kernel: [ 8595] 0 8595 8132 372 19 0 0 AliYunDunUpdate
Dec 26 20:05:30 myhost kernel: [ 9080] 0 9080 44295 11441 88 0 0 AliYunDun
Dec 26 20:05:30 myhost kernel: [58297] 38 58297 7491 158 19 0 0 ntpd
Dec 26 20:05:30 myhost kernel: [21537] 1001 21537 28726 472 12 0 0 app_watchdog.sh
Dec 26 20:05:30 myhost kernel: [21600] 1001 21600 3817967 615094 1819 0 0 java
Dec 26 20:05:30 myhost kernel: [29546] 1001 29546 28726 471 12 0 0 app_watchdog.sh
Dec 26 20:05:30 myhost kernel: [32310] 1001 32310 28726 471 12 0 0 app_watchdog.sh
Dec 26 20:05:30 myhost kernel: [32360] 1001 32360 4532386 1934665 5006 0 0 java
Dec 26 20:05:30 myhost kernel: [37926] 1001 37926 28726 471 13 0 0 app_watchdog.sh
Dec 26 20:05:30 myhost kernel: [37965] 1001 37965 3686958 387894 1320 0 0 java
Dec 26 20:05:30 myhost kernel: [40686] 1001 40686 28726 471 11 0 0 app_watchdog.sh
Dec 26 20:05:30 myhost kernel: [40735] 1001 40735 4597914 1960388 5065 0 0 java
Dec 26 20:05:30 myhost kernel: [41223] 1001 41223 28726 471 13 0 0 app_watchdog.sh
Dec 26 20:05:30 myhost kernel: [41468] 1001 41468 4596501 1862812 4877 0 0 java
Dec 26 20:05:30 myhost kernel: [43679] 1001 43679 28726 471 13 0 0 app_watchdog.sh
Dec 26 20:05:30 myhost kernel: [43836] 1001 43836 4596108 1924911 4999 0 0 java
Dec 26 20:05:30 myhost kernel: [45896] 1001 45896 28726 471 13 0 0 app_watchdog.sh
Dec 26 20:05:30 myhost kernel: [45988] 1001 45988 4595346 1896739 4951 0 0 java
Dec 26 20:05:30 myhost kernel: [56357] 1001 56357 4535046 1940786 5046 0 0 java
Dec 26 20:05:30 myhost kernel: [ 3720] 0 3720 201068 1471 13 0 0 aliyun-service
Dec 26 20:05:30 myhost kernel: [ 3822] 0 3822 4472 122 13 0 0 assist_daemon
Dec 26 20:05:30 myhost kernel: [28130] 0 28130 45614 263 46 0 0 crond
Dec 26 20:05:30 myhost kernel: [37497] 1001 37497 28325 91 11 0 0 app_log_bak.sh
Dec 26 20:05:30 myhost kernel: [37502] 1001 37502 28325 93 11 0 0 app_log_bak.sh
Dec 26 20:05:30 myhost kernel: [37506] 1001 37506 43958 188 41 0 0 scp
Dec 26 20:05:30 myhost kernel: [37507] 1001 37507 44682 252 43 0 0 ssh
Dec 26 20:05:30 myhost kernel: [54640] 1001 54640 28325 95 11 0 0 app_log_bak.sh
Dec 26 20:05:30 myhost kernel: [54645] 1001 54645 28325 97 11 0 0 app_log_bak.sh
Dec 26 20:05:30 myhost kernel: [54649] 1001 54649 43958 188 40 0 0 scp
Dec 26 20:05:30 myhost kernel: [54650] 1001 54650 44682 262 41 0 0 ssh
Dec 26 20:05:30 myhost kernel: [ 6703] 1001 6703 28325 99 11 0 0 app_log_bak.sh
Dec 26 20:05:30 myhost kernel: [ 6708] 1001 6708 28325 101 11 0 0 app_log_bak.sh
Dec 26 20:05:30 myhost kernel: [ 6711] 1001 6711 43958 200 42 0 0 scp
Dec 26 20:05:30 myhost kernel: [ 6713] 1001 6713 44682 253 43 0 0 ssh
Dec 26 20:05:30 myhost kernel: [11353] 1001 11353 28325 101 11 0 0 app_log_bak.sh
Dec 26 20:05:30 myhost kernel: [11358] 1001 11358 28325 103 11 0 0 app_log_bak.sh
Dec 26 20:05:30 myhost kernel: [11361] 1001 11361 43958 188 41 0 0 scp
Dec 26 20:05:30 myhost kernel: [11362] 1001 11362 44682 248 44 0 0 ssh
Dec 26 20:05:30 myhost kernel: [23855] 0 23855 45614 263 46 0 0 crond
Dec 26 20:05:30 myhost kernel: [23858] 1001 23858 28325 88 12 0 0 app_log_bak.sh
Dec 26 20:05:30 myhost kernel: Out of memory: Kill process 40735 (java) score 121 or sacrifice child
Dec 26 20:05:30 myhost kernel: Killed process 40735 (java), UID 1001, total-vm:18391656kB, anon-rss:7841552kB, file-rss:0kB, shmem-rss:0kB
主机信息:Linux myhost 3.10.0-1062.9.1.el7.x86_64 #1
free -lm gives:
total used free shared buff/cache available
Mem: 63239 41399 354 2 21484 21317
Low: 63239 62884 354
High: 0 0 0
Swap: 0 0 0
我们的主机故意禁用了交换。
解决方案
您的系统似乎遭受了严重的内存碎片,可能是因为您正在运行没有任何交换的旧内核。
重要的几行如下:
systemd invoked oom-killer: gfp_mask=0x3000d0, order=2, oom_score_adj=0
...
Node 0 Normal: 1280224*4kB (EM) 188481*8kB (EM) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6628744kB
的最后一位gfp_mask
是零,所以这个分配需要从Normal
内存区域完成。order=2
告诉需要分配的数量是 pagesize * 2 ^ order 或 4096 bytes * 2 * 2 = 16 kB 。
“buddyinfo”行(您可以使用 来检查当前状态cat /proc/buddyinfo
)告诉您在不同的块大小下有多少可用内存。事实证明,您有 1280224 个每个 4 kB 的块,188481 个每个 8 kB 的块和零个更大的块。该请求是针对 16 kB 块的,而您的可用块为零,因此系统进入 OOM 模式。(您的系统可能正在运行一些导致大量碎片的软件,因为这不是具有如此多总可用内存的通常状态。)
此外,Normal 区域的总可用 RAM 为 6628744 kB,而不是您想象的 9 GB。差异可能是由于您的内存统计记录器无法正确处理内存使用高峰造成的。
如果您至少使用了少量交换,我认为内核可以使用它来交换页面以将一些小页面片段合并为更大的片段,但是您完全没有交换,从而消除了这种可能性。我建议使用较小的zram
交换区域,以便在交换时您的性能不受 SSD 或 HDD 的限制。
作为替代方案,您可以尝试运行支持内存压缩 ( CONFIG_COMPACTION=y
) 的更现代的内核,并且可以按如下方式配置系统:
/proc/sys/vm/extfrag_threshold
:至少500
,最好更多/proc/sys/vm/watermark_scale_factor
:至少100
,最好更多
/proc/sys/vm/zone_reclaim_mode
:至少1
,我建议3
或7
(请注意,如果您正在实时运行,您无法更改此设置,但您也不应该用完 RAM)
您也可以考虑设置/proc/sys/vm/watermark_boost_factor
为大于零的值,但请先阅读文档。
虚拟内存管理器配置选项记录在https://github.com/torvalds/linux/blob/master/Documentation/admin-guide/sysctl/vm.rst(忽略声称此文档仅 2.6.29 版的标题,它实际上描述了最新的 Linux 内核)。
或者,如果这仅在系统运行很长时间时发生,则可以通过运行cron
定期以 root 身份运行的作业来避免碎片echo 1 > /proc/sys/vm/compact_memory
。
如果您正在运行的软件支持大页面,您可能希望将其设置/proc/sys/vm/nr_overcommit_hugepages
为大约 90% 的 RAM。实际上,在 Linux 上,您可以简单地运行
echo "$(getconf _PHYS_PAGES) * 0.9 / 512" | bc > /proc/sys/vm/nr_overcommit_hugepages
因为大页面每个大小为 2 MB,比默认的 4 KB 页面大 512 倍,您希望将限制设置为可用物理页面的 90%。对需要大量共享 RAM(例如 PostgreSQL)的进程使用大页面可减少内存碎片并减少浪费在页表上的 RAM。请注意,这是运行时更改,您必须在每次重新启动后执行此操作,然后才能启动支持使用大页面的服务。
推荐阅读
- r - 我的 R 脚本没有拾取文件夹中的所有文件
- python - Google 计算引擎 api:创建全局地址
- powershell - PowerShell 字符串插值语法
- c# - 在 DotNetStandard 2.1 项目中找不到 SignalR HubOption MaximumReceiveMessageSize 属性
- java - 启用和禁用 TextView 的可编辑模式
- python - 为什么 aiohttp 需要 await 来获取 body?
- r - 如何单独更改散点图中图例的填充以匹配标签颜色?
- node.js - 如何使用 mongoose 加载 MongoDB 集合中的所有数据?
- javascript - 如何将 Google 表单从模板连接到 Google AppsScript
- android - 比较文件的大小和该文件的字节数组的大小