Linux内核为了提高内存的使用效率采用过度分配内存(over-commit memory)的办法,造成物理内存过度紧张进而触发OOM机制来杀死一些进程回收内存。该机制会监控那些占用内存过大,尤其是瞬间很快消耗大量内存的进程,为了防止内存耗尽会把该进程杀掉。
1、oom过程
out_of_memory函数的代码逻辑还是非常简单清晰的,总共有两步
- 1.先选择一个要杀死的进程,
- 2.杀死它。
oom_kill_process函数的目的很简单,但是实现过程也有点复杂,这里就不展开分析了,大家可以自行去看一下代码。我们重点分析一下select_bad_process函数的逻辑,select_bad_process主要是依靠oom_score来进行进程选择的。
我们先来看一下和oom_score有关的三个文件。
-
/proc//oom_score 系统计算出来的oom_score值,只读文件,取值范围0 –- 1000,0代表never kill,1000代表aways kill,值越大,进程被选中的概率越大。
-
/proc//oom_score_adj 让用户空间调节oom_score的接口,root可读写,取值范围 -1000 — 1000,默认为0,若为 -1000,则oom_score加上此值一定小于等于0,从而变成never kill进程。OS可以把一些关键的系统进程的oom_score_adj设为-1000,从而避免被oom kill。
-
/proc//oom_adj 旧的接口文件,为兼容而保留,root可读写,取值范围 -16 — 15,会被线性映射到oom_score_adj,特殊值 -17代表 OOM_DISABLE。大家尽量不要再用此接口。
Linux内存管理 (21)OOM:https://www.cnblogs.com/arnoldlu/p/8567559.html
2、oom配置
2.1 /proc/sys/vm/overcommit_memory
内核参数 vm.overcommit_memory 接受三种取值:
- 0 – Heuristic overcommit handling. 这是缺省值,它允许overcommit,但过于明目张胆的overcommit会被拒绝,比如malloc一次性申请的内存大小就超过了系统总内存。Heuristic的意思是“试探式的”,内核利用某种算法猜测你的内存申请是否合理,它认为不合理就会拒绝overcommit。
- 1 – Always overcommit. 允许overcommit,对内存申请来者不拒。内核执行无内存过量使用处理。使用这个设置会增大内存超载的可能性,但也可以增强大量使用内存任务的性能。
- 2 – Don’t overcommit. 禁止overcommit。 内存拒绝等于或者大于总可用 swap 大小以及 overcommit_ratio 指定的物理 RAM 比例的内存请求。如果您希望减小内存过度使用的风险,这个设置就是最好的。
遇到问题,线程里面一直fork()执行内容再退出之后,会出现fork失败。
由于fork的时候会拷贝一份父进程的内存,所以如果一开始正常fork到后面fork失败,那就说明一定是这个进程有内存泄露问题,因为内存泄露导致进程内存变大,fork的时候拷贝的父进程内存也就更大,导致内存不足,fork失败。
linux - fork() 因内存不足错误而失败:https://www.coder.work/article/167298
2.2 /proc/sys/vm/panic_on_oom
决定系统出现oom的时候,要做的操作。接受的三种取值如下:
- 0 - 默认值,当出现oom的时候,触发oom killer
- 1 - 程序在有cpuset、memory policy、memcg的约束情况下的OOM,可以考虑不panic,而是启动OOM killer。其它情况触发 kernel panic,即系统直接重启
- 2 - 当出现oom,直接触发kernel panic,即系统直接重启
2.3 /proc/sys/vm/min_free_kbytes
- 代表系统所保留空闲内存的最低限。
min_free_kbytes设的越大,watermark的线越高,同时三个线之间的buffer量也相应会增加。这意味着会较早的启动kswapd进行回收,且会回收上来较多的内存(直至watermark[high]才会停止),这会使得系统预留过多的空闲内存,从而在一定程度上降低了应用程序可使用的内存量。极端情况下设置min_free_kbytes接近内存大小时,留给应用程序的内存就会太少而可能会频繁地导致OOM的发生。
min_free_kbytes设的过小,则会导致系统预留内存过小。kswapd回收的过程中也会有少量的内存分配行为(会设上PF_MEMALLOC)标志,这个标志会允许kswapd使用预留内存;另外一种情况是被OOM选中杀死的进程在退出过程中,如果需要申请内存也可以使用预留部分。这两种情况下让他们使用预留内存可以避免系统进入deadlock状态。
3.vm参数推荐配置
# vm
vm.min_free_kbytes=4096
vm.vfs_cache_pressure=200
vm.dirty_background_ratio=5
vm.dirty_ratio=10
vm.dirty_expire_centisecs=500
vm.dirty_writeback_centisecs=200
vm.extfrag_threshold=10
vm.panic_on_oom=1
/proc/sys/vm/ 文档介绍:https://zhuanlan.zhihu.com/p/503579974?utm_id=0
五万字 | 深入理解Linux内存管理:https://mp.weixin.qq.com/s/nlMGEhuaDUYqV6r8A4cRlA
记一次linux oom内存溢出排查过程:https://blog.csdn.net/hu_jinghui/article/details/81740575
4.都有内存回收机制了,为啥还会oom
- 后台内存回收(kswapd)
- 直接内存回收(direct reclaim)
- OOM机制(Out of Memory)
kswapd 是一个内核线程,在内存不足时负责在后台进行内存回收,这个过程发生在后台,因此是异步发生,不会阻塞进程。
- 当内存大于 pages_low 时,表示此时系统内存足够,不会进行内存回收。
- 当内存小于 pages_low 时,表示此时内存存在压力,会触发 kswapd0 进行后台内存回收,直到 pages_high 为止。
- 当内存小于 pages_min 时,表示此时用户内存耗尽,会触发直接内存回收,进程被阻塞。
如果要调整 kswapd 的触发时机,需要修改 pages_low 的值,而pages_low的值由pages_min计算,因此需要修改pages_min。
相关参数调整
1、内核参数vm.swappiness,决定回收缓存或swap机制回收内存的倾向取值范围是 0-100,
- 数值越大,越积极使用 Swap,也就是更倾向于回收匿名页;
- 数值越小,越消极使用 Swap,也就是更倾向于回收文件页。
- 0不代表不使用swap,当剩余内存 + 文件页小于页高阈值时,还是会发生 Swap。
2、内核参数vm.min_free_kbytes,调整内存水位
- pages_min = min_free_kbytes换算为page单位,
- pages_low = pages_min*5/4
- pages_high = pages_min*3/2
所以当内存消耗的速度比回收的机制快的时候就会直接触发oom,比如dhcp flood驱动没有限制住流量的时候。或者就是内存都被消耗了,实在释放不出来了,那也只能oom。
【Linux内核】内存管理——内存回收机制:https://blog.csdn.net/weixin_45636061/article/details/127184818文章来源:https://www.toymoban.com/news/detail-752739.html
出现oom实际日志:文章来源地址https://www.toymoban.com/news/detail-752739.html
[940027.316884] PTK:be634a2126f098a15a84324cb8fdc92cd05288ac3566b4d14290511de834cbc682d1ff2f7b0ca76ce0e9f956265658f3a8f5e6941538e457bbd020b27733e244
[940027.330955] peer_msg4: 33781 usec
[940029.263407] ACT - SendBSS2040CoexistMgmtAction(BSSCoexist2040=0x4)
[940029.270498] hw_ctrl_flow_v2_peer_update: wdev_idx=0
[940030.898046] p1905_managerd invoked oom-killer: gfp_mask=0x2420848, order=0, oom_score_adj=0
[940030.906882] CPU: 0 PID: 14089 Comm: p1905_managerd Not tainted 4.4.198 #0
[940030.913781] Stack : 81d168c2 0000003d 00000000 00000000 81d168c2 00000000 00000000 00000000
[940030.913781] 8358ecf4 819086e3 817d4110 00000000 00003709 81d13690 00000003 00000001
[940030.913781] 819087e0 8107446c 00000000 00000004 00000006 00000000 817db220 857cb87c
[940030.913781] 00000000 8107213c 81d168c2 0000004f 00000019 0014f000 81d168c2 007cb87c
[940030.913781] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[940030.913781] ...
[940030.949718] Call Trace:
[940030.952288] [<810188b4>] show_stack+0x54/0x88
[940030.956791] [<811e4bd4>] dump_stack+0x8c/0xc4
[940030.961275] [<810b17d8>] dump_header.isra.4+0x54/0x19c
[940030.966546] [<810b1d68>] oom_kill_process+0xf0/0x578
[940030.971643] [<810b2504>] out_of_memory+0x314/0x3a8
[940030.976555] [<810b67e0>] __alloc_pages_nodemask+0x7dc/0x824
[940030.982249] [<810ae69c>] pagecache_get_page+0x1b8/0x288
[940030.987656] [<81122114>] __getblk_slow+0x1ac/0x3e0
[940030.992635] [<81153e80>] squashfs_bio_submit+0x1e4/0x5c8
[940030.998132] [<81154938>] __squashfs_read_data+0x264/0x298
[940031.003710] [<81154a40>] squashfs_read_data_async+0x28/0x34
[940031.009470] [<81158cd0>] squashfs_readpages_block+0x330/0x378
[940031.015404] [<811565c4>] __squashfs_readpages.isra.5+0x6d4/0x914
[940031.021595] [<81156824>] squashfs_readpages+0x20/0x30
[940031.026829] [<810bab20>] __do_page_cache_readahead+0x1a8/0x288
[940031.032860] [<810b0a1c>] filemap_fault+0x1e0/0x500
[940031.037834] [<810d04fc>] __do_fault+0x64/0xd4
[940031.042371] [<810d3f60>] handle_mm_fault+0x544/0xdfc
[940031.047546] [<81021c80>] __do_page_fault+0x138/0x48c
[940031.052689] [<81005420>] ret_from_exception+0x0/0x10
[940031.057788]
[940031.060888] Mem-Info:
[940031.063598] active_anon:1071 inactive_anon:18 isolated_anon:0
[940031.063598] active_file:431 inactive_file:482 isolated_file:70
[940031.063598] unevictable:0 dirty:0 writeback:0 unstable:0
[940031.063598] slab_reclaimable:491 slab_unreclaimable:4198
[940031.063598] mapped:222 shmem:27 pagetables:110 bounce:0
[940031.063598] free:4122 free_pcp:0 free_cma:0
[940031.096924] DMA free:2744kB min:2304kB low:2880kB high:3456kB active_anon:380kB inactive_anon:16kB active_file:200kB inactive_file:236kB unevictable:0kB isolated(anon):0kB isolated(file):100kB present:16384kB managed:16380kB mlocked:0kB dirty:0kB writeback:0kB mapped:40kB shmem:20kB slab_reclaimable:396kB slab_unreclaimable:2520kB kernel_stack:104kB pagetables:44kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[940031.140948] lowmem_reserve[]: 0 97 97
[940031.145100] Normal free:13712kB min:14076kB low:17592kB high:21112kB active_anon:3904kB inactive_anon:56kB active_file:1556kB inactive_file:1488kB unevictable:0kB isolated(anon):0kB isolated(file):352kB present:114688kB managed:99936kB mlocked:0kB dirty:0kB writeback:0kB mapped:792kB shmem:88kB slab_reclaimable:1592kB slab_unreclaimable:14292kB kernel_stack:760kB pagetables:396kB unstable:0kB bounce:0kB free_pcp:100kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:376 all_unreclaimable? no
[940031.190330] lowmem_reserve[]: 0 0 0
[940031.194458] DMA: 94*4kB (U) 76*8kB (UM) 49*16kB (U) 29*32kB (U) 1*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2760kB
[940031.207537] Normal: 345*4kB (UME) 464*8kB (UM) 269*16kB (UM) 59*32kB (M) 28*64kB (UM) 7*128kB (UM) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 13972kB
[940031.221806] 987 total pagecache pages
[940031.225766] 0 pages in swap cache
[940031.229374] Swap cache stats: add 0, delete 0, find 0/0
[940031.234878] Free swap = 0kB
[940031.238101] Total swap = 0kB
[940031.241108] 32768 pages RAM
[940031.244015] 0 pages HighMem/MovableOnly
[940031.248030] 3689 pages reserved
[940031.251292] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[940031.260252] [ 654] 0 654 297 10 4 0 0 0 ubusd
[940031.269171] [ 655] 0 655 225 6 3 0 0 0 askfirst
[940031.278300] [ 1277] 0 1277 548 253 4 0 0 0 logd
[940031.287161] [ 1309] 0 1309 362 13 3 0 0 0 rpcd
[940031.295852] [ 1453] 0 1453 364 41 3 0 0 0 zmsg
[940031.304628] [ 1542] 0 1542 422 34 3 0 0 0 netifd
[940031.313645] [ 1566] 0 1566 351 11 3 0 0 0 odhcpd
[940031.322644] [ 1660] 0 1660 305 22 3 0 0 0 udhcpc
[940031.331616] [ 1672] 0 1672 305 22 4 0 0 0 udhcpc
[940031.340639] [ 1684] 0 1684 256 8 3 0 0 0 odhcp6c
[940031.349639] [ 1788] 0 1788 267 7 3 0 0 0 dropbear
[940031.358853] [ 1804] 0 1804 256 8 3 0 0 0 odhcp6c
[940031.367987] [ 1891] 0 1891 381 28 3 0 0 0 uhttpd
[940031.376973] [ 1933] 0 1933 357 12 4 0 0 0 zdisplay
[940031.386896] [ 1979] 0 1979 635 11 4 0 0 0 zdtool
[940031.396188] [ 2013] 0 2013 879 62 5 0 0 0 zmqttproxy
[940031.405512] [ 2031] 0 2031 311 29 4 0 0 0 zihomescript.sh
[940031.415776] [ 2059] 0 2059 1088 79 4 0 0 0 zapclient
[940031.425109] [ 2160] 0 2160 401 14 4 0 0 0 zdetect
[940031.434347] [ 2185] 0 2185 305 8 3 0 0 0 telnetd
[940031.443588] [ 2207] 0 2207 384 36 3 0 0 0 zdetect
[940031.452739] [ 2272] 453 2272 328 10 4 0 0 0 dnsmasq
[940031.461824] [14087] 0 14087 845 66 5 0 0 0 wapp
[940031.470593] [14088] 0 14088 820 52 5 0 0 0 wapp
[940031.479403] [14089] 0 14089 523 51 4 0 0 0 p1905_managerd
[940031.489111] [14090] 0 14090 470 47 4 0 0 0 mapd
[940031.498003] [31857] 0 31857 228 9 4 0 0 0 htpdate
[940031.507119] [26573] 0 26573 311 15 4 0 0 0 zihomescript.sh
[940031.516833] [26576] 0 26576 446 12 3 0 0 0 curl
[940031.525562] Out of memory: Kill process 1277 (logd) score 8 or sacrifice child
[940031.533175] Killed process 1277 (logd) total-vm:2192kB, anon-rss:1012kB, file-rss:0kB
[940031.677537] wapp invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0
[940031.685326] CPU: 0 PID: 14087 Comm: wapp Not tainted 4.4.198 #0
[940031.691370] Stack : 81d168c2 00000033 00000000 00000000 81d168c2 00000000 00000000 00000000
[940031.691370] 863bbdf4 819086e3 817d4110 00000000 00003707 81d13690 00000003 00000001
[940031.691370] 819087e0 8107446c 00000000 00000004 00000006 00000000 817db220 83683b6c
[940031.691370] 00000000 8107213c 81d168c2 00000045 83683b50 00000000 81d168c2 00683b6c
[940031.691370] 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[940031.691370] ...
[940031.727377] Call Trace:
[940031.729959] [<810188b4>] show_stack+0x54/0x88
[940031.734459] [<811e4bd4>] dump_stack+0x8c/0xc4
[940031.738967] [<810b17d8>] dump_header.isra.4+0x54/0x19c
[940031.744237] [<810b1d68>] oom_kill_process+0xf0/0x578
[940031.749354] [<810b2504>] out_of_memory+0x314/0x3a8
[940031.754300] [<810b66f0>] __alloc_pages_nodemask+0x6ec/0x824
[940031.760036] [<810b0b98>] filemap_fault+0x35c/0x500
[940031.764980] [<810d04fc>] __do_fault+0x64/0xd4
[940031.769475] [<810d3f60>] handle_mm_fault+0x544/0xdfc
[940031.774577] [<81021c80>] __do_page_fault+0x138/0x48c
[940031.779662] [<81005420>] ret_from_exception+0x0/0x10
[940031.784735]
[940039.905445] Mem-Info:
[940039.907867] active_anon:415 inactive_anon:18 isolated_anon:0
[940039.907867] active_file:376 inactive_file:532 isolated_file:32
[940039.907867] unevictable:0 dirty:0 writeback:0 unstable:0
[940039.907867] slab_reclaimable:478 slab_unreclaimable:4239
[940039.907867] mapped:144 shmem:27 pagetables:73 bounce:0
[940039.907867] free:3638 free_pcp:0 free_cma:0
[940039.940722] DMA free:2568kB min:2304kB low:2880kB high:3456kB active_anon:216kB inactive_anon:16kB active_file:528kB inactive_file:784kB unevictable:0kB isolated(anon):0kB isolated(file):128kB present:16384kB managed:16380kB mlocked:0kB dirty:0kB writeback:0kB mapped:252kB shmem:20kB slab_reclaimable:404kB slab_unreclaimable:2516kB kernel_stack:96kB pagetables:32kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:156 all_unreclaimable? no
[940039.984232] lowmem_reserve[]: 0 97 97
[940039.988439] Normal free:11836kB min:14076kB low:17592kB high:21112kB active_anon:1444kB inactive_anon:56kB active_file:976kB inactive_file:1516kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:114688kB managed:99936kB mlocked:0kB dirty:0kB writeback:0kB mapped:324kB shmem:88kB slab_reclaimable:1508kB slab_unreclaimable:14440kB kernel_stack:696kB pagetables:260kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:1840 all_unreclaimable? no
[940040.033091] lowmem_reserve[]: 0 0 0
[940040.036744] DMA: 143*4kB (UM) 116*8kB (U) 74*16kB (UM) 3*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2780kB
[940040.050026] Normal: 569*4kB (UME) 439*8kB (UM) 286*16kB (UM) 8*32kB (UM) 15*64kB (UM) 2*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 11836kB
[940040.064766] 909 total pagecache pages
[940040.068584] 0 pages in swap cache
[940040.072334] Swap cache stats: add 0, delete 0, find 0/0
[940040.078136] Free swap = 0kB
[940040.081589] Total swap = 0kB
[940040.084587] 32768 pages RAM
[940040.087964] 0 pages HighMem/MovableOnly
[940040.092312] 3689 pages reserved
[940040.096087] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[940040.105230] [ 654] 0 654 297 10 4 0 0 0 ubusd
[940040.114380] [ 655] 0 655 225 6 3 0 0 0 askfirst
[940040.123882] [ 1309] 0 1309 362 13 3 0 0 0 rpcd
[940040.132961] [ 1453] 0 1453 364 27 3 0 0 0 zmsg
[940040.141654] [ 1542] 0 1542 422 34 3 0 0 0 netifd
[940040.150504] [ 1566] 0 1566 351 45 3 0 0 0 odhcpd
[940040.160038] [ 1660] 0 1660 305 22 3 0 0 0 udhcpc
[940040.169236] [ 1672] 0 1672 305 22 4 0 0 0 udhcpc
[940040.179054] [ 1684] 0 1684 256 8 3 0 0 0 odhcp6c
[940040.188399] [ 1788] 0 1788 267 7 3 0 0 0 dropbear
[940040.197855] [ 1804] 0 1804 256 8 3 0 0 0 odhcp6c
[940040.207217] [ 1891] 0 1891 381 28 3 0 0 0 uhttpd
[940040.216402] [ 1933] 0 1933 357 21 4 0 0 0 zdisplay
[940040.225873] [ 1979] 0 1979 635 11 4 0 0 0 zdtool
[940040.235113] [ 2031] 0 2031 311 29 4 0 0 0 zihomescript.sh
[940040.245101] [ 2185] 0 2185 305 8 3 0 0 0 telnetd
[940040.254072] [ 2272] 453 2272 328 10 4 0 0 0 dnsmasq
[940040.263456] [14090] 0 14090 470 52 4 0 0 0 mapd
[940040.272473] [31857] 0 31857 228 9 4 0 0 0 htpdate
[940040.281790] [26573] 0 26573 311 15 4 0 0 0 zihomescript.sh
[940040.291889] Out of memory: Kill process 14090 (mapd) score 1 or sacrifice child
[940040.299716] Killed process 14090 (mapd) total-vm:1880kB, anon-rss:208kB, file-rss:0kB
[940047.227817] MacTableDeleteEntry(): wcid 8 =====
[940047.238415] hw_ctrl_flow_v2_disconnt_act: wdev_idx=3
[940085.994632] ACT - SendBSS2040CoexistMgmtAction(BSSCoexist2040=0x6)
[940086.001068] hw_ctrl_flow_v2_peer_update: wdev_idx=0
[940224.529243] ACT - SendBSS2040CoexistMgmtAction(BSSCoexist2040=0x4)
[940224.536168] hw_ctrl_flow_v2_peer_update: wdev_idx=0
[940276.725992] peer_auth_req: 44 usec
[940276.732087] IE_WLAN_EXTENSION: no handler for extension_id:35
[940276.738337] @@@ ap_cmm_peer_assoc_req_action(): (wcid=24), HTC_ICVErrCnt(0), HTC_AAD_OM_Freeze(0), HTC_AAD_OM_CountDown(0), HTC_AAD_OM_Freeze(1) is in Asso. stage!
[940276.753598] add he assoc_rsp, len=59
到了这里,关于5-Linux 内存溢出oom的文章就介绍完了。如果您还想了解更多内容,请在右上角搜索TOY模板网以前的文章或继续浏览下面的相关文章,希望大家以后多多支持TOY模板网!