时间戳

首页 > 解决方案 > MongoDB在并发更新单个文档时分配大量内存

mongodb - MongoDB在并发更新单个文档时分配大量内存

问题描述

谁能向我解释一下 mongo 在同时更新单个文档时分配大量内存的情况。

我在说什么：

我有一份结构如下的文件：

{ id, data: [{id, status}] }

数据字段里面有很多对象（在我的测试用例中是 5000 个对象）。

在测试中，我只是更新单个文档中的每个数据对象。

当我同步执行时，没有额外的内存分配。但是当我并行更新文档时，mongo 分配了超过 2GB 的额外内存。

在第一种情况下（同步）有分配：

{
    "pageheap_free_bytes" : 23552000,
    "pageheap_unmapped_bytes" : 0,
    "max_total_thread_cache_bytes" : NumberLong(1073741824),
    "current_total_thread_cache_bytes" : 994032,
    "total_free_bytes" : 7793496,
    "central_cache_free_bytes" : 4958056,
    "transfer_cache_free_bytes" : 1841408,
    "thread_cache_free_bytes" : 994032,
    "aggressive_memory_decommit" : 0,
    "pageheap_committed_bytes" : 112832512,
    "pageheap_scavenge_count" : 0,
    "pageheap_commit_count" : 85,
    "pageheap_total_commit_bytes" : 112832512,
    "pageheap_decommit_count" : 0,
    "pageheap_total_decommit_bytes" : 0,
    "pageheap_reserve_count" : 85,
    "pageheap_total_reserve_bytes" : 112832512,
    "spinlock_total_delay_ns" : NumberLong(1169495344),
    "formattedString" : "------------------------------------------------\nMALLOC:       81487592 (   77.7 MiB) Bytes in use by application\nMALLOC: +     23552000 (   22.5 MiB) Bytes in page heap freelist\nMALLOC: +      4958056 (    4.7 MiB) Bytes in central cache freelist\nMALLOC: +      1841408 (    1.8 MiB) Bytes in transfer cache freelist\nMALLOC: +       993456 (    0.9 MiB) Bytes in thread cache freelists\nMALLOC: +      2318592 (    2.2 MiB) Bytes in malloc metadata\nMALLOC:   ------------\nMALLOC: =    115151104 (  109.8 MiB) Actual memory used (physical + swap)\nMALLOC: +            0 (    0.0 MiB) Bytes released to OS (aka unmapped)\nMALLOC:   ------------\nMALLOC: =    115151104 (  109.8 MiB) Virtual address space used\nMALLOC:\nMALLOC:           2618              Spans in use\nMALLOC:             25              Thread heaps in use\nMALLOC:           4096              Tcmalloc page size\n------------------------------------------------\nCall ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).\nBytes released to the OS take up virtual address space but no physical memory.\n"
}

第二个：

{
    "pageheap_free_bytes" : 1020854272,
    "pageheap_unmapped_bytes" : 13770752,
    "max_total_thread_cache_bytes" : NumberLong(1073741824),
    "current_total_thread_cache_bytes" : 1996120,
    "total_free_bytes" : 24262888,
    "central_cache_free_bytes" : 19604560,
    "transfer_cache_free_bytes" : 2662208,
    "thread_cache_free_bytes" : 1996120,
    "aggressive_memory_decommit" : 0,
    "pageheap_committed_bytes" : NumberLong(2859118592),
    "pageheap_scavenge_count" : 51,
    "pageheap_commit_count" : 2781,
    "pageheap_total_commit_bytes" : NumberLong(2886373376),
    "pageheap_decommit_count" : 51,
    "pageheap_total_decommit_bytes" : 27254784,
    "pageheap_reserve_count" : 2690,
    "pageheap_total_reserve_bytes" : NumberLong(2872889344),
    "spinlock_total_delay_ns" : NumberLong(158962109950),
    "formattedString" : "------------------------------------------------\nMALLOC:     1814002008 ( 1730.0 MiB) Bytes in use by application\nMALLOC: +   1020854272 (  973.6 MiB) Bytes in page heap freelist\nMALLOC: +     19604560 (   18.7 MiB) Bytes in central cache freelist\nMALLOC: +      2662208 (    2.5 MiB) Bytes in transfer cache freelist\nMALLOC: +      1995544 (    1.9 MiB) Bytes in thread cache freelists\nMALLOC: +     12574976 (   12.0 MiB) Bytes in malloc metadata\nMALLOC:   ------------\nMALLOC: =   2871693568 ( 2738.7 MiB) Actual memory used (physical + swap)\nMALLOC: +     13770752 (   13.1 MiB) Bytes released to OS (aka unmapped)\nMALLOC:   ------------\nMALLOC: =   2885464320 ( 2751.8 MiB) Virtual address space used\nMALLOC:\nMALLOC:          19410              Spans in use\nMALLOC:             28              Thread heaps in use\nMALLOC:           4096              Tcmalloc page size\n------------------------------------------------\nCall ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).\nBytes released to the OS take up virtual address space but no physical memory.\n"
}

如您所见，应用程序使用的字节数和页堆空闲列表中的字节数从 77.7 / 22.5 MB 增加到 1730 / 973.6 MB。

因此，如果我多次运行并行代码，这些数字会线性增加。

谁能解释 mongo 处理并发查询时发生了什么以及如何防止分配？

在我的测试中，我使用的是 mongoDb 4.0.4（这里是一个 docker 映像）和.net core client。

标签： mongodbconcurrencyparallel-processingmongodb-.net-driver

解决方案

当您进行顺序更新时，您使用的是单个线程。这只会使用一个连接。每个连接踏板最多需要 1MB。根据文档

如果您正在进行并行更新，这将有多个连接线程，每个线程将使用 1 MB，这将推动内存使用。

推荐阅读