java - 持久化时，Ignite 无法消耗 WAL 日志并释放 OS 缓冲区
问题描述
持久化时，Ignite 无法消耗 WAL 日志并释放 OS 缓冲区
我有一台 128G 内存的 ignite 服务器，并启用了持久化功能以确保我的数据安全。
正如我从官方文档中得到的那样，我的理解是：当 Persitent 启用时，Ignite 将首先将数据更改保存到 OS 缓冲区（我在 linux 命令 free -mh 中将其检查为 buff/cache），然后写入 WAL 日志，并通过检查点定期处理以分析 WAL 日志并释放已解析的 WAL 日志磁盘空间并释放使用的 OS 缓冲区，如果我错了，请纠正我。
但是在我的测试中，当 Ignite 开始处理流量时，我发现 OS 缓冲区迅速增加，并查看 WAL 日志目录，有很多 wal 日志按顺序生成，几乎与 buff/cache 的大小相同。
[root@Redis1 apache-ignite]# free -mh
total used free shared buff/cache available
Mem: 125G 14G 109G 995M 1.7G 109G
Swap: 127G 0B 127G
127G
仅几分钟，空闲列迅速减少，而 buff/cache 增加
[root@Redis1 apache-ignite]# free -mh
total used free shared buff/cache available
Mem: 125G 15G 85G 995M 25G 108G
Swap: 127G 0B 127G
并且 WAL 日志大小和段数也在不断增加，仍然与 buff/cache 的大小几乎相同。
我检查了 ignite 日志，每 3 分钟检查一次检查点进程审计：
[05:30:05,818][INFO][db-checkpoint-thread-#107][GridCacheDatabaseSharedManager] Checkpoint started [checkpointId=9428aebc-f2b0-4d33-bed6-fb9a1ad49848, startPtr=FileWALPointer [idx=341, fileOff=50223036, len=420491], checkpointLockWait=0ms, checkpointLockHoldTime=860ms, walCpRecordFsyncDuration=245ms, pages=89627, reason='timeout']
[05:30:22,429][INFO][db-checkpoint-thread-#107][GridCacheDatabaseSharedManager] Checkpoint finished [cpId=9428aebc-f2b0-4d33-bed6-fb9a1ad49848, pages=89627, markPos=FileWALPointer [idx=341, fileOff=50223036, len=420491], walSegmentsCleared=0, markDuration=1288ms, pagesWrite=844ms, fsync=15767ms, total=17899ms]
但是对于“free -mh”命令的输出，“free”列无法释放，仍然随着流量的增加而增加，即使我停止流量，它也不会减少，如果我继续发送流量，可用内存不断减少，最后可用内存减少到数百兆字节左右，
[root@Redis1 apache-ignite]# free -mh
total used free shared buff/cache available
Mem: 125G 16G 370M 971M 108G 107G
Swap: 127G 0B 127G
当这种情况发生时（空闲内存耗尽？），我所有基于 ignite 的服务都停止处理我的新请求，对于 ignite，它挂起。
我还注意到带有 reason='timeout' 的 Checkpoint 日志，我不知道这是否代表该 ignite 无法正确解析 WAL 日志和释放 OS 缓存缓冲区？无论如何让检查点正常工作以释放内存？
我的问题是我该如何做一些事情来防止点燃耗尽可用内存，并保持我的服务在持久状态下可用，我发现如果我关闭持久性，非常快速地点燃句柄，并且使用相同的流量缓存使用少于 1G，但是当启用持久标志，操作系统缓存内存快速增加 vert 并耗尽所有可用内存，然后 ignite 无法从此条件恢复并挂起。
我尝试了很多参数，使用 WALMODE、LOG_ONLY 或背景，在 JVM 中设置 -DIGNITE_WAL_MMAP=false，设置 checkpointPageBufferSize，但没有一个可以保存我的 ignite 服务，它仍然会吃掉操作系统缓存并耗尽它。
https://apacheignite.readme.io/docs/write-ahead-log https://apacheignite.readme.io/docs/durable-memory-tuning#section-checkpointing-buffer-size
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<!-- 10 GB initial size. -->
<property name="initialSize" value="#{10L * 1024 * 1024 * 1024}"/>
<!-- 50 GB maximum size. -->
<property name="maxSize" value="#{50L * 1024 * 1024 * 1024}"/>
<property name="persistenceEnabled" value="true"/>
<property name="checkpointPageBufferSize" value="#{1024L * 1024 * 1024}"/>
</bean>
</property>
<property name="writeThrottlingEnabled" value="true"/>
<property name="walMode" value="LOG_ONLY"/>
<property name="walPath" value="/wal/ebc"/>
<property name="walArchivePath" value="/wal/ebc"/>
</bean>
</property>
贝娄是我的缓存配置：
public void createLvOneTxCache() {
CacheConfiguration<String, OrderInfo> cacheCfg =
new CacheConfiguration<>("LvOneTxCache");
cacheCfg.setCacheMode(CacheMode.REPLICATED);
//cacheCfg.setStoreKeepBinary(true);
cacheCfg.setAtomicityMode(ATOMIC);
ebcLvOneTxCache = ignite.getOrCreateCache(cacheCfg);
}
我尝试修改参数，但操作系统缓存仍在增加：
<!-- Enabling Apache Ignite native persistence. -->
<property name="dataStorageConfiguration">
<bean class="org.apache.ignite.configuration.DataStorageConfiguration">
<property name="defaultDataRegionConfiguration">
<bean class="org.apache.ignite.configuration.DataRegionConfiguration">
<!-- 10 GB initial size. -->
<property name="initialSize" value="#{4L * 1024 * 1024 * 1024}"/>
<!-- 50 GB maximum size. -->
<property name="maxSize" value="#{4L * 1024 * 1024 * 1024}"/>
<property name="persistenceEnabled" value="true"/>
<property name="checkpointPageBufferSize" value="#{4L * 1024 * 1024 * 1024}"/>
</bean>
</property>
<property name="checkpointFrequency" value="6000"/>
<property name="checkpointThreads" value="32"/>
<property name="writeThrottlingEnabled" value="true"/>
<property name="walMode" value="LOG_ONLY"/>
<property name="walPath" value="/wal/ebc"/>
<property name="walArchivePath" value="/wal/ebc"/>
</bean>
</property>
并快速启动日志显示审计，但缓存也不会被释放。
[07:51:20,165][INFO][db-checkpoint-thread-#108][GridCacheDatabaseSharedManager] Checkpoint started [checkpointId=fd0c7e68-564a-4b40-9516-bb2a451869e7, startPtr=FileWALPointer [idx=23, fileOff=47849256, len=420491], checkpointLockWait=0ms, checkpointLockHoldTime=77ms, walCpRecordFsyncDuration=233ms, pages=7744, reason='timeout']
[07:51:20,219][INFO][sys-stripe-0-#1][PageMemoryImpl] Throttling is applied to page modifications [percentOfPartTime=0.36, markDirty=16378 pages/sec, checkpointWrite=3322 pages/sec, estIdealMarkDirty=673642 pages/sec, curDirty=0.00, maxDirty=0.40, avgParkTime=21501 ns, pages: (total=7744, evicted=0, written=7744, synced=229, cpBufUsed=0, cpBufTotal=1036430)]
[07:51:22,303][INFO][db-checkpoint-thread-#108][GridCacheDatabaseSharedManager] Checkpoint finished [cpId=fd0c7e68-564a-4b40-9516-bb2a451869e7, pages=7744, markPos=FileWALPointer [idx=23, fileOff=47849256, len=420491], walSegmentsCleared=0, markDuration=317ms, pagesWrite=24ms, fsync=2114ms, total=2456ms]
[07:51:26,117][INFO][db-checkpoint-thread-#108][GridCacheDatabaseSharedManager] Checkpoint started [checkpointId=d64991bc-3d2f-4f2c-8175-d7e92f46f0bf, startPtr=FileWALPointer [idx=25, fileOff=35951286, len=420491], checkpointLockWait=0ms, checkpointLockHoldTime=49ms, walCpRecordFsyncDuration=200ms, pages=7605, reason='timeout']
[07:51:28,612][INFO][db-checkpoint-thread-#108][GridCacheDatabaseSharedManager] Checkpoint finished [cpId=d64991bc-3d2f-4f2c-8175-d7e92f46f0bf, pages=7605, markPos=FileWALPointer [idx=25, fileOff=35951286, len=420491], walSegmentsCleared=0, markDuration=266ms, pagesWrite=23ms, fsync=2472ms, total=2761ms]
[07:51:32,118][INFO][db-checkpoint-thread-#108][GridCacheDatabaseSharedManager] Checkpoint started [checkpointId=07246861-57ae-4ef5-8419-cb7710d2f72d, startPtr=FileWALPointer [idx=27, fileOff=38042090, len=420491], checkpointLockWait=6ms, checkpointLockHoldTime=60ms, walCpRecordFsyncDuration=185ms, pages=7186, reason='timeout']
[07:51:32,121][INFO][service-#232][PageMemoryImpl] Throttling is applied to page modifications [percentOfPartTime=0.24, markDirty=10738 pages/sec, checkpointWrite=2757 pages/sec, estIdealMarkDirty=310976 pages/sec, curDirty=0.00, maxDirty=0.07, avgParkTime=358945 ns, pages: (total=7186, evicted=0, written=896, synced=0, cpBufUsed=565, cpBufTotal=1036430)]
[07:51:34,534][INFO][db-checkpoint-thread-#108][GridCacheDatabaseSharedManager] Checkpoint finished [cpId=07246861-57ae-4ef5-8419-cb7710d2f72d, pages=7186, markPos=FileWALPointer [idx=27, fileOff=38042090, len=420491], walSegmentsCleared=0, markDuration=257ms, pagesWrite=29ms, fsync=2387ms, total=2679ms]
[07:51:38,169][INFO][db-checkpoint-thread-#108][GridCacheDatabaseSharedManager] Checkpoint started [checkpointId=44e6870a-e370-4bd3-8ad9-8252abb0acd3, startPtr=FileWALPointer [idx=29, fileOff=44462293, len=420491], checkpointLockWait=0ms, checkpointLockHoldTime=76ms, walCpRecordFsyncDuration=210ms, pages=7529, reason='timeout']
[07:51:40,668][INFO][db-checkpoint-thread-#108][GridCacheDatabaseSharedManager] Checkpoint finished [cpId=44e6870a-e370-4bd3-8ad9-8252abb0acd3, pages=7529, markPos=FileWALPointer [idx=29, fileOff=44462293, len=420491], walSegmentsCleared=0, markDuration=303ms, pagesWrite=24ms, fsync=2475ms, total=2802ms]
[root@Redis1 node00-296a5110-74ad-45e0-bf9c-5c075a4f5fdf]# free -mh
total used free shared buff/cache available
Mem: 125G 14G 107G 995M 3.5G 109G
Swap: 127G 0B 127G
[root@Redis1 node00-296a5110-74ad-45e0-bf9c-5c075a4f5fdf]# free -mh
total used free shared buff/cache available
Mem: 125G 14G 107G 995M 3.5G 109G
Swap: 127G 0B 127G
[root@Redis1 node00-296a5110-74ad-45e0-bf9c-5c075a4f5fdf]# free -mh
total used free shared buff/cache available
Mem: 125G 14G 107G 995M 3.5G 109G
Swap: 127G 0B 127G
[root@Redis1 node00-296a5110-74ad-45e0-bf9c-5c075a4f5fdf]# free -mh
total used free shared buff/cache available
Mem: 125G 14G 105G 995M 5.6G 109G
Swap: 127G 0B 127G
当我停止流量更新缓存时，我发现操作系统缓存回来了，但非常非常慢，需要很长时间才能释放，快速 checkpointFrequency 6s。怎么可能快速处理？
[root@Redis1 node00-296a5110-74ad-45e0-bf9c-5c075a4f5fdf]# free -mh
total used free shared buff/cache available
Mem: 125G 14G 104G 995M 6.5G 109G
Swap: 127G 0B 127G
[root@Redis1 node00-296a5110-74ad-45e0-bf9c-5c075a4f5fdf]# free -mh
total used free shared buff/cache available
Mem: 125G 14G 104G 995M 6.3G 109G
Swap: 127G 0B 127G
[root@Redis1 node00-296a5110-74ad-45e0-bf9c-5c075a4f5fdf]# free -mh
total used free shared buff/cache available
Mem: 125G 14G 104G 995M 6.3G 109G
Swap: 127G 0B 127G
[root@Redis1 node00-296a5110-74ad-45e0-bf9c-5c075a4f5fdf]# free -mh
total used free shared buff/cache available
Mem: 125G 14G 106G 995M 4.6G 109G
Swap: 127G 0B 127G
[root@Redis1 node00-296a5110-74ad-45e0-bf9c-5c075a4f5fdf]# free -mh
total used free shared buff/cache available
Mem: 125G 14G 106G 995M 4.4G 109G
解决方案
操作系统缓存磁盘数据完全没问题，这里很好解释linux ate my ram。如果你的内核支持，你总是可以设置空闲内存量，这可以减少 Ignite 分配新内存块时的暂停
推荐阅读
- python - Pandas: best way to add a total row that calculates the sum of specific (multiple) columns while preserving the data type
- azure-data-factory - 通过 ADF 处理后从 ADL 中删除文件
- pandas - 计算列中每对值的列表交集
- swift - How to get File Navigator View in MacOS App
- python - 将 git 版本控制与 jupyter 笔记本一起使用
- python - 如何在 Python 3.xx 中重用一个输入并将其添加到列表中
- java - JXLS - add formula for blank cells, formula just disappears
- azure-devops - How to update work item state after a release
- firebase - Firebase 存储文件上传参考网址安全
- javascript - how to find common name objects in json using javascript