时间戳

首页 > 解决方案 > 使用手动创建的 Buffer 的 NodeJS 内存泄漏

javascript - 使用手动创建的 Buffer 的 NodeJS 内存泄漏

问题描述

我正在尝试创建要通过套接字发送的缓冲区数据。（接收方需要它只是缓冲区）。因此，要创建要发送的最终缓冲区，我在两者之间创建不同的缓冲区，然后将它们连接起来以生成要发送的最后一个缓冲区。另一个棘手的问题是我还需要分别计算和发送缓冲区长度。请在下面找到代码：

function BufferGenerator() {
  this.generateData = generateData;
  this.sendData = sendData;
  //....
}
function generateData(requestObj, params, paramsLength) {
  let uBuf = Buffer.from(requestObj.user.login);
  let passBuf = Buffer.from(requestObj.user.pass);
  let nameBuf = Buffer.from(requestObj.user.name);
  const MsgLen = uBuf.length + passBuf.length + nameBuf.length + paramsLength;
  return Buffer.concat([uBuf, passBuf, nameBuf]);
}

function sendData(requestObj, data) {
  const pBufs = [];
  let pLength = 0;
    _.forEach(data, function (value, key) { // this usually has ~15-20 items in data.
      const kBuf = Buffer.from(key);
      const kLengthBuf = Buffer.alloc(4);
      kLengthBuf.writeInt32LE(key.length);

      const vBuf = Buffer.from(value);
      const valBufLength = Buffer.alloc(4);
      valBufLength.writeInt32LE(value.length);

      pLength += kLengthBuf.length + valBufLength.length + kBuf.length + vBuf.length;
      pBufs.push(kLengthBuf, valBufLength, kBuf, vBuf);
    });
  const finalBuf = this.generateData(requestObj, Buffer.concat(pBufs), pLength);
  this.sendToChannel(finalBuf);
}

现在，当我正常运行应用程序时，一切正常。但是当我在性能测试中运行它时，每秒 100 个并发请求，应用程序慢慢开始吃越来越多的内存，最终抛出内存溢出异常。挖掘后，我发现此内存泄漏或内存溢出有 2 种可能的原因：

  1. 我正在创建的缓冲区没有被正确地垃圾收集，因为当我注释那部分代码时，没有内存溢出或泄漏。
  2. GC 中频繁触发 Scavenge 事件，但不会清除完整分配的内存。在这期间我还看到了来自 GC 的 Mark-sweep 事件，这个事件确实清除了更多的内存，但最后仍然有一些剩余的东西慢慢加起来，最终看到内存溢出。

顺便说一句，当负载停止时，内存开始急剧下降，尽管我可以看到开始内存增加了一些内存。例如，起始内存是 18 MB，当开始加载时，我可以看到它增加并上升到 100MB，之后如果我停止加载，它会在几秒钟内回落到 ~20MB 和一堆GC 日志的打印速度非常快。

对于我的性能测试（100 个用户），我可以在大约 8 分钟内看到内存溢出异常。

到目前为止，我尝试了以下事情：

有人可以在这里提供帮助并建议一种方法来清除内存并避免超出范围的错误吗？

编辑 1

最后几个标记扫描日志：

/Users/harsh.r/.nvm/versions/node/v10.15.3/bin/node --trace_gc --expose_gc --trace-gc-ignore-scavenger --max-old-space-size=128 --max-semi-space-size=2 server.js
[20904:0x10264a000]      717 ms: Mark-sweep 11.6 (18.9) -> 9.2 (18.4) MB, 2.6 / 0.2 ms  (+ 4.4 ms in 7 steps since start of marking, biggest step 2.6 ms, walltime since start of marking 58 ms) (average mu = 1.000, current mu = 1.000) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]     1334 ms: Mark-sweep 22.0 (27.9) -> 19.1 (27.8) MB, 2.6 / 0.4 ms  (+ 3.3 ms in 9 steps since start of marking, biggest step 0.7 ms, walltime since start of marking 86 ms) (average mu = 0.991, current mu = 0.991) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]    19206 ms: Mark-sweep 28.0 (34.8) -> 23.3 (35.3) MB, 2.9 / 0.0 ms  (+ 4.7 ms in 13 steps since start of marking, biggest step 0.8 ms, walltime since start of marking 462 ms) (average mu = 0.999, current mu = 1.000) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]    39027 ms: Mark-sweep 31.8 (37.3) -> 23.9 (37.3) MB, 2.8 / 0.0 ms  (+ 3.7 ms in 8 steps since start of marking, biggest step 1.0 ms, walltime since start of marking 182 ms) (average mu = 1.000, current mu = 1.000) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]    50313 ms: Mark-sweep 28.5 (37.8) -> 22.9 (31.8) MB, 6.2 / 0.0 ms  (+ 4.6 ms in 11 steps since start of marking, biggest step 1.3 ms, walltime since start of marking 530 ms) (average mu = 0.999, current mu = 0.999) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]    61607 ms: Mark-sweep 28.4 (33.3) -> 23.1 (33.3) MB, 1.7 / 0.0 ms  (+ 6.5 ms in 14 steps since start of marking, biggest step 4.2 ms, walltime since start of marking 642 ms) (average mu = 0.999, current mu = 0.999) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]    77427 ms: Mark-sweep 30.9 (39.3) -> 24.1 (39.3) MB, 1.4 / 0.0 ms  (+ 3.9 ms in 16 steps since start of marking, biggest step 0.5 ms, walltime since start of marking 750 ms) (average mu = 1.000, current mu = 1.000) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]    95941 ms: Mark-sweep 32.2 (40.3) -> 24.5 (40.3) MB, 1.8 / 0.0 ms  (+ 7.6 ms in 14 steps since start of marking, biggest step 2.0 ms, walltime since start of marking 431 ms) (average mu = 1.000, current mu = 0.999) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]   114741 ms: Mark-sweep 34.4 (42.3) -> 26.1 (42.3) MB, 3.1 / 0.0 ms  (+ 4.7 ms in 17 steps since start of marking, biggest step 0.8 ms, walltime since start of marking 221 ms) (average mu = 1.000, current mu = 1.000) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]   137120 ms: Mark-sweep 34.3 (44.8) -> 26.2 (45.3) MB, 3.8 / 0.0 ms  (+ 11.1 ms in 17 steps since start of marking, biggest step 1.7 ms, walltime since start of marking 249 ms) (average mu = 0.999, current mu = 0.999) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]   156384 ms: Mark-sweep 36.1 (48.3) -> 27.9 (48.3) MB, 2.8 / 0.1 ms  (+ 5.2 ms in 14 steps since start of marking, biggest step 1.3 ms, walltime since start of marking 456 ms) (average mu = 1.000, current mu = 1.000) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]   173797 ms: Mark-sweep 35.0 (49.3) -> 27.2 (35.8) MB, 13.4 / 0.1 ms  (+ 6.0 ms in 16 steps since start of marking, biggest step 1.0 ms, walltime since start of marking 722 ms) (average mu = 0.999, current mu = 0.999) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]   176049 ms: Mark-sweep 29.5 (36.8) -> 28.1 (33.8) MB, 9.9 / 0.0 ms  (+ 5.5 ms in 9 steps since start of marking, biggest step 1.7 ms, walltime since start of marking 260 ms) (average mu = 0.999, current mu = 0.993) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]   187301 ms: Mark-sweep 32.6 (37.8) -> 28.3 (37.8) MB, 1.7 / 0.0 ms  (+ 20.7 ms in 9 steps since start of marking, biggest step 11.5 ms, walltime since start of marking 164 ms) (average mu = 0.998, current mu = 0.998) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]   206234 ms: Mark-sweep 36.4 (44.8) -> 28.8 (44.8) MB, 1.7 / 0.0 ms  (+ 5.8 ms in 13 steps since start of marking, biggest step 1.3 ms, walltime since start of marking 336 ms) (average mu = 0.999, current mu = 1.000) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]   226766 ms: Mark-sweep 36.9 (47.3) -> 29.3 (47.3) MB, 2.0 / 0.0 ms  (+ 5.1 ms in 15 steps since start of marking, biggest step 0.7 ms, walltime since start of marking 588 ms) (average mu = 0.999, current mu = 1.000) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]   246911 ms: Mark-sweep 39.2 (49.8) -> 30.7 (50.3) MB, 4.7 / 0.0 ms  (+ 7.1 ms in 15 steps since start of marking, biggest step 1.4 ms, walltime since start of marking 265 ms) (average mu = 0.999, current mu = 0.999) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]   275394 ms: Mark-sweep 39.0 (52.3) -> 30.9 (52.3) MB, 4.1 / 0.0 ms  (+ 5.2 ms in 16 steps since start of marking, biggest step 1.0 ms, walltime since start of marking 709 ms) (average mu = 1.000, current mu = 1.000) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]   295820 ms: Mark-sweep 38.4 (51.8) -> 31.2 (37.8) MB, 25.9 / 0.0 ms  (+ 8.1 ms in 13 steps since start of marking, biggest step 2.4 ms, walltime since start of marking 873 ms) (average mu = 0.999, current mu = 0.998) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]   297405 ms: Mark-sweep 33.2 (38.8) -> 31.8 (38.3) MB, 3.7 / 0.0 ms  (+ 5.7 ms in 16 steps since start of marking, biggest step 1.2 ms, walltime since start of marking 239 ms) (average mu = 0.999, current mu = 0.994) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]   319076 ms: Mark-sweep 40.5 (48.3) -> 32.9 (48.3) MB, 3.2 / 0.0 ms  (+ 7.1 ms in 14 steps since start of marking, biggest step 1.5 ms, walltime since start of marking 280 ms) (average mu = 0.999, current mu = 1.000) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]   340409 ms: Mark-sweep 42.6 (50.8) -> 33.7 (50.8) MB, 10.8 / 0.1 ms  (+ 9.8 ms in 14 steps since start of marking, biggest step 3.1 ms, walltime since start of marking 467 ms) (average mu = 0.999, current mu = 0.999) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]   348817 ms: Mark-sweep 37.8 (51.8) -> 33.5 (51.8) MB, 31.8 / 0.1 ms  (average mu = 0.998, current mu = 0.996) heap profiler GC in old space requested
[20904:0x10264a000]   348850 ms: Mark-sweep 33.5 (51.8) -> 33.3 (50.8) MB, 33.3 / 0.0 ms  (average mu = 0.996, current mu = 0.001) heap profiler GC in old space requested
[20904:0x10264a000]   368660 ms: Mark-sweep 42.0 (52.3) -> 34.4 (52.8) MB, 7.9 / 0.0 ms  (+ 5.1 ms in 14 steps since start of marking, biggest step 0.9 ms, walltime since start of marking 330 ms) (average mu = 0.998, current mu = 0.999) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]   389645 ms: Mark-sweep 43.2 (54.8) -> 34.5 (55.3) MB, 4.1 / 0.0 ms  (+ 4.9 ms in 15 steps since start of marking, biggest step 0.8 ms, walltime since start of marking 548 ms) (average mu = 0.999, current mu = 1.000) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]   407648 ms: Mark-sweep 43.8 (57.3) -> 35.2 (57.3) MB, 9.5 / 0.0 ms  (+ 4.5 ms in 16 steps since start of marking, biggest step 0.7 ms, walltime since start of marking 531 ms) (average mu = 0.999, current mu = 0.999) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]   421509 ms: Mark-sweep 41.3 (59.8) -> 35.7 (59.8) MB, 49.5 / 0.0 ms  (average mu = 0.998, current mu = 0.996) heap profiler GC in old space requested
[20904:0x10264a000]   421555 ms: Mark-sweep 35.7 (59.8) -> 35.7 (59.8) MB, 45.9 / 0.0 ms  (average mu = 0.995, current mu = 0.001) heap profiler GC in old space requested
[20904:0x10264a000]   444721 ms: Mark-sweep 44.3 (61.8) -> 36.5 (61.8) MB, 10.1 / 0.0 ms  (+ 6.1 ms in 14 steps since start of marking, biggest step 1.0 ms, walltime since start of marking 427 ms) (average mu = 0.998, current mu = 0.999) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]   466438 ms: Mark-sweep 44.4 (63.8) -> 37.0 (63.8) MB, 2.9 / 0.0 ms  (+ 17.9 ms in 13 steps since start of marking, biggest step 12.0 ms, walltime since start of marking 536 ms) (average mu = 0.999, current mu = 0.999) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]   490390 ms: Mark-sweep 46.5 (65.8) -> 38.2 (66.3) MB, 5.6 / 0.0 ms  (+ 5.4 ms in 15 steps since start of marking, biggest step 1.0 ms, walltime since start of marking 585 ms) (average mu = 0.999, current mu = 1.000) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]   508953 ms: Mark-sweep 45.8 (68.3) -> 38.4 (68.3) MB, 60.6 / 8.5 ms  (average mu = 0.998, current mu = 0.997) heap profiler GC in old space requested
[20904:0x10264a000]   509004 ms: Mark-sweep 38.4 (68.3) -> 38.2 (68.3) MB, 51.6 / 0.0 ms  (average mu = 0.995, current mu = 0.002) heap profiler GC in old space requested
[20904:0x10264a000]   536930 ms: Mark-sweep 47.0 (70.3) -> 39.9 (70.8) MB, 21.1 / 0.0 ms  (+ 10.5 ms in 20 steps since start of marking, biggest step 3.6 ms, walltime since start of marking 61 ms) (average mu = 0.998, current mu = 0.999) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]   572032 ms: Mark-sweep 47.9 (72.8) -> 40.2 (72.8) MB, 13.1 / 0.7 ms  (+ 13.0 ms in 9 steps since start of marking, biggest step 4.4 ms, walltime since start of marking 236 ms) (average mu = 0.999, current mu = 0.999) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]   590105 ms: Mark-sweep 49.1 (74.8) -> 41.0 (74.8) MB, 5.0 / 0.0 ms  (+ 13.7 ms in 9 steps since start of marking, biggest step 8.3 ms, walltime since start of marking 73 ms) (average mu = 0.999, current mu = 0.999) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]   609718 ms: Mark-sweep 49.2 (76.8) -> 41.3 (76.8) MB, 25.2 / 0.0 ms  (+ 10.0 ms in 10 steps since start of marking, biggest step 2.9 ms, walltime since start of marking 64 ms) (average mu = 0.999, current mu = 0.998) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]   629874 ms: Mark-sweep 50.6 (78.8) -> 41.7 (78.8) MB, 6.4 / 0.1 ms  (+ 7.2 ms in 5 steps since start of marking, biggest step 3.6 ms, walltime since start of marking 204 ms) (average mu = 0.999, current mu = 0.999) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]   649548 ms: Mark-sweep 51.2 (80.8) -> 43.0 (81.3) MB, 9.9 / 0.0 ms  (+ 6.7 ms in 13 steps since start of marking, biggest step 1.8 ms, walltime since start of marking 193 ms) (average mu = 0.999, current mu = 0.999) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]   672509 ms: Mark-sweep 51.5 (83.8) -> 43.4 (83.8) MB, 4.9 / 0.0 ms  (+ 4.7 ms in 6 steps since start of marking, biggest step 2.0 ms, walltime since start of marking 199 ms) (average mu = 0.999, current mu = 1.000) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]   684486 ms: Mark-sweep 52.0 (84.8) -> 40.2 (84.8) MB, 18.8 / 0.0 ms  (+ 11.2 ms in 23 steps since start of marking, biggest step 3.1 ms, walltime since start of marking 40 ms) (average mu = 0.999, current mu = 0.997) finalize incremental marking via stack guard GC in old space requested
[20904:0x10264a000]   685065 ms: Mark-sweep 53.3 (84.8) -> 29.9 (81.3) MB, 9.0 / 0.0 ms  (+ 9.6 ms in 25 steps since start of marking, biggest step 1.6 ms, walltime since start of marking 29 ms) (average mu = 0.998, current mu = 0.968) finalize incremental marking via task GC in old space requested

我们可以看到内存正在缓慢增长，即使标记清除事件正在发生。

一旦我停止加载，日志将变为：

[20904:0x10264a000]   685065 ms: Mark-sweep 53.3 (84.8) -> 29.9 (81.3) MB, 9.0 / 0.0 ms  (+ 9.6 ms in 25 steps since start of marking, biggest step 1.6 ms, walltime since start of marking 29 ms) (average mu = 0.998, current mu = 0.968) finalize incremental marking via task GC in old space requested
[20904:0x10264a000]   708577 ms: Mark-sweep 39.0 (81.3) -> 21.6 (31.3) MB, 45.7 / 0.0 ms  (+ 11.4 ms in 16 steps since start of marking, biggest step 1.2 ms, walltime since start of marking 58 ms) (average mu = 0.998, current mu = 0.998) finalize incremental marking via task GC in old space requested
[20904:0x10264a000]   709610 ms: Mark-sweep 21.6 (31.3) -> 21.4 (27.3) MB, 12.9 / 0.0 ms  (+ 15.9 ms in 17 steps since start of marking, biggest step 2.3 ms, walltime since start of marking 29 ms) (average mu = 0.996, current mu = 0.972) finalize incremental marking via task GC in old space requested
[20904:0x10264a000]   710632 ms: Mark-sweep 21.4 (27.3) -> 21.4 (27.3) MB, 6.6 / 0.0 ms  (+ 11.0 ms in 14 steps since start of marking, biggest step 1.6 ms, walltime since start of marking 18 ms) (average mu = 0.995, current mu = 0.983) finalize incremental marking via task GC in old space requested

记忆下来了。

标签： javascriptnode.jsmemory-managementmemory-leaksbuffer

解决方案

推荐阅读