node.js - “集群”（多进程）环境中的“JavaScript 堆内存不足”问题

Question

我正在开发一个从数据源获取大量数据的服务器功能，并且由于获取的数据的大小，我的堆内存似乎溢出了。

我正在尝试增加max-old-space-size我的子进程，但似乎子进程只分配默认空间大小而不是给定的选项大小。

我就是这样设置的。

const {THREADS, NODE_ENV} = process.env;

cluster.schedulingPolicy = cluster.SCHED_RR;

if(cluster.isMaster){
  const forks = (NODE_ENV==="development")? 1: Math.min(os.cpus().length, Math.max(THREADS || 4));

  for (let i =0; i<forks; i++){
    let worker = cluster.fork("./server/app.js", {
      execArgv: ['--max-old-space-size=4096'],
      env: Object.assign({"NODE_OPTIONS" : "--max-old-space-size=4096"})
    });
    worker.on("message", m=>{
      console.log(m);
    });

    worker.on("exit", (_,signal,__)=>{
      console.log(`Server node Died :: ${signal}`);
      cluster.fork("./server/app.js", {
        execArgv: ['--max-old-space-size=4096'],
        env: Object.assign({"NODE_OPTIONS" : "--max-old-space-size=4096"})
      });
    })

  }
  console.log("Server started..")

}

if (cluster.isWorker){

  console.log("Worker node has :: ");
  console.log(process.memoryUsage())
  console.log(`Memory usage per node :: ${Math.floor(process.memoryUsage().rss/10000000)/10} GB`);
  console.log(v8.getHeapStatistics());


  const server = new ApolloServer({
    schema: makeExecutableSchema({typeDefs, resolvers}),
    dataSources: ()=>({
      ... // my datasources
    }),
    context:({req, res})=>{
      if(!req.body.query.includes("IntrospectionQuery")){
        console.log(`${req.method} ${req.url} ${req.body.query}` );
      }
      return{...req}
    }
  });

  server.listen().then(({url})=>{
    // Do nothing
  });

}

这就是从控制台得到的。

Server started..
Worker node has :: 
{ rss: 129269760,
  heapTotal: 100978688,
  heapUsed: 67195888,
  external: 511759 }
Memory usage per node :: 1.2 GB
{ total_heap_size: 100978688,
  total_heap_size_executable: 3145728,
  total_physical_size: 98869008,
  total_available_size: 1439367280,
  used_heap_size: 67240920,
  heap_size_limit: 1526909922,
  malloced_memory: 16384,
  peak_malloced_memory: 3562032,
  does_zap_garbage: 0 }

这是我发送请求时得到的。

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
[7501:0x102808000]    35581 ms: Mark-sweep 1395.8 (1424.9) -> 1395.4 (1424.4) MB, 934.0 / 0.0 ms  (average mu = 0.095, current mu = 0.018) allocation failure scavenge might not succeed
 1: 0x10003cf99 node::Abort() [...]
[7501:0x102808000]    36519 ms: Mark-sweep 1396.2 (1424.4) -> 1395.8 (1424.4) MB, 936.2 / 0.0 ms  (average mu = 0.051, current mu = 0.003) allocation failure scavenge might not succeed

 2: 0x10003d1a3 node::OnFatalError(char const*, char const*) [~/.nvm/versions/node/v10.16.0/bin/node]

 3: 0x1001b7835 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [~/.nvm/versions/node/v10.16.0/bin/node]
<--- JS stacktrace --->

 4: 0x100585682 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [~/.nvm/versions/node/v10.16.0/bin/node]
==== JS stack trace =========================================
 5: 0x100588155 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [~/.nvm/versions/node/v10.16.0/bin/node]

    0: ExitFrame [pc: 0x22a4085dbe3d]
Security context: 0x122e5461e6e9 <JSObject>
 6: 0x100583fff v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [~/.nvm/versions/node/v10.16.0/bin/node]
...

标签： node.jschild-process