时间戳

首页 > 解决方案 > What is the best approach to autonomously maintain a state of available workers

node.js - What is the best approach to autonomously maintain a state of available workers

问题描述

Currently how my system is designed I spool up a 1-core instance that is only allowed one encoding job. This instance connects to redis and waits for a list of encoding jobs to be populated. The worker then pops the job from the list and sets itself to busy while the job is being ran, as soon as the job is done- it continues looking for more unfinished jobs.

My example worker.js

module.exports = function (redisClient) {
  const spawn = require('child_process').spawn;
  var busy = false;
  function startCoStream(job) {
    console.log(job);
    busy = true;
    var proc = spawn('./costream.sh', [job.streamKey, job.leftIngest, job.leftKey, job.rightIngest, job.rightKey]);
    proc.on('exit', function (code, signal) {
      busy = false;
      readJobBus();
    });
    proc.stderr.setEncoding('utf8');
    proc.stderr.on('data', (chunk) => { 
      console.log(chunk);
    });
  }
  function readJobBus () {
    if (busy) return;
    redisClient.lpop('jobbus', function(err, reply) {
      if (!reply && !err) {
        setTimeout(readJobBus, 1000);
      }
      else {
        var job = JSON.parse(reply);
        if (job.type==0) {
          startCoStream(job);
        }
      }
    });
  }
  readJobBus();
  return redisClient;
}

This runs great in my current setup, the only problem is I need to know ahead of time the amount of encoding processes that are anticipated and provision the instances manually. My question is this, what is the best way of keeping this system autonomous and loosely coupled while maintaining a list of busy and available workers in order for me to implement auto-scaling.

Idea 1) Register the worker on a available worker list upon start, then remove from list upon the starting of the process and adding back once the process is done. Problem 1) What if the worker dies while still on the available list, then the list would show 1 more available worker then the real state of the application.

What would be a good approach to this? The desired outcome is simple. When a worker is provisioned it connects to redis and places itself on a worker list, when the worker list has more then half the workers as busy, then provision more workers. What would be the best way to maintain this list without a dedicated system maintaining the list.

标签： node.jsredisarchitectureload-balancing

解决方案

当然，最简单的方法是拥有一个对此进行评估的主管节点。否则，您将不得不将该角色分配给您的工作节点。我建议您让每个等待工作的空闲节点检查工作人员列表并测试是否需要配置更多工作人员。

这很棘手，因为您必须使用分布式锁来防止多个工作人员同时执行此任务。

  1. 尝试获取主管锁，如果被阻塞，转到#4
  2. 检查忙碌工人的数量，并在必要时提供更多
  3. 释放主管锁
  4. 找工作

推荐阅读