javascript - 如何迭代 URL 数组并在移动到下一步之前等待响应

Question

我有一个包含 900 多个 url 的列表，我想为每天的 cron 作业获取这些 url，并且我正在使用 Express.js。我在网上查了一种方法来做到这一点，并遇到了这个 jsinclair page，所以我尝试实现它，但到目前为止还没有奏效。假设这是我的清单：

let arrayOfAsyncTasks = [
    {task: 'fetch', url: 'http://thatapi.com/api1/thing'},
    {task: 'wait', duration: 60000 },
    {task: 'fetch', url: 'http://thatapi.com/api2/thing'},
    {task: 'wait', duration: 60000 },
    ...
]

我试图做的是Array.reduce在开始获取下一个 url 之前强制迭代坚持 60 秒的延迟，如下所示：

const starterPromise = Promise.resolve(null);
const log = result => console.log(result);

arrayOfAsyncTasks.reduce(
    (p, spec) => p.then(() => {
        runTask(spec)
        .then(result => {
            console.log(`Inside reduce function after runTask! result: `, result)
            log(result)
        })
     })
     .catch(err => console.log(`err: `, err) ),
     starterPromise
    );

以下是辅助函数：

function asyncTimeout(delay) {
    console.log(`inside asyncTimeout! delay: `, delay);
    return (new Promise(resolve => {setTimeout(() => resolve(delay), delay)}))
        .then(d => `Waited ${d} seconds`);
}

function asyncFetch(url) {
    console.log(`inside asyncFetch!  url: `, url);
    scrapeEbayCategory(url)
        .then(response => (response.json()))
        .then(text => `Fetched ${url}, and got back ${text}` );
}

function runTask(spec) {
    return (spec.task === 'wait')
    ? asyncTimeout(spec.duration)
    : asyncFetch(spec.url);
}

结果是它移动得太快了到下一个项目，然后在控制台中显示了多个延迟完成的控制台日志：

Waited 60000 seconds
Inside reduce function after runTask! result:  Waited 60000 seconds
Waited 60000 seconds
Inside reduce function after runTask! result:  Waited 60000 seconds
Waited 60000 seconds
Inside reduce function after runTask! result:  Waited 60000 seconds
Waited 60000 seconds
Inside reduce function after runTask! result:  Waited 60000 seconds
Waited 60000 seconds
Inside reduce function after runTask! result:  Waited 60000 seconds
Waited 60000 seconds
Inside reduce function after runTask! result:  Waited 60000 seconds
Waited 60000 seconds
Inside reduce function after runTask! result:  Waited 60000 seconds
Waited 60000 seconds
Inside reduce function after runTask! result:  Waited 60000 seconds
...

我想要的结果是它等待一个项目的获取完成，然后再移动以获取下一个项目。

标签： javascriptnode.jsexpressasynchronousasync-await

Answer 1

似乎过于复杂。

Id 重写它以循环遍历数组并基于task，在 async/await 中进行获取或等待

//
const sleep = ms => new Promise(r => setTimeout(r, ms));

let arrayOfAsyncTasks = [{
    task: 'fetch',
    url: 'http://thatapi.com/api1/thing'
  },
  {
    task: 'wait',
    duration: 60000
  },
  {
    task: 'fetch',
    url: 'http://thatapi.com/api2/thing'
  },
  {
    task: 'wait',
    duration: 60000
  }
]

;(async() => {

  for (let task of arrayOfAsyncTasks) {
    console.log('running', task)
    if (task.task === 'wait') {
      await sleep(task.duration)
    }

    if (task.task === 'fetch') {
      try {
        // await fetch(task.url)....
      } catch (e) {}
    }
  }
})()