web-scraping - 如何将数据从多页存储到json？

Question

谢谢你的关注，所以我写了一个小项目来抓取新闻网站并存储它们的主要文本。我尝试了许多解决方案来在我的项目中添加 json 而没有 consol.log 但总是在刮掉它之后只显示一个正文。所以我向你展示我的代码，这样你就可以帮助我如何让 json 包含所有三个新闻。

const { Cluster } = require('puppeteer-cluster');
const fs = require('fs');


const launchOptions = {
    headless: false,
    args: [
        '--disable-gpu',
        '--disable-dev-shm-usage',
        '--disable-web-security',
        '--disable-xss-auditor',
        '--disable-accelerated-2d-canvas',
        '--ignore-certifcate-errors',
        '--ignore-certifcate-errors-spki-list',
        '--no-zygote',
        '--no-sandbox',
        '--disable-setuid-sandbox',
        '--disable-webgl',
    ],
    ignoreHTTPSErrors: true,
    waitUntil: 'networkidle2',
};
(async() => {
    // Create a cluster with 2 workers
    const cluster = await Cluster.launch({
        monitor: true,
        concurrency: Cluster.CONCURRENCY_PAGE,
        maxConcurrency: 2,
        puppeteerOptions: launchOptions,
    });

    // Define a task (in this case: screenshot of page)
    await cluster.task(async({ page, data: url }) => {
        await page.setRequestInterception(true);
        page.on('request', (request) => {
            if (['stylesheet', 'font', 'image', 'styles','other', 'media'].indexOf(request.resourceType()) !== -1) {
                request.abort();
            } else {
                request.continue();
            }
        });
        await page.goto(url);
        const scrapedData = await page.$eval('div[class="entry-content clearfix"]', el => el.innerText)
        fs.writeFileSync('test.json', JSON.stringify(scrapedData, null, 2))
    });
    // Add some pages to queue
    cluster.queue('https://www.ettelaat.com/?p=526642');
    cluster.queue('https://www.ettelaat.com/?p=526640');
    cluster.queue('https://www.ettelaat.com/?p=526641');

    // Shutdown after everything is done
    await cluster.idle();
    await cluster.close();
})();

标签： web-scrapingpuppeteerpuppeteer-cluster

Answer 1

为了收集所有输出，我必须将我的 fs 放在 cluster.close 的底部

kanopyDB = []
.
.
.
        kanopyDB = kanopyDB.concat(name);
.
.
.
    await cluster.idle();
    await cluster.close();
    fs.writeFileSync('output.json', kanopyDB, 'utf8');