时间戳

首页 > 解决方案 > Scrapy: Can't define bot name anymore in scrapy 2.1.0

scrapy - Scrapy: Can't define bot name anymore in scrapy 2.1.0

问题描述

I have a scrapy 2.1 project running where I have under the root directory two folders. My intention is to have 2 projects with seperate bots and settings in addition to a global setting.

Now I tried to define project names, so I used the export command. Unfortunatelly this somehow changed the bot name to "scrapybot" and it is not possible to rename it again:

merlin@192-143-0-9 spider1 % export SCRAPY_PROJECT=bid      
merlin@192-143-0-9 spider1 % scrapy settings --get BOT_NAME 
scrapybot

The project is within a git repo and doing a git status, it does show no changes at all. The spider list is empty now due to the renamed project.

This looks like a bug to me, so my main question is:

  1. How can I rename my bot back to it`s original name "bid" where all spiders are located?

Once I have it back running, the second question which led me to the current problem is:

  1. How can I specify two different projects under one root directory with their own spiders and settings?

标签： scrapy

解决方案

  • 很正常git status，没啥区别。你只是设置一个环境变量。scrapy settings将从 settings.py 文件中读取值，因此根据您所在的项目文件夹，它会给您“bid”或“scrapybot”
  • 以下结构允许您在 1 个根目录下拥有多个 scrapy 项目，每个项目都有自己的设置和蜘蛛：
root/
    scrapy.cfg
    bid/
        __init__.py
        items.py
        middlewares.py
        pipelines.py
        settings.py
        spiders/
            bidspider.py
    scrapybot/
        __init__.py
        items.py
        middlewares.py
        pipelines.py
        settings.py
        spiders/
            scrapybotspider.py

推荐阅读