scrapy - 如何修复 Docker 上 Scrapy Splash 中的“403”/“503”错误

Question

当我在 Docker 上运行我的 scrapy/splash 配置时，由于出现 403/503 错误，scrapy 无法连接到 Splash。

这是我的 docker-compose 文件：

  scraper:
    build:
      context: ./my_scraper
    image: "my_scraper"
    ports:
      - 6800:6800
    depends_on:
      - splash
  splash:
    image: "scrapinghub/splash"
    ports:
      - 8050:8050
      - 5023:5023
    expose:
      - 8050
      - 5023

刮板的 dockerfile 如下所示：

FROM python:3.7

WORKDIR /my_scraper

COPY requirements.txt /tmp/
RUN pip install -r /tmp/requirements.txt  

COPY . .

EXPOSE 6800

CMD ["scrapyd", "--pidfile="]

我的初始设置是：

SPLASH_URL = 'http://splash:8050'

我的刮板中的其余设置是由 scrapy-splash 文档推荐的。

但是，当我运行我的爬虫时，它无法连接到 Splash，并抛出此错误：

2018-12-29 13:24:14 [scrapy.core.engine] INFO: Spider opened
2018-12-29 13:24:14 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-12-29 13:24:14 [sorg] INFO: Spider opened: wiki_scraper
2018-12-29 13:24:14 [sorg] INFO: Spider opened: wiki_scraper
2018-12-29 13:24:14 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2018-12-29 13:24:14 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.wikipedia.org/ via http://splash:8050/render.html> (failed 1 times): 503 Service Unavailable
2018-12-29 13:24:14 [scrapy.core.engine] DEBUG: Crawled (403) <GET https://www.wikipedia.org/ via http://splash:8050/render.html> (referer: None)
2018-12-29 13:24:14 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <403 https://www.wikipedia.org/>: HTTP status code is not handled or not allowed
2018-12-29 13:24:14 [scrapy.core.engine] INFO: Closing spider (finished)
2018-12-29 13:24:14 [scrapy.core.engine] ERROR: Scraper close failure

在构建刮刀图像时我遇到了类似的问题，但是通过 dockerfile 中的暴露命令解决了这个问题。但是，我为 Splash 使用了现成的图像，所以我有点困惑为什么我无法连接到它。任何帮助将非常感激。

编辑：当我在 SPLASH_URL 设置中使用 docker ip '192.168.0.14' 时，我得到一个不同的错误，表明（至少我认为......）由于超时而不是没有服务而失败：

2018-12-29 14:36:28 [scrapy.downloadermiddlewares.retry] DEBUG: Retrying <GET https://www.wikipedia.org/ via http://192.168.0.14:8050/render.html> (failed 2 times): 504 Gateway Time-out
2018-12-29 14:36:30 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2018-12-29 14:37:23 [scrapy.downloadermiddlewares.retry] DEBUG: Gave up retrying <GET https://www.wikipedia.org/ via http://192.168.0.14:8050/render.html> (failed 3 times): User timeout caused connection failure: Getting http://192.168.0.14:8050/render.html took longer than 55.0 seconds..

标签： scrapydocker-composesplash-screenscrapyd