时间戳

首页 > 解决方案 > ScrapydWeb：docker-compose 中的连接被拒绝

python - ScrapydWeb：docker-compose 中的连接被拒绝

问题描述

我尝试运行几个scrapyd 服务以在我的本地主机上建立一个简单的集群，但只有第一个节点有效。对于另外 2 个人，我收到以下错误

scrapydweb_1      | [2020-11-17 07:17:32,738] ERROR    in scrapydweb.utils.check_app_config: HTTPConnectionPool(host='scrapyd_node_3', port=6802): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb060b8ef50>: Failed to establish a new connection: [Errno 111] Connection refused'))
scrapydweb_1      | [2020-11-17 07:17:32,738] ERROR    in scrapydweb.utils.check_app_config: HTTPConnectionPool(host='scrapyd_node_2', port=6801): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb060a1e650>: Failed to establish a new connection: [Errno 111] Connection refused'))

我有以下docker-compose.yml文件：

version: '3'
services:
  scrapyd_node_1:
    build:
      context: .
      dockerfile: ./crawlers/scrapyd/Dockerfile
    ports:
      - "6800:6800"
    volumes:
      - ./data:/var/lib/scrapyd
      - ./data/results:/app/results
    restart: unless-stopped

  scrapyd_node_2:
    build:
      context: .
      dockerfile: ./crawlers/scrapyd/Dockerfile
    ports:
      - "6801:6800"
    volumes:
      - ./data:/var/lib/scrapyd
      - ./data/results:/app/results
    restart: unless-stopped

  scrapyd_node_3:
    build:
      context: .
      dockerfile: ./crawlers/scrapyd/Dockerfile
    ports:
      - "6802:6800"
    volumes:
      - ./data:/var/lib/scrapyd
      - ./data/results:/app/results
    restart: unless-stopped


  scrapydweb:
    build:
      context: .
      dockerfile: ./crawlers/scrapydweb/Dockerfile
    environment:
      USERNAME: "test"
      PASSWORD: "test"
      SCRAPYD_SERVERS: "scrapyd_node_1:6800,scrapyd_node_2:6801,scrapyd_node_3:6802"
    links:
      - scrapyd_node_1
      - scrapyd_node_2
      - scrapyd_node_3
    ports:
      - "5000:5000"
    depends_on:
      - scrapyd_node_1
      - scrapyd_node_2
      - scrapyd_node_3

    restart: unless-stopped

我的 docker-compose 文件有什么问题？

标签： pythondockerdocker-composescrapyscrapyd

解决方案

问题在于：

SCRAPYD_SERVERS: "scrapyd_node_1:6800,scrapyd_node_2:6801,scrapyd_node_3:6802"

尝试将其更改为：

SCRAPYD_SERVERS: "scrapyd_node_1:6800,scrapyd_node_2:6800,scrapyd_node_3:6800"

解释：

例如，当您定义 docker 服务scrapyd_node_2时，您将端口定义为：

    ports:
      - "6801:6800"

这意味着，6800来自容器的端口被映射到6801主机上的端口。因此，当你想用 hostname 声明节点时scrapyd_node_2，你应该使用它的 port = scrapyd_node_2:6800。

推荐阅读