时间戳

首页 > 解决方案 > ScrapingHub 部署失败

scrapy - ScrapingHub 部署失败

问题描述

我正在尝试部署到 ScrapingHub，这是我得到的错误...

Deploy log last 30 lines:
  File "/app/python/lib/python3.8/site-packages/scrapy/cmdline.py", line 142, in execute
    cmd.crawler_process = CrawlerProcess(settings)
  File "/app/python/lib/python3.8/site-packages/scrapy/crawler.py", line 280, in __init__
    super(CrawlerProcess, self).__init__(settings)
  File "/app/python/lib/python3.8/site-packages/scrapy/crawler.py", line 152, in __init__
    self.spider_loader = self._get_spider_loader(settings)
  File "/app/python/lib/python3.8/site-packages/scrapy/crawler.py", line 146, in _get_spider_loader
    return loader_cls.from_settings(settings.frozencopy())
  File "/app/python/lib/python3.8/site-packages/scrapy/spiderloader.py", line 60, in from_settings
    return cls(settings)
  File "/app/python/lib/python3.8/site-packages/scrapy/spiderloader.py", line 24, in __init__
    self._load_all_spiders()
  File "/app/python/lib/python3.8/site-packages/scrapy/spiderloader.py", line 46, in _load_all_spiders
    for module in walk_modules(name):
  File "/app/python/lib/python3.8/site-packages/scrapy/utils/misc.py", line 77, in walk_modules
    submod = import_module(fullpath)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 618, in _load_backward_compatible
  File "<frozen zipimport>", line 259, in load_module
  File "/app/__main__.egg/sstkscrape/spiders/sstkscrape_spider.py", line 4, in <module>
  File "/app/__main__.egg/sstkscrape/spiders/sstkscrape_spider.py", line 7, in QuoteSpider
FileNotFoundError: [Errno 2] No such file or directory: 'urls-test.csv'
{"message": "shub-image-info exit code: 1", "details": null, "error": "image_info_error"}

{"status": "error", "message": "Internal error"}
Deploy log location: /var/folders/p7/nwmq6_4138n6t3w2spdnpzfm0000gn/T/shub_deploy_n9zmxfto.log
Error: Deploy failed: b'{"status": "error", "message": "Internal error"}'

我有一个本地 csv 文件，我试图将其包含在部署中并打开该 csv 文件，并想知道这是否会导致问题？

标签： scrapyscrapinghub

解决方案

您需要使用package_datasetup.py 文件的部分：部署非代码文件

推荐阅读