python - 使用 GCSFilesStore 在 Scrapy Cloud 上获取蜘蛛以将文件存储在 Google Cloud Storage 上并获取 ImportError

问题描述

将蜘蛛部署到 Scraping Cloud。它会收集文件的下载链接，并将这些文件保存在 Google Cloud 存储桶中。它在本地运行时有效。但是在部署到 Scraping Hub 时，它会返回以下错误：

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1299, in _inlineCallbacks
    result = g.send(result)
  File "/usr/local/lib/python2.7/site-packages/scrapy/crawler.py", line 95, in crawl
    six.reraise(*exc_info)
  File "/usr/local/lib/python2.7/site-packages/scrapy/crawler.py", line 77, in crawl
    self.engine = self._create_engine()
  File "/usr/local/lib/python2.7/site-packages/scrapy/crawler.py", line 102, in _create_engine
    return ExecutionEngine(self, lambda _: self.stop())
  File "/usr/local/lib/python2.7/site-packages/scrapy/core/engine.py", line 70, in __init__
    self.scraper = Scraper(crawler)
  File "/usr/local/lib/python2.7/site-packages/scrapy/core/scraper.py", line 71, in __init__
    self.itemproc = itemproc_cls.from_crawler(crawler)
  File "/usr/local/lib/python2.7/site-packages/scrapy/middleware.py", line 58, in from_crawler
    return cls.from_settings(crawler.settings, crawler)
  File "/usr/local/lib/python2.7/site-packages/scrapy/middleware.py", line 34, in from_settings
    mwcls = load_object(clspath)
  File "/usr/local/lib/python2.7/site-packages/scrapy/utils/misc.py", line 44, in load_object
    mod = import_module(module)
  File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/app/__main__.egg/tceq_crawler/pipelines.py", line 4, in <module>
ImportError: cannot import name GCSFilesStore

它在本地运行，并且能够将文件上传到 GCloud 存储桶。有一个专门为蜘蛛创建的服务帐户。我导出了 JSON 凭证文件并将其直接粘贴到凭证对象中。

在 YML 文件中，我指定了需求文件。在 requirements.txt 文件中，我指定了所有需要的包。

在这一点上，我不确定为什么它在导入 FilesPipeline 时无法在 Scrapy Cloud 上导入 GCSFilesStore。

这是我的代码：

pipelines.py

# -*- coding: utf-8 -*-

from scrapy.pipelines.files import FilesPipeline
from scrapy.pipelines.files import GCSFilesStore
from scrapy.http.request import Request

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://doc.scrapy.org/en/latest/topics/item-pipeline.html

class TceqCrawlerPipeline(object):
    def process_item(self, item, spider):
        return item

class CustomFileNamePipeline(FilesPipeline):

    def get_media_requests(self, item, info):
        return [Request(x, meta={'file_name': item["file_name"]})
                for x in item.get('file_urls', [])]

    def file_path(self, request, response=None, info=None):
        return request.meta['file_name']

class GCSFilesStoreJSON(GCSFilesStore):
    CREDENTIALS = {
        "type": "service_account",
        "project_id": "project_id",
        "private_key_id": "FROM JSON FILE",
        "private_key": "-----BEGIN PRIVATE KEY-----FROM JSON FILE-----END PRIVATE KEY-----\n",
        "client_email": "FROM JSON FILE",
        "client_id": "FROM JSON FILE",
        "auth_uri": "https://accounts.google.com/o/oauth2/auth",
        "token_uri": "https://oauth2.googleapis.com/token",
        "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
        "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/FROM JSON FILE"
    }
    def __init__(self, uri):
        from google.cloud import storage
        client = storage.Client.from_service_account_info(self.CREDENTIALS)
        bucket, prefix = uri[5:].split('/', 1)
        self.bucket = client.bucket(bucket)
        self.prefix = prefix

class GCSFilePipeline(FilesPipeline):
    def __init__(self, store_uri, download_func=None, settings=None):
        super(GCSFilePipeline, self).__init__(store_uri,download_func,settings)

settings.py

DOWNLOADER_MIDDLEWARES = {'scrapy_crawlera.CrawleraMiddleware': 300}
CRAWLERA_ENABLED = True
CRAWLERA_APIKEY = 'API_KEY'


ITEM_PIPELINES = {
    'tceq_crawler.pipelines.GCSFilePipeline': 1,
    'tceq_crawler.pipelines.CustomFileNamePipeline': 200
}
##GOOGLE CLOUD STORAGE
FILES_STORE = 'gs://bucket-name/'
GCS_PROJECT_ID = 'project_id'

scrapinghub.yml

project: 12345

requirements:
  file: requirements.txt

requirements.txt

pyasn1==0.4.1
google-api-core==1.8.1                     
google-auth==1.6.3                     
google-cloud-core==0.29.1                    
google-cloud-storage==1.14.0                  
google-resumable-media==0.3.2                    
googleapis-common-protos==1.5.8

标签： pythonscrapygoogle-cloud-storagescrapinghub

python - 使用 GCSFilesStore 在 Scrapy Cloud 上获取蜘蛛以将文件存储在 Google Cloud Storage 上并获取 ImportError

问题描述

解决方案

推荐阅读