python - 无法部署到 Scrapy 云,导入通用模块也有问题
问题描述
我正在尝试将蜘蛛部署到 Scrapy 云,但我反复遇到需求问题,甚至导入常用模块,如 io、re、shlex 和 PyPDF2。我正在使用 Python 3.5。我的scrapinghub.yml
文件包含以下几行:
projects:
default: 358310
stacks:
default: scrapy:1.3-py3
requirements:
file: requirements.txt
我的requirements.txt
文件包含以下几行:
io
re
shlex
PyPDF2==1.26.0
这是我得到的错误:
Login succeeded
Building an image:
Step 1/12 : FROM scrapinghub/scrapinghub-stack-scrapy:1.3-py3
[91m# Executing 2 build triggers...
[0m
Step 1/1 : ENV PIP_TRUSTED_HOST $PIP_TRUSTED_HOST PIP_INDEX_URL $PIP_INDEX_URL
---> Using cache
Step 1/1 : RUN test -n $APT_PROXY && echo 'Acquire::http::Proxy \"$APT_PROXY\";' >/etc/apt/apt.conf.d/proxy
---> Using cache
---> 8d2af0ecc1ce
Step 2/12 : ENV PYTHONUSERBASE /app/python
---> Using cache
---> c5bc537289c7
Step 3/12 : ADD eggbased-entrypoint /usr/local/sbin/
---> Using cache
---> 210ce92ef42e
Step 4/12 : ADD run-pipcheck /usr/local/bin/
---> Using cache
---> 2d0a46143fa4
Step 5/12 : RUN chmod +x /usr/local/bin/run-pipcheck
---> Using cache
---> a2eefa41c642
Step 6/12 : RUN chmod +x /usr/local/sbin/eggbased-entrypoint && ln -sf /usr/local/sbin/eggbased-entrypoint /usr/local/sbin/start-crawl && ln -sf /usr/local/sbin/eggbased-entrypoint /usr/local/sbin/scrapy-list && ln -sf /usr/local/sbin/eggbased-entrypoint /usr/local/sbin/shub-image-info && ln -sf /usr/local/sbin/eggbased-entrypoint /usr/local/sbin/run-pipcheck
---> Using cache
---> f3f5b5c713e3
Step 7/12 : ADD requirements.txt /app/requirements.txt
---> Using cache
---> e6417a1c3fea
Step 8/12 : RUN mkdir /app/python && chown nobody:nogroup /app/python
---> Using cache
---> 6720be2ef393
Step 9/12 : RUN sudo -u nobody -E PYTHONUSERBASE=$PYTHONUSERBASE pip install --user --no-cache-dir -r /app/requirements.txt
---> Running in 8d8694511a74
Collecting io (from -r /app/requirements.txt (line 1))
[91m Could not find a version that satisfies the requirement io (from -r /app/requirements.txt (line 1)) (from versions: )
[0m
[91mNo matching distribution found for io (from -r /app/requirements.txt (line 1))
[0m
[91mYou are using pip version 9.0.3, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
[0m
{"message": "The command '/bin/sh -c sudo -u nobody -E PYTHONUSERBASE=$PYTHONUSERBASE pip install --user --no-cache-dir -r /app/requirements.txt' returned a non-zero code: 1", "details": {"message": "The command '/bin/sh -c sudo -u nobody -E PYTHONUSERBASE=$PYTHONUSERBASE pip install --user --no-cache-dir -r /app/requirements.txt' returned a non-zero code: 1", "code": 1}, "error": "requirements_error"}
我哪里错了?顺便说一句,我安装了最新版本的 pip(与错误消息所述相反)。
解决方案
据我所知,io、re、shlex 是内部库,而不是包。它们不应该在需求文件中。
推荐阅读
- java - 为经常运行的代码创建私有静态对象与实例化对象是否有价值?
- php - 创建资源时出错:[消息] fopen(https://localhost:80/oauth/token):无法打开流:连接被拒绝
- python-3.x - 如何从命令行使用 spark-submit 运行 pyspark docker 应用程序?
- r - r中有没有像entry.get()这样的内置函数?
- python - 有没有像scrapy中的# spliter这样的东西?
- c# - Xamarin.Forms Firebase 广告在实际设备上崩溃
- javascript - 在 Javascript 中查找和替换,除非已经就位
- java - weblogic:使用连接池的连接已经关闭错误
- sql - SQL 填充 2 周时间表的日历日期
- scala - 如何使用 approxQuantile 方法在 Scala 中查找分位数