python - Dataflow fails due to ERROR: Could not find a version that satisfies the requirement wheel (from versions: none)
问题描述
Since the beginning of this year our python dataflow jobs result in an error on worker startup:
Processing /var/opt/google/dataflow/setuptools-51.1.1.tar.gz
ERROR: Could not find a version that satisfies the requirement wheel (from versions: none)
ERROR: No matching distribution found for wheel
ERROR: Command errored out with exit status 1: /usr/local/bin/python3 /usr/local/lib/python3.7/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-rvfx0hn9/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links /var/opt/google/dataflow -- 'setuptools>=40.8.0' wheel Check the logs for full command output."
My requirements.txt is
ndjson==0.3.1
apache-beam[gcp]==2.26.0
dataclasses-json==0.5.2
google-cloud-storage==1.35.0
google-cloud-secret-manager==2.0.0
I am using the following Dockerfile to create a Flex-Template:
FROM gcr.io/dataflow-templates-base/python3-template-launcher-base
ARG WORKDIR=/dataflow/template
RUN mkdir -p ${WORKDIR}
WORKDIR ${WORKDIR}
# Due to a change in the Beam base image in version 2.24, we need to install
# libffi-dev manually as a dependency. For more information:
# https://github.com/GoogleCloudPlatform/python-docs-samples/issues/4891
RUN apt-get update && apt-get install -y libffi-dev git && rm -rf /var/lib/apt/lists/*
ADD utils utils
COPY requirements.txt .
COPY main.py .
ENV FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILE="${WORKDIR}/requirements.txt"
ENV FLEX_TEMPLATE_PYTHON_PY_FILE="${WORKDIR}/main.py"
RUN pip install -U apache-beam[gcp]==2.24.0
RUN pip install -U -r ./requirements.txt
Has anyone a solution for the issue? I did try to set a specific setuptools version (51.1.1) as suggested in the following post leading to the same error message. Has anyone faced the same issue before
解决方案
The issue was due a conflict in the dataclasses-json (The exact reason I couldn't find out). After removing it from the requirements.txt
the image can successfully be buildt:
ndjson==0.3.1
google-cloud-storage==1.31.2
google-cloud-secret-manager==2.0.0
pycryptodome==3.9.8
推荐阅读
- javascript - 在反应钩子中,我如何将新创建的对象传递给状态对象?
- php - 将整个 Excel 行复制到表单域中
- html - 文本变换和翻译的翻转坐标系的行为
- visual-studio-code - VSCode 上的 PHP 语言服务器错误:与服务器的连接已关闭。服务器将重新启动。语言服务器以退出代码 255 退出
- apache-kafka - Kafka偏移重置而不重新启动消费者
- typescript - TypeScript:使用属性装饰器时得到错误的对象
- android - 我的 webview 在默认浏览器中打开网页
- python - torch.nn.conv* 中的 groups 参数如何影响卷积过程?
- javascript - 为什么我不能分配给动态创建的 html 文档的正文?
- javascript - 如何正确地将 Promises 串在一起以进行同步代码执行