python - 如何使用 tensorflow-nightly 和 tensorflow 对象检测研究模型构建 docker 镜像

Question

由于目前在 Google Colab 上对tensorflow-nightly 的 GPU 支持已被破坏，因此我正在尝试构建自己的 docker 映像以进行开发。但是，当我object_detection从tensorflow/models夜间 tensorflow 包安装包时，它会被作为依赖项从object_detection setup.py.

我在 Google Colab 中遵循基本相同的步骤，但我的 tensorflow nightly 并没有在那里被覆盖，所以我不确定我错过了什么......

这是我的Dockerfile：

FROM tensorflow/tensorflow:nightly-gpu-jupyter

RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"

RUN apt-get install -y \
        curl \
        git \
        less \
        zip

RUN curl -L -O https://github.com/protocolbuffers/protobuf/releases/download/v3.11.4/protoc-3.11.4-linux-x86_64.zip && unzip protoc-3.11.4-linux-x86_64.zip

RUN cp bin/protoc /usr/local/bin

RUN git clone --depth 1 https://github.com/tensorflow/models
RUN cd models/research && \
        protoc object_detection/protos/*.proto --python_out=. && \
        cp object_detection/packages/tf2/setup.py . && \
        python -m pip install .

RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"

我正在建造的：

docker pull tensorflow/tensorflow:nightly-gpu-jupyter
docker build --no-cache . -f models-tf-nightly.Dockerfile -t tf-nightly-models

第一个print()显示：

Tensorflow version: 2.5.0-dev20201129

但第二个显示：

Tensorflow version: 2.3.1

---

在 Google Colab 中，我执行的步骤基本相同：

# Install the Object Detection API
%%bash
pip install tf-nightly-gpu
[[ -d models ]] || git clone --depth 1 https://github.com/tensorflow/models
cd models/research/
protoc object_detection/protos/*.proto --python_out=.
cp object_detection/packages/tf2/setup.py .
python -m pip install .

之后

import tensorflow as tf
print(tf.__version__)

印刷2.5.0-dev20201201

所以不知何故，我的 Google Colab 步骤保留了我的夜间 Tensorflow 安装，而在 Docker 上它被 2.3.0 覆盖。

标签： pythondockertensorflowpiptensorflow-model-garden

Answer 1

如果您在pip list安装对象检测包之前查看，您会看到tf-nightly-gpu已安装但未安装tensorflow。当您安装对象检测包时，该tensorflow包作为依赖项被拉入。pip认为它没有安装，所以它安装它。

解决此问题的一种方法是欺骗 pip install 认为该tensorflow软件包已安装。可以通过将tf_nightly_gpu-VERSION.dist-info目录符号链接到dist-packages. 我在下面的 Dockerfile 中添加了执行此操作的行。在这篇文章的底部，我还包含了一个 Dockerfile，它实现了一些最佳实践来最小化图像大小。

FROM tensorflow/tensorflow:nightly-gpu-jupyter

RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"

RUN apt-get install -y \
        curl \
        git \
        less \
        zip

# Trick pip into thinking that the 'tensorflow' package is installed.
# Installing `object_detection` attempts to install the 'tensorflow' package.
# Name the symlink with the suffix from tf_nightly_gpu.
WORKDIR /usr/local/lib/python3.6/dist-packages
RUN ln -s tf_nightly_gpu-* tensorflow-$(ls -d1 tf_nightly_gpu* | sed 's/tf_nightly_gpu-\(.*\)/\1/')

WORKDIR /tf
RUN curl -L -O https://github.com/protocolbuffers/protobuf/releases/download/v3.11.4/protoc-3.11.4-linux-x86_64.zip && unzip protoc-3.11.4-linux-x86_64.zip

RUN cp bin/protoc /usr/local/bin

RUN git clone --depth 1 https://github.com/tensorflow/models
RUN cd models/research && \
        protoc object_detection/protos/*.proto --python_out=. && \
        cp object_detection/packages/tf2/setup.py . && \
        python -m pip install .

RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"

---

这是一个 Dockerfile，它导致图像稍小（未压缩 0.22 GB）。显着的变化是清除apt列表并使用--no-cache-dirin pip install。

FROM tensorflow/tensorflow:nightly-gpu-jupyter

RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"

RUN apt-get install -y --no-install-recommends \
        ca-certificates \
        curl \
        git \
        less \
        zip && \
    rm -rf /var/lib/apt/lists/*

# Trick pip into thinking that the 'tensorflow' package is installed.
# Installing `object_detection` attempts to install the 'tensorflow' package.
# Name the symlink with the suffix from tf_nightly_gpu.
WORKDIR /usr/local/lib/python3.6/dist-packages
RUN ln -s tf_nightly_gpu-* tensorflow-$(ls -d1 tf_nightly_gpu* | sed 's/tf_nightly_gpu-\(.*\)/\1/')

WORKDIR /tf
RUN curl -L -O https://github.com/protocolbuffers/protobuf/releases/download/v3.11.4/protoc-3.11.4-linux-x86_64.zip && \
    unzip protoc-3.11.4-linux-x86_64.zip && \
    cp bin/protoc /usr/local/bin && \
    rm -r protoc-3.11.4-linux-x86_64.zip bin/

# Upgrade pip.
RUN python -m pip install --no-cache-dir --upgrade pip

RUN git clone --depth 1 https://github.com/tensorflow/models
WORKDIR models/research
RUN protoc object_detection/protos/*.proto --python_out=. && \
    cp object_detection/packages/tf2/setup.py . && \
    python -m pip install  --no-cache-dir .

RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"