python-3.x - Luigi Pipeline in Dockerfile; unknown instruction PYTHONPATH

Question

I have this Dockerfile:

FROM python:3.7
COPY . /app
WORKDIR /app
RUN pip install -r requirements.txt
CMD [
'PYTHONPATH="."', 'luigi', '--module', 'pipe',
'CreateAndFillIndex',
'--index', 'test',
'--http-auth', 'username:password',
'--host', 'localhost',
'--port', '9200'
 ]

Executing docker build -t suggestpipe . && docker run -it suggestpipe with this Dockerfile I get the error:

unknown instruction: "PYTHONPATH='.'",

and if I remove that instruction, I get unknown instruction: "LUIGI", so what am I doing wrong?

In requirements.txt there is a luigi dependency. And 'PYTHONPATH='.' was needed for running locally outside of Docker on my Mac, so I moved that CMD instruction into the Dockerfile too, but I am not sure if I need it. How do I run my pipeline?

标签： python-3.xdockerdockerfileluigi

Answer 1

The Dockerfile syntax isn't actually JSON, and you can't have a line break after the [. Docker in effect rewrites this to

CMD ["/bin/sh", "-c", "["]

(which would actually be valid! You probably have a /bin/[ binary! But the container would exit immediately with status code 0.)

and then moves on to the next line

"PYTHONPATH='.'", "luigi", "--module", "pipe",

where it gets confused because this doesn't actually look like a Dockerfile directive.

Just removing that newline on its own only gets you partway there. If you run

CMD ["PYTHONPATH='.'", "luigi", ...]

Docker won't launch a shell to try to run this; instead, it will look for a binary named exactly PYTHONPATH='.' in the usual directories, and when it doesn't find e.g. /usr/bin/PYTHONPATH='.' it will complain.

You shouldn't need to set this environment variable at all (especially since the pip install step will install packages into the image's isolated global Python installation), and I'd just delete it:

CMD ["luigi", ...]

If you do need to set it, you need to use an explicit ENV directive

ENV PYTHONPATH .