docker - 气流泊坞窗操作员内部服务器错误（“b'Mounts denied：EOF'”）MacOS

Question

我正在尝试在气流管道上使用 docker 运算符。这是我正在使用的代码：

from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import datetime, timedelta
from airflow.operators.docker_operator import DockerOperator

default_args = {
        'owner'                 : 'airflow',
        'description'           : 'Use of the DockerOperator',
        'depend_on_past'        : False,
        'start_date'            : datetime(2018, 1, 3),
        'email_on_failure'      : False,
        'email_on_retry'        : False,
        'retries'               : 1,
        'retry_delay'           : timedelta(minutes=5)
}

with DAG('docker_dag', default_args=default_args, schedule_interval="5 * * * *", catchup=False) as dag:
        t1 = BashOperator(
                task_id='print_current_date',
                bash_command='date'
        )

        t2 = DockerOperator(
                task_id='docker_command',
                image='alpine:3.7',
                api_version='auto',
                auto_remove=True,
                command="/bin/sleep 30",
                docker_url="unix://var/run/docker.sock",
                network_mode="bridge"
        )

        t3 = BashOperator(
                task_id='print_hello',
                bash_command='echo "hello world"'
        )

        t1 >> t2 >> t3

原文出处是：How to use the DockerOperator in Apache Airflow

我使用气流 UI 运行我的 DAG，但在docker_command任务中我收到此错误：

*** Reading local file: /Users/SoftwareDeveloper/airflow/logs/docker_dag/docker_command/2020-10-04T01:47:06.692609+00:00/2.log
[2020-10-03 20:52:21,857] {taskinstance.py:670} INFO - Dependencies all met for <TaskInstance: docker_dag.docker_command 2020-10-04T01:47:06.692609+00:00 [queued]>
[2020-10-03 20:52:21,876] {taskinstance.py:670} INFO - Dependencies all met for <TaskInstance: docker_dag.docker_command 2020-10-04T01:47:06.692609+00:00 [queued]>
[2020-10-03 20:52:21,877] {taskinstance.py:880} INFO - 
--------------------------------------------------------------------------------
[2020-10-03 20:52:21,877] {taskinstance.py:881} INFO - Starting attempt 2 of 2
[2020-10-03 20:52:21,877] {taskinstance.py:882} INFO - 
--------------------------------------------------------------------------------
[2020-10-03 20:52:21,886] {taskinstance.py:901} INFO - Executing <Task(DockerOperator): docker_command> on 2020-10-04T01:47:06.692609+00:00
[2020-10-03 20:52:21,889] {standard_task_runner.py:54} INFO - Started process 33165 to run task
[2020-10-03 20:52:21,929] {standard_task_runner.py:77} INFO - Running: ['airflow', 'run', 'docker_dag', 'docker_command', '2020-10-04T01:47:06.692609+00:00', '--job_id', '72', '--pool', 'default_pool', '--raw', '-sd', 'DAGS_FOLDER/docker_operator.py', '--cfg_path', '/var/folders/9m/6w_b7jmn11s9yn6f3jyfd7rr0000gs/T/tmp98jxpu6w']
[2020-10-03 20:52:21,932] {standard_task_runner.py:78} INFO - Job 72: Subtask docker_command
[2020-10-03 20:52:21,977] {logging_mixin.py:112} INFO - Running %s on host %s <TaskInstance: docker_dag.docker_command 2020-10-04T01:47:06.692609+00:00 [running]> 64.1.168.192.in-addr.arpa
[2020-10-03 20:52:22,126] {docker_operator.py:210} INFO - Starting docker container from image alpine:3.7
[2020-10-03 20:52:22,192] {taskinstance.py:1150} ERROR - 500 Server Error: Internal Server Error ("b'Mounts denied: EOF'")
Traceback (most recent call last):
  File "/Users/SoftwareDeveloper/opt/anaconda3/envs/airflow/lib/python3.8/site-packages/docker/api/client.py", line 259, in _raise_for_status
    response.raise_for_status()
  File "/Users/SoftwareDeveloper/opt/anaconda3/envs/airflow/lib/python3.8/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.40/containers/49bf0d929c8d2524a778b5bdc255544c5a3e4915530bbea379b8e147d765a5c6/start

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/SoftwareDeveloper/opt/anaconda3/envs/airflow/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 984, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/Users/SoftwareDeveloper/opt/anaconda3/envs/airflow/lib/python3.8/site-packages/airflow/operators/docker_operator.py", line 277, in execute
    return self._run_image()
  File "/Users/SoftwareDeveloper/opt/anaconda3/envs/airflow/lib/python3.8/site-packages/airflow/operators/docker_operator.py", line 233, in _run_image
    self.cli.start(self.container['Id'])
  File "/Users/SoftwareDeveloper/opt/anaconda3/envs/airflow/lib/python3.8/site-packages/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/Users/SoftwareDeveloper/opt/anaconda3/envs/airflow/lib/python3.8/site-packages/docker/api/container.py", line 1108, in start
    self._raise_for_status(res)
  File "/Users/SoftwareDeveloper/opt/anaconda3/envs/airflow/lib/python3.8/site-packages/docker/api/client.py", line 261, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/Users/SoftwareDeveloper/opt/anaconda3/envs/airflow/lib/python3.8/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("b'Mounts denied: EOF'")
[2020-10-03 20:52:22,202] {taskinstance.py:1187} INFO - Marking task as FAILED. dag_id=docker_dag, task_id=docker_command, execution_date=20201004T014706, start_date=20201004T015221, end_date=20201004T015222
[2020-10-03 20:52:26,855] {local_task_job.py:102} INFO - Task exited with return code 1

更新：这个练习我是在 Mac 上做的，但我尝试在 Linux 操作系统（Ubuntu 18.04 发行版）上做同样的事情，一切正常。所以，恐怕这与我在 macOS 上的配置或我在计算机上拥有的权限有关。

在 Mac 上，我有 Catalina 版本 10.15.5。

你知道我为什么会收到这个错误吗？

标签： dockerairflow