docker - Fail to run DAG on Airflow 1.10.14 running with docker-compose on official Apache\Airflow image
问题描述
I've been trying to set Airflow 1.10.14 to execute Python based process, on docker container using docker-compose. The host is Ubuntu 18 VM.
My Dockerfile:
FROM apache/airflow:1.10.14
USER root
RUN pip install --upgrade pip
RUN pip install --user psycopg2-binary
COPY airflow.cfg /opt/airflow/
RUN apt-get update && apt-get install -y \
libodbc1 \
python3-dev\
libevent-dev\
unixodbc-dev \
freetds-dev \
freetds-bin -y \
tdsodbc -y \
build-essential
# install dependencies
ADD requirements.txt .
RUN pip install -r requirements.txt
USER airflow
Then I execute:
docker build -t learning/airflow .
And my docker-compose.yml is:
version: "3"
networks:
airflow:
services:
postgres:
image: "postgres:9.6"
container_name: "postgres"
environment:
- POSTGRES_USER=airflow
- POSTGRES_PASSWORD=airflow
- POSTGRES_DB=airflow
ports:
- "5432:5432"
networks:
- airflow
# uncomment initdb if you need initdb at first run
initdb:
image: learning/airflow
entrypoint: airflow db init
depends_on:
- postgres
networks:
- airflow
webserver:
image: learning/airflow
restart: always
depends_on:
- postgres
volumes:
- ./dags:/opt/airflow/dags
ports:
- "8080:8080"
entrypoint: airflow webserver
healthcheck:
test: ["CMD-SHELL", "[ -f /opt/airflow/airflow-webserver.pid ]"]
interval: 30s
timeout: 30s
retries: 3
networks:
- airflow
scheduler:
image: learning/airflow
restart: always
depends_on:
- postgres
- webserver
volumes:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
entrypoint: airflow scheduler
healthcheck:
test: ["CMD-SHELL", "[ -f /opt/airflow/airflow-scheduler.pid ]"]
interval: 30s
timeout: 30s
retries: 3
networks:
- airflow
I also use the airflow.cfg as appear in here (with minor changes) In the first run I execute in 3 steps in separated terminals:
docker-compose up postgres
docker-compose up initdb
docker-compose up webserver scheduler
I'm able to access the Airflow UI and turn on the DAG but first step fails immediately with the following error:
*** Log file does not exist: /opt/airflow/logs/stg_process/Process_g/2020-12-23T00:00:00+00:00/2.log
*** Fetching from: http://bf23abdeb4b0:8793/log/stg_process/Process_g/2020-12-23T00:00:00+00:00/2.log
*** Failed to fetch log file from worker. HTTPConnectionPool(host='bf23abdeb4b0', port=8793): Max retries
exceeded with url:
/log/stg_process/Process_g/2020-12-23T00:00:00+00:00/2.log
(Caused by NewConnectionError('<urllib3.connection.HTTPConnection
object at 0x7ff6f20ae898>: Failed to establish a new connection:
[Errno 111] Connection refused',))
What am I missing here? any help will be appreciated...
解决方案
尝试重置元数据数据库然后重新构建?
airflow resetdb
推荐阅读
- javascript - 对于列表中的每个元素,从相关元素中的值添加类并将其传递
- kotlin - Kotlin 中的 Base64 编码器
- reactjs - 渲染函数下的map函数下react类组件状态数据不可用
- docker - 每个命令都会导致致命错误:运行时:内存不足
- python - 将字符串传递给python中的函数
- javascript - 全屏使用 HTML5 和 JavaScript
- java - Java 扫描器 useDelimiter() 方法
- javascript - 如有必要,将 = 或 == 添加到 base64
- python - 用于聊天机器人对话的 CLI 在作为 docker 映像运行后没有出现
- unreal-engine4 - 问题存在于游戏的打包版本中,但不存在于编辑器中