时间戳

首页 > 解决方案 > 在 conda 环境中运行气流并使用 systemd 无法启动

python - 在 conda 环境中运行气流并使用 systemd 无法启动

问题描述

我想通过 conda 安装气流并使用 systemd 来控制 Ubuntu 上的气流。我已经能够通过以下步骤成功地将气流安装到 conda 环境中，但是我无法正确配置 systemd 以使用气流。

细节：

我见过SO #52310217，但问题仍未解决。

以下是我从命令行安装和运行气流的步骤：

$ conda create --name airflow -c conda-forge airflow
$ conda activate airflow
$ export AIRFLOW_HOME=~/airflow
$ airflow initdb

$ airflow scheduler
$ airflow webserver

从这里，我创建了以下文件/etc/systemd/system/：

# /etc/systemd/system/airflow-scheduler.service
[Unit]
Description=Airflow scheduler daemon
After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
Wants=postgresql.service mysql.service redis.service rabbitmq-server.service

[Service]
Environment="PATH=/home/ubuntu/python/envs/airflow/bin"
User=airflow
Group=airflow
Type=simple
ExecStart=/home/curtis/miniconda3/envs/airflow/bin/airflow scheduler
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/airflow-webserver.service
[Unit]
Description=Airflow webserver daemon
After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
Wants=postgresql.service mysql.service redis.service rabbitmq-server.service

[Service]
Environment="PATH=/home/ubuntu/python/envs/airflow/bin"
User=airflow
Group=airflow
Type=simple
ExecStart=/home/curtis/miniconda3/envs/airflow/bin/airflow webserver -p 8085 --pid /home/curtis/airflow/airflow-webserver.pid
Restart=on-failure
RestartSec=5s
PrivateTmp=true

[Install]
WantedBy=multi-user.target

创建上述文件后，我发出了以下命令，但没有任何反应。

sudo systemctl daemon-reload
sudo systemctl enable airflow-scheduler
sudo systemctl start airflow-scheduler
sudo systemctl enable airflow-webserver
sudo systemctl start airflow-webserver

如果我ExecStart直接在终端窗口中输入命令，气流调度程序和气流网络服务器将按预期启动。出于这个原因，我认为这不是气流问题，而是我在systemd配置中缺少的东西。

我不确定要从中拿走什么，但这里是气流调度程序和气流网络服务器的状态。

[curtis:~/airflow/logs/scheduler] [airflow] 4 $ sudo systemctl status airflow-scheduler
● airflow-scheduler.service - Airflow scheduler daemon
   Loaded: loaded (/etc/systemd/system/airflow-scheduler.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Mon 2019-08-19 22:36:35 PDT; 3s ago
  Process: 5623 ExecStart=/home/curtis/miniconda3/envs/airflow/bin/airflow scheduler (code=exited, status=217/USER)
 Main PID: 5623 (code=exited, status=217/USER)

[curtis:~/airflow/logs/scheduler] [airflow] 3 $ sudo systemctl status airflow-webserver
● airflow-webserver.service - Airflow webserver daemon
   Loaded: loaded (/etc/systemd/system/airflow-webserver.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Mon 2019-08-19 22:36:46 PDT; 895ms ago
  Process: 5805 ExecStart=/home/curtis/miniconda3/envs/airflow/bin/airflow webserver -p 8085 --pid /home/curtis/airflow/airflow-webserver
 Main PID: 5805 (code=exited, status=217/USER)

标签： pythoncondaairflowubuntu-18.04airflow-scheduler

解决方案

我想出了这个问题，并想在这里为遇到同样问题的其他人发布这个。

  • 应该是您当前的User用户名，或安装 Airflow 的用户名。就我而言，那是“柯蒂斯”
  • 应该是您当前的Group组，或安装 Airflow 的用户组。就我而言，那是“柯蒂斯”
  • 这Environment是您的 conda 环境的路径。就我而言，那是PATH=/home/curtis/miniconda3/envs/airflow/bin

我已包含我更正systemd的文件供您参考。

# /etc/systemd/system/airflow-scheduler.service
[Unit]
Description=Airflow scheduler daemon
After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
Wants=postgresql.service mysql.service redis.service rabbitmq-server.service

[Service]
Environment="PATH=/home/curtis/miniconda3/envs/airflow/bin"
User=curtis
Group=curtis
Type=simple
ExecStart=/home/curtis/miniconda3/envs/airflow/bin/airflow scheduler
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/airflow-webserver.service
[Unit]
Description=Airflow webserver daemon
After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
Wants=postgresql.service mysql.service redis.service rabbitmq-server.service

[Service]
Environment="PATH=/home/curtis/miniconda3/envs/airflow/bin"
User=curtis
Group=curtis
Type=simple
ExecStart=/home/curtis/miniconda3/envs/airflow/bin/airflow webserver -p 8085 --pid /home/curtis/airflow/airflow-webserver.pid
Restart=on-failure
RestartSec=5s
PrivateTmp=true

[Install]
WantedBy=multi-user.target

推荐阅读