时间戳

首页 > 解决方案 > Apache气流运行太多进程

airflow - Apache气流运行太多进程

问题描述

我有以下所有与 Airflow 一起运行的进程。我试图理解为什么有这么多重复的东西，以及它们是否都需要。我正在使用 postgres 运行本地执行程序，它systemd按照以下教程的说明运行。https://towardsdatascience.com/an-apache-airflow-mvp-complete-guide-for-a-basic-production-installation-using-localexecutor-beb10e4886b2

我想更好地了解调试它的过程，因为调度程序目前似乎没有工作。任何帮助将不胜感激。

ubuntu   30932  0.0  0.0  76692   900 ?        Ss   00:53   0:00 /lib/systemd/systemd --user
ubuntu   30933  0.0  0.1 259528  2740 ?        S    00:53   0:00 (sd-pam)
root       525  0.0  0.3 107992  7280 ?        Ss   01:26   0:00 sshd: ubuntu [priv]
ubuntu     619  0.0  0.3 108228  6120 ?        S    01:26   0:00 sshd: ubuntu@pts/0
ubuntu     626  0.0  0.2  23380  5388 pts/0    Ss+  01:26   0:00 -bash
root     19090  0.0  0.0      0     0 ?        I    04:04   0:00 [kworker/u4:1]
root     15692  0.0  0.3 107992  7340 ?        Ss   04:44   0:00 sshd: ubuntu [priv]
ubuntu   15809  0.0  0.1 107992  3684 ?        S    04:44   0:00 sshd: ubuntu@pts/1
ubuntu   15810  0.0  0.2  23380  5356 pts/1    Ss+  04:44   0:00 -bash
root     20272  0.0  0.0      0     0 ?        I    04:50   0:00 [kworker/u4:0]
root     20274  0.0  0.0      0     0 ?        I    04:50   0:00 [kworker/0:0]
root     20676  0.0  0.3 107992  7416 ?        Ss   04:51   0:00 sshd: ubuntu [priv]
ubuntu   20783  0.0  0.3 108184  6200 ?        S    04:51   0:00 sshd: ubuntu@pts/2
ubuntu   20784  0.0  0.2  23380  5376 pts/2    Ss   04:51   0:00 -bash
root     22974  0.0  0.0      0     0 ?        I    04:54   0:00 [kworker/1:0]
ubuntu   23001  1.2  4.7 374404 95696 ?        Ss   04:54   0:04 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23002  2.2  6.2 501048 126200 ?       Ss   04:54   0:07 /usr/bin/python3 /home/ubuntu/.local/bin/airflow webserver
postgres 23030  0.0  0.8 321700 16512 ?        Ss   04:54   0:00 postgres: 10/main: airflow airflow ***.*.*.*(53558) idle
postgres 23031  0.1  0.9 322608 19976 ?        Ss   04:54   0:00 postgres: 10/main: airflow airflow ***.*.*.*(53560) idle
ubuntu   23038  0.6  4.2 1636692 84520 ?       Sl   04:54   0:02 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23047  0.0  0.0      0     0 ?        Z    04:54   0:00 [/usr/bin/python] <defunct>
ubuntu   23048  0.0  0.0      0     0 ?        Z    04:54   0:00 [/usr/bin/python] <defunct>
ubuntu   23049  0.0  0.0      0     0 ?        Z    04:54   0:00 [/usr/bin/python] <defunct>
ubuntu   23052  0.0  3.9 372588 78492 ?        S    04:54   0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23053  0.0  3.9 372588 78492 ?        S    04:54   0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23056  0.0  3.9 372588 78492 ?        S    04:54   0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23058  0.0  3.9 372588 78428 ?        S    04:54   0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23059  0.0  3.9 372588 78428 ?        S    04:54   0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23063  0.0  3.9 372588 78436 ?        S    04:54   0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23064  0.0  3.9 372588 78436 ?        S    04:54   0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23069  0.0  3.9 372588 78440 ?        S    04:54   0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23072  0.0  3.9 372588 78440 ?        S    04:54   0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23074  0.0  3.9 372588 78440 ?        S    04:54   0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23077  0.0  3.9 372588 78440 ?        S    04:54   0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23080  0.0  3.9 372588 78440 ?        S    04:54   0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23082  0.0  3.9 372588 78440 ?        S    04:54   0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23086  0.0  3.9 372588 78440 ?        S    04:54   0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23089  0.0  3.9 372588 78444 ?        S    04:54   0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23091  0.0  3.9 372588 78444 ?        S    04:54   0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23093  0.0  3.9 372588 78448 ?        S    04:54   0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23097  0.0  3.9 372588 78448 ?        S    04:54   0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23099  0.0  3.9 372588 78448 ?        S    04:54   0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23103  0.0  3.9 372588 78448 ?        S    04:54   0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23104  0.0  3.9 372588 78448 ?        S    04:54   0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23108  0.0  3.9 372588 78452 ?        S    04:54   0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23112  0.0  3.9 372588 78452 ?        S    04:54   0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23113  0.0  3.9 372588 78452 ?        S    04:54   0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23117  0.0  3.9 372588 78452 ?        S    04:54   0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23118  0.0  3.9 372588 78452 ?        S    04:54   0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23122  0.0  3.9 372588 78452 ?        S    04:54   0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23125  0.0  3.9 372588 78452 ?        S    04:54   0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23128  0.0  3.9 372588 78452 ?        S    04:54   0:00 /usr/bin/python3 /home/ubuntu/.local/bin/airflow scheduler
ubuntu   23142  1.3  4.1 373364 82908 ?        S    04:54   0:04 airflow scheduler -- DagFileProcessorManager
postgres 23179  0.0  0.9 322380 18120 ?        Ss   04:54   0:00 postgres: 10/main: airflow airflow 127.0.0.1(53564) idle
ubuntu   23193  0.4  3.1 217464 63712 ?        S    04:54   0:01 gunicorn: master [airflow-webserver]
ubuntu   26167  1.3  5.9 494876 119428 ?       Sl   04:58   0:01 [ready] gunicorn: worker [airflow-webserver]
postgres 26181  0.0  0.8 321596 16640 ?        Ss   04:58   0:00 postgres: 10/main: airflow airflow 127.0.0.1(54584) idle
ubuntu   26545  1.8  5.9 494876 119428 ?       Sl   04:58   0:01 [ready] gunicorn: worker [airflow-webserver]
postgres 26559  0.0  0.8 321596 16640 ?        Ss   04:58   0:00 postgres: 10/main: airflow airflow 127.0.0.1(54714) idle
ubuntu   26910  3.6  5.9 494876 119428 ?       Sl   04:59   0:01 [ready] gunicorn: worker [airflow-webserver]
postgres 26924  0.0  0.8 321596 16640 ?        Ss   04:59   0:00 postgres: 10/main: airflow airflow 127.0.0.1(54840) idle
ubuntu   27287 14.1  5.9 494876 119428 ?       Sl   04:59   0:01 [ready] gunicorn: worker [airflow-webserver]
postgres 27301  0.0  0.8 321596 16640 ?        Ss   04:59   0:00 postgres: 10/main: airflow airflow 127.0.0.1(54966) idle
ubuntu   27411  0.0  0.0      0     0 ?        Z    05:00   0:00 [airflow schedul] <defunct>
ubuntu   27414  0.0  0.0      0     0 ?        Z    05:00   0:00 [airflow schedul] <defunct>
ubuntu   27423  0.0  0.1  40268  3884 pts/2    R+   05:00   0:00 ps -aux --sort start_time

标签： airflow

解决方案

以下问题可能重复！

运行 `airflow scheduler` 会启动 33 个调度程序进程

拥有这些许多进程是默认行为，但您可以airflow.cfg通过更改parallelism值来更改它。

推荐阅读