python - 使用气流模块的 PythonVirtualenvOperator 无法使用 AttributeError 执行：模块 'airflow' 没有属性 'utils'
问题描述
我在虚拟环境中部署了 Airflow，如果我尝试通过导入 Airflow 模块（例如获取变量）来执行 PythonVirtualenvOperator，它会给我 AttributeError。猜猜我不完全理解 Airflow 如何执行 VirtualenvOperator，因此如何克服它，所以任何建议和见解都将受到高度赞赏
我的测试 DAG 代码
from airflow import DAG
from airflow.operators.python import PythonVirtualenvOperator
from airflow.utils.dates import days_ago
args = {
'owner': 'airflow',
}
dag = DAG(
dag_id='example_python_operator_shieet',
default_args=args,
schedule_interval=None,
start_date=days_ago(2),
tags=['example'],
)
def callable_virtualenv():
from time import sleep
from colorama import Back, Fore, Style
from airflow.models import Variable
print(Fore.RED + 'some red text')
print(Back.GREEN + 'and with a green background')
print(Style.DIM + 'and in dim text')
print(Style.RESET_ALL)
for _ in range(10):
print(Style.DIM + 'Please wait...', flush=True)
sleep(10)
print('Finished')
virtualenv_task = PythonVirtualenvOperator(
task_id="virtualenv_python",
python_callable=callable_virtualenv,
requirements=["colorama==0.4.0",'apache-airflow==2.0.1'],
system_site_packages=False,
dag=dag,
)
运行，当 PythonVirtualenvOperator 使用气流模块时，所有情况下的日志都相同
*** Reading local file: /opt1/app_data/airflow/logs/weather_update/updating_forecasts/2021-04-19T12:27:09.086267+00:00/1.log
[2021-04-19 15:27:10,221] {taskinstance.py:851} INFO - Dependencies all met for <TaskInstance: weather_update.updating_forecasts 2021-04-19T12:27:09.086267+00:00 [queued]>
[2021-04-19 15:27:10,234] {taskinstance.py:851} INFO - Dependencies all met for <TaskInstance: weather_update.updating_forecasts 2021-04-19T12:27:09.086267+00:00 [queued]>
[2021-04-19 15:27:10,234] {taskinstance.py:1042} INFO -
--------------------------------------------------------------------------------
[2021-04-19 15:27:10,234] {taskinstance.py:1043} INFO - Starting attempt 1 of 1
[2021-04-19 15:27:10,234] {taskinstance.py:1044} INFO -
--------------------------------------------------------------------------------
[2021-04-19 15:27:10,241] {taskinstance.py:1063} INFO - Executing <Task(PythonVirtualenvOperator): updating_forecasts> on 2021-04-19T12:27:09.086267+00:00
[2021-04-19 15:27:10,244] {standard_task_runner.py:52} INFO - Started process 24052 to run task
[2021-04-19 15:27:10,248] {standard_task_runner.py:76} INFO - Running: ['airflow', 'tasks', 'run', 'weather_update', 'updating_forecasts', '2021-04-19T12:27:09.086267+00:00', '--job-id', '73', '--pool', 'default_pool', '--raw', '--subdir', 'DAGS_FOLDER/weather_dag.py', '--cfg-path', '/tmp/tmp2j8ueojr', '--error-file', '/tmp/tmpvh9s78fv']
[2021-04-19 15:27:10,248] {standard_task_runner.py:77} INFO - Job 73: Subtask updating_forecasts
[2021-04-19 15:27:10,289] {logging_mixin.py:104} INFO - Running <TaskInstance: weather_update.updating_forecasts 2021-04-19T12:27:09.086267+00:00 [running]> on host sk-bi-08.erkapharm.ru
[2021-04-19 15:27:10,330] {taskinstance.py:1257} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_EMAIL=burzilov.ea@erkapharm.com
AIRFLOW_CTX_DAG_OWNER=burzilov.ea
AIRFLOW_CTX_DAG_ID=weather_update
AIRFLOW_CTX_TASK_ID=updating_forecasts
AIRFLOW_CTX_EXECUTION_DATE=2021-04-19T12:27:09.086267+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2021-04-19T12:27:09.086267+00:00
[2021-04-19 15:27:10,331] {process_utils.py:135} INFO - Executing cmd: virtualenv /tmp/venvxuo1gayl
[2021-04-19 15:27:10,336] {process_utils.py:137} INFO - Output:
[2021-04-19 15:27:10,556] {process_utils.py:141} INFO - created virtual environment CPython3.7.3.final.0-64 in 134ms
[2021-04-19 15:27:10,557] {process_utils.py:141} INFO - creator CPython3Posix(dest=/tmp/venvxuo1gayl, clear=False, no_vcs_ignore=False, global=False)
[2021-04-19 15:27:10,557] {process_utils.py:141} INFO - seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/RKF/burzilov.ea/.local/share/virtualenv)
[2021-04-19 15:27:10,557] {process_utils.py:141} INFO - added seed packages: pip==21.0.1, setuptools==54.2.0, wheel==0.36.2
[2021-04-19 15:27:10,557] {process_utils.py:141} INFO - activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
[2021-04-19 15:27:10,574] {process_utils.py:135} INFO - Executing cmd: /tmp/venvxuo1gayl/bin/pip install pandas numpy pyodbc sqlalchemy apache-airflow==2.0.1
[2021-04-19 15:27:10,579] {process_utils.py:137} INFO - Output:
[2021-04-19 15:27:11,603] {process_utils.py:141} INFO - Collecting pandas
[2021-04-19 15:27:11,644] {process_utils.py:141} INFO - Using cached pandas-1.2.4-cp37-cp37m-manylinux1_x86_64.whl (9.9 MB)
[2021-04-19 15:27:12,048] {process_utils.py:141} INFO - Collecting numpy
[2021-04-19 15:27:12,112] {process_utils.py:141} INFO - Using cached numpy-1.20.2-cp37-cp37m-manylinux2010_x86_64.whl (15.3 MB)
[2021-04-19 15:27:12,287] {process_utils.py:141} INFO - Collecting pyodbc
[2021-04-19 15:27:12,287] {process_utils.py:141} INFO - Using cached pyodbc-4.0.30-cp37-cp37m-linux_x86_64.whl
[2021-04-19 15:27:12,519] {process_utils.py:141} INFO - Collecting sqlalchemy
[2021-04-19 15:27:12,530] {process_utils.py:141} INFO - Using cached SQLAlchemy-1.4.9-cp37-cp37m-manylinux2014_x86_64.whl (1.5 MB)
[2021-04-19 15:27:12,634] {process_utils.py:141} INFO - Collecting apache-airflow==2.0.1
[2021-04-19 15:27:12,699] {process_utils.py:141} INFO - Using cached apache_airflow-2.0.1-py3-none-any.whl (4.5 MB)
[2021-04-19 15:27:14,536] {process_utils.py:141} INFO - Collecting sqlalchemy-jsonfield~=1.0
[2021-04-19 15:27:14,538] {process_utils.py:141} INFO - Using cached SQLAlchemy_JSONField-1.0.0-py3-none-any.whl (10 kB)
[2021-04-19 15:27:14,603] {process_utils.py:141} INFO - Collecting attrs<21.0,>=20.0
[2021-04-19 15:27:14,606] {process_utils.py:141} INFO - Using cached attrs-20.3.0-py2.py3-none-any.whl (49 kB)
[2021-04-19 15:27:14,662] {process_utils.py:141} INFO - Collecting python-slugify<5.0,>=3.0.0
[2021-04-19 15:27:14,662] {process_utils.py:141} INFO - Using cached python_slugify-4.0.1-py2.py3-none-any.whl
[2021-04-19 15:27:14,716] {process_utils.py:141} INFO - Collecting itsdangerous>=1.1.0
[2021-04-19 15:27:14,718] {process_utils.py:141} INFO - Using cached itsdangerous-1.1.0-py2.py3-none-any.whl (16 kB)
[2021-04-19 15:27:14,799] {process_utils.py:141} INFO - Collecting graphviz>=0.12
[2021-04-19 15:27:14,801] {process_utils.py:141} INFO - Using cached graphviz-0.16-py2.py3-none-any.whl (19 kB)
[2021-04-19 15:27:14,895] {process_utils.py:141} INFO - Collecting markdown<4.0,>=2.5.2
[2021-04-19 15:27:14,897] {process_utils.py:141} INFO - Using cached Markdown-3.3.4-py3-none-any.whl (97 kB)
[2021-04-19 15:27:14,970] {process_utils.py:141} INFO - Collecting flask<2.0,>=1.1.0
[2021-04-19 15:27:14,973] {process_utils.py:141} INFO - Using cached Flask-1.1.2-py2.py3-none-any.whl (94 kB)
[2021-04-19 15:27:15,104] {process_utils.py:141} INFO - Collecting connexion[flask,swagger-ui]<3,>=2.6.0
[2021-04-19 15:27:15,107] {process_utils.py:141} INFO - Using cached connexion-2.7.0-py2.py3-none-any.whl (77 kB)
[2021-04-19 15:27:15,199] {process_utils.py:141} INFO - Collecting requests>=2.20.0
[2021-04-19 15:27:15,202] {process_utils.py:141} INFO - Using cached requests-2.25.1-py2.py3-none-any.whl (61 kB)
[2021-04-19 15:27:15,286] {process_utils.py:141} INFO - Collecting markupsafe<2.0,>=1.1.1
[2021-04-19 15:27:15,288] {process_utils.py:141} INFO - Using cached MarkupSafe-1.1.1-cp37-cp37m-manylinux2010_x86_64.whl (33 kB)
[2021-04-19 15:27:15,366] {process_utils.py:141} INFO - Collecting python-dateutil<3,>=2.3
[2021-04-19 15:27:15,370] {process_utils.py:141} INFO - Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
[2021-04-19 15:27:15,494] {process_utils.py:141} INFO - Collecting rich==9.2.0
[2021-04-19 15:27:15,498] {process_utils.py:141} INFO - Using cached rich-9.2.0-py3-none-any.whl (164 kB)
[2021-04-19 15:27:15,555] {process_utils.py:141} INFO - Collecting marshmallow-oneofschema>=2.0.1
[2021-04-19 15:27:15,556] {process_utils.py:141} INFO - Using cached marshmallow_oneofschema-2.1.0-py2.py3-none-any.whl (5.7 kB)
[2021-04-19 15:27:15,614] {process_utils.py:141} INFO - Collecting typing-extensions>=3.7.4
[2021-04-19 15:27:15,616] {process_utils.py:141} INFO - Using cached typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
[2021-04-19 15:27:15,682] {process_utils.py:141} INFO - Collecting flask-caching<2.0.0,>=1.5.0
[2021-04-19 15:27:15,684] {process_utils.py:141} INFO - Using cached Flask_Caching-1.10.1-py3-none-any.whl (34 kB)
[2021-04-19 15:27:15,744] {process_utils.py:141} INFO - Collecting tabulate<0.9,>=0.7.5
[2021-04-19 15:27:15,745] {process_utils.py:141} INFO - Using cached tabulate-0.8.9-py3-none-any.whl (25 kB)
[2021-04-19 15:27:15,813] {process_utils.py:141} INFO - Collecting jinja2<2.12.0,>=2.10.1
[2021-04-19 15:27:15,817] {process_utils.py:141} INFO - Using cached Jinja2-2.11.3-py2.py3-none-any.whl (125 kB)
[2021-04-19 15:27:15,872] {process_utils.py:141} INFO - Collecting apache-airflow-providers-http
[2021-04-19 15:27:15,874] {process_utils.py:141} INFO - Using cached apache_airflow_providers_http-1.1.1-py3-none-any.whl (20 kB)
[2021-04-19 15:27:16,201] {process_utils.py:141} INFO - Collecting cryptography>=0.9.3
[2021-04-19 15:27:16,208] {process_utils.py:141} INFO - Using cached cryptography-3.4.7-cp36-abi3-manylinux2014_x86_64.whl (3.2 MB)
[2021-04-19 15:27:16,314] {process_utils.py:141} INFO - Collecting croniter<0.4,>=0.3.17
[2021-04-19 15:27:16,316] {process_utils.py:141} INFO - Using cached croniter-0.3.37-py2.py3-none-any.whl (13 kB)
[2021-04-19 15:27:16,398] {process_utils.py:141} INFO - Collecting alembic<2.0,>=1.2
[2021-04-19 15:27:16,400] {process_utils.py:141} INFO - Using cached alembic-1.5.8-py2.py3-none-any.whl (159 kB)
[2021-04-19 15:27:16,456] {process_utils.py:141} INFO - Collecting python-nvd3~=0.15.0
[2021-04-19 15:27:16,456] {process_utils.py:141} INFO - Using cached python_nvd3-0.15.0-py3-none-any.whl
[2021-04-19 15:27:16,507] {process_utils.py:141} INFO - Collecting unicodecsv>=0.14.1
[2021-04-19 15:27:16,507] {process_utils.py:141} INFO - Using cached unicodecsv-0.14.1-py3-none-any.whl
[2021-04-19 15:27:16,568] {process_utils.py:141} INFO - Collecting setproctitle<2,>=1.1.8
[2021-04-19 15:27:16,570] {process_utils.py:141} INFO - Using cached setproctitle-1.2.2-cp37-cp37m-manylinux1_x86_64.whl (36 kB)
[2021-04-19 15:27:16,628] {process_utils.py:141} INFO - Collecting flask-login<0.5,>=0.3
[2021-04-19 15:27:16,628] {process_utils.py:141} INFO - Using cached Flask_Login-0.4.1-py2.py3-none-any.whl
[2021-04-19 15:27:16,682] {process_utils.py:141} INFO - Collecting apache-airflow-providers-imap
[2021-04-19 15:27:16,684] {process_utils.py:141} INFO - Using cached apache_airflow_providers_imap-1.0.1-py3-none-any.whl (15 kB)
[2021-04-19 15:27:16,768] {process_utils.py:141} INFO - Collecting colorlog>=4.0.2
[2021-04-19 15:27:16,770] {process_utils.py:141} INFO - Using cached colorlog-5.0.1-py2.py3-none-any.whl (10 kB)
[2021-04-19 15:27:16,874] {process_utils.py:141} INFO - Collecting pygments<3.0,>=2.0.1
[2021-04-19 15:27:16,878] {process_utils.py:141} INFO - Using cached Pygments-2.8.1-py3-none-any.whl (983 kB)
[2021-04-19 15:27:16,976] {process_utils.py:141} INFO - Collecting argcomplete~=1.10
[2021-04-19 15:27:16,978] {process_utils.py:141} INFO - Using cached argcomplete-1.12.2-py2.py3-none-any.whl (38 kB)
[2021-04-19 15:27:17,055] {process_utils.py:141} INFO - Collecting importlib-resources~=1.4
[2021-04-19 15:27:17,057] {process_utils.py:141} INFO - Using cached importlib_resources-1.5.0-py2.py3-none-any.whl (21 kB)
[2021-04-19 15:27:17,113] {process_utils.py:141} INFO - Collecting apache-airflow-providers-sqlite
[2021-04-19 15:27:17,114] {process_utils.py:141} INFO - Using cached apache_airflow_providers_sqlite-1.0.2-py3-none-any.whl (14 kB)
[2021-04-19 15:27:17,185] {process_utils.py:141} INFO - Collecting cattrs~=1.1
[2021-04-19 15:27:17,187] {process_utils.py:141} INFO - Using cached cattrs-1.5.0-py3-none-any.whl (19 kB)
[2021-04-19 15:27:17,274] {process_utils.py:141} INFO - Collecting pyjwt<2
[2021-04-19 15:27:17,275] {process_utils.py:141} INFO - Using cached PyJWT-1.7.1-py2.py3-none-any.whl (18 kB)
[2021-04-19 15:27:17,397] {process_utils.py:141} INFO - Collecting lazy-object-proxy
[2021-04-19 15:27:17,398] {process_utils.py:141} INFO - Using cached lazy_object_proxy-1.6.0-cp37-cp37m-manylinux1_x86_64.whl (55 kB)
[2021-04-19 15:27:17,457] {process_utils.py:141} INFO - Collecting flask-wtf<0.15,>=0.14.3
[2021-04-19 15:27:17,459] {process_utils.py:141} INFO - Using cached Flask_WTF-0.14.3-py2.py3-none-any.whl (13 kB)
[2021-04-19 15:27:17,603] {process_utils.py:141} INFO - Collecting flask-appbuilder~=3.1.1
[2021-04-19 15:27:17,632] {process_utils.py:141} INFO - Using cached Flask_AppBuilder-3.1.1-py3-none-any.whl (1.7 MB)
[2021-04-19 15:27:17,696] {process_utils.py:141} INFO - Collecting apache-airflow-providers-ftp
[2021-04-19 15:27:17,698] {process_utils.py:141} INFO - Using cached apache_airflow_providers_ftp-1.0.1-py3-none-any.whl (14 kB)
*some other apache-airflow dependencies installation logs*
[2021-04-19 15:27:23,603] {process_utils.py:141} INFO - Collecting certifi>=2017.4.17
[2021-04-19 15:27:23,607] {process_utils.py:141} INFO - Using cached certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
[2021-04-19 15:27:24,201] {process_utils.py:141} INFO - Collecting isodate
[2021-04-19 15:27:24,203] {process_utils.py:141} INFO - Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
[2021-04-19 15:27:26,537] {process_utils.py:141} INFO - Installing collected packages: zipp, six, pyrsistent, importlib-metadata, attrs, markupsafe, jsonschema, isodate, werkzeug, urllib3, PyYAML, pytz, openapi-schema-validator, jinja2, itsdangerous, idna, greenlet, defusedxml, click, chardet, certifi, WTForms, text-unidecode, sqlalchemy, requests, python3-openid, pyjwt, pycparser, openapi-spec-validator, marshmallow, inflection, flask, dnspython, clickclick, Babel, apispec, typing-extensions, swagger-ui-bundle, sqlalchemy-utils, pytzdata, python-slugify, python-editor, python-dateutil, pygments, prison, numpy, natsort, marshmallow-sqlalchemy, marshmallow-enum, Mako, lockfile, flask-wtf, Flask-SQLAlchemy, Flask-OpenID, flask-login, Flask-JWT-Extended, Flask-Babel, email-validator, docutils, connexion, commonmark, colorama, cffi, unicodecsv, termcolor, tenacity, tabulate, sqlalchemy-jsonfield, setproctitle, rich, python-nvd3, python-daemon, psutil, pendulum, pandas, marshmallow-oneofschema, markdown, lazy-object-proxy, iso8601, importlib-resources, gunicorn, graphviz, flask-caching, flask-appbuilder, dill, cryptography, croniter, colorlog, cattrs, cached-property, argcomplete, apache-airflow-providers-sqlite, apache-airflow-providers-imap, apache-airflow-providers-http, apache-airflow-providers-ftp, alembic, pyodbc, apache-airflow
[2021-04-19 15:27:35,766] {process_utils.py:141} INFO - Successfully installed Babel-2.9.0 Flask-Babel-1.0.0 Flask-JWT-Extended-3.25.1 Flask-OpenID-1.2.5 Flask-SQLAlchemy-2.5.1 Mako-1.1.4 PyYAML-5.4.1 WTForms-2.3.3 alembic-1.5.8 apache-airflow-2.0.1 apache-airflow-providers-ftp-1.0.1 apache-airflow-providers-http-1.1.1 apache-airflow-providers-imap-1.0.1 apache-airflow-providers-sqlite-1.0.2 apispec-3.3.2 argcomplete-1.12.2 attrs-20.3.0 cached-property-1.5.2 cattrs-1.5.0 certifi-2020.12.5 cffi-1.14.5 chardet-4.0.0 click-7.1.2 clickclick-20.10.2 colorama-0.4.4 colorlog-5.0.1 commonmark-0.9.1 connexion-2.7.0 croniter-0.3.37 cryptography-3.4.7 defusedxml-0.7.1 dill-0.3.3 dnspython-2.1.0 docutils-0.17.1 email-validator-1.1.2 flask-1.1.2 flask-appbuilder-3.1.1 flask-caching-1.10.1 flask-login-0.4.1 flask-wtf-0.14.3 graphviz-0.16 greenlet-1.0.0 gunicorn-19.10.0 idna-2.10 importlib-metadata-1.7.0 importlib-resources-1.5.0 inflection-0.5.1 iso8601-0.1.14 isodate-0.6.0 itsdangerous-1.1.0 jinja2-2.11.3 jsonschema-3.2.0 lazy-object-proxy-1.6.0 lockfile-0.12.2 markdown-3.3.4 markupsafe-1.1.1 marshmallow-3.11.1 marshmallow-enum-1.5.1 marshmallow-oneofschema-2.1.0 marshmallow-sqlalchemy-0.23.1 natsort-7.1.1 numpy-1.20.2 openapi-schema-validator-0.1.5 openapi-spec-validator-0.3.0 pandas-1.2.4 pendulum-2.1.2 prison-0.1.3 psutil-5.8.0 pycparser-2.20 pygments-2.8.1 pyjwt-1.7.1 pyodbc-4.0.30 pyrsistent-0.17.3 python-daemon-2.3.0 python-dateutil-2.8.1 python-editor-1.0.4 python-nvd3-0.15.0 python-slugify-4.0.1 python3-openid-3.2.0 pytz-2021.1 pytzdata-2020.1 requests-2.25.1 rich-9.2.0 setproctitle-1.2.2 six-1.15.0 sqlalchemy-1.4.9 sqlalchemy-jsonfield-1.0.0 sqlalchemy-utils-0.37.0 swagger-ui-bundle-0.0.8 tabulate-0.8.9 tenacity-6.2.0 termcolor-1.1.0 text-unidecode-1.3 typing-extensions-3.7.4.3 unicodecsv-0.14.1 urllib3-1.26.4 werkzeug-1.0.1 zipp-3.4.1
[2021-04-19 15:27:36,306] {process_utils.py:135} INFO - Executing cmd: /tmp/venvxuo1gayl/bin/python /tmp/venvxuo1gayl/script.py /tmp/venvxuo1gayl/script.in /tmp/venvxuo1gayl/script.out /tmp/venvxuo1gayl/string_args.txt
[2021-04-19 15:27:36,312] {process_utils.py:137} INFO - Output:
[2021-04-19 15:27:36,875] {process_utils.py:141} INFO - [[34m2021-04-19 15:27:36,875[0m] {[34mlogging_config.py:[0m59} ERROR[0m - Unable to load the config, contains a configuration error.[0m
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - Traceback (most recent call last):
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - File "/usr/lib/python3.7/logging/config.py", line 93, in _resolve
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - found = getattr(found, n)
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - AttributeError: module 'airflow' has no attribute 'utils'
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO -
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - During handling of the above exception, another exception occurred:
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO -
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - Traceback (most recent call last):
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - File "/usr/lib/python3.7/logging/config.py", line 542, in configure
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - formatters[name])
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - File "/usr/lib/python3.7/logging/config.py", line 672, in configure_formatter
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - c = _resolve(cname)
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - File "/usr/lib/python3.7/logging/config.py", line 96, in _resolve
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - found = getattr(found, n)
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO - AttributeError: module 'airflow' has no attribute 'utils'
[2021-04-19 15:27:36,876] {process_utils.py:141} INFO -
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - The above exception was the direct cause of the following exception:
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO -
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - Traceback (most recent call last):
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - File "/tmp/venvxuo1gayl/script.py", line 132, in <module>
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - res = weather_update(*arg_dict["args"], **arg_dict["kwargs"])
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - File "/tmp/venvxuo1gayl/script.py", line 40, in weather_update
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - from airflow.models import Variable
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - File "/tmp/venvxuo1gayl/lib/python3.7/site-packages/airflow/__init__.py", line 46, in <module>
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - settings.initialize()
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - File "/tmp/venvxuo1gayl/lib/python3.7/site-packages/airflow/settings.py", line 434, in initialize
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - LOGGING_CLASS_PATH = configure_logging()
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - File "/tmp/venvxuo1gayl/lib/python3.7/site-packages/airflow/logging_config.py", line 62, in configure_logging
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - raise e
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - File "/tmp/venvxuo1gayl/lib/python3.7/site-packages/airflow/logging_config.py", line 57, in configure_logging
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - dictConfig(logging_config)
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - File "/usr/lib/python3.7/logging/config.py", line 799, in dictConfig
[2021-04-19 15:27:36,877] {process_utils.py:141} INFO - dictConfigClass(config).configure()
[2021-04-19 15:27:36,878] {process_utils.py:141} INFO - File "/usr/lib/python3.7/logging/config.py", line 545, in configure
[2021-04-19 15:27:36,878] {process_utils.py:141} INFO - 'formatter %r' % name) from e
[2021-04-19 15:27:36,878] {process_utils.py:141} INFO - ValueError: Unable to configure formatter 'airflow_coloured'
[2021-04-19 15:27:37,210] {taskinstance.py:1455} ERROR - Command '['/tmp/venvxuo1gayl/bin/python', '/tmp/venvxuo1gayl/script.py', '/tmp/venvxuo1gayl/script.in', '/tmp/venvxuo1gayl/script.out', '/tmp/venvxuo1gayl/string_args.txt']' returned non-zero exit status 1.
Traceback (most recent call last):
File "/opt1/python_envs/airflow/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1112, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/opt1/python_envs/airflow/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1285, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/opt1/python_envs/airflow/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 1315, in _execute_task
result = task_copy.execute(context=context)
File "/opt1/python_envs/airflow/lib/python3.7/site-packages/airflow/operators/python.py", line 493, in execute
super().execute(context=serializable_context)
File "/opt1/python_envs/airflow/lib/python3.7/site-packages/airflow/operators/python.py", line 117, in execute
return_value = self.execute_callable()
File "/opt1/python_envs/airflow/lib/python3.7/site-packages/airflow/operators/python.py", line 531, in execute_callable
string_args_filename,
File "/opt1/python_envs/airflow/lib/python3.7/site-packages/airflow/utils/process_utils.py", line 145, in execute_in_subprocess
raise subprocess.CalledProcessError(exit_code, cmd)
subprocess.CalledProcessError: Command '['/tmp/venvxuo1gayl/bin/python', '/tmp/venvxuo1gayl/script.py', '/tmp/venvxuo1gayl/script.in', '/tmp/venvxuo1gayl/script.out', '/tmp/venvxuo1gayl/string_args.txt']' returned non-zero exit status 1.
[2021-04-19 15:27:37,212] {taskinstance.py:1503} INFO - Marking task as FAILED. dag_id=weather_update, task_id=updating_forecasts, execution_date=20210419T122709, start_date=20210419T122710, end_date=20210419T122737
[2021-04-19 15:27:37,227] {configuration.py:356} WARNING - section/key [smtp/smtp_user] not found in config
解决方案
您似乎混淆了 PythonVirtualenvOperator 和PythonOperator的用例。
如果您只是想在任务中运行 Python 可调用（callable_virtualenv()在您的情况下），您可以使用 PythonOperator。在这种情况下，无论您是在虚拟环境、系统范围内还是使用 Docker 安装 Airflow。
您的代码中发生的情况如下：PythonVirtualenvOperator创建另一个虚拟环境（与运行 Airflow 的环境完全无关），将 Airflow 安装到其中，并尝试导入Variable. 但是这个另一个 Airflow 安装没有配置，这就是你得到这些异常的原因。您可以将AIRFLOW_HOME第二个 Airflow 安装的环境变量设置为与第一个 Airflow 安装使用的目录相同的目录，这实际上应该可以工作，但对我来说这看起来有点过头了。
因此，您可以做的是安装colorama到您安装 Airflow 的同一环境中并替换PythonVirtualenvOperator为PythonOperator.
顺便说一句，print()可调用对象中的那些将被重定向到日志文件中，而不是打印到终端，因此与它们一起使用可能没有多大意义colorama。
推荐阅读
- graphicsmagick - gm convert：此图像格式没有解码委托
- laravel - GItLab 没有将 laravel 应用程序部署到 AWS ec2
- html - 调整浏览器窗口大小时，块元素变得更大并且不会保持垂直居中
- c# - 如何实现类似“如果玩家同时按下 ctrl 和 w”，移动速度（向前力）将为 2000 而不是 1000
- python - 安装 python 包时出现“确认 ssl 证书时出现问题”
- java - 如何将 bytea 转换为 joda.time.DateTime 或时间戳？
- gm-xmlhttprequest - 如何理解 GM_xmlhttpRequest 参数“二进制”
- flutter - 为flutter android项目添加google gms服务时出错并且厌倦了搜索答案
- git - Github 在我出错后检索代码
- latex - LaTeX：使用 TeXmaker 的 beamer 类中的 babel 包存在问题