python - 在 Google Cloud Composer 中浏览 DAG 任务信息时,Airflow 会抛出与时区相关的 TypeError
问题描述
我正在使用具有以下定义的Google Cloud Composer运行 Airflow DAG:
with airflow.DAG(
'build_daily_rollups',
default_args={
'owner': 'airflow',
'start_date': datetime(2019, 5, 7, 6, tzinfo=new_york_tz),
'concurrency': 1,
'retries': 1 # safe b/c this DAG is idempotent
},
schedule_interval=timedelta(hours=24),
catchup=False
) as dag:
roll_up_tasks = []
for feed_id in FEEDMAP:
task_id = f'roll_up_feed_{feed_id}'
task = PythonOperator(
task_id=task_id,
params={'feed_id': feed_id, 'ts_f_strings': ts_f_strings},
python_callable=parse_feed,
provide_context=True
)
dag.add_task(task)
roll_up_tasks.append(task)
packager = PythonOperator(
task_id='build_package',
python_callable=build_package
)
for task in roll_up_tasks:
task >> packager
(我省略了parse_feed
可调用代码,因为它与这个问题不是特别相关)
我触发了这个 DAG 的手动运行。DAG 进入“运行”状态,然后什么都没有发生——没有任务被调度甚至排队,DAG 中每个任务的状态保持不变null
。在尝试导航到此 DAG 的“任务实例详细信息”窗格时,我收到以下错误:
Traceback (most recent call last):
File "/opt/python3.6/lib/python3.6/site-packages/flask/app.py", line 1988, in wsgi_app
response = self.full_dispatch_request()
File "/opt/python3.6/lib/python3.6/site-packages/flask/app.py", line 1641, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/opt/python3.6/lib/python3.6/site-packages/flask/app.py", line 1544, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/opt/python3.6/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/opt/python3.6/lib/python3.6/site-packages/flask/app.py", line 1639, in full_dispatch_request
rv = self.dispatch_request()
File "/opt/python3.6/lib/python3.6/site-packages/flask/app.py", line 1625, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/opt/python3.6/lib/python3.6/site-packages/flask_admin/base.py", line 69, in inner
return self._run_view(f, *args, **kwargs)
File "/opt/python3.6/lib/python3.6/site-packages/flask_admin/base.py", line 368, in _run_view
return fn(self, *args, **kwargs)
File "/opt/python3.6/lib/python3.6/site-packages/flask_login.py", line 755, in decorated_view
return func(*args, **kwargs)
File "/usr/local/lib/airflow/airflow/www/utils.py", line 262, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/airflow/airflow/www/views.py", line 788, in task
dep_context=dep_context)]
File "/usr/local/lib/airflow/airflow/www/views.py", line 786, in <listcomp>
failed_dep_reasons = [(dep.dep_name, dep.reason) for dep in
File "/usr/local/lib/airflow/airflow/models.py", line 1210, in get_failed_dep_statuses
dep_context):
File "/usr/local/lib/airflow/airflow/ti_deps/deps/base_ti_dep.py", line 100, in get_dep_statuses
for dep_status in self._get_dep_statuses(ti, session, dep_context):
File "/usr/local/lib/airflow/airflow/ti_deps/deps/exec_date_after_start_date_dep.py", line 24, in _get_dep_statuses
if ti.task.start_date and ti.execution_date < ti.task.start_date:
TypeError: can't compare offset-naive and offset-aware datetimes
这个 DAG 定义必须是时区感知的,这个参数化是我从阅读 Airflow 文档的相关部分中得到的。此外,我能够在我的个人计算机上的本地 Airflow 实例中成功导航到此页面。
此错误的可能原因是什么?
解决方案
根据Google 文档,Composer 中 Airflow 的默认版本是 1.9。从 Airflow 1.10 开始支持时区感知 DAG。
您应该在 Composer 环境中使用 Airflow 1.10。
推荐阅读
- c# - 在 DFT 中对任意点进行采样?
- drupal - Drupal\Core\Asset\JsCollectionOptimizer (Flexslider) Drupal 8 中没有这样的文件或目录
- ubuntu-18.04 - Lightsail 启动脚本是否仅在新实例初始化时运行?
- http - 通过http文件打开后更改服务器上ftp文件的目录
- python-3.x - taggit 模块在生产服务器中找不到错误(python 任何地方)
- swift - 如何在collectionView中实现三个项目的网格
- android - 使用 Robolectric 测试 TextView 行数
- java - 我无法使用 Firebase 存储编辑封面图片和个人资料图片。我的程序有什么问题?
- swift - 如何在没有百分比编码的情况下在 Swift URL 中包含重音字符?
- iteration - Dymola 初始化迭代变量