docker - 无法在 Airflow UI 中编辑 spark_default
问题描述
我正在尝试使用以下存储库运行 Airflow 和 Spark 的容器化应用程序, https://github.com/cordon-thiago/airflow-spark
如此处的步骤所示,我需要编辑 spark_default 连接以将我的 DAG 提交到 Spark,但是,我似乎无法这样做。这是我尝试做的时候看到的,
____/ ( ( ) ) \___
/( ( ( ) _ )) ) )\
(( ( )( ) ) ( ) )
((/ ( _( ) ( _) ) ( () ) )
( ( ( (_) (( ( ) .((_ ) . )_
( ( ) ( ( ) ) ) . ) ( )
( ( ( ( ) ( _ ( _) ). ) . ) ) ( )
( ( ( ) ( ) ( )) ) _)( ) ) )
( ( ( \ ) ( (_ ( ) ( ) ) ) ) )) ( )
( ( ( ( (_ ( ) ( _ ) ) ( ) ) )
( ( ( ( ( ) (_ ) ) ) _) ) _( ( )
(( ( )( ( _ ) _) _(_ ( (_ )
(_((__(_(__(( ( ( | ) ) ) )_))__))_)___)
((__) \\||lll|l||/// \_))
( /(/ ( ) ) )\ )
( ( ( ( | | ) ) )\ )
( /(| / ( )) ) ) )) )
( ( ((((_(|)_))))) )
( ||\(|(|)|/|| )
( |(||(||)|||| )
( //|/l|||)|\\ \ )
(/ / // /|//||||\\ \ \ \ _)
-------------------------------------------------------------------------------
Node: 5fce0b10ba4b
-------------------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1952, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1821, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
raise value
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.6/site-packages/flask_admin/base.py", line 69, in inner
return self._run_view(f, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask_admin/base.py", line 368, in _run_view
return fn(self, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask_admin/model/base.py", line 2138, in edit_view
form = self.edit_form(obj=model)
File "/usr/local/lib/python3.6/site-packages/flask_admin/model/base.py", line 1340, in edit_form
return self._edit_form_class(get_form_data(), obj=obj)
File "/usr/local/lib/python3.6/site-packages/wtforms/form.py", line 208, in __call__
return type.__call__(cls, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask_admin/form/__init__.py", line 16, in __init__
super(BaseForm, self).__init__(formdata=formdata, obj=obj, prefix=prefix, **kwargs)
File "/usr/local/lib/python3.6/site-packages/wtforms/form.py", line 274, in __init__
self.process(formdata, obj, data=data, **kwargs)
File "/usr/local/lib/python3.6/site-packages/wtforms/form.py", line 126, in process
if obj is not None and hasattr(obj, name):
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/attributes.py", line 356, in __get__
retval = self.descriptor.__get__(instance, owner)
File "/usr/local/lib/python3.6/site-packages/airflow/models/connection.py", line 212, in get_extra
return fernet.decrypt(bytes(self._extra, 'utf-8')).decode()
File "/usr/local/lib/python3.6/site-packages/cryptography/fernet.py", line 199, in decrypt
raise InvalidToken
cryptography.fernet.InvalidToken
我真的不知道 FERNET_KEY 是什么以及它如何应用在这里。我该如何设置才能运行 Spark 操作?
UDPATE
在我的 Airflow UI 的配置选项卡下,我似乎配置了 fernet_key,
据我所知,这是通过以下命令生成的,
: "${AIRFLOW__CORE__FERNET_KEY:=${FERNET_KEY:=$(python -c "from cryptography.fernet import Fernet; FERNET_KEY = Fernet.generate_key().decode(); print(FERNET_KEY)")}}"
然后导出所有变量,
export \
AIRFLOW__CELERY__BROKER_URL \
AIRFLOW__CELERY__RESULT_BACKEND \
AIRFLOW__CORE__EXECUTOR \
AIRFLOW__CORE__FERNET_KEY \
AIRFLOW__CORE__LOAD_EXAMPLES \
AIRFLOW__CORE__SQL_ALCHEMY_CONN \
这似乎与文档中可用的内容一致。这里有什么问题?
解决方案
您需要生成一个新的 fernet 密钥并将其添加到您的气流配置中。 https://airflow.apache.org/docs/apache-airflow/stable/security/secrets/fernet.html
正如链接所示,fernet 密钥是气流用来加密存储在连接信息中的密码的密钥。在上述情况下,从未设置过 fernet 密钥,因此出现错误 - Invalid token。
Fernet 是对称加密的一种实现。但这将超出此问题的范围!
推荐阅读
- python - 如何使用 Postgres 在 gino 中按 ILIKE 词过滤
- reactjs - 关闭细节面板并在材料表上打开另一个
- postgresql - TimescaleDB 中的 Segment by 需要使用外键 id 进行分段
- firebase - 如何在 Firestore 中构建数据库
- python - 如何根据 tkinkter 菜单中的选定选项触发按钮?
- python - 从 win32com.client.CDispatch 对象获取命名元组变量
- android - Vimeo 视频无法在 android webview (ifame) 中完全适合屏幕播放(在纵向模式下不显示完全适合屏幕)
- c# - 在另一个窗口中打开文件而不阻止视图中的后续调用/并发调用
- kubernetes - 引用唯一命名的项目时,有没有办法不在 kubernetes 中指定命名空间?
- google-sheets - 在 Google 表格上捕获时间戳的公式导致了问题