时间戳

首页 > 解决方案 > apache气流窗口中的botocore错误

python - apache气流窗口中的botocore错误

问题描述

我已经在 apache 气流 webUI 中设置了变量和连接，所以当我尝试从 aiflow 挂钩导入 s3 挂钩时，我使用获取 s3 存储桶中的密钥，但它失败了，我收到以下错误：

[2021-08-24 21:59:11,720] {{dagbag.py:246}} ERROR - Failed to import: 
/usr/local/airflow/dags/lesson4.py
webserver_1  | Traceback (most recent call last):
webserver_1  |   File "/usr/local/lib/python3.7/site-packages/airflow/models/dagbag.py", line 
243, in process_file
webserver_1  |     m = imp.load_source(mod_name, filepath)
webserver_1  |   File "/usr/local/lib/python3.7/imp.py", line 171, in load_source
webserver_1  |     module = _load(spec)
webserver_1  |   File "<frozen importlib._bootstrap>", line 696, in _load
webserver_1  |   File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
webserver_1  |   File "<frozen importlib._bootstrap_external>", line 728, in exec_module
webserver_1  |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
webserver_1  |   File "/usr/local/airflow/dags/lesson4.py", line 9, in <module>
webserver_1  |     from airflow.hooks.S3_hook import S3hook
webserver_1  |   File "/usr/local/lib/python3.7/site-packages/airflow/hooks/S3_hook.py", line 
19, in <module>
webserver_1  |     from botocore.exceptions import ClientError
webserver_1  | ModuleNotFoundError: No module named 'botocore'


import datetime
import logging

from airflow import DAG
from airflow.models import Variable
from airflow.operators.python_operator import PythonOperator
#import airflow.hooks.S3_hook as S3Hook
from airflow.hooks.S3_hook import S3Hook


def list_keys():
    hook = S3Hook(aws_conn_id='aws_credentials')
    bucket = Variable.get('s3_bucket')
    prefix = Variable.get('s3_prefix')
    logging.info(f"Listing Keys from {bucket}/{prefix}")
    keys = hook.list_keys(bucket, prefix=prefix)
    for key in keys:
        logging.info(f"- s3://{bucket}/{key}")


dag = DAG(
        'lesson1.exercise4',
        start_date=datetime.datetime.now())

list_task = PythonOperator(
    task_id="list_keys",
    python_callable=list_keys,
    dag=dag

)

谁能告诉我如何解决这个问题

标签： pythonamazon-s3airflowbotocore

解决方案

看起来您没有安装Amazon 提供程序包botocore的 PIP 要求。

您可以使用 pip 在 Airflow 2.1+ 之上安装提供程序包：

pip install apache-airflow-providers-amazon

安装后，如果您airflow info从 CLI 运行，您应该会看到它列在Providers info 部分下。

...
Providers info
apache-airflow-providers-amazon          | 2.0.0
apache-airflow-providers-celery          | 2.0.0
apache-airflow-providers-cncf-kubernetes | 2.0.0
...

如果您使用 Docker Compose 运行 Airflow，此答案可能会帮助您了解如何在容器中安装 pip 包。

推荐阅读