amazon-web-services - AWS Managed Airflow 将文件夹上传到 MWAA 环境

Question

我将开始使用 AWS 管理的气流。为了让受管气流访问 dags，我需要将我的代码上传到 s3 存储桶中的 dags/ 目录，MWAA 会接收它。

但是，在我的代码库中，我在其他目录中有代码，例如tasks/目录。问题是当我将tasks文件夹上传到 s3 存储桶时，mwaa 没有提取它们，并且我的 dag 出现导入错误。

AWS 文档没有为此提供任何指导。我想知道以前有没有人这样做过？还是我必须将所有代码上传到dags/文件夹中？

标签： amazon-web-servicesairflowmwaa

Answer 1

What we do for MWAA using airflow version 1.10 is that whatever custom code we write that needs to be used in dags, is added as plugins. Currently you can only have dags deployed based on the s3 folder/key that you specifiy. We have our custom code in a separate folder that is then zipped and configured mwaa as the plugins location.

Im not sure what your tasks folder holds and what sort of code it has, but below is an example of our code structure.

src
    ------> dags (dir)
                  ----------> dag1.py
                  ----------> dag2.py
                  ----------> dag3.py
    -------> plugins(dir)
                    ----------> __init__.py
                    ----------> common (dir)
                                            -----> __init__.py
                                            -----> something.py
                    -----------> hooks (dir)
                                            ----> __init__.py
                                            ---> somehook.py
                    ------------>operators(dir)
                                           ----> __init__py
                                           ----> op1.py
                                           ----> op2.py

Using the above structure dags are deployed as is in the dags folder Plugin folder is completely zipped and uploaded into s3 to the location specified in the plugin location in mwaa