azure-devops - DBT（数据构建工具）- 在 Azure DevOps 中创建 CI/CD 管道

Question

我正在开发一个使用 Fishtown Analytics 的 DBT 进行 ELT 处理的项目。我正在尝试在 Azure DevOps 中创建一个 CI/CD 管道来自动化构建发布过程，但我无法找到合适的文档。代码已集成到 DevOps Repos 中，现在我需要一个参考来开始构建 CI/CD 管道。

标签： azure-devopsazure-pipelinesdbtcicdfishtown-analytics

Answer 1

Our team uses ADO Pipelines, here’s our doc on how we do it!

To make this example work, you will need 3+ files:

1. profiles.yml.

If you look at this file, you'll see usage of dbt-jinja's env_var macro. This lets you make ADO Pipeline secret variables for your database credentials, and make them available to dbt.

2. step_template.yml

This file is a recipe of deployment steps that are used when deploying to different environments. The beauty of the dbt CLI is that the same steps are used when:

• doing our integration tests (gatekeeper), and
• deploying to our production and staging/UAT environments. So this step_template.yml lets each pipeline re-use the same steps. It's worth noting here that because we are using the AzureCLI@2 task because it will auto-authenticate to Azure for us. If you're not using Azure, you'll need to:
• add Pipeline secrets for username and password to the db, and
• use the Bash@3 task

3. gatekeeper.yml and prod.yml

These are the actual pipelines. If you look, they are identical except for:

• the variables they deploy to different databases), and
• the trigger changes to which branch will trigger this pipeline?
• the schedule should this run even if there are no new commits to the branch? Yes, if production because we want the data to stay up-to-date for our users.