1

I am working on a project that uses DBT by Fishtown Analytics for ELT processing. I am trying to create a CI/CD pipeline in Azure DevOps to automate the build release process, but I am unable to find a suitable documentation around it. The code has been integrated in DevOps Repos, now I need a reference to start with building the CI/CD pipelines.

user961
  • 453
  • 6
  • 20

1 Answers1

2

Our team uses ADO Pipelines, here’s our doc on how we do it!

To make this example work, you will need 3+ files:

1. profiles.yml.

If you look at this file, you'll see usage of dbt-jinja's env_var macro. This lets you make ADO Pipeline secret variables for your database credentials, and make them available to dbt.

2. step_template.yml

This file is a recipe of deployment steps that are used when deploying to different environments. The beauty of the dbt CLI is that the same steps are used when:

  • doing our integration tests (gatekeeper), and
  • deploying to our production and staging/UAT environments. So this step_template.yml lets each pipeline re-use the same steps. It's worth noting here that because we are using the AzureCLI@2 task because it will auto-authenticate to Azure for us. If you're not using Azure, you'll need to:
  • add Pipeline secrets for username and password to the db, and
  • use the Bash@3 task

3. gatekeeper.yml and prod.yml

These are the actual pipelines. If you look, they are identical except for:

  • the variables they deploy to different databases), and
  • the trigger changes to which branch will trigger this pipeline?
  • the schedule should this run even if there are no new commits to the branch? Yes, if production because we want the data to stay up-to-date for our users.
Anders Swanson
  • 3,637
  • 1
  • 18
  • 43
  • Thank you so much for your help. Can you also help me with brief description of what each file is meant for, the ones present under ado_pipelines_example/build/. – user961 Aug 30 '21 at 09:18
  • 1
    just added more context! – Anders Swanson Aug 30 '21 at 21:45
  • Thank you so much for the explanation. One thing that I have observed is - after setting up profiles.yml, if I make any updates to this file, say changing the authentication, the build pipeline always picks the old version of this file. Another error that it is giving is : dbt_project.yml file [ERROR not found] – user961 Aug 31 '21 at 11:40
  • 1
    for the first error, try changing the trigger for one of the pipelines to be `*`, so that it runs for every commit on every branch? – Anders Swanson Aug 31 '21 at 14:54
  • 1
    resolved the first error, however still stuck with the issue while running the command -dbt debug --profiles-dir $(location) -- dbt_project.yml file [ERROR not found]. Both dbt_project.yml and profiles.yml are present at the exact same location – user961 Aug 31 '21 at 17:53
  • 1
    @user961 wish I could be more helpful here but my only advice is to add steps to print out your current working directory and the files in them. That’ll help you debug I bet – Anders Swanson Aug 31 '21 at 20:38
  • Thanks for your help. I resolved this error by changing the working directory for the Azure CLI task (so that it is able to locate the dbt_project.yml file) as - task: AzureCLI@2 displayName: 'dbt debug' inputs: azureSubscription: XXXX ScriptType: bash scriptLocation: inlineScript workingDirectory: '$(System.DefaultWorkingDirectory)' inlineScript: | dbt --version dbt debug --profiles-dir $(location) – user961 Sep 01 '21 at 08:32