We've used GitHub Actions as an example, since it's most popular with our users, but you can use any CI/CD tool that can run commands in a terminal.

The repo includes templates for the most popular automations:

1. Add Qyra preview projects to pull requests
2. Deploy dbt changes to your production Qyra project
3. Validate your Qyra project on pull requests
4. Compile your dbt project
5. Refresh your Qyra project

All of these workflows require secrets so that the CI/CD tool can authenticate with Qyra and your data warehouse. We'll go through that first.

Set up secrets and credentials

The workflows read their credentials from repository secrets. Follow GitHub's guide to using secrets in GitHub Actions to add each secret below to your dbt repo.

Every workflow needs these secrets:

• QYRA_API_KEY is a personal access token. Create one in Qyra by going to Settings > Personal Access Tokens.

• QYRA_PROJECT is the UUID for your project. For example, if your URL looks like https://app.qyraflow.com/projects/3538ab33-dc90-aabb-bc00-e50bba3a5f69/tables, then 3538ab33-dc90-45f0-aabb-e50bba3a5f69 is your QYRA_PROJECT.

• QYRA_URL is https://eu1.qyraflow.com or https://app.qyraflow.com for Starter customers, or something like https://your_company.qyraflow.com for dedicated instances. If you self-host, this should be your own custom domain.

Workflows that compile your dbt project in CI (previews, deploy, validate, and compile) also need a DBT_PROFILES secret with the warehouse connection dbt should use. The refresh workflow doesn't need it.

DBT_PROFILES tips:

• You might be able to copy a bunch of the information from your local profiles.yml file. You can see what's in there by typing cat ~/.dbt/profiles.yml in your terminal.
• If you have a separate prod and dev profile, you probably want to use the information from your prod profile for your GitHub action.
• If you want to have different connection settings depending on the user that opened the pull request (dev profiles), then check out this guide.

Find your data warehouse from the list below to get a profiles.yml file template. Fill out this template, and this is your DBT_PROFILES secret.

Use a CLI config file instead of environment variables

If you prefer to authenticate the Qyra CLI with its config.yaml file instead of environment variables, copy the full contents of your local ~/.config/qyra/config.yaml into a CLI_CONFIG secret, then add this step to your workflow before the Qyra CLI step:

- name: Create config file
  env:
    config: ${{ secrets.CLI_CONFIG }}
  run: |
    mkdir -p $HOME/.config/qyra
    echo -e "$config" > $HOME/.config/qyra/config.yaml

Both options are compatible, and environment variables take priority if you use both.

Now that you have your secrets set up, you can use them in the sections below to automate your Qyra CLI workflow.

Add previews to pull requests

If you've connected Qyra to GitHub, you can setup a github action and get Qyra to create new dynamic preview projects automatically when a new pull request is created, and it will automatically delete the preview project when the pull request is closed or merged.

Create preview workflow

Go to your repo, click on Actions menu, and click on Configure

If you have some GitHub actions in your repo already, click on New workflow, then select setup a workflow yourself.

Now copy this start-preview.yml file from the cli-actions repo

And save by clicking on Start commit

Do the same with this close-preview.yml file.

Use developer credentials

When developing in dbt, you typically have a different set of credentials and dataset/schema than when you are running in production. Here are two options on how to set them up based on the developer that opened the Pull Request.

- uses: actions/github-script@v6
  id: get_pr_creator
  with:
    script: |
      return (
        await github.rest.repos.listPullRequestsAssociatedWithCommit({
          commit_sha: context.sha,
          owner: context.repo.owner,
          repo: context.repo.repo,
        })
      ).data[0].user.login;
    result-encoding: string

Use profile targets

Update your DBT_PROFILES to have 1 target per developer. The target name should be their GitHub username.

jaffle_shop:
  target: prod
  outputs:
    prod:
      type: bigquery
      method: oauth
      keyfile: keyfile.json
      project: jaffle-shop
      dataset: prod
    katie: # example developer 1, should be GitHub username
      type: bigquery
      method: oauth
      keyfile: keyfile.json
      project: jaffle-shop
      dataset: dbt_katie
    jose: # example developer 2
      type: bigquery
      method: oauth
      keyfile: keyfile.json
      project: jaffle-shop
      dataset: dbt_jose

Then, update your GitHub action to use the username as the --target flag for the qyra start-preview command.

run: qyra start-preview --project-dir "$PROJECT_DIR" --profiles-dir . --name ${GITHUB_REF##*/} --target ${{ github.actor }}

Use Github environments

Setup a GitHub environment for each developer where the secrets are specifically for them. The environment name should be their GitHub username. Then, update your GitHub action to use the username as the environment.

jobs:
  preview:
    runs-on: ubuntu-latest
    environment: ${{ github.actor }}

Use dbt cloud schema

If you are using a continuous integration job in dbt cloud, you can use the schema that is created by dbt cloud (dbt_cloud_pr_<job_id>_<pr_id>) for your preview project.

First we need to add an environment variable to your profile.yml file that will be used by dbt to connect to the correct schema.

schema: "{{ env_var('DBT_SCHEMA') }}"

If you are using BigQuery, it should be dataset instead of schema.

Then we need to add a step in the GitHub action to fetch the pull request id.

- uses: actions/github-script@v6
  id: pr_id
  with:
    script: |
      if (context.issue.number) {
        // Return issue number if present
        return context.issue.number;
      } else {
        // Otherwise return issue number from commit
        return (
          await github.rest.repos.listPullRequestsAssociatedWithCommit({
            commit_sha: context.sha,
            owner: context.repo.owner,
            repo: context.repo.repo,
          })
        ).data[0].number;
      }
    result-encoding: string

After that we need to add a new env variable to the step "Qyra CLI start preview" which is the schema that dbt cloud will use.

Note that in this example we assume the job id is 1234. You will need to replace this with the actual job id.

env:
  # ... keep existing env variables
  DBT_SCHEMA: 'dbt_cloud_pr_1234_${{steps.pr_id.outputs.result}}'

Now dbt will use the correct schema when running in the preview environment.

You're done!

Everytime you create a new pull request, a Qyra preview project with your branch name will be created on your organization. You will see this link as a github-actions bot comment in the pull request conversation. Everytime you make a change to that branch, the preview environment will get updated. Once you close or merge your pull request, the preview project will get deleted.

You can see the log on your Github actions page:

Deploy changes to Qyra

If you've connected Qyra to GitHub, you can use a github action to deploy your project automatically whenever new changes get merged to your main branch. This is the easiest way to keep Qyra in sync with your changes to dbt.

Create deploy workflow

Go to your repo, click on Actions menu.

If you don't have any GitHub actions, you'll just need to click on Configure

If you have some GitHub actions in your repo already, click on New workflow, then select setup a workflow yourself.

Now copy this deploy.yml file from the cli-actions repo.

Give it a nice name like deploy-qyra.yml

And commit this to your repo by clicking on Start commit.

You're done!

Everytime you merge a change to your repo, on the main branch, it will automatically deploy your new config into your Qyra projects

You can see the log on the Github actions page

Validate your Qyra project

The qyra-validate.yml workflow runs qyra validate on every pull request and checks that your changes don't break any charts or dashboards in your Qyra project. It comments the validation results on the pull request, and the comment is informational, so it doesn't block merging.

Create validate workflow

Create a new workflow in your repo, just like you did for the preview workflow, and copy in this qyra-validate.yml file.

Every time someone opens or updates a pull request, the workflow validates your project against the changes and posts the results as a comment on the pull request.

If you want validation to block merging instead, see validating a preview environment.

Compile your dbt project

The compile.yml workflow runs qyra compile on every pull request and fails if there are any errors that would break your Qyra project. For example, a metric that references a dimension that doesn't exist.

Create compile workflow

Create a new workflow in your repo, just like you did for the preview workflow, and copy in this compile.yml file.

To understand how compilation works and how to enable strict compilation, see Compilation.

Refresh your Qyra project

The refresh.yml workflow is an alternative to deploying changes. Instead of compiling your dbt project in the workflow using your DBT_PROFILES secret, it runs qyra refresh on every merge to your main branch, and Qyra compiles your project server-side using the connection settings saved in your project.

Create refresh workflow

Create a new workflow in your repo, just like you did for the preview workflow, and copy in this refresh.yml file.

This workflow only needs the QYRA_API_KEY, QYRA_PROJECT, and QYRA_URL secrets.

Not sure whether to deploy or refresh? Read about the differences between the two commands in the CLI reference.