Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deploy dbt docs to GitHub Pages #57

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
86 commits
Select commit Hold shift + click to select a range
88494fd
Define GitHub workflows for building and testing dbt models
jeancochrane Aug 2, 2023
997f3e4
Configure AWS credentials in GitHub Actions build and test workflows
jeancochrane Aug 2, 2023
bc0e000
Centralize dbt env vars in GitHub Actions workflows
jeancochrane Aug 2, 2023
ff4fd81
Tweaks to dbt GitHub Actions workflow definition ahead of testing
jeancochrane Aug 3, 2023
297f7ec
Fix linting problems with dbt GitHub actions and workflows
jeancochrane Aug 3, 2023
0396849
Empty commit to trigger CI build
jeancochrane Aug 3, 2023
f833271
Rename local GitHub actions to match expected dir/action.yaml naming …
jeancochrane Aug 3, 2023
3fb81f6
Add permissions to interact with GitHub OIDC to dbt actions
jeancochrane Aug 3, 2023
5b37bf8
Try new format for build and test cache key on CI
jeancochrane Aug 3, 2023
e8ec9d0
Strip extraneous dollar sign from dbt workflow cache key
jeancochrane Aug 3, 2023
f4ba9de
Try different format for dbt directory paths in workflow env vars
jeancochrane Aug 3, 2023
8530109
Rename dbt workflow env vars to avoid collisions with dbt internal en…
jeancochrane Aug 3, 2023
b26cefc
Make sure STATE_ARGS env var is never empty in build_and_test_dbt wor…
jeancochrane Aug 3, 2023
3ff6d98
Try new format for reading dbt commands from env vars in GitHub workflow
jeancochrane Aug 3, 2023
0607be7
Add step to build_and_test_dbt workflow to test dbt installation
jeancochrane Aug 3, 2023
98a42ef
Try a different quoting scheme for RUN_CMD and TEST_CMD in build_and_…
jeancochrane Aug 3, 2023
03e4dc1
Define build/test commands directly instead of via env vars in dbt wo…
jeancochrane Aug 3, 2023
b99d0da
Log all conditional branches in build_and_test_dbt_models workflow
jeancochrane Aug 3, 2023
829a456
See if removing hyphens from database names appeases dbt-athena on CI
jeancochrane Aug 3, 2023
cb0299b
Merge data-catalog into jeancochrane/31-data-catalog-define-github-ac…
jeancochrane Aug 3, 2023
31de4e0
Temporarily enable dbt debugging to try to figure out AWS permissions
jeancochrane Aug 3, 2023
7b0037a
Try reverting dbt schema naming back to kebab_slugify
jeancochrane Aug 4, 2023
9bf6cac
Remove --debug flag from dbt run call in build_and_test_dbt workflow
jeancochrane Aug 4, 2023
76db394
Bump error thresholds for four dbt tests
jeancochrane Aug 4, 2023
4d552aa
Add step to cleanup resources to build_and_test_dbt workflow
jeancochrane Aug 4, 2023
27f556c
Clean up cleanup_dbt_resources.sh script for use in CI
jeancochrane Aug 7, 2023
15a86d2
Bump allowed errors in dbt tests due to data problems
jeancochrane Aug 7, 2023
7ab4b69
Update build_and_test_dbt workflow to run when PRs are closed
jeancochrane Aug 7, 2023
9304b2f
Try apt-get instead of apt for installing jq in build_and_test_dbt wo…
jeancochrane Aug 7, 2023
b291800
Temporarily disable PR event restriction on dbt cleanup install step …
jeancochrane Aug 7, 2023
b4ec438
Try sudo apt-get for installing jq in build_and_test_dbt workflow
jeancochrane Aug 7, 2023
52fc11a
Remove installation step for dbt cleanup in build_and_test_dbt workflow
jeancochrane Aug 7, 2023
640387f
Enforce jq as a requirement for cleanup_dbt_resources.sh script
jeancochrane Aug 7, 2023
dfe8785
Fix path to cleanup_dbt_resources.sh on CI
jeancochrane Aug 7, 2023
6d685a0
Revert "Remove installation step for dbt cleanup in build_and_test_db…
jeancochrane Aug 7, 2023
1e5331d
Temporarily disable PR event restriction on cleanup in build_and_test…
jeancochrane Aug 7, 2023
620877b
Revert "Enforce jq as a requirement for cleanup_dbt_resources.sh script"
jeancochrane Aug 7, 2023
f2d0508
Revert "Temporarily disable PR event restriction on cleanup in build_…
jeancochrane Aug 7, 2023
cf5995b
Revert "Temporarily disable PR event restriction on dbt cleanup insta…
jeancochrane Aug 7, 2023
d34c83b
Temporarily run test_dbt_models workflow on PRs so we can dispatch it…
jeancochrane Aug 7, 2023
d2909ea
Give more verbose names to dbt workflow jobs
jeancochrane Aug 7, 2023
7538f9e
Revert "Temporarily run test_dbt_models workflow on PRs so we can dis…
jeancochrane Aug 7, 2023
c3a7a62
Try adding push to test_dbt_models workflow definition to test dispatch
jeancochrane Aug 7, 2023
e10540f
Revert "Try adding push to test_dbt_models workflow definition to tes…
jeancochrane Aug 7, 2023
b5cec8d
Add docstring to cleanup_dbt_resources.sh
jeancochrane Aug 7, 2023
68f914c
Run `dbt run` with --defer on CI to inherit built resources
jeancochrane Aug 7, 2023
1d4ec5a
Don't use build cache in test_dbt_models workflow
jeancochrane Aug 7, 2023
8af875c
Try adding push to test_dbt_models workflow definition to test it again
jeancochrane Aug 7, 2023
87353b2
Temporarily add --debug flag to dbt call in test_dbt_models
jeancochrane Aug 7, 2023
37ba0bb
Change `push` to `pull_request` for testing test_dbt_models workflow
jeancochrane Aug 7, 2023
b36fec7
Remove --debug flag from test_dbt_models workflow
jeancochrane Aug 7, 2023
5bdf9a2
Change test_dbt_models workflow to only run on dispatch
jeancochrane Aug 7, 2023
9b4378d
Add draft GitHub Pages deployment workflow
jeancochrane Aug 7, 2023
9708b5c
Configure AWS credentials in deploy_dbt_docs workflow
jeancochrane Aug 8, 2023
687eceb
Run dbt docs generate against ci target in deploy_dbt_docs workflow
jeancochrane Aug 8, 2023
722529e
Test deploy_dbt_docs on pull_request event instead of push
jeancochrane Aug 8, 2023
cef322a
Cache dbt and Python requirements in install_dbt_requirements action
jeancochrane Aug 8, 2023
e51c677
Use sed to strip comment lines in load_environment_variables composit…
jeancochrane Aug 8, 2023
f9b5d86
Kebab case dbt build and test workflow names
jeancochrane Aug 8, 2023
704ed13
Factor out cleanup-dbt-resources into its own workflow
jeancochrane Aug 8, 2023
1738374
Set GITHUB_HEAD_REF var in test_dbt_models workflow
jeancochrane Aug 8, 2023
672a7d4
Bump threshold for vw_pin_appeal dbt test failures
jeancochrane Aug 8, 2023
de2e846
Merge branch 'jeancochrane/31-data-catalog-define-github-actions-work…
jeancochrane Aug 8, 2023
3b3aebe
Set GITHUB_HEAD_REF when running deploy-dbt-docs workflow
jeancochrane Aug 8, 2023
c232ed1
Temporarily add --debug flag to dbt docs generate call in deploy-dbt-…
jeancochrane Aug 8, 2023
86f50d7
Try passing GITHUB_HEAD_REF explicitly to dbt docs generate in deploy…
jeancochrane Aug 8, 2023
ae1be0e
Try appending GITHUB_HEAD_REF to GITHUB_ENV in deploy-dbt-docs workflow
jeancochrane Aug 8, 2023
cae3364
Temporarily log GITHUB_HEAD_REF in generate_schema_name macro for deb…
jeancochrane Aug 8, 2023
780f1d5
Try a different method of passing GITHUB_HEAD_REF to deploy-dbt-docs …
jeancochrane Aug 8, 2023
c65083a
Try using HEAD_REF instead of GITHUB_HEAD_REF in generate_schema_name…
jeancochrane Aug 8, 2023
9a77e31
Undo GITHUB_HEAD_REF -> HEAD_REF renaming in generate_schema_name macro
jeancochrane Aug 8, 2023
60c2bb1
Factor out composite GitHub action for configure_dbt_environment to s…
jeancochrane Aug 8, 2023
dcbe3a0
Merge branch 'jeancochrane/31-data-catalog-define-github-actions-work…
jeancochrane Aug 8, 2023
197e910
Rename GITHUB_HEAD_REF -> HEAD_REF in deploy_dbt_docs workflow
jeancochrane Aug 8, 2023
a3fcc87
Upload pages artifact from fully qualified path in deploy_dbt_docs wo…
jeancochrane Aug 8, 2023
eeb976f
Explicitly specify the files to upload in deploy_dbt_docs workflow
jeancochrane Aug 8, 2023
0aa549b
Only run deploy-dbt-docs workflow after successful build-and-test-dbt…
jeancochrane Aug 8, 2023
949a01b
Merge branch 'data-catalog' into 34-data-catalog-define-github-action…
jeancochrane Aug 8, 2023
ee4a760
Use top-level `if` condition in deploy-dbt-docs workflow job
jeancochrane Aug 9, 2023
ec2f703
Test running deploy-dbt-docs workflow on PR branch
jeancochrane Aug 9, 2023
7559fab
Bump error thresholds in dbt tests
jeancochrane Aug 9, 2023
291c1a5
Try removing `if` conditional from deploy-dbt-docs workflow
jeancochrane Aug 9, 2023
ab72942
Try removing workflow_dispatch from deploy-dbt-docs workflow
jeancochrane Aug 9, 2023
4d4c10b
Revert "Try removing workflow_dispatch from deploy-dbt-docs workflow"
jeancochrane Aug 9, 2023
4784ed3
Revert "Try removing `if` conditional from deploy-dbt-docs workflow"
jeancochrane Aug 9, 2023
a72801a
Revert "Test running deploy-dbt-docs workflow on PR branch"
jeancochrane Aug 9, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
60 changes: 60 additions & 0 deletions .github/workflows/deploy_dbt_docs.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
name: deploy-dbt-docs

on:
workflow_run:
dfsnow marked this conversation as resolved.
Show resolved Hide resolved
workflows: [build-and-test-dbt]
branches: [master, data-catalog]
types: [completed]
dfsnow marked this conversation as resolved.
Show resolved Hide resolved
workflow_dispatch:

jobs:
deploy-dbt-docs:
runs-on: ubuntu-latest
# These permissions are required to make a GitHub Pages deployment
permissions:
pages: write # To deploy to Pages
id-token: write # To verify the deployment comes from an valid source
if: github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success'
steps:
- name: Checkout
uses: actions/checkout@v3

- name: Install dbt requirements
uses: ./.github/actions/install_dbt_requirements

- name: Load environment variables
uses: ./.github/actions/load_environment_variables

- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v2
with:
role-to-assume: ${{ secrets.AWS_IAM_ROLE_TO_ASSUME_ARN }}
aws-region: us-east-1

- name: Generate docs
# Target is currently set to CI because we expect this action to be
# run against the long-lived data-catalog branch, but we should change
# this to prod when we merge that branch into master
run: dbt docs generate --target ci
working-directory: ${{ env.PROJECT_DIR }}
shell: bash
env:
HEAD_REF: data-catalog

- name: Package doc files for upload
run: |
mkdir _site
for file in index.html catalog.json manifest.json; do
cp "target/$file" "_site/$file"
done
dfsnow marked this conversation as resolved.
Show resolved Hide resolved
working-directory: ${{ env.PROJECT_DIR }}
shell: bash

- name: Upload docs directory artifact
uses: actions/upload-pages-artifact@v2
with:
path: ${{ format('{0}/_site', env.PROJECT_DIR) }}

- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages@v2
6 changes: 3 additions & 3 deletions dbt/models/default/schema.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ models:
- year
- case_no
config:
error_if: ">365894"
error_if: ">365905"
# `change` should be an enum
- dbt_utils.expression_is_true:
name: vw_pin_appeal_no_unexpected_change_values
Expand Down Expand Up @@ -111,7 +111,7 @@ models:
- mail_address_zipcode_1
- mail_address_zipcode_2
config:
error_if: ">880581"
error_if: ">880607"
# TODO: Mailing address changes after validated sale(?)
# TODO: Site addresses are all in Cook County
- name: vw_pin_condo_char_test
Expand Down Expand Up @@ -180,6 +180,6 @@ models:
group_column: year
count_column: class
config:
error_if: ">23"
error_if: ">24"
# TODO: Data completeness correlates with availability of spatial data
# by year