Fix dbt production deploy problems #82
Conversation
| - name: Configure dbt environment | ||
| uses: ./.github/actions/configure_dbt_environment |
There was a problem hiding this comment.
Run dbt docs generate --target "$TARGET"
18:26:[23](https://github.com/ccao-data/data-architecture/actions/runs/5905883640/job/16020814093#step:6:24) Running with dbt=1.5.5
18:[26](https://github.com/ccao-data/data-architecture/actions/runs/5905883640/job/16020814093#step:6:27):23 Encountered an error:
Runtime Error
The profile 'athena' does not have a target named ''. The valid target names for this profile are:
- dev
- ci
- prod
Error: Process completed with exit code 2.
Previously we had been hardcoding the target "ci" so I hadn't noticed that we were missing this step.
There was a problem hiding this comment.
Nice, I should've caught this too. Tricky tricky
jeancochrane
force-pushed
the
jeancochrane/fix-dbt-prod-deploy-problems
branch
from
e31e62e
to
068ddad
Compare
| - name: Move dbt manifest directory to update cache | ||
| run: mv "$TARGET_DIR" "$STATE_DIR" | ||
| working-directory: ${{ env.PROJECT_DIR }} | ||
| shell: bash |
There was a problem hiding this comment.
Run mv "$TARGET_DIR" "$STATE_DIR"
mv: cannot move 'target' to 'state/target': Directory not empty
Error: Process completed with exit code 1.
Saving the cache explicitly with the actions/cache/save action resolves this problem.
| # We have to use the separate `restore`/`save` actions instead of the | ||
| # unified `cache` action because only `restore` provides access to the | ||
| # `cache-matched-key` and `cache-primary-key` outputs as of v3 |
There was a problem hiding this comment.
This is a frustrating difference! Note the outputs of the restore action vs. the outputs of the cache action.
| # If we restore the cache from the `restore-keys` key, the `cache-hit` | ||
| # output will be 'false' but the `cache-matched-key` output will be | ||
| # the name of the `restore-keys` key; we want to count this case as a hit | ||
| - if: | | ||
| steps.cache.outputs.cache-hit == 'true' || | ||
| steps.cache.outputs.cache-matched-key == format( | ||
| '{0}-master', env.CACHE_NAME | ||
| ) |
There was a problem hiding this comment.
This workflow run has three attempts where I tested each of the possible cases:
- Attempt #1: Branch cache exists; exact cache key hit ➡️ this block evaluates True
- Attempt #2: Deleted branch cache, but
masterbranch cache exists; partial cache key hit ➡️ this block evaluates True - Attempt #3: Deleted both branch cache and
masterbranch cache; no cache key hit ➡️ this block evaluates False
There was a problem hiding this comment.
praise: Sheesh, this is very confusing behavior. And weird that the actions maintainers don't provide access to the cache- vars in actions/cache. Nice work on the fast debugging!
| - name: Configure dbt environment | ||
| uses: ./.github/actions/configure_dbt_environment |
There was a problem hiding this comment.
Nice, I should've caught this too. Tricky tricky
| # If we restore the cache from the `restore-keys` key, the `cache-hit` | ||
| # output will be 'false' but the `cache-matched-key` output will be | ||
| # the name of the `restore-keys` key; we want to count this case as a hit | ||
| - if: | | ||
| steps.cache.outputs.cache-hit == 'true' || | ||
| steps.cache.outputs.cache-matched-key == format( | ||
| '{0}-master', env.CACHE_NAME | ||
| ) |
There was a problem hiding this comment.
praise: Sheesh, this is very confusing behavior. And weird that the actions maintainers don't provide access to the cache- vars in actions/cache. Nice work on the fast debugging!
Three problems that emerged out of merging the data catalog branch into master (#78):
mvcall that is not actually performing the correct directory renaming$MODIFIED_RESOURCES_ONLYcan be set tofalsewhen we want it to betruein cases where we have to restore the cache from therestore-keysfallbackThis PR fixes those three problems.