Update dbt docs with more detail on handling table models and upgrade to dbt 1.6 #136
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jeancochrane
commented
Sep 15, 2023
jeancochrane
force-pushed
the
jeancochrane/update-dbt-documentation
branch
from
9dea20e
to
544c43e
Compare
dfsnow
approved these changes
Sep 18, 2023
|
|
||
| ``` | ||
| dbt run --select location.vw_pin10_location default.vw_pin_universe | ||
| dbt clone --state master-cache |
Comment on lines
+475
to
+477
| ``` | ||
| dbt run --select +model.vw_pin_shared_input --exclude location.* proximity.* | ||
| ``` |
There was a problem hiding this comment.
Extremely cool. Wild that dbt is this flexible.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The introduction of our spatial queries as materialized tables into the dbt DAG in #111 creates an additional layer of complexity for local dbt development, since the time to build the full DAG has increased from a couple of minutes to a couple of hours.
This PR updates the dbt README to provide detailed instructions on how to avoid building compute-intensive tables using
dbt cloneand the--excludeoption. In order to support this workflow, we upgrade to the v1.6.x lines of dbt Core and the dbt-athena adapter, which are the first versions to supportdbt clonein addition to being the latest stable minor versions of both libraries.We also take this opportunity to sneak in one small improvement to developer experience by reducing the number of times that dbt-athena will retry a failed query from 5 to 1; in my experience, intermittent Athena failures are very rare compared to SQL syntax and compilation failures, and eliminating the retry logic allows for faster debugging cycles when trying to resolve the latter class of failure.