Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add conditional uniqueness tests for iasWorld tables #98

Merged
merged 13 commits into from
Aug 25, 2023
Merged

Add conditional uniqueness tests for iasWorld tables #98

Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
1eb0a2b
Add source freshness definitions for iasworld tables
jeancochrane Aug 21, 2023
006ade8
Add test_dbt_source_freshness workflow
jeancochrane Aug 21, 2023
d9366cb
Consolidate shared dbt setup steps into setup_dbt composite action on CI
jeancochrane Aug 21, 2023
913db16
Temporarily add config to run test_dbt_source_freshness on PRs for te…
jeancochrane Aug 21, 2023
4fafd6f
Revert "Temporarily add config to run test_dbt_source_freshness on PR…
jeancochrane Aug 21, 2023
a829bfb
Adjust iasworld freshness checks to include prior year's data
jeancochrane Aug 22, 2023
671dc6d
Use YAML anchors for error/warning thresholds in iasworld source fres…
jeancochrane Aug 22, 2023
cf32372
Add `where` flag to unique_combination_of_columns macro and use it in…
jeancochrane Aug 22, 2023
2ca0f81
Add uniqueness tests for htpar and sales tables
jeancochrane Aug 24, 2023
8f0e011
Merge 'master' into jeancochrane/87-add-conditional-uniqueness-tests-…
jeancochrane Aug 24, 2023
6a0f10e
Add additional filter condition to asmt_all test that was missing fro…
jeancochrane Aug 24, 2023
a6f876a
Move where cluase in unique_combination_of_columns macro and add it t…
jeancochrane Aug 24, 2023
8e72252
Switch to using builtin config.where filter for iasworld tests
jeancochrane Aug 25, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 36 additions & 3 deletions dbt/models/iasworld/schema.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,20 @@ sources:
filter: &latest_taxyr taxyr >= date_format(current_date - interval '1' year, '%Y')
warn_after: &24_hours {count: 24, period: hour}
error_after: &48_hours {count: 48, period: hour}
tests:
- unique_combination_of_columns:
name: asmt_all_unique_by_parid_procname_and_taxyr
combination_of_columns:
- parid
- procname
- taxyr
config:
where: >-
cur = 'Y' and
deactivat is null and
procname in ('CCAOVALUE', 'CCAOFINAL', 'BORVALUE') and
valclass is null
error_if: ">125"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought (dreadful): It's going to be so much fun to figure out what all these errors are...

- name: asmt_hist
- name: cname
- name: comdat
Expand Down Expand Up @@ -73,6 +87,17 @@ sources:
filter: *latest_taxyr
warn_after: *24_hours
error_after: *48_hours
tests:
- unique_combination_of_columns:
name: htpar_unique_by_parid_caseno_taxyr_subkey
combination_of_columns:
- parid
- caseno
- taxyr
- subkey
config:
where: cur = 'Y' and deactivat is null
error_if: ">2"
- name: land
description: '{{ doc("land") }}'
tests:
Expand All @@ -89,7 +114,7 @@ sources:
name: legdat_unique_by_parid_taxyr
combination_of_columns:
- parid
- taxyr
- taxyr
- name: lpmod
- name: lpnbhd
- name: oby
Expand All @@ -109,21 +134,29 @@ sources:
name: owndat_unique_by_parid_taxyr
combination_of_columns:
- parid
- taxyr
- taxyr
- name: pardat
description: '{{ doc("pardat") }}'
tests:
- unique_combination_of_columns:
name: pardat_unique_by_parid_taxyr
combination_of_columns:
- parid
- taxyr
- taxyr
- name: permit
freshness:
filter: date_format(date_parse(permdt, '%Y-%m-%d %H:%i:%s.0'), '%Y') >= date_format(current_date - interval '1' year, '%Y')
warn_after: *48_hours
error_after: &72_hours {count: 72, period: hour}
- name: rcoby
- name: sales
tests:
- unique_combination_of_columns:
name: sales_unique_by_parid_instruno
combination_of_columns:
- parid
- instruno
config:
where: substr(saledt, 1, 4) >= '2023'
- name: splcom
- name: valclass
Loading