Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add shared / updated column documentation to default views #168

Merged
merged 57 commits into from
Oct 25, 2023
Merged
Show file tree
Hide file tree
Changes from 49 commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
0af792e
Add initial shared column definitions
dfsnow Oct 16, 2023
fc43857
Add more Cook column definitions to views
dfsnow Oct 16, 2023
08ad9ce
Add spatial column definitions
dfsnow Oct 16, 2023
72efbf0
Add initial iasWorld column comment dump
dfsnow Oct 17, 2023
5f9237c
Add separate column definition file
dfsnow Oct 17, 2023
d0ea9ed
Replace static column names with ones filled from mardown file
dfsnow Oct 17, 2023
04ea04e
Cleanup syntax errors
dfsnow Oct 17, 2023
77c8dc0
Cleanup column descriptions
dfsnow Oct 17, 2023
9ba5abf
Fix superlinter email spelling
dfsnow Oct 17, 2023
a853baf
Fix line ending
dfsnow Oct 17, 2023
3503d1c
Merge branch 'dansnow/add-iasworld-column-docs' into dansnow/add-shar…
dfsnow Oct 17, 2023
31ebfac
Rename top-level column prefix to shared_column
dfsnow Oct 17, 2023
1374f83
Alphabetize doc blocks
dfsnow Oct 17, 2023
fc68cf3
Changed CDU description
dfsnow Oct 17, 2023
f413b40
Split iasWorld schema file into tables
dfsnow Oct 17, 2023
f0d3f7c
Fix freshness check formatting
dfsnow Oct 17, 2023
553f687
Alphabetize source tests
dfsnow Oct 17, 2023
029e148
Move exposures to their own yaml + alphabetize them
dfsnow Oct 17, 2023
546d57a
Split default schema.yml into separate files
dfsnow Oct 17, 2023
f55c3cd
Add prefix to default schema files
dfsnow Oct 17, 2023
fcf87c7
Add prefix to iasworld schema files
dfsnow Oct 17, 2023
4c15ff7
Drop version: top-level key
dfsnow Oct 18, 2023
db8b0ef
Drop extra linebreak from exposures
dfsnow Oct 18, 2023
21139c1
Move loaded_at_field key to each iasworld schema file
dfsnow Oct 18, 2023
a3dfe2a
Merge branch 'dansnow/split-large-schema-files' into dansnow/add-shar…
dfsnow Oct 18, 2023
0a31ff6
Merge branch 'master' into dansnow/add-shared-column-docs
dfsnow Oct 18, 2023
b5db6d8
Move char columns to shared_columns.md
dfsnow Oct 19, 2023
7066f84
Reoganize char columns
dfsnow Oct 19, 2023
80c8d52
Fix missing chunk of shared_columns
dfsnow Oct 19, 2023
1cafc6b
Move all card_res_char descriptions to shared_columns.md
dfsnow Oct 19, 2023
90b0fdb
Add definitions for all other res char_ cols
dfsnow Oct 19, 2023
67273e8
Add missing card_res_char columns
dfsnow Oct 19, 2023
1791fe7
Merge branch 'master' into dansnow/add-shared-column-docs
dfsnow Oct 19, 2023
3e84660
Move address columns to shared_columns.md
dfsnow Oct 19, 2023
b2952be
Add address shared columns
dfsnow Oct 23, 2023
47fe95e
Add defs for PIN history and appeals
dfsnow Oct 24, 2023
d7c5d2d
Fix missing cols in vw_pin_appeal
dfsnow Oct 24, 2023
10de4e0
Add condo shared defs
dfsnow Oct 24, 2023
df833f7
Fix doc block typo
dfsnow Oct 24, 2023
3be4f95
Wrap adding descriptions to condo view
dfsnow Oct 24, 2023
bf3fbc4
Add exempt view descriptions
dfsnow Oct 24, 2023
623f064
Add land view defs
dfsnow Oct 24, 2023
75ab9c6
Add defs for vw_pin_value
dfsnow Oct 24, 2023
2292325
Add sale table column defs
dfsnow Oct 24, 2023
2867f19
Add initial PIN universe defs
dfsnow Oct 24, 2023
bf879f7
Fix col name typos
dfsnow Oct 24, 2023
d0de565
Fix spacing issue
dfsnow Oct 24, 2023
bcb9295
Fix even more spacing
dfsnow Oct 24, 2023
66cc838
Fix column def typo
dfsnow Oct 24, 2023
137c61e
Update dbt/models/default/schema/default.vw_pin_condo_char.yml
dfsnow Oct 25, 2023
9a49b8e
Update dbt/models/shared_columns.md
dfsnow Oct 25, 2023
ef888c2
Update dbt/models/shared_columns.md
dfsnow Oct 25, 2023
6caae57
Update dbt/models/shared_columns.md
dfsnow Oct 25, 2023
95c5ba8
Update dbt/models/default/schema/default.vw_pin_condo_char.yml
dfsnow Oct 25, 2023
5609dc2
Update dbt/models/default/schema/default.vw_pin_condo_char.yml
dfsnow Oct 25, 2023
e8ea7aa
Update dbt/models/default/schema/default.vw_pin_sale.yml
dfsnow Oct 25, 2023
ccd39c7
Update sale price language
dfsnow Oct 25, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions dbt/models/default/docs.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,6 @@ View that standardizes residential property characteristics for use in modeling
and reporting.

### Assumptions
`char_renovation` (`dweldat.user3`) is updated when HI permits are filed.
jeancochrane marked this conversation as resolved.
Show resolved Hide resolved

### Nuance
Land is parcel-level.
Expand Down Expand Up @@ -101,4 +100,4 @@ Source of truth view for PIN location.

{% docs vw_pin_land %}
View containing aggregated land square footage.
{% enddocs %}
{% enddocs %}
175 changes: 97 additions & 78 deletions dbt/models/default/schema/default.vw_card_res_char.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,87 +6,106 @@ models:
- type_res

columns:
- name: char_air
description: '{{ doc("shared_column_char_air") }}'
- name: char_attic_fnsh
description: '{{ doc("shared_column_char_attic_fnsh") }}'
- name: char_attic_type
description: '{{ doc("shared_column_char_attic_type") }}'
- name: char_bsmt
description: '{{ doc("shared_column_char_bsmt") }}'
- name: char_bsmt_fin
description: '{{ doc("shared_column_char_bsmt_fin") }}'
- name: char_ext_wall
description: '{{ doc("shared_column_char_ext_wall") }}'
- name: card
description: '{{ doc("shared_column_card") }}'
- name: card_proration_rate
description: |
Proration rate within a given PIN.

Some PINs have multiple improvements, known as "cards". Total value for a PIN should be allocated across cards accordingly.
description: '{{ doc("shared_column_card_proration_rate") }}'
- name: cdu
description: '{{ doc("shared_column_cdu") }}'
- name: char_age
description: '{{ doc("shared_column_char_age") }}'
- name: char_apts
description: '{{ doc("shared_column_char_apts") }}'
- name: char_beds
description: '{{ doc("shared_column_char_beds") }}'
- name: char_bldg_sf
description: '{{ doc("shared_column_char_bldg_sf") }}'
- name: char_cnst_qlty
description: '{{ doc("shared_column_char_cnst_qlty") }}'
- name: char_fbath
description: '{{ doc("shared_column_char_fbath") }}'
- name: char_frpl
description: '{{ doc("shared_column_char_frpl") }}'
- name: char_gar1_area
description: '{{ doc("shared_column_char_gar_area") }}'
- name: char_gar1_att
description: '{{ doc("shared_column_char_gar_att") }}'
- name: char_gar1_cnst
description: '{{ doc("shared_column_char_gar_cnst") }}'
- name: char_gar1_size
description: '{{ doc("shared_column_char_gar_size") }}'
- name: char_heat
description: '{{ doc("shared_column_char_heat") }}'
- name: char_hbath
description: '{{ doc("shared_column_char_hbath") }}'
- name: char_land_sf
description: '{{ doc("shared_column_char_land_sf") }}'
- name: char_ncu
description: '{{ doc("shared_column_char_ncu") }}'
- name: char_porch
description: '{{ doc("shared_column_char_porch") }}'
- name: char_recent_renovation
description: '{{ doc("shared_column_char_recent_renovation") }}'
- name: char_renovation
description: '{{ doc("shared_column_char_renovation") }}'
- name: char_repair_cnd
description: '{{ doc("shared_column_char_repair_cnd") }}'
- name: char_roof_cnst
description: '{{ doc("shared_column_char_roof_cnst") }}'
- name: char_rooms
description: '{{ doc("shared_column_char_rooms") }}'
- name: char_site
description: '{{ doc("shared_column_char_site") }}'
- name: char_tp_dsgn
description: '{{ doc("shared_column_char_tp_dsgn") }}'
- name: char_tp_plan
description: '{{ doc("shared_column_char_tp_plan") }}'
- name: char_type_resd
description: '{{ doc("shared_column_char_type_resd") }}'
- name: char_use
description: '{{ doc("shared_column_char_use") }}'
- name: char_yrblt
description: '{{ doc("shared_column_char_yrblt") }}'
- name: class
description: '{{ doc("shared_column_class") }}'
- name: pin
description: '{{ doc("shared_column_pin") }}'
- name: pin_is_multiland
description: Indicates the PIN has more than one landline
description: '{{ doc("shared_column_pin_is_multiland") }}'
- name: pin_is_multicard
description: Indicator for a PIN with more than one building (ADU, coach house, etc.)
- name: char_tp_dsgn
description: |
Deprecated.

Field has not been updated recently enough to be useful for modeling
- name: char_site
description: |
Deprecated.

This field lacks sufficient variation to be useful for modeling
- name: char_rooms
description: Number of total rooms in the building (excluding baths). Not to be confused with bedrooms
- name: char_repair_cnd
description: |
Deprecated.

This field is subjective and contains little variation. As such, it is not used for modeling
- name: char_recent_renovation
description: |
Indicates whether or not a property was renovated within the last 3 years.

Renovation is indicated by the char_renovation characteristic flipping from "NO" to "YES"
- name: char_ncu
description: |
Number of commercial units.

The vast majority are for properties with class 212
- name: char_land_sf
description: |
Square footage of the land (not just the building) of the property.

A single PIN can have multiple "land lines," meaning it can be associated with more than one 200-class land lot
- name: char_hbath
description: Number of half baths, defined as bathrooms without a shower or bathtub
- name: char_gar1_area
descripion: |
Summary forthcoming.

Is Garage 1 physically included within the building area? If yes, the garage area is subtracted from the building square feet calculation by the field agent
- name: char_frpl
description: Number of fireplaces, counted as the number of flues one can see from the outside of the building
- name: char_fbath
description: |
Number of full bathrooms, defined as having a bath or shower.

If this value is missing, the default value is set to 1
- name: char_cnst_qlty
descripion: |
Summary forthcoming.

In general, this field is not used consistently and is therefore not useful for analytical purposes
- name: char_bldg_sf
description: Square footage of the building, as measured from the exterior
- name: char_beds
description: Number of bedrooms in the building
- name: char_apts
description: |
Number of apartments for class 211 and 212 properties.

CAUTION: Note the numerically encoded values DO NOT correspond to the number of apartments i.e. code 1 means 2 apartments, code 6 means 0 apartments
- name: char_age
description: |
Deprecated.

Use year built instead
- name: cdu
description: Dozens of codes attached to a face sheet that denote a number of seemingly unrelated characteristics associated with a PIN, ranging from condition to types of subsidies
- name: card
description: |
Represents the building ID if multiple buildings or spaces exist on the same PIN

Equivalent to legacy MLT_CD (multicode)
description: '{{ doc("shared_column_pin_is_multicard") }}'
- name: pin_num_cards
description: '{{ doc("shared_column_pin_num_cards") }}'
- name: pin_num_landlines
description: '{{ doc("shared_column_pin_num_landlines") }}'
- name: pin10
description: '{{ doc("shared_column_pin10") }}'
- name: seq
description: '{{ doc("shared_column_seq") }}'
- name: tieback_key_pin
description: '{{ doc("shared_column_tieback_key_pin") }}'
- name: tieback_proration_rate
description: '{{ doc("shared_column_tieback_proration_rate") }}'
- name: township_code
description: '{{ doc("shared_column_township_code") }}'
- name: updated_at
description: '{{ doc("shared_column_updated_at") }}'
- name: updated_by
description: '{{ doc("shared_column_updated_by") }}'
- name: year
description: '{{ doc("shared_column_year") }}'

tests:
- unique_combination_of_columns:
Expand Down
46 changes: 46 additions & 0 deletions dbt/models/default/schema/default.vw_pin_address.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,52 @@ models:
- name: default.vw_pin_address
description: '{{ doc("vw_pin_address") }}'

columns:
- name: mail_address_city_name
description: '{{ doc("shared_column_mail_address_city_name") }}'
- name: mail_address_full
description: '{{ doc("shared_column_mail_address_full") }}'
- name: mail_address_name
description: '{{ doc("shared_column_mail_address_name") }}'
- name: mail_address_state
description: '{{ doc("shared_column_mail_address_state") }}'
- name: mail_address_zipcode_1
description: '{{ doc("shared_column_mail_address_zipcode_1") }}'
- name: mail_address_zipcode_2
description: '{{ doc("shared_column_mail_address_zipcode_2") }}'
- name: prop_address_city_name
description: '{{ doc("shared_column_prop_address_city_name") }}'
- name: pin
description: '{{ doc("shared_column_pin") }}'
- name: pin10
description: '{{ doc("shared_column_pin10") }}'
- name: prop_address_full
description: '{{ doc("shared_column_prop_address_full") }}'
- name: prop_address_prefix
description: '{{ doc("shared_column_prop_address_prefix") }}'
- name: prop_address_state
description: '{{ doc("shared_column_prop_address_state") }}'
- name: prop_address_street_dir
description: '{{ doc("shared_column_prop_address_street_dir") }}'
- name: prop_address_street_name
description: '{{ doc("shared_column_prop_address_street_name") }}'
- name: prop_address_street_number
description: '{{ doc("shared_column_prop_address_street_number") }}'
- name: prop_address_suffix_1
description: '{{ doc("shared_column_prop_address_suffix_1") }}'
- name: prop_address_suffix_2
description: '{{ doc("shared_column_prop_address_suffix_2") }}'
- name: prop_address_unit_prefix
description: '{{ doc("shared_column_prop_address_unit_prefix") }}'
- name: prop_address_unit_number
description: '{{ doc("shared_column_prop_address_unit_number") }}'
- name: prop_address_zipcode_1
description: '{{ doc("shared_column_prop_address_zipcode_1") }}'
- name: prop_address_zipcode_2
description: '{{ doc("shared_column_prop_address_zipcode_2") }}'
- name: year
description: '{{ doc("shared_column_year") }}'

tests:
# No superfluous whitespace
- no_extra_whitespace:
Expand Down
40 changes: 40 additions & 0 deletions dbt/models/default/schema/default.vw_pin_appeal.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,46 @@ models:
- name: default.vw_pin_appeal
description: '{{ doc("vw_pin_appeal") }}'

columns:
- name: agent_code
description: '{{ doc("shared_column_agent_code") }}'
- name: agent_name
description: '{{ doc("shared_column_agent_name") }}'
- name: appeal_type
description: '{{ doc("shared_column_appeal_type") }}'
- name: case_no
description: '{{ doc("shared_column_case_no") }}'
- name: certified_bldg
description: '{{ doc("shared_column_certified_bldg") }}'
- name: certified_land
description: '{{ doc("shared_column_certified_land") }}'
- name: certified_tot
description: '{{ doc("shared_column_certified_tot") }}'
- name: change
description: '{{ doc("shared_column_appeal_change") }}'
- name: class
description: '{{ doc("shared_column_class") }}'
- name: mailed_bldg
description: '{{ doc("shared_column_mailed_bldg") }}'
- name: mailed_land
description: '{{ doc("shared_column_mailed_land") }}'
- name: mailed_tot
description: '{{ doc("shared_column_mailed_tot") }}'
- name: pin
description: '{{ doc("shared_column_pin") }}'
- name: reason_code1
description: '{{ doc("shared_column_reason_code") }}'
- name: reason_code2
description: '{{ doc("shared_column_reason_code") }}'
- name: reason_code3
description: '{{ doc("shared_column_reason_code") }}'
- name: status
description: '{{ doc("shared_column_appeal_status") }}'
- name: township_code
description: '{{ doc("shared_column_township_code") }}'
- name: year
description: '{{ doc("shared_column_year") }}'

tests:
# If there is a `change`, the values should reflect this
- expression_is_true:
Expand Down
Loading