[3/3] Migrate Will's town close QC reports to dbt models (#557)

* Migrate 0 land value QC report

* Sketch out town close QC queries based on asmt prior year values

* Migrate "Res parcels not set to cost approach" QC report

* Migrate "OVRRCNLDs discrepancy checks" QC report

* Move qc_vw_incorrect_val_method_revcode_eq_1 data test to new cost approach model

* Migrate "Bldg parcel class mismatch" QC report

* Add docs to QC report town close models, with TODO Inquire query numbers

* Migrate "289s" QC report

* Migrate "500k increase, 1m decrease" QC report

* Migrate "Res multicodes" QC report

* Migrate "Card code 5s possibly not at 100%" QC report for comdat

* Migrate all "Card code 5s possibly not at 100%" QC reports

* Factor out vw_iasworld_sales_latest_sale_since_2021 QC view

* Migrate "Res edit" QC report with some TODOs

* Add export templates and complete migration of QC reports

* Fix "Possibly not at 100%" QC reports

* Fix formatting of "Possibly not at 100%" percent change columns

* Fix "500k increase, 1m decrease" QC report formatting

* Fix "Bldg parcel class mismatch" QC report formatting

* Fix "OVRRCNLD discrepancy checks" QC report formatting

* Fix "Res multicodes" QC report formatting

* Fix "Res parcels not set to Cost Approach" formatting

* Fix "Res edit" QC report formatting

* Document QC report models ahead of review

* Add docs for intermediate QC models

* Remove unused vw_incorrect_val_method docs

* More reader-friendly description for res_parcels_not_set_to_cost_approach view

* Refactor qc.vw_report_town_close_500k_increase_1m_decrease to use shared asmt view

* Rollback QC report models that will be added in third PR

* Rollback QC report models that will be added in second PR

* [2/3] Migrate Will's town close QC reports to dbt models
This reverts commit 5b86b18.

* [3/3] Migrate Will's town close QC reports to dbt models

This reverts commit 5e4bc7b.

* Oops, forgot to filter for current records in Res Multicodes QC report

* Filter parcels from "Bldg parcel class mismatch" if at least one card has a matching class

* Format parids as numbers in QC reports to avoid scientific notation

* Filter out leasehold properties from 0 land value QC report

* Simplify bldg_parcel_class_mismatch report and make filtering match Will's

* Remove land.sf IS NOT NULL condition from "Res Edit" QC file

* Remove unnecessary flake8 config now that we're using ruff

* Only convert QC reports to data tables if the result set is non-empty

* Reorder qc schema docs alphanumerically

* Append reason code descriptions to aprval.reascd in QC reports

* Fix edge case bug in export_models.py script that can raise an error when result set has only 1 row

* Document card code 5s query in 500k_increase_1m_decrease QC report

* Format PINs as explicitly 14-digit numbers in export_models.py script

* Fix typos in comments in card code 5 reports

* Consolidate and flesh out docs for prior_year_card_code_5s reports

* Remove stray docs block artifact from qc/docs.md

* Reorder qc/schema.yml alphanumerically

* Re-alphabetize QC docs and schema for latest QC models

* Make some small fixes to QC reports based on Dan's feedback

* Pull num_dwellings from dweldat instead of subquery in qc.vw_report_town_close_res_edit

* Remove stray whitespace in qc/docs.md

dbt/models/qc/docs.md

@@ -62,6 +62,14 @@ Test if selected classes have sales prices greater than $20,000,000 in
 
 {% enddocs %}
 
+# vw_iasworld_sales_latest_sale
+
+{% docs view_vw_iasworld_sales_latest_sale %}
+View that pulls the latest sale for each parcel from `iasworld.sales`.
+
+This view is used as the basis for a number of town close QC reports.
+{% enddocs %}
+
 # vw_iasworld_sales_null_values
 
 {% docs view_vw_iasworld_sales_null_values %}
@@ -90,16 +98,6 @@ Identify years with large numbers of unmatched sales between `sale.mydec` and
 
 {% enddocs %}
 
-# vw_incorrect_val_method
-
-{% docs view_vw_incorrect_val_method %}
-Test that no residential parcels are using a cost approach.
-
-Pulled from the following Inquire queries:
-
-- `FP Checklist - Res parcels not set to Cost Approach` (#1578) - Will C.
-{% enddocs %}
-
 # vw_neg_asmt_value
 
 {% docs view_vw_neg_asmt_value %}
@@ -149,10 +147,23 @@ Pulled from the following Inquire queries:
 - `FP Checklist - Non-EX, RR parcels with 0 value` (#1570) - Will C.
 {% enddocs %}
 
+# vw_report_town_close_289s
+
+{% docs view_vw_report_town_close_289s %}
+Check for rows in the `OBY` table where the class is 289.
+
+This view is exported as part of the QC report to check values prior to town
+closings.
+
+Pulled from the following Inquire queries:
+
+- `FP Checklist - 289s` (#1963) - Will C.
+{% enddocs %}
+
 # vw_report_town_close_500k_increase_1m_decrease
 
 {% docs view_vw_report_town_close_500k_increase_1m_decrease %}
-Check for rows in the  `ASMT` table where the total market value has increased
+Check for rows in the `ASMT` table where the total market value has increased
 by more than $500k or decreased by more than $1m in the past year.
 
 This view is exported as part of the QC report to check values prior to town
@@ -164,6 +175,20 @@ Pulled from the following Inquire queries:
 - `FP Checklist - Card Code Info` (#2160) - Will C.
 {% enddocs %}
 
+# vw_report_town_close_bldg_parcel_class_mismatch
+
+{% docs view_vw_report_town_close_bldg_parcel_class_mismatch %}
+Check for rows in the `DWELDAT`, `COMDAT`, and `OBY` tables
+where the class does not match the parcel's class in the `PARDAT` table.
+
+This view is exported as part of the QC report to check values prior to town
+closings.
+
+Pulled from the following Inquire queries:
+
+- `FP Checklist - Bldg and parcel class mismatch` (#2161) - Will C.
+{% enddocs %}
+
 # vw_report_town_close_class_does_not_equal_luc
 
 {% docs view_vw_report_town_close_class_does_not_equal_luc %}
@@ -208,6 +233,21 @@ In contrast to `qc.vw_neg_asmt_value`, this view directly performs filtering
 for negative values.
 {% enddocs %}
 
+# vw_report_town_close_ovrrcnlds_to_review
+
+{% docs view_vw_report_town_close_ovrrcnlds_to_review %}
+Check for rows in the `DWELDAT`, `COMDAT`, and `OBY` tables
+where the calculated net market value does not match the override net market
+value.
+
+This view is exported as part of the QC report to check values prior to town
+closings.
+
+Pulled from the following Inquire queries:
+
+- `FP Checklist - OVRRCNLDs to review` (#2474) - Will C.
+{% enddocs %}
+
 # vw_report_town_close_prior_year_card_code_5s
 
 {% docs view_vw_report_town_close_prior_year_card_code_5s %}
@@ -235,6 +275,56 @@ Pulled from the following Inquire queries:
 - `FP Checklist - Prior Yr Card Code 5s OBY` (#1586) - Will C.
 {% enddocs %}
 
+# vw_report_town_close_res_edit
+
+{% docs view_vw_report_town_close_res_edit %}
+Check detailed info for residential dwellings.
+
+This view is exported as part of the QC report to check values prior to town
+closings.
+
+Pulled from the following Inquire queries:
+
+- `Res Edit - ASMT Values` (#1734) - Will C.
+- `Res Edit - DWELDAT` (#1737) - Will C.
+- `Res Edit - Parcel Characteristics` (#1736) - Will C.
+- `Res Edit - Sales` (#961) - Will C.
+{% enddocs %}
+
+# vw_report_town_close_res_multicodes
+
+{% docs view_vw_report_town_close_res_multicodes %}
+Check market and assessed values for parcels with multiple cards.
+
+This view is exported as part of the QC report to check values prior to town
+closings.
+
+Pulled from the following Inquire queries:
+
+- `FP Checklist - Res multicode PIN list` (#1591) - Will C.
+- `FP Checklist - Res multicode report with sales` (#1659) - Will C.
+{% enddocs %}
+
+# vw_report_town_close_res_parcels_not_set_to_cost_approach
+
+{% docs view_vw_report_town_close_res_parcels_not_set_to_cost_approach %}
+Check for residential class parcels that do not have "1: Cost Approach" set
+for their Valuation Method.
+
+This might seem counter-intuitive since residential parcels are not generally
+valued using a cost approach, but it stems from the fact that "1: Cost Approach"
+is actually used as a flag to indicate that we valued the parcel the normal way
+using the data in the `LAND`/`DWELDAT`/`COMDAT`/`OBY` tables rather than using
+overrides in the "Land/Building/Total Value Estimate" fields.
+
+This view is exported as part of the QC report to check values prior to town
+closings.
+
+Pulled from the following Inquire queries:
+
+- `FP Checklist - Res parcels not set to Cost Approach` (#2114) - Will C.
+{% enddocs %}
+
 # vw_report_town_close_vacant_class_with_bldg_value
 
 {% docs view_vw_report_town_close_vacant_class_with_bldg_value %}

dbt/models/qc/qc.vw_iasworld_sales_latest_sale.sql

@@ -0,0 +1,21 @@
+SELECT
+    parid,
+    instruno,
+    price,
+    saledt,
+    DATE_FORMAT(
+        DATE_PARSE(saledt, '%Y-%m-%d %H:%i:%S.%f'), '%c/%e/%Y'
+    ) AS saledt_fmt
+FROM (
+    SELECT
+        parid,
+        instruno,
+        saledt,
+        price,
+        ROW_NUMBER()
+            OVER (PARTITION BY parid ORDER BY saledt DESC)
+            AS row_num
+    FROM {{ source('iasworld', 'sales') }}
+    WHERE price > 1
+) AS ranked_sales
+WHERE row_num = 1

dbt/models/qc/qc.vw_report_town_close_289s.sql

@@ -0,0 +1,14 @@
+SELECT
+    oby.parid,
+    oby.taxyr,
+    legdat.user1 AS township_code,
+    oby.class
+FROM {{ source('iasworld', 'oby') }} AS oby
+LEFT JOIN {{ source('iasworld', 'legdat') }} AS legdat
+    ON oby.parid = legdat.parid
+    AND oby.taxyr = legdat.taxyr
+    AND legdat.cur = 'Y'
+    AND legdat.deactivat IS NULL
+WHERE oby.cur = 'Y'
+    AND oby.deactivat IS NULL
+    AND oby.class = '289'

dbt/models/qc/qc.vw_report_town_close_bldg_parcel_class_mismatch.sql

@@ -0,0 +1,65 @@
+{% set comparison_tables = ['comdat', 'dweldat', 'oby'] %}
+
+SELECT
+    legdat.parid,
+    legdat.taxyr,
+    legdat.user1 AS township_code,
+    pardat.class AS parcel_class,
+{% for tablename in comparison_tables %}
+    ARRAY_JOIN({{ tablename }}.classes, ', ')
+        AS {{ tablename }}_classes{% if not loop.last %}, {% endif %}
+{% endfor %}
+FROM {{ source('iasworld', 'pardat') }} AS pardat
+LEFT JOIN {{ source('iasworld', 'legdat') }} AS legdat
+    ON pardat.parid = legdat.parid
+    AND pardat.taxyr = legdat.taxyr
+    AND legdat.cur = 'Y'
+    AND legdat.deactivat IS NULL
+{% for tablename in comparison_tables %}
+    LEFT JOIN (
+        SELECT
+            {{ tablename }}.parid,
+            {{ tablename }}.taxyr,
+            ARRAY_AGG(DISTINCT {{ tablename }}.class) AS classes
+        FROM {{ source('iasworld', tablename) }} AS {{ tablename }}
+        WHERE {{ tablename }}.cur = 'Y'
+            AND {{ tablename }}.deactivat IS NULL
+        GROUP BY {{ tablename }}.parid, {{ tablename }}.taxyr
+    ) AS {{ tablename }}
+        ON pardat.parid = {{ tablename }}.parid
+        AND pardat.taxyr = {{ tablename }}.taxyr
+{% endfor %}
+WHERE pardat.cur = 'Y'
+    AND pardat.deactivat IS NULL
+    AND pardat.class NOT IN (
+        '100', '200', '239', '240', '241', '300', '400', '500', '535', '550'
+    )
+    AND SUBSTR(pardat.class, 1, 3) NOT IN (
+        '637', '700', '742', '800', '900'
+    )
+    -- Parcels fail the check if one of the comparison tables has classes
+    -- that A) exist (aka are not null) and B) have no match to the
+    -- parcel class
+    AND (
+        {% for tablename in comparison_tables %}
+            (
+                -- Only run this check if the column is not null, since
+                -- otherwise CONTAINS() will return null as well
+                {{ tablename }}.classes IS NULL
+                OR NOT CONTAINS({{ tablename }}.classes, pardat.class)
+            )
+            {% if not loop.last %}AND{% endif %}
+        {% endfor %}
+    )
+    -- Check for the edge case where a parcel is exempt and none of the
+    -- comparison tables exist, which should not be returned but can pass
+    -- the conditional above due to all table joins returning nulls
+    AND NOT (
+        pardat.class IN ('EX', 'RR')
+        AND
+        {% for tablename in comparison_tables %}
+            {{ tablename }}.classes IS NULL
+            {% if not loop.last %}AND{% endif %}
+        {% endfor %}
+
+    )