Improve documentation for QC products (#568)

* Update README.md to clarify role of dbt generics

* Make some updates to dbt/README.md to sketch out QC doc structure

* Add link to QC tests and reports section to root README.md

* Continue fleshing out dbt test docs in dbt/README.md

* Clean up data testing section of dbt/README.md

* Update dbt/README.md to add docs for running QC reports

* Finish fleshing out docs on QC reports in dbt/README.md

* Move transform_dbt_test_results.py script to dbt/scripts/ subdirectory

* Fix small issues in dbt/README.md

* Clean up some trailing whitespace in dbt/README.md

* Run pip install step in dbt/ subdir in test_dbt_models workflow

* Clarify bug described in unit tests section of dbt/README.md

* Clarify --rebuild docs in dbt/README.md

* Small clarification in dbt/README.md

* Apply a few small nitpicks from Dan's code review to dbt/README.md

Co-authored-by: Dan Snow <31494343+dfsnow@users.noreply.github.com>

* Standardize on "Valuations staff" in dbt/README.md

* Use relative paths in dbt/README.md

* Refactor meta.export_template option to not expect file extensions for export_models script

* Refactor docs to use clearer terminology for iasWorld data tests

* Add comma after "Currently" in dbt/README.md

Co-authored-by: Dan Snow <31494343+dfsnow@users.noreply.github.com>

---------

Co-authored-by: Dan Snow <31494343+dfsnow@users.noreply.github.com>

.github/workflows/test_dbt_models.yaml

@@ -23,7 +23,8 @@ jobs:
           role-to-assume: ${{ secrets.AWS_IAM_ROLE_TO_ASSUME_ARN }}
 
       - name: Install Python requirements
-        run: pip install -r ./.github/scripts/requirements.transform_dbt_test_results.txt
+        run: pip install -r scripts/requirements.transform_dbt_test_results.txt
+        working-directory: ${{ env.PROJECT_DIR }}
         shell: bash
 
       - name: Run tests
@@ -52,7 +53,7 @@ jobs:
 
       - name: Extract test results
         run: |
-          python3 ../.github/scripts/transform_dbt_test_results.py \
+          python3 scripts/transform_dbt_test_results.py \
             --output-dir ./qc_test_results/
         working-directory: ${{ env.PROJECT_DIR }}
         shell: bash

README.md

@@ -10,8 +10,10 @@ modeling, reporting, and data integrity work.
   Documentation for all CCAO data lakehouse tables and views
 - [:nut_and_bolt: dbt README](/dbt/README.md) - How to develop CCAO data
   infrastructure using dbt
-- [:test_tube: dbt Test Documentation](/dbt/tests/generic/README.md) -
-  Definitions for CCAO generic dbt data tests
+- [:test_tube: dbt Tests and QC Reports](dbt/README.md#-how-to-add-and-run-tests-and-qc-reports) -
+  How to add and run data tests, unit tests, and QC reports using dbt
+- [:pencil: dbt Generic Test Documentation](/dbt/tests/generic/README.md) -
+  Definitions for CCAO generic dbt tests, which are functions that we use to define our QC tests
 
 ## Repository Structure
 

dbt/models/qc/schema.yml

@@ -814,7 +814,7 @@ models:
         - qc_report_town_close
     meta:
       export_name: 0 land value
-      export_template: assessed_and_market_values.xlsx
+      export_template: assessed_and_market_values
 
   - name: qc.vw_report_town_close_0_value
     description: '{{ doc("view_vw_report_town_close_0_value") }}'
@@ -823,7 +823,7 @@ models:
         - qc_report_town_close
     meta:
       export_name: 0 value
-      export_template: assessed_and_market_values.xlsx
+      export_template: assessed_and_market_values
 
   - name: qc.vw_report_town_close_289s
     description: '{{ doc("view_vw_report_town_close_289s") }}'
@@ -869,7 +869,7 @@ models:
         - qc_report_town_close
     meta:
       export_name: Improved class, no bldg value
-      export_template: assessed_and_market_values.xlsx
+      export_template: assessed_and_market_values
 
   - name: qc.vw_report_town_close_neg_asmt_value
     description: '{{ doc("view_vw_report_town_close_neg_asmt_value") }}'
@@ -983,7 +983,7 @@ models:
         - qc_report_town_close
     meta:
       export_name: Vacant class, bldg value
-      export_template: assessed_and_market_values.xlsx
+      export_template: assessed_and_market_values
 
   - name: qc.vw_sale_mydec_null_values
     description: '{{ doc("view_vw_sale_mydec_null_values") }}'

dbt/scripts/export_models.py

@@ -27,9 +27,10 @@
     * config.meta.export_name (optional): The base name of the output file that will be generated. File extensions are not necessary since all
       models are exported as .xlsx files. If unset, defaults to the name of the model.
 
-    * config.meta.export_template (optional): The filename of an Excel template to use when exporting a model. Templates should be stored in the
-      export/templates/ directory and should include header rows. If unset, will search for a template with the same name as the model; if no
-      template is found, defaults to a simple layout with filterable columns and striped rows.
+    * config.meta.export_template (optional): The base name of an Excel template to use when exporting a model. File extensions are not necessary
+      since all templates are assumed to be .xlsx files. Templates should be stored in the export/templates/ directory and should include header
+      rows. If unset, will search for a template with the same name as the model; if no template is found, defaults to a simple layout with
+      filterable columns and striped rows.
 """  # noqa: E501
 CLI_EXAMPLE = """Example usage to output the qc_report_town_close report for Hyde Park township:
 
@@ -146,13 +147,10 @@ def main():
         model_name = model["name"]
         relation_name = model["relation_name"]
         export_name = model["config"]["meta"].get("export_name") or model_name
-        template = (
-            model["config"]["meta"].get("export_template")
-            or f"{model_name}.xlsx"
-        )
+        template = model["config"]["meta"].get("export_template") or model_name
 
         # Define inputs and outputs for export based on model metadata
-        template_path = os.path.join("export", "templates", template)
+        template_path = os.path.join("export", "templates", f"{template}.xlsx")
         template_exists = os.path.isfile(template_path)
         output_path = os.path.join("export", "output", f"{export_name}.xlsx")