Add proximity.dist_pin_to_pin to the dbt DAG using intermediate transforms

[jeancochrane]

This PR changes proximity.dist_pin_to_pin from a source to a ref so that it can be tracked in the dbt DAG. In order to build the table in a reasonable amount of time, we define a series of intermediate transformations, documented in dbt/models/proximity/docs.md, that expand the PIN-to-PIN buffer gradually so that we only compute the 10km buffer for PINs that do not have neighbors within 1km.

As evidenced by this successful GitHub workflow run, this new design takes about 14 minutes to build dist_pin_to_pin.

Closes #122.

[jeancochrane]

This macro is nearly identical to the pre-existing aws-athena/ctas/proximity-dist_pin_to_pin.sql query, with three main changes:

1. Parameterization of source_model, num_neigbors, and radius_km
2. Addition of pin10 to the distinct_pins subquery per our missing data discussion on Teams (diff here)
3. Reformatting to match dbt styles using sqlfmt

Everything else should be identical!

[dfsnow]

@jeancochrane Great work! When we visit #130 it may be worth it to see what else we can abstract into macros. See my one blocking issue, but otherwise looks good.

[dfsnow]

issue (blocking): If dist_pin_to_pin_10km.sql only contains the values missing from dist_pin_to_pin_1km.sql, then shouldn't this be the union of dist_pin_to_pin_1km.sql and dist_pin_to_pin_10km.sql?

[jeancochrane]

Wow, excellent catch @dfsnow! Embarrassing mistake on my part 🙈 This is fixed in a0f7423, and I updated the table docs to reflect it 👍🏻

[jeancochrane]

@dfsnow Looks like the new table doesn't have quite the same contents as the existing one:

select count(*)
from "ci_jeancochrane-122-data-catalog-move-proximitydist-pin-to-pin-ctas-query-into-the-dbt-dag_proximity"."dist_pin_to_pin"

Results: 32370578

select count(*)
from "proximity"."dist_pin_to_pin"

Results: 32420636

That's a difference of about 50k rows; I would be fine with this if the new version had more rows, but it seems concerning that there are fewer.

Further queries confirm that the new version of the table is a subset of the old version:

select count(*)
from "proximity"."dist_pin_to_pin" as prod
full outer join "ci_jeancochrane-122-data-catalog-move-proximitydist-pin-to-pin-ctas-query-into-the-dbt-dag_proximity"."dist_pin_to_pin" as ci
on prod.pin10 = ci.pin10 and prod.year = ci.year
where prod.pin10 is null

Results: 0

select count(*)
from "proximity"."dist_pin_to_pin" as prod
full outer join "ci_jeancochrane-122-data-catalog-move-proximitydist-pin-to-pin-ctas-query-into-the-dbt-dag_proximity"."dist_pin_to_pin" as ci
on prod.pin10 = ci.pin10 and prod.year = ci.year
where ci.pin10 is null

Results: 53610

Here's a version of the query grouped by year:

select prod.year, count(*)
from "proximity"."dist_pin_to_pin" as prod
full outer join "ci_jeancochrane-122-data-catalog-move-proximitydist-pin-to-pin-ctas-query-into-the-dbt-dag_proximity"."dist_pin_to_pin" as ci
on prod.pin10 = ci.pin10 and prod.year = ci.year
where ci.pin10 is null
group by prod.year
order by prod.year desc

year	count of missing PINs
2022	2931
2021	2369
2020	2885
2019	2870
2018	2677
2017	2629
2016	2488
2015	2632
2014	2715
2013	2760
2012	2862
2011	2759
2010	2760
2009	2791
2008	2304
2007	2143
2006	2038
2005	1836
2004	1630
2003	1508
2002	1423
2001	1331
2000	1269

What do you think @dfsnow? Is this worth investigating before we merge and overwrite the existing table?

[dfsnow]

What do you think @dfsnow? Is this worth investigating before we merge and overwrite the existing table?

@jeancochrane Do a quick anti-join on pin and let's see if it's easy to diagnose. If it takes more than 30 minutes I'm comfortable just merging it as is, since the 3 neighbors are only used for filling.

[jeancochrane]

@dfsnow Not much closer to diagnosing it, but here's some interesting information for you -- only about 5k PINs are missing data for multiple years:

with missing_pins as (
    select prod.pin10 as pin10, prod.year as year
    from "proximity"."dist_pin_to_pin" as prod
    full outer join "ci_jeancochrane-122-data-catalog-move-proximitydist-pin-to-pin-ctas-query-into-the-dbt-dag_proximity"."dist_pin_to_pin" as ci
    on prod.pin10 = ci.pin10 and prod.year = ci.year
    where ci.pin10 is null
),

missing_pins_grouped_by_year as (
    select pin10, array_agg(year) as missing_years
    from missing_pins
    group by pin10
    having cardinality(array_agg(year)) > 1
)

select count(*)
from missing_pins_grouped_by_year

Results: 4940

And only 153 PINs are missing data for all years:

select count(*)
from missing_pins_grouped_by_year
where cardinality(missing_years) = 23

Results: 153

This at least makes me feel better that we're not just dropping an entire group of PINs! If you feel fine with this, I'll go ahead and merge.

[dfsnow]

@jeancochrane Likely has something to do with PIN changes across years or fake "999" PINs. Let's just go ahead and merge it.

[jeancochrane]

Sounds good, thanks @dfsnow! Mind hitting the Approve button so I can merge?