Integrate mydec sales into vw_pin_sale

[wagnerlmichael]

This PR uses brings in mydec sales into default.vw_pin_sale only if the doc_no does not already exist in iasworld. We coalesce columns with priority given to iasworld, and add a new column called source which lets us know which table the sale was pulled from; iasworld or mydec.

If a mydec doc_no does not exist in iasworld, but does exist in mydec, we will bring that sale into this view. If the doc_no exists in both tables we will use the iasworld doc_no.

We (discussed with @jeancochrane ) also changed the error threshold for the this test since we are letting in a bunch of sales that were previously no longer in iasworld. It pushes the test failures from 3997 to 4600.

We will need to make a large amount of PRs in other repos to accommodate this. Most likely a bunch of where source ='iasworld' filters.

View at FROM z_ci_583_create_a_sales_view_that_combines_iasworld_mydec_and_ccrd_sales_default.vw_pin_sale

Quick integrity checks

Total counts

SELECT source, count(*)
FROM "z_ci_583_create_a_sales_view_that_combines_iasworld_mydec_and_ccrd_sales_default"."vw_pin_sale"
GROUP BY source;

1 | mydec | 205367
2 | iasworld | 2496073

. . .

select count(*) from default.vw_pin_sale

1 | 2496073

Filter deed type

-- deed_type_filter
SELECT count(*)
FROM "z_ci_583_create_a_sales_view_that_combines_iasworld_mydec_and_ccrd_sales_default"."vw_pin_sale"
where
source = 'iasworld'
and sale_filter_deed_type

1 | 116921

. . . 

select count(*) from default.vw_pin_sale
where sale_filter_deed_type

1 | 116921

Filter same sale within 365 days

This figure is different because after a talk with Billy it was decided that we should be
using doc_no's from mydec in the calculation of this filter even for ias doc_no observations.

select count(*), sale_filter_same_sale_within_365
from "z_ci_583_create_a_sales_view_that_combines_iasworld_mydec_and_ccrd_sales_default"."vw_pin_sale"
where source = 'iasworld'
group by sale_filter_same_sale_within_365

1	2480605	false
2	15468	true


. . .

select count(*), sale_filter_same_sale_within_365
from "z_ci_sale_filter_fix_default"."vw_pin_sale"
group by sale_filter_same_sale_within_365


1 | 13188 | true
2 | 2482885 | false

is_mydec_date check

select count(*), is_mydec_date
from "z_ci_583_create_a_sales_view_that_combines_iasworld_mydec_and_ccrd_sales_default"."vw_pin_sale"
where source = 'iasworld'
group by is_mydec_date

1	913170	true
2	1582903	false

. . .

select count(*), is_mydec_date
from "default"."vw_pin_sale"
group by is_mydec_date

1	913170	true
2	1582903	false

Sales by year counts

select count(*), year
from "z_ci_583_create_a_sales_view_that_combines_iasworld_mydec_and_ccrd_sales_default"."vw_pin_sale"
where source = 'iasworld'
group by year

5	75533	2023
21	101313	2022
9	106914	2021
28	113805	2020
16	141078	2019
25	127821	2018
27	127386	2017
31	127655	2016

. . .

select count(*), year
from default.vw_pin_sale
group by year

#	_col0	year
11	75533	2023
28	101313	2022
8	106914	2021
32	113805	2020
21	141078	2019
1	127821	2018
31	127386	2017
15	127655	2016

[wrridgeway]

We should probably just use the year column we create in the inner query, huh

[wagnerlmichael]

As in this change makes sense?

[wrridgeway]

As in we define year a few lines above this, let's just use that

[wrridgeway]

Let's take care of this before we merge.

[wrridgeway]

Just gonna nibble around the edges on this for the first bite. I'll look at the data and inspect the filters once this round of changes is made.

[wagnerlmichael]

Just gonna nibble around the edges on this for the first bite. I'll look at the data and inspect the filters once this round of changes is made.

Impressively mouthy sentence

[wrridgeway]

Let's take care of this before we merge.

[wrridgeway]

Just a few tiny suggestions, we're really close.