Implemented ETL for covidtracking.com and rt.live data sources to a c… #37
Conversation
…opy of main homepage google sheet.
There was a problem hiding this comment.
Generally this looks awesome, really great work @wesmwoo ! Way to dive in and get this working quite quickly.
Honestly all of the code looks pretty solid to me -- I left some nits about naming and running the hooks, but nothing major.
My main concern at this point is that the updated sheet (https://docs.google.com/spreadsheets/d/1CKcRpPqYzWo-B_xEhrSyrFDplMxhf9PaGTlpgh91jeQ/edit#gid=237779988) no longer has its formulas working -- do you know why those are broken? If we can get them to work, this could be a big step forward in automating Ryan's daily work!
| @@ -944,6 +944,20 @@ def transform_covidtracking_data(covidtracking_df): | |||
| return covidtracking_df | |||
|
|
|||
|
|
|||
| def transform_covidtracking_data_to_states_historical(covidtracking_df): | |||
There was a problem hiding this comment.
Nice, I'm glad these are methods now.
| @@ -435,7 +435,7 @@ | |||
| ] | |||
|
|
|||
|
|
|||
| def transform_covidtracking_data(covidtracking_df): | |||
| def transform_covidtracking_data_to_cdc(covidtracking_df): | |||
There was a problem hiding this comment.
Nice, I'm glad this is renamed! Nit, maybe rename it as ...to_cdc_criteria, since that's a bit more accurate about what's happening.
| @@ -6,7 +6,8 @@ | |||
|
|
|||
| from covid.constants import PATH_TO_SERVICE_ACCOUNT_KEY | |||
| from covid.extract import extract_cdc_ili_data | |||
| from covid.extract import extract_covidtracking_historical_data | |||
| from covid.extract import extract_covidtracking_historical_data, extract_covidtracking_current_data | |||
There was a problem hiding this comment.
Would you mind install pre-commit and running it according to the instructions i just added? https://github.com/usdigitalresponse/covid-exit-strategy/pull/42/files. Not your fault, this wasn't documented yet! But it'll move these to single-line imports.
…opy of main homepage google sheet.