Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add IDOT data to data-lake #617

Open
wants to merge 76 commits into
base: master
Choose a base branch
from
Open

Add IDOT data to data-lake #617

wants to merge 76 commits into from
+435 −3

Commits on Oct 3, 2024

  1. Push for raw upload

    @Damonamajor
    Damonamajor committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    14d4edf View commit details
    Browse the repository at this point in the history

  2. Remove unnecessary code

    @Damonamajor
    Damonamajor committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    2ecffe1 View commit details
    Browse the repository at this point in the history

  3. Testing file

    @Damonamajor
    Damonamajor committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    e0c05ae View commit details
    Browse the repository at this point in the history

  4. Modify HWY so it looks back to 2012

    @Damonamajor
    Damonamajor committed Oct 3, 2024
    Configuration menu
    Copy the full SHA
    d145bae View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2024

  1. Minor simplifications

    @wrridgeway
    wrridgeway committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    b749f3a View commit details
    Browse the repository at this point in the history

  2. Add cleaning script

    @Damonamajor
    Damonamajor committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    2bfc813 View commit details
    Browse the repository at this point in the history

  3. Quick edit

    @Damonamajor
    Damonamajor committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    da0791d View commit details
    Browse the repository at this point in the history

  4. Push correct version

    @Damonamajor
    Damonamajor committed Oct 4, 2024
    Configuration menu
    Copy the full SHA
    b15eafa View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2024

  1. Remove old file

    @Damonamajor
    Damonamajor committed Oct 7, 2024
    Configuration menu
    Copy the full SHA
    bc1070c View commit details
    Browse the repository at this point in the history

  2. Working script

    @Damonamajor
    Damonamajor committed Oct 7, 2024
    Configuration menu
    Copy the full SHA
    917c48d View commit details
    Browse the repository at this point in the history

  3. Working loop

    @Damonamajor
    Damonamajor committed Oct 7, 2024
    Configuration menu
    Copy the full SHA
    936b972 View commit details
    Browse the repository at this point in the history

  4. Billy edits

    @Damonamajor
    Damonamajor committed Oct 7, 2024
    Configuration menu
    Copy the full SHA
    1bbcce4 View commit details
    Browse the repository at this point in the history

  5. text edits

    @Damonamajor
    Damonamajor committed Oct 7, 2024
    Configuration menu
    Copy the full SHA
    1e44751 View commit details
    Browse the repository at this point in the history

  6. Use correct buckets

    @Damonamajor
    Damonamajor committed Oct 7, 2024
    Configuration menu
    Copy the full SHA
    21aa438 View commit details
    Browse the repository at this point in the history

  7. Text edits

    @Damonamajor
    Damonamajor committed Oct 7, 2024
    Configuration menu
    Copy the full SHA
    83a0e4e View commit details
    Browse the repository at this point in the history

Commits on Oct 8, 2024

  1. lintr

    @Damonamajor
    Damonamajor committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    3206571 View commit details
    Browse the repository at this point in the history

  2. lintr

    @Damonamajor
    Damonamajor committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    59c180d View commit details
    Browse the repository at this point in the history

  3. Change geoparquet function

    @wrridgeway
    wrridgeway committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    de905ef View commit details
    Browse the repository at this point in the history

  4. Add filter for Cook County

    @Damonamajor
    Damonamajor committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    418ef5e View commit details
    Browse the repository at this point in the history

  5. Comment

    @Damonamajor
    Damonamajor committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    45b7929 View commit details
    Browse the repository at this point in the history

  6. Add if-else statement for County

    @Damonamajor
    Damonamajor committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    c6961d0 View commit details
    Browse the repository at this point in the history

  7. rename data to shapefile_data

    @Damonamajor
    Damonamajor committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    c0fa4a8 View commit details
    Browse the repository at this point in the history

  8. Include period instead of named file

    @Damonamajor
    Damonamajor committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    266ba86 View commit details
    Browse the repository at this point in the history

  9. period

    @Damonamajor
    Damonamajor committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    cc5a18f View commit details
    Browse the repository at this point in the history

  10. Fix s3 pathing

    @wrridgeway
    wrridgeway committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    2cb1e0c View commit details
    Browse the repository at this point in the history

  11. Start dbt schema

    @Damonamajor
    Damonamajor committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    683a551 View commit details
    Browse the repository at this point in the history

  12. Merge branch 'Investigate-IDOT-data' of github.com:ccao-data/data-arc… 

    …hitecture into Investigate-IDOT-data
    @Damonamajor
    Damonamajor committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    8eda339 View commit details
    Browse the repository at this point in the history

  13. Add year

    @Damonamajor
    Damonamajor committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    76bfe5b View commit details
    Browse the repository at this point in the history

  14. Fix path

    @Damonamajor
    Damonamajor committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    c1d57e8 View commit details
    Browse the repository at this point in the history

  15. Add renaming

    @Damonamajor
    Damonamajor committed Oct 8, 2024
    Configuration menu
    Copy the full SHA
    89c69e7 View commit details
    Browse the repository at this point in the history

Commits on Oct 9, 2024

  1. Remove unecessary code

    @wrridgeway
    wrridgeway committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    cb883e4 View commit details
    Browse the repository at this point in the history

  2. Delete file

    @Damonamajor
    Damonamajor committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    2014cbe View commit details
    Browse the repository at this point in the history

  3. Remove docs

    @Damonamajor
    Damonamajor committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    102b721 View commit details
    Browse the repository at this point in the history

  4. Use FC_NAME and FCNAME

    @Damonamajor
    Damonamajor committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    ad6ed46 View commit details
    Browse the repository at this point in the history

  5. Fix brackets

    @Damonamajor
    Damonamajor committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    47b0469 View commit details
    Browse the repository at this point in the history

  6. Get back to running

    @wrridgeway
    wrridgeway committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    20b9c4c View commit details
    Browse the repository at this point in the history

  7. Make year a character

    @Damonamajor
    Damonamajor committed Oct 9, 2024
    Configuration menu
    Copy the full SHA
    369e228 View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2024

  1. Add NA handeling

    @Damonamajor
    Damonamajor committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    f07d62f View commit details
    Browse the repository at this point in the history

  2. Reorder columns

    @Damonamajor
    Damonamajor committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    6ec7267 View commit details
    Browse the repository at this point in the history

  3. Add docs

    @Damonamajor
    Damonamajor committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    b402b77 View commit details
    Browse the repository at this point in the history

  4. better commenting

    @Damonamajor
    Damonamajor committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    1dfd84b View commit details
    Browse the repository at this point in the history

  5. Remove duplicated code

    @Damonamajor
    Damonamajor committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    22741ab View commit details
    Browse the repository at this point in the history

  6. Rename SURF_YR

    @Damonamajor
    Damonamajor committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    c0450ff View commit details
    Browse the repository at this point in the history

  7. Better renaming

    @Damonamajor
    Damonamajor committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    cfafd12 View commit details
    Browse the repository at this point in the history

  8. rename traffic, fix surface_year

    @Damonamajor
    Damonamajor committed Oct 10, 2024
    Configuration menu
    Copy the full SHA
    8af8f07 View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2024

  1. Add mean values

    @Damonamajor
    Damonamajor committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    75b1bbb View commit details
    Browse the repository at this point in the history

  2. Update commenting

    @Damonamajor
    Damonamajor committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    a86203c View commit details
    Browse the repository at this point in the history

  3. Remove extra line

    @Damonamajor
    Damonamajor committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    184b0e2 View commit details
    Browse the repository at this point in the history

  4. Run function

    @Damonamajor
    Damonamajor committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    b074eee View commit details
    Browse the repository at this point in the history

  5. Revert num_lanes

    @Damonamajor
    Damonamajor committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    4eb17fe View commit details
    Browse the repository at this point in the history

  6. Get loop working

    @Damonamajor
    Damonamajor committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    6840eb3 View commit details
    Browse the repository at this point in the history

  7. Wrapup

    @Damonamajor
    Damonamajor committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    fb86cc6 View commit details
    Browse the repository at this point in the history

  8. Wrapup

    @Damonamajor
    Damonamajor committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    540efdc View commit details
    Browse the repository at this point in the history

  9. Linting

    @wrridgeway
    wrridgeway committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    81cfdf6 View commit details
    Browse the repository at this point in the history

  10. linting

    @Damonamajor
    Damonamajor committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    7f1e4c1 View commit details
    Browse the repository at this point in the history

  11. linting

    @Damonamajor
    Damonamajor committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    fa29c88 View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2024

  1. Rename to shapefile_data

    @Damonamajor
    Damonamajor committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    5a285fd View commit details
    Browse the repository at this point in the history

  2. Add commented text

    @Damonamajor
    Damonamajor committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    6734b5d View commit details
    Browse the repository at this point in the history

  3. remove slash

    @Damonamajor
    Damonamajor committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    bc37c55 View commit details
    Browse the repository at this point in the history

  4. remove hash

    @Damonamajor
    Damonamajor committed Oct 21, 2024
    Configuration menu
    Copy the full SHA
    91fab65 View commit details
    Browse the repository at this point in the history

Commits on Oct 22, 2024

  1. Updates after viewing output

    @Damonamajor
    Damonamajor committed Oct 22, 2024
    Configuration menu
    Copy the full SHA
    9eb0fe3 View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2024

  1. Remove additional geom column from mutate

    @wrridgeway
    wrridgeway committed Oct 23, 2024
    Configuration menu
    Copy the full SHA
    8237ace View commit details
    Browse the repository at this point in the history

Commits on Nov 1, 2024

  1. lintr

    @Damonamajor
    Damonamajor committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    c87c58c View commit details
    Browse the repository at this point in the history

  2. Merge branch 'Investigate-IDOT-data' of github.com:ccao-data/data-arc… 

    …hitecture into Investigate-IDOT-data
    @Damonamajor
    Damonamajor committed Nov 1, 2024
    Configuration menu
    Copy the full SHA
    d2b07d6 View commit details
    Browse the repository at this point in the history

Commits on Nov 3, 2024

  1. Linting

    @wrridgeway
    wrridgeway committed Nov 3, 2024
    Configuration menu
    Copy the full SHA
    9f067ee View commit details
    Browse the repository at this point in the history

Commits on Nov 4, 2024

  1. Update dbt/models/spatial/docs.md 

    Co-authored-by: William Ridgeway <10358980+wrridgeway@users.noreply.github.com>
    @Damonamajor @wrridgeway
    Damonamajor and wrridgeway authored Nov 4, 2024
    Configuration menu
    Copy the full SHA
    2fb559a View commit details
    Browse the repository at this point in the history

  2. Update etl/scripts-ccao-data-raw-us-east-1/spatial/spatial-environmen… 

    …t-traffic.R

Co-authored-by: William Ridgeway <10358980+wrridgeway@users.noreply.github.com>
    @Damonamajor @wrridgeway
    Damonamajor and wrridgeway authored Nov 4, 2024
    Configuration menu
    Copy the full SHA
    fd1945b View commit details
    Browse the repository at this point in the history

  3. Update etl/scripts-ccao-data-warehouse-us-east-1/spatial/spatial-envi… 

    …ronment-traffic.R

Co-authored-by: William Ridgeway <10358980+wrridgeway@users.noreply.github.com>
    @Damonamajor @wrridgeway
    Damonamajor and wrridgeway authored Nov 4, 2024
    Configuration menu
    Copy the full SHA
    bc045b0 View commit details
    Browse the repository at this point in the history

  4. Billy changes

    @Damonamajor
    Damonamajor committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    9810b7e View commit details
    Browse the repository at this point in the history

  5. Working file with doc updates

    @Damonamajor
    Damonamajor committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    de0526c View commit details
    Browse the repository at this point in the history

  6. Final?

    @Damonamajor
    Damonamajor committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    a1b8691 View commit details
    Browse the repository at this point in the history

  7. Remove line at end

    @Damonamajor
    Damonamajor committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    02bbdcc View commit details
    Browse the repository at this point in the history

  8. Rename environ

    @Damonamajor
    Damonamajor committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    bc72c99 View commit details
    Browse the repository at this point in the history

  9. Add )

    @Damonamajor
    Damonamajor committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    87da63b View commit details
    Browse the repository at this point in the history

  10. lintr

    @Damonamajor
    Damonamajor committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    a5df331 View commit details
    Browse the repository at this point in the history

  11. styler

    @Damonamajor
    Damonamajor committed Nov 4, 2024
    Configuration menu
    Copy the full SHA
    5c60c23 View commit details
    Browse the repository at this point in the history