Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FEAT: Add 'revenue' field to Rail data pipeline #446

Merged
merged 4 commits into from
Oct 2, 2024
Merged

Conversation

rymarczy
Copy link
Collaborator

@rymarczy rymarczy commented Oct 1, 2024

This change adds the GTFS-RT revenue field to the LAMP Rail Performance Data pipeline.

  1. Create db migrations to add revenue column to vehicle_trips table
  2. Read vehicle.trip.revenue column from VehiclePositions parquet files
  3. Load revenue data from parquet files into DB table
  4. Add is_revenue column to LAMP_ALL_RT_fields OPMI export
  5. Add where clause for only revenue trips to Subway Performance Data export

The use of the revenue field differs between the OPMI export and Subway Performance Data export that TransitMatters utilizes. For the OMPI export an additional field is added so that revenue trips can be filtered out at their discretion. The Subway Performance Data export was never meant to include non-revenue event data, so it is filtered out from the export all together.

Asana Task: https://app.asana.com/0/1205827492903547/1208216161546522

Copy link

github-actions bot commented Oct 1, 2024

Coverage of commit 4c8fd25

Summary coverage rate:
  lines......: 75.8% (2483 of 3277 lines)
  functions..: no data found
  branches...: no data found

Files changed coverage rate:
                                                                                     |Lines       |Functions  |Branches    
  Filename                                                                           |Rate     Num|Rate    Num|Rate     Num
  =========================================================================================================================
  src/lamp_py/aws/s3.py                                                              |48.4%    287|    -     0|    -      0
  src/lamp_py/ingestion/light_rail_gps.py                                            |54.4%     79|    -     0|    -      0
  src/lamp_py/performance_manager/flat_file.py                                       |82.7%    104|    -     0|    -      0
  src/lamp_py/performance_manager/l0_gtfs_rt_events.py                               |95.6%    135|    -     0|    -      0
  src/lamp_py/performance_manager/l0_gtfs_static_load.py                             |88.1%    135|    -     0|    -      0
  src/lamp_py/performance_manager/l0_rt_trip_updates.py                              |92.9%     70|    -     0|    -      0
  src/lamp_py/performance_manager/l0_rt_vehicle_positions.py                         |98.6%     71|    -     0|    -      0
  src/lamp_py/performance_manager/l1_cte_statements.py                               | 100%     14|    -     0|    -      0
  src/lamp_py/performance_manager/l1_rt_trips.py                                     | 100%    154|    -     0|    -      0
  src/lamp_py/postgres/rail_performance_manager_schema.py                            | 100%    202|    -     0|    -      0

Download coverage report

Copy link

github-actions bot commented Oct 1, 2024

Coverage of commit 962bca9

Summary coverage rate:
  lines......: 75.4% (2471 of 3277 lines)
  functions..: no data found
  branches...: no data found

Files changed coverage rate:
                                                                                     |Lines       |Functions  |Branches    
  Filename                                                                           |Rate     Num|Rate    Num|Rate     Num
  =========================================================================================================================
  src/lamp_py/aws/s3.py                                                              |48.4%    287|    -     0|    -      0
  src/lamp_py/ingestion/light_rail_gps.py                                            |54.4%     79|    -     0|    -      0
  src/lamp_py/performance_manager/flat_file.py                                       |82.7%    104|    -     0|    -      0
  src/lamp_py/performance_manager/l0_gtfs_rt_events.py                               |95.6%    135|    -     0|    -      0
  src/lamp_py/performance_manager/l0_gtfs_static_load.py                             |88.1%    135|    -     0|    -      0
  src/lamp_py/performance_manager/l0_rt_trip_updates.py                              |92.9%     70|    -     0|    -      0
  src/lamp_py/performance_manager/l0_rt_vehicle_positions.py                         |98.6%     71|    -     0|    -      0
  src/lamp_py/performance_manager/l1_cte_statements.py                               | 100%     14|    -     0|    -      0
  src/lamp_py/performance_manager/l1_rt_trips.py                                     | 100%    154|    -     0|    -      0
  src/lamp_py/postgres/rail_performance_manager_schema.py                            | 100%    202|    -     0|    -      0

Download coverage report

@rymarczy rymarczy requested a review from arkadyan October 1, 2024 17:59
Copy link

@arkadyan arkadyan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@rymarczy rymarczy merged commit f0a4652 into main Oct 2, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants