Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GTFS RT date, time, and trip labeling updates #2489

Merged
merged 58 commits into from
Jul 13, 2023
Merged

Conversation

lauriemerrell
Copy link
Contributor

@lauriemerrell lauriemerrell commented Apr 20, 2023

Description

Describe your changes and why you're making them. Please include the context, motivation, and relevant dependencies.

This PR fundamentally restructures date/time handling and trip keying in GTFS RT models. See this release note doc for a full description of the issues and fix approaches.

It also deletes the dbt model for mart_ad_hoc.fct_daily_vehicle_location_trip_counts, rather than update it to make it compatible with these changes, because it was causing issue #2732 and the model generally seems a bit redundant with other models that are available.

Also adds vehicle.wheelchairAccessible to the trip updates raw messages models, I am not sure why it was missing before.

Decided not to do:

  • Add test that time zone doesn't change over time - this doesn't seem worth it

Resolves #2413
Resolves #2347
Resolves #2732 by deleting the model
Fixes CAL-ITP-DATA-INFRA-25HX (x)

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

How has this been tested?

Include commands/logs/screenshots as relevant.

Ran an unqualified dbt run and dbt test, confirmed everything runs and no new tests fail

Post-merge follow-ups

Document any actions that must be taken post-merge to deploy or otherwise implement the changes in this PR (for example, running a full refresh of some incremental model in dbt). If these actions will take more than a few hours after the merge or if they will be completed by someone other than the PR author, please create a dedicated follow-up issue and link it here to track resolution.

  • No action required
  • Actions required (specified below)
  • We are planning a full full refresh (no selectors) to capture everything here and various changes in GTFS schedule data
  • Post in Slack explaining changes
  • Update RT files Metabase dashboard to no longer need a join to get dataset name

@github-actions
Copy link

github-actions bot commented Jun 9, 2023

Warehouse report 📦

Checks/potential follow-ups

Checks indicate the following action items may be necessary.

  • For new models, do they all have a surrogate primary key that is tested to be not-null and unique?
  • For modified incremental models (or incremental models whose parents are modified), does the PR description identify whether a full refresh is needed for these tables?

New models 🌱

calitp_warehouse.intermediate.gtfs.int_gtfs_rt__service_alerts_day_map_grouping

calitp_warehouse.intermediate.gtfs.int_gtfs_rt__service_alerts_trip_day_map_grouping

calitp_warehouse.intermediate.gtfs.int_gtfs_rt__trip_updates_trip_day_map_grouping

calitp_warehouse.intermediate.gtfs.int_gtfs_rt__vehicle_positions_trip_day_map_grouping

Changed incremental models 🔀

calitp_warehouse.mart.gtfs_quality.fct_daily_service_alerts_message_age_summary

calitp_warehouse.mart.ad_hoc.fct_daily_trip_update_status_counts

calitp_warehouse.mart.gtfs_quality.fct_daily_trip_updates_message_age_summary

calitp_warehouse.mart.ad_hoc.fct_daily_vehicle_location_feed_trip_counts

calitp_warehouse.mart.gtfs_quality.fct_daily_vehicle_positions_message_age_summary

calitp_warehouse.mart.gtfs.fct_service_alerts_messages_unnested

calitp_warehouse.mart.gtfs.fct_trip_updates_no_stop_times

calitp_warehouse.mart.gtfs.fct_vehicle_locations

DAG

Legend (in order of precedence)

Resource type Indicator Resolution
Large table-materialized model Orange Make the model incremental
Large model without partitioning or clustering Orange Add partitioning and/or clustering
View with more than one child Yellow Materialize as a table or incremental
Incremental Light green
Table Green
View White

@lauriemerrell
Copy link
Contributor Author

decided this PR is getting too unwieldy, I'm going to break it up further and do each RT message type in dedicated PRs

@lauriemerrell
Copy link
Contributor Author

... forgot about fct_observed_trips, the changes need to be batched 🤦‍♀️ reopening

@lauriemerrell lauriemerrell reopened this Jun 16, 2023
@lauriemerrell lauriemerrell changed the title RT trip labeling / matching updates GTFS RT date, time, and trip labeling updates Jun 20, 2023
@lauriemerrell lauriemerrell merged commit 9ab0f58 into main Jul 13, 2023
4 of 5 checks passed
@lauriemerrell lauriemerrell deleted the rt-trip-matching branch July 13, 2023 16:17
@lauriemerrell lauriemerrell removed the do-not-merge Do not merge, even if approved label Jul 14, 2023
lauriemerrell added a commit that referenced this pull request Jul 17, 2023
* update profile based on observed behavior

* address failing test and update keying

* only test on sample

* update intermediate tests

* working fix for schedule change issue

* sampled where updates

* add schedule URL to intermediate key

* add comments

* forward date URL to dataset mapping

* get vp test passing

* fix gtfs dataset join to use new url mapping

* coalesce feed time zone and refactor schedule join to use url map

* deprecate trip status count model per slack convo

* minor refactors on vehicle locations

* working -- get dataset info from url map
@lauriemerrell lauriemerrell mentioned this pull request Jul 18, 2023
6 tasks
lauriemerrell added a commit that referenced this pull request Jul 19, 2023
* update timeout

* convert to unique proportion trips

* add docs about uniqueness violations

* turn on test sampling

* fix keying for daily service alerts model

* update comment per pr review
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants