Creating an incremental table for vendor vehicle message age by day. #3477
Conversation
fsalemi
requested review from
evansiroky,
vevetron and
tiffanychu90
as code owners
|
Warehouse report 📦 Checks/potential follow-upsChecks indicate the following action items may be necessary.
New models 🌱calitp_warehouse.mart.gtfs_quality.fct_daily_vendor_vehicle_positions_message_age_summary DAG
Legend (in order of precedence)
|
|
I'm helping @fsalemi install pre-commit hooks into his environment. |
fsalemi
force-pushed
the
vendor_vehicle_positions
branch
from
5f6b126
to
982f009
Compare
| INNER JOIN {{ ref('dim_gtfs_service_data') }} AS GSD | ||
| ON VPM.gtfs_dataset_key = GSD.gtfs_dataset_key | ||
| WHERE {{ incremental_where(default_start_var='PROD_GTFS_RT_START') }} | ||
| AND customer_facing IS TRUE |
There was a problem hiding this comment.
I don't think we want to limit this table to just customer facing datasets.
There was a problem hiding this comment.
Thanks or the hint. I deleted customer facing check.
| bridge_organization_x_gtfs_dataset AS ( | ||
| SELECT | ||
| organization_name, | ||
| gtfs_dataset_name, | ||
| MIN(_valid_from) AS valid_from, | ||
| MAX(_valid_to) AS valid_to | ||
| FROM {{ ref('bridge_organizations_x_gtfs_datasets_produced') }} | ||
| WHERE | ||
| (gtfs_dataset_name LIKE '%VehiclePositions%' | ||
| OR gtfs_dataset_name LIKE '%Vehicle Positions%' | ||
| OR gtfs_dataset_name LIKE '%VehiclePosition%' | ||
| OR gtfs_dataset_name LIKE '%Vehicle Position%') | ||
| GROUP BY | ||
| organization_name, gtfs_dataset_name | ||
| ), | ||
|
|
||
| vendor_vehicle_positions_ages AS ( | ||
| SELECT DISTINCT | ||
| dt, | ||
| organization_name, | ||
| _header_message_age, | ||
| _vehicle_message_age, | ||
| _vehicle_message_age_vs_header | ||
| FROM vehicle_positions_ages AS VPA | ||
| INNER JOIN bridge_organization_x_gtfs_dataset AS BOGD | ||
| ON VPA.gtfs_dataset_name = BOGD.gtfs_dataset_name | ||
| AND dt BETWEEN DATE(valid_from) AND DATE(valid_to) | ||
| ), |
There was a problem hiding this comment.
I don't think we may need to select into the new bridge_organization_x_gtfs_dataset from the bridge_organizations_x_gtfs_datasets_produced. Instead, it seems like the vendor_vehicle_positions_ages could join directly from bridge_organizations_x_gtfs_datasets_produced. Also instead of matching on dataset name, we should be matching on gtfs_dataset_key.
There was a problem hiding this comment.
Thanks for the hint. I used gtfs_dataset_key instead.
fsalemi
force-pushed
the
vendor_vehicle_positions
branch
from
982f009
to
5c680b6
Compare
|
@evansiroky |
There was a problem hiding this comment.
I think this is good for a first pass. I think eventually we may also want to include the trip_id and the organization key. That could be done in a later effort.
Bringing in the organization key would help do more joins to the vendor's record that may be present elsewhere. The trip_id could help analyze things related to whether the vehicle appears to be doing revenue service.
Creating a new incremental table for vendor vehicle position message age by day.
Resolves issue #3478
Type of change
How has this been tested?
Yes,
poetry run dbt run -s +fct_daily_vendor_vehicle_positions_message_age_summary
20:17:15 Finished running 12 view models, 11 table models, 1 incremental model in 0 hours 4 minutes and 39.96 seconds (279.96s). 20:17:15 20:17:15 Completed successfully 20:17:15 20:17:15 Done. PASS=24 WARN=0 ERROR=0 SKIP=0 TOTAL=24Post-merge follow-ups
Document any actions that must be taken post-merge to deploy or otherwise implement the changes in this PR (for example, running a full refresh of some incremental model in dbt). If these actions will take more than a few hours after the merge or if they will be completed by someone other than the PR author, please create a dedicated follow-up issue and link it here to track resolution.