Skip to content

Commit

Permalink
rename column per pr review and update yaml
Browse files Browse the repository at this point in the history
  • Loading branch information
Laurie Merrell committed Jul 12, 2023
1 parent 064a4b3 commit ca62470
Show file tree
Hide file tree
Showing 8 changed files with 28 additions and 18 deletions.
2 changes: 1 addition & 1 deletion warehouse/macros/gtfs_rt_messages_keying.sql
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ WITH
-- if we ever backfill v1 RT data, the reliance on _config_extract_ts for joins in this table may become problematic
SELECT
urls_to_gtfs_datasets.gtfs_dataset_key AS gtfs_dataset_key,
rt_datasets.name as name,
rt_datasets.name as gtfs_dataset_name,
schedule_datasets.key AS schedule_gtfs_dataset_key,
schedule_datasets.base64_url AS schedule_base64_url,
schedule_datasets.name AS schedule_name,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ int_gtfs_rt__service_alerts_fully_unnested AS (
base64_url,
_extract_ts,
_config_extract_ts,
name,
gtfs_dataset_name,
schedule_gtfs_dataset_key,
schedule_base64_url,
schedule_name,
Expand Down
32 changes: 21 additions & 11 deletions warehouse/models/mart/gtfs/_mart_gtfs_fcts.yml
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,7 @@ models:
value updates every time that the Airtable data extract runs, whereas `dim_gtfs_datasets` has versioning applied
and `_valid_from` only updates when an attribute within the record actually changes.
- &gtfs_rt_name
name: _gtfs_dataset_name
name: gtfs_dataset_name
description: |
Name from the associated GTFS dataset record.
- &gtfs_rt_schedule_dataset_key
Expand All @@ -175,6 +175,21 @@ models:
field: key
config:
where: '__rt_sampled__'
- &rt_service_date
name: service_date
description: |
Attempt to identify the `service_date` (corresponding to the related schedule feed) for trip activity referenced in a
GTFS RT feed. It uses the following fallback logic:
* If `trip_start_date` is populated, use that. This is assumed to be provided with respect to `schedule_feed_timezone`.
* Otherwise, for trip updates and vehicle positions, if `trip_update_timestamp` or `vehicle_timestamp` (respectively) are populated, convert that
to the `schedule_feed_timezone` and extract the date from that.
* Otherwise, use `header_timestamp` converted to `schedule_feed_timezone` and extract the date.
* Finally (and this generally should not happen, since `header_timestamp` should be populated), fall back to `_extract_ts` converted to `schedule_feed_timezone` and extract the date.
- &rt_schedule_base64_url
name: schedule_base64_url
description: |
URL-safe base64-encoded URL of the schedule feed used to validate
this RT feed.
- &gtfs_rt_schedule_dataset_name
name: schedule_name
description: |
Expand Down Expand Up @@ -371,6 +386,8 @@ models:
- *gtfs_rt_dt
- *gtfs_rt_hour
- *base64_url
- *rt_schedule_base64_url
- *rt_service_date
- *gtfs_rt_extract_ts
- *gtfs_rt_config_extract_ts
- *gtfs_rt_name
Expand Down Expand Up @@ -684,6 +701,8 @@ models:
- *gtfs_rt_dt
- *gtfs_rt_hour
- *base64_url
- *rt_schedule_base64_url
- *rt_service_date
- *gtfs_rt_extract_ts
- *gtfs_rt_config_extract_ts
- *gtfs_rt_name
Expand Down Expand Up @@ -759,16 +778,7 @@ models:
# field: key
# config:
# where: '__rt_sampled__'
- &rt_service_date
name: service_date
description: |
Attempt to identify the `service_date` (corresponding to the related schedule feed) for trip activity referenced in a
GTFS RT feed. It uses the following fallback logic:
* If `trip_start_date` is populated, use that. This is assumed to be provided with respect to `schedule_feed_timezone`.
* Otherwise, for trip updates and vehicle positions, if `trip_update_timestamp` or `vehicle_timestamp` (respectively) are populated, convert that
to the `schedule_feed_timezone` and extract the date from that.
* Otherwise, use `header_timestamp` converted to `schedule_feed_timezone` and extract the date.
* Finally (and this generally should not happen, since `header_timestamp` should be populated), fall back to `_extract_ts` converted to `schedule_feed_timezone` and extract the date.
- *rt_service_date
- *gtfs_rt_dataset_key
- *gtfs_rt_dt
- *gtfs_rt_hour
Expand Down
2 changes: 1 addition & 1 deletion warehouse/models/mart/gtfs/fct_service_alerts_messages.sql
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ fct_service_alerts_messages AS (
base64_url,
_extract_ts,
_config_extract_ts,
name,
gtfs_dataset_name,
schedule_gtfs_dataset_key,
schedule_base64_url,
schedule_name,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ fct_service_alerts_messages_unnested AS (
base64_url,
_extract_ts,
_config_extract_ts,
name,
gtfs_dataset_name,
schedule_gtfs_dataset_key,
schedule_base64_url,
schedule_name,
Expand Down
2 changes: 1 addition & 1 deletion warehouse/models/mart/gtfs/fct_trip_updates_messages.sql
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ fct_trip_updates_messages AS (
base64_url,
_extract_ts,
_config_extract_ts,
name,
gtfs_dataset_name,
schedule_gtfs_dataset_key,
schedule_base64_url,
schedule_name,
Expand Down
2 changes: 1 addition & 1 deletion warehouse/models/mart/gtfs/fct_vehicle_locations.sql
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ first_keying_and_filtering AS (
-- so you can end up intermingling multiple vehicles
-- however, not clear this issue remains if we are also dropping rows with no trip
-- since regional feed does have unique trip IDs per agency
AND name != 'Bay Area 511 Regional VehiclePositions'
AND gtfs_dataset_name != 'Bay Area 511 Regional VehiclePositions'
),

deduped AS (
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ fct_vehicle_positions_messages AS (
base64_url,
_extract_ts,
_config_extract_ts,
name,
gtfs_dataset_name,
schedule_gtfs_dataset_key,
schedule_base64_url,
schedule_name,
Expand Down

0 comments on commit ca62470

Please sign in to comment.