Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor unnesting for service alerts #2583

Merged
merged 3 commits into from
May 12, 2023
Merged

Conversation

lauriemerrell
Copy link
Contributor

@lauriemerrell lauriemerrell commented May 11, 2023

Description

Describe your changes and why you're making them. Please include the context, motivation, and relevant dependencies.

In the course of working on #2489, I realized that our current unnesting for service alerts does not seem appropriate. We have three separate tables for informed entities, active periods, and translations, but an individual "alert" actually comprises all three of these elements.

It makes more sense to have one fully unnested table, deduped on language.

Resolves #N/A

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation

How has this been tested?

Include commands/logs/screenshots as relevant.

All new tests pass on the new tables. I also did full refreshes of the two updated mart tables (daily and trip_summaries) and confirmed that for the past week my namespace copy is identical to the prod copy:

WITH prod AS (
  SELECT *
  FROM `cal-itp-data-infra.mart_gtfs.fct_service_alerts_trip_summaries`
),

tst AS (
  SELECT *
  FROM `cal-itp-data-infra-staging.laurie_mart_gtfs.fct_service_alerts_trip_summaries`
)

SELECT 
  prod.*,
  tst.*
FROM prod
FULL OUTER JOIN tst
USING (key)
WHERE COALESCE(prod.dt, tst.dt) > '2023-05-04' AND COALESCE(prod.dt, tst.dt) < CURRENT_DATE() AND (prod.key IS NULL OR tst.key IS NULL)

Post-merge follow-ups

Document any actions that must be taken post-merge to deploy or otherwise implement the changes in this PR (for example, running a full refresh of some incremental model in dbt). If these actions will take more than a few hours after the merge or if they will be completed by someone other than the PR author, please create a dedicated follow-up issue and link it here to track resolution.

  • No action required
  • Actions required (specified below)

Finish the deletion process for the deprecated mart tables; they have been added to the deprecation spreadsheet and have been announced on Slack. I have reminders set to soft- and hard-delete in 1 and 2 weeks respectively.

No full refresh is required after merge because the swap-in is identical and the partitions are the same.

@github-actions
Copy link

github-actions bot commented May 11, 2023

Warehouse report 📦

New models 🌱

mart.gtfs.fct_service_alerts_messages_unnested
intermediate.gtfs.int_gtfs_rt__service_alerts_fully_unnested

Changed models 🔀

mart.gtfs.fct_daily_service_alerts
mart.gtfs.fct_service_alerts_trip_summaries

@lauriemerrell lauriemerrell marked this pull request as ready for review May 11, 2023 22:29
@lauriemerrell lauriemerrell self-assigned this May 11, 2023
Copy link
Contributor

@atvaccaro atvaccaro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm thanks for fixing my stuff

@lauriemerrell lauriemerrell merged commit 3a9ac95 into main May 12, 2023
@lauriemerrell lauriemerrell deleted the refactor-service-alerts branch May 12, 2023 18:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants