Alerts runtime support

[begelundmuller]

Overview: Implements an Alert resource type and associated YAML parser and reconciler with the following high-level features:

• An alert can be triggered on a cron schedule and/or when its dependencies are refreshed
• An alert can either check relative to its current execution time (default) or for each of a given interval since last refresh
• An alert has a query that it runs every time it is checked
• An alert execution results in one of three states:
  • pass (query returned no rows)
  • fail (query returned 1+ rows)
  • error (query failed)
• An alert can send emails either on every trigger OR only when there is a state change

Preliminary YAML syntax:

kind: alert
title: My Alert

# Configuration of when to check the alert
refresh:
  # When a query dependency is refreshed
  ref_update: true

  # On a time schedule:
  # cron: "0 * * * *"
  # every: 24h
  # time_zone: UTC

  # Disable the alert without deleting it
  # disable: true

# Watermark to use as the execution time of the alert (default: trigger_time).
# For "inherit", it will take the lowest watermark of the alert's refs.
watermark: trigger_time|inherit

# Configure the alert to check once per elapsed time interval (instead of once for the current watermark).
# If multiple time intervals have elapsed, the alert will be checked for each interval.
# If no time intervals have elapsed, the alert will not be checked.
intervals:
  # ISO duration of each interval
  duration: P1D
  # Whether unclosed intervals should be checked
  check_unclosed: false|true
  # Maximum number of intervals to check for one invocation.
  # If the limit is exceeded, it will only check the most recent intervals.
  limit: 10

# Configuration for the query to run
query:
  name: MetricsViewAggregation
  for: 
    # The user's attributes to use for security policies
    user_email: [email protected]
  args:
    metrics_view: mv1
    dimensions:
      - country
    measures:
      - sales
    time_range:
      iso_duration: P1W
    having:
      # Condition for: sales >= 1000
      cond:
        op: OPERATION_GTE
        exprs:
          - ident: sales
          - val: 1000

# Configuration for how to send emails
email:
  recipients:
    - [email protected]

  # The alert status cases to trigger an email for:
  on_recover: true
  on_fail: true
  on_error: true

  # Whether to send an email again if the alert triggers with the same status as previously
  renotify: false,true # Default: false
  # How long to wait until renotifying
  renotify_after: P24H

Other YAML changes:
In dashboard definitions, you can now specify a watermark expression:

model: foo
timeseries:  __time
watermark: date_trunc('day', max(__time)) 
# If not specified, the watermark defaults to max({{ timeseries }})

In any refresh clause (for sources, models, reports, alerts), you now have two new options:

refresh:
  disable: true # completely disable the resource (without deleting it)
  ref_update: false # don't refresh when a dependency is refreshed

Task list:

• Define initial resource spec and state
• Implement initial YAML syntax
• Implement triggering logic
• Implement interval logic
• Implement query logic
• Implement email logic
• Experiment with watermark triggers (based on changes to max timestamp of underlying data)

Contributes to #3791

[netlify]

✅ Deploy Preview for rill-developer canceled.

Name	Link
🔨 Latest commit	15d0118
🔍 Latest deploy log	https://app.netlify.com/sites/rill-developer/deploys/65bbd2ca1080d30008431c43

[AdityaHegde]

Should the timeout be a multiple of number of each block of interval this needs to run?

[AdityaHegde]

Looking at the execution in depth, it will recover from last executed interval. But there will be a failure in reconcile no? Would that lead to resource being deleted and all the history being lost?

[begelundmuller]

Good questions. I made the timeout apply to all queries together since the purpose of it is to prevent runaway query execution that makes reconciles take a very long time.

About failure, yes it will recover and lead to a reconcile error, but that won't cause the resource to be deleted. The resource will still be there (and keep its state), it will just have .Meta.ReconcileError set until it is retried.

[AdityaHegde]

Would this cause an issue if the ref was created for the 1st time, went into error state and was fixed?

I remember this case when state version was reset and was the same if it came out of error state vs being created for the 1st time.

[begelundmuller]

Good catch! Fixed it

[begelundmuller]

Linter is failing for an unrelated reason fixed in #3996

[AdityaHegde]

Looks great!