Alert query examples

Select a database and table in the alert editor. $__table refers to that table, and $__time_interval is supplied from Check every. The alert system supplies the time bounds; do not add a rolling ago() filter or render timechart to these queries.

Return timestamp, a numeric value, and only the labels you intend to alert on. Value column can select another numeric column; otherwise Berserk uses value or the sole numeric column. Every other returned column identifies the group. An extra changing measurement can accidentally create a new group on every evaluation and prevent grace periods from completing. Duplicate bucket/group rows, ambiguous measurements, and invalid or misaligned timestamps are errors.

Maximum gauge value per interval

For a gauge named app.queue.depth, alert on the largest observed queue depth per service:

$__table
| where metric_name == "app.queue.depth"
| summarize value = max(toreal(value))
    by bin(timestamp, $__time_interval),
       service = tostring(resource["service.name"])

Replace the metric name with a gauge emitted by your application. With a critical threshold of 100 and comparison >, any sample above 100 makes that service's interval breach. Use avg for the interval's sample average, or min if every observed sample must exceed the threshold. These are sample-based aggregates, not time-weighted measurements.

For one alert across all matching samples, remove the service grouping and its preceding comma while keeping the timestamp bin.

Error log counts, including healthy zeroes

Count OTel logs with ERROR or higher severity in each interval:

$__table
| where isnotnull(body)
| summarize value = countif(severity_number >= 17)
    by bin(timestamp, $__time_interval)

This is the query shown in the backtest screenshot. The threshold is an error count per interval, not errors per second. For separate service alerts, add service = tostring(resource["service.name"]) to the grouping.

Keep non-error logs in the input so an interval containing only healthy logs returns zero. Filtering to errors first would make that interval disappear. An interval with no logs at all is still missing, not zero. Missing or unpopulated severity fields do not count as errors; check your source's log schema before using this rule.

Counter rate per service

For an OTel counter named app.requests, compute each series' rate before combining series into a service total:

$__table
| where metric_name == "app.requests"
| summarize rate = otel_rate($raw)
    by bin(timestamp, $__time_interval), metric_hash,
       service = tostring(resource["service.name"])
| summarize value = sum(rate) by timestamp, service

The threshold is requests per second. This two-stage form produces a rate per metric_hash, then combines those rates while retaining the bucket. otel_rate($raw) also tracks series internally, so explicitly grouping by metric_hash is optional when only the service total is needed. Choose an interval long enough for multiple samples per counter series; cumulative rate calculation needs at least two samples. Do not use the raw cumulative counter maximum as an interval rate.

Histogram percentile per service

For an OTel histogram named app.request.duration:

$__table
| where metric_name == "app.request.duration"
| summarize value = otel_histogram_percentile($raw, 95)
    by bin(timestamp, $__time_interval),
       service = tostring(resource["service.name"])

The threshold uses the metric's unit. If duration is recorded in seconds, 0.5 means 500 milliseconds. This computes the percentile from the histograms rather than averaging precomputed percentiles.

Express timing in an Alert document

The following example checks five-minute maximum queue depths. It requires two breaching intervals to fire and two healthy intervals to recover, with a 30-second ingestion allowance. Replace the database, table, metric, and connection names with your own. The named notification connection must already exist.

apiVersion: bzrk.dev/v1alpha1
kind: Alert
metadata:
  path: queues/high-depth
spec:
  enabled: false
  display:
    title: Queue depth high
  query:
    database: production
    targetTable: telemetry
    kql: |
      $__table
      | where metric_name == "app.queue.depth"
      | summarize value = max(toreal(value))
          by bin(timestamp, $__time_interval),
             service = tostring(resource["service.name"])
    timeColumn: timestamp
    valueColumn: value
  condition:
    operator: gt
    critical: 100
    degraded: 75
  schedule:
    every: 5m
    for: 10m
    keepFiringFor: 10m
    ingestionDelay: 30s
  notifications:
    channels:
      - connection: oncall

See evaluation and recovery for escalation between degraded and critical, interrupted grace periods, and missing measurements.