Evaluation and recovery

Each alert evaluates one numeric measurement per group per completed interval. Its query decides what that measurement means: a maximum, average, count, rate, or percentile. The alert compares that measurement with its thresholds and updates each group's state independently.

Timing and grace periods

The grace selectors display interval counts and their duration: with Check every = 5m, 2× (10m) means two consecutive five-minute measurements. Changing Check every changes the duration represented by those counts. In an Alert document, for and keepFiringFor must be whole multiples of every.

For = Immediately and Keep firing = 0× require the first eligible breaching or recovering measurement. They do not evaluate an unfinished interval or bypass ingestion delay. 1× also requires one measurement; 2× requires two, rather than two additional intervals after the first.

Grace periods count observations, not elapsed wall-clock time. Waiting ten minutes without new measurements cannot complete a ten-minute grace period.

Which interval is evaluated?

Buckets are fixed, UTC-aligned, and include their start but exclude their end. With a five-minute interval, examples are [12:00, 12:05) and [12:05, 12:10). An event at exactly 12:05 belongs to the second bucket. This is not a rolling window ending at the instant the scheduler runs.

With a 30-second ingestion delay, the first bucket becomes eligible at 12:05:30. It is processed on a scheduler run at or after that time; the delay is an eligibility boundary, not a promise of a notification at that exact second. If the backend supplies an ingestion watermark, it must also cover the bucket end plus the delay. Without a watermark, a complete query result uses the configured delay alone.

The watermark tracks ingestion progress, not event timestamps. Choose a delay that accommodates the source's arrival latency. A processed bucket is not reopened when later data arrives. The next run can catch up on unfinished buckets, and evaluating an already processed bucket cannot repeat its state transitions.

Firing, escalation, and recovery

The comparison first checks Critical, then the optional Degraded threshold. A value that satisfies neither is healthy. For >, equality does not breach; for >=, it does. For high-value alerts, the degraded threshold must be below critical; for low-value alerts, the order is reversed.

• From healthy, For controls entry into degraded or critical.
• Critical measurements also count toward degraded evidence. A critical measurement followed by a degraded one can satisfy two intervals of degraded evidence.
• Escalation to critical requires its own consecutive critical measurements.
• From critical, Keep firing controls the downgrade after consecutive measurements below critical severity. If those measurements are all healthy, the group can recover directly to healthy.
• From degraded, recovery requires consecutive healthy measurements.

A renewed breach interrupts recovery evidence. Each group tracks its own streak: a healthy host cannot clear another host's alert.

Example: recovery interrupted by missing data

Assume Check every = 5m, comparison >, critical threshold 100, no degraded threshold, For = 2×, and Keep firing = 2×. The table shows state after each eligible bucket is evaluated.

Missing data and failed evaluations

An absent group, empty bucket, or null measurement is unknown, not zero and not healthy. Missing measurements break consecutive firing and recovery evidence. A previously firing group retains its severity until actual measurements satisfy recovery.

A failed, truncated, or partial query cannot contribute evidence or clear an alert. Waiting for ingestion completeness also leaves the state unchanged. If unfinished intervals are later evaluated successfully, their evidence is processed in chronological order; a genuinely missing interval still breaks the streak.

For error-count alerts, use countif over all logs instead of filtering to errors before counting. That produces zero when logs exist but none are errors. If the group sends no logs at all, it remains missing. Detecting absence requires a query that explicitly produces a measurement for the expected group; the alert system does not invent zeroes.

Why the query still needs a time bin

Aggregation and binning do different jobs. max selects a measurement; bin(timestamp, $__time_interval) keeps each interval's measurement separate. For a query restricted to exactly one aligned interval, taking a maximum without binning would produce the same numeric value when samples exist. The current alert contract nevertheless requires the bin.

Live checks, historical preview, and catch-up use the same query. Preview and catch-up can evaluate many intervals in one execution. Without the time group, a maximum would collapse the whole selected range into one result. The validator therefore requires the first aggregation to group by bin(timestamp, $__time_interval) and subsequent aggregations to preserve that bucket column. The interval comes from Check every, not the preview range.

For a maximum-based rule, a breach means at least one observed sample exceeded the threshold in that interval. Even several consecutive breaching maxima do not prove the underlying value stayed above the threshold continuously. Choose an average or minimum when that better expresses the intended condition; measurements cannot establish what happened between samples.

See alert query examples for complete expressions.