Service Metrics

All Berserk services emit metrics via OpenTelemetry. Metrics are exported over OTLP to the configured collector endpoint and can be queried in Berserk itself.

Each metric name is prefixed with bzrk. followed by the service scope (e.g. bzrk.ui.query_duration).

A pre-built Grafana dashboard is available for download: bzrk-service-metrics.json. Import it into Grafana and select your Berserk datasource to visualize all metrics below.

CacheServer

Metrics for the same-host segment-cache server (Step 2/2.9). Emitted partly from apps/cache_server/src/metrics.rs (server-side counters and gauges) and partly from apps/query/src/cache_spawn.rs (the supervisor's respawn_total).

Two pages live here as decomposition aids for the user-facing bzrk.query.errors alert: a crash-loop signal (cache_server is panicking on startup) and a Revoke-pile signal (eviction is back-pressured by stuck client connections).

Metric	Type	Across pods	Unit	Labels	Description
`bzrk.cache_server.respawn_total`	counter	—	—	—	apps/query supervisor restarted the cache_server child, regardless of cause. Sustained > 0 indicates a crash loop (corrupt cache.log, bad config, missing filesystem permissions, …) or a wedge loop (see `liveness_kill_total` / `watchdog_exit_total`). Emitted from apps/query/cache_spawn.rs. Decomposition counters (subsets of this metric): - `liveness_kill_total` — supervisor's UDS Hello probe failed and force-killed the child. - `watchdog_exit_total` — child self-exited via its in-process liveness watchdog. Restarts that are neither (panics, OOM, deploy SIGTERM, …) show only on `respawn_total`.
`bzrk.cache_server.liveness_kill_total`	counter	—	—	—	apps/query supervisor force-killed the cache_server child after [LIVENESS_PROBE_FAIL_THRESHOLD] consecutive Hello-handshake probe failures. This is the primary alive-but-wedged signal — typically fires before `watchdog_exit_total` because the supervisor's probe window (~15s of failures) is shorter than the in-process watchdog's stall budget (30s). Bumped exactly once per kill — the resulting respawn also counts on `respawn_total`. Non-zero means the child was alive (its `child.wait()` had not returned) but its Hello pipeline (connect → accept → dispatch → reply) wasn't completing. Emitted from apps/query/cache_spawn.rs.
`bzrk.cache_server.watchdog_exit_total`	counter	—	—	—	cache_server's in-process liveness watchdog (apps/cache_server/src/liveness.rs) tripped — the child self-exited because its accept-loop heartbeat counter stopped advancing for the watchdog window. The supervisor observes the child exit and bumps this counter. The resulting respawn also counts on `respawn_total`. Distinct from `liveness_kill_total`: the watchdog catches "tokio still alive enough to fire its self-exit timer," and the supervisor probe catches "tokio is so dead it can't even self-exit." Emitted from apps/query/cache_spawn.rs.
`bzrk.cache_server.accept_active`	gauge	fleet total	—	—	Currently-open UDS connections accepted by the server. With Step 2.8 per-task sessions this tracks the number of live engine tasks holding a cache session. A persistent ceiling at the configured `max_inflight` cap signals saturation; sudden drops correlate with worker fan-in.
`bzrk.cache_server.dispatch_in_flight`	gauge	fleet total	—	—	Active dispatch tasks across all connections (one per inbound `ClientRequest` currently being processed). Sustained ceiling indicates slot starvation; correlate with `s3_fetches_in_flight` from `bzrk.segment_cache.*`.
`bzrk.cache_server.revoke_sent_total`	counter	—	—	—	`Revoke` frames fanned out by the eviction observer (Step 2.9 fire-and-forget). High rate alongside `evictions_with_active_holders_total` from `bzrk.segment_cache.*` means we're actively reclaiming disk on held keys; this is normal under bursty churn.
`bzrk.cache_server.open_fds`	gauge	worst pod	—	—	`bzrk_lib::count_open_fds()` periodic sample. Pair with `bzrk.cache_server.fd_limit` to compute `open_fds / fd_limit` on dashboards/alerts without joining against startup logs. cache_server holds N×M cache_fds + shm_fds + accepted UDS sockets; a runaway client-task count would show up here.
`bzrk.cache_server.fd_limit`	gauge	same on every pod	—	—	Current `RLIMIT_NOFILE` soft cap. Companion to `open_fds` — sampled on the same 10s tick so dashboards can show "fds: N / LIMIT (X%)" and alerts can fire on `open_fds / fd_limit > 0.8` before saturation. cache_server raises the soft limit to the hard cap at startup, so the value is effectively static; emitting it as a gauge keeps the query simple.

Gateway.proxy

Metric	Type	Across pods	Unit	Labels	Description
`bzrk.gateway.proxy.requests_total`	counter	—	—	`upstream`: ui, query, meta, ingest, other; `status_class`: 2xx, 3xx, 4xx, 5xx, other	Requests the gateway forwarded to a backend, by upstream and status class. Edge error rate = sum(4xx,5xx) / total per upstream. ui/query REST record the real upstream HTTP status; gRPC (query/meta/ingest) records transport-level status (2xx forwarded, 5xx unreachable).
`bzrk.gateway.proxy.request_duration_ms`	histogram	—	ms	`upstream`: ui, query, meta, ingest, other	Forwarded request latency in milliseconds, by upstream — time from gateway receiving the request to the upstream response headers.
`bzrk.gateway.proxy.upstream_endpoints`	gauge	worst pod	—	`upstream`: ui, query, meta, ingest, other	Resolved endpoints currently in an upstream's balanced set, sampled every 15s. Expect one per Ready pod behind a headless Service, or 1 for a normal ClusterIP name.
`bzrk.gateway.proxy.upstream_endpoints_quarantined`	gauge	worst pod	—	`upstream`: ui, query, meta, ingest, other	Endpoints excluded from balancing after repeated transport failures, sampled every 15s. Persistently non-zero means a pod is failing but still in DNS; equal to `upstream_endpoints` means every endpoint is failing and the balancer is serving on a degraded choice.

Ingest

Simplified OTLP ingest service that receives traces, metrics, and logs over HTTP/gRPC and uploads to S3 via ingest_client.

Metric	Type	Across pods	Unit	Labels	Description
`bzrk.ingest.queue_rejections`	counter	—	—	`reason`: concurrency_limit, inflight_bytes, memory_pressure, buffered_requests	Total requests rejected due to admission control (semaphore exhaustion or dead stream actor)
`bzrk.ingest.batch_flush_duration`	histogram	—	ms	`kind`: batch, keep_alive, retry	One `StreamIngestClient::upload()`, body included — ingest's own view of S3 write latency, and what the attempt budgets are calibrated against. The `kind` attribute separates `batch` from `retry` and `keep_alive`, whose latencies mean different things.
`bzrk.ingest.batch_inputs`	histogram	—	items	—	Incoming OTLP requests coalesced into one S3 batch flush. p50/p99 sizes the fan-in expected at the batch upload span links.
`bzrk.ingest.data_dropped`	counter	—	—	`reason`: concurrency_limit, inflight_bytes, memory_pressure, buffered_requests, meta_unavailable, no_capacity, budget_expired, stream_unavailable, shutting_down, router_timeout, unspecified, token_rejected, stream_stopped, upload_fatal, retry_budget_exhausted, shutdown_upload_fatal, shutdown_flush_timeout, shutdown_no_budget, shutdown_upload_failed	Requests whose data never reached S3. `reason` says which of four: admission refused it, its token was missing or rejected, nothing could carry it, or it was lost after a fast-mode 202 — only the last is silent loss.
`bzrk.ingest.lost_after_ack`	counter	—	—	`reason`: concurrency_limit, inflight_bytes, memory_pressure, buffered_requests, meta_unavailable, no_capacity, budget_expired, stream_unavailable, shutting_down, router_timeout, unspecified, token_rejected, stream_stopped, upload_fatal, retry_budget_exhausted, shutdown_upload_fatal, shutdown_flush_timeout, shutdown_no_budget, shutdown_upload_failed	Requests acknowledged with a fast-mode 202 whose data never reached S3, by drop reason: the silent subset of `data_dropped`, and the history behind the Ingest page's lost count, which shows process-lifetime totals only.
`bzrk.ingest.rejected_requests`	counter	—	—	`kind`: unauthorized, throttled, unavailable, failed	Requests answered with an error the sender can act on, counted at the pipeline's outcome. A body a transport refused before the pipeline (an oversized one, a bad gRPC frame) is not in it. The history behind the Ingest page's rejected count.
`bzrk.ingest.invalid_payloads`	counter	—	—	—	Payloads the OTLP validator refused, counted at the pipeline's outcome; `invalid_otlp_total` is the fast-path validator's own view. The history behind the Ingest page's invalid count.
`bzrk.ingest.accepted_requests`	counter	—	—	`signal`: traces, metrics, logs	Requests ingest took responsibility for: durably in S3 in durable mode, queued on a stream in fast mode. The denominator for `data_dropped`, which also counts requests. A fast-mode request lost after its 202 is counted in both. A request carrying no records (a key probe) is not traffic and is not counted.
`bzrk.ingest.accepted_records`	counter	—	—	`signal`: traces, metrics, logs	Spans, log records and metric data points in accepted requests.
`bzrk.ingest.accepted_bytes_uncompressed`	counter	—	By	`signal`: traces, metrics, logs	Size of accepted requests as OTLP protobuf, after Loki and Datadog are converted to it and before compression. The ingest volume we report to customers.
`bzrk.ingest.accepted_bytes_compressed`	counter	—	By	—	LZ4-compressed batches written to S3, counted once per batch when its upload succeeds. A batch mixes signals, so there is no `signal` label.
`bzrk.ingest.time_since_last_upload_seconds`	gauge	worst pod	s	—	Worst-silent stream's seconds since last successful S3 upload
`bzrk.ingest.inflight_requests`	gauge	fleet total	—	—	Admission permits in use across HTTP/gRPC/Loki transports
`bzrk.ingest.buffer_bytes`	gauge	fleet total	By	—	Bytes buffered across stream actors (pod-level SUM)
`bzrk.ingest.inflight_bytes`	gauge	fleet total	By	—	In-flight bytes reserved against the admission byte budget — the primary memory gate. Leading indicator of byte-budget pressure before throttling starts.
`bzrk.ingest.inflight_bytes_limit`	gauge	fleet total	By	—	Total admission byte budget (auto-sized to a fraction of the cgroup memory limit). Constant; lets dashboards compute utilization without joining config.
`bzrk.ingest.process_rss_bytes`	gauge	worst pod	By	—	Sampled process resident set size feeding the memory-ceiling admission gate
`bzrk.ingest.memory_ceiling_bytes`	gauge	same on every pod	By	—	RSS ceiling above which admission sheds (0 when the memory gate is disabled)
`bzrk.ingest.datadog_discarded`	counter	—	—	`endpoint`: check_run, intake, metadata, apm_stats	Datadog intake requests acknowledged and discarded: payloads we do not store or derive ourselves (service checks, host metadata, APM stats).
`bzrk.ingest.datadog_unrouted`	counter	—	—	—	Datadog intake requests to a path we do not serve (404). The path is logged at debug; a rise means a client sends an endpoint we have not routed yet.
`bzrk.ingest.phantom_write_retries`	counter	—	—	—	Total retry attempts after an uncertain upload (held by PendingRetry)
`bzrk.ingest.phantom_write_retry_budget_exhausted`	counter	—	—	—	Total times a PendingRetry exhausted its budget without resolving (fell back to retryable error)
`bzrk.ingest.invalid_otlp_total`	counter	—	—	`signal`: traces, metrics, logs; `verdict`: accepts_where_we_reject, rejected, skipped_rate_limited	OTLP payloads rejected by the fast-path validator. Attributes: signal (traces\|metrics\|logs), verdict (accepts_where_we_reject\|rejected\|skipped_rate_limited). `accepts_where_we_reject` > 0 indicates the fast-path validator disagrees with prost — strong parser-bug signal. See /debug/invalid-otlp-samples for the byte-level sample.

Janitor

Background service responsible for segment lifecycle management: merging small segments into larger ones and deleting tombstoned segments from cloud storage.

Metric	Type	Across pods	Unit	Labels	Description
`bzrk.janitor.segment_count`	gauge	same on every pod	—	—	Current number of segments in the cluster
`bzrk.janitor.total_data_size`	gauge	same on every pod	By	—	Total size of all segment data in cloud storage
`bzrk.janitor.segments_deleted`	counter	—	—	—	Total segments deleted from cloud storage
`bzrk.janitor.merge_cycle_duration`	histogram	—	ms	—	Duration of segment merge cycles
`bzrk.janitor.merge_failures`	counter	—	—	—	Total failed merge cycles
`bzrk.janitor.vsearch_merger_artifacts_emitted`	counter	—	—	—	Merged segments where the merger rebuilt VCEN/VTPH/VTPC/VIDF. See docs/dev/vidx-vsearch-impl-plan.md PR 8.
`bzrk.janitor.vsearch_merger_pre_feature_inputs`	counter	—	—	—	Input segments to a merge that had no vsearch artifacts (pre-feature). Each increment indicates one input was skipped during VXXX rebuild.
`bzrk.janitor.vsearch_merger_unstamped_rows`	counter	—	—	—	Rows seen during merge that lacked a template_id.FIELD stamp. Persistent nonzero rate indicates ingest-side stamping isn't keeping up with merger fan-in.
`bzrk.janitor.vsearch_merger_template_index_bytes`	gauge	worst pod	By	—	Peak transient JanitorTemplateIndex memory during a merge (sum of input VTPC embedding tables). docs/dev/vidx-vsearch.md 8.2 caps the risk; this metric flags the cap binding before OOM.
`bzrk.janitor.vsearch_merger_duration_ms`	histogram	—	ms	—	Wall-clock added to a merge by VXXX rebuild (loading input VTPCs + tier selection + writing output VCEN/VTPH/VTPC/VIDF). Excludes the base ROWS-merger time.

K8s

Cluster-state telemetry from the otel-collector's k8s_cluster and kubeletstats receivers. There is no Rust source for this scope — only alert metadata, as with apps/postgres.

These metrics keep their upstream OTel names and are emitted by a single collector for the whole shared K3s cluster, so every namespace is stamped bzrk.cluster.name == "dev". Both facts are why this scope declares a source: block: alerts here match raw metric names and select their environment by namespace instead of by cluster.

Metric	Type	Across pods	Unit	Labels	Description
`bzrk.k8s.k8s.container.ready`	gauge	one series per pod	—	—	1 when a container passes its readiness probe, 0 otherwise. Sampled continuously per container by the k8s_cluster receiver. A pod that cannot start — CrashLoopBackOff, ImagePullBackOff, a failing readiness probe, a stuck init container — holds this at 0 for as long as it keeps failing, which makes it a broader "workload cannot come up" signal than a restart count. It is also the only one of the two that catches a wedged rollout: during a rolling update the previous pod keeps serving, so `k8s.deployment.available` never drops.

Metric	Type	Across pods	Unit	Labels	Description
`bzrk.meta.db_pool_active`	gauge	worst pod	—	—	Number of active database connections in the pool
`bzrk.meta.db_pool_idle`	gauge	worst pod	—	—	Number of idle database connections in the pool
`bzrk.meta.merge_backlog`	gauge	same on every pod	—	—	Number of outstanding merge consolidation tasks in the queue (excludes 1-to-1 rewrite tasks, which are deprioritized and linger by design)
`bzrk.meta.merge_task_failures`	counter	—	—	—	Total segment merge tasks that failed and were deleted for the planner to re-evaluate
`bzrk.meta.oldest_merge_job`	gauge	same on every pod	s	—	Age in seconds of the oldest outstanding consolidation merge task (excludes 1-to-1 rewrite tasks, which are deprioritized and linger by design)
`bzrk.meta.oldest_undeleted_data_age_seconds`	gauge	same on every pod	s	—	Age in seconds of the oldest raw ingest blob the nursery still owes a delete for, over every stream whose deletion frontier is behind its floor. Read off the stream age ladder, so it is quantized to the ladder sampling interval and falls back to the stream's own age when no sample survives
`bzrk.meta.nursery_groups_reassigned`	counter	—	—	—	Stream groups that failed over: reassigned away from an owner that was not live — failover or restart handover; the new owner resumes from the committed merged frontier. Bootstrap assignment of unowned groups (a one-time event) is not counted, only logged.
`bzrk.meta.ingest_stream_transitions`	counter	—	—	—	Stream transitions the ingest control loop committed — a stream opened, joined, replaced or retired — by either entry point
`bzrk.meta.nursery_live_count`	gauge	same on every pod	—	—	Live nurseries seen by the coordinator this reconcile tick
`bzrk.meta.stream_group_count`	gauge	same on every pod	—	—	Total stream groups, owned or not
`bzrk.meta.stream_group_target_deficit`	gauge	same on every pod	—	—	Groups still to create so every live nursery owns its floor of four; a nursery above it counts nothing
`bzrk.meta.nursery_load_spread_bytes`	gauge	same on every pod	—	—	Richest-minus-poorest delivered bytes per second across the live nurseries whose load is known; how uneven the fleet is being allowed to be under the pressure line. 0 when fewer than 2 are known
`bzrk.meta.ingest_control_cas_lost`	counter	—	—	—	Planned moves whose owner CAS lost to a concurrent change; the coordinator replans from fresh state next tick
`bzrk.meta.ingest_control_grace_held`	counter	—	—	—	Planned moves suppressed by the startup grace: a live owner may just be slow to re-poll after a meta restart
`bzrk.meta.oldest_draining_stream_age_seconds`	gauge	same on every pod	—	—	Seconds since the oldest drain the ingester has reported beginning was marked. Rising means a writer cannot flush what it holds, or went away without the idle reaper sealing it.
`bzrk.meta.nursery_load_report_age_seconds`	gauge	worst pod	—	—	Seconds since the oldest live nursery's last load report. Rising means the loop is deciding on a stale picture even though the nurseries look alive
`bzrk.meta.ingest_completeness_lag_seconds`	gauge	same on every pod	s	—	Seconds between now and the cluster's query completeness watermark: the least W any stream group's owner reports, so every row ingested before then is queryable. Taken from the nurseries' own reports on each poll, from a group's current owner only. Zero while ingest_completeness_unknown is 1, which is the absence of a reading, not a healthy one
`bzrk.meta.ingest_completeness_unknown`	gauge	same on every pod	—	—	1 while the cluster's completeness watermark cannot be stated: a group with streams is unowned, its owner has not reported within the TTL, has applied no poll yet, or its newest report predates its ownership. The Ingest monitoring page names which. 0 when ingest_completeness_lag_seconds is a reading
`bzrk.meta.tombstoned_expired_rows`	gauge	same on every pod	—	—	Catalog rows in {databases, tables, views, ingest_tokens} with delete_at in the past — should drain each minute as the tombstone expiry sweep makes progress
`bzrk.meta.tombstoned_table_with_segments`	gauge	same on every pod	—	—	Tombstoned tables whose delete_at is past AND still have live segments after several expiry-sweep ticks. Non-zero means segment cascade is producing tombstones the janitor isn't draining, OR ingest_stream_tables cleanup is blocking the table's hard-delete
`bzrk.meta.chain_rebaselines`	counter	—	—	—	Segments written off by an automatic chain rebaseline — the chain believed them live but their catalog rows were confirmed gone across consecutive verification walks
`bzrk.meta.chain_adoptions`	counter	—	—	—	Catalog rows adopted by an automatic chain rebaseline — live segments the chain had no created event for, confirmed across consecutive verification walks. Adoption records the row without vouching for its provenance
`bzrk.meta.audit_shipping_lag_seconds`	gauge	same on every pod	s	—	Age in seconds of the oldest audit chain event the external archive does not yet hold, over every table; 0 when caught up or when nothing has ever been archived

Nursery

Ingestion service that receives OpenTelemetry data from the collector, converts it into segments, and manages segment merging for optimal query performance.

Metric	Type	Across pods	Unit	Labels	Description
`bzrk.nursery.streams_active`	up_down_counter	fleet total	—	—	Number of currently active stream followers
`bzrk.nursery.ingest_lag_seconds`	gauge	worst pod	s	—	How far behind the read positions are on the streams this nursery still owes reads on — the data-plane freshness signal. Deliberately excludes meta's poll bound and the watermark's skew margin, so a rise here means ingest, not coordination. A stream owed but never read counts from when this nursery took it on, so being stuck cannot read as being caught up. Zero means no stream is owed anything
`bzrk.nursery.completeness_watermark_lag_seconds`	gauge	worst pod	s	—	Seconds since the completeness watermark this nursery publishes. Includes meta's snapshot bound and the skew margin, because the watermark is capped by both — so unlike ingest_lag_seconds it never reads below meta's own delay. A nursery owning no stream group reports that bound alone, which is the most any nursery can vouch for. Zero only before the first applied poll, a one-RPC window in which the watermark is the conservative floor and nothing is capturable anyway
`bzrk.nursery.refresh_discards_consecutive`	gauge	worst pod	—	—	Meta refreshes discarded in a row since the last one installed. Ordinary staleness clears on its own: on one continuous primary a later snapshot always covers an earlier one, so non-coverage can only be a delayed in-flight response, and those stop arriving. A value that STAYS high has two causes, and the nursery log message says which — both log at ERROR once the run is sustained, so match on the message, not the level. (1) Meta's database moved backwards — a restore, a lagged failover, or a repointed DSN — logged as 'Discarding stale meta response'. The nursery deliberately does not try to recover: the cloud holds segments meta has forgotten, so there is no reconciliation to reach, and the fix is a clean-disk restart of the nurseries (replace the pods, which discards the working dir by design, rather than restarting the containers in place). (2) A covering body that would move a monotonic frontier backwards, logged as 'Discarding a refresh that would move a monotonic frontier backwards'. That is a broken co-snapshot invariant, not an operational event, and a restart will not fix it. Either way the nursery stops learning routing and its watermark freezes while this holds
`bzrk.nursery.download_duration_ms`	histogram	—	ms	—	S3 segment download duration
`bzrk.nursery.conversion_duration_ms`	histogram	—	ms	—	Protobuf to segment conversion duration
`bzrk.nursery.total_duration_ms`	histogram	—	ms	—	Total segment processing duration (download + conversion)
`bzrk.nursery.bytes_ingested`	counter	—	By	—	Total compressed bytes downloaded from S3 (use rate() for throughput)
`bzrk.nursery.bytes_ingested_uncompressed`	counter	—	By	—	Total uncompressed proto bytes ingested (use rate() for throughput)
`bzrk.nursery.segment_output_bytes`	counter	—	By	—	Total bytes of segment files produced (use rate() for throughput)
`bzrk.nursery.data_errors`	counter	—	—	—	Data errors (malformed protobuf, conversion failures)
`bzrk.nursery.infra_errors`	counter	—	—	—	Infrastructure errors (S3 failures, I/O errors)
`bzrk.nursery.active_streams`	gauge	fleet total	—	—	Number of active streams reported by Meta
`bzrk.nursery.closed_streams`	gauge	fleet total	—	—	Number of closed streams reported by Meta
`bzrk.nursery.stream_groups_released`	counter	—	—	—	Stream groups whose streams left this nursery's poll scope (ownership moved or streams deleted); local work is discarded
`bzrk.nursery.merge_count`	counter	—	—	—	Total number of completed merges
`bzrk.nursery.merge_inputs`	histogram	—	segments	—	Ingest segments consumed by one baby-segment merge. p50/p99 sizes the fan-in expected at the nursery merge span links.
`bzrk.nursery.merge_output_size`	histogram	—	By	—	Compressed output size of merged segments
`bzrk.nursery.merge_duration`	histogram	—	ms	—	Duration of segment merge operations
`bzrk.nursery.merge_speed`	histogram	—	By/s	—	Merge throughput, uncompressed input over elapsed time
`bzrk.nursery.merges_failed`	counter	—	—	`reason`: unreadable_input, panicked, other	Baby-segment merges that failed, by reason. unreadable_input: an input's bytes fail their checks or a row in it is malformed; the merge names the file, and the nursery rebuilds its (stream, table) from the stream (local_rebuilds), so a writer bug heals itself. panicked and other name no input, and the table backs off before its files are selected again. While a table's merges fail its rows do not reach table storage.
`bzrk.nursery.panics_caught`	counter	—	—	`site`: action, merge, combine, conversion, follower	Panics the nursery caught and turned into a failure, by where: an executor action, the blocking task of a merge, combine or conversion, or a follower. Every one is a bug; the panic hook's 'Thread panicked' line carries its message and location.
`bzrk.nursery.oldest_unmerged_data_age_seconds`	gauge	worst pod	s	—	Age of the oldest unmerged baby segment in seconds
`bzrk.nursery.local_backlog_segments`	gauge	fleet total	segments	—	Un-merged baby segments held locally across all streams — offsets above the merged frontier, i.e. outstanding merge work. Merged segments still on disk awaiting their retention grace are excluded. The count-driven fetch-backpressure signal — many small segments OOM a bounded pod via query-planning structures. See docs/dev/nursery-overload-backpressure.md.
`bzrk.nursery.local_backlog_bytes`	gauge	fleet total	By	—	Total bytes of un-merged baby segments held locally, excluding merged segments awaiting their retention grace. Secondary fetch-backpressure signal (the byte budget); segment count is primary. Total disk residency is `k8s.pod.filesystem.usage`, not this.
`bzrk.nursery.local_resident_segments`	gauge	fleet total	segments	—	All baby segments resident locally, merged or not — the input to the hard fetch-backpressure ceiling, which is an OOM guard on what the pod holds rather than on merge debt. Exceeds local_backlog_segments by the merged set awaiting its retention grace.
`bzrk.nursery.local_resident_bytes`	gauge	fleet total	By	—	Total bytes of all locally resident baby segments, merged or not — the byte axis of the hard fetch-backpressure ceiling. Not interchangeable with k8s.pod.filesystem.usage, which also counts merged outputs and other pod files.
`bzrk.nursery.fetch_paused`	gauge	one series per pod	—	—	1 while fetch backpressure is paused (inbound fetch stopped so the merge pipeline can drain), 0 otherwise. Sustained 1 means the nursery cannot keep up with ingest.
`bzrk.nursery.cpu_utilisation`	gauge	one series per pod	1	—	CPU time over the load window's worth of cores: 1.0 is every core busy throughout. The whole of meta's load tier — a nursery at or above coordinator_pressure_fraction (0.4) starts having its streams spread elsewhere. Under-reads on a contended machine with no cgroup quota, where the process gets less CPU than it wants and counts only what it got
`bzrk.nursery.follow_late_fraction`	gauge	one series per pod	1	—	The worst follower's share of fresh offsets finished more than 60s past their ingest time, over the last load window. A distress signal: at 0.5 meta treats the nursery as failing to keep up and hands one of its streams to another. A count rather than the deepest excursion, which one slow upload pins
`bzrk.nursery.oldest_unmerged_deadlines`	gauge	one series per pod	1	—	Age of the oldest UNCLAIMED merge work, in multiples of this nursery's own merge deadline: 1.0 is at the deadline, 1.5 is the distress line on two consecutive windows. Expect a ramp, not a level — where a table's pending backlog fits one merge, which is the normal case for the low-volume table that usually sets this, the merge claims all of it and the oldest unclaimed segment resets to a fresh arrival. Such a table reads 0 to 1 across its cycle rather than sitting at 1. A backlog past max_size_bytes leaves the remainder unclaimed, so the value stays high instead of resetting — which is the signal working. Dimensionless so meta holds no copy of the nursery's config
`bzrk.nursery.events_ingested`	counter	—	—	—	Total events ingested across all streams
`bzrk.nursery.forward_dated_events_clamped`	counter	—	—	—	Events whose OTLP timestamp exceeded the S3-derived ingest_time and was clamped on write. Rows are retained. A non-zero count can reflect object-store timestamp precision or source clock skew; it does not distinguish the two.
`bzrk.nursery.ingest_delay`	histogram	—	ms	—	Delay between event timestamp and ingest time
`bzrk.nursery.routing_unknown_table`	counter	—	—	—	Dropped segments where the routing key did not match any table in the token's database
`bzrk.nursery.oldest_routing_hold_age_seconds`	gauge	worst pod	s	—	Age of the longest-standing hold on an offset whose routing does not resolve. Excludes time spent under fetch backpressure, which stalls a hold for a reason that is not routing.
`bzrk.nursery.combine_count`	counter	—	—	—	Completed local combines, the analogue of merge_count. Zero while combining is disabled (combine_max_concurrent = 0); a rate that tracks ingest means the trigger is doing what it was set to.
`bzrk.nursery.combines_failed`	counter	—	—	—	Local combines the executor refused or could not write. Unlike a stalled merge, which shows up as local_backlog growth, a combine that stops achieves nothing visible — the files simply stay, which is indistinguishable from having nothing to combine. This is the only signal that says otherwise. Inputs that disagree on their vsearch artifacts are combined without them rather than refused, so any failure here deserves a look; the log line carries the error.
`bzrk.nursery.combine_input_bytes`	counter	—	By	—	Bytes read into local combines. Against segment_output_bytes this is the copy amplification combining costs, which is the signal that decides whether the trigger is set right — every byte is copied at most once, but v1 copies nearly all of them, so expect a ratio just under 1. Not against bytes_ingested, which is compressed bytes off S3 rather than the post-conversion local bytes a combine actually reads.
`bzrk.nursery.pending_combines`	gauge	fleet total	—	—	Combines dispatched and not yet reported. One is cleared only by its executor's terminal event, so a value stuck above zero means that event is never coming — the pair's claim is held for the life of the process and no merge can start for it.
`bzrk.nursery.local_rebuilds`	counter	—	—	—	(stream, table) pairs whose unreadable local file was dropped and is being fetched and converted again from the stream. Each follows a merges_failed or combines_failed naming an unreadable input; the same span failing again poisons the pair instead (poisoned_ranges).
`bzrk.nursery.quarantine_uploads_failed`	counter	—	—	—	Unreadable local files a rebuild could not keep a copy of, or whose copy or JSON sidecar did not reach quarantine/nursery/ in the tenant bucket. The rebuild goes ahead regardless; what is lost is the evidence of what the bad bytes were.
`bzrk.nursery.poisoned_ranges`	gauge	fleet total	—	—	(stream, table) pairs holding a local file that cannot be read and will not be rebuilt: it failed again after a rebuild, or its stream objects are gone. The pair keeps its files and stops merging and combining, so its rows do not reach table storage, its stream's floor stays put and queries needing it decline. A restart allows one more rebuild; killing the stream forfeits its unmerged rows.
`bzrk.nursery.vsearch_seal_artifacts_emitted`	counter	—	—	—	Segments whose seal wrote VCEN/VTPH/VTPC/VIDF. Increments by 1 per sealed segment that produced vsearch artifacts.
`bzrk.nursery.vsearch_seal_artifacts_skipped`	counter	—	—	—	Segments of a table with vector fields whose seal wrote no vsearch artifacts: no model, a model that failed to load, or an embedding failure. Increments by 1 per such segment; the rows still seal. A table without vector fields is not counted.
`bzrk.nursery.vsearch_embedding_cache_hits`	counter	—	—	—	Template-hash cache hits in the seal-time embedding cache. High hit rate = log data is template-clustered as expected.
`bzrk.nursery.vsearch_embedding_cache_misses`	counter	—	—	—	Template-hash cache misses — model.encode() invocations at seal time.
`bzrk.nursery.vidx_seal_segments`	counter	—	—	`vidx.format`: v1, legacy, none; `vidx.reason`: granted, no_grant, grant_expired, model_unavailable, legacy_failed, manifest_invalid	Sealed segments of tables with vector fields, by the vector index they carry: attribute vidx.format is v1 (the new format, written only under a write grant), legacy (VCEN/VTPH/VTPC, no grant) or none; vidx.reason is the grant state, or why it is not v1 (no_grant, grant_expired, model_unavailable, legacy_failed, manifest_invalid). A table without vector fields is not counted. The rows seal in every case.
`bzrk.nursery.vidx_seal_fields_unindexed`	counter	—	—	`vidx.reason`: version_mismatch, unknown_format, input_not_usable, embedding_failure, invariant, wire_limits, manifest_capacity, no_model_room	Vector fields a v1 seal wrote as not indexed, by vidx.reason (embedding_failure — including the model being unavailable —, invariant, wire_limits, manifest_capacity). Each is also logged. Persistent embedding_failure means the model or encoder is broken; the rewrite picks those segments up.
`bzrk.nursery.vidx_seal_templates_embedded`	counter	—	—	—	Templates a v1 seal embedded: one encode per distinct template of a vector field per segment, from its first row's text.
`bzrk.nursery.vidx_seal_index_bytes`	counter	—	By	—	Bytes of exact vector index v1 seals wrote, summed over their indexed fields: group and head chunks, framed. Read against vidx_seal_field_row_bytes to see what small segments of distinct short values cost (design open question 2).
`bzrk.nursery.vidx_seal_field_row_bytes`	counter	—	By	—	Compressed ROWS bytes attributed to indexed vector fields at v1 seals: each ROWS chunk's framed bytes split by the field's share of the chunk's uncompressed bytes.
`bzrk.nursery.vidx_seal_index_to_row_ratio`	histogram	—	—	—	Per v1 seal and indexed field: exact-index bytes over the field's attributed ROWS bytes. Well above 1 marks segments of distinct short values, whose one-kilobyte vectors outweigh their rows; whether seals should close at a distinct-template limit is decided from this.
`bzrk.nursery.vsearch_merger_artifacts_emitted`	counter	—	—	—	Merged segments where the merger rebuilt VCEN/VTPH/VTPC/VIDF. See docs/dev/vidx-vsearch-impl-plan.md PR 8.
`bzrk.nursery.vsearch_merger_pre_feature_inputs`	counter	—	—	—	Input segments to a merge that had no vsearch artifacts (pre-feature). Each increment indicates one input was skipped during VXXX rebuild.
`bzrk.nursery.vsearch_merger_unstamped_rows`	counter	—	—	—	Rows seen during merge that lacked a template_id.FIELD stamp. Persistent nonzero rate indicates ingest-side stamping isn't keeping up with merger fan-in.
`bzrk.nursery.vsearch_merger_template_index_bytes`	gauge	worst pod	By	—	Peak transient JanitorTemplateIndex memory during a merge (sum of input VTPC embedding tables). docs/dev/vidx-vsearch.md 8.2 caps the risk; this metric flags the cap binding before OOM.
`bzrk.nursery.vsearch_merger_duration_ms`	histogram	—	ms	—	Wall-clock added to a merge by VXXX rebuild (loading input VTPCs + tier selection + writing output VCEN/VTPH/VTPC/VIDF). Excludes the base ROWS-merger time.

Postgres

CNPG-managed PostgreSQL cluster (Hetzner dev). Metrics are scraped from the CNPG operator and per-instance pods by apps/otel-collector and renamed cnpg_* -> bzrk.postgres.* at scrape time so the alert framework's bzrk.{scope}.{metric} query template matches. There is no Rust source for this scope — only alert metadata.

Metric	Type	Across pods	Unit	Labels	Description
`bzrk.postgres.pg_replication_lag`	gauge	same on every pod	seconds	—	Each instance's replication lag behind its primary in seconds, reported by that instance's CNPG exporter. Primaries report zero.
`bzrk.postgres.collector_pg_wal_archive_status`	gauge	worst pod	—	—	CNPG gauge counting WAL files per archive state (`done`, `ready`) on each instance. A growing `ready` count is barman-cloud-wal-archive falling behind or failing outright — most often an S3 credential / endpoint / encryption misconfiguration (see GH #2585).
`bzrk.postgres.collector_backups_total`	counter	—	—	—	CNPG counter for completed backups, labelled by `phase`. Filter to `phase=completed` to count successful base backups; rate_below catches missed daily backups.
`bzrk.postgres.backends_total`	gauge	same on every pod	—	—	Currently connected client backends, partitioned by database, user, application, and state. Sum partitions per pod and scrape before comparing against capacity. `max_connections=200` (postgresql.parameters in k8s-resources.tf); alert at 160 to leave a 20% headroom for spike traffic and CNPG's own internal connections.

Query

Query execution service that receives KQL queries over HTTP and gRPC, plans and executes them against segments, and streams results back to clients.

Metric	Type	Across pods	Unit	Labels	Description
`bzrk.query.execution_duration`	histogram	—	ms	—	End-to-end query execution duration
`bzrk.query.requests`	counter	—	—	—	Total query requests received
`bzrk.query.in_flight`	up_down_counter	fleet total	—	—	Queries currently executing in the query process (incremented at request entry, decremented when the response stream completes). The saturation / admission signal: under load this is the queue-depth proxy that traffic (requests) and latency (execution_duration) can't show on their own. Recorded only by the parent query process (qc-child is not a writer).
`bzrk.query.result_rows`	histogram	—	—	—	Number of rows returned per query
`bzrk.query.warnings`	counter	—	—	—	Queries that showed the caller at least one execution warning, counted once per query however many it raised. Mixed by design, so a starting point rather than a page: most kinds are deterministic properties of the query — a memory budget shedding groups, a widened bin span — and repeat on every run against the same data, while `RowEvaluationError` is the system failing to evaluate data it holds. A chunk storage could not read is not a warning but a partial failure. Break down with the `query.warning` log record, one per kind: default \| where body == 'query.warning' \| summarize queries = count(), affected = sum(toreal(attributes['count'])) by kind = tostring(attributes['kind'])
`bzrk.query.partial_failures`	counter	—	—	—	Queries whose result was missing a unit of work it needed — a nursery that could not serve a stream group, a worker that went away, a segment that could not be opened, a chunk storage could not read. Counted once per query, not once per failure. Transient where a warning usually is not: the same query run again once the owner is back returns the missing rows. The `query.partial_failure` log record carries which ones.
`bzrk.query.errors`	counter	—	—	—	Total query errors by error type
`bzrk.query.open_fds`	gauge	worst pod	—	—	`bzrk_lib::count_open_fds()` periodic sample (10s interval). Pair with `bzrk.query.fd_limit` to compute `open_fds / fd_limit` on dashboards/alerts without joining against startup logs. apps/query in cache_mode=remote holds a UDS connection per worker task plus the SCM_RIGHTS cache_fd + shm_fd passed by cache_server, so the count tracks engine concurrency directly. Symmetric with `bzrk.cache_server.open_fds`.
`bzrk.query.fd_limit`	gauge	same on every pod	—	—	Current `RLIMIT_NOFILE` soft cap. Companion to `open_fds` — sampled on the same 10s tick so dashboards can show "fds: N / LIMIT (X%)" and alerts can fire on `open_fds / fd_limit > 0.8` before saturation. Production binaries raise the soft limit to the hard cap at startup, so this is effectively static; emitting it as a gauge keeps the query simple.
`bzrk.query.routing_decisions`	counter	—	—	`mode`: sticky, fallback_walk, fallback_round_robin, no_target_round_robin, pool_empty	Sessions opened by `PoolBackedQwsTransport`, attributed by the routing decision taken (`mode`): - `sticky` — QC supplied a target_node_id and the live member was found in the pool snapshot. Ring routing held end-to-end. - `fallback_walk` — target_node_id supplied, but the targeted member was gone from the live snapshot. Fell back to the partition ring's next-priority node for the batch's first segment. - `fallback_round_robin` — target_node_id supplied, member was gone, and the priority walk failed too (snapshot didn't cover the segment). Degraded to round-robin: NOT ring-aware. A non-zero rate here is the signature of seed-vs-read disagreement surviving the ring's protections. - `no_target_round_robin` — QC didn't supply a target_node_id at all (no partition snapshot at coordinator time → bootstrap / in-process / non-sticky). NOT ring-aware. Sustained non-zero on a fully-bootstrapped cluster means the coordinator never saw a snapshot. - `pool_empty` — no QwsCloud members visible at session-open. Session will fail on first send; pool likely re-populating after a pod rollout. The counter exists to spot the two "NOT ring-aware" rows showing up at non-trivial rates — the failure mode that would let a query land on a non-owner pod and cold-fetch a freshly-seeded segment.
`bzrk.query.vsearch_queries`	counter	—	—	—	vsearch queries handled by the coordinator (one increment per query containing a vsearch operator).
`bzrk.query.vsearch_query_latency_ms`	histogram	—	ms	—	End-to-end vsearch query latency (coordinator encode + worker scatter/gather + reducer merge).
`bzrk.query.qc_warm_pool_ready`	up_down_counter	fleet total	—	—	Ready, idle coordinator children in the warm buffer. Recorded only by the parent query process; zero means the buffer is empty or pooling is disabled.

Query.cache

Metric	Type	Across pods	Unit	Labels	Description
`bzrk.query.cache.bytes_total`	counter	—	By	—	Uncompressed footprint (scanned + filter-skipped) of all slices a query would touch — reused plus recomputed
`bzrk.query.cache.bytes_saved`	counter	—	By	—	Uncompressed footprint of slices served from the cache — the work avoided this run
`bzrk.query.cache.scan_nanos_total`	counter	—	ns	—	Chunk-scan time of all slices (reused plus recomputed)
`bzrk.query.cache.scan_nanos_saved`	counter	—	ns	—	Chunk-scan time of slices served from cache — latency avoided
`bzrk.query.cache.lookups`	counter	—	—	`outcome`: hit, miss, skip; `reason`: none, no_state_cache, bypass_requested, coordinator_driven, multi_phase, prohibiting_fn, short_circuit_reducer, no_time_bin	Cache lookups by outcome: hit / miss / skip; skip carries a reason attribute (short_circuit_reducer, prohibiting_fn, no_time_bin, ...)
`bzrk.query.cache.saved_pct`	histogram	—	%	—	Per-query saved-byte fraction (0-100) over cache-eligible queries; cold runs land at 0. Distribution of reuse, not the volume-weighted aggregate
`bzrk.query.cache.resident_bytes`	gauge	one series per pod	By	—	On-disk size of resident cache entries
`bzrk.query.cache.entries`	gauge	one series per pod	—	—	Resident cache entry count. Growth per query is the capture-churn signal: entries that never supersede pile up one generation per refresh
`bzrk.query.cache.capacity_bytes`	gauge	one series per pod	By	—	Configured size cap; static for the process lifetime
`bzrk.query.cache.eviction_frontier_age_seconds`	gauge	one series per pod	s	—	Age of the oldest resident entry's last use — the LRU eviction frontier. Low under size pressure means recently-used entries are being evicted (cache disk too small); high means headroom
`bzrk.query.cache.evictions_total`	counter	—	—	—	Entries evicted since process start (age + size-cap + self-heal). Its rate is the eviction-pressure signal
`bzrk.query.cache.peer.ops`	counter	—	—	`op`: lookup, fetch, fetch_dict, store; `outcome`: local, local_miss, local_miss_settled, remote_hit, remote_miss, no_endpoint, rejected_non_owner, error, timeout	Ring cache operations by op and outcome. Lookup fans out to every owner plus the local index, so one query emits several: the ratio of remote_hit to local tells you whether sharing is doing the work, and local_miss_settled counts the fan-outs skipped because this node has owned the partition long enough for its miss to be authoritative. Without this a ring answering everything locally is indistinguishable from one that is sharing
`bzrk.query.cache.peer.rpc_duration_ms`	histogram	—	ms	`op`: lookup, fetch, fetch_dict, store	Wall time of one ring RPC, by op. Lookup sits on the query critical path
`bzrk.query.cache.peer.bytes`	counter	—	By	`direction`: in, out	Bytes moved between pods by the ring: out (states replicated to owners), in (states fetched from an owner). The cost that scales with replica count

Qws

Query-worker supervisor process. Hosts the QueryWorker gRPC service, owns the cache_server child, and drives engine execution for the segments routed to this node.

Metric	Type	Across pods	Unit	Labels	Description
`bzrk.qws.cache_objects_resident`	gauge	fleet total	—	—	Objects held in this node's segment cache, counted by the boot-time reconciliation sweep. Measured once per process, so it is a starting point rather than a live level — cache eviction is driven by disk pressure, so an under-full cache accumulates files for long-dead segments, and this is the count that reveals it.
`bzrk.qws.cache_dead_objects_evicted`	counter	—	—	—	Cached objects invalidated because their segment is gone: a delete pushed by the janitor and applied after its delay, or one the boot-time sweep found meta no longer reports as worth caching. Files reclaimed that pressure-driven eviction would never have touched.
`bzrk.qws.cache_reap_sweep_failures`	counter	—	—	—	Boot-time reconciliation sweeps that aborted on an error (cache listing or the meta cacheability call). The sweep fails closed — nothing is evicted — so a node that trips this keeps whatever it missed while down until its next restart.
`bzrk.qws.seed_receive_outcomes`	counter	—	—	`outcome`: accepted, rejected_non_owner, error	Receiver-side outcomes of merge-side cache seed pushes, by outcome (accepted \| rejected_non_owner \| error). `rejected_non_owner` is the ring re-check rejecting a stale producer's push, which surfaces ring drift on the receiver side without segment-level logging.

Ui

Phoenix LiveView frontend. The BEAM runtime metrics below say which part of the runtime holds memory, which the OS/cgroup view (metrics/base.yaml, metrics/container.yaml) cannot: the difference between 'the ui used 900 MB' and '900 MB of it is process heaps, and one mailbox is 400 MB of that'. There is no Rust source for this scope — the Elixir mode of metrics-gen renders the catalogue this app's meter and its operator /metrics exposition both read.

Metric	Type	Across pods	Unit	Labels	Description
`bzrk.ui.beam.memory.current.total_bytes`	gauge	fleet total	By	—	Bytes allocated by the BEAM in total — the runtime's view of the number cgroup accounting kills on.
`bzrk.ui.beam.memory.current.processes_bytes`	gauge	fleet total	By	—	Bytes held by process heaps and mailboxes. A blocked consumer with a filling mailbox shows up here first.
`bzrk.ui.beam.memory.current.binary_bytes`	gauge	fleet total	By	—	Bytes held by refc binaries — large result cells, gzip blobs, protobuf payloads.
`bzrk.ui.beam.memory.current.ets_bytes`	gauge	fleet total	By	—	Bytes held by ETS tables — caches, the operator-metrics window, registries.
`bzrk.ui.beam.process.current.count`	gauge	fleet total	{process}	—	Live processes. One per LiveView, query stream, and task.
`bzrk.ui.beam.memory.peak.total_bytes`	gauge	worst pod	By	—	Trailing-window peak of the BEAM's total allocation, sampled faster than the export interval.
`bzrk.ui.beam.memory.peak.processes_bytes`	gauge	worst pod	By	—	Trailing-window peak of bytes held by process heaps and mailboxes.
`bzrk.ui.beam.memory.peak.binary_bytes`	gauge	worst pod	By	—	Trailing-window peak of bytes held by refc binaries.
`bzrk.ui.beam.memory.peak.ets_bytes`	gauge	worst pod	By	—	Trailing-window peak of bytes held by ETS tables.
`bzrk.ui.beam.process.peak.count`	gauge	worst pod	{process}	—	Trailing-window peak of the live process count.
`bzrk.ui.workflow.scheduler.enqueued`	counter	—	—	—	Scheduled workflow runs inserted by the ticker.
`bzrk.ui.workflow.scheduler.errors`	counter	—	—	—	Due schedules the ticker failed to enqueue or advance.
`bzrk.ui.workflow.scheduler.identity_revoked`	counter	—	—	—	Scheduled fires skipped because the owner is gone, disabled, or the gateway was unreachable.
`bzrk.ui.workflow.query.pushdown.evaluations`	counter	—	—	—	Merged query-condition evaluations, labelled by terminal outcome (decided_early, completed, error).
`bzrk.ui.workflow.query.early_exits`	counter	—	—	—	Workflow queries cancelled by an absorbing progressive condition.
`bzrk.ui.workflow.query.early_exit.coverage`	histogram	—	1	—	Scan coverage (0-1) at which an absorbing condition cancelled the query.
`bzrk.ui.workflow.query.early_exit.iterations`	histogram	—	{iteration}	—	Complete result iterations seen before an absorbing condition cancelled the query.
`bzrk.ui.workflow.scheduler.lag_seconds`	gauge	worst pod	s	—	Age of the oldest due next_fire_at among enabled schedules (0 when none are overdue).
`bzrk.ui.workflow.scheduler.skipped_overlap`	counter	—	—	—	Scheduled fires skipped because the workflow still had an active run (queued, running, retrying, or waiting).
`bzrk.ui.workflow.recovery.sweeps`	counter	—	—	—	Completed reconciler sweeps; one per reconcile_interval_ms (10s) per ui pod while recovery is healthy.
`bzrk.ui.workflow.recovery.overdue`	gauge	worst pod	{run}	—	Runs needing recovery when a sweep started: expired lease or deadline, ownerless active rows, expired queue waits, due retries. Sustained nonzero means recovery is not converging.
`bzrk.ui.workflow.recovery.outcomes`	counter	—	—	—	Reconciler transitions, labelled by outcome: execution_lost, deadline_exceeded, queue_timeout, retry_scheduled, retry_dispatched, dispatch_repaired, dispatch_failed, dispatched, error.
`bzrk.ui.workflow.delivery.outcomes`	counter	—	—	—	Alert notification delivery attempts, labelled by status: sent, unknown (may have reached the receiver, never replayed), failed, retry, skipped.
`bzrk.ui.workflow.attempt.duration`	histogram	—	ms	—	Workflow attempt operation duration, labelled by bounded status.
`bzrk.ui.workflow.delivery.duration`	histogram	—	ms	—	Workflow delivery operation duration, labelled by bounded status.
`bzrk.ui.workflow.recovery.duration`	histogram	—	ms	—	Workflow recovery operation duration, labelled by bounded status.
`bzrk.ui.workflow.cleanup.duration`	histogram	—	ms	—	Workflow cleanup operation duration, labelled by bounded status.
`bzrk.ui.workflow.attempt.outcomes`	counter	—	—	—	Workflow attempt outcomes, labelled by bounded status.
`bzrk.ui.workflow.recovery.errors`	counter	—	—	—	Recovery operation errors, labelled by bounded phase.
`bzrk.ui.workflow.recovery.deferred`	counter	—	—	—	Recovery candidates deferred by contention, capacity or sweep budget, labelled by bounded reason.
`bzrk.ui.workflow.cleanup.outcomes`	counter	—	—	—	Durable job cancellation outcomes, labelled by bounded status.
`bzrk.ui.workflow.cleanup.pending`	gauge	worst pod	{job}	—	Durable transport cancellation candidates seen in the bounded cleanup batch (capped at 100).
`bzrk.ui.source_control.operations`	counter	—	—	—	Native source-control operations by provider and operation.
`bzrk.ui.source_control.failures`	counter	—	—	—	Failed native source-control operations; never labeled with repository or credential data.
`bzrk.ui.source_control.operation.duration_seconds`	histogram	—	s	—	Native source-control API operation duration.
`bzrk.ui.source_control.webhook.deliveries`	counter	—	—	—	Native provider webhook delivery outcomes.
`bzrk.ui.source_control.ci.exports`	counter	—	—	—	Customer CI telemetry export outcomes.
`bzrk.ui.query.response_bytes`	counter	—	By	—	Wire bytes of query-service responses, summed before protobuf decode. This is what crossed the socket, including gRPC framing and any transport compression, which a measurement taken after decode cannot see. Against the query count it gives the size of a typical result; against time it gives the read volume one UI pod pulls out of the query service.
`bzrk.ui.cache.lookups`	counter	—	{lookup}	—	Cache checks by cache, kind and outcome (hit, miss, error). Decoding after a miss does not count as a hit.
`bzrk.ui.cache.bytes`	gauge	fleet total	By	—	Accounted resident cache bytes. Prepared visualization bytes are a subset of disk bytes; decoded bytes are the admitted term weight, not total BEAM memory.
`bzrk.ui.cache.entries`	gauge	fleet total	{entry}	—	Resident entries by cache. Disk includes result generations and stored result blobs.
`bzrk.ui.cache.capacity_bytes`	gauge	fleet total	By	—	Configured admission budget for decoded memory or the shared disk store.
`bzrk.ui.cache.reserved_bytes`	gauge	fleet total	By	—	Bytes reserved for active disk writers and spools.
`bzrk.ui.cache.pending_bytes`	gauge	fleet total	By	—	Disk bytes pending deletion, still charged against capacity.

Log

What every Berserk process writes down about its own logging, shipped over OTLP by all of them and queryable from Berserk.

One counter, deliberately generic. Individual faults — a stream whose object store is unreachable, a wedged merge, a misconfigured bucket — each already say so at error!, but each would otherwise need a metric of its own before anyone could notice without a shell on the box. That is the gap this closes: an operator who does not know what to look for can still see that something is wrong, and the logs then say what.

Metric	Type	Across pods	Unit	Labels	Description
`bzrk.log.records`	counter	—	—	`level`: warn, error	Log records this process emitted, by `level`. Only `warn` and `error` are counted: `info` and below are volume rather than signal, and counting them would make the value depend on each install's `log_level`, so it would stop meaning the same thing across installs. Measures the records the process was configured to see — the global log filter sits above the counting layer, so a target an operator has explicitly silenced is not counted.

S3

Object-store signals under s3.*, shipped over OTLP by every service that reads or writes segments, and queryable from Berserk.

The same signals are also published under bzrk.s3.* on each pod's Prometheus endpoint. That tier is scraped rather than ingested, so it is not queryable from Berserk — alerts name the s3.* instruments here.

Metric	Type	Across pods	Unit	Labels	Description
`bzrk.s3.s3.connect.failures`	counter	—	—	—	Candidate addresses that failed to yield a connection, labelled by `family` and `reason`. Counted per address, so this is not a failed-operation count: establishment walks the addresses the endpoint publishes and a later one usually answers.

Engine

Query-execution metrics under bzrk.engine.*, emitted by the query service while it runs a query.

They decompose where a query's time and chunk work actually went — planning, fetching, scanning, waiting — so a slow or expensive query can be attributed to a phase rather than guessed at. Chunk outcomes additionally show how much data the pre-filters skipped before any rows were read.

These carry no alerts of their own: they are triage signal. The page for user-visible query failure is bzrk.query.errors.

Metric	Type	Across pods	Unit	Labels	Description
`bzrk.engine.chunk_outcomes_total`	counter	—	—	`phase`: plan, exec; `outcome`: scanned, skipped_range, skipped_bloom, skipped_shar, skipped_field_types, skipped_required_fields, empty_scan, errored	Total chunks observed by the engine, attributed by `phase` (`plan` for pre-filter / chunk-skip during planning; `exec` for per-row scan during execution) and `outcome` (`scanned`, `skipped_range`, `skipped_bloom`, `skipped_shar`, `skipped_field_types`, `skipped_required_fields`, `empty_scan`, `errored`). Ratios decompose what dominated a query: e.g. `outcome=skipped_bloom / outcome=scanned` is the bloom prune rate; `outcome=empty_scan / outcome=scanned` flags false-positive bloom matches.
`bzrk.engine.worker_cancellations_total`	counter	—	—	`reason`: coordinator_gone	Worker sessions told to stop scheduling new work, by `reason`. `coordinator_gone` is the directive channel closing: the QC holds that channel open for the query's lifetime, so a close means the coordinator went away — client disconnect, QC crash, or a transport reset surfaced as a stream close. Cancellation is cooperative: the token is flipped and the executor's checkpoints decline the next unit of work, so work already in flight finishes. This counter is how you tell that the path still fires at all — before it existed, a cancellation path that silently stopped working looked exactly like one that was never exercised.
`bzrk.engine.short_circuit_triggered_total`	counter	—	—	`reducer`: take, tail, top, summarize, getschema, fieldstats, fieldtypes, otellogstats, joinhashfilter, joinhashfilter_binned, joinside, tracefind, tracefind_discovery, tophitters, toptemplates, partition, reduce	Count of times a branch first transitioned to short-circuited (`take`, `top`, `tail` satisfied). Attributed by `reducer` (closed enum from `ReduceFunction::metric_kind()`). Deduped to first state transition — multiple racing threads can hit the short-circuit site for the same branch, but this counter increments once per branch.
`bzrk.engine.deferred_segments_planned_total`	counter	—	—	—	Count of deferred-segment plans completed. Pair with `chunk_outcomes_total{phase=plan}` to attribute planning-phase chunk activity to deferred work.
`bzrk.engine.worker_processing_duration_ms`	histogram	—	ms	—	Per-thread wall time spent in the `Chunks` arm of the worker work-batch (actual scan loop). Recorded once per worker thread at thread exit from `ExecutionStats::worker_processing_time_nanos`.
`bzrk.engine.worker_fetch_duration_ms`	histogram	—	ms	—	Per-thread wall time spent in `reader.fetch_chunk_data().await` (cache_server fetch path) summed across chunks. Recorded once per worker thread at thread exit from `ExecutionStats::worker_fetching_chunk_nanos`. Jumps if the cache layer regresses.
`bzrk.engine.worker_chunk_body_duration_ms`	histogram	—	ms	—	Per-thread wall time inside the synchronous chunk-body closure (decompress + per-row scan + branch accounting). Recorded once per worker thread at thread exit from `ExecutionStats::worker_chunk_body_nanos`. Pair with `worker_fetch_duration_ms`: their sum should approximate `worker_processing_duration_ms` modulo scaffolding overhead.
`bzrk.engine.worker_waiting_duration_ms`	histogram	—	ms	—	Per-thread wall time the worker spent parked waiting for deferred-planning work (sum of throttle-wait and queue-empty-wait nanos). Recorded once per worker thread at thread exit. Sustained non-zero indicates the deferred-planner pipeline is the bottleneck, not chunk scanning.
`bzrk.engine.worker_compute_slot_duration_ms`	histogram	—	ms	—	Per-thread wall time blocked on the compute-slot semaphore before the chunk-body closure runs. Recorded once per worker thread at thread exit. Sustained non-zero means threads are overcommitted against compute slots, which neither `worker_fetch_duration_ms` nor `worker_chunk_body_duration_ms` can show — both measure work that has already been admitted.
`bzrk.engine.blom_lumps_processed_total`	counter	—	—	—	BLOM lumps processed by worker threads, added once per thread at exit. Ratio against `chunk_outcomes_total{phase=plan}` is whether bloom work still arrives batched: a ratio approaching one lump per chunk means the lumping degenerated and each chunk pays its own scheduling round.
`bzrk.engine.worker_wait_wakeup_timeouts_total`	counter	—	—	—	Bounded `WaitForDeferred` waits that expired on their 100 ms timeout instead of being woken. `notify_waiters` does not buffer, so a producer that notifies before the waiter registers drops the wake; the timeout turns that hang into a latency penalty. Non-zero means the race is live and should be root-caused — the timeout is the backstop, not the mechanism.
`bzrk.engine.summarize_state_bytes`	histogram	—	By	—	Per-operator `summarize` intra-group aggregate state size on the final merged state, recorded once per operator per query at the coordinator's terminal delivery (`is_done`) — never per row, and gated to the final result so the engine's incremental re-finalization doesn't multi-count. Compare the distribution against the per-operator budget (`BZRK_AGG_BUDGET_DEFAULT_MIB`, default 128 MiB) to see how close queries run to the memory ceiling — the primary load-safety signal, and the input for tuning the default. The same final size is also recorded as the `summarize_state_bytes` attribute on the `coordinator.stream` span, so a heavy aggregation is visible per-query in the trace as well as in aggregate.
`bzrk.engine.summarize_budget_events_total`	counter	—	—	`outcome`: sampled, failed; `strategy`: sample, heaviest	Times a `summarize` operator hit its memory budget, attributed by `outcome` and `strategy`: - `outcome=sampled` — the operator stayed under budget by dropping whole series (graceful degradation); `strategy` is the active `hint.sample` (`sample` / `heaviest`). - `outcome=failed` — the operator had no series axis to sample (e.g. `by bin(timestamp,1h)` alone) and failed the query loud. A rising `failed` rate means budgets are too tight (or queries too heavy) for the offered load — the signal to raise `BZRK_AGG_BUDGET_DEFAULT_MIB` or narrow the queries. A rising `sampled` rate means results are increasingly partial.
`bzrk.engine.vsearch_segment_ctx_builds_total`	counter	—	—	`outcome`: indexed, unindexed	Per-segment vsearch scoring contexts built, by `outcome`: `indexed` when the segment carries VCEN+VTPH for the request's field, `unindexed` when it does not (pre-feature segment, or the field is not indexed), so its rows score 0. Once per (segment, request) per query. A rising `unindexed` share means `similarto` / `vector-search` is scanning segments that cannot answer it.
`bzrk.engine.vsearch_precompute_duration_us`	histogram	—	us	—	Time to build one context's template score tables (`SegmentPrecompute::build`), once per (segment, request) per query. docs/dev/vidx-vsearch.md 5.6 estimates ~10-30 us at defaults.
`bzrk.engine.vsearch_lineage_broken_total`	counter	—	—	—	`similarto` calls bound with an upstream `parse` / `mv-expand` that may rewrite the `on=` value, so their rows score 0. Once per call at bind time, not per row.

QueryCoord

Query admission-queue metrics under bzrk.query_coord.*, emitted by the query service.

bzrk.query.in_flight counts waiting and running queries together, so a saturated pod and a concurrency limit that is not holding look identical on it. These two split that apart: one counts queries waiting for a slot, the other counts queries actually executing.

Metric	Type	Across pods	Unit	Labels	Description
`bzrk.query_coord.admission_queued`	up_down_counter	fleet total	—	—	Queries waiting for an execution slot. Non-zero means `max_concurrent_queries` is binding.
`bzrk.query_coord.admission_executing`	up_down_counter	fleet total	—	—	Queries holding an execution slot. Must never exceed the configured `max_concurrent_queries` — above it, the limit is not holding, which is otherwise indistinguishable from a deep queue. Recorded by the coordinator in the parent process; qc-child and qw are not writers, so a crashed worker cannot leak a permanent +1.

QwSpawn

Query-worker admission and startup, under bzrk.qw_spawn.*. Emitted by whichever long-lived process owns the spawn factory — the qws in a query pod, the embedded qws in a nursery pod — never by a spawned worker, which does not outlive its own query.

Metric	Type	Across pods	Unit	Labels	Description
`bzrk.qw_spawn.slots_in_use`	up_down_counter	fleet total	—	—	Query workers running on this node, against `slots_total`. The slot count is the first ceiling a burst of queries meets: coordinators admit far more than any one node can serve workers for, deliberately, so that a long query cannot block new ones from starting — and a spike then queues here. Sustained saturation is the signal to add nodes or memory. On the nursery lane that means more nursery replicas or a bigger pod, since slots are the pod's memory limit over `NURSERY_BYTES_PER_QW_SLOT`.
`bzrk.qw_spawn.slots_total`	gauge	fleet total	—	—	Query-worker slots this node will run at once, derived at startup from its memory limit. The denominator for `slots_in_use`, reported because the numerator alone cannot say whether a node is full, and because the value moves when a pod is resized.
`bzrk.qw_spawn.warm_pool_ready`	up_down_counter	fleet total	—	—	Booted query workers waiting for a single-pass session. A session that finds the pool empty forks its worker on the query's critical path, so a pool that sits at zero while queries arrive is smaller than the rate it serves. An idle member holds no slot, only its memory.
`bzrk.qw_spawn.slot_wait_us`	histogram	—	us	—	How long a query waited for a worker slot. The leading indicator for saturation: it rises while queries still succeed, whereas end-to-end latency only moves once the queue is deep, and the nursery lane's open deadline turns the wait into a partial result after that. Measured on dev, a 500-query burst pushed server-side p50 from 4.4s to 12.8s as queueing alone, with nothing recording why.
`bzrk.qw_spawn.startup_us`	histogram	—	us	—	Successful worker startup from slot acquisition through process launch. The worker's own start, until it serves, is spent by the session's first request, which waits in its socket. Excludes admission wait, measured separately by slot_wait_us, and failed or cancelled startups.

SegmentCache

Segment-cache metrics under bzrk.segment_cache.*, emitted by the query service. The segment cache is what keeps recently-read segment data local instead of re-fetching it from object storage.

They decompose a slow or failing query into its storage-side causes: which fetch outcomes dominated, whether fetch slots are saturated, and whether fetch latency is pinned at its ceiling.

These carry no alerts of their own: they are triage signal. The page for user-visible query failure is bzrk.query.errors.

Metric	Type	Across pods	Unit	Labels	Description
`bzrk.segment_cache.s3_fetches_started`	counter	—	—	`mode`: first_block, streaming	Total S3 fetches initiated by the coordinator, attributed by mode (first_block \| streaming). Use `otel_rate()` for per-second activity.
`bzrk.segment_cache.range_fetch_retries`	counter	—	—	—	Block-range fetches re-dispatched after a retriable backend failure. Read against `range_fetch_giveups`: retries without giveups is the backend being flaky and the cache absorbing it, which is the point. Read against `s3_fetch_outcomes{outcome=timeout}` to see what share of timeouts the retry is covering.
`bzrk.segment_cache.range_fetch_giveups`	counter	—	—	—	Block-range fetches with at least one block that exhausted its retry budget, counted once per fetch like `range_fetch_retries`. A reader still waiting on such a block sees a `ChunkScanError` warning on a partial result, so this is the closest signal to user-visible storage loss; a reader that already left sees nothing.
`bzrk.segment_cache.s3_fetch_outcomes`	counter	—	—	`outcome`: success, error, timeout, not_found, throttled, write_error, abandoned; `mode`: first_block, streaming	S3 fetches completed, attributed by mode and terminal outcome (success \| error \| timeout \| not_found \| throttled \| write_error \| abandoned). When query.errors is firing, break this down by `attributes.outcome` to see which outcome class dominates. `abandoned` is a fetch dropped because every reader had gone before it finished (#4145) — a saving, not a fault, and deliberately absent from `bzrk.s3.failures_total`.
`bzrk.segment_cache.s3_fetch_bytes_total`	counter	—	By	`mode`: first_block, streaming	Cumulative bytes fetched from S3 by the coordinator
`bzrk.segment_cache.s3_fetch_duration_ms`	histogram	—	ms	`mode`: first_block, streaming; `outcome`: success, error, timeout, not_found, throttled, write_error, abandoned	Time between S3 fetch start and completion/error, attributed by mode and outcome. The p95 on the `first_block` mode is what callers block on for cold reads — filter via `attributes.mode == 'first_block'` and `attributes.outcome == 'success'` when investigating latency.
`bzrk.segment_cache.s3_fetches_in_flight`	up_down_counter	fleet total	—	—	Current number of S3 fetches holding a coordinator slot, summed over the process's coordinators (one per bucket). Max is `SegmentCache` config `max_s3_fetches` (8 in prod) per coordinator. Sustained ceiling + non-zero pending queue depth is the slot-starvation signal.
`bzrk.segment_cache.global_s3_permits_in_use`	gauge	fleet total	—	—	Process-wide S3 fetch permits currently held, across all coordinators. Ceiling is `CacheConfig::max_total_concurrent_s3_requests` (32 in prod). Sustained ceiling + rising `global_s3_permit_wait_ms` means the global cap is binding (see #3249 for the calibration playbook).
`bzrk.segment_cache.global_s3_permit_wait_ms`	histogram	—	ms	`outcome`: acquired, abandoned	Time a fetch task spent blocked acquiring a process-wide S3 permit, attributed by outcome (acquired \| abandoned). Queueing here is invisible to `s3_fetch_duration_ms`, whose timer starts after the permit is acquired — this histogram is the "ceiling too low" signal. `abandoned` samples are waits cancelled by the fetch deadline, i.e. permit-starvation events; recorded via a drop guard so deadline cancellation cannot censor the longest waits.
`bzrk.segment_cache.pending_first_blocks_depth`	up_down_counter	fleet total	—	—	Number of first-block fetches queued because all coordinator slots were in use, summed over the process's coordinators. A sustained non-zero value plus saturated slots is the backpressure-into-the-caller signal.
`bzrk.segment_cache.space_used_bytes`	gauge	fleet total	—	—	Bytes the space tracker believes the cache holds on disk. The tracker accumulates from reservations, so it can only ever drift upward; each eviction cycle reconciles it against a sparse-aware scan of the cache directory. Approaching `space_capacity_bytes` while eviction frees nothing is the shape of a cache that has stopped admitting data.
`bzrk.segment_cache.space_reserved_bytes`	gauge	fleet total	—	—	Bytes reserved for in-flight fetches and seeds. Counted against capacity alongside `space_used_bytes`, so a large single reservation can refuse admission on its own.
`bzrk.segment_cache.space_capacity_bytes`	gauge	fleet total	—	—	Configured cache capacity. Present so used/reserved can be read as a fraction without hardcoding the deployment's size in a panel.
`bzrk.segment_cache.abandoned_waiters_dropped`	counter	—	—	—	Queued first-block waiters dropped because the caller had already given up before the fetch was dispatched. A client going away closes its reply port silently, so nothing else tells the coordinator. Sustained non-zero means requests are timing out before they reach S3 at all.
`bzrk.segment_cache.space_reserve_denied`	counter	—	—	`purpose`: first_block, coalesce, seed	Reservations refused for want of cache space, by what the space was for. Sustained non-zero means fetches are not reaching S3 at all: callers queue and time out while the cache serves only what it already holds. Counted rather than logged — the dispatch retry runs at the rate limiter's rate, so a log line here emits thousands per second.
`bzrk.segment_cache.reply_port_timeouts`	counter	—	—	`site`: get_or_create_object, first_block_waiter, coordinator_rpc	Coordinator reply-port sends that failed because the caller had already timed out. This is the smoking gun for the Apr-22 incident class — the coordinator's answer arrives after the 30s ractor `call_t!` deadline and the receiver has been dropped. Attribute `site` (get_or_create_object \| first_block_waiter \| coordinator_rpc) tells you which coordinator message class dropped.
`bzrk.segment_cache.range_lock_acquire_duration_ms`	histogram	—	ms	`kind`: read, write; `outcome`: ok, poisoned, error	Time spent acquiring a per-block range lock from entry of `read_blocks` / `write_blocks_for_eviction` to the returned guard. Attributed by `kind` (read \| write) and `outcome` (ok \| poisoned \| error). Reader p99 is the warm-cache fast-path lock-acquire tail latency; large writer p99 + non-zero kernel retry rate indicates eviction is contending with hot reads.
`bzrk.segment_cache.range_lock_kernel_retries`	counter	—	—	`kind`: read, write	Count of kernel `F_OFD_SETLK` conflicts that triggered a bounded backoff + retry. Attributed by `kind` (read \| write). Within a single process, in-process readers and writers coordinate via the per-block state machine BEFORE issuing fcntl, so non-zero here implies a foreign holder (cross-process or peer contention).
`bzrk.segment_cache.range_lock_poison`	counter	—	—	`op`: read_lock, read_unlock, write_lock, write_unlock	Count of per-block range-lock poison events. Attribute `op` identifies the failing operation: `read_lock` (F_RDLCK syscall error), `read_unlock` (F_UNLCK on read release), `write_lock` (F_WRLCK syscall error), `write_unlock` (F_UNLCK on writer drop). Any non-zero rate is a kernel/fs-corruption signal — the affected object is invalidated and re-fetched.
`bzrk.segment_cache.punch_races`	counter	—	—	`reason`: epoch_moved, not_resident, writer_pending, io_error	Lock-free reads that declined the bytes they had just read and fell back to the locked path, by `reason`: `epoch_moved` (a hole-punch landed on the object during the read; under the lock the bytes turned out fine), `not_resident` (the blocks were gone and had to be fetched again), `writer_pending` (a writer held the range, so the read waited), `io_error` (the open or read failed, and the locked path reissues it so the error is reported properly). Normally a trickle against `cache_reads` — it takes an eviction punch landing inside a read. Zero while eviction is punching means the guard is not being exercised, not that it is working, which is what to check after a deploy on a cluster whose cache never fills.
`bzrk.segment_cache.local_punch_races`	counter	—	—	`reason`: epoch_moved, not_resident, writer_pending, io_error	The same event as `punch_races`, seen by the in-process cache handle rather than by the out-of-process client, with the same `reason` values. A separate series on purpose. Both can live in the same binary, and one name for two vantage points does not give a combined view — each side would overwrite the other's. Sum them when you want the total; keep them apart when you want to know which side raced.
`bzrk.segment_cache.client_uds_frames_sent`	counter	—	—	—	Control messages the query service sent to the segment cache. Reading cached segment data does not use these, so this tracks how often segments are opened and closed, not how much data is read. It should stay small relative to query volume.
`bzrk.segment_cache.client_uds_frames_received`	counter	—	—	—	Control messages the query service received from the segment cache, including notices that cached data was evicted.
`bzrk.segment_cache.cache_read_nanos`	counter	—	—	`path`: fast, slow	Time spent reading cached segment data, split by `path` the same way as `cache_reads`. Divide by `cache_reads` for the same `path` to get an average read time. The two paths differ by orders of magnitude, so a combined average is misleading — read them apart.
`bzrk.segment_cache.slow_path_entries`	counter	—	—	`reason`: not_resident, writer_pending, revoked	Reads that could not be served from local cache storage and had to ask the cache service first, by `reason`: `not_resident` (the data was not marked as present locally), `writer_pending` (another reader was already bringing it in), `revoked` (the local copy was withdrawn). These cost a round-trip to the cache service, so they are far more expensive than an ordinary read. A large share of `not_resident` on data that has already been queried once is worth investigating.
`bzrk.segment_cache.handle_reopens`	counter	—	—	—	Objects a reader had to open again because the server evicted them out from under an open handle. Eviction punches regardless of who holds an object, so its id, cache fd and shm mapping all go stale together; re-opening is how a query survives an eviction that lands on a segment it is reading. Expected to be non-zero wherever the cache is small relative to the data — which is every customer, and neither of our clusters. Each one is a read that recovered rather than a query that failed.
`bzrk.segment_cache.handle_acquisitions`	counter	—	—	`path`: resident, coordinator	Cache handles acquired for a segment open, attributed by `path`: `resident` (served from the object map the coordinator publishes, with no message to the actor) or `coordinator` (object not registered, so admission — usually an S3 first-block fetch — ran on the actor). A `coordinator` share that tracks the total open rate means residency lookups are serializing through the actor mailbox again (#5222).
`bzrk.segment_cache.handle_acquire_duration_ms`	histogram	—	ms	—	What a contended segment open waited for its cache handle, recorded by the reader for whichever cache backend it holds — in-process or the cache_server round trip. Only opens slower than 5 ms are recorded: a resident segment opens in microseconds, and instrumenting every open would cost more than it tells us. So the count is "how many opens queued" — the alertable signal, normally zero — and the buckets are how badly. Do not read the percentiles as overall open latency. During #5222 this population was ~90k per 30 min averaging 571 ms.
`bzrk.segment_cache.block_hits`	counter	—	—	`path`: fast, slow	Cumulative count of cache blocks served from local disk (resident at read time). Pair with `block_misses` to compute block-level hit rate.
`bzrk.segment_cache.block_misses`	counter	—	—	—	Cumulative count of cache blocks that triggered an S3 fetch (not resident at read time, dispatched through the coordinator).
`bzrk.segment_cache.bytes_read_from_cache`	counter	—	By	`path`: fast, slow	Cumulative bytes returned to callers from local disk (fast or slow path, after blocks landed). Pair with `s3_fetch_bytes_total` to compute byte-level hit rate.
`bzrk.segment_cache.cache_reads`	counter	—	—	`path`: fast, slow	Cache-read attempts terminated. Attribute `path` distinguishes `fast` (all blocks resident, direct pread) from `slow` (at least one block missing, went via coordinator). Ratio fast/(fast+slow) is the read-path fast-path hit rate.

Process

Metric	Type	Across pods	Unit	Labels	Description
`bzrk.process.cpu_seconds_total`	counter	—	s	—	Total CPU time consumed by the process, in seconds (use rate()).
`bzrk.process.start_time_seconds`	gauge	one series per pod	s	—	Unix start time of the process, in seconds. Uptime = time() - this.
`bzrk.process.resident_memory_bytes`	gauge	one series per pod	bytes	—	Resident set size of the process. Pod total is the sum across the pod's processes.
`bzrk.process.open_fds`	gauge	one series per pod	—	—	Open file descriptors held by the process.
`bzrk.process.max_fds`	gauge	same on every pod	—	—	Soft RLIMIT_NOFILE — the file-descriptor ceiling.

CacheSeeder

Metric	Type	Across pods	Unit	Labels	Description
`bzrk.cache_seeder.seed_outcomes`	counter	—	—	`outcome`: seeded, no_owner, skipped; `kind`: nursery, janitor	Producer-side merge cache-seed attempts, by outcome (seeded \| no_owner \| skipped) and kind (nursery \| janitor). A rising no_owner/skipped share means merged segments are landing cold, so the first query pays an S3 fetch.

Container

Metric	Type	Across pods	Unit	Labels	Description
`bzrk.container.memory_bytes`	gauge	worst pod	bytes	—	Container memory working set (cgroup v2 memory.current minus inactive_file), matching kubelet's container_memory_working_set_bytes; excludes reclaimable page cache. Read from the container's OWN cgroup (via cgroup namespace) and measured against the same cgroup as the container memory limit — the correct OOM reference. May read below the otel-agent's pod-level k8s.pod.memory.working_set (parent cgroup + sandbox); that scope difference is expected, not a discrepancy.
`bzrk.container.cpu_seconds_total`	counter	—	s	—	Cumulative container CPU time (cgroup v2 cpu.stat usage_usec); use rate().

Http

Metric	Type	Across pods	Unit	Labels	Description
`bzrk.http.requests_total`	counter	—	—	`method`: GET, POST, PUT, DELETE, PATCH, HEAD, OPTIONS, other; `status_class`: 2xx, 3xx, 4xx, 5xx, other	Total HTTP requests the service handled, by method and status class (2xx/3xx/4xx/5xx/other). Error rate = sum(4xx,5xx) / total.
`bzrk.http.request_duration_ms`	histogram	—	ms	—	HTTP request handling latency in milliseconds (overall, unlabeled — use histogram_quantile).

S3

Metric	Type	Across pods	Unit	Labels	Description
`bzrk.s3.upload_bytes_total`	counter	—	By	—	Bytes uploaded to S3 (PUT / multipart part bodies). Throughput = rate()/1048576 MiB/s.
`bzrk.s3.download_bytes_total`	counter	—	By	—	Bytes downloaded from S3 (GET / ranged-read bodies). Throughput = rate()/1048576 MiB/s.
`bzrk.s3.failures_total`	counter	—	—	`operation`: upload, upload_part, download, head, delete; `kind`: throttled, timeout, error	S3 operations that failed, by operation + failure kind. Excludes not_found (normal for nursery poll-misses; see requests_total). Reliability signal for the S3 backend.
`bzrk.s3.requests_total`	counter	—	—	`operation`: upload, upload_part, download, head, delete; `outcome`: success, not_found, throttled, timeout, error, abandoned	All S3 operations attempted, by operation + terminal outcome. Denominator for error ratio (failures_total / requests_total) and the only place not_found is visible. `upload` counts whole-object operations — a single PUT, or a completed multipart upload — so it reads as files per second. Individual parts count under `upload_part`, which keeps the failures/requests ratio coherent within each, since a part can fail without failing its file. There is deliberately no part latency to go with it — see `upload_duration_ms`. `abandoned` is a download dropped because every reader had gone before it finished (#4145); it stays out of failures_total. Not alerted directly (the alert sums all series).
`bzrk.s3.ttfb_ms`	histogram	—	ms	`operation`: download	Time to an S3 GET's response headers: connect plus the store's own think time, body excluded. Pair it with `body_transfer_ms` to separate "the store is slow to respond" from "the path is slow to transfer", which is the decomposition that identifies a TCP slow-start-bound read. Only a read has a separately measurable headers phase, so `download` is the only value — a PUT has no headers-then-body split and lands in `upload_duration_ms`. The label stays because a HEAD, if it is ever timed, is a headers-only operation and belongs here rather than there. Recorded on all three S3 paths, which matters because only one of them reaches the `RateLimitedStore` wrapper that the OTLP `s3.request.duration_seconds` is emitted from: the wrapper itself, the segment cache's raw client, and the segment fetcher/uploader, which also hold a raw client plus a standalone limiter and carry the highest-volume transfers. Coverage here is therefore wider than on the OTLP metric.
`bzrk.s3.upload_duration_ms`	histogram	—	ms	—	One complete PUT of a whole object, body included. Unlabelled: a write has no second phase to name, and no other operation has a whole-operation duration worth keeping. Read it as a latency, not a throughput. The value is roughly `size / rate + roundtrip`, so its percentile is largely a percentile of object sizes; it earns its place at the small end, where the duration is the roundtrip and a few-hundred-byte control write has no meaningful bandwidth. For anything moved in bulk, `transfer_bytes_per_second{operation=upload}` is the size-independent view. A multipart part has no duration of its own. Its payload is a fixed `part_size`, so a part's duration and its bandwidth are the same measurement — `transfer_bytes_per_second` states it directly, and stays interpretable if `part_size` ever varies with file size. Nothing times a multipart operation end to end either, so a large segment's total upload time is unmeasured by design: it would be size-dependent and no more interpretable than this number is.
`bzrk.s3.body_transfer_ms`	histogram	—	ms	`operation`: download	Time to drain an S3 GET response body after the headers arrive. High here with a low `ttfb_ms` means the store answers promptly and the path is the constraint — for a block-sized read that is usually RTT times the number of TCP slow-start round trips, which no single latency number would reveal. Only the download direction has a separately measurable body phase; uploads are covered by `upload_duration_ms`. Covers both cache read shapes — the buffered reads and the streaming multi-block drain, which carries the largest ranges. On the streaming path only time blocked on the network is counted, excluding our own CRC and write work, so the two shapes stay comparable. Recorded on a fully drained stream only: a truncated transfer would report less time than the transfer it failed to finish, and those show up as `fetch_duration_ms{outcome=timeout\|error}` instead.
`bzrk.s3.fetch_duration_ms`	histogram	—	ms	`mode`: first_block, streaming; `outcome`: success, error, timeout, not_found, throttled, write_error, abandoned	End-to-end cold-read latency in the segment cache, by mode and outcome — what a query actually blocks on when a block is not resident. The `first_block` + `success` series is the user-facing number. Bucket bounds are kept identical to the OTLP `bzrk.segment_cache.s3_fetch_duration_ms` it promotes, so percentiles are comparable across the two tiers.
`bzrk.s3.permit_wait_ms`	histogram	—	ms	`outcome`: acquired, abandoned	Time a fetch spent waiting for a process-wide S3 permit. This queue is invisible to `fetch_duration_ms`, whose timer starts only once the permit is held, so it is the signal that distinguishes "the object store is slow" from "we are throttling ourselves and the store was never asked". Rising here with `permits_in_use` pinned at its ceiling means the concurrency cap is binding. `abandoned` samples are waits cancelled by the fetch deadline, i.e. permit starvation rather than a slow store.
`bzrk.s3.permits_in_use`	gauge	worst pod	—	—	S3 fetch permits currently held process-wide. Read against the configured ceiling (`max_total_concurrent_s3_requests`): sustained saturation with a rising `permit_wait_ms` is the "concurrency cap too low" signal, whereas idle permits during slow fetches point at the path rather than at us.
`bzrk.s3.retries_total`	counter	—	—	`operation`: upload, upload_part, download, head, delete; `kind`: throttled, timeout, error	S3 requests retried after the store returned a retryable failure. Genuine backend trouble: the request reached S3 and S3 (or the network) failed it. Excludes retries caused by our own rate limiter — see local_backoff_total. Recorded in every retry loop, not just `RateLimitedStore`'s: the segment fetcher and uploader run their own, and they carry the bulk of the traffic, so omitting them would leave this alert quiet through a retry storm on segment transfers. `CreateMultipartUpload` and `CompleteMultipartUpload` have no inner retry loop — the whole upload is reattempted by an outer wrapper — so they are attributed at the failure site, which is the only place the S3 error kind is still known. That accounting counts the failure that triggers the retry, so an exhausted sequence contributes one more sample than the inner loops would; at most one per upload, and immaterial to a rate-based alert.
`bzrk.s3.local_backoff_total`	counter	—	—	`operation`: upload, download, head, delete	Backoff delays applied by OUR OWN AIMD rate limiter before the request reached S3. Counts delays, not requests — one request held back several times increments this once per delay, which is what makes it comparable with retries_total. Not an S3 fault and deliberately not alerted: it is the "we are the bottleneck" counterpart to retries_total. A high ratio of this to retries_total means the limiter has backed itself down and the store is barely being asked — tune the limiter, not the object store. Recorded in exactly one place — `S3RateLimiter::try_acquire`, on the branch where it denies a token — so every shape of local backpressure is counted by construction: the blocking `acquire` poll (segment fetcher, single-PUT uploads), the non-blocking deferrals in the multipart uploader and on the cache cold-read dispatch path, and the outright rejection returned by the rate-limited multipart `put_part`. Callers must not count it again. Speculative `has_capacity` checks consume no token and are deliberately not counted. The unit is one per denied unit of work, not one per scheduled wake-up.
`bzrk.s3.connections_opened_total`	counter	—	—	—	TCP connections established to the object store. Against `requests_total` this is the reuse rate: `1 - rate(connections_opened_total)/rate(requests_total)`. A pooled connection never reaches the connector, so counting the connector is counting new connections. Counted on success, not when the attempt starts: a connect that fails or is abandoned mid-flight never becomes a connection and so never produces a matching close, and counting it here would make `connections_opened_total - connections_closed_total` — the live count — climb without bound. A rate approaching `requests_total` means the pool is not reusing anything, which costs a TCP and TLS handshake per request; the usual cause is `pool_idle_timeout` above the endpoint's own idle timeout, or bodies being abandoned (see `connections_closed_total{reason="body_abandoned"}`).
`bzrk.s3.connections_closed_total`	counter	—	—	`reason`: idle_timeout, idle_evicted, server_closed, body_abandoned, cancelled, error, pool_dropped	Connections retired, by cause. The split is what makes churn diagnosable rather than merely visible. `server_closed` is the endpoint hanging up first — if it dominates, our `pool_idle_timeout` is longer than theirs and every idle connection is a request waiting to be replayed. `body_abandoned` is a response body dropped before it drained, which costs the whole connection because an HTTP/1.1 connection cannot be reused mid-message; the segment cache does this deliberately when every reader has gone (#4145), so it must stay distinguishable — otherwise a rise reads as an endpoint fault. `idle_timeout` is the reaper doing its job and is not a fault at all. `idle_evicted` means `pool_max_idle_per_host` is binding.
`bzrk.s3.connect_duration_ms`	histogram	—	ms	`phase`: resolve, tcp, proxy_tls, tunnel, tls, total; `via`: direct, proxy	Cost of establishing a connection, split by phase, recorded only on a pool miss. `total` is carried explicitly rather than left to be summed from the others: it spans the whole establishment, so it also carries time spent on candidate addresses that stalled before a later one succeeded, and summing the phases would report a millisecond connection after a multi-second failover. The split earns its series: DNS runs on the shared blocking pool, the same one the segment cache uses for its writes, so resolution can queue behind write-back and present as a slow object store — `resolve` is where that becomes visible instead of mysterious. `tls` is absent for plaintext endpoints rather than recorded as zero. `via` says whether the connection went through an egress proxy, and when it did, `resolve` and `tcp` are the proxy's rather than the endpoint's and `tunnel` is the `CONNECT` exchange with it — without the split, a slow proxy reads as a slow object store. `proxy_tls` is the handshake with an `https://` proxy, which is a different peer from the one `tls` measures: two certificates are verified on such a connection, the proxy's on the way in and the endpoint's inside the tunnel, and only the second says anything about the store. Multiplied by `connections_opened_total` this is what poor reuse actually costs.
`bzrk.s3.connection_age_ms`	histogram	—	ms	—	How long a connection lived, sampled when it closes. The pool-churn signal: a distribution clustered near `pool_idle_timeout` means connections are being retired by our own reaper before the next request wants them, while a cluster well below it means something else is ending them — read `connections_closed_total{reason}` to find out which.
`bzrk.s3.connect_failures_total`	counter	—	—	`family`: ipv4, ipv6; `reason`: tcp, timeout, tls, tls_timeout, proxy; `via`: direct, proxy	Candidate addresses that failed to yield a connection, by address family and cause. Counted per address, so a non-zero rate is not by itself a failed request: establishment walks every resolved address, both families at once, and a later candidate usually succeeds. That is exactly why this has to be counted — without it a family that is entirely unreachable is invisible, because the request still completes on the other one and no other metric moves. The `family` split is the point: a rate concentrated on one family means the endpoint publishes addresses this pod has no route to, which costs a failed connect on every pool miss and turns into hard errors wherever the caller disabled retries. `timeout` rather than `tcp` means the address is blackholed instead of refused; `tls` means the peer accepted the connection and then failed the handshake, and `tls_timeout` that it accepted and then simply stopped — reachable and wedged, a fault above the network rather than in it. `reason=proxy` is an egress proxy that refused to open the tunnel, which is its policy or its credentials rather than anything about the endpoint. `via` carries the rest of that distinction: when it is `proxy` the address that failed is the proxy's, so without it a proxy outage reads as the object store being unreachable — and an install can have some endpoints proxied and others exempted by `NO_PROXY`.
`bzrk.s3.request_replays_total`	counter	—	—	—	Requests re-sent on a fresh connection because a pooled one turned out to be dead. Only ever counted when hyper hands the request back, which it does only if not one byte was serialised, so a replay is safe even for a PUT. This is the race a liveness check cannot close: the connection passed screening and the peer's FIN landed before the write. Small and non-zero is normal; a rate that tracks `requests_total` means nearly every pooled connection is stale by the time it is used.
`bzrk.s3.transfer_bytes_per_second`	histogram	—	By/s	`operation`: upload, download	Throughput of one S3 transfer. Size-independent, unlike a duration, so it is comparable across services and stores where `upload_duration_ms` on a whole object is largely a percentile of object sizes — and it is the only view of download transfer speed, since the body drain reaches no other metric. Covers every production transfer: whole-object uploads, one multipart part, ingest's batch PUTs, both segment download paths (the fetcher and `get_to_file`), and the segment cache's own reads — streaming and buffered alike. What is left uncovered on the write side is control-plane records — stream STOP markers, the rewrite journal — not payload. A `RateLimitedStore` GET draws no sample, because it is reported before its body is read; the body it goes on to drain is sampled by whichever path drains it. Only a duration that is the transfer produces a sample — callers declare what their clock covers (`TransferTiming`), since a duration that also spans retry backoff or cache-file writes would report our own delays as a slow store. On `download` that declaration makes this a store-blocked rate, and the distinction is not academic: the sample divides by time spent waiting on the store, excluding the local writes that overlap it, so it answers "how fast does the store deliver when asked" rather than how fast the fetch went. It is a ceiling, never achieved throughput. The gap is largest exactly where a reader is most tempted to trust the number — a service reading small objects spends most of each drain writing, so little of the transfer is visible to the clock. Measured on dev across the four services, one read p50 sat above 150 MB/s and its throughput estimate peaked near 800 MB/s on a path nowhere near that capable, while a service reading large objects over the same link reported a fifth of it. Read as a speed those numbers are nonsense; read as ceilings they are correct. So the informative quantity is the gap against wall-clock throughput, not this figure alone: a high rate here beside a slow fetch is the signature of our own serialisation rather than a slow store, which is the whole question in #4226. One number cannot separate the two. Samples below 256 KiB are dropped: under that size the elapsed time is mostly round trip, so the ratio measures latency in the wrong units. The floor sits just under the smallest object moved in bulk (a baby segment). Bounds run from 0.25 MiB/s to 1 GiB/s: the low end is where a path is unusable rather than merely slow, and the top is above a 1 Gbps NIC, so a saturated link lands inside the range instead of in `+Inf`.
`bzrk.s3.allowed_rate`	gauge	one series per pod	—	—	Requests per second the AIMD limiter currently permits itself — the "what we allow" half of capable-vs-delivered, where `ttfb_ms`, `upload_duration_ms` and `transfer_bytes_per_second` are the "what the path gave us" half. Falling here while the store looks healthy means we backed ourselves off; read it with `local_backoff_total`, which counts the delays this ceiling caused. The minimum across the process's limiters that are not idle, rather than one series per limiter: the operator tier admits no `endpoint`/`bucket` label (open cardinality), and the lowest active ceiling is the one actually holding traffic back. A rate recovers only when a request is weighed against it and registry entries are never removed, so an idle bucket would otherwise hold a stale low rate and report a busy one as constrained. Per-target detail stays on the OTLP `s3.throttle.*` instruments. Reads 0 on a pod holding no limiter, and before the emitting process's first S3 request: every declared series renders whether or not it has been written, so there is no way to express "not applicable". Under the production `grpc` transport that includes the query pod itself — its fetches run in the qws pod's `cache_server`, which is where to read this.

Service Metrics

CacheServerCopy Anchor Link

Gateway.proxyCopy Anchor Link

IngestCopy Anchor Link

JanitorCopy Anchor Link

K8sCopy Anchor Link

MetaCopy Anchor Link

NurseryCopy Anchor Link

PostgresCopy Anchor Link

QueryCopy Anchor Link

Query.cacheCopy Anchor Link

QwsCopy Anchor Link

UiCopy Anchor Link

LogCopy Anchor Link

S3Copy Anchor Link

EngineCopy Anchor Link

QueryCoordCopy Anchor Link

QwSpawnCopy Anchor Link

SegmentCacheCopy Anchor Link

ProcessCopy Anchor Link

CacheSeederCopy Anchor Link

ContainerCopy Anchor Link

HttpCopy Anchor Link