Service Metrics

All Berserk services emit metrics via OpenTelemetry. Metrics are exported over OTLP to the configured collector endpoint and can be queried in Berserk itself.

Each metric name is prefixed with bzrk. followed by the service scope (e.g. bzrk.ui.query_duration).

A pre-built Grafana dashboard is available for download: bzrk-service-metrics.json. Import it into Grafana and select your Berserk datasource to visualize all metrics below.

Ingest

Simplified OTLP ingest service that receives traces, metrics, and logs over HTTP/gRPC and uploads to S3 via ingest_client.

Metric	Type	Across pods	Unit	Description
`bzrk.ingest.queue_rejections`	counter	—	—	Total requests rejected due to admission control (semaphore exhaustion or dead stream actor)
`bzrk.ingest.batch_flush_duration`	histogram	—	ms	Duration of batch flush operations (S3 upload latency)
`bzrk.ingest.batch_inputs`	histogram	—	items	Incoming OTLP requests coalesced into one S3 batch flush. p50/p99 sizes the fan-in expected at the batch upload span links.
`bzrk.ingest.data_dropped`	counter	—	—	Total requests dropped before reaching a stream — missing/unresolvable ingest token, or routing failed because the target stream actor died
`bzrk.ingest.time_since_last_upload_seconds`	gauge	worst pod	s	Worst-silent stream's seconds since last successful S3 upload
`bzrk.ingest.inflight_requests`	gauge	fleet total	—	Admission permits in use across HTTP/gRPC/Loki transports
`bzrk.ingest.buffer_bytes`	gauge	fleet total	bytes	Bytes buffered across stream actors (pod-level SUM)
`bzrk.ingest.inflight_bytes`	gauge	fleet total	bytes	In-flight bytes reserved against the admission byte budget — the primary memory gate. Leading indicator of byte-budget pressure before throttling starts.
`bzrk.ingest.inflight_bytes_limit`	gauge	fleet total	bytes	Total admission byte budget (auto-sized to a fraction of the cgroup memory limit). Constant; lets dashboards compute utilization without joining config.
`bzrk.ingest.process_rss_bytes`	gauge	worst pod	bytes	Sampled process resident set size feeding the memory-ceiling admission gate
`bzrk.ingest.memory_ceiling_bytes`	gauge	same on every pod	bytes	RSS ceiling above which admission sheds (0 when the memory gate is disabled)
`bzrk.ingest.phantom_write_retries`	counter	—	—	Total retry attempts after an uncertain upload (held by PendingRetry)
`bzrk.ingest.phantom_write_retry_budget_exhausted`	counter	—	—	Total times a PendingRetry exhausted its budget without resolving (fell back to retryable error)
`bzrk.ingest.invalid_otlp_total`	counter	—	—	OTLP payloads rejected by the fast-path validator. Attributes: signal (traces\|metrics\|logs), verdict (accepts_where_we_reject\|rejected\|skipped_rate_limited). `accepts_where_we_reject` > 0 indicates the fast-path validator disagrees with prost — strong parser-bug signal. See /debug/invalid-otlp-samples for the byte-level sample.

Background service responsible for segment lifecycle management: merging small segments into larger ones, deleting tombstoned segments from cloud storage, and optionally running configured probe queries.

Metric	Type	Across pods	Unit	Description
`bzrk.janitor.segment_count`	gauge	same on every pod	—	Current number of segments in the cluster
`bzrk.janitor.total_data_size`	gauge	same on every pod	bytes	Total size of all segment data in cloud storage
`bzrk.janitor.segments_deleted`	counter	—	—	Total segments deleted from cloud storage
`bzrk.janitor.merge_cycle_duration`	histogram	—	ms	Duration of segment merge cycles
`bzrk.janitor.merge_failures`	counter	—	—	Total failed merge cycles
`bzrk.janitor.probes_rejected`	counter	—	—	Probe queries the query service answered by rejecting — an unknown table, or a query invalid against this tenant's schema. Reachability is proven (see probes_completed), so this is not an outage; it means the probe reads nothing here and the read path is therefore unmeasured. Non-zero on a tenant whose configured probe table has never been written to.
`bzrk.janitor.probe_duration`	histogram	—	ms	Duration of probe query executions. Only emitted where probe queries are configured (janitor.config.probeJobs); off by default.
`bzrk.janitor.vsearch_merger_artifacts_emitted`	counter	—	—	Merged segments where the merger rebuilt VCEN/VTPH/VTPC/VIDF. See docs/dev/vidx-vsearch-impl-plan.md PR 8.
`bzrk.janitor.vsearch_merger_pre_feature_inputs`	counter	—	—	Input segments to a merge that had no vsearch artifacts (pre-feature). Each increment indicates one input was skipped during VXXX rebuild.
`bzrk.janitor.vsearch_merger_unstamped_rows`	counter	—	—	Rows seen during merge that lacked a template_id.FIELD stamp. Persistent nonzero rate indicates ingest-side stamping isn't keeping up with merger fan-in.
`bzrk.janitor.vsearch_merger_template_index_bytes`	gauge	worst pod	By	Peak transient JanitorTemplateIndex memory during a merge (sum of input VTPC embedding tables). docs/dev/vidx-vsearch.md 8.2 caps the risk; this metric flags the cap binding before OOM.
`bzrk.janitor.vsearch_merger_duration_ms`	histogram	—	ms	Wall-clock added to a merge by VXXX rebuild (loading input VTPCs + tier selection + writing output VCEN/VTPH/VTPC/VIDF). Excludes the base ROWS-merger time.
`bzrk.janitor.probes_completed`	counter	—	—	Probe queries the query service answered — including a rejection such as an unknown table, because reaching that verdict still took the HTTP hop, the parser, the binder and a catalog lookup through meta. A rejected answer is counted here and also in probes_rejected. Only emitted where probe queries are configured (janitor.config.probeJobs); off by default, so absence is the normal state and carries no alert.

Nursery

Ingestion service that receives OpenTelemetry data from the collector, converts it into segments, and manages segment merging for optimal query performance.

Metric	Type	Across pods	Unit	Description
`bzrk.nursery.streams_active`	up_down_counter	fleet total	—	Number of currently active stream followers
`bzrk.nursery.ingest_lag_seconds`	gauge	worst pod	s	How far behind the read positions are on the streams this nursery still owes reads on — the data-plane freshness signal. Deliberately excludes meta's poll bound and the watermark's skew margin, so a rise here means ingest, not coordination. A stream owed but never read counts from when this nursery took it on, so being stuck cannot read as being caught up. Zero means no stream is owed anything
`bzrk.nursery.completeness_watermark_lag_seconds`	gauge	worst pod	s	Seconds since the completeness watermark this nursery publishes. Includes meta's snapshot bound and the skew margin, because the watermark is capped by both — so unlike ingest_lag_seconds it never reads below meta's own delay. A nursery owning no stream group reports that bound alone, which is the most any nursery can vouch for. Zero only before the first applied poll, a one-RPC window in which the watermark is the conservative floor and nothing is capturable anyway
`bzrk.nursery.refresh_discards_consecutive`	gauge	worst pod	—	Meta refreshes discarded in a row since the last one installed. Ordinary staleness clears on its own: on one continuous primary a later snapshot always covers an earlier one, so non-coverage can only be a delayed in-flight response, and those stop arriving. A value that STAYS high has two causes, and the nursery log message says which — both log at ERROR once the run is sustained, so match on the message, not the level. (1) Meta's database moved backwards — a restore, a lagged failover, or a repointed DSN — logged as 'Discarding stale meta response'. The nursery deliberately does not try to recover: the cloud holds segments meta has forgotten, so there is no reconciliation to reach, and the fix is a clean-disk restart of the nurseries (replace the pods, which discards the working dir by design, rather than restarting the containers in place). (2) A covering body that would move a monotonic frontier backwards, logged as 'Discarding a refresh that would move a monotonic frontier backwards'. That is a broken co-snapshot invariant, not an operational event, and a restart will not fix it. Either way the nursery stops learning routing and its watermark freezes while this holds
`bzrk.nursery.download_duration_ms`	histogram	—	ms	S3 segment download duration
`bzrk.nursery.conversion_duration_ms`	histogram	—	ms	Protobuf to segment conversion duration
`bzrk.nursery.total_duration_ms`	histogram	—	ms	Total segment processing duration (download + conversion)
`bzrk.nursery.bytes_ingested`	counter	—	By	Total compressed bytes downloaded from S3 (use rate() for throughput)
`bzrk.nursery.bytes_ingested_uncompressed`	counter	—	By	Total uncompressed proto bytes ingested (use rate() for throughput)
`bzrk.nursery.segment_output_bytes`	counter	—	By	Total bytes of segment files produced (use rate() for throughput)
`bzrk.nursery.data_errors`	counter	—	—	Data errors (malformed protobuf, conversion failures)
`bzrk.nursery.infra_errors`	counter	—	—	Infrastructure errors (S3 failures, I/O errors)
`bzrk.nursery.active_streams`	gauge	fleet total	—	Number of active streams reported by Meta
`bzrk.nursery.closed_streams`	gauge	fleet total	—	Number of closed streams reported by Meta
`bzrk.nursery.stream_groups_released`	counter	—	—	Stream groups whose streams left this nursery's poll scope (ownership moved or streams deleted); local work is discarded
`bzrk.nursery.merge_count`	counter	—	—	Total number of completed merges
`bzrk.nursery.merge_inputs`	histogram	—	segments	Ingest segments consumed by one baby-segment merge. p50/p99 sizes the fan-in expected at the nursery merge span links.
`bzrk.nursery.merge_output_size_mb`	histogram	—	MB	Compressed output size of merged segments
`bzrk.nursery.merge_duration`	histogram	—	ms	Duration of segment merge operations
`bzrk.nursery.merge_speed_mbps`	histogram	—	MB/s	Merge throughput in megabytes per second
`bzrk.nursery.oldest_unmerged_data_age_seconds`	gauge	worst pod	s	Age of the oldest unmerged baby segment in seconds
`bzrk.nursery.local_backlog_segments`	gauge	fleet total	segments	Un-merged baby segments held locally across all streams — offsets above the merged frontier, i.e. outstanding merge work. Merged segments still on disk awaiting their retention grace are excluded. The count-driven fetch-backpressure signal — many small segments OOM a bounded pod via query-planning structures. See docs/dev/nursery-overload-backpressure.md.
`bzrk.nursery.local_backlog_bytes`	gauge	fleet total	By	Total bytes of un-merged baby segments held locally, excluding merged segments awaiting their retention grace. Secondary fetch-backpressure signal (the byte budget); segment count is primary. Total disk residency is `k8s.pod.filesystem.usage`, not this.
`bzrk.nursery.local_resident_segments`	gauge	fleet total	segments	All baby segments resident locally, merged or not — the input to the hard fetch-backpressure ceiling, which is an OOM guard on what the pod holds rather than on merge debt. Exceeds local_backlog_segments by the merged set awaiting its retention grace.
`bzrk.nursery.local_resident_bytes`	gauge	fleet total	By	Total bytes of all locally resident baby segments, merged or not — the byte axis of the hard fetch-backpressure ceiling. Not interchangeable with k8s.pod.filesystem.usage, which also counts merged outputs and other pod files.
`bzrk.nursery.fetch_paused`	gauge	one series per pod	—	1 while fetch backpressure is paused (inbound fetch stopped so the merge pipeline can drain), 0 otherwise. Sustained 1 means the nursery cannot keep up with ingest.
`bzrk.nursery.events_ingested`	counter	—	—	Total events ingested across all streams
`bzrk.nursery.forward_dated_events_clamped`	counter	—	—	Events whose OTLP timestamp was in the future relative to ingest_time and got clamped to ingest_time on write. Never drops the row. Non-zero indicates clock skew at the source — the map-reduce-state cache's monotonic-ingest invariant still holds because the row's `timestamp` is now ≤ its `ingest_time`.
`bzrk.nursery.ingest_delay`	histogram	—	ms	Delay between event timestamp and ingest time
`bzrk.nursery.routing_unknown_table`	counter	—	—	Dropped segments where the routing key did not match any table in the token's database
`bzrk.nursery.oldest_routing_hold_age_seconds`	gauge	worst pod	s	Age of the longest-standing hold on an offset whose routing does not resolve. Excludes time spent under fetch backpressure, which stalls a hold for a reason that is not routing.
`bzrk.nursery.combine_count`	counter	—	—	Completed local combines, the analogue of merge_count. Zero while combining is disabled (combine_max_concurrent = 0); a rate that tracks ingest means the trigger is doing what it was set to.
`bzrk.nursery.combines_failed`	counter	—	—	Local combines the executor refused or could not write. Unlike a stalled merge, which shows up as local_backlog growth, a combine that stops achieves nothing visible — the files simply stay, which is indistinguishable from having nothing to combine. This is the only signal that says otherwise. A refusal over inputs that disagree on their vsearch config is expected and benign; an I/O failure is not, and the log line carries the error that tells them apart.
`bzrk.nursery.combine_input_bytes`	counter	—	By	Bytes read into local combines. Against segment_output_bytes this is the copy amplification combining costs, which is the signal that decides whether the trigger is set right — every byte is copied at most once, but v1 copies nearly all of them, so expect a ratio just under 1. Not against bytes_ingested, which is compressed bytes off S3 rather than the post-conversion local bytes a combine actually reads.
`bzrk.nursery.pending_combines`	gauge	fleet total	—	Combines dispatched and not yet reported. One is cleared only by its executor's terminal event, so a value stuck above zero means that event is never coming — the pair's claim is held for the life of the process and no merge can start for it.
`bzrk.nursery.vsearch_seal_artifacts_emitted`	counter	—	—	Segments whose seal wrote VCEN/VTPH/VTPC/VIDF. Increments by 1 per sealed segment that produced vsearch artifacts.
`bzrk.nursery.vsearch_seal_artifacts_skipped`	counter	—	—	Segments where seal skipped vsearch artifact emission (no model configured, or no vsearch_fields). Increments by 1 per such segment.
`bzrk.nursery.vsearch_embedding_cache_hits`	counter	—	—	Template-hash cache hits in the seal-time embedding cache. High hit rate = log data is template-clustered as expected.
`bzrk.nursery.vsearch_embedding_cache_misses`	counter	—	—	Template-hash cache misses — model.encode() invocations at seal time.
`bzrk.nursery.vsearch_merger_artifacts_emitted`	counter	—	—	Merged segments where the merger rebuilt VCEN/VTPH/VTPC/VIDF. See docs/dev/vidx-vsearch-impl-plan.md PR 8.
`bzrk.nursery.vsearch_merger_pre_feature_inputs`	counter	—	—	Input segments to a merge that had no vsearch artifacts (pre-feature). Each increment indicates one input was skipped during VXXX rebuild.
`bzrk.nursery.vsearch_merger_unstamped_rows`	counter	—	—	Rows seen during merge that lacked a template_id.FIELD stamp. Persistent nonzero rate indicates ingest-side stamping isn't keeping up with merger fan-in.
`bzrk.nursery.vsearch_merger_template_index_bytes`	gauge	worst pod	By	Peak transient JanitorTemplateIndex memory during a merge (sum of input VTPC embedding tables). docs/dev/vidx-vsearch.md 8.2 caps the risk; this metric flags the cap binding before OOM.
`bzrk.nursery.vsearch_merger_duration_ms`	histogram	—	ms	Wall-clock added to a merge by VXXX rebuild (loading input VTPCs + tier selection + writing output VCEN/VTPH/VTPC/VIDF). Excludes the base ROWS-merger time.

Query

Query execution service that receives KQL queries over HTTP and gRPC, plans and executes them against segments, and streams results back to clients.

Metric	Type	Across pods	Unit	Description
`bzrk.query.execution_duration`	histogram	—	ms	End-to-end query execution duration
`bzrk.query.requests`	counter	—	—	Total query requests received
`bzrk.query.in_flight`	up_down_counter	fleet total	—	Queries currently executing in the query process (incremented at request entry, decremented when the response stream completes). The saturation / admission signal: under load this is the queue-depth proxy that traffic (requests) and latency (execution_duration) can't show on their own. Recorded only by the parent query process (qc-child is not a writer).
`bzrk.query.result_rows`	histogram	—	—	Number of rows returned per query
`bzrk.query.errors`	counter	—	—	Total query errors by error type
`bzrk.query.open_fds`	gauge	worst pod	—	`bzrk_lib::count_open_fds()` periodic sample (10s interval). Pair with `bzrk.query.fd_limit` to compute `open_fds / fd_limit` on dashboards/alerts without joining against startup logs. apps/query in cache_mode=remote holds a UDS connection per worker task plus the SCM_RIGHTS cache_fd + shm_fd passed by cache_server, so the count tracks engine concurrency directly. Symmetric with `bzrk.cache_server.open_fds`.
`bzrk.query.fd_limit`	gauge	same on every pod	—	Current `RLIMIT_NOFILE` soft cap. Companion to `open_fds` — sampled on the same 10s tick so dashboards can show "fds: N / LIMIT (X%)" and alerts can fire on `open_fds / fd_limit > 0.8` before saturation. Production binaries raise the soft limit to the hard cap at startup, so this is effectively static; emitting it as a gauge keeps the query simple.
`bzrk.query.routing_decisions`	counter	—	—	Sessions opened by `PoolBackedQwsTransport`, attributed by the routing decision taken (`mode`): - `sticky` — QC supplied a target_node_id and the live member was found in the pool snapshot. Ring routing held end-to-end. - `fallback_walk` — target_node_id supplied, but the targeted member was gone from the live snapshot. Fell back to the partition ring's next-priority node for the batch's first segment. - `fallback_round_robin` — target_node_id supplied, member was gone, and the priority walk failed too (snapshot didn't cover the segment). Degraded to round-robin: NOT ring-aware. A non-zero rate here is the signature of seed-vs-read disagreement surviving the ring's protections. - `no_target_round_robin` — QC didn't supply a target_node_id at all (no partition snapshot at coordinator time → bootstrap / in-process / non-sticky). NOT ring-aware. Sustained non-zero on a fully-bootstrapped cluster means the coordinator never saw a snapshot. - `pool_empty` — no QwsCloud members visible at session-open. Session will fail on first send; pool likely re-populating after a pod rollout. The counter exists to spot the two "NOT ring-aware" rows showing up at non-trivial rates — the failure mode that would let a query land on a non-owner pod and cold-fetch a freshly-seeded segment.
`bzrk.query.vsearch_queries`	counter	—	—	vsearch queries handled by the coordinator (one increment per query containing a vsearch operator).
`bzrk.query.vsearch_query_latency_ms`	histogram	—	ms	End-to-end vsearch query latency (coordinator encode + worker scatter/gather + reducer merge).
`bzrk.query.vsearch_segment_ctx_built`	counter	—	—	Successful per-segment SegmentVsearchContext builds during query execution (segment had VCEN+VTPH for the queried field).
`bzrk.query.vsearch_segment_ctx_skipped`	counter	—	—	Per-segment context builds that returned None (segment lacked VCEN or VTPH for the queried field — pre-feature or wrong field_id). Worker degrades to BM25-only for these.
`bzrk.query.vsearch_chunk_gate_admits`	counter	—	—	ROWS chunks admitted by the vsearch tier-1 gate (composite alphatemplate_sim_ub + (1-alpha)bm25_max >= tau_chunk).
`bzrk.query.vsearch_chunk_gate_drops`	counter	—	—	ROWS chunks dropped by the vsearch tier-1 gate before row scan.
`bzrk.query.vsearch_degraded_to_bm25`	counter	—	—	Queries where the binder set degraded_to_bm25=true on the VSearchScore op (lineage-modifying upstream op — parse/mv-expand). Per-query, not per-row.
`bzrk.query.vsearch_precompute_duration_us`	histogram	—	us	Per-(segment, query) SegmentPrecompute build time. docs/dev/vidx-vsearch.md 5.6 estimates ~10-30 us at defaults.

Query.cache

Metric	Type	Across pods	Unit	Description
`bzrk.query.cache.bytes_total`	counter	—	By	Uncompressed footprint (scanned + filter-skipped) of all slices a query would touch — reused plus recomputed
`bzrk.query.cache.bytes_saved`	counter	—	By	Uncompressed footprint of slices served from the cache — the work avoided this run
`bzrk.query.cache.scan_nanos_total`	counter	—	ns	Chunk-scan time of all slices (reused plus recomputed)
`bzrk.query.cache.scan_nanos_saved`	counter	—	ns	Chunk-scan time of slices served from cache — latency avoided
`bzrk.query.cache.lookups`	counter	—	—	Cache lookups by outcome: hit / miss / skip; skip carries a reason attribute (short_circuit_reducer, prohibiting_fn, no_time_bin, ...)
`bzrk.query.cache.saved_pct`	histogram	—	%	Per-query saved-byte fraction (0-100) over cache-eligible queries; cold runs land at 0. Distribution of reuse, not the volume-weighted aggregate
`bzrk.query.cache.resident_bytes`	gauge	one series per pod	By	On-disk size of resident cache entries
`bzrk.query.cache.entries`	gauge	one series per pod	—	Resident cache entry count. Growth per query is the capture-churn signal: entries that never supersede pile up one generation per refresh
`bzrk.query.cache.capacity_bytes`	gauge	one series per pod	By	Configured size cap; static for the process lifetime
`bzrk.query.cache.eviction_frontier_age_seconds`	gauge	one series per pod	s	Age of the oldest resident entry's last use — the LRU eviction frontier. Low under size pressure means recently-used entries are being evicted (cache disk too small); high means headroom
`bzrk.query.cache.evictions_total`	counter	—	—	Entries evicted since process start (age + size-cap + self-heal). Its rate is the eviction-pressure signal
`bzrk.query.cache.peer.ops`	counter	—	—	Ring cache operations by op and outcome. Lookup fans out to every owner plus the local index, so one query emits several: the ratio of remote_hit to local tells you whether sharing is doing the work, and local_miss_settled counts the fan-outs skipped because this node has owned the partition long enough for its miss to be authoritative. Without this a ring answering everything locally is indistinguishable from one that is sharing
`bzrk.query.cache.peer.rpc_duration_ms`	histogram	—	ms	Wall time of one ring RPC, by op. Lookup sits on the query critical path
`bzrk.query.cache.peer.bytes`	counter	—	By	Bytes moved between pods by the ring: out (states replicated to owners), in (states fetched from an owner). The cost that scales with replica count

Qws

Query-worker supervisor process. Hosts the QueryWorker gRPC service, owns the cache_server child, and drives engine execution for the segments routed to this node. Emits engine-side vsearch lifecycle counters that the coordinator (apps/query) can't see — those fire inside time_slicing/executor.rs and the binder, on the worker side of the QC↔QWS boundary.

Metric	Type	Across pods	Unit	Description
`bzrk.qws.vsearch_segment_ctx_built`	counter	—	—	Successful per-segment SegmentVsearchContext builds on this worker (segment had VCEN+VTPH for the queried field).
`bzrk.qws.vsearch_segment_ctx_skipped`	counter	—	—	Per-segment context builds that returned None (segment lacked VCEN or VTPH for the queried field — pre-feature or wrong field_id). Worker degrades to BM25-only for these.
`bzrk.qws.vsearch_chunk_gate_admits`	counter	—	—	ROWS chunks admitted by the vsearch tier-1 gate on this worker (composite alphatemplate_sim_ub + (1-alpha)bm25_max >= tau_chunk).
`bzrk.qws.vsearch_chunk_gate_drops`	counter	—	—	ROWS chunks dropped by the vsearch tier-1 gate on this worker before row scan.
`bzrk.qws.vsearch_degraded_to_bm25`	counter	—	—	Queries where this worker's binder set degraded_to_bm25=true on the VSearchScore op (lineage-modifying upstream op — parse/mv-expand). Per-query, not per-row.
`bzrk.qws.vsearch_precompute_duration_us`	histogram	—	us	Per-(segment, query) SegmentPrecompute build time on this worker. docs/dev/vidx-vsearch.md 5.6 estimates ~10-30 us at defaults.
`bzrk.qws.cache_objects_resident`	gauge	fleet total	—	Objects held in this node's segment cache, counted by the boot-time reconciliation sweep. Measured once per process, so it is a starting point rather than a live level — cache eviction is driven by disk pressure, so an under-full cache accumulates files for long-dead segments, and this is the count that reveals it.
`bzrk.qws.cache_dead_objects_evicted`	counter	—	—	Cached objects invalidated because their segment is gone: a delete pushed by the janitor and applied after its delay, or one the boot-time sweep found meta no longer reports as worth caching. Files reclaimed that pressure-driven eviction would never have touched.
`bzrk.qws.cache_reap_sweep_failures`	counter	—	—	Boot-time reconciliation sweeps that aborted on an error (cache listing or the meta cacheability call). The sweep fails closed — nothing is evicted — so a node that trips this keeps whatever it missed while down until its next restart.
`bzrk.qws.seed_receive_outcomes`	counter	—	—	Receiver-side outcomes of merge-side cache seed pushes, by outcome (accepted \| rejected_non_owner \| error). `rejected_non_owner` is the ring re-check rejecting a stale producer's push, which surfaces ring drift on the receiver side without segment-level logging.

Ui

Phoenix LiveView frontend. The BEAM runtime metrics below say which part of the runtime holds memory, which the OS/cgroup view (metrics/base.yaml, metrics/container.yaml) cannot: the difference between 'the ui used 900 MB' and '900 MB of it is process heaps, and one mailbox is 400 MB of that'. There is no Rust source for this scope — the Elixir mode of metrics-gen renders the catalogue this app's meter and its operator /metrics exposition both read.

Metric	Type	Across pods	Unit	Description
`bzrk.ui.beam.memory.current.total_bytes`	gauge	fleet total	By	Bytes allocated by the BEAM in total — the runtime's view of the number cgroup accounting kills on.
`bzrk.ui.beam.memory.current.processes_bytes`	gauge	fleet total	By	Bytes held by process heaps and mailboxes. A blocked consumer with a filling mailbox shows up here first.
`bzrk.ui.beam.memory.current.binary_bytes`	gauge	fleet total	By	Bytes held by refc binaries — large result cells, gzip blobs, protobuf payloads.
`bzrk.ui.beam.memory.current.ets_bytes`	gauge	fleet total	By	Bytes held by ETS tables — caches, the operator-metrics window, registries.
`bzrk.ui.beam.process.current.count`	gauge	fleet total	{process}	Live processes. One per LiveView, query stream, and task.
`bzrk.ui.beam.memory.peak.total_bytes`	gauge	worst pod	By	Trailing-window peak of the BEAM's total allocation, sampled faster than the export interval.
`bzrk.ui.beam.memory.peak.processes_bytes`	gauge	worst pod	By	Trailing-window peak of bytes held by process heaps and mailboxes.
`bzrk.ui.beam.memory.peak.binary_bytes`	gauge	worst pod	By	Trailing-window peak of bytes held by refc binaries.
`bzrk.ui.beam.memory.peak.ets_bytes`	gauge	worst pod	By	Trailing-window peak of bytes held by ETS tables.
`bzrk.ui.beam.process.peak.count`	gauge	worst pod	{process}	Trailing-window peak of the live process count.
`bzrk.ui.workflow.scheduler.enqueued`	counter	—	—	Scheduled workflow runs inserted by the ticker.
`bzrk.ui.workflow.scheduler.errors`	counter	—	—	Due schedules the ticker failed to enqueue or advance.
`bzrk.ui.workflow.scheduler.identity_revoked`	counter	—	—	Scheduled fires skipped because the owner is gone, disabled, or the gateway was unreachable.
`bzrk.ui.workflow.query.pushdown.evaluations`	counter	—	—	Merged query-condition evaluations, labelled by terminal outcome (decided_early, completed, error).
`bzrk.ui.workflow.query.early_exits`	counter	—	—	Workflow queries cancelled by an absorbing progressive condition.
`bzrk.ui.workflow.query.early_exit.coverage`	histogram	—	1	Scan coverage (0-1) at which an absorbing condition cancelled the query.
`bzrk.ui.workflow.query.early_exit.iterations`	histogram	—	{iteration}	Complete result iterations seen before an absorbing condition cancelled the query.
`bzrk.ui.workflow.scheduler.lag_seconds`	gauge	worst pod	s	Age of the oldest due next_fire_at among enabled schedules (0 when none are overdue).
`bzrk.ui.source_control.operations`	counter	—	—	Native source-control operations by provider and operation.
`bzrk.ui.source_control.failures`	counter	—	—	Failed native source-control operations; never labeled with repository or credential data.
`bzrk.ui.source_control.operation.duration_seconds`	histogram	—	s	Native source-control API operation duration.
`bzrk.ui.source_control.webhook.deliveries`	counter	—	—	Native provider webhook delivery outcomes.
`bzrk.ui.source_control.ci.exports`	counter	—	—	Customer CI telemetry export outcomes.

S3

Metric	Type	Across pods	Unit	Description
`bzrk.s3.upload_bytes_total`	counter	—	By	Bytes uploaded to S3 (PUT / multipart part bodies). Throughput = rate()/1048576 MiB/s.
`bzrk.s3.download_bytes_total`	counter	—	By	Bytes downloaded from S3 (GET / ranged-read bodies). Throughput = rate()/1048576 MiB/s.
`bzrk.s3.failures_total`	counter	—	—	S3 operations that failed, by operation + failure kind. Excludes not_found (normal for nursery poll-misses; see requests_total). Reliability signal for the S3 backend.
`bzrk.s3.requests_total`	counter	—	—	All S3 operations attempted, by operation + terminal outcome. Denominator for error ratio (failures_total / requests_total) and the only place not_found is visible. `upload` counts whole-object operations — a single PUT, or a completed multipart upload — so it reads as files per second. Individual parts count under `upload_part`, which keeps the failures/requests ratio coherent within each, since a part can fail without failing its file. There is deliberately no part latency to go with it — see `upload_duration_ms`. `abandoned` is a download dropped because every reader had gone before it finished (#4145); it stays out of failures_total. Not alerted directly (the alert sums all series).
`bzrk.s3.ttfb_ms`	histogram	—	ms	Time to an S3 GET's response headers: connect plus the store's own think time, body excluded. Pair it with `body_transfer_ms` to separate "the store is slow to respond" from "the path is slow to transfer", which is the decomposition that identifies a TCP slow-start-bound read. Only a read has a separately measurable headers phase, so `download` is the only value — a PUT has no headers-then-body split and lands in `upload_duration_ms`. The label stays because a HEAD, if it is ever timed, is a headers-only operation and belongs here rather than there. Recorded on all three S3 paths, which matters because only one of them reaches the `RateLimitedStore` wrapper that the OTLP `s3.request.duration_seconds` is emitted from: the wrapper itself, the segment cache's raw client, and the segment fetcher/uploader, which also hold a raw client plus a standalone limiter and carry the highest-volume transfers. Coverage here is therefore wider than on the OTLP metric.
`bzrk.s3.upload_duration_ms`	histogram	—	ms	One complete PUT of a whole object, body included. Unlabelled: a write has no second phase to name, and no other operation has a whole-operation duration worth keeping. Read it as a latency, not a throughput. The value is roughly `size / rate + roundtrip`, so its percentile is largely a percentile of object sizes; it earns its place at the small end, where the duration is the roundtrip and a few-hundred-byte control write has no meaningful bandwidth. For anything moved in bulk, `transfer_bytes_per_second{operation=upload}` is the size-independent view. A multipart part has no duration of its own. Its payload is a fixed `part_size`, so a part's duration and its bandwidth are the same measurement — `transfer_bytes_per_second` states it directly, and stays interpretable if `part_size` ever varies with file size. Nothing times a multipart operation end to end either, so a large segment's total upload time is unmeasured by design: it would be size-dependent and no more interpretable than this number is.
`bzrk.s3.body_transfer_ms`	histogram	—	ms	Time to drain an S3 GET response body after the headers arrive. High here with a low `ttfb_ms` means the store answers promptly and the path is the constraint — for a block-sized read that is usually RTT times the number of TCP slow-start round trips, which no single latency number would reveal. Only the download direction has a separately measurable body phase; uploads are covered by `upload_duration_ms`. Covers both cache read shapes — the buffered reads and the streaming multi-block drain, which carries the largest ranges. On the streaming path only time blocked on the network is counted, excluding our own CRC and write work, so the two shapes stay comparable. Recorded on a fully drained stream only: a truncated transfer would report less time than the transfer it failed to finish, and those show up as `fetch_duration_ms{outcome=timeout\|error}` instead.
`bzrk.s3.fetch_duration_ms`	histogram	—	ms	End-to-end cold-read latency in the segment cache, by mode and outcome — what a query actually blocks on when a block is not resident. The `first_block` + `success` series is the user-facing number. Bucket bounds are kept identical to the OTLP `bzrk.segment_cache.s3_fetch_duration_ms` it promotes, so percentiles are comparable across the two tiers.
`bzrk.s3.permit_wait_ms`	histogram	—	ms	Time a fetch spent waiting for a process-wide S3 permit. This queue is invisible to `fetch_duration_ms`, whose timer starts only once the permit is held, so it is the signal that distinguishes "the object store is slow" from "we are throttling ourselves and the store was never asked". Rising here with `permits_in_use` pinned at its ceiling means the concurrency cap is binding. `abandoned` samples are waits cancelled by the fetch deadline, i.e. permit starvation rather than a slow store.
`bzrk.s3.permits_in_use`	gauge	worst pod	—	S3 fetch permits currently held process-wide. Read against the configured ceiling (`max_total_concurrent_s3_requests`): sustained saturation with a rising `permit_wait_ms` is the "concurrency cap too low" signal, whereas idle permits during slow fetches point at the path rather than at us.
`bzrk.s3.retries_total`	counter	—	—	S3 requests retried after the store returned a retryable failure. Genuine backend trouble: the request reached S3 and S3 (or the network) failed it. Excludes retries caused by our own rate limiter — see local_backoff_total. Recorded in every retry loop, not just `RateLimitedStore`'s: the segment fetcher and uploader run their own, and they carry the bulk of the traffic, so omitting them would leave this alert quiet through a retry storm on segment transfers. `CreateMultipartUpload` and `CompleteMultipartUpload` have no inner retry loop — the whole upload is reattempted by an outer wrapper — so they are attributed at the failure site, which is the only place the S3 error kind is still known. That accounting counts the failure that triggers the retry, so an exhausted sequence contributes one more sample than the inner loops would; at most one per upload, and immaterial to a rate-based alert.
`bzrk.s3.local_backoff_total`	counter	—	—	Backoff delays applied by OUR OWN AIMD rate limiter before the request reached S3. Counts delays, not requests — one request held back several times increments this once per delay, which is what makes it comparable with retries_total. Not an S3 fault and deliberately not alerted: it is the "we are the bottleneck" counterpart to retries_total. A high ratio of this to retries_total means the limiter has backed itself down and the store is barely being asked — tune the limiter, not the object store. Recorded in exactly one place — `S3RateLimiter::try_acquire`, on the branch where it denies a token — so every shape of local backpressure is counted by construction: the blocking `acquire` poll (segment fetcher, single-PUT uploads), the non-blocking deferrals in the multipart uploader and on the cache cold-read dispatch path, and the outright rejection returned by the rate-limited multipart `put_part`. Callers must not count it again. Speculative `has_capacity` checks consume no token and are deliberately not counted. The unit is one per denied unit of work, not one per scheduled wake-up.
`bzrk.s3.connections_opened_total`	counter	—	—	TCP connections established to the object store. Against `requests_total` this is the reuse rate: `1 - rate(connections_opened_total)/rate(requests_total)`. A pooled connection never reaches the connector, so counting the connector is counting new connections. Counted on success, not when the attempt starts: a connect that fails or is abandoned mid-flight never becomes a connection and so never produces a matching close, and counting it here would make `connections_opened_total - connections_closed_total` — the live count — climb without bound. A rate approaching `requests_total` means the pool is not reusing anything, which costs a TCP and TLS handshake per request; the usual cause is `pool_idle_timeout` above the endpoint's own idle timeout, or bodies being abandoned (see `connections_closed_total{reason="body_abandoned"}`).
`bzrk.s3.connections_closed_total`	counter	—	—	Connections retired, by cause. The split is what makes churn diagnosable rather than merely visible. `server_closed` is the endpoint hanging up first — if it dominates, our `pool_idle_timeout` is longer than theirs and every idle connection is a request waiting to be replayed. `body_abandoned` is a response body dropped before it drained, which costs the whole connection because an HTTP/1.1 connection cannot be reused mid-message; the segment cache does this deliberately when every reader has gone (#4145), so it must stay distinguishable — otherwise a rise reads as an endpoint fault. `idle_timeout` is the reaper doing its job and is not a fault at all. `idle_evicted` means `pool_max_idle_per_host` is binding.
`bzrk.s3.connect_duration_ms`	histogram	—	ms	Cost of establishing a connection, split by phase, recorded only on a pool miss. `total` is carried explicitly rather than left to be summed from the others: it spans the whole establishment, so it also carries time spent on candidate addresses that stalled before a later one succeeded, and summing the phases would report a millisecond connection after a multi-second failover. The split earns its series: DNS runs on the shared blocking pool, the same one the segment cache uses for its writes, so resolution can queue behind write-back and present as a slow object store — `resolve` is where that becomes visible instead of mysterious. `tls` is absent for plaintext endpoints rather than recorded as zero. `via` says whether the connection went through an egress proxy, and when it did, `resolve` and `tcp` are the proxy's rather than the endpoint's and `tunnel` is the `CONNECT` exchange with it — without the split, a slow proxy reads as a slow object store. `proxy_tls` is the handshake with an `https://` proxy, which is a different peer from the one `tls` measures: two certificates are verified on such a connection, the proxy's on the way in and the endpoint's inside the tunnel, and only the second says anything about the store. Multiplied by `connections_opened_total` this is what poor reuse actually costs.
`bzrk.s3.connection_age_ms`	histogram	—	ms	How long a connection lived, sampled when it closes. The pool-churn signal: a distribution clustered near `pool_idle_timeout` means connections are being retired by our own reaper before the next request wants them, while a cluster well below it means something else is ending them — read `connections_closed_total{reason}` to find out which.
`bzrk.s3.connect_failures_total`	counter	—	—	Candidate addresses that failed to yield a connection, by address family and cause. Counted per address, so a non-zero rate is not by itself a failed request: establishment walks every resolved address, both families at once, and a later candidate usually succeeds. That is exactly why this has to be counted — without it a family that is entirely unreachable is invisible, because the request still completes on the other one and no other metric moves. The `family` split is the point: a rate concentrated on one family means the endpoint publishes addresses this pod has no route to, which costs a failed connect on every pool miss and turns into hard errors wherever the caller disabled retries. `timeout` rather than `tcp` means the address is blackholed instead of refused; `tls` means the peer accepted the connection and then failed the handshake, and `tls_timeout` that it accepted and then simply stopped — reachable and wedged, a fault above the network rather than in it. `reason=proxy` is an egress proxy that refused to open the tunnel, which is its policy or its credentials rather than anything about the endpoint. `via` carries the rest of that distinction: when it is `proxy` the address that failed is the proxy's, so without it a proxy outage reads as the object store being unreachable — and an install can have some endpoints proxied and others exempted by `NO_PROXY`.
`bzrk.s3.request_replays_total`	counter	—	—	Requests re-sent on a fresh connection because a pooled one turned out to be dead. Only ever counted when hyper hands the request back, which it does only if not one byte was serialised, so a replay is safe even for a PUT. This is the race a liveness check cannot close: the connection passed screening and the peer's FIN landed before the write. Small and non-zero is normal; a rate that tracks `requests_total` means nearly every pooled connection is stale by the time it is used.
`bzrk.s3.transfer_bytes_per_second`	histogram	—	By/s	Throughput of one S3 transfer. Size-independent, unlike a duration, so it is comparable across services and stores where `upload_duration_ms` on a whole object is largely a percentile of object sizes — and it is the only view of download transfer speed, since the body drain reaches no other metric. Covers every production transfer: whole-object uploads, one multipart part, ingest's batch PUTs, both segment download paths (the fetcher and `get_to_file`), and the segment cache's own reads — streaming and buffered alike. What is left uncovered on the write side is control-plane records — stream STOP markers, the rewrite journal — not payload. A `RateLimitedStore` GET draws no sample, because it is reported before its body is read; the body it goes on to drain is sampled by whichever path drains it. Only a duration that is the transfer produces a sample — callers declare what their clock covers (`TransferTiming`), since a duration that also spans retry backoff or cache-file writes would report our own delays as a slow store. On `download` that declaration makes this a store-blocked rate, and the distinction is not academic: the sample divides by time spent waiting on the store, excluding the local writes that overlap it, so it answers "how fast does the store deliver when asked" rather than how fast the fetch went. It is a ceiling, never achieved throughput. The gap is largest exactly where a reader is most tempted to trust the number — a service reading small objects spends most of each drain writing, so little of the transfer is visible to the clock. Measured on dev across the four services, one read p50 sat above 150 MB/s and its throughput estimate peaked near 800 MB/s on a path nowhere near that capable, while a service reading large objects over the same link reported a fifth of it. Read as a speed those numbers are nonsense; read as ceilings they are correct. So the informative quantity is the gap against wall-clock throughput, not this figure alone: a high rate here beside a slow fetch is the signature of our own serialisation rather than a slow store, which is the whole question in #4226. One number cannot separate the two. Samples below 256 KiB are dropped: under that size the elapsed time is mostly round trip, so the ratio measures latency in the wrong units. The floor sits just under the smallest object moved in bulk (a baby segment). Bounds run from 0.25 MiB/s to 1 GiB/s: the low end is where a path is unusable rather than merely slow, and the top is above a 1 Gbps NIC, so a saturated link lands inside the range instead of in `+Inf`.
`bzrk.s3.allowed_rate`	gauge	one series per pod	—	Requests per second the AIMD limiter currently permits itself — the "what we allow" half of capable-vs-delivered, where `ttfb_ms`, `upload_duration_ms` and `transfer_bytes_per_second` are the "what the path gave us" half. Falling here while the store looks healthy means we backed ourselves off; read it with `local_backoff_total`, which counts the delays this ceiling caused. The minimum across the process's limiters that are not idle, rather than one series per limiter: the operator tier admits no `endpoint`/`bucket` label (open cardinality), and the lowest active ceiling is the one actually holding traffic back. A rate recovers only when a request is weighed against it and registry entries are never removed, so an idle bucket would otherwise hold a stale low rate and report a busy one as constrained. Per-target detail stays on the OTLP `s3.throttle.*` instruments. Reads 0 on a pod holding no limiter, and before the emitting process's first S3 request: every declared series renders whether or not it has been written, so there is no way to express "not applicable". Under the production `grpc` transport that includes the query pod itself — its fetches run in the qws pod's `cache_server`, which is where to read this.

CacheSeeder

Metric	Type	Across pods	Unit	Description
`bzrk.cache_seeder.seed_outcomes`	counter	—	—	Producer-side merge cache-seed attempts, by outcome (seeded \| no_owner \| skipped) and kind (nursery \| janitor). A rising no_owner/skipped share means merged segments are landing cold, so the first query pays an S3 fetch.

Service Metrics

On this page