Observability

A Berserk installation exposes two independent surfaces for monitoring its own health:

The two are independent — enable either or both. Use the Prometheus endpoint to feed the monitoring stack you already run; use OTLP export for full-fidelity telemetry, including sending Berserk's telemetry back into Berserk itself.

Prometheus metrics endpoint

Every Berserk pod serves GET /metrics in Prometheus text exposition format on an internal HTTP port. A Prometheus (or any compatible scraper) running in your cluster can ingest Berserk's health metrics into your existing monitoring — no Prometheus Operator, no CRDs, and no Berserk-side configuration required.

The endpoint is served unauthenticated on the pod IP, inside the cluster only: every service is ClusterIP, and the ports that can be exposed publicly (the gateway edge and ingest's OTLP ports) deliberately do not carry /metrics.

The gateway's listener port is configurable via gateway.metricsPort (set it to "" to disable the listener entirely); if you change it, update the prometheus.io/port annotation in gateway.podAnnotations to match.

What the endpoint exposes

/metrics carries a curated operator tier — a deliberately small set chosen for dashboards and alerting rather than the full instrumentation Berserk emits over OTLP:

• Request and error rates, ingest throughput, bytes in and out
• Queue depths, merge backlog, in-flight query gauges
• Cache occupancy and hit rates
• S3 request, failure, and bandwidth counters
• Latency histograms (query execution, S3, upstream calls)
• Process and container basics: CPU, working-set memory, file-descriptor usage, uptime

Values are pod totals. On the multi-process pods (query, nursery) the short-lived per-query worker processes record into shared memory owned by the pod's main process, so their contributions are included even though a scrape interval could never catch them alive. Each pod exposes only the metrics of its own role.

The full-fidelity metric set — per-query and per-segment detail the operator tier deliberately drops — continues to flow over OTLP export; the Service Metrics reference lists it in full.

Naming conventions on the scrape surface:

• Every metric is prefixed bzrk_, and dots in the OpenTelemetry name become underscores: bzrk.query.errors → bzrk_query_errors.
• Counters generally do not carry a _total suffix (bzrk_query_errors, bzrk_nursery_bytes_ingested); a few do (bzrk_ingest_invalid_otlp_total, bzrk_*_cpu_seconds_total). Use the names as the endpoint reports them.
• Histograms are classic explicit-bucket: _bucket{le=...}, _sum, _count — use histogram_quantile() over the _bucket rate for percentiles.

Scrape configuration

Each pod carries the conventional scrape annotations, set by default in the Helm chart's per-service podAnnotations:

prometheus.io/scrape: "true"
prometheus.io/path: /metrics
prometheus.io/port: "<port from the table above>"

If you already run annotation-based pod discovery cluster-wide, Berserk pods are picked up without any changes. Otherwise, a vanilla kubernetes_sd_configs pod-discovery job is all you need:

scrape_configs:
  - job_name: berserk
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names: [<berserk-namespace>]
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels:
          [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      - source_labels: [__meta_kubernetes_pod_label_app]
        action: replace
        target_label: service

The final rule attaches a service label (gateway, meta, query, ...) to every sample — the label to aggregate alerts and dashboards on.

Metrics ports under network policies

When network policies are enforced, the metrics ports are not opened by default: the correct ingress source is your monitoring namespace, which the chart cannot know, and an allow-any rule would defeat the policy. Add an allowance scoped to your monitoring namespace:

global:
  networkPolicy:
    enabled: true
    additionalIngress:
      - from:
          - namespaceSelector:
              matchLabels:
                name: monitoring
        ports:
          - port: 9502 # gateway, janitor
            protocol: TCP
          - port: 9511 # query
            protocol: TCP
          - port: 9530 # nursery
            protocol: TCP
          - port: 9550 # ingest
            protocol: TCP
          - port: 9560 # meta
            protocol: TCP
          - port: 9571 # ui
            protocol: TCP
          - port: 9580 # permissions
            protocol: TCP

For tighter scoping, use each service's networkPolicy.additionalIngress with just that service's port — see Custom Rules.

Exporting Berserk's own telemetry

Berserk services emit their own logs, metrics, and traces via OpenTelemetry. You can export this telemetry to any OTLP-compatible endpoint for full-fidelity monitoring: debugging query performance, tracking ingestion throughput, and tracing requests across services.

Export is disabled by default. Enable it in your Helm values:

global:
  observability:
    otlpEnabled: true
    otlpEndpoint: "your-collector:4317"

When enabled, all Berserk services — including the UI — send telemetry to the configured endpoint over OTLP gRPC. The OTLP stream carries the complete metric set (a superset of the Prometheus operator tier, with per-query and per-segment detail) plus distributed traces and logs. The Service Metrics reference lists every metric with its OpenTelemetry name.

When network policies are enabled, egress to the default ingest:4317 endpoint is allowed automatically; an external collector needs an explicit otlpEgress rule — see OTLP Telemetry.