Observability

Berserk has two monitoring surfaces. Enable either or both:

The Prometheus endpoint feeds an existing monitoring stack. OTLP export carries everything, and can send Berserk's telemetry back into Berserk.

Prometheus metrics endpointCopy Anchor Link

Every Berserk pod serves GET /metrics in Prometheus text format on an internal HTTP port. A Prometheus, or any compatible scraper, in your cluster can collect it; it needs no Prometheus Operator, no CRDs and no Berserk configuration.

The endpoint is unauthenticated and reachable only inside the cluster: every service is ClusterIP, and the ports that can be public (the gateway edge and ingest's API ports) do not serve /metrics.

gateway.metricsPort sets the gateway's listener port; "" turns the listener off. After changing it, update the prometheus.io/port annotation in gateway.podAnnotations to match.

What the endpoint exposesCopy Anchor Link

/metrics carries the operator tier, a small set of metrics for dashboards and alerts:

• Request and error rates, ingest throughput, bytes in and out
• Queue depths, merge backlog, in-flight query gauges
• Cache occupancy and hit rates
• S3 request, failure and bandwidth counters
• Latency histograms (query execution, S3, upstream calls)
• Process and container basics: CPU, working-set memory, file-descriptor usage, uptime

Values are pod totals. On the multi-process pods (query, nursery), the short-lived per-query worker processes record into shared memory owned by the pod's main process, so their values are included. Each pod exposes only its own role's metrics.

The full metric set, with the per-query and per-segment detail the operator tier leaves out, goes over OTLP export; Service Metrics lists it. Every operator-tier metric also reaches OTLP under its dotted name, so the bzrk_s3_* metrics behind the UI's Monitoring → Storage page are queryable as bzrk.s3.*. Metrics on only one surface have different names and labels: bzrk.segment_cache.s3_fetch_* and s3.request.* are OTLP-only and do not replace them in queries.

The UI's Monitoring pages read the operator tier through an in-product collector, which scrapes every pod and keeps a short window in memory, lost on restart. For longer history, scrape /metrics with a Prometheus.

Names on the scrape surface:

• Every metric is prefixed bzrk_, and dots in the OpenTelemetry name become underscores: bzrk.query.errors → bzrk_query_errors.
• Most counters have no _total suffix (bzrk_query_errors, bzrk_nursery_bytes_ingested); a few do (bzrk_ingest_invalid_otlp_total, bzrk_*_cpu_seconds_total). Use the names the endpoint reports.
• Histograms are classic explicit-bucket: _bucket{le=...}, _sum, _count. Use histogram_quantile() over the _bucket rate for percentiles.

Scrape configurationCopy Anchor Link

The chart sets the conventional scrape annotations on every pod, in each service's podAnnotations:

prometheus.io/scrape: "true"
prometheus.io/path: /metrics
prometheus.io/port: "<port from the table above>"

Cluster-wide annotation-based pod discovery picks Berserk pods up as they are. Otherwise, use a kubernetes_sd_configs pod-discovery job:

scrape_configs:
  - job_name: berserk
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names: [<berserk-namespace>]
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels:
          [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      - source_labels: [__meta_kubernetes_pod_label_app]
        action: replace
        target_label: service

The last rule adds a service label (gateway, meta, query, …) to every sample, for alerts and dashboards to aggregate on.

Metrics ports under network policiesCopy Anchor Link

With network policies on, the metrics ports are open by default only to the peers the chart ships: the UI's in-product collector, and the bundled Prometheus when prometheus.enabled (global.networkPolicy.scrape.from). The chart does not know your monitoring namespace, so allow it:

global:
  networkPolicy:
    enabled: true
    additionalIngress:
      - from:
          - namespaceSelector:
              matchLabels:
                name: monitoring
        ports:
          - port: 9502 # gateway, janitor
            protocol: TCP
          - port: 9511 # query
            protocol: TCP
          - port: 9530 # nursery
            protocol: TCP
          - port: 9550 # ingest
            protocol: TCP
          - port: 9560 # meta
            protocol: TCP
          - port: 9571 # ui
            protocol: TCP
          - port: 9580 # permissions
            protocol: TCP

For a narrower rule, use each service's networkPolicy.additionalIngress with only that service's port: Custom Rules.

Exporting Berserk's own telemetryCopy Anchor Link

Berserk services emit their own logs, metrics and traces over OpenTelemetry, and can export them to any OTLP endpoint. Export is off by default:

global:
  observability:
    otlpEnabled: true
    otlpEndpoint: "your-collector:4317"

With export on, every Berserk service, the UI included, sends to the endpoint over OTLP gRPC. The stream carries the full metric set (the operator tier plus per-query and per-segment detail), traces and logs. Service Metrics lists every metric with its OpenTelemetry name.

With network policies on, egress to the default ingest:4317 is allowed; an external collector needs an otlpEgress rule: OTLP Telemetry.