OpenTelemetry Collector

The OpenTelemetry Collector is the reference shipper for Berserk: it speaks OTLP natively for all three signals, and its persistent queue provides the durability layer that Ingestion expects from a client. This page is the recommended configuration and the reasoning behind each setting.

For the ingest contract itself — tokens, ports, response codes, the wire cap — see Ingestion. For Fluent Bit, see Fluent Bit.

Recommended Configuration

Below is the recommended default configuration for sending data to Berserk. Your setup may vary depending on your environment and use case, but these settings are a good starting point.

# Disk-backed queue so buffered data survives collector restarts.
extensions:
  file_storage/queue:
    directory: /var/lib/otel/queue

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

exporters:
  otlp/berserk:
    # TLS endpoint (https): leave TLS enabled — do not set insecure: true.
    endpoint: "<your-ingest-endpoint>:4317"
    # Plaintext endpoint only (e.g. in-cluster ingest:4317):
    #   tls:
    #     insecure: true
    headers:
      authorization: "Bearer <your-ingest-token>"

    # OTLP payloads typically compress 3-5x with gzip (ratio depends on
    # payload redundancy). This cuts network egress, shortens per-request
    # wall time, and usually lets larger batches fit under the ingest
    # service's 16 MiB wire cap.
    compression: gzip

    # Berserk ingest service may hold each request until its batch window closes (up to 2s)
    # and the S3 upload completes. 30s stays above the ingest service's
    # internal ack timeout so it can return a real retryable error
    # instead of the collector timing out first.
    timeout: 30s

    sending_queue:
      # Disk-backed so buffered data survives collector restarts.
      storage: file_storage/queue

      # Concurrency ceiling, not a constant: idles at ~1, ramps only under
      # backlog. Keep num_consumers × signals (fleet-wide) under ingest capacity.
      num_consumers: 4

      # Outage buffer: peak_rate × tolerated_outage × headroom.
      queue_size: 1073741824
      sizer: bytes

      # The single batching layer (exporter queue batcher, not a `batch`
      # processor): send Berserk fewer, larger requests.
      batch:
        sizer: bytes
        min_size: 4194304     # 4 MiB
        flush_timeout: 1s     # ~3s end-to-end (server adds ~2s)
        max_size: 10485760    # under the 16 MiB wire cap

    retry_on_failure:
      enabled: true
      initial_interval: 1s
      max_interval: 60s
      # 0 = retry forever. The default 5min limit drops data after timeout.
      max_elapsed_time: 0
      multiplier: 2

processors:
  # Backpressure on receivers when approaching memory limit.
  # Prevents OOM before the disk queue absorbs everything.
  memory_limiter:
    check_interval: 1s
    limit_mib: 256
    spike_limit_mib: 64

service:
  extensions: [file_storage/queue]
  # Batching lives in the exporter's sending_queue.batch (above), not a
  # `batch` processor — see "Why these settings matter" below.
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [otlp/berserk]
    logs:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [otlp/berserk]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [otlp/berserk]

Why These Settings Matter

timeout: 30s — Berserk batches incoming data for up to 2 seconds before uploading to S3. The collector's default 5-second timeout will cause spurious failures during normal operation. 30 seconds gives headroom for S3 uploads under load and stays above the ingest service's internal 25s ack timeout — during a sustained S3 outage the ingest service answers within that window with an HTTP 429 + Retry-After (jittered, 30–45s), which the collector's exporter respects so it doesn't keep hammering the backend. Shortening the collector timeout below 25s risks dropping this signal and converting intentional throttles into generic timeouts.

file_storage/queue — Berserk's ingest service has no local durability — your collector is the durability layer. If the collector restarts with an in-memory queue, all buffered data is lost. The file_storage extension persists the queue to disk.

max_elapsed_time: 0 — Disables the default 5-minute retry limit. The ingest service returns retryable errors for backpressure (429), transient S3 failures (503), and stream recovery (503). With a disk-backed queue, the collector should retry indefinitely. Setting a limit means data is silently dropped after the timeout.

num_consumers: 4 — A ceiling on parallel export workers, not a constant level of parallelism: actual concurrency is min(num_consumers, batches ready in the queue), so it sits at ~1 when load is light and ramps toward the ceiling only under backlog. Berserk accepts a limited number of concurrent requests, so keep num_consumers × number-of-signals (summed across your collector fleet) modest, and prefer larger batches (below) over more consumers for throughput. Raise it only if a backlog won't drain after an outage — and note the queue also fills when Berserk is slow, not just when you're busy, so an oversized value mainly piles extra requests onto an already-busy endpoint.

sending_queue.batch — Batches on the collector so Berserk receives fewer, larger requests. Berserk holds each request open until its data is durably stored, so fewer requests means less waiting and less throttling under load. min_size coalesces into fuller requests; flush_timeout: 1s bounds the added latency (≈3 s end-to-end, since Berserk adds up to ~2 s of its own batching); max_size keeps requests under the 16 MiB wire cap. This is the exporter-helper queue batcher — use it instead of the legacy batch processor (next).

memory_limiter — Applies backpressure to receivers when the collector approaches its memory limit. Without this, if the ingest service is slow and the queue is filling, the collector can OOM before the disk queue absorbs everything.

No batch processor — Batch via the exporter's sending_queue.batch (above), not the legacy batch processor. The processor buffers data before the persistent queue, so anything it holds is lost on a collector restart, and it is on the upstream deprecation path in favor of exporter-helper batching. This is not "don't batch on the collector" — batching there means Berserk handles fewer requests; it just belongs in the durable queue layer, not a pre-queue processor.

compression: gzip — OTLP payloads typically compress 3–5× with gzip (ratio varies with payload redundancy). Enabling it cuts network egress, shortens per-request wall time, and usually lets larger batches fit under the per-request wire cap.

Configuration Variations and Their Consequences

The recommended config above is one point in a space. The two knobs that actually decide what you can lose are the queue storage and the retry budget — here is what each combination buys you:

Two further consequences that follow from the semantics above:

• The retry loop occupies a queue consumer for its whole wait. With max_elapsed_time: 0 during a long outage, up to num_consumers batches sit in memory retrying while the backlog accumulates in the queue behind them. That's by design — but it's why num_consumers, memory_limiter, and queue_size should be set together, not independently.
• Receivers have their own loss modes. A persistent exporter queue only protects data after it is enqueued. If you tail files (filelog receiver), also configure its storage: for offset persistence — otherwise a collector restart can re-read (duplicates) or skip (loss) file content regardless of how good your exporter queue is.

Staying Under the Wire Cap

Berserk accepts OTLP requests up to 16 MiB on the wire. Two exporter knobs keep you there:

• compression: gzip — the easy win, and already in the config above.
• max_size on the exporter's sending_queue.batch — cap each request below the ceiling.

Requests over the cap come back as 413 (HTTP) or InvalidArgument (gRPC), which the exporter treats as permanent and drops.