Ingestion

Berserk ingests telemetry — logs, traces, and metrics — via the OpenTelemetry Protocol (OTLP). Anything that speaks OTLP can send to it: an OpenTelemetry Collector, Fluent Bit, a language SDK, or your own client. Everything after that is handled automatically.

This page is the ingest contract — tokens, ports, response codes, size limits — that any shipper has to satisfy. The per-shipper pages linked above are the configurations that satisfy it.

How Data Flows

1. Your collector or agent sends data to Berserk's Ingest component over OTLP (gRPC or HTTP).
2. Your shipper includes an ingest token in each request. Ingest validates it with the Meta service, which authenticates the token. Ingest then batches incoming data and uploads it to S3.
3. The query component (Nursery) follows each stream, downloads batches from S3, routes data to the correct tables, and makes them searchable. Nursery also merges small batches into larger optimized segments in the background.

This page covers the shipper-facing half. For what happens to a record after it is accepted — when it becomes queryable, how it is compacted, and when it ages out — see Life of a Row.

Ingest holds each request open until S3 confirms the upload, then returns that result to the collector. A success response means the data is durably stored. If S3 is temporarily unavailable, Ingest returns a failure to the collector rather than buffer locally. Your shipper is the durability layer — it is responsible for retrying failed requests and persistently queuing data until Ingest accepts it.

Ingest Tokens

Every request to Ingest must carry an ingest token for authentication and routing:

Authorization: Bearer ing_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Each token is bound to a table. All data sent with that token routes to that table. You can override routing per-record by setting the bzrk.table resource attribute.

Create a token with the CLI:

bzrk ingest-token create --table default my-token

Default Ingest Token (Kubernetes)

When deploying with the Helm chart, the ingest service can be configured with a default ingest token via a Kubernetes Secret. This token is used to authenticate incoming data when no other token is provided.

Managed mode (recommended): Set global.ingestToken.managed: true in your Helm values. An init container will automatically create the token by calling Meta's API and store it in a Kubernetes Secret before Ingest starts. This is idempotent — if the secret already exists, the init container is skipped entirely.

global:
  ingestToken:
    managed: true

Manual mode: Create the secret yourself and reference it in the chart:

kubectl create secret generic ingest-token \
  --from-literal=default_ingest_token="ing_<your-token-value>"

The Helm chart references this secret by default (ingest-token with key default_ingest_token).

Streams

A stream is a sequential write path in S3. Ingest registers a stream with Meta on startup and writes all incoming data — from any number of collectors and ingest tokens — to that single stream. Data from different tokens targeting different tables is batched together in the same upload; Nursery handles the routing.

In some cases Meta may assign more than one stream to an Ingest instance (e.g., after a restart or during scaling), but typically there is just one. Streams are created and managed automatically — you do not need to configure or interact with them directly.

Protocols

Ingest accepts OTLP over both gRPC and HTTP:

Fluent Bit and other OTLP clients that address gRPC by method path use the standard OTLP service paths, all three of which Ingest serves:

OTLP profiles are not accepted; a shipper configured to send them gets an error on that signal only.

Request payloads may be compressed with gzip or zstd on either transport (grpc-encoding for gRPC, content-encoding for HTTP). Do it where your shipper supports it — see request size limits.

Delivery Semantics

What a response from the ingest endpoint means, precisely:

Three consequences worth internalizing:

• An acknowledgment is a durability guarantee. Ingest holds each request open until S3 confirms the upload — there is no window where acknowledged data exists only in a local buffer. This also means a slow S3 makes acks slower, not less safe; backpressure (429) is how Ingest tells your collector to hold data on its side, which is why the collector's queue is the durability layer for unacknowledged data.
• Delivery is at-least-once. If a response is lost in transit (network failure, timeout) after the write actually landed, your collector retries and Ingest deduplicates the resend in the common case (it verifies suspected duplicates against content hashes on S3). In rare partial-failure scenarios — S3 accepting a write whose confirmation never arrived, across a sustained outage — the same data can be stored twice. This is a known, accepted limitation; if your pipeline is sensitive to duplicates, deduplicate downstream on record identity.
• Retry-After values are jittered. Ingest spreads the delays it hands out (e.g. 30–45s during a sustained S3 outage) so that a fleet of throttled collectors does not return in one synchronized wave. Don't be surprised by varying values for the "same" condition.

Latency, Error Recovery and Durability

Request Size Limits

Berserk's ingest service accepts OTLP requests up to 16 MiB on the wire. With gzip compression enabled that typically carries 40–80 MiB of uncompressed telemetry per request — ample for busy collectors.

If your shipper emits requests above this ceiling, the ingest service returns 413 (HTTP) or InvalidArgument (gRPC) — a permanent error, so the batch is dropped rather than retried. Two fixes, in order:

• Enable compression (gzip or zstd) on the exporter — this is the easy win, where your shipper supports it (Fluent Bit does not compress on its gRPC path).
• Lower the shipper's batch ceiling so each request stays under the cap. See the per-shipper pages for the exact knob (collector, Fluent Bit).

Ingress sizing: if you terminate TLS or proxy through nginx/Envoy/Istio in front of the ingest service, check your ingress body limits. The most common trap is nginx-ingress, whose default client_max_body_size is 1 MiB — set the annotation nginx.ingress.kubernetes.io/proxy-body-size: "16m" on the ingest Ingress resource. Other ingresses (Envoy, Contour, AWS ALB, GCP HTTP(S) LB) either have no body limit or a limit well above 16 MiB, but it is worth confirming for your specific setup.

Choosing a Shipper

Verifying Ingestion

After configuring your shipper, verify data is flowing:

bzrk search "<your table> | take 10" --since "5m ago"

The table you query must exist and your ingest token must route to it — data sent with a token bound to table foo won't show up under any other table. If the query returns unknown table, create the table (or fix the name) and confirm your token's routing. If your CLI isn't pointed at the cluster yet, set up a profile and sign in first — see AI Agent Setup.

If no data appears, check:

• The ingest token is correct and not revoked, and is bound to the table you're querying
• The collector can reach your ingest endpoint on 4317 (gRPC) or 4318 (HTTP)
• The collector logs for export errors or retries