Ingestion
How data reaches Berserk, where to send it, and what the ingest endpoint answers
Berserk ingests logs, traces and metrics through three APIs: the OTLP API (OpenTelemetry Protocol) over gRPC or HTTP, the Loki Push API and the Datadog API. The OTLP API over gRPC is recommended: it is more efficient than over HTTP. The OpenTelemetry Collector, Fluent Bit, OpenTelemetry SDKs and your own clients use the OTLP API; Loki clients use the Loki Push API, and the Datadog Agent and Vector the Datadog API.
This page covers tokens, endpoints, responses and size limits. The shipper pages configure each shipper to match.
How Data Flows
- Your collector or agent sends data to Ingest.
- Each request carries an ingest token, which Ingest validates with Meta. Ingest batches the data and uploads it to S3.
- The Nursery follows each stream, downloads the batches from S3, routes rows to their tables and makes them searchable. It also merges small batches into larger segments.
What happens to a row after that: Life of a Row.
Ingest answers a request after S3 confirms the upload. When S3 is unavailable, Ingest returns an error instead of buffering, so the shipper retries and keeps the data queued until Ingest accepts it.
Ingest Tokens
Every request carries an ingest token:
Authorization: Bearer ing_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxA token routes each signal (traces, logs, metrics) to a table, all to the same table by default. The bzrk.table resource attribute overrides the routing per record.
Create a token on Settings → Databases → Tokens, or with the CLI:
bzrk ingest-token create --table default my-token--traces-table, --logs-table and --metrics-table give a signal its own table. This token sends logs and traces to logs, and metrics to metrics:
bzrk ingest-token create my-token --table logs --metrics-table metricsThe token value is shown once, when it is created.
Managed Berserk requires a token on every request. A self-hosted install can accept data without one: anonymous ingest.
APIs
Each API has its own port. Settings → Endpoints shows your installation's addresses, with a configuration snippet for each.
| API | Default Port | Senders | Managed Berserk |
|---|---|---|---|
| OTLP API over gRPC | 4317 | Recommended. Collectors, Fluent Bit and SDKs | ingest.<your-domain>:4317, TLS |
| OTLP API over HTTP | 4318 | Senders without gRPC, such as AWS Lambda | https://ingest.<your-domain>:4318 |
| Loki Push API | 3100 | Grafana Alloy, Promtail and Vector: Loki clients | Not published; ask the Berserk team |
| Datadog API | 9552 | The Datadog Agent and Vector's datadog_* sinks: Datadog Agent & Vector | Off by default; ask the Berserk team to turn it on. Then https://ingest.<your-domain>:9552 |
On a self-hosted install the operator chooses which APIs run and publishes their addresses: Ingest Endpoints.
gRPC methods and HTTP paths
These describe the wire protocol. A gRPC exporter takes an authority or SDK URL without them; an HTTP exporter appends the path to its base URL.
| Signal | gRPC method path | HTTP path |
|---|---|---|
| Logs | /opentelemetry.proto.collector.logs.v1.LogsService/Export | /v1/logs |
| Metrics | /opentelemetry.proto.collector.metrics.v1.MetricsService/Export | /v1/metrics |
| Traces | /opentelemetry.proto.collector.trace.v1.TraceService/Export | /v1/traces |
Ingest does not accept OTLP profiles; a shipper that sends them gets an error on that signal only.
Payloads may be compressed with gzip or zstd on either transport (grpc-encoding for gRPC, content-encoding for HTTP).
Delivery Semantics
This holds in the default acknowledgement mode, which Managed Berserk uses. A self-hosted install can acknowledge on receipt instead: the OTLP API over HTTP and the Loki Push API then answer 202, and an acknowledgement means only that Ingest received the data (acknowledgement).
| Response | Meaning | What your collector does |
|---|---|---|
| 200 / OK | The batch is stored in S3. | Drops the data from its queue. |
429 + Retry-After: N (gRPC: UNAVAILABLE + RetryInfo) | Backpressure. N is the server's estimate of when capacity returns. | Retries after N seconds. Standard OTLP exporters do this. |
503 (gRPC: UNAVAILABLE) | A transient server error, with no delay estimate. | Retries with exponential backoff. Standard exporters do this. |
400 / 401 / 413 (gRPC: INVALID_ARGUMENT / UNAUTHENTICATED) | Permanent: a bad payload, a bad token or an oversized request. | Drops the batch. Fix the cause. |
- An acknowledgement means the data is in S3. A slow S3 makes acknowledgements slower. Unacknowledged data stays in the collector's queue.
- Delivery is at-least-once. When a response is lost after the write landed, the collector retries, and Ingest usually drops the resend after checking content hashes on S3. If S3 stored a write whose confirmation never arrived, the data can be stored twice.
Retry-Aftervalues are jittered: 30–45 seconds during a sustained S3 outage, for example.
Latency
Ingest batches data for up to 2 seconds or 16 MiB before uploading it to S3; a self-hosted install can lower both (batching). Data is typically searchable 1–10 seconds after the collector sends it.
Request Size Limits
Ingest accepts requests up to 16 MiB on the wire. With gzip, a request typically carries 40–80 MiB of uncompressed telemetry.
A larger request gets 413 (HTTP) or InvalidArgument (gRPC). Both are permanent: the batch is dropped, not retried. To stay under the limit:
- Compress with
gziporzstdon the exporter. Fluent Bit does not compress on its gRPC path. - Lower the shipper's batch size: collector, Fluent Bit.
A proxy or ingress in front of Ingest must accept 16 MiB bodies. On a self-hosted install, see request body size.
Choosing a Shipper
| Shipper | When |
|---|---|
| OpenTelemetry Collector | The default: OTLP over gRPC for all three signals, a disk-backed queue, most processors |
| Fluent Bit | You already collect logs with Fluent Bit, or need a small footprint on an edge node |
| An OpenTelemetry SDK | Exporting from an application without a collector |
| Loki clients | You already ship logs with Grafana Alloy, Promtail or Vector's loki sink |
| Datadog Agent or Vector | You already ship to Datadog: change the endpoint and API key |
| Your own client | Any OTLP client that meets the requirements below |
Your own client
Honor Retry-After on 429, retry 503 with backoff, treat other 4xx as
permanent, keep the per-request timeout above 25 seconds, and keep
unacknowledged data queued: Delivery Semantics applies
to every client.
Verifying Ingestion
bzrk search "<your table> | take 10" --since "5m ago"The table must exist and your token must route to it: data sent with a token bound to table foo lands only in foo. unknown table means the table does not exist; create the table or fix the name. To point the CLI at your cluster, see Berserk CLI.
If no data appears, check that:
- the ingest token is valid, not revoked, and routes to the table you query
- the collector reaches the ingest endpoint on
4317(gRPC) or4318(HTTP) - the collector's logs show no export errors or retries