Ingest Endpoints

Senders reach Berserk through the ingest service, on a host separate from the gateway. Ingest serves the OTLP API, the Loki Push API and the Datadog API, each on a listener: a port of the ingest Service. An Ingress routes to a listener, and Settings → Endpoints shows the published address. Sender configuration: Ingest.

ListenersCopy Anchor Link

An Ingress path selects a listener with portName.

IngressesCopy Anchor Link

The chart has three Ingresses for ingest, all off by default: ingest.ingress, ingest.httpIngress and ingest.datadogIngress. The OTLP API over gRPC needs an Ingress that speaks HTTP/2 to the backend:

• nginx: annotate ingest.ingress with nginx.ingress.kubernetes.io/backend-protocol: "GRPC". The annotation applies to every path of that Ingress, so route the OTLP API over HTTP and the Loki Push API through ingest.httpIngress, on the same host.
• Traefik: set traefik.ingress.kubernetes.io/service.serversscheme: h2c on the Service. Traefik ignores it on an Ingress, and every external request then fails the HTTP/2 preface check with a 500. The chart sets it on the ingest Service by default, so one Ingress can route to every listener, with a portName per path.

For nginx, on one host:

ingest:
  ingress:
    enabled: true
    className: nginx
    annotations:
      nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
      nginx.ingress.kubernetes.io/proxy-body-size: "16m"
    hosts:
      - host: ingest.example.com
        paths:
          - path: /
            pathType: Prefix
            portName: otlp-grpc
    tls:
      - hosts: [ingest.example.com]
        secretName: ingest-tls
  httpIngress:
    enabled: true
    className: nginx
    annotations:
      nginx.ingress.kubernetes.io/proxy-body-size: "16m"
    hosts:
      - host: ingest.example.com
        paths:
          - path: /v1/traces
            pathType: Exact
            portName: otlp-http
          - path: /v1/logs
            pathType: Exact
            portName: otlp-http
          - path: /v1/metrics
            pathType: Exact
            portName: otlp-http
          - path: /loki/api/v1/push
            pathType: Exact
            portName: loki
    tls:
      - hosts: [ingest.example.com]
        secretName: ingest-tls

nginx sends the Exact paths to their HTTP listeners, and every other request on the host, the gRPC methods included, to the gRPC listener.

Request body sizeCopy Anchor Link

Ingest accepts requests up to 16 MiB on the wire (request size limits), and an Ingress or proxy in front of it must accept bodies that large. nginx-ingress defaults to 1 MiB: set nginx.ingress.kubernetes.io/proxy-body-size: "16m" on each ingest Ingress, as in the example above. Envoy, Contour, AWS ALB and the GCP HTTP(S) load balancer have no body limit, or one above 16 MiB.

Loki Push APICopy Anchor Link

The Loki Push API listener is on by default and accepts POST /loki/api/v1/push. To publish it, add that path to ingest.httpIngress with portName: loki, as in the example above; the chart has no Ingress of its own for it. Keep the path off an nginx Ingress annotated backend-protocol: "GRPC", which would send gRPC to the listener. To turn the listener off, set ingest.config.lokiEnabled: false.

Datadog APICopy Anchor Link

The Datadog API listener is off by default. Turn it on and route it:

ingest:
  config:
    datadogEnabled: true
  datadogIngress:
    enabled: true
    className: <same as ingest.ingress.className>   # optional; empty uses the cluster default
    hosts:
      - host: datadog.example.com
        paths:
          - path: /
            pathType: Prefix
            portName: datadog
    tls:
      - hosts: [datadog.example.com]
        secretName: datadog-tls

The example gives the Datadog API a host of its own, since Datadog clients append many paths to their endpoint. To share a host, route these Exact paths to portName: datadog:

/api/v2/logs, /api/v2/series, /api/intake/metrics/v3/series, /api/beta/sketches, /api/v0.2/traces, /api/v1/validate, /_health, /api/v1/check_run, /intake/, /api/v1/metadata, /api/v0.2/stats

The chart refuses datadogIngress.enabled without datadogEnabled. Senders use an ingest token as their Datadog API key: Datadog Agent & Vector.

Endpoints in SettingsCopy Anchor Link

Settings → Endpoints shows senders the addresses set under ui.config.ingest, with a configuration snippet for each. Set them to the addresses your Ingresses publish, which can differ from the ingest Service ports. An empty one is listed as not published.

ui:
  config:
    ingest:
      otlpGrpc:
        host: ingest.example.com
        port: 443
        tls: true
      otlpHttp: "https://ingest.example.com"
      loki: "https://ingest.example.com"
      datadog: "https://datadog.example.com"

• otlpGrpc is the authority a gRPC exporter dials; tls says whether that address terminates TLS.
• otlpHttp, loki and datadog take a base URL: scheme, host and port. The page appends each API's path.

Anonymous ingestCopy Anchor Link

Every sender presents an ingest token of its own, made with bzrk ingest-token create, and by default ingest refuses data without one. With anonymous ingest on, Meta answers such data with the default ingest token, which routes to defaultDatabase and defaultTable and which nothing outside Meta holds:

global:
  ingestToken:
    enabled: true
    tokenName: "default_ingest_token"
ingest:
  config:
    requireAuth: false

Both settings are needed: requireAuth: false lets ingest pass such data on, and enabled: true lets Meta accept it. An ingest reachable from outside keeps requireAuth: true, so it refuses anonymous data while an internal one accepts it.

Turning anonymous ingest off retires the rows Meta registered; the API cannot delete them. Tokens from the former ingest-token Secret, used before release 1.1.12, are ordinary tokens and work until you delete them.

AcknowledgementCopy Anchor Link

ingest.config.ackMode sets when ingest acknowledges a request and what it does when its buffer is full. Unset, it is durable.

The fast modes are at-most-once: a request acknowledged and then lost before its upload, in a crash for example, is counted by ingest but not reported to the sender. They remove head-of-line blocking for Loki Push API clients that push for many senders, where a slow acknowledgement to one sender stalls them all. They need a pod memory limit, which sizes ingest's buffer; the chart sets ingest.resources.limits.memory to 2Gi by default. Without a limit, ingest sizes the buffer from the node's memory and logs a warning at startup.

In durable mode an acknowledgement means the data is stored (delivery semantics). In the fast modes it means only that ingest received the data, and the OTLP API over HTTP and the Loki Push API answer 202.

BatchingCopy Anchor Link

Ingest collects requests into batches and uploads each batch to S3 once it reaches a size or an age:

maxBatchAgeSecs can only be lowered. Ingest refuses to start unless the batch age, 20 seconds of upload retries and 2 seconds of slack fit inside its 25-second acknowledgement timeout, and 2 is the largest value that fits.

StreamsCopy Anchor Link

A stream is a sequential write path in S3. Ingest registers a stream with Meta at startup and writes all incoming data to it, from any number of senders and tokens. Data for different tables goes into the same upload; the Nursery routes it. Meta can assign an ingest instance more than one stream, for example after a restart or while scaling. Streams need no configuration.

Reading ingest's logsCopy Anchor Link

S3 answers a small share of requests with 500 or 503. Ingest keeps the batch and retries it, up to 4 times within 20 seconds, and logs one WARN line per affected batch:

transient S3 upload failure; batch retained and retried automatically, no data lost
  stream_id=… bytes=… num_requests=… retry_budget_ms=… cause=Generic S3 error: …

These lines alone do not mean data was lost. cause carries S3's response on one line, which tells a 500 from throttling or a credentials problem.

Two signals mean something is wrong:

• retry budget exhausted; throttling collector (ERROR): the retries did not succeed in time, so the collector is told to hold the data and retry later. The data stays in the collector's queue. A sustained run of these means S3 is unhealthy.
• A rising upload_fatal on the heartbeat: an upload failed in a way retrying cannot fix.

The heartbeat is one INFO line per interval, tagged tick=ingest, with the rates upload_transient, retry_budget_exhausted, pending_retries and oldest_pending_age_s.