API Introduction

Berserk exposes two query protocols: gRPC (native streaming) and HTTP (ADX v2 REST). Both run KQL on the same engine; they differ in how results are delivered.

ConnectingCopy Anchor Link

Clients connect to the gateway, the cluster's public edge, never to a query pod. The gateway authenticates the bearer token — a CLI access token from bzrk login or a service account's token — and forwards the request to the query service with a token the gateway signed. The query service refuses any query request without one: 401 over HTTP, UNAUTHENTICATED over gRPC.

The gateway forwards these query surfaces:

• gRPC under /api/grpc/, as /api/grpc/<package>.<Service>/<Method>. For example /api/grpc/query.QueryService/ExecuteQuery.
• HTTP: the Kusto REST endpoints POST /v1/rest/query and POST /v2/rest/query, at the gateway's root.

The query service's other HTTP routes are internal; the gateway does not forward them.

gRPC Streaming ProtocolCopy Anchor Link

The gRPC API returns a server-side stream of typed frames. Each frame carries one payload:

Schema → RowBatch → RowBatch → Metadata → Progress → RowBatch → Metadata → Progress → Completion

Frame TypesCopy Anchor Link

Result IterationsCopy Anchor Link

For long-running queries, the engine delivers incremental results as data is processed. Each update is a whole snapshot of the answer so far, named by a result_iteration_id on the frame — a UUID that changes with every snapshot.

When the id changes, discard all previously accumulated rows and start fresh. Each snapshot ends with one metadata frame carrying its warnings, advice and partial failures; replace what you hold with it, even when it is empty.

Snapshot "abc-123":
  RowBatch(table="PrimaryResult", rows=[...], is_iteration_complete=false)
  RowBatch(table="PrimaryResult", rows=[...], is_iteration_complete=true)
  Metadata(warnings=[...])

Snapshot "def-456":           ← new id → clear all previous rows
  RowBatch(table="PrimaryResult", rows=[...], is_iteration_complete=true)
  Metadata()                  ← no warnings now: the earlier ones are gone

RowBatch ConcatenationCopy Anchor Link

A single result iteration may produce multiple RowBatch frames for the same table because a gRPC frame is limited to ~4 MiB. The server packs rows into a batch until the encoded frame would exceed that limit and then starts the next batch, so rows per batch depend on row size, not on a fixed row count. A single row too large to fit a frame is dropped, with a warning.

Concatenate all RowBatch frames that share the same table_name and result_iteration_id:

Batch 1: table="PrimaryResult", iteration="abc", rows=[row0..row999], is_iteration_complete=false
Batch 2: table="PrimaryResult", iteration="abc", rows=[row1000..row1234], is_iteration_complete=true
→ Result: 1235 rows

The is_iteration_complete flag on the last batch signals that no more batches will arrive for this table in this snapshot. Clients should:

1. Show the first batch immediately
2. Accumulate subsequent batches
3. Treat the snapshot as whole once its metadata frame arrives
4. On a new snapshot id, clear and restart

Multi-Table ResultsCopy Anchor Link

Some queries produce multiple tables (e.g., fork queries or queries with extra diagnostic tables):

• PrimaryResult — the main result table (always first)
• ExtraTable_0, ExtraTable_1, ... — additional tables
• Fork queries use branch names (e.g., "Totals", "TopTwo")

Schema frames are sent once per table name. RowBatch frames reference their table by table_name. When an iteration ID changes, all tables are cleared together — not individually.

Schema(name="PrimaryResult", columns=[...])
Schema(name="Totals", columns=[...])
RowBatch(table="PrimaryResult", iteration="abc", rows=[...])
RowBatch(table="Totals", iteration="abc", rows=[...])

Progress FramesCopy Anchor Link

Progress frames are sent periodically during execution, independent of row delivery. They contain:

• Scan statistics: rows processed, chunks total/scanned/skipped
• Skip reasons: range, bloom filter, shard (hash), required fields
• Timing: chunk scan time, query time, merge time (nanoseconds)
• Bin progress: per-bin completion for summarize ... by bin() queries
• Planning progress: segment planning completion count
• Operator diagnostics: per-operator key-value telemetry

Use the latest progress frame — each one represents cumulative statistics.

Metadata FrameCopy Anchor Link

Exactly one metadata frame closes every snapshot, after its row batches. It is whole — sent even when empty — and replaces the previous one, so a warning a later snapshot does not repeat is gone.

• Visualization: type and properties from the render operator (e.g., timechart, piechart)
• Warnings: execution warnings like "summarize memory limit reached" or "result truncated"
• Partial failures: how many segments could not be read (segment_count) and why. The legacy segment_ids field is always empty.

HeadersCopy Anchor Link

Example: Processing a gRPC StreamCopy Anchor Link

use tokio_stream::StreamExt;

let mut stream = client.execute_query(request).await?.into_inner();
let mut tables: HashMap<String, Vec<Row>> = HashMap::new();
let mut current_iteration: Option<String> = None;

while let Some(frame) = stream.next().await {
    let frame = frame?;
    match frame.payload {
        Some(Payload::Schema(s)) => { /* store column definitions */ }
        Some(Payload::Batch(b)) => {
            // New iteration? Clear everything.
            if current_iteration.as_ref() != Some(&b.result_iteration_id) {
                tables.clear();
                current_iteration = Some(b.result_iteration_id.clone());
            }
            // Append rows to the correct table
            tables.entry(b.table_name).or_default().extend(b.rows);
        }
        Some(Payload::Progress(p)) => { /* update stats display */ }
        Some(Payload::Error(e)) => { /* handle error */ }
        Some(Payload::Done(_)) => break,
        _ => {}
    }
}

HTTP REST Protocol (ADX v2)Copy Anchor Link

The HTTP API implements the Kusto v2 frame format, making it compatible with Azure Data Explorer tooling (Grafana ADX plugin, Kusto SDKs, etc.).

Send requests to the gateway, which mounts the endpoint at its root and requires a bearer token (a CLI access token or a service account's token). Anonymous calls get 401 with a WWW-Authenticate: Bearer challenge — never a login redirect. A non-empty db is required.

Non-Progressive Mode (Default)Copy Anchor Link

Returns all frames in a single JSON array:

curl -X POST https://<gateway>/v2/rest/query \
  -H 'Authorization: Bearer <token>' \
  -H 'Content-Type: application/json' \
  -d '{"db": "default", "csl": "my_table | take 10"}'

Response:

[
  {"FrameType": "DataSetHeader", "IsProgressive": false, "Version": "v2.0"},
  {"FrameType": "DataTable", "TableKind": "QueryProperties", "TableName": "@ExtendedProperties", ...},
  {"FrameType": "DataTable", "TableKind": "PrimaryResult", "TableName": "PrimaryResult", "Columns": [...], "Rows": [...]},
  {"FrameType": "DataTable", "TableKind": "QueryCompletionInformation", ...},
  {"FrameType": "DataSetCompletion", "HasErrors": false}
]

Progressive ModeCopy Anchor Link

Enabled by setting results_progressive_enabled: true in request properties. Streams frames via chunked HTTP transfer:

curl -X POST https://<gateway>/v2/rest/query \
  -H 'Authorization: Bearer <token>' \
  -H 'Content-Type: application/json' \
  -d '{
    "db": "default",
    "csl": "my_table | summarize count() by bin(timestamp, 1m)",
    "properties": {"Options": {"results_progressive_enabled": true}}
  }'

Progressive frames use replace semantics — each TableFragment supersedes all previous fragments for the same table:

{"FrameType": "DataSetHeader", "IsProgressive": true, "IsFragmented": true}
{"FrameType": "TableHeader", "TableId": 0, "TableName": "PrimaryResult", "Columns": [...]}
{"FrameType": "TableFragment", "TableId": 0, "TableFragmentType": "DataReplace", "Rows": [[...]]}
{"FrameType": "TableProgress", "TableId": 0, "TableProgress": 45.0}
{"FrameType": "TableFragment", "TableId": 0, "TableFragmentType": "DataReplace", "Rows": [[...]]}
{"FrameType": "TableProgress", "TableId": 0, "TableProgress": 100.0}
{"FrameType": "TableCompletion", "TableId": 0, "RowCount": 150}
{"FrameType": "DataSetCompletion", "HasErrors": false}

ErrorsCopy Anchor Link

A query-level failure (parse error, bind error, execution error) that happens before the response body starts streaming returns HTTP 4xx with a JSON error body, not v2 frames:

{
  "type": "https://api.bzrk.dev/problems/query-error",
  "message": "Syntax error: ...",
  "code": "ParseError",
  "error_details": { "message": "..." }
}

Failing to resolve the database uses a different envelope, with the HTTP status repeated in the body:

{
  "error": {
    "code": "DatabaseNotFound",
    "message": "database 'nope' not found",
    "@type": "BzrkClient/v1"
  },
  "status": 404
}

In progressive mode a failure that happens after frames have already been sent cannot change the HTTP status. The response ends instead with a DataSetCompletion frame whose HasErrors is true.

HeadersCopy Anchor Link

Column TypesCopy Anchor Link

Both protocols use the same type system:

Dynamic Values (gRPC)Copy Anchor Link

In the gRPC protocol, cell values are encoded as BqlValue protobuf messages with a oneof for each scalar type plus arrays and property bags. See the proto definition for the full schema.

In the HTTP protocol, values are JSON-native — strings, numbers, booleans, nulls, arrays, and objects.