Running Queries
Inspect what the cluster is executing right now with bzrk ps, tell a slow query from a stuck one, see where each query came from, and stop one
bzrk ps lists the queries your cluster is executing at this moment, and
bzrk kill stops one. Between them they answer the two questions an operator
has when a cluster is busy: what is running, and should it be.
The listing is cluster-wide. A query runs on exactly one query pod, so the
gateway asks every pod and merges the answers — a pod that cannot be reached is
left out of the result rather than failing it, which shows up as that pod's
absence from the POD column.
Reading a row
bzrk psQUERY ID POD ELAPSED QUEUED IDLE SPAN PROGRESS CHUNKS scan/need ROWS
0199f3a2-7c41-7db3-9f0e-2a1b4c6d8e10 10.0.1.5:9510 1m2s - 200ms 7d 42.0% 1200/3000 1204331
who: alice@example.com (session) from 203.0.113.7
claims: bzrk-cli 1.0.19 · cli
window: 7d 2026-08-27T09:12:00Z → 2026-09-03T09:12:00Z
search "error" | summarize count() by bin(timestamp, 1m)| Column | Meaning |
|---|---|
QUERY ID | Server-side id, and the handle bzrk kill takes |
POD | Which query pod is running it |
ELAPSED | How long since the query arrived |
QUEUED | How long it waited for an execution slot; - when it never waited |
IDLE | Since the last frame reached the client; - before the first one |
SPAN | How much time the query covers — see The window a query scans |
PROGRESS | Share of planned work done; - while still planning |
CHUNKS scan/need | Chunks scanned against chunks still to scan |
ROWS | Rows processed so far |
ELAPSED, QUEUED and IDLE are all durations of the execution; SPAN is
the duration the query asks about. A row reading 1m2s elapsed over 30d of
data is doing a lot of work; the same elapsed time over 10m of data is not.
The column that earns its place is IDLE. Elapsed time alone cannot tell a
slow query from a stuck one: a query four minutes in whose last frame was 200 ms
ago is working, and one whose last frame was three minutes ago is not. QUEUED
makes the same distinction at the other end — a query a minute old that has not
started is waiting for a slot, not running slowly, and it has no progress to
report because planning has not happened yet.
Buffered surfaces show no idle time
IDLE reads - for the whole life of a query on a buffered surface (the
Kusto REST and Loki endpoints), which delivers its result once at the end.
Nothing is wrong; there are simply no frames to be idle since.
The window a query scans
SPAN is how much time the query covers, and it is often the difference between
an expensive query and a mistake. Nothing else in the row distinguishes a
ten-minute lookback from a thirty-day one: both can sit at the same elapsed
time, the same progress and the same chunk counts for a while.
The window: line under the row gives the bounds behind that span, because a
30-day window ending last week is not the same query as one ending now. A query
bounding the ingest-time axis gets an ingest: line on the same shape, and the
column marks an ingest-only span as i30d so it is not read as an event window.
The window is the server's own resolution of what the request carried — relative
bounds like 24h ago are already resolved against the now captured when the
request arrived — so it is not a client claim and is not grouped with one.
A self-bounded query reports no span
SPAN reads - when the request imposed no window and the query carries its
own bounds in KQL (| where timestamp > ago(1h)). The coordinator derives
that window while planning, after the listing entry is made, so the pod
answering bzrk ps genuinely does not know it. Passing --since (or the
equivalent on your surface) is what makes the span visible.
--full widens the table with every execution statistic the engine reports —
slices, chunk coverage, skip reasons by kind, bytes scanned and skipped, fetch
and snapshot time, state-cache reuse — the same set bzrk search --stats prints
for a finished query.
Where a query came from
Under each row are up to three lines: who asked, what the client says it is, and the query itself.
who: grafana-dev (service_token) from 203.0.113.7
claims: grafana-plugin 1.4.0 · dashboard · Ingest health › p99 latency · user alice
search "error" | summarize count() by bin(timestamp, 1m)who: is what the server established. The gateway authenticated the
caller, resolved the credential, and injected the result on its way to the query
pod. A client cannot assert any of it: the header namespace these travel in is
stripped from every inbound request at the public edge, so a caller that sends
them has them removed before anything reads them.
| Field | |
|---|---|
| principal | Email for a user, principal name for a service |
| authn method | session, cli_token, service_token or trusted_proxy |
| client address | Left-most X-Forwarded-For from the trusted edge proxy, else the TCP peer |
claims: is what the client said about itself, and nothing verifies it.
It is worth reading anyway — a dashboard title is usually the fastest answer to
"why is this running" — but it is a self-report, which is why the listing labels
it rather than mixing it into the line above.
| Field | |
|---|---|
| app + version | The program that issued the query |
| surface | cli, dashboard, explore, alerting, … |
| dashboard › panel | Title when the client sent one, uid or panel number otherwise |
| user | The human behind a shared credential |
| agent | The agent driving the client, when one is |
The user row is what makes Grafana legible. The Berserk datasource plugin
authenticates the whole organization with one service principal, so who:
says grafana-dev for every panel in the company; claims: … user alice is the
only thing distinguishing one person's dashboard from another's. The plugin
takes the login from Grafana itself rather than from anything the browser can
set, but it is still the plugin's word, not the server's — hence the label.
Either line is omitted when there is nothing to say. A service calling a query
pod directly, inside the cluster, passes no gateway and so has no who: at all;
a client that reports nothing about itself has no claims:.
The listing is not private
Any authenticated caller sees every running query, including its full text and
the identity that submitted it. Treat bzrk ps output the way you would treat
the query log.
Structured output
--json keeps the two halves in separate objects, so nothing consuming the
output can mistake a self-reported user for a verified one by reaching for the
wrong key:
{
"query_id": "0199f3a2-7c41-7db3-9f0e-2a1b4c6d8e10",
"pod": "10.0.1.5:9510",
"query": "search \"error\" | summarize count() by bin(timestamp, 1m)",
"attested": {
"principal_type": "service",
"principal_name": "grafana-dev",
"authn_method": "service_token",
"client_ip": "203.0.113.7"
},
"claimed": {
"app": "grafana-plugin",
"app_version": "1.4.0",
"surface": "dashboard",
"end_user": "alice",
"dashboard_uid": "abc123",
"dashboard_title": "Ingest health",
"panel_title": "p99 latency"
},
"window": {
"span_ms": 604800000,
"since": "2026-08-27T09:12:00Z",
"until": "2026-09-03T09:12:00Z",
"ingest_span_ms": null,
"ingest_since": null,
"ingest_until": null,
"now": "2026-09-03T09:12:00Z",
"bypass_state_cache": false
},
"elapsed_ms": 62000,
"idle_ms": 200
}--csv and --tsv carry the same fields, flattened to attested.principal_name
and claimed.dashboard_title — the prefix survives, so a spreadsheet keeps the
distinction too.
The query text is stored truncated (with a trailing …) so that one
pathological client cannot make the listing unreadable, and the table collapses
it to a single line. --json gives you what the pod holds.
Attributing your own client
Every field under claims: comes from one optional message on the query
request, so any client can fill it in — and none has to. Berserk's own clients
already do: the CLI reports itself, its version and its driving agent, and the
Grafana plugin reports the dashboard, panel and Grafana login on both its
streaming and its one-shot paths.
Two things are worth knowing before you add your own:
- It travels on the request, not in a header. The gateway drops the entire
x-bzrk-*header namespace on inbound so that nothing can forge the identity headers it injects itself, which means a header is not a channel a client can use. The Loki- and Kusto-compatible HTTP endpoints are the exception: a stock Grafana datasource pointed at them is attributed automatically from the dashboard headers Grafana already sends. - Nothing branches on it. Claims are display strings. They are capped in length and stripped of control characters on arrival, and no planning, routing or authorization decision reads them.
Stopping a query
bzrk kill 0199f3a2-7c41-7db3-9f0e-2a1b4c6d8e10Cancellation is cooperative. The command returns as soon as the signal is
delivered — reported as signalled, not "stopped" — and the query stops
starting new work at its next boundary rather than being interrupted mid-chunk.
The client running it sees its stream end as cancelled rather than completed, so
it can report the query as killed rather than done.
not_found means no pod is running that id, which usually means the query
finished on its own between your bzrk ps and your bzrk kill.
After the fact
bzrk ps only sees queries in flight. The same provenance is recorded on each
query's completion event, so the historical question — which dashboard is
generating the load, which principal is behind the expensive queries — is
answerable from your telemetry rather than by watching the listing. See
Observability for where those events land, and
Query Usage Reporting for the
anonymized per-query record, which deliberately carries none of this.