Query Usage Reporting
The anonymized per-query usage record — what it contains, why it cannot contain your data, and where it goes
Every query that finishes — completed, failed, cancelled or abandoned — produces one usage record: the shape of the query, how the engine ran it, and what it cost. Records show how a cluster is used, and can be shared with the Berserk team without sharing data.
A record contains no query text, table names, field names, literal values or user identity; the code that writes it has no access to the query text (Why a record cannot contain your data).
What a record looks like
Given this query:
otel-logs
| where resource['service.name'] == "checkout"
and severity_text == "ERROR"
and timestamp > ago(1h)
| summarize errors = count() by resource['service.name'], bin(timestamp, 5m)the record's query.shape is:
T1 - F1 | where F2.F3.F4 == "S1" and F5 == "S2" and F6 > ago(timespan(TS1))
| summarize A1 = count() by F2.F3.F4, bin(F6, timespan(TS2))Operators, keywords and built-in function names stay. Everything else becomes a
marker: T for a table, F for a field, A for an alias, L for a let
binding, FN for a function that is not built in, and S / N / TS / DT /
G for a string, number, timespan, datetime or guid constant.
Markers are assigned per name, not per position, so one name gets one marker
everywhere it appears. Above, F2.F3.F4 is the same field in the where and
the summarize by: the record shows that the query filtered on the field it
grouped by, without naming the field.
Alongside the shape, a dictionary describes each marker, in derived facts only:
{
"token": "S1",
"class": "string_const",
"occurrences": 1,
"stage": 1,
"phase": "pre_reduce",
"len": 8
}| Field | What it says |
|---|---|
token, class | The marker and what kind of thing it replaced |
occurrences | How many times it appeared in the query |
stage, phase | Which pipeline stage it first appeared in, and whether that was before or after the first aggregating operator |
len | For a string constant: how many characters it had |
labels | For a string constant: coarse labels from a fixed list — hex, uuid, ipv4, digits, dotted, has_space, wildcard, non_ascii, empty |
bucket | For a number or timespan: a magnitude band from a fixed list, e.g. <1e3, 1h..1d |
len and phase explain a slow query without its text: a three-character
literal cannot use the five-gram bloom filter, and a literal in a post_reduce
stage never reaches the scan, so no index applies to it.
The rest of the record
| Field | What it says |
|---|---|
outcome | completed, failed, or abandoned (a client hang-up, a cancel, or a deadline) |
error_code | The error's class — ParseError, Timeout, … — never its message |
| (no warnings) | Query warnings are left out: they name columns and quote predicates |
surface, duration_ms, rows | Which entry point served it, how long it took, how many rows came back |
partial_failures | Segments the query could not read, so an incomplete result is visible as such |
planned_quality | The quality score the planner expected before reading a chunk |
quality | The score the client was shown. Its distance from planned_quality is the planner's calibration error |
quality_inputs | The primitives behind the planned score: bloom selectivity, range and shard term counts, reducer weight |
plan | A name-free digest of the plan that ran: source kind, mapper operator kinds in order, reducer, time bounds |
stats | The engine's execution-stats bag — chunks scanned and skipped, bytes, timings, cache reuse — filtered to numbers and booleans |
plan and quality_inputs describe what the engine did; query.shape
describes what the query asked for.
Where the records go
Nothing leaves your cluster unless you configure it. The second destination below is off by default and sends wherever you point it, usually your own Berserk.
1. The service log stream (always on). The query service writes each record
as a log record with body and target query.usage, the record in its
attributes. It goes wherever the query service's logs go. With Berserk's own
telemetry exported into Berserk (Observability),
you can query the records:
bzrk search 'search "query.usage"
| where resource["service.name"] == "query"
| summarize count() by tostring(attributes.outcome)' --since "24h ago"2. A usage stream (off by default). Given an OTLP endpoint and an ingest token, the query service also sends each record there, usually to your own ingest service with a token bound to a table of its own, which gives the records their own retention:
query:
config:
usageReportEndpoint: "http://ingest:4317"Create the table and the token, and put the token in a Secret; the chart passes it to the service's environment, not to a ConfigMap:
bzrk table create bzrk-usage
bzrk ingest-token create usage_reporting --table bzrk-usage --simple
kubectl create secret generic query-usage-report --from-literal=token="<token>"The Secret's name and key are query.config.usageReportTokenSecret and
usageReportTokenSecretKey. Without a token the records go to the endpoint's
default table.
Export failures do not affect queries: records go through a bounded queue drained in the background, and are dropped rather than slowing a query.
Sending records to the Berserk team
Nothing is uploaded automatically: you copy the records and send them. They contain only markers, numbers and labels.
bzrk -P <profile> search "search 'query.usage'" --since "7d ago" --no-stream --json > usage.json
bzrk admin verify-usage-report usage.jsonverify-usage-report checks that every query shape in the file uses only KQL
vocabulary and markers, and exits non-zero if anything is not anonymized. Then
do not send the file; report it to the Berserk team as a bug.
The verbatim companion line
The query service also logs one query.completed line per query, with the query
text and the calling principal. It stays inside your install and answers who ran
what. To keep query text out of your logs, drop the query.completed target in
your log pipeline; the usage stream does not depend on it.
Why a record cannot contain your data
The anonymizer's output text comes from one function, which takes no query text, only a list of atoms, each one of:
- a token kind, printed from a compiled-in table of keyword and operator spellings;
- a built-in function name, printed from the compiled-in function registry;
- a marker class plus an ordinal, printed as a fixed prefix and a decimal number.
Every byte of a shape comes from a table compiled into the binary or from a counter. The code that reads the query produces only atoms and derived numbers; it cannot print. A separate check re-reads each shape and requires every token to be compiled-in vocabulary or a well-formed marker. The test suite runs that check over hostile inputs: marker-shaped literals, nested quotes, comment tricks, unterminated strings, and bytes the lexer cannot read.
The stats bag is filtered to numbers and booleans: every string value is dropped, and maps not on an allowlist are dropped whole, so no string survives as a key either. A scan error tally, for example, is keyed by the error's message, which can quote the row that failed to convert. A new engine counter appears in the record; a new string-keyed map is left out until it is allowlisted.
Two limits:
- A field whose name is also a KQL keyword renders as that keyword.
project countstaysproject count. The bytes are still compiled-in, but the shape does reveal that a field with that name exists. Only the ~110 words in the KQL vocabulary can appear this way. - A shape is a template, not a runnable query. It lexes cleanly and mostly parses, so it can be analyzed as a query, but a position that syntactically requires a literal carries a marker word instead.
Turning it off
Leave usageReportEndpoint unset and the second destination never opens. To
stop the records entirely, filter the query.usage target out of your log
pipeline, or set the query service's log level so that target is suppressed.