Design Principles

Berserk's query language, BerserkQL, uses the syntax of the Kusto Query Language (KQL): the pipe, the operators and the function names you already know from Azure Data Explorer. What it is built for is different. This page explains the principles behind it, so that its behavior is predictable, including the places where it deliberately differs from Microsoft's KQL. The specific differences are listed in Compared to Microsoft KQL.

Built for interactive questions over very large dataCopy Anchor Link

BerserkQL is designed for asking questions of large volumes of logs, traces and metrics, whether the person asking is an engineer at a prompt or an AI agent working through an investigation. Both work best with an answer in seconds. So BerserkQL is built to:

• answer quickly, with a first result while the query is still running;
• read as little data as possible, by letting indexes skip data that cannot match;
• tell you exactly what each answer covers, so a fast answer is never mistaken for something it is not.

Your data does not need a schema. Every field in every record can be queried as soon as it arrives, and BerserkQL works out types as it goes.

What it feels likeCopy Anchor Link

• The obvious query is the fast one. where status == 500, where level =~ "error" and summarize count() by service work directly on raw fields, with no casts and no schema, and they use the indexes. You do not need a performance cheat sheet.
• Results arrive right away and fill in. A first result appears early and is refined as more data is read. Each update is a complete answer for the data read so far.
• Every answer states its coverage. The result tells you whether it covers the whole time range, whether it is exact, and if not, how it was computed.
• Mistakes in the query are caught early; messy data is handled gracefully. Comparing a number to a date is reported when you write the query. A field that holds a number in one record and text in the next is handled, not reported as an error.
• Costs are visible. Anything that reads every record or turns off index pruning, such as an explicit conversion, is written in the query, never added behind your back.

How a query runsCopy Anchor Link

A query has two parts. The first part, up to and including the first aggregation such as summarize, top or count, runs in parallel across all of your data, close to where it is stored. This is the part that has to be fast at any scale: its filters use the indexes, and the aggregation condenses billions of records into a result small enough to work with.

Everything after the first aggregation runs on that condensed result, in one place. Sorting, joining aggregated results, window functions and further calculations there are exact and work on the whole intermediate result.

default
| where severity_text == "ERROR"                       // first part: runs across all data
| summarize errors = count() by service = resource['service.name']
| sort by errors desc                                  // second part: runs on the summary
| take 10

Most of the behavior on this page follows from that split. Limits, sampling and estimates apply at the first aggregation, because that is where the amount of data is decided. Sorting raw records without aggregating, for example default | sort by duration desc, keeps the top 2,000 rows unless you ask for a different number with take or top. Sorting the result of an aggregation is not limited.

The principlesCopy Anchor Link

Every answer states what it coversCopy Anchor Link

An answer from BerserkQL is one of:

• exact;
• exact for a stated part of the data (for example, a consistent sample of the services when there are more than the memory budget allows);
• an estimate with a known accuracy (for example, a distinct count from a sketch);
• an error that tells you how to change the query.

When data was left out of an answer, for example because a grouped query had more groups than its memory budget allows, the result carries a warning saying what was left out. A warning always calls for your attention. Estimates that stay within their documented accuracy are not warnings: that accuracy is what the function promises, just as dcount() promises in Microsoft's KQL. When a query could run more cheaply, the result carries advice; advice never means the answer is wrong.

Early results, then complete resultsCopy Anchor Link

Large queries stream their results. You see a complete answer for the data read so far, updated as the query progresses, and you can stop at any point and keep what you have. The final result is marked as final.

A few operators need all of their input before they can answer. A plain join, for example, cannot know that a record has no match until both sides have been read, so it shows progress but no rows until it finishes. For the most common reason to join telemetry, correlating events in time or within a trace, use join-with-time or trace-find, which stream their results like other queries.

Sensible limits by default, and you can raise themCopy Anchor Link

Every query has a time range, a limit on the number of rows it returns, and a memory budget. The defaults suit someone reading a screen or an agent reading a response: for example, a query without an explicit limit returns the newest 100 rows. Any limit can be raised by saying so in the query, with take 10000, a longer time range or a larger budget. Defaults never grow on their own as your data grows.

Precise estimates instead of slow exact answersCopy Anchor Link

Some questions, such as how many distinct users there were or what the 99th-percentile latency was, normally require keeping every value in memory. BerserkQL answers them with well-studied estimation techniques (sketches) that need a fixed amount of memory and stay close to the exact answer with high probability. As in Microsoft's KQL, dcount() and the percentile functions work this way. Their accuracy is part of the function's definition, so using them does not produce a warning.

When a grouped query has more groups than its memory budget allows, BerserkQL keeps a consistent selection of whole groups, chosen by a hash of the group key, and computes each of them exactly. A group is never partly counted, and the result carries a warning that groups were left out.

Queries are designed around the indexesCopy Anchor Link

Berserk indexes every chunk of data so that most of it can be skipped without being read (see Best Practices). BerserkQL is designed so that the natural way to write a filter is the way that uses these indexes:

• filters on raw fields compare values by their stored type, and stay index-backed;
• has, contains, =~ and startswith cost about the same, so choose the one that means what you want;
• for many of the filters that cannot use an index, the result includes advice saying so.

Flexible data, typed queriesCopy Anchor Link

Fields arrive without declared types. BerserkQL reads each value as what it is: a number, a string, a timestamp, a nested object. Within a query, types are checked as early as possible. If you compare two values whose types can never match, you get a clear error that names both types and suggests a fix, instead of a query that quietly returns nothing.

When you know what a field holds, you can say so with annotate. Annotating is free and keeps index pruning, unlike a conversion.

Values are used as stored, not silently convertedCopy Anchor Link

BerserkQL does not convert values between types on its own. A number is not turned into text because it was passed to a text function, and a nested object is not turned into JSON text because it was searched. When a function needs a string and the field holds a string, it is used directly. When it holds something else, the function sees an empty value.

When you do want a conversion, write it, for example tostring(x) or todatetime(x). This keeps searches index-backed and makes the cost of a conversion visible in the query.

Mistakes in the query stop it; unexpected data does notCopy Anchor Link

If a query is wrong, for example a type mismatch or an invalid argument, it stops with an error that points at the problem and suggests a fix, before any data is read whenever possible.

If the data is unexpected, for example a missing field or a value of an unexpected type, the query keeps running. The affected expression produces an empty (null) value, the same way a function such as todatetime("not a date") returns null. Running the same query on different data should not decide whether it succeeds.

Familiar where possible, different where it mattersCopy Anchor Link

BerserkQL keeps KQL's names, syntax and behavior wherever they fit these principles. Where it differs, the difference serves one of them, and it is documented in Compared to Microsoft KQL.