Semantic Search (similarto)

similarto ranks rows by how close a field's text is, in meaning, to a natural-language phrase — so you can find events that look like a description without knowing the exact words, error codes, or templates in your data.

default
| where timestamp > ago(1h)
| top 100 by body similarto 'database connection timeouts'

This returns the 100 closest-matching rows, each with a _score column (a real in [0, 1], higher = more similar), sorted from most to least similar.

SyntaxCopy Anchor Link

<field> similarto "<text>" evaluates to a real score in [0, 1].

• <field> — a single string column or field path (e.g. body, attributes['exception.message']). A dot means a nested path — attributes.prompt descends the attributes bag to its prompt key. To target a field whose name literally contains a dot, bracket-escape that segment: attributes['db.statement'] reads the db.statement key inside attributes, and ['http.target'] is a single top-level field named http.target. The field must be one your data is indexed on (see Requirements).
• "<text>" — the search phrase, a string literal. Both '…' and "…" work.

Where you can write itCopy Anchor Link

The score is an ordinary value, so it composes like one:

// Rank: `top` names the score `_score` for you
default | top 20 by body similarto 'payment declined'

// Bind it, then use it like any other column
default | extend score = body similarto 'payment declined' | where score > 0.6

// Filter directly — this is the fast path, see below
default | where body similarto 'payment declined' > 0.6

// Project it, sort by it, or nest it in a larger expression
default | project body, score = body similarto 'payment declined'
default | sort by body similarto 'payment declined' desc
default | extend weighted = (body similarto 'payment declined') * priority

In top … by, similarto must be the only sort key and the order must be descending. Everywhere else it is an ordinary expression. The last three work, but the index cannot skip anything for them — see Which shapes are fast.

Several phrases in one queryCopy Anchor Link

Each similarto scores against its own phrase, so one query can compare a row against several, and on different fields:

default
| extend auth = body similarto 'authentication failure',
         db   = body similarto 'database connection error'
| where auth > 0.6 or db > 0.6

A condition over several scores is not a threshold the index can apply, so this query scores every row and says so (see Which shapes are fast).

A query may use up to 4 distinct (phrase, field) pairs; repeating the same phrase on the same field counts once. Each extra pair adds its own scoring work to every segment the query reads.

The vector-search operatorCopy Anchor Link

Searching for a phrase usually means "what kinds of event are like this", not "every line like this" — twenty copies of one log line are one answer, not twenty. So vector-search returns one result per kind of match (per log template), most similar first, each with an example: its most recent matching row (the best-scoring one when the input has no timestamp), led by these columns:

default | vector-search 'payment declined'

The row's columns follow, as search returns them: over one table every column; over several, the searched fields, the columns every table has, and $raw for the rest. A $ prefix marks a column the engine supplies about the row ($table, $raw); a _ prefix marks a value computed for each result (_score, _count, _value, _field).

• on= — the field to search. Leave it out to search the vector fields configured on the tables the query scans (at most 4 in all), so you don't need to know where the text is — _field tells you. When none of those tables configures any, the query is an error: name the field with on=, or configure the table's vector fields with bzrk table vector-fields set.
• k= — how many results to keep. Default 100, the default query limit.
• min_score= — a score floor, inclusive (>= S), in [0, 1]. Default 0.
• output= — what each result is:
  • examples (default) — the most recent matching row of each kind of match, as above. Only the columns the rest of the query reads are kept, so a project after the search keeps it small. The search fails, rather than returning fewer results, when the kept rows of its k results outgrow its memory budget; ask for values or a smaller k then.
  • values — just the matched text, as _field, _value, _score and _count. Compact and cheapest: one string per kind of match.
  • matches — every matching row, ranked by _score like top … by … similarto, led by $table and _score (and _field when searching several fields), with the row's columns as for examples.

On a table scan, only rows the vector index scores count: rows of segments that do not index the field are left out of _count (the query's statistics show those segments as chunks unindexed); with output=matches such rows score 0. A kind of match scores the same on every row of a segment, but can score differently in another segment, written at another precision or embedded from another row's text; _count and the example then cover only the segments that score it at or above the k-th best result while they are read.

output=examples returns what this pipeline returns, with ties at the k-th score broken by field and template, which you can write yourself to change it — to aggregate differently, or keep a different sample:

default
| where isnotempty(tostring(body))
| extend s = body similarto 'payment declined'
| summarize _score = max(s), _count = count(), arg_max(timestamp, *)
    by _field = 'body', t = log_template_hash(tostring(body))
| project-away t, s
| top 100 by _score desc

Which shapes are fastCopy Anchor Link

Scores are approximate by design: the index trades exactness for size and speed, and every indexed surface returns its estimate. Before the first aggregate, three shapes let the index skip data it can prove cannot match:

• A ranking key — top K by <field> similarto '…', a top K by s over a score s bound with extend, and vector-search.
• A threshold — where <field> similarto '…' > S, or where s > S directly after the extend that binds s. It also serves semantic counting, such as a summarize count() by bin(timestamp, 1h) after it.
• The best templates — summarize m = max(s) by log_template_hash(tostring(<field>)) followed by top K by m, which is what vector-search does. Between the score and the summarize, a where may only drop empty values of the field or threshold the score.

Searching several fields at once (vector-search without on=) is one scan that ranks every field's matches together. The index serves each field, a chunk is read only when some field could still rank in it, and this holds anywhere in a query (a union arm, a fork branch, a join side) and at any k=. With output=matches a row that matches in two fields is two results, one per field, and min_score applies to each field on its own.

Any other use before the first aggregate — a score projected over every row, a sort without a top, summarize avg(s), arithmetic on the score — works, but every row is scored, and the query carries a VSearchUnoptimized warning saying so. It is the only warning similarto gives itself. Ranked and thresholded searches never warn, whatever the precision of their scores. After an aggregate the score is computed over the rows that remain (see After a reducer) and nothing warns.

The query's statistics report how the index served each field: templates and rows per precision, and the chunks where the field carried no index (with the reason, for data written in the current index format).

Thresholds are pushed into the searchCopy Anchor Link

A score floor is applied while ranking, not afterwards:

default | where body similarto 'payment declined' > 0.6
default | extend s = body similarto 'payment declined' | where s > 0.6
default | vector-search 'payment declined' on=body min_score=0.6

All three drop sub-threshold rows inside the search operator, so they never occupy a top K slot or travel back from the workers. > and >= mean what they say — a row scoring exactly 0.6 survives >= 0.6 and not > 0.6 — and a min_score= combined with a where narrows to whichever is tighter.

Only a lower bound on the score folds in. An upper bound (where s < 0.6), a bound outside [0, 1], or a threshold combined with other conditions in one where are evaluated as ordinary filters over the ranked rows. They give the same answer, just later.

ExamplesCopy Anchor Link

// Narrow with a fast filter first, then rank semantically
default
| where Service == 'checkout-api' and timestamp > ago(6h)
| top 50 by body similarto 'payment provider returned 5xx'
| project timestamp, _score, severity_text, body

// Rank a nested attribute instead of the log body
default
| top 50 by attributes['exception.message'] similarto 'null pointer in serializer'

// Rank wide, then post-filter and re-rank by a domain signal
default
| top 1000 by body similarto 'rate limited by upstream'
| where severity_number >= 17
| top 50 by _score

Put filters you want to narrow the search before the ranking — they shrink the candidate set it runs over (and let Berserk skip data). A filter on the score is the exception: it is pushed into the operator wherever you write it.

Rank kinds of event, not linesCopy Anchor Link

Scores are per-template (see How scoring works), so a top K can fill every slot from a single kind of message — twenty rows that are the same line twenty times. The query's statistics report it when it happens (collapse beside the field).

To rank kinds of event instead, collapse by template first, then rank the templates:

default
| where timestamp > ago(1h)
| extend score = body similarto 'database connection timeouts'
| summarize score = max(score), n = count(), sample = take_any(body)
    by template = log_template_hash(tostring(body))
| top 20 by score desc
| project sample, n, score

Each row is now one kind of event, with n telling you how common it is and sample showing a representative line.

How scoring works (and what that means for you)Copy Anchor Link

Every row is scored, but the similarity is measured against the row's template, not its raw text. Berserk reduces each row to a template with the same extract_log_template function used elsewhere in KQL — stripping the variable parts (ids, numbers, paths, timestamps) down to a skeleton — and compares your search phrase against those skeletons. Matching is therefore per-row, but the signal is template-level. The practical consequences:

• Rows that share a template share a score. "Found 10 products" and "Found 20 products" reduce to the same template, so they rank identically — the ranking is over kinds of events, not individual lines. This holds when the score is read from the index; see After a reducer for the exception.
• Paraphrases of the same kind of event rank together. "auth refused", "credentials rejected", and "401 from idp" land near each other even though they share no words, because their templates embed close together.
• It's a similarity ranking, not a filter. Without a threshold, top K returns up to K rows; low-relevance rows still appear (with low _score) once the closer ones run out.
• It's strongest for topics, weakest for one-off lines. Finding "events like database timeouts" works well; finding a single rare line whose template appears nowhere else has little for the ranking to grip on.

After a reducerCopy Anchor Link

Downstream of an operator that bounds the rows — summarize, count, take, tail, top, sort — there is no index to read, so Berserk embeds each row's own text instead and compares it with your phrase. The operator has already capped how many rows reach it, which keeps this affordable. Two things differ from the indexed case:

• Each row is scored by its own text, so two rows of one template can score slightly differently ("Found 10 products" and "Found 20 products" are close, not equal).
• The field doesn't need to be indexed for these rows to score.

// The 20 most recent errors, scored against a phrase
default
| where severity_number >= 17
| top 20 by timestamp
| extend score = body similarto 'database connection timeouts'

When to use similarto vs. lexical searchCopy Anchor Link

For exact strings, has is both faster and more precise — similarto is for when you don't know the exact wording.

RequirementsCopy Anchor Link

similarto only produces meaningful scores when the field is indexed for semantic search for that dataset (an admin enables it per-field at ingest). When it isn't:

• The query still runs and returns rows, but _score is 0 for rows with no indexed template — there's no semantic signal to rank on. The exception is a similarto after a reducer, which embeds the rows' text and needs no index.
• A field name that doesn't exist in the schema is a bind-time error (unknown column), same as anywhere else.

If your scores come back all-zero, the field is most likely not one of the table's vector fields — ask whoever administers it. The query's statistics count the chunks it read where the field carried no index.

LimitationsCopy Anchor Link

• At most 4 distinct (phrase, field) pairs per query. A fifth is a bind-time error.
• Not after a top … by … similarto. That top already chose the rows, so a second similarto downstream of it is rejected; score it before the top.
• The phrase must be a literal string — you can't pass a column or variable as the search text.
• Single field. Searching several fields at once ((body, attributes…)) is not supported.
• Not inside an aggregate. summarize max(body similarto '…') is rejected; bind the score with extend first, then aggregate the column.
• Not after join, fork, or trace-find, where the row's lineage back to an indexed segment is ambiguous.
• Not inside an mv-apply subquery. Score the rows before or after the mv-apply instead.
• In top, descending and sole sort key only. top K by … similarto … asc and a second sort key are both rejected. Use sort by if you want ascending.
• Rewriting the searched field disables scoring. If an upstream operator replaces the field's value — parse body …, mv-expand, or an extend/project that assigns to the field you're searching — Berserk can no longer trust the indexed template for it, those rows score 0, and the query carries a VSearchUnoptimized warning naming the field. Rank before you reshape the field. Rewriting a different column is fine, as is an identity rename (project body = body).
• English-centric by default. The default embedding model is English-trained; quality on other languages is poor until a multilingual model is configured.
• Topic search, not anomaly detection. similarto finds rows like a phrase; it doesn't surface "unusual" events.

See alsoCopy Anchor Link

• Best Practices — Berserk's indexing and filter-pushdown model.
• top operator — one of the hosts for similarto.