reduce
Groups string values into patterns, where `*` stands for a run of words that varies within a group.
A word stays literal when at least threshold of the rows (and at least two) contain it; rows whose literal words and gaps line up form a group when there are at least as many of them. Values — datetimes, UUIDs, IPv4 addresses, long hex strings and words that start with a digit, such as 404 or 12ms — are always *, however often they repeat; identifiers like s3 or x86_64 are words. Every other row is counted in an others row. Runs on rows that are already bounded: it must follow a reducer such as take, summarize, top or sort, or read a datatable.
Syntax
reduce [kind=source] by Expr [with [threshold=T] [, characters=C]]Returns one row per pattern with Pattern, Count and Representative,
the first row of the group, most frequent pattern first, then others
with an empty representative. With kind=source, returns every input
row with a Pattern column appended instead, empty for a row in no
group.
Parameters
| Name | Description |
|---|---|
| kind | source keeps the input rows and appends Pattern. |
| Expr | The string value to reduce. |
| threshold | A number in [0, 1], default 0.01. The fraction of rows a word must occur in to stay literal, and a group must reach to be reported. Higher values give fewer, more general patterns. |
| characters | Characters that separate words, in addition to everything that is not a letter or digit. |
Examples
Example 1
datatable(text:string)[
"connection to db-1 failed after 30 ms",
"connection to db-2 failed after 41 ms",
"connection to db-7 failed after 12 ms",
"user alice logged in",
"user bob logged in"
]
| reduce by text| Pattern (string) | Count (long) | Representative (string) |
|---|---|---|
| connection to db-* failed after * ms | 3 | connection to db-1 failed after 30 ms |
| user * logged in | 2 | user alice logged in |
Example 2
datatable(id:long, text:string)[
1, "GET /api/users/1 200",
2, "GET /api/users/2 404",
3, "GET /api/users/3 500"
]
| reduce kind=source by text| id (long) | text (string) | Pattern (string) |
|---|---|---|
| 1 | GET /api/users/1 200 | GET /api/users/* |
| 2 | GET /api/users/2 404 | GET /api/users/* |
| 3 | GET /api/users/3 500 | GET /api/users/* |