Extracting Metrics from Logs

In a real-world production environment, logs are a mess of structured JSON, text logs with embedded numbers, syslog lines, nginx access logs, and more. Not everything conforms to OpenTelemetry conventions. Berserk is built to handle unstructured logs efficiently — you can extract fields on the fly and use them in meaningful ways without pre-processing.

This guide shows you how to pull numeric values out of free-text log lines with parse, aggregate them with summarize, and build gap-filled time series with make-series.

Sample data

The examples below use these application logs with embedded candidate counts and size ranges:

Notice the gap: no log lines exist between 10:10 and 10:14. This is common in real data — processes go idle, services restart, or log volume simply drops to zero.

Step 1: Extract values with parse

Use parse to pull the candidate count and size range out of each log line:

...
| parse body with "fetched " candidate_count:long " candidates for size range " size_range

This adds two new columns to each row:

The :long type suffix tells parse to convert the extracted text to an integer. Rows that don't match the pattern get null for both fields.

Step 2: Quick summary with summarize

The simplest way to aggregate is summarize with bin():

...
| parse body with "fetched " candidate_count:long " candidates for size range " size_range
| summarize avg(candidate_count) by bin($time, 5m)

The 10:10 bucket is missing entirely — summarize only produces rows for buckets that contain data. If you chart this directly, the line jumps from 10:05 straight to 10:15 with no indication that a gap exists. For dashboards and alerting, that silent gap is a problem.

Step 3: Build time series with make-series

make-series solves this by producing arrays with every bucket present, filling gaps with a default value:

...
| parse body with "fetched " candidate_count:long " candidates for size range " size_range
| make-series
    avg_candidates = avg(candidate_count),
    sample_count = count()
    default = 0
  on $time step 5m

Notice the columns are all dynamic — in KQL, arrays are represented as the dynamic type regardless of element type.

The 10:10 bucket now appears with 0 — the default value. When from/to are omitted, the time range is inferred from the data's min/max timestamps, so every 5-minute bucket within it is guaranteed to be present. Charts render a continuous axis and downstream series functions (smoothing, anomaly detection) get a regular time grid to work with.

Step 4: Group by extracted dimension

Add by size_range to get one series per size range:

...
| parse body with "fetched " candidate_count:long " candidates for size range " size_range
| make-series
    avg_candidates = avg(candidate_count),
    sample_count = count()
    default = 0
  on $time step 5m
  by size_range

Now each size range gets its own row, each with gap-filled arrays. The gap at 10:10 is filled with 0 independently per group:

From here you can pipe into series functions (series_fir, series_decompose_anomalies) or visualize with | render timechart.

When to use which approach