Tables

A table is the top-level unit of data organization in Berserk. Each table is an independent collection of logs, traces, and metrics — typically representing a service, environment, or team.

Every cluster comes with a default table ready for ingestion. You can create additional tables to separate data by any boundary that makes sense for your organization.

The table name is the first element of every query:

default
| where level == "ERROR"
| take 10

Listing TablesCopy Anchor Link

bzrk table list

Tables:
  NAME     ID                                    RETENTION
  default  019ae8e5-f2aa-79c2-a0a6-41d49b609f9e  30d

Inspecting a TableCopy Anchor Link

table get shows everything set on a table — retention, merge span, sharding and vector fields — and what refers to it: the views it is a member of and the ingest tokens routing to it. Add the global --json flag for a machine-readable answer.

bzrk table get default

Table: default
  ID: 019ae8e5-f2aa-79c2-a0a6-41d49b609f9e
  Database: default (ID: 019ae8e5-f1b0-7c3e-9d2a-0b6c1e2f3a4d)
  Retention: 30d
  Max merge span: default (derived from retention)
  Sharding fields:
    metric_name               10
    resource['service.name']  10
    trace_id                  2
  Vector fields: none
  Member of views: none
  Ingest tokens routing here:
    my-ingest-token (traces, logs, metrics)

Creating a TableCopy Anchor Link

bzrk table create myapp

Created table 'myapp' (019cd6a4-ab94-7a90-bcc1-c380ae443daf) in database 'default'

ShardingCopy Anchor Link

Berserk groups a table's rows into chunks by a set of shard fields you choose. A query that then filters a shard field with == or in (…) skips whole chunks instead of scanning them, so choosing shard fields that match how you query a table is one of the biggest levers on query speed.

Set the fields with the CLI. Each is a field=weight pair, or a bare field name for a weight of 1; a higher weight gives the field more influence on the layout. set replaces the table's fields outright and needs at least one field:

bzrk table sharding set myapp \
  "resource['service.name']=10 metric_name=10 trace_id=2 span_id"

Inspect them, remove them all, or restore the default shard set:

bzrk table sharding list myapp
bzrk table sharding clear myapp
bzrk table sharding reset myapp

Recommended fields for OpenTelemetry dataCopy Anchor Link

This is the default shard set seeded on every new table — a good starting point for any table carrying OpenTelemetry-shaped logs, traces, and metrics. To re-apply it to an existing table (or after clear), run bzrk table sharding reset myapp. The set is:

A shard field names a key exactly as the data holds it. OpenTelemetry attribute keys contain dots, so service.name under resource is resource['service.name'], while resource.service.name names a service object holding a name. A query skips chunks by a shard field when it spells the field the same way: resource['service.name'] == "checkout". A query spelled resource.service.name reads the nested path, which OpenTelemetry data does not hold, so it finds no rows.

A shard field spelled resource.service.name names that nested path, so over data that holds the flat key service.name it records no values. On upgrade, Berserk renames such dotted names under resource, scope and attributes to their bracket form in every table's shard set, keeping weights and every other field as it is.

Note that fields that are configured for sharding but are not present in the row being written or merged do no harm. The set should not exceed ~40 fields.

Vector FieldsCopy Anchor Link

A table's vector fields are the fields semantic search indexes, so similarto and vector-search can score them from the index. A table has at most 4, and each is a field name such as body or attributes.prompt.

Set them with the CLI. set replaces the table's fields outright and prints them as Berserk stored them, in canonical form:

bzrk table vector-fields set myapp body attributes.prompt

Inspect them:

bzrk table vector-fields list myapp

Remove them, so the table's new data is no longer indexed for semantic search:

bzrk table vector-fields clear myapp

Ingest TokensCopy Anchor Link

An ingest token authenticates an OpenTelemetry Collector and routes each signal (traces, logs, metrics) to a table — all to the same table by default. Each table can have multiple ingest tokens (e.g., one per collector or environment).

Creating an Ingest TokenCopy Anchor Link

bzrk ingest-token create --table myapp my-ingest-token

Created ingest token 'my-ingest-token' (019cd6a6-5fe5-7a80-be74-9dba27894af8) in database 'default'; it routes all signals to table 'myapp'

Token: ing_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Save this token now; it cannot be retrieved again.

To route a signal elsewhere, give it its own table with --traces-table, --logs-table or --metrics-table. --table covers every signal without its own flag, so it is required unless all three are given. This token sends logs and traces to logs, and metrics to metrics:

bzrk ingest-token create my-token --table logs --metrics-table metrics

Listing Ingest TokensCopy Anchor Link

list shows each token's status and where each signal goes; get shows one token:

bzrk ingest-token list
bzrk ingest-token get my-token

Ingest tokens:
  my-token (ID: 019cd6a6-5fe5-7a80-be74-9dba27894af8) (Token: ing_...x7Qa) [active]
    traces → logs, logs → logs, metrics → metrics

Deleting an Ingest TokenCopy Anchor Link

bzrk ingest-token delete my-token --dry-run   # shows what would be deleted
bzrk ingest-token delete my-token

Ingesters learn of the delete at their next reconcile with the catalog, so a deleted token keeps working until then — normally within the interval the command prints (30 seconds by default). Data sent until then is still stored. The token can be resurrected during its grace window.

Querying a TableCopy Anchor Link

bzrk search "myapp | take 10" --since "1h ago"

See Query Reference for the full query language reference.