Release notes

Changes that ask something of operators or API clients, newest release first. Fixes and query-engine improvements that need no action are not listed.

1.1.13Copy Anchor Link

Dotted spellings no longer read flat OpenTelemetry keysCopy Anchor Link

OpenTelemetry attribute keys contain dots: service.name is one key in the resource bag. Until now Berserk also stored such keys under the dotted path, so resource.service.name found them. That alias is gone. A dotted path reads nested objects only, and a key that contains dots is named in brackets: resource['service.name']. resource.service.name now means a name inside a service object, which OpenTelemetry data does not hold, so it finds nothing.

Action needed: rewrite every query that names an OpenTelemetry attribute with dots to the bracket form, in saved queries, dashboards, alerts, workflows, panel filters and columns, and anything that queries Berserk from outside, such as Grafana or scripts:

• resource.service.name becomes resource['service.name']
• attributes.http.method becomes attributes['http.method']

Nothing is rewritten for you. An alert whose query uses a dotted spelling matches nothing, so a threshold alert on it never fires. The bracket form works on earlier releases too, so you can rewrite queries before you upgrade. A query that still reads a dotted path under resource, scope or attributes returns advice that names the bracket form.

After the upgrade Berserk rewrites existing data in the background to remove the stored aliases. Until that finishes, a dotted spelling may still find rows written before the upgrade, so its results are mixed. The bracket form returns the same rows throughout.

Every segment is rewritten once after the upgradeCopy Anchor Link

The background rewrite above touches every stored segment once. While it runs, each janitor merges continuously at about one CPU core and up to its memory limit, so it lasts hours on a large dataset. Queries return correct results throughout.

Action needed if janitors autoscale: a rewrite keeps every janitor busy, so the autoscaler grows the janitor deployment to its maxReplicas and holds it there until the rewrite finishes. If janitors share nodes with other workloads, lower janitor.autoscaling.maxReplicas for the upgrade, or make sure the nodes have that much spare CPU and memory.

Text operators search for a stringCopy Anchor Link

has, contains, startswith, endswith, hasprefix, hassuffix, their _cs and ! forms, like, notlike and matches regex now require a string on their right side, as in Microsoft Kusto. body has 5 used to match nothing; it is now a type error. A dynamic value is still accepted and is searched for when it holds a string. The left side still takes any type.

Action needed: a saved query, dashboard panel or alert that compares a text operator with a number or a bool fails after the upgrade. Write the value as a string: body has "5".

Ingest holds the master key againCopy Anchor Link

Ingest is a service inside the cluster like the others: it reads BZRK_MASTER_KEY and presents the token derived from it. With it, ingest registers in the fleet registry again, and the UI's operator metrics list ingest pods again. The Helm chart and the Docker Compose file set the key; set it yourself if you run ingest another way. An ingest without it does not start.

Every account is local or managed by SSOCopy Anchor Link

An account signs in one way: a local account with a password, or an SSO account through the identity provider. A cluster admin configures single sign-on on the Single sign-on settings page, adds SSO members by email there or on the Members page, and can switch a local account to SSO; the Members page shows which kind each account is. The identity provider never signs in a local account, and a local account signs in the same whether SSO is on or off. The OIDC settings live in Berserk's database now: the deployment configuration's oidc_* fields are imported once at the first start after the upgrade and not read again.

Action needed for an account that had both a password and an identity-provider identity: it is a local account now and keeps its password. Its next "Continue with SSO" is refused with "this account signs in with a password". To move it to SSO, an admin switches it on the Members page: the password goes, sessions and roles stay.

1.1.12Copy Anchor Link

Every query goes through the gatewayCopy Anchor Link

The query service now answers only requests the gateway forwards. The gateway signs each request it authenticates, and the query service refuses one without a valid signature: 401 over HTTP, UNAUTHENTICATED over gRPC. Clients already going through the gateway change nothing.

Action needed if anything calls the query service (port 9510) directly — a script, a health check, a dashboard datasource, a deployment gate. Point it at the gateway with a bearer token instead. For a workload, use a service account's token:

bzrk service-account create ci
bzrk service-account grant ci observer
bzrk service-account token create ci ci-token --expires 90d

curl -X POST https://<gateway>/v1/rest/query \
  -H 'Authorization: Bearer <token>' \
  -H 'Content-Type: application/json' \
  -d '{"db": "default", "csl": "my_table | where timestamp > ago(15m) | count"}'

See the Query HTTP API and the API introduction.

Also part of this change:

• The gateway forwards gRPC under /api/grpc/ and the Kusto REST endpoints POST /v1/rest/query and POST /v2/rest/query. The query service's other HTTP routes (/query, /plan, /v1/databases, /timeline-query) are internal and no longer reach clients.
• /health* and /info on the query service stay open for probes.
• While a rolling upgrade from 1.1.11 mixes old and new pods, a 1.1.12 query service refuses queries a 1.1.11 gateway forwards. They succeed once the gateway has rolled.
• The profiling endpoints (/debug/pprof/*) moved from port 9510 to the query service's operator port, 9511, beside /metrics.
• The Helm chart no longer opens the query service's port to every source (query.networkPolicy.allowExternal). With network policies enabled, only the gateway, the UI and the other query pods reach it.
• A gateway signs with a key it registers with meta, so it needs meta's address (meta_endpoint). The Helm chart and the Docker Compose file set it; set it yourself if you run the gateway another way. A gateway reports ready only once meta lists its key.
• helm test runs its query through the gateway, signed in as the bootstrap admin (gateway.bootstrap.adminEmail with adminPassword or existingSecret). An install with no bootstrap password, such as an OIDC-only one, has no query test. If the admin's password has changed since the bootstrap, the test cannot sign in and fails.

The Loki query API is removedCopy Anchor Link

The query service's Loki-compatible read API (/loki/api/v1/*) is gone. Query with KQL through the gateway instead. Loki ingest — pushing logs with the Loki protocol — is unchanged.

Workflows and alerts act for their ownerCopy Anchor Link

A workflow or alert run now queries as the workflow, acting for its owner, rather than as the owner. On the query service's query.completed log line, the workflow is the principal and the owner appears in the new on_behalf_of_type, on_behalf_of_id and on_behalf_of_name fields. A report that counts a user's queries by principal no longer includes their workflows' runs.

Everyone signs in again onceCopy Anchor Link

The UI's session cookie is now encrypted, so existing browser sessions end at the upgrade.

The UI trusts only the gateway's tokenCopy Anchor Link

The UI takes who is signed in only from the token the gateway signs, and checks it against the keys the gateway publishes at GET /api/gateway-keys. The gateway sends no other identity headers, and drops the whole x-bzrk-* header namespace from incoming requests.

• Expect brief errors while upgrading. While a rolling upgrade from 1.1.11 mixes old and new pods, a 1.1.11 UI behind a 1.1.12 gateway answers every request with 403, and a 1.1.12 UI behind a 1.1.11 gateway treats signed-in users as signed out. Both stop once the gateway and the UI have rolled.
• Run exports need a signed-in caller. GET /api/runs/<id> and its export downloads used to answer anyone who knew a run's id. They now need the same credentials as the rest of the API: a browser session, or Authorization: Bearer with a CLI or service-account token.
• A tab left open reloads itself a few minutes before the token it loaded with expires, 24 hours after it opened, to get a fresh one.

Services authenticate each otherCopy Anchor Link

Every service now presents a token derived from the master key on its calls to the others, and meta, the query workers and the query pods' shared state cache refuse calls without it. Nothing outside the cluster holds the token, so a caller elsewhere on the network can no longer register with meta, read cluster state from it or reach a worker.

• Every service needs the master key. The query service, nursery, janitor and audit-shipper now read BZRK_MASTER_KEY, like the gateway, UI, permissions and meta. The Helm chart and the Docker Compose file set it; set it yourself if you run them another way. A service without it does not start.
• Meta answers only the cluster's services and what the gateway forwards. A CLI pointed straight at meta (bzrk --endpoint <meta>:9560) is refused, unless it runs inside the cluster with the master key, as the chart's setup job does. Through the gateway, users and service accounts reach what the CLI and the UI offer — tables, databases, views, ingest tokens, the janitor and stream commands — and none of meta's internal operations.
• Rotating the master key starts with a restart. Add the new key as secondary_master_key and restart every service before switching; see Rotating the master key.
• Ingest no longer registers in the fleet registry, so the UI's operator metrics no longer list ingest pods.
• Expect refusals while upgrading. Until every pod runs 1.1.12, meta and upgraded workers refuse pods on the old version. No acknowledged data is lost: ingested data waits in object storage until a nursery on the new version commits it. A pod on the old version that restarts during the upgrade does not start again until it is replaced.

Changing the catalog takes an admin or a maintainerCopy Anchor Link

Creating, changing, deleting or restoring a database, table, view or ingest token now takes an admin or a maintainer, a role users and service accounts can hold. So do the janitor, stream and vector-index commands that change anything. Anyone else is refused: PERMISSION_DENIED over gRPC, 403 over gRPC-web. Reading the catalog and querying are unchanged. The UI used to allow these changes to admins only, and now allows them to maintainers too; the CLI and the API used to allow them to anyone signed in.

Action needed if anyone but an admin changes the catalog, such as a CI job that creates tables. Make the person an admin, or grant the service account the role:

bzrk service-account grant ci maintainer

Ingest needs a token by default, and holds no secretCopy Anchor Link

Data sent to ingest without a token is now refused by default: 401 over HTTP, UNAUTHENTICATED over gRPC, 403 for the Datadog intake. The chart used to answer it with the default ingest token. Now both settings that allow it default to off: global.ingestToken.enabled, read by meta, and ingest.config.requireAuth, read by ingest.

Action needed if anything sends without a token, such as a collector inside the cluster. Give it a token of its own with bzrk ingest-token create, or turn anonymous ingest back on:

global:
  ingestToken:
    enabled: true
ingest:
  config:
    requireAuth: false

Also part of this change:

• Ingest no longer holds the master key. Meta answers a sender with no token itself, so the chart mounts no key into ingest, and ingest's DEFAULT_INGEST_TOKEN_ENABLED setting and ingest.config.ingestToken value are gone. If you run the services another way, set ingest's REQUIRE_AUTH and meta's META_DEFAULT_INGEST_TOKEN_ENABLED yourself; both now default to refusing data without a token.
• What an ingest reports to meta counts only with proof. It names itself with a key it makes at start, and lists the tokens it holds by their value, so a caller can no longer close another ingester's streams or claim tokens it does not hold. Ingester ids in bzrk admin stream listings are now 32 hex characters rather than UUIDs.
• Upgrade ingest with the rest of the cluster. An ingest from an earlier release cannot report to a 1.1.12 meta, nor a 1.1.12 ingest to an earlier meta. Until both have rolled, ingest treats data it cannot place as it does when meta is unreachable: 503 in the default durable ack mode, which senders retry, and dropped in fast_drop. An ingest you run outside the cluster has to be upgraded at the same time.

CLICopy Anchor Link

• bzrk auth token prints the active profile's gateway token, refreshed first when it is about to expire: curl -H "Authorization: Bearer $(bzrk auth token)" ….
• With no profile and no --endpoint, the CLI connects to the gateway at http://localhost:9500 rather than to the query service.
• bzrk admin segment write-off <segment-id> --reason "<why>" writes off a segment whose data is lost or corrupt, so merges and queries stop failing on it. Its events are no longer queryable, and the table's audit chain records the write-off with who did it and why. It takes an admin or a maintainer. See Tamper Evidence.

GatewayCopy Anchor Link

The gateway refuses cluster membership changes (meta.ClusterService/RegisterMember and DeregisterMember) from its callers. Only Berserk's own services register with meta.

JanitorCopy Anchor Link

The janitor runs no probe queries. Its queryEndpoint and probeJobs chart values and its probe metrics are gone; values left in an override are ignored.