Segment Placement

Every segment lives in S3, and every query node keeps a local segment cache. If any node could serve any segment, each node would eventually cache everything: the working set would be duplicated N times and the cache hit rate would fall as you scaled out.

So Berserk gives each segment an owner. One query node is responsible for a given segment: the coordinator routes scans of it there, the nursery and janitor push freshly merged copies there, and each node's background warmer prefetches only its own share. The cache is partitioned across the cluster instead of replicated, so adding nodes adds cache capacity rather than just adding copies.

How ownership is decided

Each segment id is hashed against the current set of query nodes, weighted by each node's CPU capacity. The result is a ranking of nodes for that segment; the highest-ranked node that is currently live owns it, and the ones below it are the standby order for when it isn't.

Three things follow from this, and they are the properties worth remembering when operating a cluster:

• Ownership is computed, not stored. The query coordinator, the nursery, the janitor, and the query nodes all derive the same answer from the segment id and the current member list. There is no placement table to keep in sync, no lookup service, and no rebalance job.
• A segment's owner is stable. It only changes when the cluster's membership or weighting changes — never as a side effect of queries, restarts of other services, or the segment being read.
• Membership changes move as little as possible. Adding or removing a node disturbs only that node's share, as the sections below spell out.

Weighting by CPU

The goal is equal GB of scanned data per core, not equal GB per node. A 32-vCPU node should hold and scan twice as much as a 16-vCPU one, so that both finish their part of a query at the same time.

Each query node declares its effective vCPU count when it registers with meta, read from the container's CPU quota (minimum 1). Under Kubernetes that is the pod's CPU limit, not the host's core count — a pod limited to 24 CPUs on a 64-core machine declares 24, which is what the rest of the cluster should judge it by. The declaration is refreshed on every heartbeat, so retuning a limit propagates on its own.

A node's share of the data is its share of the cluster's total weight:

Because segment ids hash uniformly and segments are of similar size, share-of-segments tracks share-of-bytes, which is what determines both cache footprint and scan work. Ingest recency doesn't skew this: consecutive segments scatter across the cluster, so a query over the last hour still spreads in the same proportions.

Adding a node

When you scale from N nodes to N+1, no segment moves between the existing N nodes. The new node takes a slice of each existing node's share, and nothing else changes.

The fraction of segments that move is exactly the new node's share, weight(new) / Σ weight — and every one of them moves to the new node. Continuing the example above, adding a 16-vCPU node D:

28.6% of segments changed owner, and all of them landed on D. A never handed anything to B or C.

What this costs in practice:

• No data is copied and no rebalance runs. S3 is the source of truth; ownership only decides who caches and scans what.
• Segments newly owned by D are cold there. Its warmer starts prefetching the index regions of its share, and the first query touching a not-yet-warm segment fetches it from S3 once, after which it is resident.
• The now-stale copies on A, B, and C are simply never read again and age out of their caches.
• The other 71.4% of the cluster's cached data is untouched and stays hot throughout.

Scaling out is a replicaCount change on the query service; there is no placement step to run afterwards.

Removing a node

Removal is the mirror image. Only the departing node's segments move; every segment it did not own keeps the owner it had. Its share is spread over the survivors in proportion to their weights.

Removing C from the four-node cluster above:

Only C's 14.3% changed hands. A, B, and D each pick up roughly a third of it and keep everything they already had warm.

The same mechanism covers failures and restarts, which are just membership changes:

• A node goes away (crash, eviction, drain). Meta drops it after 30 seconds without a heartbeat (nodes heartbeat every 5s), and the rest of the cluster notices on its next registry poll, up to 10 seconds later. In the gap, work aimed at the missing node falls to the next node in that segment's ranking — a cache miss, not an error.
• A node comes back with the same identity. Each query node persists its identity in its cache directory, so a container restart on the same volume reclaims exactly the segments it had and finds its cache still warm. A pod replaced onto a fresh volume gets a new identity, which the cluster sees as one removal plus one addition.
• A rolling upgrade is a sequence of those events, one node at a time, each disturbing only the share of the node being replaced.

Changing a node's CPU

Retuning a node's CPU limit changes its weight, and ownership re-settles within a poll interval. This is also minimally disruptive: only the retuned node gains or loses segments, and the unchanged nodes never trade segments with each other. Raising a limit pulls a proportional slice of the others' segments onto that node; lowering it pushes a slice back out.

What ownership affects

Three parts of the system act on it, which is why they agree without coordinating:

• The query coordinator groups a query's segments by owner and sends each group to the node that has it cached. If the owner has left the cluster the work falls to the next node in that segment's ranking, and during early startup — before the coordinator has seen the member list — it simply spreads work round-robin. Both fallbacks cost cache hits, never correctness.
• The nursery and janitor push each newly merged segment into its owner's cache before making it queryable, so the first query finds it resident instead of cold-fetching a just-written object. A push that is declined or fails just means the segment is fetched from S3 on first use.
• Each query node's background warmer prefetches the index regions of the segments it owns, and skips the rest.

Operating notes

• Scale by changing query.replicaCount. Everything above happens automatically as the new pods register.
• Size caches per node, not per cluster. A node's cached working set is roughly its weight share of the total. If the cache sizing table says the cluster needs 600 GB, three equally-weighted nodes need ~200 GB each — plus headroom for the churn that follows a membership change.
• Expect a temporary dip in cache hit rate after scaling, proportional to the share that moved, until the new node is warm. There is no way — and no need — to pre-move data: recovery is just normal S3 reads.
• Heterogeneous nodes are fine. Mixing a 64-vCPU and a 32-vCPU node gives a 2:1 split of the data, not an even one, which is what keeps per-core load equal. What the weighting cannot capture is per-core speed: two nodes with the same vCPU count are treated as equals even if one is meaningfully faster per core.