Segment Placement

Every segment lives in S3, and every query node keeps a local segment cache. Each segment has one owner, a query node: the coordinator routes scans of the segment there, the nursery and janitor push new merged copies there, and each node's background warmer prefetches only its own share. The cache is partitioned across the nodes, not replicated, so adding a node adds cache capacity.

How ownership is decidedCopy Anchor Link

Each segment id is hashed against the current set of query nodes, weighted by each node's CPU capacity. The result is a ranking of nodes for that segment; the highest-ranked live node owns it, and the nodes below it are the standby order.

Three consequences:

• Ownership is computed, not stored. The query coordinator, the nursery, the janitor and the query nodes derive the same answer from the segment id and the current member list. There is no placement table, no lookup service and no rebalance job.
• A segment's owner is stable. It changes only when the cluster's membership or weighting changes, never as a side effect of queries, restarts of other services, or the segment being read.
• Membership changes move as little as possible. Adding or removing a node moves only that node's share.

Weighting by CPUCopy Anchor Link

The aim is equal GB of scanned data per core, not per node: a 32-vCPU node holds and scans twice as much as a 16-vCPU one, so both finish their part of a query at the same time.

Each query node declares its effective vCPU count when it registers with meta, read from the container's CPU quota (minimum 1). Under Kubernetes that is the pod's CPU limit, not the host's core count: a pod limited to 24 CPUs on a 64-core machine declares 24. The declaration is refreshed on every heartbeat.

A node's share of the data is its share of the cluster's total weight:

Segment ids hash uniformly and segments are of similar size, so the share of segments tracks the share of bytes, which sets both cache footprint and scan work. Consecutive segments scatter across the nodes, so a query over the last hour spreads in the same proportions.

Adding a nodeCopy Anchor Link

When you scale from N nodes to N+1, no segment moves between the existing N nodes. The new node takes a slice of each existing node's share, and nothing else changes.

The segments that move are exactly the new node's share, weight(new) / Σ weight, and all of them move to the new node. Adding a 16-vCPU node D to the example above:

28.6% of segments changed owner, and all of them landed on D. A handed nothing to B or C.

The cost:

• No data is copied and no rebalance runs. S3 is the source of truth; ownership only decides who caches and scans what.
• Segments newly owned by D are cold there. Its warmer starts prefetching the index regions of its share, and the first query that touches a segment not yet warm fetches it from S3 once.
• The old copies on A, B and C are not read again and age out of their caches.
• The other 71.4% of the cached data stays where it is, warm.

Scaling out is a replicaCount change on the query service, with no placement step afterwards.

Removing a nodeCopy Anchor Link

Only the departing node's segments move; every segment it did not own keeps its owner. Its share is spread over the remaining nodes in proportion to their weights.

Removing C from the four-node cluster above:

Only C's 14.3% changed hands. A, B and D each take about a third of it and keep their warm segments.

Failures and restarts are membership changes too:

• A node goes away (crash, eviction, drain). Meta drops it after 30 seconds without a heartbeat (nodes heartbeat every 5 seconds), and the other nodes notice on their next registry poll, up to 10 seconds later. Meanwhile, work for the missing node goes to the next node in that segment's ranking: a cache miss, not an error.
• A node comes back with the same identity. Each query node keeps its identity in its cache directory, so a container restart on the same volume reclaims the same segments and finds its cache warm. A pod replaced onto a new volume gets a new identity: one removal plus one addition.
• A rolling upgrade is a sequence of those events, one node at a time, each moving only the share of the node being replaced.

Changing a node's CPUCopy Anchor Link

Changing a node's CPU limit changes its weight; ownership settles within a poll interval. Only that node gains or loses segments, and the other nodes never trade segments with each other. Raising a limit pulls a proportional slice of the others' segments onto the node; lowering it pushes a slice back out.

What ownership affectsCopy Anchor Link

Three parts of the system act on ownership, each computing it the same way:

• The query coordinator groups a query's segments by owner and sends each group to the node that has it cached. If the owner has left, the work goes to the next node in that segment's ranking; before the coordinator has seen the member list at startup, it spreads work round-robin. Both fallbacks cost cache hits, not correctness.
• The nursery and janitor push each new merged segment into its owner's cache before making it queryable, so the first query finds it cached. A declined or failed push means the segment is fetched from S3 on first use.
• Each query node's background warmer prefetches the index regions of the segments it owns, and skips the rest.

Operating notesCopy Anchor Link

• Scale by changing query.replicaCount. Ownership follows as the new pods register.
• Size caches per node, not per cluster. A node's cached working set is about its weight share of the total. If the cache sizing table gives 600 GB for the cluster, three equally weighted nodes need about 200 GB each, plus headroom for the moves after a membership change.
• The cache hit rate dips after scaling, in proportion to the share that moved, until the new node is warm. Data cannot be moved ahead; the new node reads it from S3.
• Nodes can differ in size. A 64-vCPU and a 32-vCPU node split the data 2:1, which keeps per-core load equal. The weighting does not account for per-core speed: two nodes with the same vCPU count are treated as equal even if one is faster per core.