Hardware Sizing
Cores, memory, disk and object storage for a Berserk cluster, from ingest volume and retention
What drives the size
The size follows from three properties of your workload:
- Ingest per day: how much telemetry you send, measured as uncompressed OTLP data.
- Retention: how long you keep data. This sets object storage capacity.
- Query cache retention: how many days you want on local disk for fast searches. This sets the disk on query nodes.
The coefficients below come from measured installations and scale linearly. Check them against your cluster's metrics.
Calculator
Compute
Storage and network
Example machines
The numbers
Compression: 10x, depending on your data. Ingest is measured as uncompressed OTLP protobuf; stored segments are about 10x smaller. Repetitive logs compress better, high-cardinality traces worse.
Ingest: 1 TB/day per core. One physical core ingests 1 TB a day through the whole pipeline: receiving, converting to segments and merging. A core is a physical core; on x86 cloud instances that is two vCPUs, on ARM instances (Graviton, Axion) one vCPU is one core.
Query: start at 4x the ingest cores. Query cost depends on the search load, so there is no formula. Four times the ingest cores fits normal interactive use; heavy dashboards or many concurrent users need more.
Memory: 2 GB per core. Many parallel queries, or queries with large state, use more memory than the ingest pipeline (ingest, nursery and janitor).
Query cache disk: ingest / compression x cache days. Query nodes cache segments from object storage on local SSD or NVMe. Size the cache for the number of days you want available for fast searches. Older data is still searchable, read from object storage.
| Resource | Rule |
|---|---|
| Compression | 10x, data dependent |
| Ingest CPU | 1 TB/day per core |
| Query CPU | 4x ingest cores, starting point |
| Memory | 2 GB per core |
| Query cache disk | ingest / compression x cache days, SSD or NVMe |
| Minimum cluster | 4 cores, 8 GB RAM |
The coefficients are for average ingest. If your peak hour is twice your average, size for the peak.
- PostgreSQL: a small instance, 2 cores and 2 GB, with a little disk. Do not use a burstable instance type.
- Object storage: ingest / compression x retention days of capacity (details).
- Network: 10 TB/day of ingest is about 1 Gbit/s inbound.
Examples
| Example | Cores (ingest + query) | RAM | Cache disk | S3 | Example machines |
|---|---|---|---|---|---|
| 1 TB/day, 30 days retention, 3 days cache | 1 + 4 = 5 | 10 GB | 300 GB | 3 TB | Single node: 8 cores (16 vCPU), 16 GB RAM, 400 GB NVMe Highly available: 3 nodes of 4 cores (8 vCPU), 8 GB RAM, 200 GB NVMe |
| 10 TB/day, 1 year retention, 7 days cache | 10 + 40 = 50 | 100 GB | 7 TB | 365 TB | Larger nodes: 4 nodes of 16 cores (32 vCPU), 64 GB RAM, 2 TB NVMe Smaller nodes: 7 nodes of 8 cores (16 vCPU), 16 GB RAM, 1.5 TB NVMe |
| 100 TB/day, 90 days retention, 7 days cache | 100 + 400 = 500 | 1 TB | 70 TB | 900 TB | Larger nodes: 16 nodes of 32 cores (64 vCPU), 128 GB RAM, 5 TB NVMe Smaller nodes: 32 nodes of 16 cores (32 vCPU), 64 GB RAM, 2.5 TB NVMe |
Machine suggestions assume x86 (vCPU counts are two per core) and one query node per machine; the NVMe per node is its share of the query cache with 10% headroom. Where the whole cluster fits on one machine, the table shows that option first; a single node is not highly available. Three or more nodes tolerate the loss of one; see Dependencies for the rest of a highly available setup.
Details per service
Ingest runs as three kinds of pods. For Helm values, the ingest cores split roughly 60% to the nursery, 40% to the janitor and 5% to the collector.
Ingest collector
Receives OTLP, batches it, and uploads raw batches to object storage. About 5% of the ingest cores; scale it for burst, not for average rate. Memory absorbs bursts: in-flight data is capped at 60% of the pod's memory limit, so the limit decides how many seconds of incoming data the collector can hold while object storage is slow. Plan for at least 1 GB. No local disk.
Nursery
Converts raw batches into segments and merges them locally. About 60% of the ingest cores, in a few large pods rather than many small ones. Memory is 2 GB per core. It needs 50 GB of local working disk per pod for baby segments; node storage is enough.
Keep a CPU limit on nursery pods. The nursery reads its limit as the cores it has, and meta sizes the nursery's capacity from it, at about 10 MB/s of uncompressed telemetry per CPU; a nursery over half of that hands work to a less loaded one, and under it keeps its data together for better merges. A pod with no limit reports the whole node and never spreads. Outside Kubernetes, or where the limit does not reflect the intended share, set --available-cores on the nursery to the cores it should count as its own.
Janitor
Merges segments in object storage into larger ones. About 40% of the ingest cores, but a merge is single-threaded, so run it as replicas of one core each rather than one large pod. Each replica uses whatever memory limit it is given for merge buffers and does more passes with less; plan for 2 to 4 GB per replica. Each replica needs 30 GB of scratch disk at the default settings, and that is a hard cap: a replica that overruns it is evicted mid-merge.
Query
Executes searches and holds the segment cache. Cores and memory are the 4x and 2 GB per core starting points above. Each query pod owns a share of the cache proportional to its CPU limit, so with one query pod per node the cache disk is spread evenly across nodes; provision each node's volume a little larger than its share. Memory beyond what queries use becomes page cache over the segment cache. See Storage and Segment Placement.
Meta
The metadata service. It tracks tables, segments and ingest streams in PostgreSQL and answers the lookups every query starts with, so its latency is on the path of every query. It needs a fraction of a core and a few hundred MB of memory at any scale; keep it close to the database.
Control plane
The UI, gateway and permissions service together use a fraction of a core and under 1 GB of memory at any scale.
PostgreSQL
Holds metadata: segments, streams, users and permissions. It is small and grows with the number of segments, but it sits on the path of every query, so a slow or throttled database stalls the whole cluster. 2 cores and 2 GB on a non-burstable instance is enough for the clusters on this page.
Object storage
Capacity is ingest / compression x retention days, plus the raw batches waiting to be converted, which is a day or less of ingest. Every byte is written and read more than once as it moves from raw batch to segment to merged segment, so sustained throughput matters as much as capacity. Choosing and configuring the object store is not covered here.