Tamper Evidence

Berserk detects telemetry that was changed or removed after it was written. Every stored segment carries hashes of its own contents, and the metadata database keeps both each segment's digest and an append-only, hash-chained record of what was written, merged and deleted.

What is recordedCopy Anchor Link

• In each segment. When a segment is written to object storage, each of its chunks is hashed with a cryptographic hash function, and the hashes are stored in the segment along with hashes of its header and chunk index. The hash of that record, the segment's digest, covers every byte of the file.
• In the catalog. The digest is stored with the segment's catalog entry when the segment is registered, and cannot be changed afterwards.
• In the audit chain. Creating, merging or deleting a segment adds an event to an append-only chain kept per table. Each event includes the hash of the one before it, so an edited, removed or reordered event breaks the chain.

The daily check re-derives every link.

When it is checkedCopy Anchor Link

A query never returns data that fails these checks. A segment that fails at open is reported as a partial failure. A chunk that fails is first fetched again from object storage, in case the local cache was damaged; if it still fails, it is reported as a warning on the result.

The daily chain checkCopy Anchor Link

The daily check attests what it can still prove. If the chain itself has been altered, it withholds the attestation and logs an error. If a segment disagrees with the chain — missing, present without a recorded creation, or with a different digest — it logs that segment and goes on attesting the rest. A missing or unrecorded segment is also written into the chain as an event of its own once the disagreement persists.

Writing off a lost segmentCopy Anchor Link

When a segment's data is lost or corrupt, an admin or a maintainer can write it off with bzrk admin segment write-off <segment-id> --reason "<why>". The segment leaves the catalog and its events are no longer queryable. The chain records the write-off as an event that names who did it and the reason they gave, so the chain and the catalog still agree and the rest of the table stays verifiable.

Read-time verificationCopy Anchor Link

The query checks above are controlled by audtVerify, which is true by default.

With false, queries do not pay for hashing the data they read. The checksum still catches accidental corruption, but not a deliberate change that also rewrites the checksum.

Set it on both charts, because the nursery also answers queries:

query:
  config:
    audtVerify: false
nursery:
  config:
    audtVerify: false

Row structure checkCopy Anchor Link

Hashes prove that stored bytes did not change after they were written, not that they were right when written. Every row is also checked for structure: when it is written, at ingest and in merges, and again when a merge verifies its output. The check walks each value at every depth and each entry of the row's field index, and fails a row when:

• a value has an unknown type,
• a length runs past the value, field set or array that holds it,
• an array holds a different number of elements than it declares,
• a field index points anywhere but at the start of a value, or a value has no index entry,
• the rows of a stored block do not add up to the block, or to the number of rows it declares.

A row that fails when written is refused, and the write reports an error, so a malformed row is never stored. A merge whose output fails verification fails as a whole, and its inputs stay as they were.

The check is controlled by rowStructureCheck, which is true by default. It costs time on every row written and verified, mostly in merges. Turn it off only if merges cannot keep up, and set it on every chart that writes segments:

query:
  config:
    rowStructureCheck: false
nursery:
  config:
    rowStructureCheck: false
janitor:
  config:
    rowStructureCheck: false

LimitsCopy Anchor Link

Tamper evidence covers data after it is written. It cannot tell whether a compromised Berserk service wrote the wrong data in the first place.