RustFS Chunk Format Specification — M0 (EN)
RustFS Chunk Format Specification — M0 (EN)
Document: RUSTFS-SPEC-CHUNK-001
Status: DRAFT — FREEZE TARGET for milestone M0
Version: format_version = 1
Scope: The on-disk shard file format, streaming bitrot, chunk↔shard
mapping, the chunk-key path scheme, and the durable write/read
protocols. This is the foundation of chunk-store and the
self-verifying integrity model.
Language: English (中文镜像见 rustfs-chunk-format-ZH.md)
Endianness: little-endian, unless stated otherwise
1. Terminology and the Storage Hierarchy
File a POSIX file / S3 object / block volume's logical bytes.
Chunk a fixed logical slice of a file: chunk_size bytes (default 4 MiB).
Identified by (file_id, version, chunk_index).
Shard the unit actually stored on one disk. A chunk is transformed into
shards by the storage policy:
· EC(k,m) → k data shards + m parity shards
· Replica(r) → r identical shards (k=1, m=0 conceptually)
Shard file the on-disk file holding one shard, in the format defined here.
Bitrot block the unit over which one integrity hash is computed inside a shard
file (default 16 KiB). A shard file's data region is a sequence of
bitrot blocks; each has a HighwayHash-256 entry in the hash table.
EC block the erasure-coding unit (default 1 MiB) over original chunk bytes;
each EC block produces one shard_fragment per data shard.
File ──split──▶ Chunks ──EC or Replica──▶ Shards ──stored as──▶ Shard files
│
[ Header | HashTable | Data ]
Data = bitrot blocks, each hashed
Authority note: the inode Layout in the KV is authoritative for placement
and decode parameters. The shard-file header (below) duplicates the essential
parameters so each shard is self-describing for verification, scrub, and
disaster recovery. If header and inode disagree, that is a corruption signal —
the header is never used to override placement on the normal path.
2. Shard File Layout (top level)
A shard file consists of three contiguous, 4 KiB-aligned regions:
Offset Region
───────────────────────────── ──────────────────────────────────────────────
0 Header (exactly 4096 bytes)
4096 Hash Table (align_up(block_count*32, 4096) B)
4096 + hash_table_size Data (the shard's bytes; bitrot blocks)
Rationale for the split layout (hashes separated from data, not interleaved):
the Data region begins on a 4 KiB boundary and every bitrot block is a multiple
of 4 KiB, so reads are O_DIRECT-friendly and the byte at shard-offset O is
always at file offset data_start + O — clean math for random reads. The hash
for the bitrot block containing O is at 4096 + (O / bitrot_block) * 32.
3. Header (4096 bytes, fixed)
Off Size Field Notes
─── ──── ──────────────────── ────────────────────────────────────────────
0 8 magic ASCII "RFSHARD1"
8 2 format_version (u16) = 1
10 2 flags (u16) see §3.1
12 4 header_crc32c (u32) CRC32C (Castagnoli) over bytes [16, 4096)
16 8 file_id (u64)
24 8 chunk_index (u64)
32 8 version (u64) object/file version (1 if unversioned)
40 2 shard_index (u16) 0..k+m-1 (EC) or 0..r-1 (replica)
42 2 ec_k (u16) data shards (1 for replica)
44 2 ec_m (u16) parity shards (0 for replica)
46 1 ec_algo (u8) 0=none/replica, 1=rs-vandermonde
47 1 csum_algo (u8) 1=HighwayHash256
48 4 bitrot_block (u32) bytes per hashed block; multiple of 4096
52 8 shard_data_len (u64) logical bytes of THIS shard's data region
60 8 chunk_logical_len(u64) logical bytes of the original chunk
68 4 block_count (u32) = ceil(shard_data_len / bitrot_block)
72 8 created_unix_nanos(u64)
80 8 ec_block_size (u64) erasure block size over chunk bytes (e.g. 1 MiB)
88 8 placement_seed (u64) echo of Layout.placement_seed (cross-check)
96 8 cluster_map_epoch(u64) epoch under which this shard was placed
104 8 chunk_size (u64) file's chunk_size (echo of Layout)
112 8 content_len (u64) pre-transform logical length of chunk payload
(before compression/encryption), for the last
chunk's short tail
120 4904 reserved MUST be zero on write; ignored on read
3.1 flags (u16, bitfield)
bit 0 mode_ec 1 = EC(k,m) data/parity shard; 0 = replica copy
bit 1 compressed payload was compressed before hashing (algo in Layout)
bit 2 encrypted payload was encrypted before hashing (key ref in Layout)
bit 3 last_block_short the final bitrot block is < bitrot_block bytes
bit 4 parity_shard this shard is a parity shard (shard_index >= ec_k)
bits 5..15 reserved (0)
Transform order (when flags set): the stored payload = encrypt(compress(raw)). Bitrot hashes are computed over the stored payload (post-transform), so integrity protects exactly the on-disk bytes.
4. Hash Table (region 2)
block_count entries, each 32 bytes (HighwayHash-256 digest), in block order.
Entry i covers Data bytes [i*bitrot_block, min((i+1)*bitrot_block, shard_data_len)).
Region padded with zeros to the next 4096 boundary.
4.1 HighwayHash-256 keying (position-bound)
Each block's hash uses a 256-bit key derived from the cluster bitrot key
K = (K0,K1,K2,K3) (4×u64, configured per cluster) XOR-mixed with the block's
position, so a correct block at the wrong position fails verification:
key0 = K0 ^ file_id
key1 = K1 ^ chunk_index
key2 = K2 ^ ((shard_index as u64) << 32 | block_index as u64)
key3 = K3 ^ version
digest_i = HighwayHash256(key=(key0,key1,key2,key3), data = data_block_i)
This binds every block to (file, chunk, shard, version, block index), catching misdirected reads/writes in addition to bit rot.
5. Data Region (region 3)
shard_data_len bytes, conceptually block_count bitrot blocks laid out
contiguously. All blocks are bitrot_block bytes except possibly the last
(flags.last_block_short). The Data region is NOT padded between blocks; only the
whole file may be padded to 4 KiB at EOF for O_DIRECT writes (trailing pad bytes
beyond shard_data_len are ignored).
6. Chunk ↔ Shard Byte Mapping (EC mode)
For EC(k,m) with ec_block_size (EB), the chunk's original bytes are processed
in EB-sized erasure blocks; each EB is split into k equal shard_fragments:
shard_fragment = ceil(EB / k) (last data shard zero-padded if needed)
To locate chunk byte at chunk-offset C (0-based within the chunk):
eb_index = C / EB # which erasure block
off_in_eb = C % EB
data_shard = off_in_eb / shard_fragment # which DATA shard (0..k-1)
off_in_fragment = off_in_eb % shard_fragment
shard_offset = eb_index * shard_fragment + off_in_fragment
So a contiguous chunk range can touch up to k data shards; a small range
(< shard_fragment, within one EB) touches exactly ONE data shard → 1x read
amplification (the random-read fast path; no parity, no reconstruction unless a
needed data shard is unhealthy).
For Replica mode the chunk bytes are stored verbatim in each shard (shard_offset = chunk_offset); reads pick any healthy replica.
7. Chunk-Key and On-Disk Path Scheme
Logical chunk-key: (file_id, version, chunk_index)
Shard identity: (file_id, version, chunk_index, shard_index)
On-disk path:
{disk_root}/.chunks/{p0}/{p1}/{file_id:016x}.v{version}/c{chunk_index}/{shard_index}.shard
where:
p0 = hex byte 0 of h = xxh3_64(file_id) # 00..ff
p1 = hex byte 1 of h # 00..ff
→ two-level fan-out (≤ 65536 leaf groups) to bound entries per directory.
Temp during write:
{...}/{shard_index}.shard.tmp.{random_u64:016x}
The directory fan-out keys on file_id only (not chunk_index) so all chunks of
a file cluster together on a disk — good for sequential scrub/heal locality.
8. Durable Write Protocols
8.1 Whole-shard write (new chunk; dataset write path)
1. Compute bitrot hashes for all blocks (over the post-transform payload).
2. Write Header, Hash Table, Data to {path}.tmp.{rand}.
3. fsync(tmp).
4. rename(tmp → final) # atomic on the same filesystem
5. fsync(parent directory) # persist the rename
8.2 Partial write (block volumes / random in-place writes)
Writes must be handled at bitrot_block granularity (read-modify-write):
For each bitrot block B overlapping the write range:
1. read block B's data + its hash entry; verify (skip verify if block is being
fully overwritten).
2. apply the new bytes into B's buffer.
3. recompute digest for B.
4. write the data block (O_DIRECT-aligned) ; write the 32-byte hash entry.
5. fdatasync (or O_DSYNC).
Torn-write window between the data block and its hash entry is tolerated: on a later read, a mismatch triggers reconstruction (EC) or an alternate replica read + repair. Block volumes SHOULD use Replica mode (see RUSTFS-MD-DESIGN-005) precisely so partial writes avoid EC read-modify-write amplification.
9. Read Protocol (verified byte-range read of a shard)
Input: shard-relative range [O, O+L)
first = O / bitrot_block ; last = (O + L - 1) / bitrot_block
data_start = 4096 + align_up(block_count*32, 4096)
for i in first..=last:
blk = read(data_start + i*bitrot_block, this_block_len(i)) # O_DIRECT ok
h = read_hash_entry(i) # 32 bytes
if HighwayHash256(key_i, blk) != h: return BitrotError(i)
return concat(blocks)[ O - first*bitrot_block .. + L ]
On BitrotError the chunk-store does NOT fail the client read: for EC it reads
the other shards and reconstructs the affected EC block(s); for Replica it reads
another replica. The bad shard is queued for heal (background).
10. Defaults and Tunables
Parameter Default Notes
─────────────── ─────────── ──────────────────────────────────────────────
chunk_size 4 MiB per-file (Layout)
ec_block_size 1 MiB EC encode unit; chunk_size multiple of it
bitrot_block 16 KiB 4×4 KiB; SMALLER for point-read-heavy files
(less verify amplification), LARGER for
sequential (less hash-table overhead).
MUST be a multiple of 4096.
csum_algo HighwayHash256
ec_algo rs-vandermonde
dir fan-out 2 levels xxh3_64(file_id) bytes 0,1
Overhead at defaults: hash table = 32 B per 16 KiB ≈ 0.195% of shard size.
11. Container & Slice Layout (small-file packing)
Small files (AI samples/tokens; node_modules, pip/conda, build temp — billions of them) are NOT stored as their own shards. They are PACKED as needles into a container, and the file's inode holds a Slice pointer (see RUSTFS-MD-DESIGN-005 §3.5). This section defines the on-disk needle layout, the Slice pointer fields, and how a slice read maps onto the §6 chunk↔shard mapping and the §9 verified read — so packing reuses the existing chunk format, EC, and bitrot unchanged.
11.1 Container
A container is a large INTERNAL file (default capacity 256 MiB) in a reserved
namespace. It is stored EXACTLY as a normal large file:
· its own file_id (= container_id),
· its own Layout (formula placement + EC, or Replica) — §3.2 of DESIGN-005,
· its bytes are chunks in the §2 shard format, EC'd / bitrot'd as usual.
Containers are append-only until SEALED at capacity; sealed containers are
immutable (ideal for EC). Container storage policy follows the size×access-mode
rule at the CONTAINER level (write-once dataset packs → EC; high-churn temp packs
→ Replica). This is opaque to the needle layer below.
11.2 Needle (one small file inside a container)
A container's logical byte stream is a sequence of needles:
Needle = [ NeedleHeader (32 B) | Data (data_len bytes) | pad to needle_align ]
NeedleHeader (little-endian, 32 bytes):
Off Size Field Notes
0 4 magic "RFND" (0x52464E44)
4 4 flags bit0 deleted, bit1 compressed, bit2 encrypted
8 8 file_id owning inode (enables KV-light compaction & scrub)
16 4 data_len logical bytes of Data
20 4 cookie random; must match the Slice pointer (anti-stale/-corrupt)
24 4 data_crc32c CRC32C over Data (fast needle-level check)
28 4 reserved 0
needle_align : default 8 bytes. Needle starts are multiples of needle_align in
the container's logical stream. Waste ≤ needle_align-1 per needle.
ALIGNMENT LAYERING (important): needle_align is for PACKING DENSITY only.
Device / O_DIRECT / EC-fragment alignment is handled entirely at the chunk
layer (§2/§5/§6) when the container bytes are read. Needles do NOT need device
alignment — so tiny files pack tightly (8 B granularity) while reads stay
O_DIRECT-friendly at the chunk layer beneath.
11.3 Slice pointer (in the small file's inode)
inode.body = Slice {
container_id : u64, // the container's file_id
offset : u64, // byte offset of the NeedleHeader in the container stream
length : u32, // = data_len (size the read without parsing the header)
cookie : u32, // must equal NeedleHeader.cookie
}
This is the ONLY per-small-file data structure, and it lives in the inode row (KV). There is no separate small-file index and no bulk data in the KV.
11.4 Slice read → EC partial read (offset mapping)
A small-file read reuses §6 + §9 entirely; nothing new on the data path:
Given Slice{container_id, offset, length} and the container's cached Layout:
1. Container byte range to fetch: [offset, offset + 32 + length)
(header + data; the header lets us validate magic/file_id/cookie).
2. Map container offset Oc → container chunk & EC shard (this IS §6):
chunk_index = Oc / container.chunk_size
off_in_chunk = Oc % container.chunk_size
(eb_index, data_shard, shard_offset) = §6( off_in_chunk, EB, k )
3. Issue the §9 verified byte-range read on the container's shard(s):
bitrot-verified; EC-partial (touches only the data shard(s) covering the
range; NO reconstruction unless a needed data shard is unhealthy).
4. Parse NeedleHeader; check magic == "RFND", file_id == inode, cookie ==
Slice.cookie; optionally verify data_crc32c.
5. Return Data[0 .. length].
Amplification: a needle smaller than shard_fragment (= ceil(EB/k)) lands within
ONE EC data-shard fragment in the common case → 1× read (one data shard). If it
straddles a fragment boundary it touches 2 data shards — still no parity, no
reconstruction. Small-file reads thus inherit the LLD-002 random-read fast path.
The MDS is NOT on this path (the client caches the container Layout, and
containers are few and hot).
11.5 Append, seal, delete, compaction
Append : reserve a needle slot at the container's current append offset (one
atomic bump per write; one OPEN container per writer/session avoids
cross-writer contention). Write the needle into the open container's
chunk(s); then commit inode.body = Slice{...}. Data is durable (container
chunk write, §8) BEFORE the inode commit (data-first; crash GC per
DESIGN-005 §3.4).
Seal : when used ≥ capacity, mark CMTA.sealed; the container becomes immutable
and is finalized as a normal write-once EC file.
Delete : logical only — drop the small inode and bump CMTA.dead_bytes. The sealed
container is NOT modified in place (immutable / EC).
Compact: when CMTA.dead_bytes/capacity ≥ threshold (Q6), the compactor scans the
container's needles; for each needle with file_id F at offset O it checks
inode F: if F.body == Slice{this_container, O, …} the needle is LIVE →
copy into a fresh open container and update F's Slice transactionally;
else DEAD → skip. When done, free the old container. (The needle's
file_id makes this liveness check possible without a reverse index.)
11.6 Defaults
container_capacity 256 MiB (Q6 tunable; smaller = faster compaction turnover)
container_policy EC(k,m) for write-once packs; Replica for high-churn packs
needle_align 8 bytes
NeedleHeader 32 bytes (fixed)
compaction trigger dead_bytes/capacity ≥ ~0.4 (Q6 tunable), rate-limited vs I/O
12. Format Versioning & Compatibility
· format_version in the header gates all parsing. Readers MUST reject versions
they do not understand.
· reserved header bytes MUST be zero on write and ignored on read, allowing
additive fields in future minor revisions without a version bump ONLY if the
field's absence (zero) is a valid default.
· Any change to region layout, hashing, or mapping REQUIRES bumping
format_version and providing a reader for the new version.
· There is no migration from older formats: this is a greenfield system
(RUSTFS-MD-DESIGN-005). format_version 1 is the first and only format at GA.
13. Golden Test Vectors (interop contract)
Two independent implementations MUST agree byte-for-byte. The reference
implementation generates and commits golden vectors under testdata/chunk/,
each vector = { params.json, input.bin, expected.shard }.
A vector MUST exercise, at minimum:
G1 Replica, single full bitrot block (data_len = bitrot_block).
G2 Replica, short last block (data_len = 1.5 × bitrot_block).
G3 EC(4,2) data shard 0, multi-EC-block chunk (chunk = 4 MiB, EB = 1 MiB).
G4 EC(4,2) parity shard (shard_index = 4), same chunk as G3.
G5 Compressed + encrypted payload (flags bits 1,2), verifying bitrot is over
the post-transform bytes.
G6 Position-binding: a valid block from G3 placed at the wrong block_index MUST
fail HighwayHash verification (negative vector).
G7 Packing (Replica container): 3 needles written into one container; each reads
back bit-exact via its Slice; NeedleHeader magic/file_id/cookie validated; a
Slice with a wrong cookie MUST be rejected (negative vector).
G8 Packing (EC container): a needle wholly inside one shard_fragment reads from
exactly ONE data shard (EC partial read, no reconstruction); a needle that
straddles a fragment boundary reads from exactly TWO data shards.
13.1 Worked structural example (G2, tiny sizes for illustration)
Using illustrative bitrot_block = 16 bytes (real default is 16 KiB), Replica
mode, shard_data_len = 24 (one full block + an 8-byte short block):
block_count = ceil(24/16) = 2
Header : bytes [0,4096) magic="RFSHARD1", format_version=1, flags=0x0008
(last_block_short), bitrot_block=16, shard_data_len=24,
block_count=2, ec_k=1, ec_m=0, ec_algo=0, csum_algo=1, ...
HashTable: bytes [4096,8192) (align_up(2*32=64, 4096)=4096)
[4096,4128) = HighwayHash256(key_0, data[0..16]) # 32 B
[4128,4160) = HighwayHash256(key_1, data[16..24]) # 32 B
[4160,8192) = 0x00 padding
Data : bytes [8192, 8192+24) = the 24 payload bytes
file padded to 8192+4096 (next 4 KiB) for O_DIRECT; pad ignored.
(The actual 32-byte digests are produced by the reference HighwayHash256 with
the keying of §4.1 and committed in expected.shard; they cannot be hand-
computed here. The structural offsets above are the normative contract.)
14. Conformance Checklist (M0 exit)
□ Header encodes/decodes all fields; header_crc32c validated on read.
□ Split layout offsets exactly as §2; Data starts 4 KiB-aligned.
□ HighwayHash256 keying per §4.1 (position-bound) implemented.
□ Chunk↔shard mapping (§6) matches rustfs-placement's expectation.
□ Path scheme (§7) and atomic write (§8.1) implemented.
□ Partial RMW (§8.2) for Replica mode; torn-write tolerated via repair.
□ Verified read (§9) returns BitrotError on mismatch; never silent.
□ NeedleHeader (§11.2) encode/decode; magic + file_id + cookie + data_crc32c
validated; Slice read (§11.4) maps to §6/§9 (container range → EC partial read).
□ A needle < shard_fragment touches ≤ 2 data shards, no reconstruction on healthy
data; compaction liveness via needle.file_id ↔ inode.Slice; sealed container
never modified in place.
□ All golden vectors G1–G8 pass in two independent decoders.
□ format_version gate rejects unknown versions.
Chinese mirror: rustfs-chunk-format-ZH.md. This spec is the M0 freeze artifact for RUSTFS-PLAN-001; it underpins chunk-store (Track B) and rustfs-placement.