v0.7.0
Home GitHub

Persistence

Last updated

This page explains ToriiDB's on-disk layout, the record format shared by snapshots and logs, and how each write is made durable.

Storage Layout

Each database has a directory containing numbered snapshot and log files:

<dir>/
└── db_0/
    ├── 00000002.snap
    ├── 00000003.log
    ├── 00000003.snap
    └── 00000004.log
File Content
NNNNNNNN.snap Full state: one SET line per live entry at compaction time
NNNNNNNN.log Operations appended after the snapshot with the previous number
*.tmp Snapshot being written; removed on the next load if a crash left it behind

Snapshot N contains everything up to and including log N. Names must be exactly eight digits followed by .snap or .log; anything else in the directory is ignored.

There is no per-key file mirror. The snapshot and logs are the only on-disk source of truth.

Record Format

Snapshots and logs share one line format, a JSON AOFRecord followed by \n:

Field Meaning
ts Unix timestamp of the operation (in snapshots: the entry's CreatedAt)
cmd SET, DEL, EXPIRE, EXPIREAT, or PERSIST
key Affected key
value Value for SET; seconds or timestamp text for EXPIRE/EXPIREAT
expire_at Optional absolute expiry
vector Optional base64 little-endian float32 vector

Write Path

Every mutation runs under the database write lock:

  1. Update the in-memory entry.
  2. Open the current log with O_CREATE|O_APPEND|O_WRONLY if it is not open yet.
  3. Append the record and call File.Sync().
  4. If log size >= max(snapshot size, 1 MiB) * 2, compact.

The threshold is relative to the snapshot on purpose: a fixed threshold would rewrite the whole database on almost every write once live data grows close to it.

Vector attachment and embedding-cache writes add their own SET records. The once-per-minute expiry sweep only deletes from memory and writes nothing.

中文