Documentation
Home GitHub

Persistence

Understand how ToriiDB combines in-memory state, append-only logs, per-key files, replay, compaction, and expiration cleanup.

Storage Layout

A storage root contains one directory per logical database:

<root>/
└── db_0/
    ├── record.aof
    └── xx/yy/zz/<md5>.json

The default root is ./temp. store.New(path) can select another root. All 16 database descriptors are created at startup, but individual database directories and AOF handles are initialized lazily.

In-Memory State

Each database owns a map[string]*Entry protected by a sync.RWMutex. This map is the runtime source of truth. An entry contains its value, detected type, timestamps, optional expiration, optional vector, and parsed JSON cache.

Databases are independent: selecting another database changes which map and persistence directory a session uses.

Per-Key JSON Files

Every write stores the complete entry in a JSON file. The path is derived from the key's MD5 hash and split into three two-character directory levels:

<root>/db_N/ab/cd/ef/abcdef....json

utils.WriteFile creates parent directories, writes to a sibling temporary file, then renames it over the target. Entry writers call Entry.JSON() instead of marshaling Entry directly so the unexported raw value is included.

Deleting a key removes its entry file. Periodic expiration cleanup also removes the corresponding file.

Append-Only Log

record.aof contains one JSON object per line. Each AOFRecord may include:

Field Meaning
ts Unix timestamp of the operation
cmd SET, DEL, EXPIRE, EXPIREAT, or PERSIST
key Affected key
value Value for SET records
expire_at Optional absolute expiration timestamp
vector Optional base64 little-endian float32 vector

Each append is synchronized with File.Sync. Write handlers normally update memory, the per-key file, and then the AOF while holding the database write lock.

Replay

The first access to a database invokes replayAOF once. Replay:

  1. Reads newline-delimited records.
  2. Ignores empty or malformed lines.
  3. Reconstructs SET values and vectors.
  4. Applies deletion and expiration operations.
  5. Warms the parsed cache for JSON values.
  6. Records the valid log size as the compaction baseline.

If no AOF exists, the database starts empty. Per-key JSON files are cache artifacts; startup reconstruction is driven by the AOF.

Compaction

Automatic compaction starts when current AOF size reaches twice the larger of:

Compaction emits one current SET record for each unexpired entry, including expiration and vector data, and discards superseded history. If no entries remain, it removes the AOF.

Store.Close() waits for background vector jobs, stops cleanup, and compacts every loaded database concurrently. Call it during graceful shutdown.

Expiration Cleanup

Expiration is enforced at several layers:

Path Behavior
Point reads Treat expired entries as missing and may remove them from memory
Scan operations Skip expired entries
Cleanup timer Once per minute, removes expired entries and files from loaded databases
Compaction Excludes expired entries from the rewritten AOF

A key therefore becomes logically absent at its expiration timestamp even before its file is physically removed.

Backup and Recovery

中文