Replay and Recovery

Last updated 2026-10-03

This page explains how ToriiDB loads a database from its snapshot and logs, falls back to an older generation, and how to back the data up.

Replay

A database is loaded on first use (sync.Once):

1. Remove leftover *.tmp files.
2. List snapshot numbers from newest to oldest; with no snapshot, start from generation 0 (logs only).
3. For generation N, replay N.snap (if N > 0), then N+1.log, N+2.log, ... while they exist.
4. On an I/O error, remember it as loadErr and try the next older snapshot.
5. On success, clear loadErr and keep appending to the last existing log.

Lines are read with bufio.Reader.ReadBytes('\n'), so line length is unlimited. bufio.Scanner is not used because its 64 KB default line limit made a 70 KB value hide the entire database. Empty or malformed JSON lines are skipped; replay only fails on real read errors, such as a .snap path that is a directory.

Replay rebuilds SET entries with their expiry and vector, applies DEL/EXPIRE/EXPIREAT/PERSIST, and warms the parsed cache of JSON values.

Backup and Recovery

• Back up the whole data directory, including every db_N/ directory.
• Close the store (or stop writes) before copying if you need a consistent point-in-time copy.
• When the newest snapshot cannot be read, the database loads from the previous generation automatically.
• When no generation can be replayed, the process keeps accepting appends but refuses to compact, so no snapshot is overwritten.

Related Pages

• Files and Writes
• Compaction and GC
• Removed APIs