Compaction and Garbage Collection
Last updated
This page explains how ToriiDB rewrites a database into a new snapshot, removes superseded files, and what Store.Close() does at shutdown.
Compaction
compact() runs only while the database write lock is held (from the write path or Store.Close()):
| Step | Action |
|---|---|
| 1 | Refuse with loadErr if replay failed, so an incomplete map never overwrites a good snapshot |
| 2 | Skip if there is no data and the database directory does not exist |
| 3 | Serialize unexpired entries into N.snap.tmp with write + fsync + close |
| 4 | os.Rename it to N.snap, then fsync the directory |
| 5 | Open N+1.log, swap it in, and only then close the old log |
| 6 | Garbage-collect in the background |
N is the current log number. os.Rename is used directly because the go-pkg move helper falls back to copy + remove across devices, which is not atomic. Opening the new log before closing the old one means the database never has a nil log handle.
Serialization writes CreatedAt as the timestamp and does not carry UpdatedAt, so after a restart from a snapshot, entries report no update time.
Garbage Collection
After compacting to snapshot N, snapshots < N-1 and logs <= N-1 are deleted. The previous generation (N-1.snap + N.log) stays on disk as the fallback for replay.
Store.Close() is idempotent (sync.Once) and for each loaded database:
- Waits for background vector jobs.
- Stops the expiry timer.
- Compacts under the write lock.
- Only if compaction succeeded, runs garbage collection synchronously, because a background goroutine would not run before the process exits.
- Closes the log file.
The first compaction error is returned. Skipping garbage collection after a failed compaction prevents a bogus high-numbered snapshot from causing valid older files to be deleted.