Architecture

This page describes how ToriiDB routes commands, stores entries, coordinates concurrent access, and persists data.

System Overview

graph TB
    A[REPL Client] --> B[Store or Session]
    B --> C[Exec Router]
    C --> D[Key-Value Commands]
    C --> E[JSON Field Commands]
    C --> F[Query Commands]
    C --> G[Vector Commands]
    D --> H[Active DB 0-15]
    E --> H
    F --> H
    G --> H
    H --> I[In-Memory Entry Map]
    I --> J[AOF]
    I --> K[Per-Key JSON Files]
    G --> L[OpenAI Embedder]
    L --> M[Internal Vector Cache]
    M --> H

Components

Component	Source	Responsibility
REPL	`cmd/test/main.go`	Reads commands, displays the active database, and forwards input to `Exec`
Store lifecycle	`core/store/store.go`	Creates 16 databases, sessions, cleanup timer, and shutdown compaction
Command router	`core/store/exec.go`	Parses command tokens and dispatches key, field, TTL, query, and vector operations
Entry engine	`core/store/set.go`	Detects value types and keeps raw values, parsed JSON, metadata, and vectors synchronized
Persistence	`core/store/aof.go`	Appends AOF records, replays startup state, and compacts logs
Query engine	`core/store/find.go`, `query.go`, `filter/`	Scans primitive values and evaluates JSON filter expressions
Vector engine	`core/store/vector.go`, `vcache.go`, `vsearch.go`, `vsim.go`	Encodes embeddings, caches them, and performs cosine-similarity search

Store and Database Lifecycle

store.New() creates one Store with slots for databases 0 through 15. Each database owns an independent sync.RWMutex, map[string]*Entry, directory, AOF handle, and lazy-load guard. Database files are not opened until the database is first accessed or written.

A Session shares the same database array, embedder, and background-work wait group while maintaining its own selected database index. Select only changes that session's index.

A cleanup timer runs once per minute and removes expired entries from loaded databases. Close waits for vector attachment jobs, cancels cleanup, and compacts every loaded database concurrently.

Command Data Flow

The REPL sends an input string to Exec.
Exec tokenizes the input with strings.Fields and selects a command handler.
Dot-notation keys are split into a top-level key and nested JSON path when applicable.
The handler loads the active database and takes a read or write lock.
Write handlers update the in-memory entry, its per-key JSON file, and the AOF.
Exec converts the handler result to the command-line response format.

Library callers may bypass Exec and call promoted Store or Session methods directly.

Storage Model

Layer	Representation	Behavior
Memory	`map[string]*Entry` per database	Primary runtime state protected by the database mutex
AOF	`<root>/db_N/record.aof`	Newline-delimited JSON records replayed when a database loads
Entry file	`<root>/db_N/xx/yy/zz/<md5>.json`	Full serialized entry written atomically through a temporary file and rename
Vector cache	`__torii:embed:<sha256>` entries	Stores deterministic embeddings by model, dimension, and text

The default root is ./temp; callers can pass a different root to store.New(path). The three-level entry-file path is derived from the MD5 hash of the key.

AOF Replay and Compaction

AOF records contain a timestamp, command, key, optional value, optional expiration, and optional base64-encoded vector. Replay rebuilds in-memory entries, restores expirations and vectors, and warms the parsed cache for JSON values.

The log is compacted when its size reaches twice the larger of its post-load baseline and 1 MiB. Compaction writes one current SET record per live entry and drops expired or superseded history. Shutdown also compacts each loaded database.

Concurrency and Cache Invariants

Database maps, entry mutation, and parsed JSON objects are protected by the per-database mutex.
parseAndCache mutates the parsed cache and is used only under a write lock or during single-threaded replay.
cached is read-only and is used while a read lock remains held.
JSON write paths call setParsed, keeping the raw value and parsed object synchronized.
Entry files are serialized with Entry.JSON() so the unexported raw value is included.
Embedding HTTP requests run without a database lock; attachment occurs later under the write lock and only if the entry value still matches the embedded text.

Expiration and Internal Keys

Point reads treat expired entries as missing and may remove them lazily. Scan operations skip expired entries without returning them. The periodic cleanup process removes expired files from loaded databases.

Keys beginning with __torii: are reserved for internal embedding cache entries. Scan-style commands such as KEYS, FIND, QUERY, and VSEARCH hide them, while point operations remain available for debugging.