v0.7.0
Home GitHub

Architecture

Last updated

This page shows how ToriiDB's layers fit together: the socket daemon, the embedded store, per-database state, persistence, and the embedding client.

System Overview

graph TB
    APP[Application process] --> NEW["daemon.New(dir)"]
    NEW -->|socket answers| CLI[ModeClient: pooled unix socket client]
    NEW -->|no live socket| SRV[ModeServer: owns store.Store]
    CLI -->|4-byte length + JSON frame| SOCK["/tmp/toriidb-uid-hash.sock"]
    SOCK --> DISP["dispatch: Session() + Select(db)"]
    SRV --> DISP
    REPL[cmd/test REPL] --> EXEC["Store.Exec command router"]
    DISP --> CORE[store core: KV, JSON fields, TTL, scans, vectors]
    EXEC --> CORE
    CORE --> DBS["db_0 ... db_15: RWMutex + map of Entry"]
    DBS --> DISK["NNNNNNNN.snap + NNNNNNNN.log generations"]
    CORE --> EMB["openai.Client (text-embedding-3-small)"]
    EMB --> CACHE["__torii:embed:* cache entries"]
    CACHE --> DBS

Layers

Layer Source Responsibility
Socket daemon core/daemon/daemon.go, socket.go One server per data directory; later New calls on the same directory become clients
Wire protocol core/daemon/protocol.go, client.go, handler.go Length-prefixed JSON frames, connection pool, op dispatch, error kinds
Command router core/store/exec.go Parses Redis-like text commands for the REPL and Exec callers
Store lifecycle core/store/store.go 16 lazy databases, sessions, expiry timer, shutdown compaction
Entry engine core/store/set.go, get.go, setField.go, del.go, incr*.go, ttl.go Type detection, JSON field mutation, TTL, parsed-cache invariants
Scans core/store/keys.go, find.go, query.go, filter/ Glob key listing, value comparison, JSON filter expressions
Persistence core/store/aof.go, gc.go Append + fsync, snapshot compaction, generation fallback, garbage collection
Vectors core/store/vector.go, vcache.go, vsearch.go, vsim.go, core/openai/client.go Embedding, cache, cosine top-K search

Cross-Cutting Principles

Principle Where it shows up
Socket identity is a pure function of location resolveDir runs CheckDir → filepath.Abs → filepath.EvalSymlinks; the socket lives in hard-coded /tmp, never $TMPDIR
Callers never branch on mode Daemon exposes one method set; server mode dispatches in-process, client mode sends a frame
Every request is isolated Each daemon request creates its own Session and Selects its database, because the selected index is an unlocked int
Memory is authoritative, disk is replayable Writes update the map, then append a synced log line; startup replays snapshot + logs
Destructive rewrites are guarded Compaction refuses to run after a failed replay and only garbage-collects old generations after it succeeds
Internal keys stay internal __torii:* keys are skipped by KEYS, FIND, QUERY, and VSEARCH; point reads still see them

Further Reading

中文