Architecture

Understand how KuraDB separates lifecycle management, read-only queries, indexing writes, durable storage, and rebuildable caches.

System overview

graph TB
    CLI["kura CLI"] --> Registry["Registry<br/>db.json"]
    CLI --> Daemon["Background daemon"]
    Daemon --> API["Gin read-only API"]
    Daemon --> Watcher["Filesystem watcher"]
    Watcher --> Parser["Document parsers"]
    Parser --> PerDB["Per-database SQLite<br/>file_data"]
    PerDB --> Embedder["Embedding scheduler"]
    Embedder --> OpenAI["OpenAI embeddings"]
    Embedder --> Vector["In-memory vector cache"]
    API --> Keyword["Keyword search"]
    API --> Semantic["Semantic search"]
    Keyword --> PerDB
    Semantic --> QueryCache["Memory + global SQLite<br/>query_cache"]
    Semantic --> Vector
    Vector --> PerDB
    Registry --> Daemon

Runtime composition

Layer	Primary code	Responsibility
Entry and lifecycle	`cmd/app`	Dispatch CLI commands, daemonize, initialize subsystems, handle shutdown
HTTP API	`internal/api`	Expose local read-only routes and validate database selection
Persistence	`internal/database`	Open SQLite stores, maintain registry, and execute all content writes
File ingestion	`internal/filesystem`	Poll inboxes, detect changes, parse supported files, dismiss removals
Embeddings	`internal/openai`	Read credentials, call OpenAI, encode vectors, cache query embeddings
Vector retrieval	`internal/vector`	Maintain rebuildable buckets and perform two-stage cosine search
Tokenization	`internal/utils/segmenter`	Tokenize and deduplicate keyword queries with gse
Process state	`internal/runtime`	Write PID metadata, detect liveness, and stop the daemon

Startup sequence

runServer launches the current executable with --daemon, redirects output to daemon.log, and waits for the endpoint file. runServerDaemon then performs the authoritative initialization sequence:

Create or refresh runtime.uid.
Initialize the KuraDB keychain context.
Load the database registry.
Construct the OpenAI embedder.
Open global.db and preload query embeddings of the expected dimension.
Initialize the tokenizer and process-wide vector cache.
Open every registered per-database SQLite file.
Rebuild each vector bucket from embedded, non-dismissed rows.
Start one watcher and one embedding scheduler per loaded database.
Start the local HTTP server and wait for SIGINT or SIGTERM.

Storage topology

KuraDB uses several local state files under ~/.config/kuradb/:

Store	Authority	Contents
`db.json`	Durable registry	Database names and creation timestamps
`config.json`	Durable configuration	Optional pinned HTTP `port`
`global.db`	Durable cache store	Query text to embedding blobs
`{db}/data.db`	Source of truth	Parsed chunks, soft-delete state, and embeddings
`{db}/record.json`	Watcher snapshot	File size, mtime, type, and child snapshots
`runtime.uid`	Ephemeral process state	UID, PID, and start time
`endpoint`	Ephemeral discovery	Current local HTTP base URL
`daemon.log`	Operational log	Daemon stdout and stderr

vector.Cache and openai.Cache are accelerators, not authoritative stores. They must remain reconstructible from SQLite.

Concurrency model

Each loaded database owns independent watcher and embedder goroutines. HTTP keyword and semantic branches run concurrently when /api/search has no target. SQLite provides a read pool of eight connections and a separate write connection through go-sqlkit. Vector cache maps use process, database bucket, and query-cache mutexes to isolate concurrent access.

Trust boundaries and invariants

The HTTP surface contains GET query routes only; no mutation endpoint may bypass ingestion.
Content writes flow through watcher → parser → databaseHandler.Upsert → SQLite.
Keyword and semantic reads exclude rows where dismiss = TRUE.
All loaded embedding blobs must match the process-wide openai.Dim() expectation.
API responses expose source, chunk, and content, but not internal IDs, scores, hit counts, or totals.
Images and other skipped binary formats do not enter the text embedding pipeline.
Newly registered databases become available only after restart.

Shutdown

A termination signal cancels the shared context. The HTTP server removes endpoint and performs a five-second graceful shutdown. Watchers and embedders leave their select loops, runtime.uid is cleared, and all per-database plus global SQLite connectors are closed.