Core Concepts

Learn the state, consistency, security, and retrieval concepts that define KuraDB behavior.

Read-only service boundary

KuraDB is read-only from the consumer's perspective. The HTTP API accepts health, list, and search requests, but never content mutations. Filesystem ingestion is the only content-write path:

watched inbox → parser → databaseHandler.Upsert → SQLite

This boundary prevents API clients from modifying the retrieval corpus outside the watcher-controlled pipeline.

Registry versus loaded databases

The registry in db.json records desired databases. The daemon opens those entries only during startup. Therefore:

kura add changes durable registry state and creates directories.
/api/list distinguishes registered entries from currently loaded names.
A newly added database is unavailable to search until restart.
Rename and removal should be performed while understanding that an already running process retains its startup-time map.

SQLite as source of truth

Each database stores parsed chunks and embeddings in {db}/data.db. SQLite controls whether a row is active, pending embedding, or dismissed. In-memory caches improve query speed but are disposable:

State	Durable source	Rebuild behavior
Chunk content	`file_data.content`	Never sourced from vector cache
Embedding	`file_data.embedding`	Loaded into vector buckets at startup
Query embedding	`query_cache.embedding`	Preloaded into `openai.Cache`
Source vectors	Derived in memory	Rebuilt by averaging and normalizing chunk vectors
Watch snapshot	`record.json`	Loaded before the next directory comparison

Chunk lifecycle

A parsed file produces numbered chunks. Upsert first marks existing active rows for the source as dismissed, then inserts or updates current chunks in one transaction. If a chunk's content is unchanged, its embedding is retained; changed content clears the embedding and resets is_embed to FALSE.

Deleted files are not physically removed. The watcher calls Dismiss, which marks matching rows with dismiss = TRUE. Every active query path filters those rows.

Embedding consistency

The OpenAI client requests text-embedding-3-small vectors with 512 dimensions. Encoded vectors use little-endian float32, so one valid blob is 2,048 bytes. KuraDB enforces consistency at several points:

OpenAI responses must contain the requested number of vectors.
Every returned vector must have exactly 512 elements.
Query-cache preload skips blobs whose byte length differs from openai.Dim() * 4.
Vector search ignores vectors whose dimensions differ from the query.
Embedding updates include the original content in the SQL predicate, preventing stale results from being applied after a file changes.

Dual retrieval

A search can execute keyword retrieval, semantic retrieval, or both.

Strategy	Query preparation	Retrieval	Ranking
Keyword	gse tokenization, trimming, lowercase, deduplication	SQLite `LIKE` across active content	Number of matched tokens, then row ID
Semantic	Query embedding from cache or OpenAI	Two-stage in-memory cosine search, then SQLite hydration	Cosine score, with a `0.3` minimum

When both branches run, they execute concurrently and remain separate in the response. KuraDB does not merge their rankings into a single score.

Source-level vector filtering

Semantic search derives one normalized vector per source from its chunk vectors. It first ranks all source vectors, keeps approximately 5% with a minimum of 20 candidates, then calculates chunk-level cosine similarity only within those sources. If source vectors are unavailable, it falls back to searching all chunk vectors.

Query cache

openai.Cache maps the exact query string to an embedding. On a miss, semantic search calls OpenAI and stores the vector in memory. An OnSet callback asynchronously persists the encoded vector to global.db with a five-second timeout. Startup preloads valid entries without re-triggering persistence.

API response contract

Search results are grouped by source and expose only:

{
  "source": "/path/to/file.md",
  "matches": [
    {"chunk": 1, "content": "..."}
  ]
}

Internal fields such as row ID, semantic score, keyword hit count, chunk total, and cache state remain implementation details.

Platform assumptions

The watcher compares direct directory entries using size and modification time. KuraDB supports local macOS and Linux filesystems with reliable POSIX semantics. It does not claim correctness on Windows, SMB, NFS, or FUSE mounts where those assumptions can fail.