Storage Writes

See how parsed chunks are written to SQLite and how deleted files are soft-deleted.

Transactional upsert

databaseHandler.Upsert is the sole content-write entry point used by the filesystem layer. In one SQLite transaction it:

1. Marks all currently active rows for the source as dismiss = TRUE.
2. Inserts each current chunk or updates the (source, chunk) conflict, including the chunk total.
3. Restores current chunks to dismiss = FALSE.
4. Preserves the embedding and is_embed only when chunk content is unchanged.
5. Clears changed embeddings and resets is_embed = FALSE.

This sequence handles shortened files without returning stale trailing chunks and avoids re-embedding unchanged chunks. A parse that yields zero chunks, such as a sheet with only a header row, leaves every row for that source dismissed.

The file_data table enforces UNIQUE (source, chunk) and keeps a partial index on rows where is_embed = FALSE AND dismiss = FALSE, which is the embedding scheduler's selection predicate.

Removed files

After scanning current entries, the walker compares the previous snapshot. Missing file nodes are recursively passed to databaseHandler.Dismiss, which soft-deletes active rows by setting dismiss = TRUE. Data remains in SQLite but is excluded by indexing and query reads.

A removed directory is walked through its recorded children, so every file it contained is dismissed individually.