Indexing Pipeline
Last updated
How the watcher detects added, changed, and removed files in a database inbox and turns them into
file_data rows.Polling and Change Detection
Each loaded database runs its own watcher goroutine (
runWatcher in cmd/app/main.go) with a 10-second ticker. On every tick filesystem.WalkFiles reads the inbox recursively and compares each entry with the previous snapshot:| Condition | Treated as |
|---|---|
| Name not in the snapshot, or file and directory type differ | Changed |
| Same name, same size, same modification time | Unchanged, skipped |
| Same name but size or modification time differs | Changed |
| Name in the snapshot but no longer on disk | Removed |
Subdirectories are always descended into, so files at any depth are indexed. Unchanged files are never re-read.
Snapshot File
After each walk the new tree (size, modification time, directory flag, children) is written to
<db>/record.json in a background goroutine, serialized by a mutex. On startup the watcher loads this snapshot, so a daemon restart only re-indexes files that changed while it was down. When no snapshot exists, every file is indexed on the first tick and the removal pass is skipped.Writing Changed Files
A changed file is routed by extension to a parser (see Supported File Formats), then written by
databaseHandler.Upsert in one transaction:- Every live row of that source path is flagged
dismiss = TRUE. - Each new chunk is inserted or updated on
(source, chunk)withdismiss = FALSE.
Chunks that no longer exist after an edit therefore stay dismissed, and a reader never sees a mix of old and new chunks from the same file. The
source column holds the absolute path of the file inside the inbox.| Outcome | Log line in daemon.log |
|---|---|
| Stored | saved kind=<type> path=<path> chunks=<n> |
| Parser error | parser error=...; existing rows are kept |
| Write error | store.Save error=... |
| Skipped type | No write and no log beyond changed |
A file that fails to parse is recorded in the snapshot with its current size and time, so it is retried only after it changes again.
Removed Files
For every name that disappeared since the last walk,
databaseHandler.Dismiss flags all live rows of that path as dismiss = TRUE. A removed directory dismisses every file recorded under it. Rows are never physically deleted, and every search query excludes dismissed rows.