# Indexing Pipeline

How the watcher detects added, changed, and removed files in a database inbox and turns them into `file_data` rows.

## Polling and Change Detection

Each loaded database runs its own watcher goroutine (`runWatcher` in `cmd/app/main.go`) with a 10-second ticker. On every tick `filesystem.WalkFiles` reads the inbox recursively and compares each entry with the previous snapshot:

| Condition | Treated as |
|---|---|
| Name not in the snapshot, or file and directory type differ | Changed |
| Same name, same size, same modification time | Unchanged, skipped |
| Same name but size or modification time differs | Changed |
| Name in the snapshot but no longer on disk | Removed |

Subdirectories are always descended into, so files at any depth are indexed. Unchanged files are never re-read.

## Snapshot File

After each walk the new tree (size, modification time, directory flag, children) is written to `<db>/record.json` in a background goroutine, serialized by a mutex. On startup the watcher loads this snapshot, so a daemon restart only re-indexes files that changed while it was down. When no snapshot exists, every file is indexed on the first tick and the removal pass is skipped.

## Writing Changed Files

A changed file is routed by extension to a parser (see [Supported File Formats](/file-formats)), then written by `databaseHandler.Upsert` in one transaction:

1. Every live row of that source path is flagged `dismiss = TRUE`.
2. Each new chunk is inserted or updated on `(source, chunk)` with `dismiss = FALSE`.

Chunks that no longer exist after an edit therefore stay dismissed, and a reader never sees a mix of old and new chunks from the same file. The `source` column holds the absolute path of the file inside the inbox.

| Outcome | Log line in `daemon.log` |
|---|---|
| Stored | `saved kind=<type> path=<path> chunks=<n>` |
| Parser error | `parser error=...`; existing rows are kept |
| Write error | `store.Save error=...` |
| Skipped type | No write and no log beyond `changed` |

A file that fails to parse is recorded in the snapshot with its current size and time, so it is retried only after it changes again.

## Removed Files

For every name that disappeared since the last walk, `databaseHandler.Dismiss` flags all live rows of that path as `dismiss = TRUE`. A removed directory dismisses every file recorded under it. Rows are never physically deleted, and every search query excludes dismissed rows.

## Related Pages

- [Supported File Formats](/file-formats)
- [Data Layout](/data-layout)