perf(storage): bound startup maintenance to changed content

AGENTS.md

@@ -78,7 +78,7 @@ internal/
 ├── hybrid/            — Reciprocal Rank Fusion helpers for merged lexical retrieval
 ├── sequences/         — F4 PrefixSpan mining (deterministic discovery of frequent tool-call sequences per session)
 ├── services/          — Concrete typed read-only query orchestration over storage
-└── storage/           — SQLite adapter (dual FTS5 indexes: tool_fts + messages_fts, BM25, WAL mode, migrations v1–v20, search_items, session_tags, tool_events, message_templates, template_matches, correction_signals, annotations, AggregateCommands, AggregateFailures, AggregateTemplates, AggregateCorrections, UpsertAnnotation, LoadToolSequences)
+└── storage/           — SQLite adapter (dual FTS5 indexes: tool_fts + messages_fts, BM25, WAL mode, migrations v1–v21, search_items, session_tags, tool_events, message_templates, template_matches, correction_signals, annotations, AggregateCommands, AggregateFailures, AggregateTemplates, AggregateCorrections, UpsertAnnotation, LoadToolSequences)
 scripts/
 └── recall-eval/       — isolated legacy/synthetic recall evaluator and cohort reporter
 ```
@@ -142,8 +142,9 @@ External knowledge sources are configured with active `*.inputs.toml` manifests
 - **Startup coordination**: `github.com/gofrs/flock` via `internal/startuplock` — persistent `<db>.startup-sync.lock` sidecar with `0600` permissions, OS-owned advisory lock, local-host-only WAL snapshot coordination, and read-only followers that never delete the sidecar.
 - **Connection pragmas via `_pragma`**: `modernc.org/sqlite` honors DSN pragmas only in the `_pragma=name(value)` form; the mattn-style `_name=value` (e.g. `_busy_timeout=5000`, `_journal_mode=WAL`) is silently ignored, which had left the DB in rollback (delete) journal mode with a zero busy timeout — the root cause of `database is locked` (SQLITE_BUSY) errors. Both connections in `internal/storage/storage.go` set `_pragma=journal_mode(WAL)`, `_pragma=synchronous(NORMAL)`, and `_pragma=busy_timeout(5000)` (read-only sets only the busy timeout; journal mode is persisted in the file). Always use `_pragma=name(value)` for any new connection pragma here.
 - **Autoupdate**: Mandatory for valid executable release commands, with no runtime opt-out. `newUpdater()` in `cmd/backscroll/main.go` is the single wiring point and calls private `internal/autoupdate.New` with no runtime opt-out API. Apply/fetch starts only after Cobra parses flags, selects an executable command, and passes Args, required-flag and flag-group validation, before ordinary startup/index side effects. Invalid invocations, help, version and the `--skill` fast path do no update work or waiting. Dev identity (`version="dev"`, plain `go build`) remains exempt in the private backend. Apply and completed fetch failures are nonfatal, reported once on stderr as `warning: autoupdate: ...`; stdout/JSON/robot and command errors are preserved. After execution, staging waits at most 10s; the cap returns command success with a warning that staging did not complete within 10s. Cancellation returns promptly without inventing a network failure. HTTP reads, extraction, and unique synced publication carry context and byte limits; requests retain a 60s timeout. Cancellation/the staging cap cancels and joins the concrete backend, including bounded cleanup, so no staging mutation occurs after return. The private backend implements the Apache-2.0 picokit v1.0.0 release-format contract because released upstream updaters swallow errors. Publication uses per-attempt O_EXCL temporary files, never a shared `.tmp`. Apply uses a separate temporary copy and retains valid staged candidates on every failure; see `internal/autoupdate/README.md`. `scripts/eval.sh` builds its own dev binary to avoid live updates.
-- **Startup phase diagnostics (issues #47/#155, C16)**: `BACKSCROLL_STARTUP_DIAGNOSTICS=1` reports applicable startup phases and item counts to stderr, including with `--json`/`--robot`; stdout is unchanged. One invocation-local monotonic origin is captured at startup-policy entry, before configuration/preflight, lock acquisition, index preparation and sync. `Total` is elapsed from that origin to diagnostics emission after successful startup sync and the read-owner lease handoff (mutation owners retain the lease). Sequential measured phases are Lock Acquisition (initial attempt plus any wait), Index Prepare, Discovery, Metadata (files checked), Hashing (files hashed/skipped and MB), Parsing and Database (writes and derived maintenance). Metadata/hash/parse/discovery intervals accumulate across files/inputs but do not overlap; Database begins after parsing. `Unattributed` is always printed, including zero, and covers only uninstrumented work inside that common interval (config/preflight, additional opens/queries, bookkeeping, close/flush and handoff). Measured phase sum + nonnegative Unattributed = Total; any inconsistent/overlapping supplied timing is explicitly printed as `Overlap`, deducted from the sum, never silently hidden. Snapshot followers report lock attempt plus read-only snapshot preparation, metadata followers only the lock attempt, and neither reports sync phases. Remediation omits ordinary Index Prepare and emits once after recovery, including post-install sync only for apply; recovery work is Unattributed. Finalization is installed at startup-policy entry before preflight can return: every attempted executable startup emits exactly one report, including malformed config/manifests; help/version/invalid invocations that never start policy emit none. Completed sync phase measurements are retained immediately on later failure. A failing/canceled lock or preparation phase is timed to its return boundary; an active failing/canceled sync phase is timed through sync return (including connection cleanup), reported under its normal phase name, and identified by `Phase Outcome: <phase> incomplete`; not-yet-entered phases are omitted. No failed work is silently rolled into Unattributed. Metadata counts checks entered; hashed counts indicate successful hashes. Standalone legacy sync starts its own interval and has no outer lock/preparation measurements unless explicitly supplied. Diagnostics are off by default: no diagnostic clock reads or report allocation when disabled, and the environment is checked once when creating the invocation's sync service. Deterministic clock accounting tests live in `cmd/backscroll/startup_accounting_test.go`; `BenchmarkStartupAccountingDisabled` guards the disabled coordination path and `BenchmarkStartupDiagnostics` exercises enabled CLI startup.
+- **Startup phase diagnostics (issues #47/#155, C16)**: `BACKSCROLL_STARTUP_DIAGNOSTICS=1` reports applicable startup phases and item counts to stderr, including with `--json`/`--robot`; stdout is unchanged. One invocation-local monotonic origin is captured at startup-policy entry, before configuration/preflight, lock acquisition, index preparation and sync. `Total` is elapsed from that origin to diagnostics emission after successful startup sync and the read-owner lease handoff (mutation owners retain the lease). Sequential measured phases are Lock Acquisition (initial attempt plus any wait), Index Prepare, Discovery, Metadata (files checked), Hashing (files hashed/skipped and MB), Parsing (reader extraction, project attribution, session tagging and indexed-record construction) and Database (metadata/replay-queue reads before discovery, then writes and derived maintenance after parsing through sync connection close/checkpoint). Phase intervals accumulate without overlap. `Unattributed` is always printed, including zero, and covers only uninstrumented work inside that common interval (config/preflight, additional opens, bookkeeping and handoff). Measured phase sum + nonnegative Unattributed = Total; any inconsistent/overlapping supplied timing is explicitly printed as `Overlap`, deducted from the sum, never silently hidden. Snapshot followers report lock attempt plus read-only snapshot preparation, metadata followers only the lock attempt, and neither reports sync phases. Remediation omits ordinary Index Prepare and emits once after recovery, including post-install sync only for apply; recovery work is Unattributed. Finalization is installed at startup-policy entry before preflight can return: every attempted executable startup emits exactly one report, including malformed config/manifests; help/version/invalid invocations that never start policy emit none. Completed sync phase measurements are retained immediately on later failure. A failing/canceled lock or preparation phase is timed to its return boundary; an active failing/canceled sync phase is timed through sync return (including connection cleanup), reported under its normal phase name, and identified by `Phase Outcome: <phase> incomplete`; not-yet-entered phases are omitted. Metadata counts checks entered; hashed counts indicate successful hashes. Standalone legacy sync starts its own interval and has no outer lock/preparation measurements unless explicitly supplied. Diagnostics are off by default: no diagnostic clock reads or report allocation when disabled, and the environment is checked once when creating the invocation's sync service. Deterministic clock accounting tests live in `cmd/backscroll/startup_accounting_test.go`; `BenchmarkStartupAccountingDisabled` guards the disabled coordination path and `BenchmarkStartupDiagnostics` exercises enabled CLI startup.
 - **Schema migration rule**: Every new table or column MUST be introduced as a new migration version (increment the version number and add a new version-check block in `SetupSchema()`). Never modify existing migration blocks — existing databases that already passed that version will never re-run them. Migration v5 drops the phantom `session_events` table (and its indexes `idx_session_events_order` and `idx_session_events_project`) — the table was write-only dead weight after structured-stats filtering was removed. Migration v6 drops the phantom `search_items.source_metadata` column via `ALTER TABLE ... DROP COLUMN` — it had a setter but zero production callers and was never read. Migration v13 adds indexes on `template_matches(source_path)` and `correction_signals(source_path)` to optimize backfill discovery queries (reduces O(N·M) subquery scans to O(N·log M) index lookups). Migration v15 adds `search_echo`; NULL on existing session rows queues source-backed reparsing without changing the general extraction epoch. Provenance-only updates preserve perennial IDs and text. See `docs/eval/paired-search-echo.md` for the migration and recovery evidence.
+- **V21 incremental FTS maintenance**: Only text, content type, or row-ID changes maintain the external-content FTS indexes; metadata enrichment leaves them untouched. Independent old/new routing supports moves between prose and tool indexes. Partial indexes bound pending echo discovery to NULL provenance and the broad echo-zero candidate population; the shared Go predicate remains authoritative. Per-path legacy cleanup selects the path index, UUID enrichment selects unique identity, and metadata-only perennial replay skips stopword regeneration when the prose corpus is unchanged. See `docs/search.md`.
 - **V17 project attribution proof**: `projects.toml` roots and explicit absolute `aliases` share cleaned, case-sensitive root/descendant matching. Implicit suffix/case-fold equivalence is not proof; ambiguous registry matches fall back without registry attribution. `search_items.project_evidence` distinguishes historical unknown, path fallback, registry, and native structured cwd. Rebuild/re-resolution repair only unknown/path-fallback rows; native cwd attribution retains its row identity and payload after source expiry. Available historical sources replay under the shared 200-path cap; expired sources remain unknown evidence. Recovery preserves project proof. See `docs/configuration.md#project-identity-registry`.
 - **V16 parser-backed message origin**: `search_items.origin` is constrained to `human|assistant|system|automation|unknown`, with an independent `origin_version`. Readers prove origin only from native structured envelopes and tool boundaries; stored text, historical `role`, Markdown headings, and frontmatter are never inference sources. Migration defaults historical rows to `unknown`; `PendingOriginPaths` reparses only available configured sources under the shared 200-path startup cap, so expired sources remain unknown. Stable identities may enrich `unknown` to a proven origin without replacing payload or degrading prior proof, and contradictory proof is rejected. Recovery preserves matching proof, enriches compatible unknown/proven duplicates, and rejects contradictory proven origins. `patterns --kind corrections --origin ...` is opt-in, filters before pagination, and emits origin only when requested; exact reader mappings and conditional output fields are owned by `docs/input-contract.md` and `docs/patterns.md`.
 - **F0a rich capture (migration v8)**: readers extract per-message identity and tool metadata BEFORE serialization/cleaning destroys the evidence — `uuid` (record uuid; tool blocks get stable `#tN`/`#rN` suffixes by block index), `tool_name`, `command_head`, `is_error` (`*bool`, three-valued: tool_result blocks carry it and it is paired back onto the tool_use message cross-record via `tool_use_id`), and `was_interrupted` (detected on raw content before `CleanContent` strips "Request interrupted"). Persisted to `search_items` (`extraction_version`, `was_interrupted` columns) and the perennial `tool_events` satellite table (`UNIQUE(source_path, ordinal)`, no CASCADE lifecycle or deletion path). Claude and OMP readers; OpenCode emits native UUIDs, tool names, and command heads; UUID-less historical records retain legacy behavior. Design: `docs/superpowers/specs/2026-07-17-pattern-discovery-northstar-design.md`.
@@ -260,7 +261,7 @@ github.com/pablontiv/backscroll/internal/categories    — F4 category map loade
 github.com/pablontiv/backscroll/internal/hybrid        — Reciprocal Rank Fusion helpers for merged lexical retrieval
 github.com/pablontiv/backscroll/internal/sequences     — F4 PrefixSpan mining (deterministic pattern discovery per session)
 github.com/pablontiv/backscroll/internal/services      — Concrete typed read-only query orchestration over storage
-github.com/pablontiv/backscroll/internal/storage       — Database schema, migrations v1–v20, FTS5 indexes
+github.com/pablontiv/backscroll/internal/storage       — Database schema, migrations v1–v21, FTS5 indexes
 github.com/pablontiv/backscroll/internal/projects      — Project identity registry
 github.com/pablontiv/backscroll/internal/readers       — SessionReader interface, Registry, ClaudeReader (text+tool_use+tool_result), PiReader (text+toolCall+custom results), OMPReader (native v3 text+thinking+toolCall+toolResult; parser-backed identities and origin), OpenCodeReader (text+tool state.input+state.output), CodexReader (rollout response items; docs/input-contract.md), MarkdownDocumentReader (`markdown_document`), MarkdownSectionsReader (`markdown_sections`); toolfmt serializer
 github.com/pablontiv/backscroll/internal/recovery      — Stranded database recovery orchestration, durable backup, atomic replacement, and post-install sync

cmd/backscroll/startup_accounting_failure_test.go

@@ -131,7 +131,8 @@ func TestStartupAccountingFailedSyncKeepsMeasuredPhases(t *testing.T) {
 				}
 				var out bytes.Buffer
 				d.render(&out)
-				want := map[string]time.Duration{"Discovery": 100 * time.Millisecond}
+				// Replay-queue database reads precede discovery on every attempt.
+				want := map[string]time.Duration{"Database": 0, "Discovery": 100 * time.Millisecond}
 				if phase != "Discovery" {
 					want["Metadata"] = 0
 					want["Hashing"] = 200 * time.Millisecond

cmd/backscroll/startup_accounting_test.go

@@ -328,6 +328,12 @@ func TestStartupAccountingSequentialPhasesAndHandoff(t *testing.T) {
 		advance(500 * time.Millisecond)
 		return open(ctx, path)
 	}
+	metadata := s.getFileMetadata
+	s.getFileMetadata = func(path string) (*int64, *string, error) {
+		// Metadata snapshots add time during hashing and parsed-record construction.
+		advance(50 * time.Millisecond)
+		return metadata(path)
+	}
 	sync := s.syncFiles
 	s.syncFiles = func(ctx context.Context, db *storage.Database, files []storage.IndexedFile) error {
 		advance(400 * time.Millisecond)
@@ -347,7 +353,7 @@ func TestStartupAccountingSequentialPhasesAndHandoff(t *testing.T) {
 	if result.Failure != nil {
 		t.Fatal(result.Failure)
 	}
-	for phase, want := range map[string]time.Duration{"Lock Acquisition": time.Second, "Index Prepare": time.Second, "Discovery": 100 * time.Millisecond, "Metadata": 0, "Hashing": 200 * time.Millisecond, "Parsing": 300 * time.Millisecond, "Database": 400 * time.Millisecond, "Unattributed": 1800 * time.Millisecond, "Total": 4800 * time.Millisecond} {
+	for phase, want := range map[string]time.Duration{"Lock Acquisition": time.Second, "Index Prepare": time.Second, "Discovery": 100 * time.Millisecond, "Metadata": 0, "Hashing": 250 * time.Millisecond, "Parsing": 350 * time.Millisecond, "Database": 400 * time.Millisecond, "Unattributed": 1800 * time.Millisecond, "Total": 4900 * time.Millisecond} {
 		if got := diagnosticDuration(t, out.String(), phase); got != want {
 			t.Fatalf("%s=%v want %v: %s", phase, got, want, out.String())
 		}

cmd/backscroll/sync_helpers.go

@@ -464,6 +464,12 @@ func (s *startupSyncService) syncWithProgress(ctx context.Context, cfg *config.C
 		return err
 	}
 
+	// Metadata reads and durable replay discovery are database work too. Keep
+	// them in the existing phase so the elapsed interval remains non-overlapping.
+	if diag {
+		report.beginPhase("Database", &report.database, &report.hasDatabase)
+	}
+
 	// Get existing file metadata for prefiltering
 	existingMetadata, err := db.GetFileMetadataContext(ctx)
 	if err != nil {
@@ -489,6 +495,9 @@ func (s *startupSyncService) syncWithProgress(ctx context.Context, cfg *config.C
 		return fmt.Errorf("discover project proof paths: %w", err)
 	}
 	originPaths = append(originPaths, projectPaths...)
+	if diag {
+		report.finishPhase()
+	}
 	const replayParsesCap = 200
 
 	// Build reader registry
@@ -665,10 +674,6 @@ func (s *startupSyncService) syncWithProgress(ctx context.Context, cfg *config.C
 		if err := ctx.Err(); err != nil {
 			return err
 		}
-		if diag {
-			report.finishPhase()
-		}
-
 		// Use session cwd for project identification; fall back to file path if cwd is empty
 		identPath := pf.Cwd
 		if identPath == "" {
@@ -750,6 +755,9 @@ func (s *startupSyncService) syncWithProgress(ctx context.Context, cfg *config.C
 			FileSize:             fileSize,
 			FileMtime:            fileMtime,
 		})
+		if diag {
+			report.finishPhase()
+		}
 	}
 
 	// Phase 5: Database
@@ -813,9 +821,5 @@ func (s *startupSyncService) syncWithProgress(ctx context.Context, cfg *config.C
 		return err
 	}
 
-	if diag {
-		report.finishPhase()
-	}
-
 	return nil
 }

docs/search.md

@@ -231,6 +231,13 @@ recognized, and only in that reader. Text that mentions Backscroll and
 unrelated tool commands are unchanged. There is no opt-in flag or shell
 parsing, query relaxation, or whole-session exclusion.
 
+Migration v21 maintains the split external-content indexes only when indexed
+text, row ID, or content type changes. Metadata-only enrichment preserves the
+FTS rows and search results; content-type changes move rows between the tool
+and prose indexes atomically. Pending echo discovery scans indexed NULL and
+echo-zero candidates once, retaining the same strict Go predicate and replay
+order. Perennial replays with unchanged prose skip dynamic-stopword refresh.
+
 Migration v15 adds nullable `search_echo` provenance. Already-indexed Claude
 session rows with surviving sources reparse through the existing bounded
 incremental backfill, even when hashes and file metadata match. This updates

internal/compat/catalog_test.go

@@ -331,7 +331,7 @@ func withReleaseSchemaFS(t *testing.T, fsys fs.FS) {
 func loadPublishedCurrentMigrationRows(t *testing.T) map[int]migrationRow {
 	t.Helper()
 
-	fixtureSQL, err := fs.ReadFile(releaseSchemaFS, "testdata/release-schemas/v20.sql")
+	fixtureSQL, err := fs.ReadFile(releaseSchemaFS, "testdata/release-schemas/v21.sql")
 	if err != nil {
 		t.Fatal(err)
 	}