diff --git a/AGENTS.md b/AGENTS.md index 9d711ff..2eeaa55 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -78,7 +78,7 @@ internal/ ├── hybrid/ — Reciprocal Rank Fusion helpers for merged lexical retrieval ├── sequences/ — F4 PrefixSpan mining (deterministic discovery of frequent tool-call sequences per session) ├── services/ — Concrete typed read-only query orchestration over storage -└── storage/ — SQLite adapter (dual FTS5 indexes: tool_fts + messages_fts, BM25, WAL mode, migrations v1–v20, search_items, session_tags, tool_events, message_templates, template_matches, correction_signals, annotations, AggregateCommands, AggregateFailures, AggregateTemplates, AggregateCorrections, UpsertAnnotation, LoadToolSequences) +└── storage/ — SQLite adapter (dual FTS5 indexes: tool_fts + messages_fts, BM25, WAL mode, migrations v1–v21, search_items, session_tags, tool_events, message_templates, template_matches, correction_signals, annotations, AggregateCommands, AggregateFailures, AggregateTemplates, AggregateCorrections, UpsertAnnotation, LoadToolSequences) scripts/ └── recall-eval/ — isolated legacy/synthetic recall evaluator and cohort reporter ``` @@ -142,8 +142,9 @@ External knowledge sources are configured with active `*.inputs.toml` manifests - **Startup coordination**: `github.com/gofrs/flock` via `internal/startuplock` — persistent `.startup-sync.lock` sidecar with `0600` permissions, OS-owned advisory lock, local-host-only WAL snapshot coordination, and read-only followers that never delete the sidecar. - **Connection pragmas via `_pragma`**: `modernc.org/sqlite` honors DSN pragmas only in the `_pragma=name(value)` form; the mattn-style `_name=value` (e.g. `_busy_timeout=5000`, `_journal_mode=WAL`) is silently ignored, which had left the DB in rollback (delete) journal mode with a zero busy timeout — the root cause of `database is locked` (SQLITE_BUSY) errors. Both connections in `internal/storage/storage.go` set `_pragma=journal_mode(WAL)`, `_pragma=synchronous(NORMAL)`, and `_pragma=busy_timeout(5000)` (read-only sets only the busy timeout; journal mode is persisted in the file). Always use `_pragma=name(value)` for any new connection pragma here. - **Autoupdate**: Mandatory for valid executable release commands, with no runtime opt-out. `newUpdater()` in `cmd/backscroll/main.go` is the single wiring point and calls private `internal/autoupdate.New` with no runtime opt-out API. Apply/fetch starts only after Cobra parses flags, selects an executable command, and passes Args, required-flag and flag-group validation, before ordinary startup/index side effects. Invalid invocations, help, version and the `--skill` fast path do no update work or waiting. Dev identity (`version="dev"`, plain `go build`) remains exempt in the private backend. Apply and completed fetch failures are nonfatal, reported once on stderr as `warning: autoupdate: ...`; stdout/JSON/robot and command errors are preserved. After execution, staging waits at most 10s; the cap returns command success with a warning that staging did not complete within 10s. Cancellation returns promptly without inventing a network failure. HTTP reads, extraction, and unique synced publication carry context and byte limits; requests retain a 60s timeout. Cancellation/the staging cap cancels and joins the concrete backend, including bounded cleanup, so no staging mutation occurs after return. The private backend implements the Apache-2.0 picokit v1.0.0 release-format contract because released upstream updaters swallow errors. Publication uses per-attempt O_EXCL temporary files, never a shared `.tmp`. Apply uses a separate temporary copy and retains valid staged candidates on every failure; see `internal/autoupdate/README.md`. `scripts/eval.sh` builds its own dev binary to avoid live updates. -- **Startup phase diagnostics (issues #47/#155, C16)**: `BACKSCROLL_STARTUP_DIAGNOSTICS=1` reports applicable startup phases and item counts to stderr, including with `--json`/`--robot`; stdout is unchanged. One invocation-local monotonic origin is captured at startup-policy entry, before configuration/preflight, lock acquisition, index preparation and sync. `Total` is elapsed from that origin to diagnostics emission after successful startup sync and the read-owner lease handoff (mutation owners retain the lease). Sequential measured phases are Lock Acquisition (initial attempt plus any wait), Index Prepare, Discovery, Metadata (files checked), Hashing (files hashed/skipped and MB), Parsing and Database (writes and derived maintenance). Metadata/hash/parse/discovery intervals accumulate across files/inputs but do not overlap; Database begins after parsing. `Unattributed` is always printed, including zero, and covers only uninstrumented work inside that common interval (config/preflight, additional opens/queries, bookkeeping, close/flush and handoff). Measured phase sum + nonnegative Unattributed = Total; any inconsistent/overlapping supplied timing is explicitly printed as `Overlap`, deducted from the sum, never silently hidden. Snapshot followers report lock attempt plus read-only snapshot preparation, metadata followers only the lock attempt, and neither reports sync phases. Remediation omits ordinary Index Prepare and emits once after recovery, including post-install sync only for apply; recovery work is Unattributed. Finalization is installed at startup-policy entry before preflight can return: every attempted executable startup emits exactly one report, including malformed config/manifests; help/version/invalid invocations that never start policy emit none. Completed sync phase measurements are retained immediately on later failure. A failing/canceled lock or preparation phase is timed to its return boundary; an active failing/canceled sync phase is timed through sync return (including connection cleanup), reported under its normal phase name, and identified by `Phase Outcome: incomplete`; not-yet-entered phases are omitted. No failed work is silently rolled into Unattributed. Metadata counts checks entered; hashed counts indicate successful hashes. Standalone legacy sync starts its own interval and has no outer lock/preparation measurements unless explicitly supplied. Diagnostics are off by default: no diagnostic clock reads or report allocation when disabled, and the environment is checked once when creating the invocation's sync service. Deterministic clock accounting tests live in `cmd/backscroll/startup_accounting_test.go`; `BenchmarkStartupAccountingDisabled` guards the disabled coordination path and `BenchmarkStartupDiagnostics` exercises enabled CLI startup. +- **Startup phase diagnostics (issues #47/#155, C16)**: `BACKSCROLL_STARTUP_DIAGNOSTICS=1` reports applicable startup phases and item counts to stderr, including with `--json`/`--robot`; stdout is unchanged. One invocation-local monotonic origin is captured at startup-policy entry, before configuration/preflight, lock acquisition, index preparation and sync. `Total` is elapsed from that origin to diagnostics emission after successful startup sync and the read-owner lease handoff (mutation owners retain the lease). Sequential measured phases are Lock Acquisition (initial attempt plus any wait), Index Prepare, Discovery, Metadata (files checked), Hashing (files hashed/skipped and MB), Parsing (reader extraction, project attribution, session tagging and indexed-record construction) and Database (metadata/replay-queue reads before discovery, then writes and derived maintenance after parsing through sync connection close/checkpoint). Phase intervals accumulate without overlap. `Unattributed` is always printed, including zero, and covers only uninstrumented work inside that common interval (config/preflight, additional opens, bookkeeping and handoff). Measured phase sum + nonnegative Unattributed = Total; any inconsistent/overlapping supplied timing is explicitly printed as `Overlap`, deducted from the sum, never silently hidden. Snapshot followers report lock attempt plus read-only snapshot preparation, metadata followers only the lock attempt, and neither reports sync phases. Remediation omits ordinary Index Prepare and emits once after recovery, including post-install sync only for apply; recovery work is Unattributed. Finalization is installed at startup-policy entry before preflight can return: every attempted executable startup emits exactly one report, including malformed config/manifests; help/version/invalid invocations that never start policy emit none. Completed sync phase measurements are retained immediately on later failure. A failing/canceled lock or preparation phase is timed to its return boundary; an active failing/canceled sync phase is timed through sync return (including connection cleanup), reported under its normal phase name, and identified by `Phase Outcome: incomplete`; not-yet-entered phases are omitted. Metadata counts checks entered; hashed counts indicate successful hashes. Standalone legacy sync starts its own interval and has no outer lock/preparation measurements unless explicitly supplied. Diagnostics are off by default: no diagnostic clock reads or report allocation when disabled, and the environment is checked once when creating the invocation's sync service. Deterministic clock accounting tests live in `cmd/backscroll/startup_accounting_test.go`; `BenchmarkStartupAccountingDisabled` guards the disabled coordination path and `BenchmarkStartupDiagnostics` exercises enabled CLI startup. - **Schema migration rule**: Every new table or column MUST be introduced as a new migration version (increment the version number and add a new version-check block in `SetupSchema()`). Never modify existing migration blocks — existing databases that already passed that version will never re-run them. Migration v5 drops the phantom `session_events` table (and its indexes `idx_session_events_order` and `idx_session_events_project`) — the table was write-only dead weight after structured-stats filtering was removed. Migration v6 drops the phantom `search_items.source_metadata` column via `ALTER TABLE ... DROP COLUMN` — it had a setter but zero production callers and was never read. Migration v13 adds indexes on `template_matches(source_path)` and `correction_signals(source_path)` to optimize backfill discovery queries (reduces O(N·M) subquery scans to O(N·log M) index lookups). Migration v15 adds `search_echo`; NULL on existing session rows queues source-backed reparsing without changing the general extraction epoch. Provenance-only updates preserve perennial IDs and text. See `docs/eval/paired-search-echo.md` for the migration and recovery evidence. +- **V21 incremental FTS maintenance**: Only text, content type, or row-ID changes maintain the external-content FTS indexes; metadata enrichment leaves them untouched. Independent old/new routing supports moves between prose and tool indexes. Partial indexes bound pending echo discovery to NULL provenance and the broad echo-zero candidate population; the shared Go predicate remains authoritative. Per-path legacy cleanup selects the path index, UUID enrichment selects unique identity, and metadata-only perennial replay skips stopword regeneration when the prose corpus is unchanged. See `docs/search.md`. - **V17 project attribution proof**: `projects.toml` roots and explicit absolute `aliases` share cleaned, case-sensitive root/descendant matching. Implicit suffix/case-fold equivalence is not proof; ambiguous registry matches fall back without registry attribution. `search_items.project_evidence` distinguishes historical unknown, path fallback, registry, and native structured cwd. Rebuild/re-resolution repair only unknown/path-fallback rows; native cwd attribution retains its row identity and payload after source expiry. Available historical sources replay under the shared 200-path cap; expired sources remain unknown evidence. Recovery preserves project proof. See `docs/configuration.md#project-identity-registry`. - **V16 parser-backed message origin**: `search_items.origin` is constrained to `human|assistant|system|automation|unknown`, with an independent `origin_version`. Readers prove origin only from native structured envelopes and tool boundaries; stored text, historical `role`, Markdown headings, and frontmatter are never inference sources. Migration defaults historical rows to `unknown`; `PendingOriginPaths` reparses only available configured sources under the shared 200-path startup cap, so expired sources remain unknown. Stable identities may enrich `unknown` to a proven origin without replacing payload or degrading prior proof, and contradictory proof is rejected. Recovery preserves matching proof, enriches compatible unknown/proven duplicates, and rejects contradictory proven origins. `patterns --kind corrections --origin ...` is opt-in, filters before pagination, and emits origin only when requested; exact reader mappings and conditional output fields are owned by `docs/input-contract.md` and `docs/patterns.md`. - **F0a rich capture (migration v8)**: readers extract per-message identity and tool metadata BEFORE serialization/cleaning destroys the evidence — `uuid` (record uuid; tool blocks get stable `#tN`/`#rN` suffixes by block index), `tool_name`, `command_head`, `is_error` (`*bool`, three-valued: tool_result blocks carry it and it is paired back onto the tool_use message cross-record via `tool_use_id`), and `was_interrupted` (detected on raw content before `CleanContent` strips "Request interrupted"). Persisted to `search_items` (`extraction_version`, `was_interrupted` columns) and the perennial `tool_events` satellite table (`UNIQUE(source_path, ordinal)`, no CASCADE lifecycle or deletion path). Claude and OMP readers; OpenCode emits native UUIDs, tool names, and command heads; UUID-less historical records retain legacy behavior. Design: `docs/superpowers/specs/2026-07-17-pattern-discovery-northstar-design.md`. @@ -260,7 +261,7 @@ github.com/pablontiv/backscroll/internal/categories — F4 category map loade github.com/pablontiv/backscroll/internal/hybrid — Reciprocal Rank Fusion helpers for merged lexical retrieval github.com/pablontiv/backscroll/internal/sequences — F4 PrefixSpan mining (deterministic pattern discovery per session) github.com/pablontiv/backscroll/internal/services — Concrete typed read-only query orchestration over storage -github.com/pablontiv/backscroll/internal/storage — Database schema, migrations v1–v20, FTS5 indexes +github.com/pablontiv/backscroll/internal/storage — Database schema, migrations v1–v21, FTS5 indexes github.com/pablontiv/backscroll/internal/projects — Project identity registry github.com/pablontiv/backscroll/internal/readers — SessionReader interface, Registry, ClaudeReader (text+tool_use+tool_result), PiReader (text+toolCall+custom results), OMPReader (native v3 text+thinking+toolCall+toolResult; parser-backed identities and origin), OpenCodeReader (text+tool state.input+state.output), CodexReader (rollout response items; docs/input-contract.md), MarkdownDocumentReader (`markdown_document`), MarkdownSectionsReader (`markdown_sections`); toolfmt serializer github.com/pablontiv/backscroll/internal/recovery — Stranded database recovery orchestration, durable backup, atomic replacement, and post-install sync diff --git a/cmd/backscroll/startup_accounting_failure_test.go b/cmd/backscroll/startup_accounting_failure_test.go index 86c6f62..67c7107 100644 --- a/cmd/backscroll/startup_accounting_failure_test.go +++ b/cmd/backscroll/startup_accounting_failure_test.go @@ -131,7 +131,8 @@ func TestStartupAccountingFailedSyncKeepsMeasuredPhases(t *testing.T) { } var out bytes.Buffer d.render(&out) - want := map[string]time.Duration{"Discovery": 100 * time.Millisecond} + // Replay-queue database reads precede discovery on every attempt. + want := map[string]time.Duration{"Database": 0, "Discovery": 100 * time.Millisecond} if phase != "Discovery" { want["Metadata"] = 0 want["Hashing"] = 200 * time.Millisecond diff --git a/cmd/backscroll/startup_accounting_test.go b/cmd/backscroll/startup_accounting_test.go index f38f6b2..612a73b 100644 --- a/cmd/backscroll/startup_accounting_test.go +++ b/cmd/backscroll/startup_accounting_test.go @@ -328,6 +328,12 @@ func TestStartupAccountingSequentialPhasesAndHandoff(t *testing.T) { advance(500 * time.Millisecond) return open(ctx, path) } + metadata := s.getFileMetadata + s.getFileMetadata = func(path string) (*int64, *string, error) { + // Metadata snapshots add time during hashing and parsed-record construction. + advance(50 * time.Millisecond) + return metadata(path) + } sync := s.syncFiles s.syncFiles = func(ctx context.Context, db *storage.Database, files []storage.IndexedFile) error { advance(400 * time.Millisecond) @@ -347,7 +353,7 @@ func TestStartupAccountingSequentialPhasesAndHandoff(t *testing.T) { if result.Failure != nil { t.Fatal(result.Failure) } - for phase, want := range map[string]time.Duration{"Lock Acquisition": time.Second, "Index Prepare": time.Second, "Discovery": 100 * time.Millisecond, "Metadata": 0, "Hashing": 200 * time.Millisecond, "Parsing": 300 * time.Millisecond, "Database": 400 * time.Millisecond, "Unattributed": 1800 * time.Millisecond, "Total": 4800 * time.Millisecond} { + for phase, want := range map[string]time.Duration{"Lock Acquisition": time.Second, "Index Prepare": time.Second, "Discovery": 100 * time.Millisecond, "Metadata": 0, "Hashing": 250 * time.Millisecond, "Parsing": 350 * time.Millisecond, "Database": 400 * time.Millisecond, "Unattributed": 1800 * time.Millisecond, "Total": 4900 * time.Millisecond} { if got := diagnosticDuration(t, out.String(), phase); got != want { t.Fatalf("%s=%v want %v: %s", phase, got, want, out.String()) } diff --git a/cmd/backscroll/sync_helpers.go b/cmd/backscroll/sync_helpers.go index ce0dad8..d538bf8 100644 --- a/cmd/backscroll/sync_helpers.go +++ b/cmd/backscroll/sync_helpers.go @@ -464,6 +464,12 @@ func (s *startupSyncService) syncWithProgress(ctx context.Context, cfg *config.C return err } + // Metadata reads and durable replay discovery are database work too. Keep + // them in the existing phase so the elapsed interval remains non-overlapping. + if diag { + report.beginPhase("Database", &report.database, &report.hasDatabase) + } + // Get existing file metadata for prefiltering existingMetadata, err := db.GetFileMetadataContext(ctx) if err != nil { @@ -489,6 +495,9 @@ func (s *startupSyncService) syncWithProgress(ctx context.Context, cfg *config.C return fmt.Errorf("discover project proof paths: %w", err) } originPaths = append(originPaths, projectPaths...) + if diag { + report.finishPhase() + } const replayParsesCap = 200 // Build reader registry @@ -665,10 +674,6 @@ func (s *startupSyncService) syncWithProgress(ctx context.Context, cfg *config.C if err := ctx.Err(); err != nil { return err } - if diag { - report.finishPhase() - } - // Use session cwd for project identification; fall back to file path if cwd is empty identPath := pf.Cwd if identPath == "" { @@ -750,6 +755,9 @@ func (s *startupSyncService) syncWithProgress(ctx context.Context, cfg *config.C FileSize: fileSize, FileMtime: fileMtime, }) + if diag { + report.finishPhase() + } } // Phase 5: Database @@ -813,9 +821,5 @@ func (s *startupSyncService) syncWithProgress(ctx context.Context, cfg *config.C return err } - if diag { - report.finishPhase() - } - return nil } diff --git a/docs/search.md b/docs/search.md index 940f886..71f723a 100644 --- a/docs/search.md +++ b/docs/search.md @@ -231,6 +231,13 @@ recognized, and only in that reader. Text that mentions Backscroll and unrelated tool commands are unchanged. There is no opt-in flag or shell parsing, query relaxation, or whole-session exclusion. +Migration v21 maintains the split external-content indexes only when indexed +text, row ID, or content type changes. Metadata-only enrichment preserves the +FTS rows and search results; content-type changes move rows between the tool +and prose indexes atomically. Pending echo discovery scans indexed NULL and +echo-zero candidates once, retaining the same strict Go predicate and replay +order. Perennial replays with unchanged prose skip dynamic-stopword refresh. + Migration v15 adds nullable `search_echo` provenance. Already-indexed Claude session rows with surviving sources reparse through the existing bounded incremental backfill, even when hashes and file metadata match. This updates diff --git a/internal/compat/catalog_test.go b/internal/compat/catalog_test.go index ba53769..42320b3 100644 --- a/internal/compat/catalog_test.go +++ b/internal/compat/catalog_test.go @@ -331,7 +331,7 @@ func withReleaseSchemaFS(t *testing.T, fsys fs.FS) { func loadPublishedCurrentMigrationRows(t *testing.T) map[int]migrationRow { t.Helper() - fixtureSQL, err := fs.ReadFile(releaseSchemaFS, "testdata/release-schemas/v20.sql") + fixtureSQL, err := fs.ReadFile(releaseSchemaFS, "testdata/release-schemas/v21.sql") if err != nil { t.Fatal(err) } diff --git a/internal/compat/schema.go b/internal/compat/schema.go index 12ac708..424468c 100644 --- a/internal/compat/schema.go +++ b/internal/compat/schema.go @@ -409,6 +409,7 @@ var allMigrationSteps = []MigrationStep{ {Version: 18, Name: "V18 ingestion semantics fingerprint"}, {Version: 19, Name: "V19 native tool call identity"}, {Version: 20, Name: "V20 native session identity"}, + {Version: 21, Name: "V21 incremental FTS maintenance"}, } // MigrationSteps returns the authoritative ordered migration registrations. diff --git a/internal/compat/schema_test.go b/internal/compat/schema_test.go index 3da2dc6..c549303 100644 --- a/internal/compat/schema_test.go +++ b/internal/compat/schema_test.go @@ -71,15 +71,15 @@ func TestInspectIndexRecognizesPartialV6LineageAndPlansV7(t *testing.T) { } func TestInspectIndexCurrentShapeIsIdempotent(t *testing.T) { - db := openFixtureCopy(t, "v20.sql") + db := openFixtureCopy(t, "v21.sql") defer db.Close() plan, diag, err := InspectIndex(context.Background(), db) if err != nil || diag != nil { t.Fatalf("plan error=%v diagnostic=%+v", err, diag) } - if plan.From.AppliedVersion != 20 { - t.Fatalf("applied version = %d, want 20", plan.From.AppliedVersion) + if plan.From.AppliedVersion != 21 { + t.Fatalf("applied version = %d, want 21", plan.From.AppliedVersion) } if len(plan.Steps) != 0 { t.Fatalf("current shape has pending steps: %+v", plan.Steps) @@ -371,8 +371,8 @@ func TestInspectIndexRecognizesObservedDevelopmentV13Shape(t *testing.T) { if err != nil || diag != nil { t.Fatalf("inspect error=%v diagnostic=%+v", err, diag) } - if plan.From.AppliedVersion != 13 || len(plan.Steps) != 7 || plan.Steps[0].Version != 14 || plan.Steps[1].Version != 15 || plan.Steps[2].Version != 16 || plan.Steps[3].Version != 17 || plan.Steps[4].Version != 18 || plan.Steps[5].Version != 19 || plan.Steps[6].Version != 20 { - t.Fatalf("development V13 plan = %+v, want v13 with v14/v15/v16/v17/v18/v19/v20 migration steps", plan) + if plan.From.AppliedVersion != 13 || len(plan.Steps) != 8 || plan.Steps[0].Version != 14 || plan.Steps[1].Version != 15 || plan.Steps[2].Version != 16 || plan.Steps[3].Version != 17 || plan.Steps[4].Version != 18 || plan.Steps[5].Version != 19 || plan.Steps[6].Version != 20 || plan.Steps[7].Version != 21 { + t.Fatalf("development V13 plan = %+v, want v13 with v14 through v21 migration steps", plan) } } @@ -405,8 +405,8 @@ func TestInspectIndexRecognizesCanonicalAndExplicitLegacyV13Shapes(t *testing.T) if plan.From.AppliedVersion != 13 { t.Fatalf("applied version = %d, want 13", plan.From.AppliedVersion) } - if len(plan.Steps) != 7 || plan.Steps[0].Version != 14 || plan.Steps[1].Version != 15 || plan.Steps[2].Version != 16 || plan.Steps[3].Version != 17 || plan.Steps[4].Version != 18 || plan.Steps[5].Version != 19 || plan.Steps[6].Version != 20 { - t.Fatalf("V13-compatible shape has pending steps: %+v, want v14/v15/v16/v17/v18/v19/v20 migrations", plan.Steps) + if len(plan.Steps) != 8 || plan.Steps[0].Version != 14 || plan.Steps[1].Version != 15 || plan.Steps[2].Version != 16 || plan.Steps[3].Version != 17 || plan.Steps[4].Version != 18 || plan.Steps[5].Version != 19 || plan.Steps[6].Version != 20 || plan.Steps[7].Version != 21 { + t.Fatalf("V13-compatible shape has pending steps: %+v, want v14 through v21 migrations", plan.Steps) } }) } diff --git a/internal/compat/testdata/release-schemas/manifest.json b/internal/compat/testdata/release-schemas/manifest.json index 477444c..24682a1 100644 --- a/internal/compat/testdata/release-schemas/manifest.json +++ b/internal/compat/testdata/release-schemas/manifest.json @@ -667,6 +667,14 @@ "AppliedVersion": 20, "HasSourceMetadata": false, "Provenance": "Migration v20 adds canonical nullable native session identity and retained-payload identity-scoped tags." + }, + { + "Fixture": "v21.sql", + "ProvenanceSHA256": "b5dda144af458b187d0ca6d66508adfa2b71b7e904749eb42e7a9b64c89a8097", + "Signature": "sha256:6826bae924cada3504875d9d4b54c035afe368393ca1063516a11093e3de1f53", + "AppliedVersion": 21, + "HasSourceMetadata": false, + "Provenance": "Migration v21 restricts FTS updates to indexed payload changes and indexes pending echo candidates." } ] } diff --git a/internal/compat/testdata/release-schemas/v21.sql b/internal/compat/testdata/release-schemas/v21.sql new file mode 100644 index 0000000..47f4c1e --- /dev/null +++ b/internal/compat/testdata/release-schemas/v21.sql @@ -0,0 +1,241 @@ +-- Backscroll schema fixture: v21.sql +-- Hermetic schema-only fixture extending V20 with incremental FTS maintenance. + +BEGIN TRANSACTION; +CREATE TABLE annotations ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + item_uuid TEXT, + source_path TEXT NOT NULL, + ordinal INTEGER NOT NULL, + kind TEXT NOT NULL, + label TEXT NOT NULL, + source TEXT NOT NULL DEFAULT 'agent', + created_at TEXT NOT NULL, + UNIQUE(source_path, ordinal, kind) +); + +CREATE INDEX idx_annotations_kind ON annotations(kind); +CREATE INDEX idx_annotations_uuid ON annotations(item_uuid); + +CREATE TABLE chunks ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + source_id TEXT NOT NULL, + chunk_idx INTEGER NOT NULL, + content TEXT NOT NULL, + token_count INTEGER NOT NULL, + created_at INTEGER NOT NULL, embedding BLOB, + UNIQUE(source_id, chunk_idx) +); +CREATE INDEX idx_chunks_source_id ON chunks (source_id); + +CREATE TABLE correction_signals ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + item_uuid TEXT, + source_path TEXT NOT NULL, + ordinal INTEGER NOT NULL, + detector TEXT NOT NULL, + confidence REAL NOT NULL, + extraction_version INTEGER NOT NULL, + UNIQUE(source_path, ordinal, detector) +); +CREATE INDEX idx_correction_signals_confidence ON correction_signals(confidence DESC); +CREATE INDEX idx_correction_signals_detector ON correction_signals(detector); +CREATE INDEX idx_correction_signals_source ON correction_signals(source_path); + +CREATE TABLE dynamic_stopwords (term TEXT PRIMARY KEY); + +CREATE TABLE embedding_metadata ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + chunk_id INTEGER NOT NULL REFERENCES chunks(id) ON DELETE CASCADE, + model_name TEXT NOT NULL, + model_version TEXT NOT NULL, + dimensions INTEGER NOT NULL, + created_at INTEGER NOT NULL +); + +CREATE TABLE indexed_files ( + path TEXT PRIMARY KEY, + hash TEXT NOT NULL, + last_indexed DATETIME DEFAULT CURRENT_TIMESTAMP +, file_size INTEGER, file_mtime TEXT); + +CREATE TABLE message_templates ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + signature TEXT UNIQUE NOT NULL, + normalization_version INTEGER NOT NULL, + template_text TEXT NOT NULL, + occurrence_count INTEGER NOT NULL DEFAULT 1, + first_seen TEXT, + last_seen TEXT +); +CREATE INDEX idx_templates_sig ON message_templates(signature); +CREATE INDEX idx_templates_version ON message_templates(normalization_version); + +CREATE TABLE schema_migrations ( + version INTEGER PRIMARY KEY, + name TEXT NOT NULL, + applied_on TEXT NOT NULL, + checksum TEXT NOT NULL + ); + +CREATE TABLE search_items ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + source TEXT NOT NULL DEFAULT 'session', + source_path TEXT NOT NULL, + ordinal INTEGER NOT NULL, + role TEXT NOT NULL, + text TEXT NOT NULL, + timestamp TEXT, + uuid TEXT UNIQUE, + project TEXT, + content_type TEXT NOT NULL DEFAULT 'text', + extraction_version INTEGER, + was_interrupted INTEGER +); +CREATE INDEX idx_search_items_project ON search_items(project); +CREATE INDEX idx_search_items_source_path ON search_items(source_path); + +CREATE VIRTUAL TABLE messages_fts USING fts5( + text, + content=search_items, + content_rowid=id, + tokenize='porter unicode61' +); +CREATE VIRTUAL TABLE messages_vocab USING fts5vocab(messages_fts, 'row'); + +CREATE TRIGGER search_items_ad_msg AFTER DELETE ON search_items +WHEN old.content_type IN ('text', 'code', 'reasoning') BEGIN + INSERT INTO messages_fts(messages_fts, rowid, text) VALUES('delete', old.id, old.text); +END; +CREATE TRIGGER search_items_ad_tool AFTER DELETE ON search_items +WHEN old.content_type = 'tool' BEGIN + INSERT INTO tool_fts(tool_fts, rowid, text) VALUES('delete', old.id, old.text); +END; +CREATE TRIGGER search_items_ai_msg AFTER INSERT ON search_items +WHEN new.content_type IN ('text', 'code', 'reasoning') BEGIN + INSERT INTO messages_fts(rowid, text) VALUES (new.id, new.text); +END; +CREATE TRIGGER search_items_ai_tool AFTER INSERT ON search_items +WHEN new.content_type = 'tool' BEGIN + INSERT INTO tool_fts(rowid, text) VALUES (new.id, new.text); +END; +CREATE TRIGGER search_items_au_msg AFTER UPDATE ON search_items +WHEN old.content_type IN ('text', 'code', 'reasoning') BEGIN + INSERT INTO messages_fts(messages_fts, rowid, text) VALUES('delete', old.id, old.text); + INSERT INTO messages_fts(rowid, text) VALUES (new.id, new.text); +END; +CREATE TRIGGER search_items_au_tool AFTER UPDATE ON search_items +WHEN old.content_type = 'tool' BEGIN + INSERT INTO tool_fts(tool_fts, rowid, text) VALUES('delete', old.id, old.text); + INSERT INTO tool_fts(rowid, text) VALUES (new.id, new.text); +END; + +CREATE TABLE session_tags ( + source_path TEXT NOT NULL, + tag TEXT NOT NULL, + PRIMARY KEY (source_path, tag) +); +CREATE INDEX idx_session_tags_tag ON session_tags(tag); + +CREATE TABLE template_matches ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + template_id INTEGER NOT NULL, + item_uuid TEXT, + source_path TEXT NOT NULL, + ordinal INTEGER NOT NULL, + UNIQUE(source_path, ordinal, template_id), + FOREIGN KEY(template_id) REFERENCES message_templates(id) +); +CREATE INDEX idx_matches_template ON template_matches(template_id); +CREATE INDEX idx_matches_uuid ON template_matches(item_uuid); +CREATE INDEX idx_template_matches_source ON template_matches(source_path); + +CREATE TABLE tool_events ( + id INTEGER PRIMARY KEY AUTOINCREMENT, + message_uuid TEXT, + source_path TEXT NOT NULL, + ordinal INTEGER NOT NULL, + tool_name TEXT NOT NULL, + command_head TEXT, + is_error INTEGER, + exit_code INTEGER, + extraction_version INTEGER NOT NULL, + UNIQUE(source_path, ordinal) +); +CREATE INDEX idx_tool_events_tool ON tool_events(tool_name); +CREATE INDEX idx_tool_events_uuid ON tool_events(message_uuid); +CREATE UNIQUE INDEX idx_tool_events_uuid_unique ON tool_events(message_uuid) WHERE message_uuid IS NOT NULL; + +CREATE VIRTUAL TABLE tool_fts USING fts5( + text, + content=search_items, + content_rowid=id, + tokenize='trigram' +); +CREATE VIRTUAL TABLE tool_vocab USING fts5vocab(tool_fts, 'row'); + +INSERT INTO schema_migrations (version, name, applied_on, checksum) VALUES (1, 'V1 core schema', '2026-08-22 18:41:06', '4e07949ccd3912fb3c0e149be9a2e05fdd51f8cedb8df1f28b3bb5ac5afe532a'); +INSERT INTO schema_migrations (version, name, applied_on, checksum) VALUES (2, 'V2 embedding tables', '2026-08-22 18:41:06', '37dc9627f01f0e2d0fbea6bba5cd9f609d5da05089eeb9541a057dd2290cf8af'); +INSERT INTO schema_migrations (version, name, applied_on, checksum) VALUES (3, 'V3 embedding blob column', '2026-08-22 18:41:06', '36cd183f10ff84ab4753be027078cdd710b46efdc830053d4a641af725006ea5'); +INSERT INTO schema_migrations (version, name, applied_on, checksum) VALUES (4, 'V4 tool_fts trigram index', '2026-08-22 18:41:06', '77e59cf515f33c466282e0f3e921377f584794d0b7cade1704f566374a569a55'); +INSERT INTO schema_migrations (version, name, applied_on, checksum) VALUES (5, 'V5 drop phantom session_events', '2026-08-22 18:41:06', 'aedaab81efb6bc34d3f664468b71c1a716f5b55d5132156cb43bd7462d549c7b'); +INSERT INTO schema_migrations (version, name, applied_on, checksum) VALUES (6, 'V6 drop phantom source_metadata column', '2026-08-22 18:41:06', 'a327b9b6e7b8f5fe369c9fc08093ac80a87640daa515890af94b269d430e9378'); +INSERT INTO schema_migrations (version, name, applied_on, checksum) VALUES (7, 'V7 reasoning content_type routes to messages_fts', '2026-08-22 18:41:06', 'a80704442c2a0084f98e4bc53978364b14d1c6bd3bd99ba6119a2a9ecbd685e7'); +INSERT INTO schema_migrations (version, name, applied_on, checksum) VALUES (8, 'V8 perennity: extraction_version, was_interrupted, tool_events', '2026-08-22 18:41:06', '6853d72ded3bdc775b52507321c31df44bf35e8277719432ca8aa126ee16cec1'); +INSERT INTO schema_migrations (version, name, applied_on, checksum) VALUES (9, 'V9 tool_events uuid uniqueness index', '2026-08-22 18:41:06', 'b16094805a4e08f6e0dd56bce5266c7c5fd71934389da9d13c4132076e546ca2'); +INSERT INTO schema_migrations (version, name, applied_on, checksum) VALUES (10, 'V10 template mining: message_templates, template_matches', '2026-08-22 18:41:06', '0e548d0cb6c47147726f943bfc860500ca9a9bc821df0601998876ea5e9652c2'); +INSERT INTO schema_migrations (version, name, applied_on, checksum) VALUES (11, 'V11 correction detection: correction_signals', '2026-08-22 18:41:06', '5a7180f901c5feacb67cd104a61e7b7ba9cec69aaeab1d2b5d723459dc590456'); +INSERT INTO schema_migrations (version, name, applied_on, checksum) VALUES (12, 'V12 agent classification: annotations (free-form labels; enum freeze deferred)', '2026-08-22 18:41:06', 'b3fb66fd2924a9f07a3e4ec0ba253fc1d6965664615a795be6b22d01ab292108'); +INSERT INTO schema_migrations (version, name, applied_on, checksum) VALUES (13, 'V13 backfill discovery indexes', '2026-08-22 18:41:06', '2172ce531c670806933ffe3005fdc0a2ebb8eb3f84d2bd0d8fa608dddb5d136e'); +INSERT INTO schema_migrations (version, name, applied_on, checksum) VALUES (14, 'V14 file metadata prefilter', '2026-08-22 18:41:06', '1f276c51041635b661890341d5de7b3f19b2aa0ed6056ec00179e8406a62da7f'); +COMMIT; + +ALTER TABLE search_items ADD COLUMN search_echo INTEGER DEFAULT 0; +UPDATE search_items SET search_echo = NULL WHERE source = 'session'; +INSERT INTO schema_migrations VALUES (15, 'V15 search echo provenance', '2026-09-10 00:00:00', '9b7b241b37c48b808134c909df64eba2a5bbb3a592a754dad662a8c4826aba09'); + +ALTER TABLE search_items ADD COLUMN origin TEXT NOT NULL DEFAULT 'unknown' + CHECK (origin IN ('human', 'assistant', 'system', 'automation', 'unknown')); +ALTER TABLE search_items ADD COLUMN origin_version INTEGER; +INSERT INTO schema_migrations VALUES (16, 'V16 parser-backed message origin', '2026-09-11 00:00:00', 'e80730f36197ce73a4c5d8d900863914cf38b516d742ce66217e9d3c81aed6cf'); + +ALTER TABLE search_items ADD COLUMN project_evidence TEXT + CHECK(project_evidence IN ('unknown','path_fallback','registry','native_cwd')); +INSERT INTO schema_migrations VALUES(17,'V17 project attribution evidence','2026-10-13','f29dc8d36c05df4ccba1e63f709b919d032b03a94ab95e282ce8449050cb9c86'); + +ALTER TABLE indexed_files ADD COLUMN ingestion_fingerprint TEXT; +INSERT INTO schema_migrations VALUES(18,'V18 ingestion semantics fingerprint','2026-10-15','979fffaf1ca510d1213df712d0d43fef57aca88c4da3b77f4115fe27f35e0b72'); + +ALTER TABLE tool_events ADD COLUMN native_call_id TEXT; +INSERT INTO schema_migrations VALUES(19,'V19 native tool call identity','2026-10-20','b4272408f1d908ec5cce6f18b87af6b9ada909c337455d2da1c651e6e4f2efea'); + +ALTER TABLE search_items ADD COLUMN session_id TEXT CHECK(session_id IS NULL OR length(session_id)>0); +CREATE TABLE session_identity_tags(source_path TEXT NOT NULL,session_id TEXT NOT NULL CHECK(length(session_id)>0),tag TEXT NOT NULL,PRIMARY KEY(source_path,session_id,tag)); +CREATE INDEX idx_session_identity_tags_tag ON session_identity_tags(tag); +CREATE VIEW scoped_session_tags AS SELECT source_path,NULL AS session_id,tag FROM session_tags UNION ALL SELECT source_path,session_id,tag FROM session_identity_tags; +INSERT INTO schema_migrations VALUES(20,'V20 native session identity','2026-10-20','441a23ed3bc6d1ca9366e54ee011ca1a33f0e2e8c1fee1dbdeefba844ffa57aa'); + +DROP TRIGGER search_items_au_tool; +DROP TRIGGER search_items_au_msg; +CREATE TRIGGER search_items_au_tool AFTER UPDATE ON search_items +WHEN (old.id IS NOT new.id OR old.text IS NOT new.text OR old.content_type IS NOT new.content_type) +BEGIN + INSERT INTO tool_fts(tool_fts, rowid, text) + SELECT 'delete', old.id, old.text WHERE old.content_type = 'tool'; + INSERT INTO tool_fts(rowid, text) + SELECT new.id, new.text WHERE new.content_type = 'tool'; +END; +CREATE TRIGGER search_items_au_msg AFTER UPDATE ON search_items +WHEN (old.id IS NOT new.id OR old.text IS NOT new.text OR old.content_type IS NOT new.content_type) +BEGIN + INSERT INTO messages_fts(messages_fts, rowid, text) + SELECT 'delete', old.id, old.text WHERE old.content_type IN ('text','code','reasoning'); + INSERT INTO messages_fts(rowid, text) + SELECT new.id, new.text WHERE new.content_type IN ('text','code','reasoning'); +END; +CREATE INDEX idx_search_items_echo_pending ON search_items(source_path) +WHERE source = 'session' AND search_echo IS NULL; +CREATE INDEX idx_search_items_echo_zero ON search_items(source_path) +WHERE content_type = 'tool' AND COALESCE(search_echo,0) = 0 AND text LIKE '%backscroll%'; + +INSERT INTO schema_migrations VALUES(21,'V21 incremental FTS maintenance','2026-10-22','e48f2723c248182d0d4d532976b9b6c195b3a006d7a1fd8b284eb89e257c9ede'); diff --git a/internal/storage/migration_plan_test.go b/internal/storage/migration_plan_test.go index ccc790f..b545c7b 100644 --- a/internal/storage/migration_plan_test.go +++ b/internal/storage/migration_plan_test.go @@ -69,7 +69,7 @@ func TestEveryCatalogFixtureReachesCurrentSemanticHead(t *testing.T) { "v4.sql", "v5-with-source-metadata.sql", "v5-without-source-metadata.sql", "v6.sql", "v7.sql", "v8.sql", "v9.sql", "v10.sql", "v11.sql", "v12.sql", "v13.sql", "v13-legacy-existing-schema-migrations.sql", - "v13-legacy-alter-built.sql", "v13-development-alter-built.sql", "v14.sql", "v15.sql", "v16.sql", "v17.sql", "v18.sql", "v19.sql", "v20.sql", + "v13-legacy-alter-built.sql", "v13-development-alter-built.sql", "v14.sql", "v15.sql", "v16.sql", "v17.sql", "v18.sql", "v19.sql", "v20.sql", "v21.sql", } for _, name := range required { if !seen[name] { @@ -927,6 +927,7 @@ func TestLockedMigrationPlanFromEmptySchemaCreatesCurrentShape(t *testing.T) { {Version: 18, Name: "V18 ingestion semantics fingerprint"}, {Version: 19, Name: "V19 native tool call identity"}, {Version: 20, Name: "V20 native session identity"}, + {Version: 21, Name: "V21 incremental FTS maintenance"}, } if err := applyMigrationPlanForTest(ctx, db, plan); err != nil { t.Fatalf("apply full plan: %v", err) @@ -2144,6 +2145,7 @@ func authoritativeCurrentMigrationRows() []storageMigrationRow { {Version: 18, Name: "V18 ingestion semantics fingerprint", Checksum: "979fffaf1ca510d1213df712d0d43fef57aca88c4da3b77f4115fe27f35e0b72"}, {Version: 19, Name: "V19 native tool call identity", Checksum: "b4272408f1d908ec5cce6f18b87af6b9ada909c337455d2da1c651e6e4f2efea"}, {Version: 20, Name: "V20 native session identity", Checksum: "441a23ed3bc6d1ca9366e54ee011ca1a33f0e2e8c1fee1dbdeefba844ffa57aa"}, + {Version: 21, Name: "V21 incremental FTS maintenance", Checksum: "e48f2723c248182d0d4d532976b9b6c195b3a006d7a1fd8b284eb89e257c9ede"}, } } diff --git a/internal/storage/migrations.go b/internal/storage/migrations.go index 1710027..b3f49f7 100644 --- a/internal/storage/migrations.go +++ b/internal/storage/migrations.go @@ -26,10 +26,11 @@ func newDatabaseMigrations() []migrationApplier { applyV9, applyV10, applyV11, applyV12, applyV13, applyV14, applyV15, applyV16, applyV17, applyV18, applyV19, applyV20, + applyV21, } } -// schema objects and migration records from V1 through V20 are committed as a +// schema objects and migration records from V1 through V21 are committed as a // single unit. Existing databases still migrate through OpenCompatible. func (d *Database) setupNewDatabaseSchema(ctx context.Context, migrations []migrationApplier) (err error) { tx, err := d.db.BeginTx(ctx, nil) diff --git a/internal/storage/migrations_v16_test.go b/internal/storage/migrations_v16_test.go index 9b0baf6..4f97655 100644 --- a/internal/storage/migrations_v16_test.go +++ b/internal/storage/migrations_v16_test.go @@ -35,8 +35,8 @@ func TestV16MigrationAddsConstrainedMessageOrigin(t *testing.T) { if err != nil || diag != nil { t.Fatalf("inspect v15 error=%v diagnostic=%+v", err, diag) } - if len(plan.Steps) != 5 || plan.Steps[0] != (compat.MigrationStep{Version: 16, Name: "V16 parser-backed message origin"}) { - t.Fatalf("v15 migration plan = %+v, want v16 through v20", plan.Steps) + if len(plan.Steps) != 6 || plan.Steps[0] != (compat.MigrationStep{Version: 16, Name: "V16 parser-backed message origin"}) { + t.Fatalf("v15 migration plan = %+v, want v16 through v21", plan.Steps) } if err := applyMigrationPlanForTest(context.Background(), db, plan); err != nil { t.Fatalf("apply v16: %v", err) diff --git a/internal/storage/migrations_v17_test.go b/internal/storage/migrations_v17_test.go index 1af06db..6bd0a36 100644 --- a/internal/storage/migrations_v17_test.go +++ b/internal/storage/migrations_v17_test.go @@ -21,7 +21,7 @@ func TestV17DefaultsHistoricalProjectEvidenceToUnknown(t *testing.T) { if err != nil || diag != nil { t.Fatalf("%v %+v", err, diag) } - if len(plan.Steps) != 4 || plan.Steps[0].Version != 17 || plan.Steps[1].Version != 18 || plan.Steps[2].Version != 19 || plan.Steps[3].Version != 20 { + if len(plan.Steps) != 5 || plan.Steps[0].Version != 17 || plan.Steps[1].Version != 18 || plan.Steps[2].Version != 19 || plan.Steps[3].Version != 20 || plan.Steps[4].Version != 21 { t.Fatal(plan.Steps) } if err := applyMigrationPlanForTest(context.Background(), db, plan); err != nil { diff --git a/internal/storage/migrations_v18_test.go b/internal/storage/migrations_v18_test.go index 6e30bf0..06eb13d 100644 --- a/internal/storage/migrations_v18_test.go +++ b/internal/storage/migrations_v18_test.go @@ -23,8 +23,8 @@ func TestV18AddsNullableIngestionFingerprintWithoutGuessingHistory(t *testing.T) if err != nil || diag != nil { t.Fatalf("inspect v17 error=%v diagnostic=%+v", err, diag) } - if len(plan.Steps) != 3 || plan.Steps[0] != (compat.MigrationStep{Version: 18, Name: "V18 ingestion semantics fingerprint"}) || plan.Steps[1].Version != 19 || plan.Steps[2].Version != 20 { - t.Fatalf("v17 migration plan = %+v, want v18 then v19 then v20", plan.Steps) + if len(plan.Steps) != 4 || plan.Steps[0] != (compat.MigrationStep{Version: 18, Name: "V18 ingestion semantics fingerprint"}) || plan.Steps[1].Version != 19 || plan.Steps[2].Version != 20 || plan.Steps[3].Version != 21 { + t.Fatalf("v17 migration plan = %+v, want v18 through v21", plan.Steps) } if err := applyMigrationPlanForTest(context.Background(), db, plan); err != nil { t.Fatal(err) diff --git a/internal/storage/migrations_v19_test.go b/internal/storage/migrations_v19_test.go index aeef7bd..109135c 100644 --- a/internal/storage/migrations_v19_test.go +++ b/internal/storage/migrations_v19_test.go @@ -24,7 +24,7 @@ func TestV19AddsNullableNativeCallIdentity(t *testing.T) { if err != nil || diag != nil { t.Fatalf("inspect v18: %v %+v", err, diag) } - if len(plan.Steps) != 2 || plan.Steps[0] != (compat.MigrationStep{Version: 19, Name: "V19 native tool call identity"}) || plan.Steps[1] != (compat.MigrationStep{Version: 20, Name: "V20 native session identity"}) { + if len(plan.Steps) != 3 || plan.Steps[0] != (compat.MigrationStep{Version: 19, Name: "V19 native tool call identity"}) || plan.Steps[1] != (compat.MigrationStep{Version: 20, Name: "V20 native session identity"}) || plan.Steps[2] != (compat.MigrationStep{Version: 21, Name: "V21 incremental FTS maintenance"}) { t.Fatalf("v18 plan: %+v", plan.Steps) } if err := applyMigrationPlanForTest(context.Background(), db, plan); err != nil { diff --git a/internal/storage/migrations_v20_test.go b/internal/storage/migrations_v20_test.go index 620054c..14d7355 100644 --- a/internal/storage/migrations_v20_test.go +++ b/internal/storage/migrations_v20_test.go @@ -21,7 +21,7 @@ func TestV20RetainsCanonicalLegacyPayloadAndFingerprint(t *testing.T) { t.Fatal(err) } plan, diagnostic, err := compat.InspectIndex(context.Background(), db.DB()) - if err != nil || diagnostic != nil || len(plan.Steps) != 1 || plan.Steps[0].Version != 20 { + if err != nil || diagnostic != nil || len(plan.Steps) != 2 || plan.Steps[0].Version != 20 || plan.Steps[1].Version != 21 { t.Fatalf("V19 plan=%+v diagnostic=%+v err=%v", plan, diagnostic, err) } if err := applyMigrationPlanForTest(context.Background(), db, plan); err != nil { diff --git a/internal/storage/migrations_v21.go b/internal/storage/migrations_v21.go new file mode 100644 index 0000000..5502c3c --- /dev/null +++ b/internal/storage/migrations_v21.go @@ -0,0 +1,44 @@ +package storage + +import ( + "context" + "database/sql" + "fmt" + + "github.com/pablontiv/backscroll/internal/compat" +) + +// Both external-content indexes store only text, keyed by search_items.id. +// Delete from the old route and insert into the new route independently so +// content-type moves (including unsupported types) preserve index membership. +const sqlV21 = ` +DROP TRIGGER search_items_au_tool; +DROP TRIGGER search_items_au_msg; +CREATE TRIGGER search_items_au_tool AFTER UPDATE ON search_items +WHEN (old.id IS NOT new.id OR old.text IS NOT new.text OR old.content_type IS NOT new.content_type) +BEGIN + INSERT INTO tool_fts(tool_fts, rowid, text) + SELECT 'delete', old.id, old.text WHERE old.content_type = 'tool'; + INSERT INTO tool_fts(rowid, text) + SELECT new.id, new.text WHERE new.content_type = 'tool'; +END; +CREATE TRIGGER search_items_au_msg AFTER UPDATE ON search_items +WHEN (old.id IS NOT new.id OR old.text IS NOT new.text OR old.content_type IS NOT new.content_type) +BEGIN + INSERT INTO messages_fts(messages_fts, rowid, text) + SELECT 'delete', old.id, old.text WHERE old.content_type IN ('text','code','reasoning'); + INSERT INTO messages_fts(rowid, text) + SELECT new.id, new.text WHERE new.content_type IN ('text','code','reasoning'); +END; +CREATE INDEX idx_search_items_echo_pending ON search_items(source_path) +WHERE source = 'session' AND search_echo IS NULL; +CREATE INDEX idx_search_items_echo_zero ON search_items(source_path) +WHERE content_type = 'tool' AND COALESCE(search_echo,0) = 0 AND text LIKE '%backscroll%'; +` + +func applyV21(ctx context.Context, tx *sql.Tx, _ compat.SchemaShape) error { + if _, err := tx.ExecContext(ctx, sqlV21); err != nil { + return fmt.Errorf("apply v21 incremental FTS maintenance: %w", err) + } + return recordMigration(ctx, tx, 21, "V21 incremental FTS maintenance", sqlV21, "record migration v21") +} diff --git a/internal/storage/migrations_v21_test.go b/internal/storage/migrations_v21_test.go new file mode 100644 index 0000000..71cec6a --- /dev/null +++ b/internal/storage/migrations_v21_test.go @@ -0,0 +1,163 @@ +package storage + +import ( + "context" + "fmt" + "strings" + "testing" + + "github.com/pablontiv/backscroll/internal/models" +) + +func TestV21FTSUpdates(t *testing.T) { + t.Parallel() + db, cleanup := newTestDB(t) + defer cleanup() + for i, kind := range []string{"text", "code", "reasoning", "tool"} { + if _, err := db.db.Exec(`INSERT INTO search_items(id,source_path,ordinal,role,text,content_type,uuid) VALUES(?, '/fts', ?, 'assistant', 'quartzmarker', ?, ?)`, i+1, i, kind, fmt.Sprint(i)); err != nil { + t.Fatal(err) + } + } + match := func(index, term string, want int) { + t.Helper() + var got int + if err := db.db.QueryRow(`SELECT COUNT(*) FROM `+index+` WHERE `+index+` MATCH ?`, term).Scan(&got); err != nil || got != want { + t.Fatalf("%s MATCH %s = %d, want %d: %v", index, term, got, want, err) + } + } + integrity := func() { + t.Helper() + for _, index := range []string{"messages_fts", "tool_fts"} { + if _, err := db.db.Exec(`INSERT INTO ` + index + `(` + index + `) VALUES('integrity-check')`); err != nil { + t.Fatal(err) + } + } + } + match("messages_fts", "quartzmarker", 3) + match("tool_fts", "quartzmarker", 1) + searchUUIDs := func() string { + t.Helper() + results, _, err := db.Search("quartzmarker", models.SearchOptions{AllProjects: true, Limit: 10}) + if err != nil { + t.Fatal(err) + } + var uuids []string + for _, result := range results { + uuids = append(uuids, *result.UUID) + } + return strings.Join(uuids, ",") + } + beforeUUIDs := searchUUIDs() + var before, after int + if err := db.db.QueryRow("SELECT total_changes()").Scan(&before); err != nil { + t.Fatal(err) + } + if _, err := db.db.Exec(`UPDATE search_items SET project='proven',project_evidence='registry',origin='assistant',origin_version=1,search_echo=0,extraction_version=3,was_interrupted=1,session_id='native'`); err != nil { + t.Fatal(err) + } + if err := db.db.QueryRow("SELECT total_changes()").Scan(&after); err != nil { + t.Fatal(err) + } + if after-before != 4 { + t.Fatalf("metadata update changed %d rows including derived writes, want exactly 4", after-before) + } + match("messages_fts", "quartzmarker", 3) + match("tool_fts", "quartzmarker", 1) + if got := searchUUIDs(); got != beforeUUIDs { + t.Fatalf("metadata update changed ranked search UUIDs: %s -> %s", beforeUUIDs, got) + } + integrity() + for _, step := range []struct { + sql string + proseOld, toolOld, proseNew, toolNew int + }{ + {`UPDATE search_items SET text='cobaltmarker' WHERE id=1`, 2, 1, 1, 0}, + {`UPDATE search_items SET content_type='tool' WHERE id=1`, 2, 1, 0, 1}, + {`UPDATE search_items SET content_type='reasoning',text='cobaltmarker' WHERE id=4`, 2, 0, 1, 1}, + {`UPDATE search_items SET id=40 WHERE id=4`, 2, 0, 1, 1}, + {`UPDATE search_items SET content_type='other' WHERE id=40`, 2, 0, 0, 1}, + {`UPDATE search_items SET content_type='code' WHERE id=40`, 2, 0, 1, 1}, + } { + if _, err := db.db.Exec(step.sql); err != nil { + t.Fatal(err) + } + match("messages_fts", "quartzmarker", step.proseOld) + match("tool_fts", "quartzmarker", step.toolOld) + match("messages_fts", "cobaltmarker", step.proseNew) + match("tool_fts", "cobaltmarker", step.toolNew) + integrity() + } + if _, err := db.db.Exec(`DELETE FROM search_items`); err != nil { + t.Fatal(err) + } + match("messages_fts", "quartzmarker OR cobaltmarker", 0) + match("tool_fts", "quartzmarker OR cobaltmarker", 0) + integrity() +} + +func TestStartupMutationAccessPaths(t *testing.T) { + t.Parallel() + db, cleanup := newTestDB(t) + defer cleanup() + for _, tc := range []struct { + name, query, access string + args []any + }{ + {"legacy", legacySearchCleanupSQL, "idx_search_items_source_path (source_path=?)", []any{"/path"}}, + {"project", `UPDATE search_items INDEXED BY sqlite_autoindex_search_items_1 SET project=?,project_evidence=? WHERE source_path=? AND uuid=? AND text=? AND content_type=? AND (session_id IS NULL OR session_id IS ?) AND COALESCE(project_evidence,'unknown') IN ('unknown','path_fallback') AND ? IN ('path_fallback','registry','native_cwd')`, "sqlite_autoindex_search_items_1 (uuid=?)", []any{"p", "registry", "/path", "uuid", "text", "text", nil, "registry"}}, + {"origin", `UPDATE search_items INDEXED BY sqlite_autoindex_search_items_1 SET origin=CASE WHEN origin='unknown' THEN ? ELSE origin END,origin_version=? WHERE uuid=?`, "sqlite_autoindex_search_items_1 (uuid=?)", []any{"assistant", 1, "uuid"}}, + } { + t.Run(tc.name, func(t *testing.T) { + rows, err := db.db.Query("EXPLAIN QUERY PLAN "+tc.query, tc.args...) + if err != nil { + t.Fatal(err) + } + defer rows.Close() + found := false + for rows.Next() { + var id, parent, unused int + var detail string + if err := rows.Scan(&id, &parent, &unused, &detail); err != nil { + t.Fatal(err) + } + t.Log(detail) + found = found || strings.Contains(detail, tc.access) + } + if err := rows.Err(); err != nil { + t.Fatal(err) + } + if !found { + t.Fatalf("want bounded access %s", tc.access) + } + }) + } +} + +func TestMetadataReplayPreservesStopwords(t *testing.T) { + t.Parallel() + db, cleanup := newTestDB(t) + defer cleanup() + file := IndexedFile{SourcePath: "/perennial", Source: "session", Hash: "one", Messages: []IndexedMessage{{UUID: "retained", Text: "quartzmarker", ContentType: "text", Role: "assistant"}}} + if err := db.SyncFiles([]IndexedFile{file}); err != nil { + t.Fatal(err) + } + if _, err := db.db.Exec(`CREATE TEMP TRIGGER observe_stopword_refresh BEFORE DELETE ON dynamic_stopwords BEGIN SELECT RAISE(ABORT,'prose vocabulary refresh'); END`); err != nil { + t.Fatal(err) + } + file.Hash = "two" + if err := db.SyncFilesContext(context.Background(), []IndexedFile{file}); err != nil { + t.Fatalf("unchanged prose replay: %v", err) + } + file.Messages = append(file.Messages, IndexedMessage{UUID: "tool", Ordinal: 1, Text: "toolmarker", ContentType: "tool", Role: "assistant"}) + if err := db.SyncFiles([]IndexedFile{file}); err != nil { + t.Fatalf("tool-only addition: %v", err) + } + file.Messages = append(file.Messages, IndexedMessage{UUID: "new-prose", Ordinal: 2, Text: "cobaltmarker", ContentType: "text", Role: "assistant"}) + if err := db.SyncFiles([]IndexedFile{file}); err == nil || !strings.Contains(err.Error(), "prose vocabulary refresh") { + t.Fatalf("new prose must refresh atomically: %v", err) + } + var count int + if err := db.db.QueryRow(`SELECT COUNT(*) FROM search_items WHERE uuid='new-prose'`).Scan(&count); err != nil || count != 0 { + t.Fatalf("refresh failure rollback: count=%d err=%v", count, err) + } +} diff --git a/internal/storage/origin.go b/internal/storage/origin.go index 7a2b86f..8e8ad8e 100644 --- a/internal/storage/origin.go +++ b/internal/storage/origin.go @@ -51,7 +51,7 @@ func enrichOrigin(ctx context.Context, tx *sql.Tx, uuid string, incoming models. return nil } _, err := tx.ExecContext(ctx, ` - UPDATE search_items + UPDATE search_items INDEXED BY sqlite_autoindex_search_items_1 SET origin = CASE WHEN origin = 'unknown' THEN ? ELSE origin END, origin_version = ? WHERE uuid = ? diff --git a/internal/storage/queries.go b/internal/storage/queries.go index 93b98ba..f6ed752 100644 --- a/internal/storage/queries.go +++ b/internal/storage/queries.go @@ -919,7 +919,7 @@ func (d *Database) StalePaths(currentVersion int) ([]string, error) { // StalePathsContext is StalePaths with cancellation propagated through all // discovery queries and row iteration. func (d *Database) StalePathsContext(ctx context.Context, currentVersion int) ([]string, error) { - return d.stalePathsContext(ctx, currentVersion, false) + return d.stalePathsContext(ctx, currentVersion) } // PendingSearchEchoPaths distinguishes the v15 backlog from older extraction @@ -931,7 +931,7 @@ func (d *Database) StalePathsContext(ctx context.Context, currentVersion int) ([ // Candidate discovery uses a broad SQL prefilter (searchEchoZeroPrefilterSQL: // any echo-zero tool row containing the literal substring "backscroll") and // the strict per-shape decision happens in Go via -// directsearch.IsSerializedDirectSearchCall (filterEchoZeroPaths). Enumerating +// directsearch.IsSerializedDirectSearchCall. Enumerating // the accepted separator byte-sequences in SQL is provably unbounded — // strings.Fields accepts every unicode.IsSpace rune and the shell argv adds // JSON escaping on top — so no SQL-side shape matching exists to drift out of @@ -940,36 +940,64 @@ func (d *Database) PendingSearchEchoPaths() ([]string, error) { return d.PendingSearchEchoPathsContext(context.Background()) } -// PendingSearchEchoPathsContext is PendingSearchEchoPaths with cancellation -// propagated through discovery and strict echo filtering. +// PendingSearchEchoPathsContext streams the indexed candidate population once. +// The SQL predicate is a broad superset; only the shared Go predicate decides +// whether an echo-zero serialized call requires parser-backed pairing. func (d *Database) PendingSearchEchoPathsContext(ctx context.Context) ([]string, error) { - return d.stalePathsContext(ctx, 0, true) + if err := ctx.Err(); err != nil { + return nil, err + } + rows, err := d.db.QueryContext(ctx, ` + SELECT candidates.source_path, candidates.text, candidates.pending + FROM ( + SELECT source_path, '' AS text, 1 AS pending + FROM search_items WHERE source = 'session' AND search_echo IS NULL + UNION ALL + SELECT source_path, text, 0 AS pending + FROM search_items WHERE source = 'session' AND search_echo IS NOT NULL + AND `+searchEchoZeroPrefilterSQL("search_items")+` + ) candidates + LEFT JOIN indexed_files ON candidates.source_path = indexed_files.path + ORDER BY indexed_files.last_indexed ASC, candidates.source_path ASC`) + if err != nil { + return nil, fmt.Errorf("query pending search echoes: %w", err) + } + defer func() { _ = rows.Close() }() + var paths []string + kept := make(map[string]bool) + for rows.Next() { + if err := ctx.Err(); err != nil { + return nil, err + } + var path, text string + var pending bool + if err := rows.Scan(&path, &text, &pending); err != nil { + return nil, fmt.Errorf("scan pending search echo: %w", err) + } + if !kept[path] && (pending || directsearch.IsSerializedDirectSearchCall(text)) { + kept[path] = true + paths = append(paths, path) + } + } + if err := rows.Err(); err != nil { + return nil, fmt.Errorf("iterate pending search echoes: %w", err) + } + return paths, ctx.Err() } -func (d *Database) stalePathsContext(ctx context.Context, currentVersion int, echoOnly bool) ([]string, error) { +func (d *Database) stalePathsContext(ctx context.Context, currentVersion int) ([]string, error) { if err := ctx.Err(); err != nil { return nil, err } - query := ` + rows, err := d.db.QueryContext(ctx, ` SELECT DISTINCT search_items.source_path FROM search_items LEFT JOIN indexed_files ON search_items.source_path = indexed_files.path WHERE search_items.source = 'session' - AND ((? AND (search_items.extraction_version IS NULL OR search_items.extraction_version < ?)) - OR search_items.search_echo IS NULL` - if echoOnly { - // Broad prefilter for echo-zero candidates: any tool row whose stored - // text contains the literal "backscroll" substring. The strict - // per-shape check happens in Go (filterEchoZeroPaths) via - // directsearch.IsSerializedDirectSearchCall. - query += ` - OR (` + searchEchoZeroPrefilterSQL("search_items") + `)` - } - query += `) + AND (search_items.extraction_version IS NULL OR search_items.extraction_version < ? + OR search_items.search_echo IS NULL) ORDER BY indexed_files.last_indexed ASC, search_items.source_path ASC - ` - - rows, err := d.db.QueryContext(ctx, query, !echoOnly, currentVersion) + `, currentVersion) if err != nil { return nil, fmt.Errorf("query stale paths: %w", err) } @@ -994,97 +1022,9 @@ func (d *Database) stalePathsContext(ctx context.Context, currentVersion int, ec return nil, fmt.Errorf("iterate stale paths: %w", err) } - if echoOnly { - filtered, err := d.filterEchoZeroPathsContext(ctx, paths) - if err != nil { - return nil, err - } - paths = filtered - } - return paths, nil } -// filterEchoZeroPaths drops paths whose only echo-zero candidate rows are -// rejected by the strict serialized-text predicate. A path survives if it has -// a v15 NULL search_echo row (those are always kept), or at least one -// echo-zero row that directsearch.IsSerializedDirectSearchCall accepts — for -// any of the three stored shapes (bash, exec_command, shell). -func (d *Database) filterEchoZeroPathsContext(ctx context.Context, paths []string) ([]string, error) { - if err := ctx.Err(); err != nil { - return nil, err - } - if len(paths) == 0 { - return paths, nil - } - keep := make(map[string]bool, len(paths)) - for _, path := range paths { - if err := ctx.Err(); err != nil { - return nil, err - } - // Reason 1: v15 NULL search_echo backlog — always keep. - var nullHit int - err := d.db.QueryRowContext(ctx, ` - SELECT 1 FROM search_items - WHERE source_path = ? AND search_echo IS NULL - LIMIT 1 - `, path).Scan(&nullHit) - if err != nil && err != sql.ErrNoRows { - return nil, fmt.Errorf("check NULL backlog for %s: %w", path, err) - } - if nullHit == 1 { - keep[path] = true - continue - } - // Reason 2: echo-zero serialized candidate. Apply the strict - // chokepoint predicate — keep the path if ANY row matches. - rows, err := d.db.QueryContext(ctx, ` - SELECT text FROM search_items - WHERE source_path = ? - AND `+searchEchoZeroPrefilterSQL("search_items")+` - `, path) - if err != nil { - return nil, fmt.Errorf("query echo-zero rows for %s: %w", path, err) - } - matched := false - for rows.Next() { - if err := ctx.Err(); err != nil { - _ = rows.Close() - return nil, err - } - var text string - if err := rows.Scan(&text); err != nil { - _ = rows.Close() - return nil, fmt.Errorf("scan echo-zero row for %s: %w", path, err) - } - if directsearch.IsSerializedDirectSearchCall(text) { - matched = true - break - } - } - _ = rows.Close() - if err := ctx.Err(); err != nil { - return nil, err - } - if err := rows.Err(); err != nil { - return nil, fmt.Errorf("iterate echo-zero rows for %s: %w", path, err) - } - if matched { - keep[path] = true - } - } - out := make([]string, 0, len(keep)) - for _, p := range paths { - if err := ctx.Err(); err != nil { - return nil, err - } - if keep[p] { - out = append(out, p) - } - } - return out, nil -} - // ReresolveProjects retains the context-taking API used by existing callers. func (d *Database) ReresolveProjects(ctx context.Context, resolver func(sourcePath string) string) (int64, error) { return d.ReresolveProjectsContext(ctx, resolver) diff --git a/internal/storage/storage_open_context_test.go b/internal/storage/storage_open_context_test.go index b9f2c3c..8c49d5a 100644 --- a/internal/storage/storage_open_context_test.go +++ b/internal/storage/storage_open_context_test.go @@ -318,8 +318,8 @@ func TestOpenWrapperMatchesOpenContext(t *testing.T) { if err := db.DB().QueryRow(`SELECT COUNT(*) FROM schema_migrations`).Scan(&versions); err != nil { t.Fatalf("%s migration count: %v", label, err) } - if versions != 20 { - t.Fatalf("%s migration count = %d, want 20", label, versions) + if versions != 21 { + t.Fatalf("%s migration count = %d, want 21", label, versions) } } } diff --git a/internal/storage/sync.go b/internal/storage/sync.go index c75c652..1920832 100644 --- a/internal/storage/sync.go +++ b/internal/storage/sync.go @@ -134,6 +134,14 @@ func reconcilePerennialOrdinals(ctx context.Context, tx *sql.Tx, file IndexedFil return messages, nil } +// The UUID unique index contains the global legacy NULL population; cleanup +// owns one path and must use its bounded access path instead. +const legacySearchCleanupSQL = "DELETE FROM search_items INDEXED BY idx_search_items_source_path WHERE source_path = ? AND uuid IS NULL" + +func isProseContentType(contentType string) bool { + return contentType == "text" || contentType == "code" || contentType == "reasoning" +} + // SyncFiles syncs a batch of files into the database without cancellation. // It is retained for compatibility with existing callers. func (d *Database) SyncFiles(files []IndexedFile) error { @@ -174,6 +182,8 @@ func (d *Database) SyncFilesContext(ctx context.Context, files []IndexedFile) (r return fmt.Errorf("validate sync UUIDs: %w", err) } + proseChanged := false + for _, file := range files { if err := ctx.Err(); err != nil { return err @@ -215,16 +225,28 @@ func (d *Database) SyncFilesContext(ctx context.Context, files []IndexedFile) (r if _, err := tx.ExecContext(ctx, "DELETE FROM template_matches WHERE source_path = ?", file.SourcePath); err != nil { return fmt.Errorf("delete old template_matches for %s: %w", file.SourcePath, err) } - if _, err := tx.ExecContext(ctx, "DELETE FROM search_items WHERE source_path = ?", file.SourcePath); err != nil { + result, err := tx.ExecContext(ctx, "DELETE FROM search_items WHERE source_path = ?", file.SourcePath) + if err != nil { return fmt.Errorf("delete old search_items for %s: %w", file.SourcePath, err) } + deleted, err := result.RowsAffected() + if err != nil { + return err + } + proseChanged = proseChanged || deleted > 0 if _, err := tx.ExecContext(ctx, "DELETE FROM tool_events WHERE source_path = ?", file.SourcePath); err != nil { return fmt.Errorf("delete old tool_events for %s: %w", file.SourcePath, err) } } else if legacyTransition { - if _, err := tx.ExecContext(ctx, "DELETE FROM search_items WHERE source_path = ? AND uuid IS NULL", file.SourcePath); err != nil { + result, err := tx.ExecContext(ctx, legacySearchCleanupSQL, file.SourcePath) + if err != nil { return fmt.Errorf("delete legacy rows for %s: %w", file.SourcePath, err) } + deleted, err := result.RowsAffected() + if err != nil { + return err + } + proseChanged = proseChanged || deleted > 0 if _, err := tx.ExecContext(ctx, "DELETE FROM tool_events WHERE source_path = ? AND message_uuid IS NULL", file.SourcePath); err != nil { return fmt.Errorf("delete legacy tool_events for %s: %w", file.SourcePath, err) } @@ -260,6 +282,9 @@ func (d *Database) SyncFilesContext(ctx context.Context, files []IndexedFile) (r if owner == nil || !owner.crossPath { msg = effectiveSyncMessage(file, msg) } + if (!perennial || msg.UUID == "" || !retained) && isProseContentType(msg.ContentType) { + proseChanged = true + } if err := syncIndexedMessage(ctx, tx, file, msg, perennial, retained); err != nil { return err } @@ -406,10 +431,12 @@ func (d *Database) SyncFilesContext(ctx context.Context, files []IndexedFile) (r } } - // Refresh dynamic stopwords before commit so a refresh failure or - // cancellation rolls back the complete synchronization batch. - if err := refreshStopwordsContext(ctx, tx); err != nil { - return fmt.Errorf("refresh stopwords: %w", err) + // Metadata-only perennial replays and tool-only additions leave the prose + // vocabulary unchanged. Refresh atomically only when corpus membership changes. + if proseChanged { + if err := refreshStopwordsContext(ctx, tx); err != nil { + return fmt.Errorf("refresh stopwords: %w", err) + } } if err := gate.commit(); err != nil { return err @@ -482,11 +509,11 @@ func syncIndexedMessage(ctx context.Context, tx *sql.Tx, file IndexedFile, msg I } } if msg.SessionID != "" && evidence == "native_cwd" { - if _, err := tx.ExecContext(ctx, `UPDATE search_items SET project=?,project_evidence='native_cwd' WHERE source_path=? AND uuid=? AND session_id=? AND COALESCE(project_evidence,'unknown') IN ('unknown','path_fallback')`, project, file.SourcePath, msg.UUID, msg.SessionID); err != nil { + if _, err := tx.ExecContext(ctx, `UPDATE search_items INDEXED BY sqlite_autoindex_search_items_1 SET project=?,project_evidence='native_cwd' WHERE source_path=? AND uuid=? AND session_id=? AND COALESCE(project_evidence,'unknown') IN ('unknown','path_fallback')`, project, file.SourcePath, msg.UUID, msg.SessionID); err != nil { return fmt.Errorf("enrich native project proof for %s: %w", file.SourcePath, err) } } - if _, err := tx.ExecContext(ctx, `UPDATE search_items SET project = ?, project_evidence = ? + if _, err := tx.ExecContext(ctx, `UPDATE search_items INDEXED BY sqlite_autoindex_search_items_1 SET project = ?, project_evidence = ? WHERE source_path = ? AND uuid = ? AND text = ? AND content_type = ? AND (session_id IS NULL OR session_id IS ?) AND COALESCE(project_evidence,'unknown') IN ('unknown','path_fallback') AND ? IN ('path_fallback','registry','native_cwd')`,