fix(db): schema 114 — idempotent dependency edges + ops metrics that tell the truth by runyourempire · Pull Request #558 · 4DA-Systems/4DA

runyourempire · 2026-08-30T17:17:58Z

What this is

First real burn-down of the ghost-command backlog (scripts/ghost-command-backlog.json): 116 entries seeded 2026-08-14, zero retired in 17 days, and a schema that could not even express progress. This PR (1) makes progress measurable, (2) deletes the first tranche — 8 commands that are pure residue of the STREETS/playbook feature retired from the app in June 2026 (doctrine: .claude/rules/intelligence-doctrine.md rule 2 retirement note, rule 8 "dead code is deleted") — and (3) records the 5 standing-query commands that PR #556 wired to real UI as how: "wired" in the new ledger.

1. Progress mechanism

ghost-command-backlog.json gains a top-level retired ledger — {command, retired_on, pr, how: "deleted"|"wired"} — and scripts/ghost-commands.cjs now:

prints a burn-down line in the summary: Burn-down: 103 backlogged, 13 retired
flags any how: "deleted" entry whose command still exists in Rust (STALE RETIRED LEDGER — a retirement must be real)
persists ghost_retired / retired_stale counts and the full ledger into .claude/wisdom/ghost-commands.json

The gate's contract is unchanged: it still blocks on NEW ghosts and unregistered commands only, and the retired ledger never feeds the live/ghost classification — a resurrected command with no caller still blocks as a NEW ghost, so the ledger cannot be used to smuggle a command past the gate. 5 new unit tests in scripts/ghost-commands.test.cjs pin all of this (21 tests total, all green).

2. Tranche 1 — 8 STREETS-residue commands deleted

Verification protocol per command: (a) no frontend call site — search across src/, public/, index.html finds the name only in src/lib/commands.ts CommandMap (type coverage, not usage — the exact artifact the 08-14 detector fix stopped counting); (b) no MCP-server reference — search across mcp-4da-server/src finds nothing (only a streets_engine DB column name, unrelated); (c) no e2e reference; (d) feature retired — STREETS/playbook was removed from the app in June 2026 (playbook_commands.rs header documents the command removal; curriculum now publishes on 4da.ai).

STREETS Localization cluster — module `streets_localization.rs` DELETED

command	evidence
`get_regional_data`	Loads `docs/streets/regions/*.json` price/context data for playbook lessons. No caller in src/, public/, e2e/, MCP. Only refs: CommandMap key, victauri allowlist, generate_handler.
`format_currency`	USD-to-locale currency formatting for STREETS lesson prices (static quarterly exchange rates). Same zero-caller evidence.
`calculate_electricity_cost`	STREETS "cost of running hardware" lesson helper. Same zero-caller evidence.

The whole module served only these three commands; deleted outright. RuntimePaths::streets_regions_dir() stays — suns/price_tracker.rs still uses it. docs/streets/ content untouched (kept per the retirement decision: content lives on for the website).

Sovereign Profile (STREETS Module S) cluster — module `sovereign_profile.rs` DELETED

The module's own header: "accumulates hardware/system facts from STREETS commands ... generates a 'Sovereign Stack Document' (STREETS Lesson 6 deliverable)". The STREETS commands that fed it were removed with the playbook UI.

command	evidence
`get_sovereign_profile`	Reads `sovereign_profile` table for the retired STREETS profile view. Zero frontend/MCP callers. Was also invoked from `tests/victauri_dogfood.rs` (3 sites) — test-only usage does not keep a command alive; those test references deleted with it.
`get_sovereign_profile_completeness`	Category-completeness gauge for the retired STREETS profile view. Zero callers.
`generate_sovereign_stack_document`	The STREETS Lesson 6 deliverable generator ("Run STREETS Module S commands to auto-populate"). Zero callers.
`save_sovereign_fact`	Manual fact entry for the retired profile view. Zero callers.
`get_execution_log`	Reads `command_execution_log` keyed by STREETS `module_id`/`lesson_idx` — the log written by the already-deleted playbook execution commands. Zero callers.

What survives: sovereign_facts::store_facts_from_execution is live infrastructure — the suns monitors (hardware_monitor, uptime_monitor) write hardware/uptime facts through it. They called it via a sovereign_profile re-export shim; they now call crate::sovereign_facts:: directly (doctrine rule 8: no compat shims). The sovereign_profile DB table and its writers are untouched, and the live unified profile (sovereign_developer_profile::export_sovereign_profile_markdown/json, feeds scoring's skill-gap boost) is untouched.

Also removed

lib.rs: the 8 generate_handler![] registrations, 2 mod declarations, 2 section comments (pure removals — no other lib.rs churn)
src/lib/commands.ts: the 8 CommandMap keys + 5 now-orphaned interfaces (SovereignProfileData, ProfileCompleteness, ExecutionLogEntry, RegionalData, ElectricityCostResult) and their export-list entries
victauri_commands.rs::REGISTERED_COMMANDS: the 8 allowlist entries
tests/victauri_dogfood.rs: sovereign_profile_returns_data test + 2 list entries
Cargo.toml: the dead streets-execution = [] feature — it gated "3 commands" that were already deleted with the playbook UI; nothing in the tree or CI references it

Explicitly NOT in this tranche (and why)

Achievements trio (get_achievement_state, get_achievements, check_daily_streak): feature-gated behind experimental with a dated owner note in lib.rs ("REMOVE BY 2026-09-15 ... its owner decides wiring vs drop"). That decision is assigned; not mine to pre-empt.
Toolkit cluster: toolkit_test_feed is live, the rest have locale keys and read as an unshipped-but-intended surface, not marked retired.
get_advantage_history: smells like Momentum-tab residue (sparkline feeder, CompoundAdvantage neighborhood) but sits in the live decision-windows module and is not marked retired — needs git-history proof first. Tranche 2 candidate.
Tech-convergence / temporal / accuracy / ai_costs (Audit remediation: remove ~53k lines of dead code, fix legal/tier accuracy, close enforcement gaps #421 orphans): backlog reasons explicitly say SIGNAL-TIER — "need UI built or the marketing pulled, not just allowlisting". (The standing-query members of that cluster got their UI in feat(settings): Standing Queries management panel — list, create with suggestions, delete #556 — see below.)
team/org/sso, translation, export surfaces: unreleased-but-intended, per tranche scope.

3. Ledger addendum — 5 commands wired by #556

PR #556 (Standing Queries management panel) gave list_standing_queries, create_standing_query, delete_standing_query, get_standing_query_suggestions, get_standing_query_matches real cmd() call sites in StandingQueriesSection.tsx / StandingQueryRow.tsx (verified in its diff — component code, not just test mocks). Their backlog entries move to the ledger as how: "wired", pr: "#556". No Rust or frontend changes for these commands in this PR — the wiring is #556's work; this PR records it.

Numbers

Backlog: 116 -> 103 (first movement since seeding)
Retired ledger: 13 entries — 8 deleted (this PR) + 5 wired (feat(settings): Standing Queries management panel — list, create with suggestions, delete #556)
Handler registrations: 382 -> 374

Validation

cargo build clean · cargo test --lib green · cargo fmt --check clean · clippy -D warnings clean
node --test scripts/ghost-commands.test.cjs — 21/21
node scripts/ghost-commands.cjs — 0 new ghosts, 0 unregistered, burn-down line prints
node scripts/validate-commands.cjs — consistent in both directions
pnpm run test (frontend) — green
node scripts/check-file-sizes.cjs — clean

🤖 Generated with Claude Code

https://claude.ai/code/session_01LrXvdHoDGUj99Fqf1fCYJY

…tell the truth Schema 114: rebuild dependency_edges to one row per logical edge (project, ecosystem, parent@version, child@version; newest wins) — the writer re-appended the whole graph every ACE scan since Phase 84, measured live at ~642k rows for ~12k distinct edges (52.7x, ~168 MB). A COALESCE-keyed UNIQUE index + matching ON CONFLICT upsert make the duplication structurally impossible. No VACUUM in the migration; the weekly job reclaims the pages. Ops metrics made truthful: - autophagy_cycles.db_size_after_bytes is measured (was hardcoded 0). - scheduler_state.last_outcome/last_duration_ms are written after every job body (9 GUI jobs + engine db_maintenance); cve_scan, chain_notify and temporal_snapshot stay NULL by design — documented in the PR. - dep_epoch_hash moves out of scheduler_state.last_run_unix (a 63-bit hash in a timestamp column poisoned MAX(last_run_unix) staleness math) into kv_store; the poisoned row is deleted in the migration. - source_health.last_error resets to NULL on a successful check. - DataHealth gains recommendation: Option<String> naming the concrete action when health_status is needs_attention. Co-Authored-By: Claude Fable 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01LrXvdHoDGUj99Fqf1fCYJY

…nally ask "why" after the fact (#564) ## The gap (E5010, standing since the 2026-08-21/23 audits) Scoring explanations were session-only. The LLM lane persists its explanations (`llm_judgments.explanation`), but the deterministic scorer's per-item breakdown — which axes fired, which dependency matches grounded it, which caps applied — evaporated with the process. Twice during audits the question "why did zustand score 0.092" was unanswerable post-hoc: `db/cache.rs` stored only score/version/signal columns, and `source_items.rank_factors` covers only the display-rank lane. `mcp_score_autopsy` itself documented the dead end: *"a restart destroys exactly the state the autopsy needs."* ## What ships **Schema 115 — `scoring_explanations`** (migration follows the #558 pattern; `TARGET_VERSION` 114 → 115): ```sql CREATE TABLE scoring_explanations ( source_item_id INTEGER PRIMARY KEY, -- one row per item, newest evaluation wins pipeline_version INTEGER NOT NULL, breakdown TEXT NOT NULL, -- bounded JSON envelope, hard-capped 8 KB scored_at TEXT NOT NULL DEFAULT (datetime('now')), FOREIGN KEY (source_item_id) REFERENCES source_items(id) ON DELETE CASCADE ); ``` ### Shape chosen The stored value is a self-contained envelope, not the raw struct: ```json {"score": 0.42, "breakdown": { ...full ScoreBreakdown... }, "truncated": {"breakdown.matched_deps": 57}} ``` - `score` is the raw score the scorer produced for THIS evaluation — embedded because the durable `relevance_score` can lag it by up to the 0.05 hysteresis band, and the batch layer re-ranks on top. The row answers "why" without joins. - `breakdown` is the complete `ScoreBreakdown` (every axis, dep match, cap, degraded-input marker), serialized compact. - `truncated` is the loud marker the size bound requires: each shortened path mapped to its original length. Arrays are shortened homogeneously — never replaced with marker strings — so the stored object still deserializes into the typed struct. ### Size bounds Two-pass bounding in `db/scoring_explanations.rs`: arrays > 8 entries and strings > 400 chars truncate (recorded in `truncated`); if the envelope still exceeds **8192 bytes** a second pass at 3/80 runs; the unconditional floor is a `{"score", "elided": true}` scalar envelope. Typical real breakdowns serialize at 1.5–3 KB and are stored verbatim. ### Write path (and its cost) The envelope rides the existing persist boundary — `persist_analysis_scores` grew a fifth tuple element (`ScorePersistRow` alias) and upserts the explanation **in the same transaction as the score write**, one prepared statement. All three persist paths feed it: the analysis cycle (via a new `EvaluatedItem.breakdown_json`, serialized at capture time from `evidence_score` — the batch layer may have deleted the full result by persist time), and backfill/drain (via `persistable()`, serialized after `finalize_scores` so the envelope score is exactly what persists). Measured (`measure_explanation_write_overhead`, ignored/manual, 1000 items, dev profile): **+5.6 µs/item inside the transaction** (10.6 ms → 16.2 ms per 1000-row persist batch) plus ~129 µs/item serialization that runs on the scoring threads *outside* the transaction — against a scoring path that spends ~100 ms+ per item on KNN, well under 0.2% of batch wall-clock. Release builds shrink the serialization further. Two deliberate skip rules: - **Hysteresis-suppressed writes keep the old explanation.** The durable score kept its old value, so the explanation that produced that value stands; replacing it would attach a breakdown whose score doesn't match the durable column. - **0-row score UPDATEs never touch the explanation table**, so a stale id can never violate the FK. **NO PIPELINE_VERSION bump** — this is write-only additional data; scores are byte-identical with or without it. ### Retention `ON DELETE CASCADE` from `source_items` (the `source_item_dependencies` precedent; `PRAGMA foreign_keys = ON` is set in `Database::new`). Every existing prune path (`prune_noise`, the retention sweeps, husk deletion) deletes the explanation in the same statement — nothing leaks, no new cleanup code to forget. ### Read path — provenance-labeled fallback, no new command `mcp_score_autopsy` keeps its live lane (full session context: similar items, live ACE matching) and now labels it `"provenance": {"kind": "live_session"}`. On a session miss — a restart, or an item only a background cycle scored — it falls back to the persisted envelope instead of erroring, labeled: > `"provenance": {"kind": "persisted", "label": "persisted at scoring time (<ts> UTC, pipeline vN)", ...}` The persisted response reuses the same measured-axis component builder (extracted as `breakdown_components`, shared by both lanes), derives matched deps/confirmed signals from the breakdown itself, and deliberately does **not** replay ACE matching against today's context — that would mislabel its provenance. The raw stored breakdown is included (`raw_breakdown`) so audits get the full record even if typed parsing of a future envelope fails. Response changes are purely additive — existing keys unchanged. ## Tests - `test_phase_115_scoring_explanations_table` — migration: columns, FK targets `source_items` with `CASCADE`, idempotent re-run keeps rows (repo phase-test pattern). - `test_explanation_write_then_read_roundtrip` — persist → read; pipeline version stamped; second evaluation replaces (PRIMARY KEY upsert, exactly one row). - `test_bounded_breakdown_json_caps_size_and_marks_truncation` — 500 matched deps + 14k-char llm_reason stays ≤ 8 KB, `truncated` records original lengths, typed re-read still works. - `test_pruned_item_cascades_explanation` — `DELETE FROM source_items` takes the explanation with it. - `test_hysteresis_suppressed_write_keeps_old_explanation` — a Δ<0.05 re-score does not replace the durable score's explanation. - `measure_explanation_write_overhead` — `#[ignore]`d manual timing (numbers above). - Existing persist/churn/hysteresis tests updated to the 5-tuple; semantics untouched. Gates: `cargo fmt` clean, `cargo clippy -- -D warnings` clean (lib + both binaries compile), `cargo test --lib` green, `node scripts/check-file-sizes.cjs` passes (warnings only; `commands.rs` at 966 — next touch should split the autopsy module out). 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01LrXvdHoDGUj99Fqf1fCYJY Co-authored-by: Claude Fable 5 <[email protected]>

runyourempire enabled auto-merge (squash) August 30, 2026 17:18

runyourempire force-pushed the fix/schema-114-dep-edges-ops-metrics branch from bfa52f6 to 2d6da09 Compare August 30, 2026 17:18

runyourempire merged commit f1f9423 into main Aug 30, 2026
15 checks passed

runyourempire deleted the fix/schema-114-dep-edges-ops-metrics branch August 30, 2026 17:44

runyourempire mentioned this pull request Aug 31, 2026

feat(scoring): schema 115 — score explanations persist, audits can finally ask "why" after the fact #564

Merged

Navigation Menu

Uh oh!

fix(db): schema 114 — idempotent dependency edges + ops metrics that tell the truth - #558

fix(db): schema 114 — idempotent dependency edges + ops metrics that tell the truth#558
runyourempire merged 1 commit into
mainfrom
fix/schema-114-dep-edges-ops-metrics

runyourempire commented Aug 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

runyourempire commented Aug 30, 2026

What this is

1. Progress mechanism

2. Tranche 1 — 8 STREETS-residue commands deleted

STREETS Localization cluster — module streets_localization.rs DELETED

Sovereign Profile (STREETS Module S) cluster — module sovereign_profile.rs DELETED

Also removed

Explicitly NOT in this tranche (and why)

3. Ledger addendum — 5 commands wired by #556

Numbers

Validation

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

STREETS Localization cluster — module `streets_localization.rs` DELETED

Sovereign Profile (STREETS Module S) cluster — module `sovereign_profile.rs` DELETED