Skip to content

MIT-648: propose an admin-only entry flag classifier - #180

Merged
ZachFara merged 4 commits into
feature/eval-setupfrom
docs/MIT-648
Nov 12, 2026
Merged

ZachFara merged 4 commits into
feature/eval-setupfrom
docs/MIT-648

Conversation

@ZachFara

@ZachFara ZachFara commented Nov 12, 2026 •

Copy link
Copy Markdown
Owner

What changed

Adds one Proposed decision record, docs/decisions/2026-11-08-MIT-648-entry-flag-classifier.md. It covers a small admin-only model that flags likely-problem entries on the dashboard, as the owner's answer to item 1 of Question MIT-613 (no agent-facing harmful=true). Docs only. No code changes.

The record answers the ticket's seven questions, citing file:line for every claim about the schema and data:

  1. Targets. Three: unsafe, wrong and noise. It recommends one binary model per target over a single multi-label model.
  2. Labels. It lists where labels come from, and how many exist locally.
  3. Features. What the model would use, and how to avoid leakage.
  4. Where it acts. Admin-only, with each claim checked against 0026, 0028 and 0029.
  5. Measures. Offline and online measures, with shadow mode and a 10% hold-back so the online numbers aren't inflated.
  6. MLOps. Registry, drift, label delay, retrain cadence, kill switch, OTel metrics and a dashboard panel.
  7. Costs and risks, with options A to D, including do-nothing.

It recommends B: reason categories and a rules score now, then a model per target once that target's labels clear a gate.

Two findings change the ticket's premise:

  • Commons stores no embeddings. There is no vector column, no CREATE EXTENSION vector, and no embedding dependency. Decision 0025 step 3 was never built, and it left the model and dimension unchosen. The record proposes TF-IDF features first and embeddings once 0025 lands. For pending text, those embeddings must come from a local model.
  • Reject and retire reasons are free text with no category, and the latest reason shows on /mine. Today no stored field says why an entry was bad. That is why the record starts with reason categories, stored apart from the reason text.
    • The category is required only on the human dashboard forms (source human).
    • validate_review, the MCP reject tool (source agent) and the server text stay unchanged, so agent rejects stay uncategorised and are not labels.
    • Any change to MCP reject would need its own record and an eval run (decisions 0023 and 0034).

Why

Ticket: MIT-648

How it was tested

  • Label counts, local Postgres only. A read-only script that refuses any host but localhost:
    • commons: 7 entries and 1 human reject, but migrated only to 0006 of 21.
    • commons_demo: 14 entries and 1 human reject.
    • commons_eval: 0 entries.
  • The record's label query runs on a fresh database with all 21 migrations (scripts/test-db.sh) and returns zeros, so it matches the current schema. Nothing touched the live database or any hosted credential.
  • make check: passed twice in the first round, once in pre-commit and once after merging origin/feature/eval-setup (1613 tests). In the fix round it passed once, in pre-commit on the fix commit. The later merge of origin/feature/eval-setup (MIT-681: kill-only same-task verdict, smoke test, store review sheet #179) changed only eval tooling and docs, and wasn't re-checked locally; CI checks it.

Learned

  • Commons has no embeddings: entries has no vector column, and only the Postgres image ships pgvector. Any ticket that assumes stored embeddings depends on 0025 step 3 first.
  • This machine's local databases (commons, commons_demo, commons_eval) are migrated only to 0006, so counts there say little about the current schema. A scripts/test-db.sh database is the way to check that a query matches the current schema.
  • scripts/test-db.sh exports COMMONS_DATABASE_URL in libpq key=value form (make_conninfo), not as a URL. To check its host, use psycopg.conninfo.conninfo_to_dict; urlsplit returns no host.
  • The worktree guard refuses a heredoc-written script chained with && to the command that runs it. Write scratch scripts with the Write tool, then run them in a separate call.
  • A ticket's expected file name can break the YYYY-MM-DD-<ticket>-<slug>.md rule in docs/decisions/README.md. The rule wins, and an agent ticket's ID is allowed in the name. The structure check does not enforce this rule, so only review catches it.
  • Agent-facing tools share validation with the dashboard (validate_review serves both the dashboard and the MCP reject tool). A proposal that requires a new field on a review has to say which entry point gets it, or it reads as an MCP tool change.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y

ZachFara and others added 4 commits November 11, 2026 21:35
A Proposed record for flagging likely-problem entries on the admin
dashboard instead of an agent-facing harmful flag: targets, labels and
their local counts, features (Commons stores no embeddings yet), where
flags act, offline and online measures, registry, drift and kill switch,
costs, and options. Recommends rules and reason categories first, then a
model per target once its labels clear a gate.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y
Rename the entry flag classifier record to the YYYY-MM-DD-<ticket>-<slug>
convention. Say that the reason category is required on the human
dashboard forms only: validate_review and the MCP reject tool stay
unchanged, agent rejects are not labels, and any change to MCP reject
needs its own record and an eval run.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y
@ZachFara
ZachFara merged commit a43a376 into feature/eval-setup Nov 12, 2026
2 checks passed
@ZachFara
ZachFara deleted the docs/MIT-648 branch November 12, 2026 02:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant