Repository navigation
MIT-648: propose an admin-only entry flag classifier - #180
Merged
Merged
Conversation
A Proposed record for flagging likely-problem entries on the admin dashboard instead of an agent-facing harmful flag: targets, labels and their local counts, features (Commons stores no embeddings yet), where flags act, offline and online measures, registry, drift and kill switch, costs, and options. Recommends rules and reason categories first, then a model per target once its labels clear a gate. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y
Rename the entry flag classifier record to the YYYY-MM-DD-<ticket>-<slug> convention. Say that the reason category is required on the human dashboard forms only: validate_review and the MCP reject tool stay unchanged, agent rejects are not labels, and any change to MCP reject needs its own record and an eval run. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Adds one Proposed decision record,
docs/decisions/2026-11-08-MIT-648-entry-flag-classifier.md. It covers a small admin-only model that flags likely-problem entries on the dashboard, as the owner's answer to item 1 of QuestionMIT-613(no agent-facingharmful=true). Docs only. No code changes.The record answers the ticket's seven questions, citing file:line for every claim about the schema and data:
It recommends B: reason categories and a rules score now, then a model per target once that target's labels clear a gate.
Two findings change the ticket's premise:
CREATE EXTENSION vector, and no embedding dependency. Decision 0025 step 3 was never built, and it left the model and dimension unchosen. The record proposes TF-IDF features first and embeddings once 0025 lands. For pending text, those embeddings must come from a local model./mine. Today no stored field says why an entry was bad. That is why the record starts with reason categories, stored apart from the reason text.human).validate_review, the MCPrejecttool (sourceagent) and the server text stay unchanged, so agent rejects stay uncategorised and are not labels.rejectwould need its own record and an eval run (decisions 0023 and 0034).Why
Ticket:
MIT-648How it was tested
commons: 7 entries and 1 human reject, but migrated only to 0006 of 21.commons_demo: 14 entries and 1 human reject.commons_eval: 0 entries.scripts/test-db.sh) and returns zeros, so it matches the current schema. Nothing touched the live database or any hosted credential.make check: passed twice in the first round, once in pre-commit and once after mergingorigin/feature/eval-setup(1613 tests). In the fix round it passed once, in pre-commit on the fix commit. The later merge oforigin/feature/eval-setup(MIT-681: kill-only same-task verdict, smoke test, store review sheet #179) changed only eval tooling and docs, and wasn't re-checked locally; CI checks it.Learned
entrieshas no vector column, and only the Postgres image ships pgvector. Any ticket that assumes stored embeddings depends on 0025 step 3 first.commons,commons_demo,commons_eval) are migrated only to 0006, so counts there say little about the current schema. Ascripts/test-db.shdatabase is the way to check that a query matches the current schema.scripts/test-db.shexportsCOMMONS_DATABASE_URLin libpqkey=valueform (make_conninfo), not as a URL. To check its host, usepsycopg.conninfo.conninfo_to_dict;urlsplitreturns no host.&&to the command that runs it. Write scratch scripts with the Write tool, then run them in a separate call.YYYY-MM-DD-<ticket>-<slug>.mdrule indocs/decisions/README.md. The rule wins, and an agent ticket's ID is allowed in the name. The structure check does not enforce this rule, so only review catches it.validate_reviewserves both the dashboard and the MCPrejecttool). A proposal that requires a new field on a review has to say which entry point gets it, or it reads as an MCP tool change.🤖 Generated with Claude Code
https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y