diff --git a/CHANGELOG.md b/CHANGELOG.md index 7dd9ab4..30d6b61 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,12 @@ All notable changes to `@fusengine/harness`. Format: [Keep a Changelog](https:// ## [Unreleased] +## [0.1.96] - 2026-09-07 + +### Added + +- **Verification receipts for 20 tools across 7 ecosystems** (`src/tracking/receipt-runners.ts`, `receipt-runners-parse.ts`, `receipt-runners-parse-php.ts`, `receipt-runners-parse-static.ts`, `src/tracking/receipt-command.ts`) — the TaskCompleted receipt gate previously only recognised `tsc` and `bun test`/`vitest`/`jest`/`npm test`, matched anywhere in the raw command, so Python, Go, Rust, PHP, Swift and Dart projects could never satisfy it and a commit message merely mentioning `jest` forged a receipt. Now a runners table covers `bun test`, `vitest`, `jest`, `npm`/`pnpm`/`yarn`/`bun run test`, `pytest`, `go test`, `cargo test`, `phpunit`, `pest`, `php artisan test`, `swift test`, `dart`/`flutter test`, plus `tsc`, `mypy`, `pyright`, `phpstan`, `go vet`, `go build`, `cargo check`, `cargo clippy`, `swift build`, each with a per-tool summary parser calibrated on real output (cargo pads `Finished` to 12 columns, `pytest -q` prints an undecorated summary, PHPUnit's OK-but banners count skipped tests in `Tests`). Commands are matched on the quote/heredoc-stripped text, anchored at command position, with the runner required to be the last command of the list; redirected or piped-away output is not evidence, no-run and informational flags never count, and a receipt now needs positive evidence — tests executed above zero with zero failures, the tool's own success line, or documented silence with intact output. A missing exit code means no capture. This is a guard against forgetting to verify and against honest shortcuts, not a security control against deliberate output fabrication. `bash-command-anchor.ts` now exports its parts (`CMD` unchanged); gate messages were generalised for the new tool set. Tests: 1779 to 1837. + ## [0.1.95] - 2026-09-06 ### Fixed diff --git a/MEMORY/LESSON.md b/MEMORY/LESSON.md index 1905332..45e1fab 100644 --- a/MEMORY/LESSON.md +++ b/MEMORY/LESSON.md @@ -22,20 +22,6 @@ - [2026-08-12 12:07] Un exécuteur s'est déclaré « idle/available » sans avoir écrit une seule ligne : le gate de fraîcheur APEX DU HARNAIS QU'ON CORRIGEAIT avait bloqué son `Write` (y compris dans le scratchpad), il avait lancé un `research-expert` pour se débloquer — et la notification de fin de CE sous-agent est remontée au LEAD, pas à son parent. L'exécuteur attendait donc un signal déjà arrivé, ailleurs. Interblocage silencieux : aucune erreur, aucun timeout, juste un agent qui ne repart jamais. Débloqué en lui renvoyant le verdict à la main. → Un « idle » sans livrable n'est pas une fin de tâche, c'est une alarme : demander l'état réel AVANT de conclure quoi que ce soit, et vérifier soi-même `git status` + le scratchpad plutôt que de croire un statut. Corollaire structurel : la notification d'un sous-agent lancé par un sous-agent remonte au lead — quand un agent délégué en spawne un autre, prévoir que c'est le lead qui recevra le signal et devra le relayer (arrivé 3 fois dans la même session ; à chaque fois le parent attendait un verdict déjà chez moi). SUITE 12:28 — l'autre bout du même tuyau : DEUX challengers d'affilée ont rédigé leur rapport en SORTIE TEXTE, qui ne remonte pas au lead. Le premier avait terminé son analyse complète depuis longtemps ; je l'ai cru muet, puis mort — `ListAgents` répondait « No reachable agents » alors qu'il a répondu normalement à la sonde suivante. → Deux règles : (a) inscrire dans le BRIEF INITIAL de tout agent que son SEUL canal de retour est `SendMessage` vers `team-lead`, sa sortie texte étant invisible ; (b) ne jamais conclure à la mort d'un agent sur `ListAgents` — le sonder par message d'abord, c'est gratuit et ça a détrompé deux fois. SUITE 12:15 — corollaire opérationnel confirmé deux fois dans la même session : un agent annoncé terminé ou « idle » peut encore écrire (cf. la leçon 12:15 : le sniper a réécrit `src/runtime/mcp-tool-name.ts` à 12:10:36 APRÈS m'avoir notifié « terminé, zéro modification »). D'où deux gardes systématiques : (a) tout mandat qui affirme « reproduit à l'instant » impose à son destinataire de re-vérifier la prémisse sur DISQUE avant sa première écriture et de s'arrêter si elle est tombée — c'est ce réflexe, et lui seul, qui a évité une écriture concurrente ce jour-là ; (b) encadrer toute mesure d'un `stat` des mtimes AVANT et APRÈS — mtimes identiques = mesure valide, sinon elle est à refaire. [TRIGGERS tool:Agent keyword:idle,available,bloqué,notification,sous-agent,attente,freshness,gate,concurrent,mtime,prémisse] -- [2026-08-03 13:29] L'issue #87 dormait OUVERTE depuis 14 jours, sans un seul commentaire : une exécution de commande arbitraire dans `harness check`, reproduite par son auteur sur `0.1.79` (`execSync(\`git show ":${path}"\`)` avec `path` = nom de fichier stagé, donc contrôlé par l'auteur du commit ; `core.quotepath` échappe `"` et `\` mais PAS les backticks ni `$(...)`). Correctif = une ligne (`execFileSync` + argv array), rapport de qualité, PR proposée par l'auteur. Pendant ces deux semaines on a publié 0.1.87 puis 0.1.88 sans jamais regarder les issues, et le README recommande justement `harness check` en pre-commit — le scénario exact de l'exploitation. → Les issues du repo font partie de l'état du projet, pas d'un canal annexe : `gh issue list --state open` AVANT toute release, et un rapport de sécurité reproduit se traite avant les fonctionnalités, pas après. Corollaire d'audit : quand une injection est trouvée à un endroit, balayer TOUS les sites de la même forme et prouver l'origine de chaque donnée interpolée (ici 9 sites revus, un seul exploitable — mais le verdict « sûr » s'établit en remontant les appelants, jamais à vue). [TRIGGERS keyword:issue,sécurité,injection,execSync,shell,release,publier,CVE] - -- [2026-08-03 12:57] J'ai annoncé au proprio « 1094 tests, 0 échec, tsc propre » puis lancé le commit — alors qu'entre-temps le code NE COMPILAIT PLUS : `tsc` sortait `handle.ts(84,5): Cannot find name 'handleConfirmSubmit'`, l'import ayant disparu quand le sniper a scindé `confirm-state.ts`. Mon vert datait d'AVANT son refactor ; je l'ai reporté comme s'il était encore vrai. Deux filets ont sauvé la mise : l'agent commit n'a rien commité, et le binaire déployé chez le proprio avait été buildé avant la casse — donc rien de cassé côté machine, mais c'est de la chance, pas de la méthode. → Un résultat de test/typecheck a une DATE : dès qu'un agent a retouché `src/` après lui, il est périmé. Re-lancer `tsc` + tests JUSTE AVANT d'annoncer un état ou de déclencher un commit, jamais citer une mesure prise avant la dernière écriture. Corollaire : après tout refactor qui déplace/scinde des modules, `tsc --noEmit` est le contrôle minimal — un import perdu ne se voit ni au diff relu ni aux tests si la suite ne couvre pas ce chemin. SUITE 2026-08-12 12:07 : la règle vaut AUSSI pour les ARTEFACTS DE PREUVE, pas seulement pour tsc/tests. Un exécuteur a livré une baseline de caractérisation « après » comme pièce maîtresse de non-régression — générée à 12:01:51, alors qu'il avait retouché le module à 12:02:54. Il avait consciencieusement relancé tsc et les tests après sa dernière écriture, mais pas sa propre preuve. Attrapé en comparant les mtimes de l'artefact et des fichiers source (`stat -f "%Sm %N"`), puis régénéré : sha256 identique, donc conclusion inchangée — mais elle reposait sur de la chance, pas sur une mesure valide. → Un artefact de preuve (baseline, snapshot, capture différentielle) se date et se re-génère comme un test. Contrôle systématique avant d'accepter une preuve d'un agent : `stat` l'artefact vs les fichiers qu'il prétend caractériser, et le régénérer soi-même si l'ordre ne tient pas. Un agent qui relance ses tests mais pas sa preuve croit sincèrement être à jour. SUITE 12:50 — variante bénigne mais répétée deux fois le même jour : j'ai rapporté « 1107 pass » au proprio alors que la sortie disait `1106 pass / 1 skip / Ran 1107 tests`. `Ran N` compte les tests EXÉCUTÉS (pass + skip), pas les réussis. Sans conséquence ici, mais c'est un chiffre qu'on cite au proprio à chaque livraison. → Lire la ligne de résumé `bun test` en entier et citer `pass` et `skip` séparément, jamais le total de `Ran`. Un chiffre de livraison se recopie depuis la sortie, il ne se reconstitue pas de mémoire. [TRIGGERS keyword:tsc,typecheck,vert,tests,commit,refactor,scinder,import,périmé,baseline,snapshot,preuve,mtime] - -- [2026-08-03 11:48] J'ai lancé sniper (sur `src/`+`test/`) et un agent doc (sur `*.md`) EN PARALLÈLE, périmètres de fichiers strictement disjoints — donc aucun conflit d'écriture, et sur ce plan ça a marché. Mais le sniper a SCINDÉ `confirm-state.ts` en cours de route pendant que l'agent doc écrivait `docs/config.md` et `docs/adapters.md` avec des chemins pointant vers l'ancien fichier : deux livrables cohérents chacun, incohérents ENSEMBLE. Rattrapé de justesse par un grep croisé avant le déploiement. → Des périmètres disjoints protègent des collisions d'écriture, PAS des dépendances sémantiques : quand un agent documente/référence ce qu'un autre restructure, la doc doit passer APRÈS le refactor, ou le refactoreur doit avoir interdiction de renommer/déplacer. Contrôle systématique avant de conclure un lot parallèle : grep des chemins/symboles cités par l'un dans ce que l'autre a produit. [TRIGGERS tool:Agent keyword:parallèle,disjoint,périmètre,doc,refactor,scinder,renommer,chemin] - -- [2026-08-03 11:42] Le challenger a REFUTÉ « un agent ne peut pas s'auto-approuver » sur le mécanisme `CONFIRM`, avec un bypass reproduit sur le binaire déployé : `consumeConfirmToken` ne consulte JAMAIS G0 (G0 ne vit qu'à la POSE du jeton), donc l'agent saute l'étape humaine — il calcule lui-même le hash (algo en clair dans le repo), écrit `{confirmToken}` dans `~/.fuse-harness/cache/sessions/*.json` via un `node -e`, et rejoue l'action : autorisée. Le seul rempart, `SESSION_STATE_FRAGMENT`, est un match de sous-chaîne littérale, évité par `"cache/ses" + "sions"` — un filtre lexical, pas un contrôle d'intégrité. Constat de fond : TOUT gate dont l'état vit dans un fichier que l'agent peut écrire est un garde-fou de DISCIPLINE (contre l'erreur, l'agent coopératif), jamais un contrôle de SÉCURITÉ — le harnais n'est pas une sandbox, et ça vaut aussi pour le pipeline design et la fraîcheur APEX. → Ne jamais présenter un tel mécanisme comme une barrière : documenter noir sur blanc ce qu'il garantit et ce qu'il ne garantit pas, et vérifier que le garde est évalué à CHAQUE point qui accorde le privilège (poser ET consommer), pas seulement au premier auquel on a pensé. Corollaire attrapé par le sniper dans le même lot : sur trois points d'entrée ajoutés à des chemins de hook, un seul avait son `try/catch` — une exception dans un hook fait tomber le harnais ENTIER, donc la protection se met sur tous les points d'entrée d'un coup, pas sur celui qu'on écrit en dernier. [TRIGGERS keyword:auto-approbation,bypass,jeton,token,état,sandbox,sécurité,discipline,try/catch,hook] - -- [2026-08-03 11:12] G0 (le garde-fou qui interdit de poser un jeton `CONFIRM` pendant qu'un sous-agent tourne) a d'abord été un compteur `depth ± 1` : un `SubagentStop` dupliqué par le fan-out multi-plugins le fait tomber à 0 pendant qu'un agent tourne encore, et le garde-fou S'OUVRE. Remplacé par une écriture monotone `seenAt = Math.max(prev, now)` — idempotente, sans décrément, donc rien à désynchroniser. Correction réglée. MAIS l'exécuteur a fixé la fenêtre à 30 min, et ça rend le mécanisme inutilisable pour un proprio qui lance des agents en continu : le moindre sous-agent gèle toute confirmation une demi-heure. Le piège de fond : sans décrément, la fenêtre doit couvrir la DURÉE DE VIE du sous-agent, pas la latence d'écriture — d'où une valeur énorme, techniquement juste et pratiquement morte. → Un garde-fou se valide sur DEUX axes, jamais un seul : (1) ne peut-il jamais s'ouvrir à tort, (2) reste-t-il utilisable dans le rythme réel du proprio. Quand la sûreté n'est atteinte qu'au prix d'un paramètre qui neutralise la fonctionnalité, ce n'est pas un réglage à choisir seul — remonter l'arbitrage, ou supprimer le paramètre (ici : deux compteurs croissants `starts`/`stops`, actif tant que `starts > stops`, aucune fenêtre — écarté finalement, car un fan-out asymétrique N≠M gèlerait le mécanisme POUR TOUJOURS, pire que la fenêtre). SUITE 11:18 : j'allais faire écrire la nouvelle valeur en constante nue ; le proprio a demandé « ça doit être basé sur notre système TTL non ? » — et il avait raison, `src/config/ttl.ts` expose déjà `resolveTtlSec(env, key)` + `ttlLabel()`. → Avant d'introduire le moindre paramètre de durée/seuil, chercher l'infrastructure de configuration DÉJÀ présente et y brancher une clé DÉDIÉE : réutiliser le helper (DRY, réglable sans recompiler) sans réutiliser la clé d'un autre besoin (`FUSE_ENFORCE_TTL_SEC` gouverne la fraîcheur des preuves — coupler les deux ferait qu'allonger le TTL de recherche allongerait le gel des confirmations). SUITE 11:24 : j'avais prescrit `resolveTtlSec(env, "MA_CLE")` en croyant le helper paramétrable — il ne l'est qu'à moitié, son fallback est figé sur `DEFAULT_TTL_SEC = 120` quelle que soit la clé passée, donc un défaut de 300 était inexprimable. L'exécuteur est descendu d'un cran sur `parseEnvInt` (le primitif que `resolveTtlSec` utilise lui-même) et l'a documenté. → Avant de prescrire la réutilisation d'un helper, LIRE son corps et pas seulement sa signature : un paramètre exposé ne garantit pas que tout le comportement suit (ici la clé est paramétrable, le défaut non). Le bon repli est le primitif sous-jacent, jamais une réécriture parallèle. [TRIGGERS keyword:garde-fou,fail-closed,fenêtre,window,compteur,monotone,sous-agent,G0,inutilisable] - -- [2026-08-03 10:53] Sur le point [8] (confirmation `CONFIRM `), j'ai enchaîné les mesures — `ask` sous Kimi, `systemMessage` sous Codex, payloads, P0 — sans jamais rien construire, jusqu'à ce que le proprio explose : « on est dans du code, c'est un hack à faire, simple ». Il a nommé la bonne étape que je ne voyais pas : un PROTOTYPE hors harnais, branché dans son vrai Codex. Écrit en 20 min, il a prouvé le mécanisme en conditions réelles (deny → l'humain tape le code → allow → fichier créé), et il a révélé ce qu'aucune mesure n'aurait donné : Codex utilise `apply_patch`, pas `touch`, donc le gate DOIT juger le contenu de l'action et jamais le nom d'outil. Bonus : le prototype a exposé un défaut de conception avant qu'il n'entre dans le harnais — vérifier le jeton sur le code court de 4 hex (16 bits) autorise une action qui COLLISIONNE ; le jeton doit porter le hash complet, les 4 caractères ne sont qu'un élément d'interface. → Quand une fonctionnalité est bloquée par une chaîne de mesures qui n'en finit pas, arrêter de mesurer et écrire un prototype JETABLE hors du système à modifier : il tranche plus vite, il se teste en vrai, et il fait tomber les défauts de design gratuitement. Signal d'alarme à s'appliquer : trois tours sans une ligne de code livrée = changer de mode. Corollaire vu au portage (11:04) : quand un chemin de code ne propage pas `home`, un témoin écrit dans le HOME RÉEL et y laisse des fichiers d'état orphelins (`~/.fuse-harness/cache/sessions/session--*.json`) — un test doit soit isoler son HOME, soit nettoyer ce qu'il crée, jamais laisser des résidus sur la machine du proprio. [TRIGGERS keyword:mesure,prototype,hack,simple,tourner en rond,tranche,confirmation,CONFIRM,orphelin,HOME] - -- [2026-08-03 08:57] Un exécuteur a livré un contre-témoin « avant/après » concluant `IDENTICAL` sur `node dist/cli/bin.mjs hook ` — sauf que le harnais DÉDUPLIQUE : deux invocations successives dans le MÊME cwd renvoient 11212 octets puis **0 octet** (vérifié : même avec des `session_id` différents, c'est le cwd qui porte la dédup). Son `diff` comparait donc très probablement du vide à du vide. Refait avec un cwd NEUF par capture : 11212 (claude-code) et 19983 (codex) octets, identiques avant/après — la conclusion tenait, la preuve non. → Un témoin différentiel doit d'abord prouver qu'il MESURE quelque chose : exiger la TAILLE non nulle de chaque capture avant de lire le `diff`, et isoler l'état (cwd/session neufs) entre les deux branches. Un `diff` vide sur deux sorties vides est le faux vert le plus facile à produire et le plus difficile à voir. [TRIGGERS keyword:contre-témoin,avant/après,differential,IDENTICAL,diff,capture,dédup,burst] - - [2026-09-02 12:11] Un mandat détaillé (RED commands, « 25 fichiers modifiés », « version 0.1.90 ») décrivait un état ANTÉRIEUR à la PR #100 : arbre propre, 0.1.91 déjà publiée, les 3 RED déjà verts. Re-mesurer git status + version npm + chaque RED command AVANT le moindre brief a évité de relancer 7 « points restants » déjà faits. → Un mandat écrit n'est pas une mesure : rejouer ses commandes de preuve sur le HEAD courant d'abord, et re-baseliner le périmètre sur l'écart réel. [TRIGGERS keyword:mandat,RED,re-baseline,déjà fait,périmètre,prompt obsolète] - [2026-09-02 12:11] Payloads Cursor AUTHENTIQUES : Cursor journalise chaque exécution de hook (INPUT JSON complet + OUTPUT + diagnostics) dans `~/Library/Application Support/Cursor/logs/**/cursor.hooks*.log` ; le runtime réel des hooks est dans le worker `~/Library/Application Support/Cursor/User/globalStorage/anysphere.cursor-agent-worker/agent-cli/.local/share/cursor-agent/versions//{index.js,190.index.js}`, pas seulement dans Cursor.app. Piège : `find | xargs grep` casse sur l'espace de « Application Support » (résultat vide silencieux) → `-print0 | xargs -0` ou glob Python. [TRIGGERS keyword:cursor,hooks log,payload authentique,capture,agent-worker,Application Support,xargs] @@ -105,3 +91,17 @@ - [2026-09-06 22:38] Sept Edit envoyés d'un coup à 22:37 alors que la fenêtre de fraîcheur APEX (ouverte ~22:20) venait d'expirer : le premier est passé, les six autres bloqués → `integrity.ts` laissé CASSÉ (déclarations remplacées, usages orphelins) le temps de repayer le garde. → Avant un lot d'éditions, vérifier `date` contre l'heure d'ouverture de la fenêtre ; s'il reste moins de 3 minutes, relancer research+explore D'ABORD, puis éditer. Et ordonner le lot pour que chaque édition laisse le fichier compilable seule (ajouter les nouvelles définitions AVANT de retirer les anciennes). [TRIGGERS tool:Edit error:APEX keyword:fraîcheur,15min,lot,replace_all] - [2026-09-06 22:47] Rouge CI Linux uniquement, jamais reproductible sur macOS, apparu avec une nouvelle image de runner : trois tests laissaient `process.env.HOME` sur un tmp sans le restaurer ; `integrity.ts` figeait `homedir()` au premier import ; le fils `bun -e` héritait du HOME muté → clé HMAC différente → événements rejetés en silence (append fail-open). Invisible sur macOS car Bun y ignore un HOME muté dans `homedir()` ; l'ordre des fichiers `bun test` est l'ordre `readdir`, qui change avec chaque nouveau système de fichiers d'image. J'ai perdu 40 min sur Bun 1.4.0→1.4.2 (le pin n'a rien changé) avant de lire l'ordre réel des fichiers dans le log. → Un rouge « dépendant de l'environnement » se lit d'abord comme un problème d'ORDRE et d'ÉTAT PARTAGÉ : extraire l'ordre des fichiers du log CI, chercher qui mute `process.env` sans restaurer, et bannir toute capture de HOME au niveau module (résolution à l'appel). Tout test qui touche `process.env.HOME` restaure dans `afterAll`. [TRIGGERS keyword:CI,Linux,homedir,process.env.HOME,readdir,ordre,flaky,image runner,integrity,HMAC] + +- [2026-09-06 23:20] Fumée apply_patch « grow » rejouée à la main depuis la racine du dépôt : fixture sans `cwd`, cible `big.ts` relative, fichier absent → stdout vide, indiscernable d'un « allow ». Elle n'était valide dans la chaîne que parce que la chaîne se plaçait dans un dossier contenant `big.ts`. → Une fumée qui dépend du cwd n'est pas une fumée : le script pose lui-même sa fixture dans un `mktemp -d` et s'y place ; et tout témoin « allow = sortie vide » est accompagné du témoin « deny » sur le même état, sinon la sortie vide ne prouve rien. [TRIGGERS keyword:fumée,smoke,fixture,cwd,stdout vide,apply_patch,témoin] + +- [2026-09-07 14:24] Même avec un brief disant « lance les 2 agents, attends-les, puis édite », l'exécuteur `typescript-expert` a rendu la main sur « j'attends mes agents » sans écrire, et SendMessage n'est pas disponible pour le réveiller. Le relancement qui a marché : garde payé PAR LE LEAD juste avant (research + explore terminés), brief « AUCUN sous-agent, commence à éditer dans les 2 minutes, si bloqué lis les refs SOLID et réessaie ». → Ne jamais déléguer le paiement du garde APEX à l'exécuteur : le payer soi-même, puis lancer l'exécuteur avec interdiction de sous-agents et délai d'édition explicite. [TRIGGERS keyword:typescript-expert,j'attends,waiting for,sous-agents,garde APEX,fraîcheur] + +- [2026-09-07 14:50] J'ai fait réutiliser l'ancre `CMD` du garde d'écriture pour reconnaître les reçus de vérification : elle sur-apparie (séparateurs `\n` et backtick) parce que pour un garde, un faux positif ne coûte qu'un blocage. Pour un REÇU, un faux positif est une fausse preuve de succès : un corps de heredoc contenant « bun test » satisfaisait le garde de clôture. Le challenger l'a vu, pas moi ni le sniper. → Avant de réutiliser un motif entre deux gardes, vérifier la POLARITÉ de l'erreur : un motif conçu fail-closed (sur-apparier = bloquer) devient fail-open s'il sert à prouver quelque chose. Et toute preuve issue d'une commande passe par le texte déquoté (lexer), jamais par la chaîne brute. [TRIGGERS keyword:CMD,ancre,reçu,receipt,polarité,fail-open,fail-closed,réutiliser,DRY] + +- [2026-09-07 14:59] Deux passes de « durcissement » des reçus refusées de suite par le challenger : j'ai corrigé les cas trouvés (heredoc, drapeaux sans exécution, masquage du code de sortie) au lieu de la règle fautive : `isPassing` acceptait « rien observé » comme preuve, donc un simple `--version` ou une sortie supprimée restaient verts. → Pour un mécanisme de PREUVE, formuler d'abord l'invariant positif (« un reçu exige une preuve observée : compte de tests supérieur à zéro ou ligne de succès de l'outil ») puis dériver les cas ; une liste noire de drapeaux ne fait que déplacer la faille. Lancer le challenger sur l'invariant, pas sur la liste. [TRIGGERS keyword:receipt,reçu,isPassing,preuve,liste noire,--version,challenger REFUTED] + +- [2026-09-07 15:25] Troisième REFUTED d'affilée sur les reçus : sans modèle de menace écrit, chaque passe corrige les sondes de la précédente et le challenger en trouve d'autres (un `; echo` final forge un vert). Un agent avec un shell peut toujours fabriquer une sortie : sans périmètre déclaré, la boucle n'a pas de fin. → Avant la première passe d'un mécanisme de preuve, écrire le modèle de menace (ici : erreur ou raccourci honnête, pas agent adverse), en dériver les règles STRUCTURELLES (runner en dernière position de la liste, sortie non redirigée, résumé lu depuis la sortie de l'outil) et briefer le challenger sur ce périmètre. Trois passes en une après-midi valent moins qu'une passe cadrée. [TRIGGERS keyword:modèle de menace,threat model,challenger REFUTED,boucle,preuve,reçu] + +- [2026-09-07 16:16] J'ai affirmé au propriétaire que ses agents tournaient sur le modèle de session, déduit de l'absence de champ `model` dans les définitions lues. Faux : les transcriptions réelles (`"model":` dans tasks/*.output) montrent Sonnet pour les exécuteurs et Opus pour le challenger. Même faute de méthode que les reçus : conclure de l'absence d'une trace au lieu de lire la mesure disponible. → Toute question « quel modèle / quelle version / quel binaire a tourné » se répond dans la transcription ou le log d'exécution, jamais par déduction depuis la configuration. Et les parseurs de sortie d'outil se calibrent sur des captures RÉELLES (cargo aligne `Finished` à 12 colonnes, `pytest -q` n'a pas de `=`, PHPUnit dit « OK, but… »), pas sur des exemples de doc. [TRIGGERS keyword:quel modèle,sonnet,opus,transcription,capture réelle,format de sortie,parseur] + +- [2026-09-07 16:27] Dans le brief de la 5e passe j'ai dicté moi-même la regex des options (`(?:--?[\w-]+(?:[= ]\S+)?\s+)*`) en affirmant « pas de forme (X*)* » : elle avait DEUX ambiguïtés (`--?` contre `[\w-]+` sur le tiret, valeur `[= ]\S+` avalant le drapeau suivant) → 17,8 s sur 9 000 caractères, trouvé par le sniper au chronomètre. → Une regex répétée sur des tokens reçoit toujours un test de temps (entrée hostile de 10 000 caractères sans correspondance finale, budget < 50 ms) dans les tests, pas une assertion verbale dans le brief ; et chaque alternance doit rendre le premier caractère de chaque token non ambigu (`--?[\w][\w-]*`, valeur `(?!-)`). [TRIGGERS keyword:regex,ReDoS,backtracking,quantificateur,options,PREFIX,OPTS] diff --git a/README.md b/README.md index 49d56a9..6d5ed90 100644 --- a/README.md +++ b/README.md @@ -198,9 +198,45 @@ Features shipped since 0.1.44, each with its own test: - **One-shot gate metric** — every gate outcome (deny or its later fix) lands in a 7-day sidecar keyed by a content-free op hash, so a deny→allow transition is visible (`src/tracking/one-shot.ts`, `test/one-shot.test.ts`). -- **Verification receipts** — a `tsc`/test run is captured from PostToolUse Bash - output; `TaskCompleted` **refuses** a "done" over modified code files without a - fresh passing receipt (`src/tracking/receipts.ts`, `test/receipts.test.ts`). +- **Verification receipts** — a static-check or test run is captured from + PostToolUse Bash output, spanning TS/JS (`tsc`, `bun test`/`vitest`/`jest`/ + `npm|pnpm|yarn test`), Python (`mypy`, `pyright`, `pytest`), Go (`go vet`/ + `build`/`test`), Rust (`cargo check`/`clippy`/`test`), PHP (`phpstan`, + `phpunit`/`pest`/`php artisan test`), Swift (`swift build`/`test`), and Dart/ + Flutter (`dart`/`flutter test`); `TaskCompleted` **refuses** a "done" over + modified code files without a fresh passing receipt. Commands are matched + after quote/heredoc stripping (a tool name mentioned in a commit message or + heredoc body is never a receipt) — and the recognised runner must be the + LAST command of the line: the unquoted text is split into shell list + segments on `;`/`&&`/`||`/a lone `&`/newline (never a bare pipe, which still + carries the runner's real output forward), and only the LAST segment is + searched, so `bun test && git commit -m x` and `bun test || true` both null + while `cd x && bun test` and `bun test 2>&1 | tail -5` both still resolve. A + zero-test invocation (`--watch`, `--collect-only`, `0 passed; 0 failed`) is + never treated as proof, and static checkers (`tsc`, `go vet`/`build`, `cargo + check`/`clippy`, `swift build`) parse their OWN diagnostics so a masked exit + code doesn't hide a real failure. A residual trade-off: a truncating pipe + (`cargo test --no-fail-fast 2>&1 | tail -5`) can still cut away a failing + target's own `test result: FAILED` line — `cargo test`'s parser also counts + cargo's own trailing `error: N targets failed` line as a fail floor, but a + pipe that removes ALL evidence remains an accepted risk, in exchange for + correctly resolving genuine `| tail`/`| grep` pipelines instead of nulling + every piped command. A receipt needs POSITIVE evidence (a + parsed summary or the tool's success line): output suppression + (`2>/dev/null`, ANY output redirection after the runner in its own + segment — not just `/dev/null` — for a static checker, a stdout redirect for + a test runner, or a pipe after a static checker), `--version`/`--help`/other + no-run flags, and silent/ambiguous output never count. **Threat model**: a + receipt guards against forgetting to verify and against honest shortcuts (a + discarded exit code, an informational invocation, a filtered run) — it is + NOT a security control against an agent that deliberately fabricates tool + output, same caveat as the `CONFIRM` code above. + (`src/tracking/receipt-runners.ts`, `src/tracking/receipt-command.ts`, + `src/tracking/receipts.ts`, `test/receipt-runners.test.ts`, + `test/receipts.test.ts`, `test/receipt-runners-forgery.test.ts`, + `test/receipt-runners-hardening.test.ts`, + `test/receipt-runners-evidence.test.ts`, + `test/receipt-runners-structural.test.ts`). - **Decision-time lessons** — a `MEMORY/LESSON.md` bullet tagged with `[TRIGGERS tool:… path:… error:… keyword:…]` is injected as `additionalContext` the moment a matching call is about to repeat a known mistake, cooldown-guarded diff --git a/package.json b/package.json index 714c74f..7d64e1f 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@fusengine/harness", - "version": "0.1.95", + "version": "0.1.96", "description": "Harness-agnostic toolkit for AI coding agents: runtime harness detection (Claude Code, Codex, Cursor, Cline, Gemini, Aider...), pure policy core (env config, project/framework detection, SOLID/file-size limits, APEX freshness, guard patterns, portable prompts), cache, project memory, ref routing, state/locks, statusline, per-harness adapters (Claude/Cursor/Cline/Gemini) and a cli-mode harness-check binary. Bun-native, with a built dist for Node + bundlers.", "type": "module", "module": "src/index.ts", diff --git a/src/policy/guards/bash-command-anchor.ts b/src/policy/guards/bash-command-anchor.ts index 05bfcd0..c8cfec8 100644 --- a/src/policy/guards/bash-command-anchor.ts +++ b/src/policy/guards/bash-command-anchor.ts @@ -2,6 +2,12 @@ * Command-position anchoring for the Bash write guard — the regex prefix that * tells a real mutator invocation apart from a quoted/argument mention of the * same token. Consumed by bash-write-patterns.ts (CODE_MUTATORS, CODE_COMMAND_WRITE). + * {@link WRAP}/{@link WRAP_ARG}/{@link ENV_PREFIX} are also exported for + * `src/tracking/receipt-runners.ts`, which composes its OWN anchor from the + * same parts minus the backtick separator — a backtick never actually starts + * a new command (it opens command substitution), so over-matching it as one + * is safe for a write guard (defense-in-depth) but forges a false PROOF of + * success for a verification receipt. */ /** @@ -9,7 +15,7 @@ * (`env sed -i …`, `timeout 5 patch …`, `sudo -n tee …`, `xargs sed -i`). A code * mutator behind one is still a code mutator — the wrapper is not a shield. */ -const WRAP = "(?:env|timeout|nice|nohup|sudo|command|stdbuf|time|ionice|exec|xargs)"; +export const WRAP = "(?:env|timeout|nice|nohup|sudo|command|stdbuf|time|ionice|exec|xargs)"; /** * A wrapper's own leading arg tokens: `VAR=val` assignments, `-f`/`--flag` @@ -19,7 +25,7 @@ const WRAP = "(?:env|timeout|nice|nohup|sudo|command|stdbuf|time|ionice|exec|xar * alternative consumes ≥1 char and is followed by a mandatory `\s+`, so the * repeated group is never zero-width (no `(X*)*` catastrophic shape). */ -const WRAP_ARG = "(?:\\w+=\\S+|--?[^\\s-]\\S*|\\d+[smhd]?)"; +export const WRAP_ARG = "(?:\\w+=\\S+|--?[^\\s-]\\S*|\\d+[smhd]?)"; /** * Command-position anchor: line/segment start or a `;&|(` \` separator, then any @@ -49,6 +55,6 @@ const WRAP_ARG = "(?:\\w+=\\S+|--?[^\\s-]\\S*|\\d+[smhd]?)"; * match a mutator/write pattern (recognizing a previously-invisible wrapped * invocation); it can never turn an existing block/ask into an allow. */ -const ENV_PREFIX = "(?:\\w+=\\S*\\s+)*"; +export const ENV_PREFIX = "(?:\\w+=\\S*\\s+)*"; export const CMD: string = `(?:^|[\\n;&|(\x60])\\s*${ENV_PREFIX}(?:${WRAP}\\s+(?:${WRAP_ARG}\\s+)*)*`; diff --git a/src/runtime/handle-post.ts b/src/runtime/handle-post.ts index eb35c5c..308c61f 100644 --- a/src/runtime/handle-post.ts +++ b/src/runtime/handle-post.ts @@ -68,8 +68,9 @@ export async function handlePost(ctx: PreContext): Promise { // Codex multi_agent_v2 `spawn_agent` -> same session track (no-op for every // other harness / non-spawn tool / missing `agent_type`; see module doc). await recordCodexSpawnEvidence(file, id, event.tool, event.input, opts.now); - // Verification receipts (tsc/bun test runs) — structured responses only - // (Kimi's string `tool_output` would forge a success receipt; see module). + // Verification receipts (static-check/test runs, see receipt-runners) — + // structured responses only (Kimi's string `tool_output` would forge a + // success receipt; see module). await captureBashReceipt(file, event.tool, event.command, payload.tool_result, response, opts.now); if (id === "codex") recordCodexPostFailure(event.tool, payload.tool_result ?? response, { now: opts.now, dir: defaultStateDir(opts.cwd), sessionId: event.sessionId }); // Codex `apply_patch` and Cursor `afterFileEdit` fan into per-file events for diff --git a/src/runtime/lifecycle/agent-memory.ts b/src/runtime/lifecycle/agent-memory.ts index 2fd6483..26c1bbd 100644 --- a/src/runtime/lifecycle/agent-memory.ts +++ b/src/runtime/lifecycle/agent-memory.ts @@ -73,7 +73,7 @@ export function trackAgentMemory(data: Record, home: string = h // Window = TTL×5, matching the TaskCompleted receipt gate. const windowMs = resolveTtlSec(process.env) * 1000 * 5; const noReceipt = freshReceiptFromFile(trackFile(sessionId, defaultStateDir(process.cwd())), windowMs, now) === null; - const note = noReceipt ? " NO VERIFICATION RECEIPT — run tsc + tests before reporting done." : ""; + const note = noReceipt ? " NO VERIFICATION RECEIPT — run your static checker + test suite before reporting done." : ""; return contextResponse("SubagentStop", `SNIPER VALIDATION REQUIRED: Agent '${agentType}' modified ${present.length} code file(s): ${present.join(", ")}. Run sniper agent now.${note}`); } } diff --git a/src/runtime/lifecycle/task-completed.ts b/src/runtime/lifecycle/task-completed.ts index e472064..f5cfdb4 100644 --- a/src/runtime/lifecycle/task-completed.ts +++ b/src/runtime/lifecycle/task-completed.ts @@ -42,8 +42,10 @@ function receiptGate(sid: string, files: string[], now: number, stateDir: string const windowMs = resolveTtlSec(process.env) * 1000 * RECEIPT_TTL_MULTIPLIER; if (freshReceiptFromFile(trackFile(sid, stateDir), windowMs, now)) return null; const stopReason = - "VERIFICATION RECEIPT REQUIRED: code files changed but no fresh passing tsc/test receipt " + - "exists. Run `bun test` + `tsc --noEmit` (exit 0, 0 failures) and re-complete."; + "VERIFICATION RECEIPT REQUIRED: code files changed but no fresh passing verification receipt " + + "exists. Run your test suite and static checker (bun test + tsc, pytest + mypy, go test + go vet, " + + "cargo test + cargo check, phpunit/pest + phpstan, swift test, dart test) with exit 0 and 0 failures, " + + "then re-complete."; return JSON.stringify({ continue: false, stopReason }); } diff --git a/src/runtime/receipt-capture.ts b/src/runtime/receipt-capture.ts index 109086b..a521e3d 100644 --- a/src/runtime/receipt-capture.ts +++ b/src/runtime/receipt-capture.ts @@ -1,7 +1,7 @@ /** - * Bash verification-receipt extraction for PostToolUse: a `tsc`/`bun test` - * run is parsed (exit code + output) into a signed receipt the TaskCompleted - * gate later demands. Structured responses ONLY: Kimi Code CLI sends + * Bash verification-receipt extraction for PostToolUse: a static-check/test + * run (see `receipt-runners`) is parsed (exit code + output) into a signed + * receipt the TaskCompleted gate later demands. Structured responses ONLY: Kimi Code CLI sends * `tool_output` as a truncated STRING (no exit_code, no stdout/stderr * channels) — parsing it would forge a success receipt (`exit 0`, empty * output) for possibly-failed runs, so a non-object source is skipped. @@ -28,7 +28,10 @@ export async function captureBashReceipt( if (tool !== "Bash" || !command) return; const r = (toolResult ?? response) as { exit_code?: unknown; stdout?: unknown; stderr?: unknown } | undefined; if (!r || typeof r !== "object") return; + // D4: a missing/non-finite exit_code is NOT the same as exit 0 — crediting + // it would forge a passing receipt for a response shape that never + // actually reported success. Skip capture entirely rather than guess. + if (typeof r.exit_code !== "number" || !Number.isFinite(r.exit_code)) return; const out = `${typeof r.stdout === "string" ? r.stdout : ""}\n${typeof r.stderr === "string" ? r.stderr : ""}`; - const exit = Number(r.exit_code ?? 0); - await captureReceipt(file, command, out, Number.isFinite(exit) ? exit : 0, now); + await captureReceipt(file, command, out, r.exit_code, now); } diff --git a/src/tracking/receipt-command.ts b/src/tracking/receipt-command.ts new file mode 100644 index 0000000..8517d97 --- /dev/null +++ b/src/tracking/receipt-command.ts @@ -0,0 +1,130 @@ +/** + * @module receipt-command + * Command-SHAPE rules for {@link classifyReceipt} (`./receipts`): which + * command text is even eligible to be scored as a receipt at all, and which + * runner (if any) it resolves to. Split out of `./receipts` to keep it under + * the SOLID line ceiling. + * + * Threat model (explicit): these rules guard against FORGETTING to verify + * and against honest shortcuts — a run whose output/exit code was + * accidentally discarded, an informational invocation, a filtered run. They + * are NOT a security control against an agent that deliberately fabricates + * tool output, same caveat as the `CONFIRM` code in the README ("a guard + * against accidental/hasty …, not a security control against an adversarial + * agent"). + * + * Structural rule (replaces the old `EXIT_MASK_RE` blacklist): the recognised + * runner must be the LAST command of the line — {@link lastListSegment} + * splits the unquoted command on `;`, `&&`, `||`, a lone job-control `&`, and + * newline (never a bare pipe, which still carries the runner's real output + * forward) and only the LAST segment is searched. `bun test && git commit -m + * x` and `bun test || true` both null: the runner sits in an EARLIER segment + * only. `cd x && bun test` and `bun test 2>&1 | tail -5` both still resolve: + * the runner is in (or is) the last segment either way. + */ +import type { RunnerSpec } from "./receipts"; +import { RUNNERS } from "./receipt-runners"; + +/** + * A flag/subcommand that runs zero (or an unbounded, still-running) tests, or + * is purely INFORMATIONAL (version/help/config banner, no real job done): + * `--collect-only`/`--co` (pytest), `--dry-run`, `--list`/`--listTests`/ + * `--list-tests` (jest/swift/dart), `--watch`/`--watchAll`/`--watch-all` + * (bun/vitest/jest never exit), `--no-run` (cargo test), `--version`/`-V`, + * `--help`/`-h`, `--init`/`--showConfig` (jest), `--fixtures` (pytest), + * `-count=0` (go test), `go build -n` (dry-run). `--skip-build` (swift test) + * is NOT here: it skips the BUILD step only, the tests still run — treating + * it as informational was a false negative (real defect, fixed). + * + * `(?!=false)` excludes `--watch=false`/`--watchAll=false`/`--watch-all=false` + * — Vitest's own config docs list `--watch=false` alongside `-w`/`--watch` as + * an explicit CLI form (the option defaults to `true`), and Jest's + * `--watchAll[=false]` is the documented way to force run-once mode on a + * boolean CLI flag. + */ +export const NO_RUN_RE: RegExp = + /(?:^|\s)--?(?:no-run|collect-only|co|watch(?:All|-all)?(?!=false)|listTests|list-tests|list|dry-run|version|V|help|h|init|showConfig|fixtures)\b|\btest:watch\b|(?:^|\s)-count=0\b|\bgo\s+build\s+-n\b/; + +/** `>/dev/null` in any of its `>`/`2>`/`&>` forms — discarding a test + * runner's summary or a static checker's diagnostics destroys the only + * proof a receipt can offer, on either stream, regardless of kind. */ +const DEV_NULL_RE = />\s*\/dev\/null/; + +/** A single pipe (`|`, never `||`) — invalidates a STATIC-kind runner's + * "silence proves success" evidence (a pipe stage can reorder, truncate, or + * recount diagnostics before they reach the receipt). Harmless for a + * TEST-kind runner, whose counts come from parsing the piped-through text. */ +const PIPE_RE = /(?&1`, `1>&2`, …): a bare `>`/`>>`, an fd-qualified `2>`/`2>>`, or the + * combined `&>`/`&>>` form. A STATIC-kind runner's only proof is silence — + * redirecting ANY stream after it, to ANY target (not just `/dev/null`), + * destroys that proof exactly like a pipe does. Subsumes the old `>&-` + * (fd-close) check: `(?!&\d)` only excludes a DIGIT target, so `>&-` still + * matches this. + */ +const REDIRECT_RE = /\d*>{1,2}(?!&\d)|&>{1,2}/; + +/** + * A STDOUT-directed redirect only (`>`/`>>` with no leading fd number, or the + * combined `&>`/`&>>` form) — a TEST-kind runner's counts live in whatever + * the harness captured as its output; redirecting that away destroys the + * evidence exactly like a `/dev/null` redirect does. A bare `2>` (stderr-only, + * non-`/dev/null`) is deliberately left alone: {@link DEV_NULL_RE} already + * covers the one case (any stream to `/dev/null`) that matters regardless of + * which fd a given runner actually writes its summary to, and `2>&1` (fd + * duplication, no stream lost) must stay allowed. + */ +const TEST_REDIRECT_RE = /(?{1,2}(?!&\d)|&>{1,2}/; + +/** Shell LIST separators this rule splits on: `;`, `&&`, `||`, a lone + * job-control `&` (never part of `&&`, nor a `>&`/`&>` redirect form like + * `2>&1`/`>&-`/`&>file` — the lookaround excludes both), or a newline. A + * bare pipe `|` is deliberately absent: it still delivers the runner's real + * output into the next stage, so it never starts a new list segment here. */ +const LIST_SEP_RE = /&&|\|\||;|(?)&(?!>)|\n/g; + +/** The LAST shell list segment of `unquoted` (see module doc): the only + * segment a recognised runner is allowed to be found in. */ +export function lastListSegment(unquoted: string): string { + const parts = unquoted.split(LIST_SEP_RE); + return (parts[parts.length - 1] ?? unquoted).trim(); +} + +/** Strips ANSI SGR escapes (`\x1b[...m`) and `\r` from a captured output + * once, before any pass/fail parser runs — a colorized `bun test` summary + * (`\x1b[32m 12 pass\x1b[0m`) must parse identically to a plain one. */ +export function stripAnsi(output: string): string { + return output.replace(/\x1b\[[0-9;]*[A-Za-z]/g, "").replace(/\r/g, ""); +} + +/** Whether output suppression makes a matched runner's evidence untrustworthy, + * scoped to `tail` — the text of its OWN list segment AFTER the runner + * match (never earlier text, which belongs to a prior command). */ +function isSuppressed(tail: string, kind: "tsc" | "test"): boolean { + if (DEV_NULL_RE.test(tail)) return true; + return kind === "tsc" ? PIPE_RE.test(tail) || REDIRECT_RE.test(tail) : TEST_REDIRECT_RE.test(tail); +} + +/** + * Resolve `unquoted` to a recognised {@link RunnerSpec}, or `null` when it + * matches none — including a {@link NO_RUN_RE} zero-test/informational + * invocation, a runner that only appears in an EARLIER list segment (module + * doc structural rule), or a runner whose own segment suppresses its output + * after the match ({@link isSuppressed}). First matching `RUNNERS` entry + * (table order) within the last segment wins. + * @param unquoted - `unquotedShellText(command)` — never the raw command. + */ +export function matchRunner(unquoted: string): RunnerSpec | null { + if (NO_RUN_RE.test(unquoted)) return null; + const segment = lastListSegment(unquoted); + for (const runner of RUNNERS) { + const m = runner.cmd.exec(segment); + if (!m) continue; + const tail = segment.slice(m.index + m[0].length); + return isSuppressed(tail, runner.kind) ? null : runner; + } + return null; +} diff --git a/src/tracking/receipt-runners-parse-php.ts b/src/tracking/receipt-runners-parse-php.ts new file mode 100644 index 0000000..2f85339 --- /dev/null +++ b/src/tracking/receipt-runners-parse-php.ts @@ -0,0 +1,58 @@ +/** + * @module receipt-runners-parse-php + * Pass/fail count parsers for {@link RunnerSpec.counts} (`./receipt-runners`) + * covering PHP test runners (`phpunit`, `pest`). Split out of + * `./receipt-runners-parse` (the other test-runner formats), which still + * owns {@link parseFailedPassedSummary}, to keep both files under the SOLID + * line ceiling. + */ +import { parseFailedPassedSummary } from "./receipt-runners-parse"; + +/** + * phpunit / `php artisan test` — three success banners on exit 0, all + * starting with `OK`: `OK (N tests, M assertions)`, `OK, but there were + * issues!`, and `OK, but some tests were skipped!` (PHPUnit 10-13; the last + * two are followed by a `Tests: N, Assertions: M, Skipped: …`/ + * `…, Deprecations: …` detail line, D3). Failure markers (`FAILURES!`, + * `ERRORS!`, PHPUnit 10+'s bare `FAILED`) take PRECEDENCE over any `OK` + * banner and are checked first; `Tests: N, Assertions: M[, Errors: E] + * [, Failures: F].` on failure. When no `Tests:` detail line follows a + * failure marker, `fail: 1` still records a definite failure rather than + * reporting nothing. `Tests: N` on an `OK, but …` banner INCLUDES skipped + * tests, so `pass` subtracts `Skipped: S` (tolerating any intervening + * `[, Errors:][, Failures:][, Warnings:][, Deprecations:]` clause) — an + * all-skipped suite (`Tests: 3, Skipped: 3`) must report `pass: 0`, never 3. + */ +export function parsePhpunit(output: string): { pass?: number; fail?: number } { + const detail = output.match(/Tests: (\d+), Assertions: \d+(?:, Errors: (\d+))?(?:, Failures: (\d+))?/); + if (/^(?:FAILURES|ERRORS|FAILED)\b/m.test(output)) { + if (!detail) return { fail: 1 }; + const total = Number(detail[1]); + const errors = detail[2] !== undefined ? Number(detail[2]) : 0; + const failures = detail[3] !== undefined ? Number(detail[3]) : 0; + const fail = errors + failures || 1; + return { pass: total - fail, fail }; + } + if (/^OK\b/m.test(output)) { + const okM = output.match(/^OK \((\d+) tests?/m); + if (okM) return { pass: Number(okM[1]), fail: 0 }; + if (detail) { + const skippedM = output.match(/\bSkipped: (\d+)/); + const skipped = skippedM ? Number(skippedM[1]) : 0; + return { pass: Math.max(0, Number(detail[1]) - skipped), fail: 0 }; + } + return { pass: 1, fail: 0 }; + } + return {}; +} + +/** + * Pest — `Tests: 2 failed, 10 passed (30 assertions)` (`skipped` clause + * optional). The `passed` clause is itself optional: a failed-only summary + * (`Tests: 2 failed (5 assertions)`, no trailing comma) still needs `fail` + * recognised — only the `Tests:` header + parenthesised assertions anchor it. + */ +export function parsePest(output: string): { pass?: number; fail?: number } { + const m = output.match(/Tests:\s+(?:(\d+) failed,?\s*)?(?:(\d+) skipped,\s*)?(?:(\d+) passed)?\s*\(/); + return parseFailedPassedSummary(m, 3); +} diff --git a/src/tracking/receipt-runners-parse-static.ts b/src/tracking/receipt-runners-parse-static.ts new file mode 100644 index 0000000..ac6b096 --- /dev/null +++ b/src/tracking/receipt-runners-parse-static.ts @@ -0,0 +1,102 @@ +/** + * @module receipt-runners-parse-static + * Pass/fail count parsers for {@link RunnerSpec.counts} (`./receipt-runners`) + * covering STATIC checkers (`kind: "tsc"` — the literal is historical, it now + * means "static check"). Split out of `./receipt-runners-parse` (test-runner + * formats) to keep both files under the SOLID line ceiling. Each returns + * `{ pass?, fail? }` — `undefined` `fail` means "not reported" (exit code + * alone decides); a defined `fail` (0 or more) means the OUTPUT itself proved + * a pass/fail count, closing the "masked exit code" gap (`tsc ... | head`, + * `go vet` piped through a filter) where `exitCode` alone can lie. + */ + +/** + * Marker-first static-checker contract shared by mypy/phpstan: `fail: 0` + * when `successRe` matches; else the diagnostic count when `countRe` + * matches; else `undefined` (no positive evidence). + */ +function markerOrCount(output: string, successRe: RegExp, countRe: RegExp): { fail?: number } { + if (successRe.test(output)) return { fail: 0 }; + const m = output.match(countRe); + return { fail: m ? Number(m[1]) : undefined }; +} + +/** mypy — `Found N errors` or `Success: no issues found`. */ +export function parseMypy(output: string): { pass?: number; fail?: number } { + return markerOrCount(output, /Success: no issues found/, /Found (\d+) errors?/); +} + +/** + * pyright — `N errors, M warnings, K informations`, but the `warnings` clause + * is not required to recognise the error count (`1 error ` alone, e.g. under + * `--outputjson` or a truncated summary, still parses). + */ +export function parsePyright(output: string): { pass?: number; fail?: number } { + const m = output.match(/(\d+) errors?\b/); + return { fail: m ? Number(m[1]) : undefined }; +} + +/** phpstan — `[ERROR] Found N errors` or `[OK]`. */ +export function parsePhpstan(output: string): { pass?: number; fail?: number } { + return markerOrCount(output, /\[OK\]/, /\[ERROR\] Found (\d+) errors?/); +} + +/** + * Diagnostic-count-first static-checker contract shared by tsc/go vet/go + * build/cargo check/clippy: `fail` = the count of `diagnosticRe` matches + * when present; else `0` only when `isClean(output)` holds (truly empty + * output, or the tool's own success banner); else `undefined` (no positive + * evidence). + */ +function diagnosticsOrClean(output: string, diagnosticRe: RegExp, isClean: (output: string) => boolean): { fail?: number } { + const m = output.match(diagnosticRe); + if (m) return { fail: m.length }; + return { fail: isClean(output) ? 0 : undefined }; +} + +/** + * tsc — SILENT ON SUCCESS: no summary line at all. `fail` is a POSITIVE + * signal only — the diagnostic count when `error TSxxxx:` lines are present, + * `0` only when the output is truly EMPTY (after trim), else `undefined` + * (D1/D2: `tsc --version`'s banner, a Node crash trace, or any other + * non-empty non-diagnostic text proves nothing was actually type-checked — + * `classifyReceipt`'s {@link isSuppressed} already nulls a piped/redirected + * invocation outright, closing the "masked exit + truncated output" gap this + * used to paper over with a bare `fail: 0` default). + */ +export function parseTsc(output: string): { pass?: number; fail?: number } { + return diagnosticsOrClean(output, /\berror TS\d+:/g, (o) => o.trim() === ""); +} + +/** + * `go vet` / `go build` — SILENT ON SUCCESS, same positive-evidence contract + * as {@link parseTsc}: diagnostic count when present, `0` only on truly + * empty output, else `undefined`. + */ +export function parseGoStatic(output: string): { pass?: number; fail?: number } { + return diagnosticsOrClean(output, /^(?:#\s|vet:\s|.+\.go:\d+:\d+:)/gm, (o) => o.trim() === ""); +} + +/** + * `cargo check` / `cargo clippy` — fail = count of `error[Exxxx]:`/`error:` + * lines when present; otherwise `0` ONLY on cargo's own success banner + * (`Finished …`), else `undefined` (D1: `cargo clippy --version`'s banner + * has neither an error line nor `Finished`, so it proves nothing). Cargo + * right-aligns status verbs to 12 columns, so the REAL banner is + * ` Finished \`dev\` profile [...] target(s) in 0.02s` — 4 leading + * spaces, never bare `Finished` at column 0. + */ +export function parseCargoStatic(output: string): { pass?: number; fail?: number } { + return diagnosticsOrClean(output, /^error(?:\[E\d+\])?:/gm, (o) => /^\s*Finished\b/m.test(o)); +} + +/** + * `swift build` — `Build complete!` on success (`fail: 0`); diagnostic count + * when `error: ` lines are present; otherwise `undefined` (D1: `swift build + * --help`'s usage text has neither, so it proves nothing). + */ +export function parseSwiftBuild(output: string): { pass?: number; fail?: number } { + if (/Build complete!/.test(output)) return { fail: 0 }; + const m = output.match(/(?:^|\s)error:\s/gm); + return { fail: m ? m.length : undefined }; +} diff --git a/src/tracking/receipt-runners-parse.ts b/src/tracking/receipt-runners-parse.ts new file mode 100644 index 0000000..efc4b2a --- /dev/null +++ b/src/tracking/receipt-runners-parse.ts @@ -0,0 +1,188 @@ +/** + * @module receipt-runners-parse + * Pass/fail count parsers for {@link RunnerSpec.counts} (`./receipt-runners`), + * one per TEST-runner output format — static-checker parsers (`mypy`, + * `pyright`, `phpstan`, `tsc`, `go vet`/`build`, `cargo check`/`clippy`, + * `swift build`) live in `./receipt-runners-parse-static`, and PHP-runner + * parsers (`phpunit`, `pest`) live in `./receipt-runners-parse-php` (which + * imports {@link parseFailedPassedSummary} from here), to keep every file + * under the SOLID line ceiling. Each returns `{ pass?, fail? }` — + * `undefined` fields mean "not reported", never "zero"; `classifyReceipt` + * (`./receipts`) falls back to `exitCode` alone via `isPassing` when both are + * absent. Formats verified against each tool's real summary-line output. + */ + +/** + * `bun test` — ` 8 pass\n 0 fail\n`. Anchored to LINE START (`^\s*(\d+)\s+ + * pass|fail\b`, `/m`) and the LAST such line, never the first match anywhere + * in the output (D3: a test NAME like "returns 3 fail codes" is not at a + * line start after a `\s*` prefix, so it can never match; a later summary + * line, if bun ever prints more than one block, wins over an earlier one). + */ +export function parseBunTest(output: string): { pass?: number; fail?: number } { + const passMatches = [...output.matchAll(/^\s*(\d+)\s+pass\b/gm)]; + const failMatches = [...output.matchAll(/^\s*(\d+)\s+fail\b/gm)]; + return { + pass: passMatches.length ? Number(passMatches[passMatches.length - 1]![1]) : undefined, + fail: failMatches.length ? Number(failMatches[failMatches.length - 1]![1]) : undefined, + }; +} + +/** + * Shared "`N failed`, `M passed`" summary-line contract for vitest/pest: + * capture group 1 is always the optional `failed` clause; `passGroupIndex` + * is this format's `passed` clause index. Both clauses are optional in the + * underlying regex — a failed-only summary (no `passed` clause at all) must + * still report `fail`, so `{}` is returned only when NEITHER clause matched. + * Exported for `./receipt-runners-parse-php`'s `parsePest`. + */ +export function parseFailedPassedSummary(m: RegExpMatchArray | null, passGroupIndex: number): { pass?: number; fail?: number } { + if (!m || (m[1] === undefined && m[passGroupIndex] === undefined)) return {}; + return { pass: m[passGroupIndex] !== undefined ? Number(m[passGroupIndex]) : undefined, fail: m[1] !== undefined ? Number(m[1]) : 0 }; +} + +/** + * Vitest — ` Tests 1 failed | 1 passed (2)` (the `failed |` clause is absent + * on an all-pass run). The `passed` clause itself is optional: a failed-only + * summary (` Tests 2 failed (2)`) reports no `passed` count at all — the + * `Tests` header plus a parenthesised total is still required so an + * unrelated line never matches. + */ +export function parseVitest(output: string): { pass?: number; fail?: number } { + const m = output.match(/Tests\s+(?:(\d+) failed(?:\s*\|\s*)?)?(?:(\d+) passed)?\s*\(/); + return parseFailedPassedSummary(m, 2); +} + +/** + * Jest — `Tests: 2 failed, 12 passed, 14 total` (`skipped` clause + * optional). The `passed` clause is itself optional: `Tests: 2 failed, 2 + * total` (no passing tests at all) still needs `fail` recognised — only the + * `Tests:` header + mandatory `total` count anchor the match. + */ +export function parseJest(output: string): { pass?: number; fail?: number } { + const m = output.match(/Tests:\s+(?:(\d+) failed,\s*)?(?:(\d+) skipped,\s*)?(?:(\d+) passed,\s*)?(\d+) total/); + if (!m) return {}; + return { pass: m[3] !== undefined ? Number(m[3]) : undefined, fail: m[1] !== undefined ? Number(m[1]) : 0 }; +} + +/** `npm|pnpm|yarn test` — the underlying runner is unknown from the command alone; try each format in turn. */ +export function parseNpmGeneric(output: string): { pass?: number; fail?: number } { + const vitest = parseVitest(output); + if (vitest.pass !== undefined) return vitest; + const jest = parseJest(output); + if (jest.pass !== undefined) return jest; + return parseBunTest(output); +} + +/** + * pytest — restricted to the summary line ending `in s` (e.g. + * `===== 2 failed, 12 passed, 1 skipped in 0.53s =====`), the LAST such line + * when several appear (an earlier `===== ERRORS =====` section header has no + * `in Ns` suffix, so it never qualifies). D2: the `=` padding is COSMETIC, + * never mandatory — a bare `-q`/`-x`/`-k` run prints the same summary + * undecorated (`2 passed in 0.01s`, `1 failed, 2 passed in 0.02s`, + * `no tests ran in 0.01s`); the last form reports `pass: 0` as POSITIVE + * evidence pytest ran and matched nothing, never `{}`. Fail sums `failed` + + * `errors`. + */ +export function parsePytest(output: string): { pass?: number; fail?: number } { + const isSummary = (line: string) => + /\b\d+ (?:passed|failed|errors?|skipped|xfailed|xpassed|deselected)\b/.test(line) || /\bno tests ran\b/.test(line); + const lines = [...output.matchAll(/^.*\bin [\d.]+s.*$/gm)].map((m) => m[0]).filter(isSummary); + const summary = lines.length ? lines[lines.length - 1] : undefined; + if (summary === undefined) return {}; + if (/\bno tests ran\b/.test(summary)) return { pass: 0 }; + const passM = summary.match(/(\d+) passed/); + const failM = summary.match(/(\d+) failed/); + const errM = summary.match(/(\d+) errors?\b/); + const fail = failM || errM ? (failM ? Number(failM[1]) : 0) + (errM ? Number(errM[1]) : 0) : undefined; + return { pass: passM ? Number(passM[1]) : undefined, fail }; +} + +/** + * `go test` — fail = `--- FAIL: ` line count (leading indentation included — + * a subtest's `--- FAIL:` is tab-indented under its parent `--- FAIL:`/ + * `--- PASS:` line), else package-level `FAIL` lines, else 0. Pass = count of + * `^ok\s` lines (one per successful package, `(cached)` included) — POSITIVE + * evidence a package actually ran, closing the gap where a piped/filtered + * invocation (`go test ./... | grep -c ok`) kept `fail: 0` with no real proof + * anything executed (D2) — EXCEPT an `ok` line itself tagged `[no tests to + * run]` (a filtered package matched zero tests), which doesn't count as + * executed evidence. A `? \tpkg\t[no test files]` line never starts with + * `ok`, so it is already excluded without a separate check. `pass: 0` only + * when there is no other `ok` line at all (real M2 "tests ran" floor); with + * NO `ok` line whatsoever (e.g. a compile failure), `pass` stays `undefined` + * — no evidence either way. + */ +export function parseGoTest(output: string): { pass?: number; fail?: number } { + const okLines = output.match(/^ok\s.*$/gm); + const executed = okLines ? okLines.filter((l) => !l.includes("[no tests to run]")) : []; + const failLines = output.match(/^\s*--- FAIL: /gm); + const pkgFail = output.match(/^FAIL\b/gm); + const fail = failLines ? failLines.length : pkgFail ? pkgFail.length : 0; + return { pass: okLines ? executed.length : undefined, fail }; +} + +/** + * `cargo test` — sums every `test result: ok|FAILED. N passed; M failed;` + * line (one per test binary). Cargo's own `error: N targets failed` line + * only ADDS to `fail` when no `test result:` line reported a failure yet + * (D5: `cargo test --no-fail-fast 2>&1 | tail -N` can truncate away a + * failing target's `test result: FAILED` line while this trailing line + * survives — a fail floor even with zero `test result:` lines); when a + * `test result: FAILED` line already counted the failure, the marker is + * the SAME failure restated, not an additional one, and must not double it. + */ +export function parseCargoTest(output: string): { pass?: number; fail?: number } { + const re = /test result: (?:ok|FAILED)\. (\d+) passed; (\d+) failed;/g; + let pass = 0; + let fail = 0; + let matched = false; + let m: RegExpExecArray | null; + while ((m = re.exec(output)) !== null) { + matched = true; + pass += Number(m[1]); + fail += Number(m[2]); + } + const targetsFailed = output.match(/^error: (\d+) targets? failed\b/m); + if (targetsFailed) { + matched = true; + if (fail === 0) fail += Number(targetsFailed[1]); + } + return matched ? { pass, fail } : {}; +} + +/** `swift test` — sums the LAST XCTest `Executed N tests, with M failures` line and every swift-testing `Test run with N tests ... passed|failed after` line. */ +export function parseSwiftTest(output: string): { pass?: number; fail?: number } { + let pass: number | undefined; + let fail: number | undefined; + const xcRe = /Executed (\d+) tests?, with (\d+) failures?/g; + let xcLast: RegExpExecArray | null = null; + let m: RegExpExecArray | null; + while ((m = xcRe.exec(output)) !== null) xcLast = m; + if (xcLast) { + const executed = Number(xcLast[1]); + const failures = Number(xcLast[2]); + pass = (pass ?? 0) + (executed - failures); + fail = (fail ?? 0) + failures; + } + const stRe = /Test run with (\d+) tests?(?: in \d+ suites?)? (passed|failed) after/g; + let sm: RegExpExecArray | null; + while ((sm = stRe.exec(output)) !== null) { + if (sm[2] === "failed") fail = (fail ?? 0) + 1; + else pass = (pass ?? 0) + Number(sm[1]); + } + return { pass, fail }; +} + +/** + * `dart test` / `flutter test` — only an overall pass/fail banner, no real + * counts: `All tests passed!` reports `pass: 1` as BOOLEAN positive evidence + * (D1: without it, the test-kind floor — pass > 0 — would reject every + * genuinely green dart/flutter run for lack of a numeric count). + */ +export function parseDartTest(output: string): { pass?: number; fail?: number } { + if (/All tests passed!/.test(output)) return { pass: 1, fail: 0 }; + if (/Some tests failed\./.test(output)) return { fail: 1 }; + return {}; +} diff --git a/src/tracking/receipt-runners.ts b/src/tracking/receipt-runners.ts new file mode 100644 index 0000000..b9ab7a8 --- /dev/null +++ b/src/tracking/receipt-runners.ts @@ -0,0 +1,136 @@ +/** + * @module receipt-runners + * The ordered table of recognised verification commands (`RUNNERS`) consumed + * by `classifyReceipt` (`./receipts`, first match wins). Every command regex + * is {@link RECEIPT_ANCHOR}-anchored (command position: start/segment + * separator, then any wrapper chain) plus an optional runtime {@link PREFIX} + * (`bunx`, `npx`, `vendor/bin/`, `uv run`, `poetry run`, `python -m`, …), so a + * real invocation matches while a quoted/argument MENTION of the same tool + * name (`git commit -m "fix jest flake"`) never does. `classifyReceipt` tests + * these regexes against `unquotedShellText(command)` (quote/heredoc-stripped), + * never the raw command — matching the raw text would let a forged mention + * hidden in a commit message or heredoc body pass as a real invocation. Spans + * TS/JS, Python, Go, Rust, PHP, Swift, and Dart toolchains — the extension + * list this widens on is `task-completed.ts`'s `CODE_EXTENSIONS`. + */ +import { ENV_PREFIX, WRAP, WRAP_ARG } from "../policy/guards/bash-command-anchor"; +import type { RunnerSpec } from "./receipts"; +import { + parseBunTest, parseVitest, parseJest, parseNpmGeneric, parsePytest, parseGoTest, + parseCargoTest, parseSwiftTest, parseDartTest, +} from "./receipt-runners-parse"; +import { parsePhpunit, parsePest } from "./receipt-runners-parse-php"; +import { + parseMypy, parsePyright, parsePhpstan, parseTsc, parseGoStatic, parseCargoStatic, parseSwiftBuild, +} from "./receipt-runners-parse-static"; + +/** + * Receipt-specific command-position anchor: the same env-prefix/wrapper-chain + * composition as `CMD` (bash-command-anchor.ts) but WITHOUT the backtick in + * its separator class. A backtick never actually starts a new command — it + * opens command substitution, which `unquotedShellText` already lexes and + * re-wraps in literal backticks — so treating it as a separator here would + * over-match a tool name sitting AFTER a real substitution + * (`echo \`date\` mypy src`) as if it were its own invocation. The write + * guard's `CMD` intentionally keeps the backtick (defense-in-depth: a + * mutator hidden behind one is still caught); a verification receipt is a + * PROOF of success, so over-matching here is unsafe, not merely redundant. + * Real multi-line scripts stay recognised: `\n` is still a separator. + */ +const RECEIPT_ANCHOR = `(?:^|[\\n;&|(])\\s*${ENV_PREFIX}(?:${WRAP}\\s+(?:${WRAP_ARG}\\s+)*)*`; + +/** + * Runtime-option tokens tolerated between a wrapper and the tool name itself + * (`uv run --frozen pytest`, `pnpm --filter api test`, `pnpm -r test`): each + * iteration is one flag (`-r`, `--frozen`) with an optional space- or + * `=`-joined value (`--filter api`, `--reporter=json`). The value clause + * carries a mandatory `(?!-)` right after `[= ]`: without it, `\S+` can + * swallow a FOLLOWING flag's own characters as this iteration's "value" + * (`-o` then `-o` reinterpreted as `-o` + value `-o`), giving every adjacent + * flag pair two indistinguishable partitions and turning the outer `*` into + * an exponential-partition ReDoS — measured: `pnpm ` + `-o `×1500 + a + * non-matching tail took 17.8s before this lookahead (vs <5ms after). The + * lookahead forces a value to never itself look like the start of a new + * flag, so each token has exactly one parse and backtracking stays linear. + * A value that legitimately starts with `-` (e.g. `--exclude=-x`) is out of + * scope for this best-effort recognizer — real test-runner invocations don't + * use that shape for the options this tolerates (`--filter`, `--frozen`, + * `--quiet`, `-r`). + * + * A second, independent ambiguity lived in the flag-name class itself: + * `--?[\w-]+` let a leading dash be claimed by EITHER `--?` or by `[\w-]+` + * (which also contains `-`) — e.g. `--flag` parses as `--?`="-" + + * `[\w-]+`="-flag" OR `--?`="--" + `[\w-]+`="flag", two splits producing the + * same consumed text. Across k flag tokens that is a 2^k-way split of + * identical total input, and once the surrounding tool-name literal fails to + * match, the engine re-explores every split — measured: `uv run ` + + * `--flag=val `×400 + a non-matching tail took 2034ms through just the `bun + * test` alternative alone (vs <5ms after). Fix: the flag name's FIRST + * character must be a plain word character (`\w`, never `-`), so `--?` is + * forced to claim every leading dash deterministically before the name + * starts — only dashes WITHIN a multi-word flag (`--dry-run`) stay in + * `[\w-]*`. Standard ReDoS-elimination technique (OWASP: remove + * ambiguous/overlapping alternation so only one parse exists, same principle + * as javascript.info's `(\w+\s?)*` → `(\w+\s)*\w*` rewrite). + */ +const OPTS = "(?:--?[\\w][\\w-]*(?:[= ](?!-)\\S+)?\\s+)*"; + +/** + * Optional runtime wrapper accepted between the {@link RECEIPT_ANCHOR} and + * the tool name itself: package-manager runners (`bunx`, `npx`, `pnpm + * [opts] [exec]`, `yarn`), PHP's `vendor/bin/`, and Python's `uv run [opts]` + * / `poetry run [opts]` / `python[3] -m` invocation forms (D4: `uv run + * --frozen pytest`, `poetry run --quiet pytest`). + */ +const PREFIX = `(?:(?:\\.\\/)?vendor\\/bin\\/|bunx\\s+|npx\\s+|pnpm\\s+${OPTS}(?:exec\\s+)?|yarn\\s+|uv\\s+run\\s+${OPTS}|poetry\\s+run\\s+${OPTS}|python3?\\s+-m\\s+)?`; + +/** Build an anchored command regex: {@link RECEIPT_ANCHOR} + {@link PREFIX} + a tool-specific pattern. */ +function anchored(pattern: string): RegExp { + return new RegExp(RECEIPT_ANCHOR + PREFIX + pattern); +} + +/** + * `php artisan test` — Laravel's runner is Pest-or-PHPUnit under the hood, and + * a Pest/Collision-style project prints Pest's `Tests: N passed (…)` summary + * (see {@link parsePest}), not PHPUnit's `OK (N tests, …)`/`Tests: N, + * Assertions: …` shapes. Try Pest's format FIRST; a bare `phpunit`/ + * `vendor/bin/phpunit` invocation (its own `RUNNERS` entry below) never goes + * through here, so it keeps the plain {@link parsePhpunit} contract. + */ +function parseArtisanTest(output: string): { pass?: number; fail?: number } { + const pest = parsePest(output); + return pest.pass !== undefined || pest.fail !== undefined ? pest : parsePhpunit(output); +} + +/** + * Recognised verification commands, first match wins. `kind: "tsc"` covers + * every static checker (the literal is historical); `kind: "test"` covers + * every test runner. + */ +export const RUNNERS: readonly RunnerSpec[] = [ + { tool: "bun test", kind: "test", cmd: anchored("bun\\s+test\\b"), counts: parseBunTest }, + { tool: "vitest", kind: "test", cmd: anchored("vitest\\b"), counts: parseVitest }, + { tool: "jest", kind: "test", cmd: anchored("jest\\b"), counts: parseJest }, + { + tool: "npm/pnpm/yarn test", kind: "test", + cmd: anchored(`(?:(?:npm|pnpm|yarn)\\s+(?:run\\s+)?${OPTS}test\\b|bun\\s+run\\s+${OPTS}test\\b)`), + counts: parseNpmGeneric, + }, + { tool: "pytest", kind: "test", cmd: anchored("pytest\\b"), counts: parsePytest }, + { tool: "go test", kind: "test", cmd: anchored("go\\s+test\\b"), counts: parseGoTest }, + { tool: "cargo test", kind: "test", cmd: anchored("cargo\\s+test\\b"), counts: parseCargoTest }, + { tool: "phpunit", kind: "test", cmd: anchored("php\\s+artisan\\s+test\\b"), counts: parseArtisanTest }, + { tool: "phpunit", kind: "test", cmd: anchored("phpunit\\b"), counts: parsePhpunit }, + { tool: "pest", kind: "test", cmd: anchored("pest\\b"), counts: parsePest }, + { tool: "swift test", kind: "test", cmd: anchored("swift\\s+test\\b"), counts: parseSwiftTest }, + { tool: "dart/flutter test", kind: "test", cmd: anchored("(?:flutter|dart)\\s+test\\b"), counts: parseDartTest }, + { tool: "tsc", kind: "tsc", cmd: anchored("tsc\\b"), counts: parseTsc }, + { tool: "mypy", kind: "tsc", cmd: anchored("mypy\\b"), counts: parseMypy }, + { tool: "pyright", kind: "tsc", cmd: anchored("pyright\\b"), counts: parsePyright }, + { tool: "phpstan", kind: "tsc", cmd: anchored("phpstan\\s+analy[sz]e\\b"), counts: parsePhpstan }, + { tool: "go vet", kind: "tsc", cmd: anchored("go\\s+vet\\b"), counts: parseGoStatic }, + { tool: "go build", kind: "tsc", cmd: anchored("go\\s+build\\b"), counts: parseGoStatic }, + { tool: "cargo check", kind: "tsc", cmd: anchored("cargo\\s+check\\b"), counts: parseCargoStatic }, + { tool: "cargo clippy", kind: "tsc", cmd: anchored("cargo\\s+clippy\\b"), counts: parseCargoStatic }, + { tool: "swift build", kind: "tsc", cmd: anchored("swift\\s+build\\b"), counts: parseSwiftBuild }, +]; diff --git a/src/tracking/receipts.ts b/src/tracking/receipts.ts index 194a7ce..95755da 100644 --- a/src/tracking/receipts.ts +++ b/src/tracking/receipts.ts @@ -1,67 +1,112 @@ /** * @module receipts - * Verification receipts: capture `tsc`/test runs at PostToolUse and query the - * freshest PASSING one for the TaskCompleted (hard) and SubagentStop (advisory) - * gates. A "done" over modified code files is refused unless such a receipt - * exists — mechanising "no proof, no done". + * Verification receipts: capture a static-check or test run at PostToolUse and + * query the freshest PASSING one for the TaskCompleted (hard) and SubagentStop + * (advisory) gates. A "done" over modified code files is refused unless such a + * receipt exists — mechanising "no proof, no done". Command recognition spans + * many ecosystems (TS/JS, Python, Go, Rust, PHP, Swift, Dart) via the ordered + * {@link RunnerSpec} table in `./receipt-runners`; which command text is even + * eligible, and the structural "last segment" / output-suppression rules, + * live in `./receipt-command`. + * + * Threat model (explicit): a receipt guards against FORGETTING to verify and + * against honest shortcuts (a run whose output/exit code was accidentally + * discarded, an informational invocation, a filtered run) — it is NOT a + * security control against an agent that deliberately fabricates tool + * output, same caveat as the `CONFIRM` code in the README. * @packageDocumentation */ import { withTrack, trackJournalEnabled } from "./store"; import { readTrackSync } from "./track-compact"; +import { matchRunner, stripAnsi } from "./receipt-command"; +import { unquotedShellText } from "../policy/guards/bash-write-unquoted"; import type { SessionTrack } from "./session-state"; /** * A verification receipt captured at PostToolUse from a Bash verification * command. Feeds the TaskCompleted receipt gate: a "done" over modified code * files is refused unless a fresh, passing receipt (`exitCode === 0`, `fail === 0`) - * exists. `pass`/`fail` are parsed only for test runs; `tsc` carries the code alone. + * exists. `pass`/`fail` are parsed only for test runs; a static check (`kind: + * "tsc"` — the literal is historical, it now means "static check": `tsc`, + * `mypy`, `pyright`, `phpstan`, `go vet`/`build`, `cargo check`/`clippy`, + * `swift build`) carries the exit code alone. */ export interface Receipt { kind: "tsc" | "test"; + /** The matched runner's tool name (`bun test`, `pytest`, `cargo test`, …). Absent on receipts predating this field — still verifiable via `kind`/`exitCode`/`fail` under the legacy branch of {@link isPassing}. */ + tool?: string; exitCode: number; pass?: number; fail?: number; ts: number; } -/** `tsc` / `bunx tsc` / `npx tsc -p .` — a type-check invocation. */ -const TSC_RE = /(?:^|\s|\/)(?:bunx\s+|npx\s+|pnpm\s+|yarn\s+)?tsc\b/; -/** `bun test`, `vitest`, `jest`, `npm test`, `npm run test`, … — a test run. */ -const TEST_RE = /\b(?:bun\s+test|vitest|jest|(?:npm|pnpm|yarn)\s+(?:run\s+)?test)\b/; +/** + * A single recognised verification command: which {@link RunnerSpec.kind} it + * produces, its anchored command regex, and an optional pass/fail parser for + * its output format. + */ +export interface RunnerSpec { + tool: string; + kind: "tsc" | "test"; + cmd: RegExp; + counts?: (output: string) => { pass?: number; fail?: number }; +} /** Append a verification receipt to the track. Immutable. */ export function recordReceipt(track: SessionTrack, receipt: Receipt): SessionTrack { return { ...track, receipts: [...(track.receipts ?? []), receipt] }; } -/** Parse `N pass` / `M fail` counts from bun/vitest/jest output (undefined when absent). */ -function parseCounts(output: string): { pass?: number; fail?: number } { - const pass = output.match(/(\d+)\s+pass/i); - const fail = output.match(/(\d+)\s+fail/i); - return { pass: pass ? Number(pass[1]) : undefined, fail: fail ? Number(fail[1]) : undefined }; -} - /** - * Classify a Bash command as a verification receipt, or `null` when it is not a - * recognised `tsc`/test invocation. Test runs additionally carry parsed - * pass/fail counts. + * Classify a Bash command as a verification receipt via {@link matchRunner} + * (`./receipt-command`: ordered `RUNNERS` table, first match wins, scoped to + * the last shell list segment), or `null` when it resolves to none — a + * zero-test/informational invocation, a runner sitting only in an EARLIER + * list segment (`bun test && git commit`, `bun test || true`), or a runner + * whose own segment suppresses its output after the match, none of which is + * proof of anything. Runner regexes are tested against + * `unquotedShellText(command)` (quote/heredoc/comment-stripped), never the + * raw command, so a tool name mentioned inside a commit message, heredoc + * body, or `echo`d string is never mistaken for a real invocation (H1). + * `output` is ANSI/CR-stripped once ({@link stripAnsi}) before any + * test-kind runner parses its pass/fail counts. * @param command - The Bash command line. * @param output - Combined stdout+stderr (bun writes its summary to stderr). * @param exitCode - The command's exit code. * @param now - Capture timestamp (epoch ms). */ export function classifyReceipt(command: string, output: string, exitCode: number, now: number): Receipt | null { - if (TEST_RE.test(command)) { - const { pass, fail } = parseCounts(output); - return { kind: "test", exitCode, pass, fail, ts: now }; - } - if (TSC_RE.test(command)) return { kind: "tsc", exitCode, ts: now }; - return null; + const unquoted = unquotedShellText(command); + const runner = matchRunner(unquoted); + if (!runner) return null; + const counts = runner.counts?.(stripAnsi(output)) ?? {}; + return { kind: runner.kind, tool: runner.tool, exitCode, pass: counts.pass, fail: counts.fail, ts: now }; } -/** A receipt PROVES success: exit 0 and (for tests) zero reported failures. */ +/** + * A receipt PROVES success only via POSITIVE evidence (D1: "no counts + * observed" is no longer treated as proof): exit 0, plus — for `kind: + * "test"`, a reported pass count strictly greater than 0 AND zero reported + * failures (an unreported pass count proves nothing was ever executed); for + * `kind: "tsc"` (static check), an EXPLICIT `fail === 0` (an undefined + * `fail` — no success signature and no diagnostic count observed — proves + * nothing either). + * + * Legacy compat: a receipt recorded before the `tool` field existed (`tool + * === undefined`, the historical marker) keeps the PRE-fix contract so old + * signed tracks are not retroactively invalidated: `test` kind passes on + * `pass > 0 && fail === 0` (unchanged), `tsc` kind passes on `exitCode === 0` + * alone (the old contract never inspected `fail` for a static check). + */ function isPassing(r: Receipt): boolean { - return r.exitCode === 0 && (r.fail ?? 0) === 0; + if (r.exitCode !== 0) return false; + if (r.tool === undefined) { + if (r.kind === "tsc") return true; + return (r.pass ?? 0) > 0 && (r.fail ?? 0) === 0; + } + if (r.kind === "test") return r.pass !== undefined && r.pass > 0 && (r.fail ?? 0) === 0; + return r.fail === 0; } /** The newest passing receipt within `windowMs`, or `null`. */ diff --git a/src/tracking/session-state.ts b/src/tracking/session-state.ts index e979725..5dde5dc 100644 --- a/src/tracking/session-state.ts +++ b/src/tracking/session-state.ts @@ -23,7 +23,7 @@ export interface SessionTrack { agents: { name: string; ts: number; quality?: AgentQuality }[]; trivialEdits: number[]; brainstormRequired?: boolean; - /** Verification receipts (tsc/test) at PostToolUse; absent/empty reads as unverified in the TaskCompleted gate (backward compat, fail-closed). See {@link Receipt}. */ + /** Verification receipts (static-check/test runs, see `tracking/receipt-runners`) at PostToolUse; absent/empty reads as unverified in the TaskCompleted gate (backward compat, fail-closed). See {@link Receipt}. */ receipts?: Receipt[]; /** PRD (task/agent ownership coordination) — `agent_id` bound to its resolved agent-report name. Optional: absent on any track predating the PRD module (backward compat). */ prdOwners?: Record; diff --git a/test/receipt-runners-evidence.test.ts b/test/receipt-runners-evidence.test.ts new file mode 100644 index 0000000..49e216d --- /dev/null +++ b/test/receipt-runners-evidence.test.ts @@ -0,0 +1,78 @@ +import { test, expect } from "bun:test"; +import { tmpdir } from "node:os"; +import { mkdtempSync } from "node:fs"; +import { join } from "node:path"; +import { classifyReceipt, captureReceipt, freshReceiptFromFile, recordReceipt, freshPassingReceipt } from "../src/tracking/receipts"; +import { captureBashReceipt } from "../src/runtime/receipt-capture"; +import { emptyTrack } from "../src/tracking/session-state"; + +const T = 1_000_000_000_000; +const WIN = 365 * 24 * 3600 * 1000; +const file = (): string => join(mkdtempSync(join(tmpdir(), "fh-ev-")), "track.json"); + +/** Whether `command` would leave a receipt that grants "done" right now. */ +async function passes(command: string, output: string, exitCode: number): Promise { + const f = file(); + await captureReceipt(f, command, output, exitCode, T); + return freshReceiptFromFile(f, WIN, T + 1) !== null; +} + +test("D1: informational invocations are never a receipt at all", () => { + const cmds = [ + "tsc --version", "bunx tsc --version", "pytest --version", "bun test --help", + "mypy --version", "cargo test --help", "go vet -h", "jest --showConfig", + "pyright --version", "vitest --version", "swift build --help", "pytest --fixtures", + "cargo clippy --version", "go build -n ./...", "phpunit --version", "npx jest --init", + "go test -count=0 ./...", + ]; + for (const cmd of cmds) expect(classifyReceipt(cmd, "", 0, T)).toBeNull(); +}); + +test("D2: output suppression / exit decoupling never grants a passing receipt", async () => { + expect(await passes("bun test 2>/dev/null | cat", "", 0)).toBe(false); + expect(await passes("mypy src 2>/dev/null | cat", "", 0)).toBe(false); + expect(await passes("bun test 2>/dev/null; git commit -m x", "", 0)).toBe(false); + expect(await passes("bun test | grep -c pass", "12", 0)).toBe(false); + expect(await passes('echo "$(pytest -q 2>&1 >/dev/null)"', "", 0)).toBe(false); + expect(await passes("go test ./... 2>&1 | grep -c ok", "2", 0)).toBe(false); + expect(await passes("tsc --noEmit 2>&1 | wc -l", "42", 0)).toBe(false); + expect(await passes("timeout 5 bun test 2>/dev/null | cat", "", 0)).toBe(false); + expect(await passes('true && echo "$(tsc --noEmit)"', "TypeError: crashed\n", 0)).toBe(false); + expect(await passes('echo "$(bun test 2>/dev/null)"', "", 0)).toBe(false); +}); + +test("D3: a fail-shaped test NAME never masks the real (green) summary", () => { + const out = "(pass) parse > returns 3 fail codes [0.10ms]\n\n 12 pass\n 0 fail"; + expect(classifyReceipt("bun test", out, 0, T)).toMatchObject({ tool: "bun test", pass: 12, fail: 0 }); +}); + +test("D3: not passing when the real summary line is genuinely red", async () => { + const out = "(pass) parse > returns 3 fail codes [0.10ms]\n\n 12 pass\n 0 fail"; + expect(await passes("bun test", out, 0)).toBe(true); +}); + +test("D4: a missing exit_code is never credited as exit 0 (no receipt recorded)", async () => { + const f = file(); + const response = { stdout: " 8 pass\n 0 fail\n", stderr: "" }; // no exit_code field + await captureBashReceipt(f, "Bash", "bun test", undefined, response, T); + expect(freshReceiptFromFile(f, WIN, T + 1)).toBeNull(); +}); + +test("regression: every previously-green invocation still passes", async () => { + expect(await passes("bun test", " 12 pass\n 0 fail\n", 0)).toBe(true); + expect(await passes("bun test 2>&1 | tail -5", " 12 pass\n 0 fail\n", 0)).toBe(true); + expect(await passes("bunx tsc --noEmit", "", 0)).toBe(true); + expect(await passes("go test ./...", "ok \tpkg1\t0.1s\nok \tpkg2\t0.1s", 0)).toBe(true); + expect(await passes("mypy src", "Success: no issues found in 3 source files", 0)).toBe(true); + expect(await passes("cargo check", "Finished dev [unoptimized + debuginfo] target(s) in 0.52s", 0)).toBe(true); + expect(await passes("swift build", "Build complete! (1.23s)", 0)).toBe(true); + expect(await passes("dart test", "All tests passed!", 0)).toBe(true); +}); + +test("regression: legacy (no `tool` field) receipts keep the pre-fix contract", () => { + const legacyTsc = recordReceipt(emptyTrack(), { kind: "tsc", exitCode: 0, ts: T }); + expect(freshPassingReceipt(legacyTsc, WIN, T + 1)).not.toBeNull(); + + const legacyTest = recordReceipt(emptyTrack(), { kind: "test", exitCode: 0, pass: 12, fail: 0, ts: T }); + expect(freshPassingReceipt(legacyTest, WIN, T + 1)).not.toBeNull(); +}); diff --git a/test/receipt-runners-forgery.test.ts b/test/receipt-runners-forgery.test.ts new file mode 100644 index 0000000..285439c --- /dev/null +++ b/test/receipt-runners-forgery.test.ts @@ -0,0 +1,87 @@ +import { test, expect } from "bun:test"; +import { classifyReceipt } from "../src/tracking/receipts"; + +const T = 1_000_000_000_000; + +test("H1: heredoc body mentioning a runner is never a receipt", () => { + const cmd = "cat <<'E' > notes.md\nbun test is green\nE"; + expect(classifyReceipt(cmd, "", 0, T)).toBeNull(); +}); + +test("H1: a multi-line commit message mentioning a runner is never a receipt", () => { + expect(classifyReceipt('git commit -m "fix\nbun test now green"', "", 0, T)).toBeNull(); +}); + +test("H1: a tool name sitting after a real backtick substitution is never a receipt", () => { + expect(classifyReceipt("echo `date` mypy src", "", 0, T)).toBeNull(); +}); + +test("H1: a real newline-separated multi-line script is still recognised", () => { + expect(classifyReceipt("cd x\nbun test", " 1 pass\n 0 fail\n", 0, T)).toMatchObject({ tool: "bun test", pass: 1, fail: 0 }); +}); + +test("M2: every no-run/collect-only/watch/list flag is never a receipt", () => { + expect(classifyReceipt("cargo test --no-run", "", 0, T)).toBeNull(); + expect(classifyReceipt("pytest --collect-only", "", 0, T)).toBeNull(); + expect(classifyReceipt("pytest --co", "", 0, T)).toBeNull(); + expect(classifyReceipt("bun test --watch", "", 0, T)).toBeNull(); + expect(classifyReceipt("npm run test:watch", "", 0, T)).toBeNull(); + expect(classifyReceipt("npm test -- --watch", "", 0, T)).toBeNull(); + expect(classifyReceipt("jest --listTests", "", 0, T)).toBeNull(); + expect(classifyReceipt("jest --watchAll", "", 0, T)).toBeNull(); + expect(classifyReceipt("swift test --list-tests", "", 0, T)).toBeNull(); + expect(classifyReceipt("flutter test --dry-run", "", 0, T)).toBeNull(); + expect(classifyReceipt("jest --watchAll", "", 0, T)).toBeNull(); + expect(classifyReceipt("vitest run --watch", "", 0, T)).toBeNull(); +}); + +test("M2: --watch=false / --watchAll=false explicitly disable watch mode and are still a receipt", () => { + expect(classifyReceipt("vitest --watch=false", " Tests 2 passed (2)", 0, T)).toMatchObject({ tool: "vitest", pass: 2 }); + expect(classifyReceipt("jest --watchAll=false", "Tests: 2 passed, 2 total", 0, T)).toMatchObject({ tool: "jest", pass: 2 }); + // still rejected when watch mode is explicitly ON or unqualified + expect(classifyReceipt("vitest --watch=true", "", 0, T)).toBeNull(); + expect(classifyReceipt("jest --watchAll=true", "", 0, T)).toBeNull(); +}); + +test("M2: a zero-executed test run is classified (pass: 0) but rejected by isPassing's floor", () => { + const cargoZero = classifyReceipt("cargo test", "test result: ok. 0 passed; 0 failed; 0 ignored", 0, T); + expect(cargoZero).toMatchObject({ tool: "cargo test", pass: 0, fail: 0, exitCode: 0 }); + + // Real `go test` tags a filtered zero-match package on the `ok` line itself + // (`ok \tpkg\t0.001s [no tests to run]`) — that tag, not a separate warning + // line anywhere in the output, is what excludes it from the executed count. + const goNoTests = classifyReceipt("go test -run ZZZ ./...", "ok \tpkg\t0.001s [no tests to run]", 0, T); + expect(goNoTests).toMatchObject({ tool: "go test", pass: 0, fail: 0, exitCode: 0 }); +}); + +test("D2: a pipe after a STATIC-kind runner is never a receipt, even with real diagnostics in the piped output", () => { + // Superseded hardening: a pipe stage can reorder/truncate/recount whatever + // a static checker printed (D2's `tsc --noEmit 2>&1 | wc -l` forging a + // PASS is the same shape) — a static check's only proof is silence, which + // a pipe can no longer guarantee, so classifyReceipt nulls it outright + // instead of trusting output-parsed diagnostics through it. + const out = "src/a.ts(1,1): error TS2304: Cannot find name 'x'.\nsrc/b.ts(2,2): error TS2304: Cannot find name 'y'.\n"; + const r = classifyReceipt("bunx tsc --noEmit 2>&1 | head -20", out, 0, T); + expect(r).toBeNull(); +}); + +test("M3: explicit exit-masking forms are never a receipt", () => { + expect(classifyReceipt("bunx tsc --noEmit || true", "error TS2304: x", 0, T)).toBeNull(); + expect(classifyReceipt("bunx tsc --noEmit || :", "error TS2304: x", 0, T)).toBeNull(); + expect(classifyReceipt("bunx tsc --noEmit || echo failed", "error TS2304: x", 0, T)).toBeNull(); + expect(classifyReceipt("bunx tsc --noEmit; true", "error TS2304: x", 0, T)).toBeNull(); +}); + +test("M3: go vet / cargo clippy / swift build diagnostics parse to a non-zero fail", () => { + const vet = classifyReceipt("go vet ./...", "./main.go:10:2: unreachable code", 0, T); + expect(vet).toMatchObject({ tool: "go vet", fail: 1 }); + + const clippy = classifyReceipt("cargo clippy", "error[E0308]: mismatched types\n --> src/main.rs:2:5", 0, T); + expect(clippy).toMatchObject({ tool: "cargo clippy", fail: 1 }); + + const swiftOk = classifyReceipt("swift build", "Compiling...\nBuild complete! (1.23s)", 0, T); + expect(swiftOk).toMatchObject({ tool: "swift build", fail: 0 }); + + const swiftFail = classifyReceipt("swift build", "/path/File.swift:3:1: error: expected expression", 1, T); + expect(swiftFail).toMatchObject({ tool: "swift build", fail: 1 }); +}); diff --git a/test/receipt-runners-hardening.test.ts b/test/receipt-runners-hardening.test.ts new file mode 100644 index 0000000..46ddc03 --- /dev/null +++ b/test/receipt-runners-hardening.test.ts @@ -0,0 +1,32 @@ +import { test, expect } from "bun:test"; +import { classifyReceipt } from "../src/tracking/receipts"; + +const T = 1_000_000_000_000; + +test("L4: go test indented (subtest) --- FAIL: lines are counted", () => { + const out = "--- FAIL: TestParent\n --- FAIL: TestParent/child\nFAIL\tpkg\t0.1s"; + expect(classifyReceipt("go test ./...", out, 1, T)).toMatchObject({ tool: "go test", fail: 2 }); +}); + +test("L4: pest failed-only summary (no passed clause) parses to fail>0", () => { + const out = "Tests: 2 failed (5 assertions)"; + expect(classifyReceipt("pest", out, 1, T)).toMatchObject({ tool: "pest", fail: 2, pass: undefined }); +}); + +test("L4: jest failed-only summary (no passed clause) parses to fail>0", () => { + const out = "Tests: 2 failed, 2 total"; + expect(classifyReceipt("jest", out, 1, T)).toMatchObject({ tool: "jest", fail: 2, pass: undefined }); +}); + +test("L4: vitest failed-only summary (no passed clause) parses to fail>0", () => { + const out = "Tests 2 failed (2)"; + expect(classifyReceipt("vitest run", out, 1, T)).toMatchObject({ tool: "vitest", fail: 2, pass: undefined }); +}); + +test("L4: pyright '1 error ' without the warnings clause still parses", () => { + expect(classifyReceipt("pyright", "1 error ", 1, T)).toMatchObject({ tool: "pyright", fail: 1 }); +}); + +test("L4: PHPUnit 10+ FAILED banner is recognised (fail >= 1) even with no Tests: detail line", () => { + expect(classifyReceipt("phpunit", "FAILED", 1, T)).toMatchObject({ tool: "phpunit", fail: 1 }); +}); diff --git a/test/receipt-runners-realworld-2.test.ts b/test/receipt-runners-realworld-2.test.ts new file mode 100644 index 0000000..adc2d40 --- /dev/null +++ b/test/receipt-runners-realworld-2.test.ts @@ -0,0 +1,45 @@ +/** + * Regressions for two false positives found on real tool output, split from + * `receipt-runners-realworld.test.ts` to stay under the SOLID line ceiling: + * `parsePhpunit` (`./receipt-runners-parse-php`) counting skipped tests as + * passing on an all-skipped `OK` suite, and `parseCargoTest` + * (`./receipt-runners-parse`) double-counting a failure already reported by a + * `test result: FAILED` line via cargo's trailing `error: N targets failed` + * marker. + */ +import { test, expect } from "bun:test"; +import { classifyReceipt, recordReceipt, freshPassingReceipt } from "../src/tracking/receipts"; +import { emptyTrack } from "../src/tracking/session-state"; + +const T = 1_000_000_000_000; +const WIN = 365 * 24 * 3600 * 1000; + +test("phpunit all-skipped 'OK' suite is NOT a passing receipt (Tests includes skipped)", () => { + const r = classifyReceipt("phpunit", "OK, but some tests were skipped!\nTests: 3, Assertions: 0, Skipped: 3.", 0, T); + expect(r).toMatchObject({ tool: "phpunit", pass: 0, fail: 0 }); + const track = recordReceipt(emptyTrack(), r!); + expect(freshPassingReceipt(track, WIN, T + 1)).toBeNull(); +}); + +test("phpunit 'OK, but some tests were skipped!' subtracts Skipped from Tests", () => { + expect( + classifyReceipt("phpunit", "OK, but some tests were skipped!\nTests: 3, Assertions: 4, Skipped: 1.", 0, T), + ).toMatchObject({ tool: "phpunit", pass: 2, fail: 0 }); +}); + +test("phpunit 'OK, but there were issues!' with Deprecations (no Skipped) keeps pass = Tests", () => { + expect( + classifyReceipt("phpunit", "OK, but there were issues!\nTests: 2, Assertions: 3, Deprecations: 1.", 0, T), + ).toMatchObject({ tool: "phpunit", pass: 2, fail: 0 }); +}); + +test("cargo test: 'error: N targets failed' is not added on top of a per-binary FAILED count", () => { + const output = "test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out\nerror: 1 target failed"; + expect(classifyReceipt("cargo test", output, 1, T)).toMatchObject({ tool: "cargo test", fail: 1 }); +}); + +test("cargo test: 'error: N targets failed' alone (summary line scrolled off) still reports fail 1", () => { + expect(classifyReceipt("cargo test --no-fail-fast", "error: 1 target failed", 1, T)).toMatchObject({ + tool: "cargo test", fail: 1, + }); +}); diff --git a/test/receipt-runners-realworld.test.ts b/test/receipt-runners-realworld.test.ts new file mode 100644 index 0000000..bf1fdf5 --- /dev/null +++ b/test/receipt-runners-realworld.test.ts @@ -0,0 +1,57 @@ +/** + * Challenger v4 D1-D5 false negatives, measured on REAL tool output — + * regressions for `receipt-runners-parse.ts`, `receipt-runners-parse-php.ts`, + * `receipt-runners-parse-static.ts`, and `receipt-runners.ts`'s + * `PREFIX`/`OPTS`/generic-runner regexes. + */ +import { test, expect } from "bun:test"; +import { classifyReceipt } from "../src/tracking/receipts"; + +const T = 1_000_000_000_000; + +test("D1: cargo check/clippy real padded 'Finished' banner (4 leading spaces) parses to fail 0", () => { + const real = " Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.02s"; + expect(classifyReceipt("cargo check", real, 0, T)).toMatchObject({ tool: "cargo check", fail: 0 }); + expect(classifyReceipt("cargo clippy", real, 0, T)).toMatchObject({ tool: "cargo clippy", fail: 0 }); +}); + +test("D2: pytest -q undecorated summary variants (no '=' padding)", () => { + expect(classifyReceipt("pytest -q", "2 passed in 0.01s", 0, T)).toMatchObject({ tool: "pytest", pass: 2, fail: undefined }); + expect(classifyReceipt("pytest -x", "1 failed, 2 passed in 0.02s", 1, T)).toMatchObject({ tool: "pytest", pass: 2, fail: 1 }); + expect(classifyReceipt("pytest -k missing", "no tests ran in 0.01s", 5, T)).toMatchObject({ tool: "pytest", pass: 0 }); +}); + +test("D3: phpunit 10-13 'OK, but ...' banners still report success with the right pass count", () => { + expect(classifyReceipt("phpunit", "OK (12 tests, 30 assertions)", 0, T)).toMatchObject({ tool: "phpunit", pass: 12, fail: 0 }); + expect( + classifyReceipt("phpunit", "OK, but there were issues!\nTests: 12, Assertions: 30, Skipped: 1.", 0, T), + ).toMatchObject({ tool: "phpunit", pass: 11, fail: 0 }); + expect( + classifyReceipt("phpunit", "OK, but some tests were skipped!\nTests: 12, Assertions: 30, Deprecations: 1.", 0, T), + ).toMatchObject({ tool: "phpunit", pass: 12, fail: 0 }); +}); + +test("D3: phpunit failure markers still take precedence and still fail", () => { + expect(classifyReceipt("phpunit", "FAILURES!\nTests: 12, Assertions: 30, Failures: 2.", 1, T)).toMatchObject({ + tool: "phpunit", pass: 10, fail: 2, + }); + expect(classifyReceipt("phpunit", "FAILED", 1, T)).toMatchObject({ tool: "phpunit", fail: 1 }); +}); + +test("D4: option tokens tolerated between a wrapper/runner and the tool name", () => { + expect(classifyReceipt("uv run --frozen pytest -q", "1 passed in 0.1s", 0, T)).toMatchObject({ tool: "pytest", pass: 1 }); + expect(classifyReceipt("pnpm -r test", "Tests: 2 failed, 12 passed, 14 total", 1, T)).toMatchObject({ + tool: "npm/pnpm/yarn test", pass: 12, fail: 2, + }); + expect(classifyReceipt("pnpm --filter api test", " Tests 1 failed | 1 passed (2)", 1, T)).toMatchObject({ + tool: "npm/pnpm/yarn test", pass: 1, fail: 1, + }); + expect(classifyReceipt("bun run test", " 8 pass\n 0 fail\n", 0, T)).toMatchObject({ + tool: "npm/pnpm/yarn test", pass: 8, fail: 0, + }); +}); + +test("D5: cargo test's own 'error: N targets failed' line is a fail floor a truncating pipe can't hide", () => { + const truncated = "error: 1 target failed:\n `tests::a`"; + expect(classifyReceipt("cargo test --no-fail-fast", truncated, 1, T)).toMatchObject({ tool: "cargo test", fail: 1 }); +}); diff --git a/test/receipt-runners-structural.test.ts b/test/receipt-runners-structural.test.ts new file mode 100644 index 0000000..cddfc59 --- /dev/null +++ b/test/receipt-runners-structural.test.ts @@ -0,0 +1,49 @@ +import { test, expect } from "bun:test"; +import { classifyReceipt } from "../src/tracking/receipts"; + +const T = 1_000_000_000_000; + +test("structural: a trailing echo of fake counts never resets a real runner into a receipt", () => { + expect(classifyReceipt('bun test; echo " 12 pass"; echo " 0 fail"', "", 0, T)).toBeNull(); + expect(classifyReceipt('pytest; echo "===== 12 passed in 0.1s ====="', "", 0, T)).toBeNull(); + expect(classifyReceipt('mypy src; echo "Success: no issues found in 1 source file"', "", 0, T)).toBeNull(); +}); + +test("structural: any output redirection after the runner in its own segment nulls it", () => { + expect(classifyReceipt("bunx tsc --noEmit > tsc.log 2>&1; :", "", 0, T)).toBeNull(); + expect(classifyReceipt("go vet ./... > v.log 2>&1", "", 0, T)).toBeNull(); + expect(classifyReceipt("bun test > out.txt", " 1 pass\n 0 fail\n", 0, T)).toBeNull(); +}); + +test("structural: the runner must be the LAST list segment, a pipe is not a list separator", () => { + expect(classifyReceipt("cd x && bun test", " 3 pass\n 0 fail\n", 0, T)).toMatchObject({ tool: "bun test", pass: 3, fail: 0 }); + expect(classifyReceipt("bun test 2>&1 | tail -5", " 3 pass\n 0 fail\n", 0, T)).toMatchObject({ tool: "bun test", pass: 3, fail: 0 }); + expect(classifyReceipt("bun test && git commit -m x", "", 0, T)).toBeNull(); +}); + +test("regression: parseGoTest ignores [no test files] lines and [no tests to run] ok lines", () => { + const mixed = "? \tex/foo\t[no test files]\nok \tex/bar\t0.012s"; + expect(classifyReceipt("go test ./...", mixed, 0, T)).toMatchObject({ tool: "go test", pass: 1 }); + + const onlyNoFiles = "? \tex/foo\t[no test files]\n? \tex/bar\t[no test files]"; + const r = classifyReceipt("go test ./...", onlyNoFiles, 0, T); + expect(r?.pass === undefined || r.pass === 0).toBe(true); +}); + +test("regression: php artisan test routes Pest-style output to parsePest first", () => { + expect(classifyReceipt("php artisan test", "Tests: 2 passed (2 assertions)", 0, T)).toMatchObject({ tool: "phpunit", pass: 2, fail: 0 }); + expect(classifyReceipt("php artisan test", "Tests: 1 failed, 1 passed (3 assertions)", 1, T)).toMatchObject({ tool: "phpunit", pass: 1, fail: 1 }); +}); + +test("regression: swift test --skip-build still runs the tests (no longer informational)", () => { + expect(classifyReceipt("swift test --skip-build", "Executed 12 tests, with 0 failures", 0, T)).toMatchObject({ + tool: "swift test", + pass: 12, + fail: 0, + }); +}); + +test("regression: ANSI-colorized bun summary parses identically to plain text", () => { + const out = "\x1b[32m 12 pass\x1b[0m\n 0 fail"; + expect(classifyReceipt("bun test", out, 0, T)).toMatchObject({ tool: "bun test", pass: 12, fail: 0 }); +}); diff --git a/test/receipt-runners.test.ts b/test/receipt-runners.test.ts new file mode 100644 index 0000000..5380b7f --- /dev/null +++ b/test/receipt-runners.test.ts @@ -0,0 +1,96 @@ +import { test, expect } from "bun:test"; +import { classifyReceipt } from "../src/tracking/receipts"; + +const T = 1_000_000_000_000; + +test("bun test: pass/fail", () => { + expect(classifyReceipt("bun test", " 8 pass\n 0 fail\n", 0, T)).toMatchObject({ kind: "test", tool: "bun test", pass: 8, fail: 0 }); + expect(classifyReceipt("bun test", " 0 pass\n 8 fail\n", 1, T)).toMatchObject({ kind: "test", tool: "bun test", pass: 0, fail: 8 }); +}); + +test("vitest: pass/fail", () => { + expect(classifyReceipt("vitest run", " Tests 2 passed (2)", 0, T)).toMatchObject({ tool: "vitest", pass: 2, fail: 0 }); + expect(classifyReceipt("vitest run", " Tests 1 failed | 1 passed (2)", 1, T)).toMatchObject({ tool: "vitest", pass: 1, fail: 1 }); +}); + +test("jest: pass/fail", () => { + expect(classifyReceipt("jest", "Tests: 14 passed, 14 total", 0, T)).toMatchObject({ tool: "jest", pass: 14, fail: 0 }); + expect(classifyReceipt("jest", "Tests: 2 failed, 12 passed, 14 total", 1, T)).toMatchObject({ tool: "jest", pass: 12, fail: 2 }); +}); + +test("npm/pnpm/yarn test: falls back through vitest -> jest -> bun formats", () => { + expect(classifyReceipt("npm test", "Tests: 2 failed, 12 passed, 14 total", 1, T)).toMatchObject({ tool: "npm/pnpm/yarn test", pass: 12, fail: 2 }); + expect(classifyReceipt("npm run test", " Tests 1 failed | 1 passed (2)", 1, T)).toMatchObject({ tool: "npm/pnpm/yarn test", pass: 1, fail: 1 }); +}); + +test("pytest: pass, fail sums failed+errors", () => { + expect(classifyReceipt("pytest -q", "===== 12 passed in 0.53s =====", 0, T)).toMatchObject({ tool: "pytest", pass: 12, fail: undefined }); + expect(classifyReceipt("pytest -q", "===== 2 failed, 12 passed, 1 skipped in 0.53s =====", 1, T)).toMatchObject({ tool: "pytest", pass: 12, fail: 2 }); +}); + +test("go test: fail from --- FAIL: lines, else package FAIL line; pass = count of ok lines", () => { + // D2 fix: pass is now POSITIVE evidence (count of `^ok ` lines), not + // undefined-by-default — a single successful package reports pass: 1. + expect(classifyReceipt("go test ./...", "ok \tpkg\t0.1s", 0, T)).toMatchObject({ tool: "go test", pass: 1, fail: 0 }); + expect(classifyReceipt("go test ./...", "--- FAIL: TestX\n--- FAIL: TestY\nFAIL\tpkg\t0.1s", 1, T)).toMatchObject({ tool: "go test", fail: 2 }); +}); + +test("cargo test: sums every 'test result:' line", () => { + const out = "test result: ok. 12 passed; 0 failed; 0 ignored\ntest result: FAILED. 3 passed; 1 failed; 0 ignored"; + expect(classifyReceipt("cargo test", out, 1, T)).toMatchObject({ tool: "cargo test", pass: 15, fail: 1 }); +}); + +test("phpunit / php artisan test: OK on success, Tests:/Failures: on failure", () => { + expect(classifyReceipt("phpunit", "OK (12 tests, 30 assertions)", 0, T)).toMatchObject({ tool: "phpunit", pass: 12, fail: 0 }); + expect(classifyReceipt("php artisan test", "FAILURES!\nTests: 12, Assertions: 30, Failures: 2.", 1, T)).toMatchObject({ tool: "phpunit", pass: 10, fail: 2 }); +}); + +test("pest: Tests: N failed, M passed (...)", () => { + expect(classifyReceipt("pest", "Tests: 2 failed, 10 passed (30 assertions)", 1, T)).toMatchObject({ tool: "pest", pass: 10, fail: 2 }); +}); + +test("swift test: XCTest last-summary + swift-testing summary, summed", () => { + const out = "Executed 5 tests, with 1 failure\nTest run with 3 tests in 1 suite passed after 0.2 seconds"; + expect(classifyReceipt("swift test", out, 1, T)).toMatchObject({ tool: "swift test", pass: 4 + 3, fail: 1 }); +}); + +test("dart/flutter test: pass/fail banner only", () => { + expect(classifyReceipt("flutter test", "All tests passed!", 0, T)).toMatchObject({ tool: "dart/flutter test", fail: 0 }); + expect(classifyReceipt("dart test", "Some tests failed.", 1, T)).toMatchObject({ tool: "dart/flutter test", fail: 1 }); +}); + +test("mypy: Found N errors / Success", () => { + expect(classifyReceipt("mypy .", "Success: no issues found in 5 source files", 0, T)).toMatchObject({ tool: "mypy", fail: 0 }); + expect(classifyReceipt("mypy .", "Found 3 errors in 2 files", 1, T)).toMatchObject({ tool: "mypy", fail: 3 }); +}); + +test("pyright: N errors, M warnings", () => { + expect(classifyReceipt("pyright", "1 error, 0 warnings, 0 informations", 1, T)).toMatchObject({ tool: "pyright", fail: 1 }); +}); + +test("phpstan: [ERROR] Found N errors / [OK]", () => { + expect(classifyReceipt("phpstan analyse", "[OK] No errors", 0, T)).toMatchObject({ tool: "phpstan", fail: 0 }); + expect(classifyReceipt("phpstan analyse", "[ERROR] Found 3 errors", 1, T)).toMatchObject({ tool: "phpstan", fail: 3 }); +}); + +test("go vet / go build / cargo check / cargo clippy / swift build: exit code only, no counts", () => { + expect(classifyReceipt("go vet ./...", "", 0, T)).toMatchObject({ tool: "go vet", kind: "tsc" }); + expect(classifyReceipt("go build ./...", "", 0, T)).toMatchObject({ tool: "go build", kind: "tsc" }); + expect(classifyReceipt("cargo check", "", 0, T)).toMatchObject({ tool: "cargo check", kind: "tsc" }); + expect(classifyReceipt("cargo clippy", "", 0, T)).toMatchObject({ tool: "cargo clippy", kind: "tsc" }); + expect(classifyReceipt("swift build", "", 0, T)).toMatchObject({ tool: "swift build", kind: "tsc" }); +}); + +test("anchoring negatives: an argument/quoted mention is never a receipt", () => { + expect(classifyReceipt('git commit -m "fix: jest flake"', "", 0, T)).toBeNull(); + expect(classifyReceipt("echo pytest done", "", 0, T)).toBeNull(); + expect(classifyReceipt('grep -rn "cargo test" docs/', "", 0, T)).toBeNull(); +}); + +test("anchoring positives: wrappers and prefixes still resolve to the right tool", () => { + expect(classifyReceipt("cd api && vendor/bin/phpunit", "OK (1 tests, 1 assertions)", 0, T)).toMatchObject({ tool: "phpunit" }); + expect(classifyReceipt("uv run pytest -q", "1 passed in 0.1s", 0, T)).toMatchObject({ tool: "pytest" }); + expect(classifyReceipt("time bun test", " 1 pass\n 0 fail\n", 0, T)).toMatchObject({ tool: "bun test" }); + expect(classifyReceipt("FOO=1 go test ./...", "ok\tpkg\t0.1s", 0, T)).toMatchObject({ tool: "go test" }); + expect(classifyReceipt("bun test 2>&1 | tail -5", " 1 pass\n 0 fail\n", 0, T)).toMatchObject({ tool: "bun test" }); +}); diff --git a/test/receipts.test.ts b/test/receipts.test.ts index ea83483..2fe9285 100644 --- a/test/receipts.test.ts +++ b/test/receipts.test.ts @@ -25,18 +25,18 @@ function session(sid: string): { home: string; stateDir: string; file: string } } test("classifyReceipt: parses bun test pass/fail counts", () => { - expect(classifyReceipt("bun test", " 8 pass\n 0 fail\n", 0, T)).toEqual({ kind: "test", exitCode: 0, pass: 8, fail: 0, ts: T }); + expect(classifyReceipt("bun test", " 8 pass\n 0 fail\n", 0, T)).toEqual({ kind: "test", tool: "bun test", exitCode: 0, pass: 8, fail: 0, ts: T }); }); -test("classifyReceipt: tsc carries exit code (no counts); non-verification is null", () => { - expect(classifyReceipt("bunx tsc --noEmit", "", 0, T)).toEqual({ kind: "tsc", exitCode: 0, ts: T }); +test("classifyReceipt: tsc carries exit code + output-parsed fail count; non-verification is null", () => { + expect(classifyReceipt("bunx tsc --noEmit", "", 0, T)).toEqual({ kind: "tsc", tool: "tsc", exitCode: 0, pass: undefined, fail: 0, ts: T }); expect(classifyReceipt("ls -la", "whatever", 0, T)).toBeNull(); }); test("captureReceipt: persists a parsed receipt into the signed track", async () => { const file = join(tmp("fh-rcpt-cap-"), "track.json"); await captureReceipt(file, "bun test", " 12 pass\n 3 fail\n", 1, T); - expect((await loadTrack(file)).receipts?.[0]).toEqual({ kind: "test", exitCode: 1, pass: 12, fail: 3, ts: T }); + expect((await loadTrack(file)).receipts?.[0]).toEqual({ kind: "test", tool: "bun test", exitCode: 1, pass: 12, fail: 3, ts: T }); }); test("validateTaskSolid: a fresh passing receipt lets completion pass", async () => {