Repository navigation
v1.3: separate case outcome from coverage verdict, split policy actions out of the verdict vocabulary, fix payload digest to §2.3 - #8
Merged
Conversation
…ns out of the verdict vocabulary; fix payload digest to §2.3 Three defects found while reviewing the case set for FG-TIDA/use-cases#27 against the 2026-10-23 review comment. None of them is a change of position; all three are the same position applied consistently. 1) `verdict` is six-domain evidence coverage, not a case outcome. Case A — a valid, current, in-registration credential — reports "Insufficient evidence（证据不足）" because only 2 of the 6 domains are exercised. A program that reads `verdict` as the conclusion gets a false negative on the positive case. v1.3 adds `outcome.verification` (verified / unresolved / rejected) as the field to read, keeps `verdict` with its original meaning, and states both scopes inside every conclusion in a machine-readable `scopes` object. The field was not wrong; it was being asked the wrong question. 2) The verdict vocabulary mixed policy actions with verification states. `quarantined` was a fourth verdict, but it describes what an operator does with an unsettled field, not what was verified. It is now `policy_action: exclude`, and `policy_action` reuses the four operator meanings already published in EVALUATION-STANDARD.md §2.2 (allow / defer / deny / exclude) instead of introducing a fifth word set. `reason_code` becomes a required machine-readable field (none / authorization_unresolved / authorization_rejected / status_unavailable). 3) The payload digest did not match the convention in EVALUATION-STANDARD.md §2.3. The v1.2 implementation computed the digest with default JSON separators and insertion order; §2.3 specifies sort_keys=True with compact separators. Any third party following §2.3 could not reproduce a published digest. This was verified against the v1.2 files before changing anything: recomputed under §2.3 they hash to ab56339d…, be1677e8…, ff4ff95a…, 6c37e632…, none of which matches the published values. The implementation now matches the text. All four digests change (schema lgd-eval/conclusion/v1.3): Case A fa8b1420cca29307 Case B1 a353ad64f911030a Case B2 d709c4ec85ef5d6e Case C 982c390107058485 Selftest: 35 invariants (was 15). New invariants cover B1/B2 outcome separation, reason_code consistency with the verdict, rejection of `quarantined` as a verdict, digest equality with an independently computed §2.3 value, and key-order independence. Reproduction: from a clean copy of this tree with the conclusions deleted, re-running the grader on the four inputs reproduces all four published digests (4/4 MATCH), and the selftest passes in that copy. Also in scope: demo/evidence-grading-demo.html was stale (three cases, "12 invariants", and the pre-split digests) and is regenerated from the four conclusions; the "external verification" wording is replaced with what actually happened (a reproduction run from a fresh download, made by the maintainers), since claiming third-party verification for our own run would be the exact overstatement this project exists to avoid. No files removed. The `three_cases_outcome_flow.svg` asset keeps its file name so that links from existing threads do not break; the caption and the page state that it shows four cases.
🤖 LGD-Eval PR Review BotBlocking issues/阻断: 0 — Warnings/提示: 2
|
21 of 42 tasks
The digest quoted in three places was computed over the local CRLF copy: 34,795 bytes + 647 lines = 35,442, which is the size that was hashed. No historical version of tools/evidence-grader.py has that digest. Replaced with the SHA-256 of the bytes GitHub actually serves, and stated the measurement basis inline so the mistake cannot recur silently.
🤖 LGD-Eval PR Review BotBlocking issues/阻断: 0 — Warnings/提示: 2
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
v1.3 — case outcome, one vocabulary, corrected digest
Responds to the review in FG-TIDA/use-cases#27 (2026-10-23).
What this PR is not
It is not a change of position. B1/B2 were already split on 2026-10-16 following the
themes#42 review; that split is not being
re-argued here. This PR fixes three places where the repository failed to say what it
already meant, and one place where it contradicted its own published standard.
Three defects, three fixes
1.
verdictwas being read as the case outcomeverdictcounts how many of the six evaluation domains reach grade V/S. A case set thatexercises two domains is therefore
Insufficient evidence— including Case A, a valid,current, in-registration credential whose authorization verdict is
verified. Anyonereading
verdictas "the conclusion" gets a false negative on the one positive case.v1.3 adds
outcome.verification(verified/unresolved/rejected) as the field toread, keeps
verdictwith its original meaning, and carries ascopesobject insideevery conclusion stating what each summary field does and does not answer.
2. Policy actions were inside the verdict vocabulary
quarantinedused to be a fourth authorization verdict, but it describes what anoperator does with an unsettled field — not what was verified. Verdicts are now three
(
verified/unresolved/rejected), and policy actions live in their own field,policy_action, reusing the four operator meanings this repository already publishes inEVALUATION-STANDARD.md §2.2:
allowverifieddeferunresolveddenyrejectedexcluderejectis deliberately unavailable as an action name, so that the action and theverdict cannot be confused in a single glance.
reason_codebecomes required andmachine-readable.
3. The digest did not match §2.3
§2.3 has specified
sort_keys=Truewith compact separators since v0.1. The v1.2implementation used default separators and insertion order. Recomputed under §2.3, the
four v1.2 conclusions hash to
ab56339d…,be1677e8…,ff4ff95a…,6c37e632…— noneof them the published values. A third party following the standard could not reproduce
any digest we published. The implementation now matches the text.
New digests
reason_codepolicy_actionverifiednoneallowfa8b1420cca29307unresolvedauthorization_unresolveddefera353ad64f911030arejectedauthorization_rejecteddenyd709c4ec85ef5d6eunresolvedstatus_unavailabledefer982c390107058485Verification
reason_codeconsistency with the verdict, rejection of
quarantinedas a verdict, digest equalityagainst an independently computed §2.3 value, and key-order independence of the digest.
inputs, compare — 4/4 MATCH, and the selftest passes in that copy.
.mdper case alongside each JSON), noneremoved.
Review notes
demo/evidence-grading-demo.htmlwas stale (three cases, "12 invariants", pre-splitdigests). It is regenerated from the four conclusions, and its "external verification"
wording is replaced with what actually happened — a reproduction run from a fresh
download, made by the maintainers. Claiming third-party verification for our own run
would be precisely the overstatement this project exists to catch.
assets/three_cases_outcome_flow.svgkeeps its file name so existing links fromthreads do not break; the page and the caption state that it shows four cases.
the summary fields, the vocabulary and the digest computation changed.