feat(analyze): the enhancement non-verdicts are visible, out of the prompt, counted, and migrated on re-enhancement (#611) - #635
Merged
Conversation
gadievron
requested review from
dgeyshis,
shahar-davidson and
sounil
as code owners
September 14, 2026 21:24
…rompt, counted, and migrated on re-enhancement (#611) Enhancement non-verdicts (the sentinel set incomplete/error/unknown) leaked into analysis, disappeared under filtering, and survived re-enhancement in analyze checkpoints. Four fixes, one sentinel contract (SENTINEL_CLASSIFICATIONS in core/verdict_taxonomy — the vocabulary's home; a circular-import found in-flight moved it out of analysis_core): 1. THE SHAPE: EnhanceResult carries incomplete_count + total_units (the three-bucket identity total = units_enhanced + incomplete + errors, recoverable without summing classifications — which omits errors by construction); the step-report summary threads them. 2. THE HINT: the sentinel set omits the Stage-1 'Pre-analysis hint' line ENTIRELY — a non-verdict presented to the model as a classification is the wrong channel; the durable signal is the per-row stamp. 3. THE CENSUS + COUNTER: a sentinel-stamped unit is NOT classified for the --exploitable warning's census (an all-sentinel dataset fires the loud warning); the metrics publish enhance_unclassified_analyzed from the per-row stamp — NOT a third analyze state (#293 stands: an input-provenance counter, placed after the consistency check's counts rebuild). The --exploitable filter's own retention policy is deliberately unchanged (the written contract; the drop is now VISIBLE). 4. THE RECOVERY BLOCKER: the two-sided _cp_is_stale predicate — a sentinel-stamped analyze checkpoint re-analyzes when re-enhancement CHANGES its classification to completed; a still-sentinel checkpoint stays adopted (matching the dataset). _seed_summary excludes the stale row from completed (the migration-resume double-count) while its usage still accumulates (the spend happened — the same shape as the errored-row gate). serves: I-ana, I-enh, I-out. de-balances-checked: the #293 three-bucket ruling stands; the counter's agentic-only scope documented (the single-shot row carries no stamp today). Refs #611
…te hardened, all three sentinels pinned, the CHANGELOG (#611) The combined 2d+DU seat's findings, all dispositioned: - the --exploitable warning text said 'NONE carry a security_classification' — FALSE on the new all-sentinel branch (every unit carries one; none carries a COMPLETED one). Reworded to name both conditions (unenhanced or non-verdict sentinel). - _cp_is_stale gained the isinstance(str) guard its sibling _result_security_classification already carried (a non-str/unhashable stamp must never raise out of the predicate); the behavioral truth-table test now covers the edge terms: the unstamped checkpoint (single-shot row), result:null, a nested-dict stamp, a None stamp — never stale, never a raise. - the hint-omission test now drives ALL THREE sentinel values through the REAL get_analysis_prompt (was 'incomplete' only). - the CHANGELOG entry added (the repo's stated convention) with the scope disclosures: the counter is agentic-only today and publishes into results.json's metrics only (not AnalysisMetrics/the step report/Go types); incomplete_count counts the 'incomplete' bucket only — the sentinel vocabulary and the identity buckets are two documented views. RED on base: signature-RED (the collection ImportError on SENTINEL_CLASSIFICATIONS — receipt red-receipt-635.log; the honest shape, disclosed); GREEN at head: 11/11. The full suite at the rebased head: 4192 passed + 34 skipped + the 2 pre-existing SDK-pin drift failures (suite-635.log — the body's '4160 + 33 skipped' was the pre-rebase authoring-env figure; no artifact matched it). Semgrep: 0 findings.
… CLI writer carries the identity, the guards everywhere, the pins tightened (#611) The two deep-refute seats' findings, all dispositioned: - the ORPHAN #615 heading in the CHANGELOG (an insertion artifact of the pipeline round — a bodyless section that would duplicate on the #636 merge) deleted. - the standalone CLI's enhance step summary now threads incomplete_count/total_units (the scanner's writer already did — the two-writer drift the #615 round closed for its own field); the pin covers BOTH writers. - _unit_security_classification and the prompt gate gained the isinstance(str) guard the predicate readers already carried (a non-str stamp raised TypeError at 2 of the 4 membership sites where base tolerated it). - the census test's mirror now backed by SOURCE pins on the real condition (the sentinel exclusion in analyzer.py) and the real loop's branch (the incomplete_count increment inside cls == 'incomplete'). 11/11 at head; ruff clean.
gadievron
force-pushed
the
fix/issue611-sentinel-handling
branch
from
September 17, 2026 10:20
843d2b9 to
0e4e8a5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #611.
What was wrong
Enhancement non-verdicts (45 agent-loop incompletes + 12 raise-class errors of 1,730 units in the referenced run) leaked into analysis, disappeared under filtering, and survived re-enhancement in analyze checkpoints — four gaps sharing one missing contract: the enhance sentinel values are not verdicts, and nothing treated them accordingly.
The fix (four sub-fixes, one sentinel contract)
The shared constant:
SENTINEL_CLASSIFICATIONS = {incomplete, error, unknown}incore/verdict_taxonomy(the vocabulary's home; both consumers import directly).EnhanceResultcarriesincomplete_count+total_units— the three-bucket identity (total = units_enhanced + incomplete + errors) is recoverable without summingclassifications(which omits errors by construction); BOTH step-report summary writers thread them (the scanner's pipeline summary and the standalone CLI's enhance summary).get_analysis_prompt).--exploitablewarning's census (an all-sentinel dataset fires the loud warning — the same un-enhanced shape; the warning text names both conditions: unenhanced or non-verdict sentinel); the metrics publishenhance_unclassified_analyzedfrom the per-row stamp — not a third analyze state (Progress/summary counters use a two-brancherrorvselsesplit over three states, so incomplete work is counted as completed (5 sites) #293 stands), placed after the consistency check's counts rebuild so it survives correction-runs. The filter's own retention policy is deliberately unchanged (the written contract; the drop is now visible)._cp_is_stalepredicate — a sentinel-stamped analyze checkpoint re-analyzes when re-enhancement changes its classification to completed; a still-sentinel checkpoint stays adopted (matching the run's own dataset)._seed_summaryexcludes the stale row fromcompleted(the migration-resume double-count,completed > total) while its usage still accumulates (the spend happened — the same shape as the errored-row gate). The predicate carries theisinstance(str)guard at every membership site (a non-str stamp from a hand-edited dataset never raises;_unit_security_classificationand the prompt gate included).Evidence
ImportErroronSENTINEL_CLASSIFICATIONS(the constant did not exist; receiptred-receipt-635.log; the shape disclosed: it proves the constant, and the behavioral rows' base failure is masked by the collection error).pytest -q tests/test_issue611_sentinel_handling.py→11 passed); the full suite locally:pytest -q→2 failed, 4192 passed, 34 skipped, 1 warning in 160.63s— the 2 aretest_llm_sdk_contract_floorSDK-pin host drift, identical on master; CI is the authoritative suite. Ruff clean; Semgrep 0 findings.result: null, a nested-dict stamp, a None stamp — never stale, never a raise); the migration-resume seed (the stale row excluded from completed, its usage retained; the legacy no-unitssignature unchanged); the hint gate for all three sentinels + the completed control; the three-bucket identity.analyzer.py; the loop'sincomplete_countincrement inside thecls == "incomplete"branch; both summary writers' field threading.The de-balancing check (the intent trace)
serves: I-ana, I-out;de-balances-checked: the filter's retention line is byte-identical to master (only the census that drives the warning changed); the hint gate only widens the omission (real classifications render identically); a completed-classification checkpoint's adoption is unchanged.The adversarial record
The authoring rounds (the seed double-count HIGH, the counter-vanishing-on-correction catch, the tautology extractions) + this PR-processing round (a combined 2d+DU seat + a 4-seat panel — 4/4 approve — + two deep-refute seats, every finding dispositioned): the orphan
#615CHANGELOG heading (an insertion artifact — deleted); the CLI writer's missing identity fields (the two-writer drift — added,getattr-guarded for stub callers; both writers pinned); the warning text made true on the all-sentinel branch; theisinstanceguard at the remaining 2 membership sites; the census and loop source pins (the mirror tests alone could not catch a regression); the CHANGELOG entry with the scope disclosures.Notes for the reviewer
security_classificationstamp (extending the stamp tollm_contextis a separate change).results.json's metrics only — notAnalysisMetrics, the step report, or the Go types; andverifier.pyrebuilds metrics from scratch, soresults_verified.jsondrops it too.incomplete_countcounts the"incomplete"bucket only (errors carry their ownerror_count) — the sentinel vocabulary and the identity buckets are two documented views of the same stage.--exploitablefilter still drops sentinel units — the issue's "retain" ask is a policy reversal against the filter's written contract; this PR makes the drop visible and leaves the retention question to a deliberate maintainer decision.checkpoint-statusremains dataset-blind (it reports a stale row as completed while analyze re-queues it — the divergence is the stale-row count on a migration resume).