Skip to content

feat(analyze): the enhancement non-verdicts are visible, out of the prompt, counted, and migrated on re-enhancement (#611) - #635

Merged
gadievron merged 4 commits into
masterfrom
fix/issue611-sentinel-handling
Sep 17, 2026
Merged

gadievron merged 4 commits into
masterfrom
fix/issue611-sentinel-handling

Conversation

@gadievron

@gadievron gadievron commented Sep 14, 2026

Copy link
Copy Markdown
Collaborator

Fixes #611.

What was wrong

Enhancement non-verdicts (45 agent-loop incompletes + 12 raise-class errors of 1,730 units in the referenced run) leaked into analysis, disappeared under filtering, and survived re-enhancement in analyze checkpoints — four gaps sharing one missing contract: the enhance sentinel values are not verdicts, and nothing treated them accordingly.

The fix (four sub-fixes, one sentinel contract)

The shared constant: SENTINEL_CLASSIFICATIONS = {incomplete, error, unknown} in core/verdict_taxonomy (the vocabulary's home; both consumers import directly).

  1. The shape: EnhanceResult carries incomplete_count + total_units — the three-bucket identity (total = units_enhanced + incomplete + errors) is recoverable without summing classifications (which omits errors by construction); BOTH step-report summary writers thread them (the scanner's pipeline summary and the standalone CLI's enhance summary).
  2. The hint: the sentinel set omits the Stage-1 "Pre-analysis hint" line entirely — a non-verdict presented to the model as a classification is the wrong channel; the durable signal is the per-row stamp (pinned for all three sentinel values through the real get_analysis_prompt).
  3. The census + counter: a sentinel-stamped unit is not classified for the --exploitable warning's census (an all-sentinel dataset fires the loud warning — the same un-enhanced shape; the warning text names both conditions: unenhanced or non-verdict sentinel); the metrics publish enhance_unclassified_analyzed from the per-row stamp — not a third analyze state (Progress/summary counters use a two-branch error vs else split over three states, so incomplete work is counted as completed (5 sites) #293 stands), placed after the consistency check's counts rebuild so it survives correction-runs. The filter's own retention policy is deliberately unchanged (the written contract; the drop is now visible).
  4. The recovery blocker: the two-sided _cp_is_stale predicate — a sentinel-stamped analyze checkpoint re-analyzes when re-enhancement changes its classification to completed; a still-sentinel checkpoint stays adopted (matching the run's own dataset). _seed_summary excludes the stale row from completed (the migration-resume double-count, completed > total) while its usage still accumulates (the spend happened — the same shape as the errored-row gate). The predicate carries the isinstance(str) guard at every membership site (a non-str stamp from a hand-edited dataset never raises; _unit_security_classification and the prompt gate included).

Evidence

  • RED (pristine master, this file copied in): signature-RED — the collection ImportError on SENTINEL_CLASSIFICATIONS (the constant did not exist; receipt red-receipt-635.log; the shape disclosed: it proves the constant, and the behavioral rows' base failure is masked by the collection error).
  • GREEN: 11/11 on the file (pytest -q tests/test_issue611_sentinel_handling.py11 passed); the full suite locally: pytest -q2 failed, 4192 passed, 34 skipped, 1 warning in 160.63s — the 2 are test_llm_sdk_contract_floor SDK-pin host drift, identical on master; CI is the authoritative suite. Ruff clean; Semgrep 0 findings.
  • The behavioral receipts: the four-cell + edge-term staleness truth table (the unstamped checkpoint, result: null, a nested-dict stamp, a None stamp — never stale, never a raise); the migration-resume seed (the stale row excluded from completed, its usage retained; the legacy no-units signature unchanged); the hint gate for all three sentinels + the completed control; the three-bucket identity.
  • The source pins (tightly matched, regression-catching): the census's sentinel exclusion in analyzer.py; the loop's incomplete_count increment inside the cls == "incomplete" branch; both summary writers' field threading.

The de-balancing check (the intent trace)

serves: I-ana, I-out; de-balances-checked: the filter's retention line is byte-identical to master (only the census that drives the warning changed); the hint gate only widens the omission (real classifications render identically); a completed-classification checkpoint's adoption is unchanged.

The adversarial record

The authoring rounds (the seed double-count HIGH, the counter-vanishing-on-correction catch, the tautology extractions) + this PR-processing round (a combined 2d+DU seat + a 4-seat panel — 4/4 approve — + two deep-refute seats, every finding dispositioned): the orphan #615 CHANGELOG heading (an insertion artifact — deleted); the CLI writer's missing identity fields (the two-writer drift — added, getattr-guarded for stub callers; both writers pinned); the warning text made true on the all-sentinel branch; the isinstance guard at the remaining 2 membership sites; the census and loop source pins (the mirror tests alone could not catch a regression); the CHANGELOG entry with the scope disclosures.

Notes for the reviewer

  • The counter's scope is agentic-only today — the single-shot row carries no security_classification stamp (extending the stamp to llm_context is a separate change).
  • The counter publishes into results.json's metrics only — not AnalysisMetrics, the step report, or the Go types; and verifier.py rebuilds metrics from scratch, so results_verified.json drops it too.
  • incomplete_count counts the "incomplete" bucket only (errors carry their own error_count) — the sentinel vocabulary and the identity buckets are two documented views of the same stage.
  • The --exploitable filter still drops sentinel units — the issue's "retain" ask is a policy reversal against the filter's written contract; this PR makes the drop visible and leaves the retention question to a deliberate maintainer decision.
  • The adoption/seed id-key divergence is pre-existing (analyze resume narration counts error checkpoint rows as 'already done': prints '0 units to process' while one runs, progress counter reaches 3/2 #435's lane). checkpoint-status remains dataset-blind (it reports a stale row as completed while analyze re-queues it — the divergence is the stale-row count on a migration resume).

Comment thread libs/openant-core/tests/test_issue611_sentinel_handling.py Fixed
…rompt, counted, and migrated on re-enhancement (#611)

Enhancement non-verdicts (the sentinel set incomplete/error/unknown)
leaked into analysis, disappeared under filtering, and survived
re-enhancement in analyze checkpoints. Four fixes, one sentinel
contract (SENTINEL_CLASSIFICATIONS in core/verdict_taxonomy — the
vocabulary's home; a circular-import found in-flight moved it out of
analysis_core):

1. THE SHAPE: EnhanceResult carries incomplete_count + total_units
   (the three-bucket identity total = units_enhanced + incomplete +
   errors, recoverable without summing classifications — which omits
   errors by construction); the step-report summary threads them.
2. THE HINT: the sentinel set omits the Stage-1 'Pre-analysis hint'
   line ENTIRELY — a non-verdict presented to the model as a
   classification is the wrong channel; the durable signal is the
   per-row stamp.
3. THE CENSUS + COUNTER: a sentinel-stamped unit is NOT classified
   for the --exploitable warning's census (an all-sentinel dataset
   fires the loud warning); the metrics publish
   enhance_unclassified_analyzed from the per-row stamp — NOT a
   third analyze state (#293 stands: an input-provenance counter,
   placed after the consistency check's counts rebuild). The
   --exploitable filter's own retention policy is deliberately
   unchanged (the written contract; the drop is now VISIBLE).
4. THE RECOVERY BLOCKER: the two-sided _cp_is_stale predicate — a
   sentinel-stamped analyze checkpoint re-analyzes when
   re-enhancement CHANGES its classification to completed; a
   still-sentinel checkpoint stays adopted (matching the dataset).
   _seed_summary excludes the stale row from completed (the
   migration-resume double-count) while its usage still accumulates
   (the spend happened — the same shape as the errored-row gate).

serves: I-ana, I-enh, I-out. de-balances-checked: the #293
three-bucket ruling stands; the counter's agentic-only scope
documented (the single-shot row carries no stamp today).

Refs #611
…te hardened, all three sentinels pinned, the CHANGELOG (#611)

The combined 2d+DU seat's findings, all dispositioned:

- the --exploitable warning text said 'NONE carry a
  security_classification' — FALSE on the new all-sentinel branch (every
  unit carries one; none carries a COMPLETED one). Reworded to name
  both conditions (unenhanced or non-verdict sentinel).
- _cp_is_stale gained the isinstance(str) guard its sibling
  _result_security_classification already carried (a non-str/unhashable
  stamp must never raise out of the predicate); the behavioral
  truth-table test now covers the edge terms: the unstamped checkpoint
  (single-shot row), result:null, a nested-dict stamp, a None stamp —
  never stale, never a raise.
- the hint-omission test now drives ALL THREE sentinel values through
  the REAL get_analysis_prompt (was 'incomplete' only).
- the CHANGELOG entry added (the repo's stated convention) with the
  scope disclosures: the counter is agentic-only today and publishes
  into results.json's metrics only (not AnalysisMetrics/the step
  report/Go types); incomplete_count counts the 'incomplete' bucket
  only — the sentinel vocabulary and the identity buckets are two
  documented views.

RED on base: signature-RED (the collection ImportError on
SENTINEL_CLASSIFICATIONS — receipt red-receipt-635.log; the honest
shape, disclosed); GREEN at head: 11/11. The full suite at the rebased
head: 4192 passed + 34 skipped + the 2 pre-existing SDK-pin drift
failures (suite-635.log — the body's '4160 + 33 skipped' was the
pre-rebase authoring-env figure; no artifact matched it). Semgrep: 0
findings.
… CLI writer carries the identity, the guards everywhere, the pins tightened (#611)

The two deep-refute seats' findings, all dispositioned:

- the ORPHAN #615 heading in the CHANGELOG (an insertion artifact of
  the pipeline round — a bodyless section that would duplicate on the
  #636 merge) deleted.
- the standalone CLI's enhance step summary now threads
  incomplete_count/total_units (the scanner's writer already did — the
  two-writer drift the #615 round closed for its own field); the pin
  covers BOTH writers.
- _unit_security_classification and the prompt gate gained the
  isinstance(str) guard the predicate readers already carried (a
  non-str stamp raised TypeError at 2 of the 4 membership sites where
  base tolerated it).
- the census test's mirror now backed by SOURCE pins on the real
  condition (the sentinel exclusion in analyzer.py) and the real loop's
  branch (the incomplete_count increment inside cls == 'incomplete').

11/11 at head; ruff clean.
@gadievron
gadievron force-pushed the fix/issue611-sentinel-handling branch from 843d2b9 to 0e4e8a5 Compare September 17, 2026 10:20
@gadievron
gadievron merged commit b74de4b into master Sep 17, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enhancement non-verdicts leak into analysis hints, disappear under --exploitable filtering, and survive re-enhancement in analyze checkpoints

1 participant