Skip to content

feat(enhance): the degenerate exits get their own diagnostic channel — exit_kind + the incomplete_summary histogram (#615) - #636

Merged
gadievron merged 3 commits into
masterfrom
fix/issue615-exit-kind-diagnostics
Sep 17, 2026
Merged

gadievron merged 3 commits into
masterfrom
fix/issue615-exit-kind-diagnostics

Conversation

@gadievron

@gadievron gadievron commented Sep 14, 2026

Copy link
Copy Markdown
Collaborator

Fixes #615.

What was wrong

The agentic enhance loop's four degenerate exits are conflated under one classification: "incomplete", but they differ in kind and remedy: three cheap model-behavior exits (end-turn-without-finish, finish truncated at max_tokens, no-tool-calls — confidence 0.3, remedied by retry/prompt work) vs one budget-exhaustion exit (MAX_ITERATIONS — confidence 0.2, remedied by raising the budget, not retrying). In the referenced run the split was 41 "Analysis incomplete" + 4 "Could not complete analysis within iteration limit" — a 4-unit class a consumer reading only classification cannot distinguish from the 41.

The fix

No fourth classification (that would break every == "incomplete" consumer — checkpoint.py, enhancer.py, the degenerate-exit tests). Instead:

  • exit_kind on AgentResult — the four exit sites stamp their kinds; a completed analysis carries the empty string and serializes without the key (present-only — the completed-record byte-identity).
  • The incomplete_summary histogram beside the existing error_summary: the enhancer's counting loop aggregates exit_kind (legacy unstamped rows fall back to "unstamped"; single-shot rows cannot reach the branch — their whitelist has no "incomplete"); EnhanceResult carries the histogram (present-only in to_dict); and both step-report writers — the scanner's pipeline summary and the standalone CLI's enhance summary — thread it, as FLAT siblings of the error_summary gate (a zero-error run with incompletes carries the histogram too — the pipeline round caught the CLI writer nesting it inside the error gate, which dropped it).
  • The classification and every existing consumer are unchanged (pinned by the classification-unchanged test and the branch-adjacency source pin).

Evidence

  • RED (pristine master, this file copied in): 7 failed + 1 passed (the 8-item set; the green row is the histogram mirror-test — noted).
  • GREEN: 8/8 on the file — the four kinds + the completed negative (present-only), an incomplete serializing its kind, the classification-unchanged pin, the histogram census, the EnhanceResult carry + the present-only negative, the flat-writer pin on BOTH writers, and all FOUR real-exit receipts driving the actual agent loop (end-turn-without-finish, finish-truncated, no-tool-calls, max-iterations — real AgentResults, not hand-constructed; the harness already drove all four, the "two cheaply-simulable" claim was false). The full suite locally 4189 passed + 34 skipped + 2 failed — the 2 are test_llm_sdk_contract_floor SDK-pin host drift, identical on master; CI is the authoritative suite. Ruff clean; Semgrep 0 findings.

The de-balancing check (the intent trace)

serves: I-enh; de-balances-checked: the four == "incomplete" consumers are byte-identical (nothing reads exit_kind downstream except the histogram); the completed record serializes key-for-key identical to master (the present-only splat).

The adversarial record

The authoring rounds (the CLI second-writer catch, the real-exit receipts) + this PR-processing round (a combined 2d+DU seat that caught the CLI nesting bug — the histogram inside if result.error_summary:, a zero-error run with incompletes writing no histogram to the standalone enhance step report; a 4-seat panel — 3/4 approve with the CHANGELOG duplicate-heading fix, applied; two deep-refute seats): the dedent (both writers' flat placement pinned); the four real-exit receipts; the CHANGELOG entry (the repo's stated convention — the PR carried none) with the duplicated-heading artifact fixed; the branch-adjacency source pin (the histogram line inside the cls == "incomplete" branch — the placement-blind grep passed if the line moved outside and counted every unit's kind); the schemas comment corrected.

Notes for the reviewer

gadievron added a commit that referenced this pull request Sep 17, 2026
… CLI writer carries the identity, the guards everywhere, the pins tightened (#611)

The two deep-refute seats' findings, all dispositioned:

- the ORPHAN #615 heading in the CHANGELOG (an insertion artifact of
  the pipeline round — a bodyless section that would duplicate on the
  #636 merge) deleted.
- the standalone CLI's enhance step summary now threads
  incomplete_count/total_units (the scanner's writer already did — the
  two-writer drift the #615 round closed for its own field); the pin
  covers BOTH writers.
- _unit_security_classification and the prompt gate gained the
  isinstance(str) guard the predicate readers already carried (a
  non-str stamp raised TypeError at 2 of the 4 membership sites where
  base tolerated it).
- the census test's mirror now backed by SOURCE pins on the real
  condition (the sentinel exclusion in analyzer.py) and the real loop's
  branch (the incomplete_count increment inside cls == 'incomplete').

11/11 at head; ruff clean.
@gadievron
gadievron force-pushed the fix/issue615-exit-kind-diagnostics branch from c97485e to addc55b Compare September 17, 2026 10:20
…— exit_kind + the incomplete_summary histogram (#615)

The four degenerate exits are conflated under one classification
'incomplete', but they differ in kind and remedy: three cheap
model-behavior exits (end-turn-without-finish, finish truncated at
max_tokens, no-tool-calls — confidence 0.3, remedied by retry/prompt
work) vs ONE budget-exhaustion exit (MAX_ITERATIONS — confidence 0.2,
remedied by raising the budget). A consumer reading only
classification cannot split the 4-vs-41.

NO fourth classification (that would break every =='incomplete'
consumer): an exit_kind field on AgentResult (present-only in
to_dict; empty = a completed analysis; the four sites stamp their
kinds), and the incomplete_summary kind histogram beside the existing
error_summary — EnhanceResult carries it, and BOTH step-report
writers (the scanner and the standalone CLI) thread it, so the split
is visible without stderr mining. Legacy unstamped rows fall back to
'unstamped' in the aggregate; the per-record to_dict stays present-
only. The classification and its consumers are UNCHANGED (pinned).

serves: I-enh, I-out. de-balances-checked: every ==incomplete
consumer keeps its semantics (the classification-unchanged test);
the completed analysis serializes byte-identically (present-only).

Refs #615
…gram is NOT gated on errors), all four real-exit receipts, the CHANGELOG (#615)

The combined 2d+DU seat's findings, all dispositioned:

- THE HIGH: the CLI's enhance summary nested the histogram inside
  'if result.error_summary:' — a zero-error run with incompletes wrote
  no incomplete_summary to the standalone enhance step report (the
  scanner's twin is flat). Dedented to the sibling shape; the source
  pin now covers BOTH writers' flat placement.
- the real-exit receipts extended to ALL FOUR stamps (the harness
  already drove no_tool_calls and max_iterations — the 'two
  cheaply-simulable' claim was false): end_turn_without_finish,
  finish_truncated, no_tool_calls, max_iterations on real AgentResults.
- the schemas comment corrected ('the empty dict', not 'None').
- the CHANGELOG entry added (the repo's stated convention — the PR
  carried none).

RED on base: 7 failed + 1 passed (the 8-item set, receipt
red-receipt-636.log; the green row is the histogram copy-test — noted);
GREEN at head: 8/8. The full suite at the rebased head: 4189 passed +
34 skipped + the 2 pre-existing SDK-pin drift failures (suite-636.log —
the body's '4171 + 33 skipped' was the pre-rebase authoring-env figure;
no artifact matched it). Semgrep: 0 findings.
The two deep-refute seats' findings, all dispositioned:

- the histogram source pin now asserts the BRANCH ADJACENCY (the kind
  line inside the cls == 'incomplete' branch beside the increment — the
  placement-blind grep passed if the line moved outside the branch and
  counted every unit's kind); the duplicate docstring cleaned.

8/8 at head; ruff clean.
@gadievron
gadievron force-pushed the fix/issue615-exit-kind-diagnostics branch from addc55b to 4f6dccd Compare September 17, 2026 10:35
@gadievron
gadievron merged commit b2e32c2 into master Sep 17, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant