feat(enhance): the degenerate exits get their own diagnostic channel — exit_kind + the incomplete_summary histogram (#615) - #636
Merged
Conversation
gadievron
requested review from
dgeyshis,
shahar-davidson and
sounil
as code owners
September 14, 2026 22:56
gadievron
added a commit
that referenced
this pull request
Sep 17, 2026
… CLI writer carries the identity, the guards everywhere, the pins tightened (#611) The two deep-refute seats' findings, all dispositioned: - the ORPHAN #615 heading in the CHANGELOG (an insertion artifact of the pipeline round — a bodyless section that would duplicate on the #636 merge) deleted. - the standalone CLI's enhance step summary now threads incomplete_count/total_units (the scanner's writer already did — the two-writer drift the #615 round closed for its own field); the pin covers BOTH writers. - _unit_security_classification and the prompt gate gained the isinstance(str) guard the predicate readers already carried (a non-str stamp raised TypeError at 2 of the 4 membership sites where base tolerated it). - the census test's mirror now backed by SOURCE pins on the real condition (the sentinel exclusion in analyzer.py) and the real loop's branch (the incomplete_count increment inside cls == 'incomplete'). 11/11 at head; ruff clean.
gadievron
force-pushed
the
fix/issue615-exit-kind-diagnostics
branch
from
September 17, 2026 10:20
c97485e to
addc55b
Compare
…— exit_kind + the incomplete_summary histogram (#615) The four degenerate exits are conflated under one classification 'incomplete', but they differ in kind and remedy: three cheap model-behavior exits (end-turn-without-finish, finish truncated at max_tokens, no-tool-calls — confidence 0.3, remedied by retry/prompt work) vs ONE budget-exhaustion exit (MAX_ITERATIONS — confidence 0.2, remedied by raising the budget). A consumer reading only classification cannot split the 4-vs-41. NO fourth classification (that would break every =='incomplete' consumer): an exit_kind field on AgentResult (present-only in to_dict; empty = a completed analysis; the four sites stamp their kinds), and the incomplete_summary kind histogram beside the existing error_summary — EnhanceResult carries it, and BOTH step-report writers (the scanner and the standalone CLI) thread it, so the split is visible without stderr mining. Legacy unstamped rows fall back to 'unstamped' in the aggregate; the per-record to_dict stays present- only. The classification and its consumers are UNCHANGED (pinned). serves: I-enh, I-out. de-balances-checked: every ==incomplete consumer keeps its semantics (the classification-unchanged test); the completed analysis serializes byte-identically (present-only). Refs #615
…gram is NOT gated on errors), all four real-exit receipts, the CHANGELOG (#615) The combined 2d+DU seat's findings, all dispositioned: - THE HIGH: the CLI's enhance summary nested the histogram inside 'if result.error_summary:' — a zero-error run with incompletes wrote no incomplete_summary to the standalone enhance step report (the scanner's twin is flat). Dedented to the sibling shape; the source pin now covers BOTH writers' flat placement. - the real-exit receipts extended to ALL FOUR stamps (the harness already drove no_tool_calls and max_iterations — the 'two cheaply-simulable' claim was false): end_turn_without_finish, finish_truncated, no_tool_calls, max_iterations on real AgentResults. - the schemas comment corrected ('the empty dict', not 'None'). - the CHANGELOG entry added (the repo's stated convention — the PR carried none). RED on base: 7 failed + 1 passed (the 8-item set, receipt red-receipt-636.log; the green row is the histogram copy-test — noted); GREEN at head: 8/8. The full suite at the rebased head: 4189 passed + 34 skipped + the 2 pre-existing SDK-pin drift failures (suite-636.log — the body's '4171 + 33 skipped' was the pre-rebase authoring-env figure; no artifact matched it). Semgrep: 0 findings.
The two deep-refute seats' findings, all dispositioned: - the histogram source pin now asserts the BRANCH ADJACENCY (the kind line inside the cls == 'incomplete' branch beside the increment — the placement-blind grep passed if the line moved outside the branch and counted every unit's kind); the duplicate docstring cleaned. 8/8 at head; ruff clean.
gadievron
force-pushed
the
fix/issue615-exit-kind-diagnostics
branch
from
September 17, 2026 10:35
addc55b to
4f6dccd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #615.
What was wrong
The agentic enhance loop's four degenerate exits are conflated under one
classification: "incomplete", but they differ in kind and remedy: three cheap model-behavior exits (end-turn-without-finish, finish truncated at max_tokens, no-tool-calls — confidence 0.3, remedied by retry/prompt work) vs one budget-exhaustion exit (MAX_ITERATIONS — confidence 0.2, remedied by raising the budget, not retrying). In the referenced run the split was 41 "Analysis incomplete" + 4 "Could not complete analysis within iteration limit" — a 4-unit class a consumer reading onlyclassificationcannot distinguish from the 41.The fix
No fourth classification (that would break every
== "incomplete"consumer —checkpoint.py,enhancer.py, the degenerate-exit tests). Instead:exit_kindonAgentResult— the four exit sites stamp their kinds; a completed analysis carries the empty string and serializes without the key (present-only — the completed-record byte-identity).incomplete_summaryhistogram beside the existingerror_summary: the enhancer's counting loop aggregatesexit_kind(legacy unstamped rows fall back to"unstamped"; single-shot rows cannot reach the branch — their whitelist has no"incomplete");EnhanceResultcarries the histogram (present-only into_dict); and both step-report writers — the scanner's pipeline summary and the standalone CLI's enhance summary — thread it, as FLAT siblings of the error_summary gate (a zero-error run with incompletes carries the histogram too — the pipeline round caught the CLI writer nesting it inside the error gate, which dropped it).Evidence
EnhanceResultcarry + the present-only negative, the flat-writer pin on BOTH writers, and all FOUR real-exit receipts driving the actual agent loop (end-turn-without-finish, finish-truncated, no-tool-calls, max-iterations — realAgentResults, not hand-constructed; the harness already drove all four, the "two cheaply-simulable" claim was false). The full suite locally 4189 passed + 34 skipped + 2 failed — the 2 aretest_llm_sdk_contract_floorSDK-pin host drift, identical on master; CI is the authoritative suite. Ruff clean; Semgrep 0 findings.The de-balancing check (the intent trace)
serves: I-enh;de-balances-checked: the four== "incomplete"consumers are byte-identical (nothing readsexit_kinddownstream except the histogram); the completed record serializes key-for-key identical to master (the present-only splat).The adversarial record
The authoring rounds (the CLI second-writer catch, the real-exit receipts) + this PR-processing round (a combined 2d+DU seat that caught the CLI nesting bug — the histogram inside
if result.error_summary:, a zero-error run with incompletes writing no histogram to the standalone enhance step report; a 4-seat panel — 3/4 approve with the CHANGELOG duplicate-heading fix, applied; two deep-refute seats): the dedent (both writers' flat placement pinned); the four real-exit receipts; the CHANGELOG entry (the repo's stated convention — the PR carried none) with the duplicated-heading artifact fixed; the branch-adjacency source pin (the histogram line inside thecls == "incomplete"branch — the placement-blind grep passed if the line moved outside and counted every unit's kind); the schemas comment corrected.Notes for the reviewer
EnhanceResultedits meet feat(analyze): the enhancement non-verdicts are visible, out of the prompt, counted, and migrated on re-enhancement (#611) #635'sincomplete_count/total_unitsedits on merge — the union carries both (additive resolution); the post-merge invariant to pin:sum(incomplete_summary.values()) == incomplete_count(both increments sit in the samecls == "incomplete"branch — holds by construction at the merge).context_enhancer's internal_summary_incompletecount and the resume path are kind-blind by design (they count the classification, not the kind) — out of this issue's stated scope.