feat: configurable retry policy as a top-level retry: root#44
Open
uipreliga wants to merge 1 commit into
Open
Conversation
uipreliga
requested review from
akshaylive,
bai-uipath and
tmatup
as code owners
July 22, 2026 21:32
|
Claude finished @uipreliga's task in 1m 25s —— View job Code Review in Progress
|
uipreliga
added a commit
that referenced
this pull request
Jul 22, 2026
- weight: 0 now makes a criterion purely informational — excluded from both the weighted score AND the pass/fail gate (previously it could still flip a task to FAILURE). New BaseSuccessCriterion.is_gating drives the single-source gate (all_criteria_passed) and the `coder-eval evaluate` exit code. A validator rejects weight:0 combined with stop_when or suite_thresholds (arming a non-gating criterion for a pass/fail gate is incoherent). - reports.py failure-reason sampling uses pass_threshold, not score < 1.0. - Doc drift: dropped the non-existent `rephrase` mutation; corrected the stale "max_memory_mb NOT enforced" claim (docker --memory/--cpus/--pids-limit). Split out per review: the configurable retry policy moved to #44, and the evalboard OSS-edition change is dropped entirely (see @bai-uipath's review comment — the panels need core-produced fields that do not exist yet). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
uipreliga
force-pushed
the
fix/weight-zero-gating-and-retry-policy
branch
from
July 22, 2026 21:40
c746087 to
8c02dc6
Compare
uipreliga
force-pushed
the
feat/configurable-retry-policy
branch
from
July 22, 2026 21:41
863f7dd to
cc54a8b
Compare
Split out of #34 per review, with the placement feedback applied. - New RetryPolicy (models/retry.py): max_retries / initial_delay / backoff_multiplier, all optional. It lives at the task's TOP LEVEL (`retry:`), not under `run_limits` — `run_limits` holds caps that abort a run, while a retry policy is how hard the run tries to survive a transient error. - `retry` is a first-class 4th `-D` root alongside agent/run_limits/sandbox: `-D retry.max_retries=0` to fail fast while debugging. It merges through the SAME resolve_root engine at every layer (experiment defaults, task, variant, CLI), so the unification invariant holds — covered by new test_merge_unification cases. - All retry decisions resolve through one seam, errors/retry.py:: resolve_retry_config. A category that is non-retryable in the built-in table (auth/billing) is never made retryable by a per-run override — retryability is a property of the error, not the run. - Persisted error_details.is_retryable / retry_delay_seconds now reflect the run's actual policy instead of the built-in defaults. - Docs: TASK_DEFINITION_GUIDE gains a `retry` section that documents the placement rationale AND the in-place retry semantics raised in review — a retried communicate() re-sends into the same sandbox and session (the working directory is NOT reset), and the crashed attempt's partial turn is preserved, so trajectory-observing criteria see both attempts. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
uipreliga
force-pushed
the
feat/configurable-retry-policy
branch
from
July 22, 2026 22:03
cc54a8b to
3ec8a05
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
Split out of #34, addressing @akshaylive's review on that PR:
All three points are taken:
run_limitsretry:block and its own-Droot — see below.Stacked on
fix/weight-zero-gating-and-retry-policy(#34) — retarget tomainonce that merges.Configurable retry policy
RetryPolicy(models/retry.py):max_retries/initial_delay/backoff_multiplier, all optional.retry:), not underrun_limits. Per review:run_limitsholds caps that abort a run, whereas a retry policy is how hard the run tries to survive a transient error.retryis a first-class 4th-Droot alongsideagent/run_limits/sandbox:-D retry.max_retries=0to fail fast while debugging. It merges through the sameresolve_rootengine at every layer (experiment defaults → task → variant → CLI), so the unification invariant holds — two newtests/test_merge_unification.pycases pin it.errors/retry.py::resolve_retry_config. A category that is non-retryable in the built-in table (auth/billing) is never made retryable by a per-run override — retryability is a property of the error, not the run.error_details.is_retryable/retry_delay_secondsintask.jsonnow reflect the run's actual policy instead of the built-in defaults.Review question: does this clear the working directory between retries?
No — and the guide now says so explicitly. A retried
communicate()re-sends the prompt into the same sandbox and the same agent session; the working directory is not reset, so files the failed attempt already wrote survive into the retry. The crashed attempt's partial turn is also preserved on the result (deliberate, for crash forensics), which means trajectory-observing criteria (command_executed,commands_efficiency,skill_triggered) see both attempts.I chose to document this rather than add a reset seam: resetting mid-task would discard work the agent already did and budget already spent.
retry.max_retries: 0is the escape hatch when you need a clean single-attempt trajectory. Happy to add the reset if reviewers prefer it.Test plan
make verifygreen — 3450 passed, 2 skipped, ruff + pyright + custom CE lint clean.error_detailspolicy fidelity,-D retry.*override + field-merge + did-you-mean on an unknown key, and merge-unification parity for the new root.🤖 Generated with Claude Code