Repository navigation
Conversation
Reviewer's GuideThe PR standardizes production and native-eval agent execution on the concrete claude-opus-5-5 model at explicit medium effort, adds automated fail-closed protection against pin drift and child overrides, documents the model-management and pre-merge proof workflow, and updates the pinned Fullsend harness dependencies and lockfile. Sequence diagram for Fullsend model and effort resolutionsequenceDiagram
participant Operator
participant Run as Fullsend run
participant Harness as Harness config
participant Agent as Agent frontmatter
participant Claude as Claude runtime
participant Vertex
Operator->>Run: fullsend run
Run->>Harness: Read model and effort
alt model flag or FULLSEND_MODEL is set
Run->>Run: EffectiveModel
else no higher-priority model
Run->>Agent: Read fallback model
end
Run->>Claude: --model claude-opus-5-5
Run->>Claude: --effort medium
Claude->>Vertex: Inference request
Vertex-->>Claude: Model response
Claude-->>Run: Run result and metrics
Flow diagram for validating model pins before mergeflowchart TD
Edit[Edit production and eval model pins] --> Pin[Set claude-opus-5-5 and medium effort]
Pin --> Test[test_model_pins.py]
Test -->|pass| Repin[Re-pin .fullsend harness bases]
Test -->|fail| Fix[Fix alias, drift, or child override]
Fix --> Test
Repin --> Lock[Run fullsend lock]
Lock --> Proof[Run verify-pr, triage-security, and native eval proofs]
Proof --> Metrics[Confirm provider model in metrics.json]
Metrics --> Merge[Merge with a merge commit]
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
|
🤖 Finished Verify Pr · ✅ Success · Started 8:34 AM UTC · Completed 8:41 AM UTC Commit: Runtime: claude · Model: claude-opus-4-8 · Effort: high · Cost: $4.37 |
Verification Report for TC-7278 (commit aeddaf8)
Overall: WARNAll intent, security, CI, and test-quality checks pass. The implementation switches all production and eval agent pins to This comment was AI-generated by sdlc-workflow/verify-pr v0.13.10. |
|
🤖 Finished Verify Pr · ❌ Failure (validation failed after 2 iteration(s)) · Started 9:01 AM UTC · Completed 9:03 AM UTC Commit: Runtime: claude · Model: claude-opus-5-5 · Effort: medium |
…ffort Pin verify-pr, triage-security and the native eval agent to the concrete claude-opus-5-5 model ID with explicit effort medium, replace the floating opus alias in both agent frontmatters, add a fail-closed model-pin consistency test, and document how the agent model is managed. Implements TC-7278 Assisted-by: Claude Code Co-Authored-By: Claude Opus 5.5 <[email protected]>
…4ce00b Move both composing children's base: to the rebased model-switch commit and regenerate lock.yaml with the fullsend v0.52.0 CLI (second lock run unchanged). Implements TC-7278 Assisted-by: Claude Code Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
🤖 Finished Verify Pr · ❌ Failure (post-script /home/runner/work/sdlc-plugins/sdlc-plugins/.fullsend/.fullsend-cache/resources/sha256/709bb41caf28726c869c28b47ac2d2035d56553d89fd6909ed42edfe6c4f4d33/scripts/post-verify-pr.sh failed: ex…) · Started 9:18 AM UTC · Completed 9:23 AM UTC Commit: Runtime: claude · Model: claude-opus-5-5 · Effort: medium · Cost: $2.31 |
Verification Report for TC-7278 (commit a1136ce)
Overall: FAIL
This comment was AI-generated by sdlc-workflow/verify-pr v0.13.10. |
Pre-merge live proof — verify-pr (TC-7278)Run: https://github.com/RHEcosystemAppEng/sdlc-plugins/actions/runs/41419304266 (temporary
The agent's verdict on this run (FAIL) only reflects that the live-proof evidence was not yet attached when it ran. Earlier failed attempts (recorded as failures): run 41029966351. Attempt 1 lost the concurrency race; attempt 2 hit Remaining: triage-security local report-only run, native eval local paid run. |
|
🤖 Finished Verify Pr · ❌ Failure (post-script /home/runner/work/sdlc-plugins/sdlc-plugins/.fullsend/.fullsend-cache/resources/sha256/1d2419c69738b091b2383b4ef779dc224f0e605d975959f75e7471b9f1333c85/scripts/post-verify-pr.sh failed: ex…) · Started 9:27 AM UTC · Completed 9:36 AM UTC Commit: Runtime: claude · Model: claude-opus-4-8 · Effort: high · Cost: $4.14 |
Verification Report for TC-7278 (commit d005d47)
Overall: WARNThe PR cleanly implements the The single WARN is on Acceptance Criteria: all eight criteria are satisfied in code, but the three pre-merge live-proof criteria (verify-pr, triage-security, native eval) rest on human-attached run links/logs that cannot be independently reproduced in the read-only sandbox. The verify-pr proof is evidenced by the PR report comments showing No review feedback, CI failures, or eval failures were found, so no sub-tasks or root-cause tasks were created. Note: this verification ran in a sandbox whose working tree was checked out at This comment was AI-generated by sdlc-workflow/verify-pr v0.13.10. Previous runVerification Report for TC-7278 (commit d005d47)
Overall: WARNThe PR cleanly implements the switch of the verify-pr, triage-security, and native eval agents to claude-opus-5-5 at explicit effort medium: the floating Overall WARN is driven solely by the Acceptance Criteria row: the three "pre-merge live proof" criteria (verify-pr temporary-workflow run, local triage-security run, local paid native-eval run) require paid inference and network and cannot be confirmed by static analysis in the sandbox. They are a human/process obligation rather than a code defect — and the PR comment thread shows the author already posted live-proof evidence of runs on claude-opus-5-5. No code changes were requested; no sub-tasks were created. This comment was AI-generated by sdlc-workflow/verify-pr v0.13.10. Previous run (2)Verification Report for TC-7278 (commit d005d47)
Overall: WARNThe PR cleanly implements the model switch to This comment was AI-generated by sdlc-workflow/verify-pr v0.13.10. |
|
Note on the red The run itself succeeds: the agent exits 0, the output passes schema validation, and the report is posted on this PR. Only the final Jira step fails, when it tries to update the existing verify-pr sticky comment on TC-7278: The service account can create comments but not edit its own, so every re-run on a task that already has a sticky comment fails this way. PR #364 (TC-7300) showed the same failure. The fix is the permission grant tracked in TC-7313. |
|
🤖 Finished Verify Pr · ❌ Failure (post-script /home/runner/work/sdlc-plugins/sdlc-plugins/.fullsend/.fullsend-cache/resources/sha256/1d2419c69738b091b2383b4ef779dc224f0e605d975959f75e7471b9f1333c85/scripts/post-verify-pr.sh failed: ex…) · Started 1:27 PM UTC · Completed 1:36 PM UTC Commit: Runtime: claude · Model: claude-opus-4-8 · Effort: high · Cost: $4.84 |
|
🤖 Finished Verify Pr · ✅ Success · Started 2:45 PM UTC · Completed 2:55 PM UTC Commit: Runtime: claude · Model: claude-opus-4-8 · Effort: high · Cost: $4.77 |
Fullsend's Claude runtime exports FULLSEND_OUTPUT_DIR without creating it. claude-opus-4-8 created it implicitly; claude-opus-5-5 follows the skill's single-write rule and stops, so 036-valid/037-release produced no agent-result.json (native proof 0/14). The Step 0.6 sandbox branch now runs mkdir -p; interactive mode is unchanged. Update GATE_COMMANDS_SHA256 and cover the directory creation. Implements TC-7278 Assisted-by: Claude Code Co-Authored-By: Claude Opus 5.5 <[email protected]>
…c6c74a Move both composing children's base: to the triage-security output-dir fix and regenerate lock.yaml with the fullsend v0.52.0 CLI (second lock run unchanged). Implements TC-7278 Assisted-by: Claude Code Co-Authored-By: Claude Opus 5.5 <[email protected]>
Pre-merge live proof: native eval agent + triage-security model path (TC-7278)Ran in CI against CI's Vertex project, using a temporary Final run, PASS: https://github.com/RHEcosystemAppEng/sdlc-plugins/actions/runs/41474951603 (head
This run exercises the same triage-security agent path as the production harness: same skill, policy, provider, image ( Bug found and fixed in this PR (
|
Eval ResultsSource head: 10c001c Eval Results: triage-security
Pass rate: 100% · Tokens: 128,619 · Duration: 51s Baseline ( Generated by sdlc-workflow/run-evals v0.13.10 |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Native Fullsend Eval Results
Head: 10c001c
Merge: d8fd51b
Trusted workflow: 50974cb
Reviewed eval source: 50974cb
Native job: success; complete: true.
Execution evidence: valid; quality score (advisory): 24/24 passed.
| Case | Execution evidence | Quality score (advisory) |
|---|---|---|
| 034-empty | valid | 5/5 |
| 035-malformed | valid | 5/5 |
| 036-valid | valid | 7/7 |
| 037-release | valid | 7/7 |
|
🤖 Finished Verify Pr · ✅ Success · Started 4:39 PM UTC · Completed 4:48 PM UTC Commit: Runtime: claude · Model: claude-opus-4-8 · Effort: high · Cost: $5.28 |
Verification Report for TC-7278 (commit 10c001c)
Overall: WARNOverall WARN — the single non-PASS check is Scope Containment (WARN), which is informational here: the 3 out-of-scope files are the direct, justified consequence of the model switch (claude-opus-5-5 follows the skill's single-write rule and no longer implicitly creates FULLSEND_OUTPUT_DIR, so commit dc6c74a adds a This comment was AI-generated by sdlc-workflow/verify-pr v0.13.10. |
Summary
harness/verify-pr.yaml,harness/triage-security.yamland the native eval agent (evals/fullsend/run.pydefaults, suiteharness.yaml/eval.yaml) toclaude-opus-5-5at expliciteffort: medium, on the fullsend v0.52.0 CLI and image from TC-7300.opusalias in both agent frontmatters with the concreteclaude-opus-5-5ID.plugins/sdlc-workflow/scripts/test_model_pins.py. It fails if any production/eval pin disagrees with the others, uses a floating alias, or if a.fullsend/composing child overrides the base model/effort.FULLSEND_OUTPUT_DIRin Step 0.6's sandbox-mode branch only. fullsend's Claude runtime does not create this directory. claude-opus-4-8 created it on its own, but claude-opus-5-5 follows the single-write rule and stopped without writing a result. Covered by a new test;GATE_COMMANDS_SHA256updated..fullsend/harness/*.yamltodc6c74ae(upstream branch) and regenerate.fullsend/lock.yamlwith the fullsend v0.52.0 CLI (second lock run unchanged).fullsend.mdsection "Managing the agent model" (where model/effort are set, which setting wins, why concrete IDs and explicit effort, change procedure including the sandbox image's Claude Code version, Vertex prerequisite, pre-merge proof recipe, forced-tool_choicenote); native eval model/effort source documented indocs/testing/fullsend-gate-evals.md.Precedence, verified in fullsend source:
--model>FULLSEND_MODEL> configagents:entry > harnessmodel:> agent frontmatter. In production the harness value wins.Out of scope: the remaining
claude-opus-4-8references are the eval judge model, which is tracked in TC-7279.Pre-merge live proofs
3055d4f5…andclaude-opus-5-5/medium; schema validated and report posted. The Jira sticky-comment update returns 403, which is TC-7313.claude-opus-5-5andexecution_valid: true.Verification
python3 -m pytest plugins/sdlc-workflow/scripts/: 812 passed, 3 skippeduvx skillsaw: 0 errors (triage-security SKILL.md 15,911/16k tokens);claude plugin validate plugins/sdlc-workflow: passedThe red
verify-pr / Harness runcheck is TC-7313 (the Jira service account cannot edit its own comments) and is unrelated to this PR.Merge with a merge commit, so the pinned SHA survives.
Implements TC-7278
🤖 Generated with Claude Code