Repository navigation
fix(lanes): the CI API key is out of credit, so 45% of model legs fail instantly while reporting green #12526
Description
Refs: #6958
Problem
The API key the CI lanes try first has no credit left. Every key-credit model leg fails at once with Credit balance is too low, and the subscription fallback then does the real work. The lanes spend a job on each failed leg, report it green, and no lane has API credit to draw on. This cannot bill anything (the promotional credits are capacity), so it is a throughput and measurement defect.
Evidence
From the first pull of the lane transcript store (#7676): 218 model legs created 2026-11-02 18:08Z to 22:05Z. Metrics only, no transcript content.
| Job | Legs | Errored in one turn |
|---|---|---|
check-intent |
61 | 61 |
check-intent-subscription |
60 | 0 |
review-claude-review |
18 | 18 |
review-claude-security-review |
18 | 18 |
review-subscription-claude-review |
17 | 0 |
review-subscription-claude-security-review |
17 | 0 |
adjudicate-feedback-0 |
2 | 2 |
In total, 99 of 218 legs (45%) ended this way, in 161 to 614 ms. The result message on one sampled leg per job reads "result":"Credit balance is too low", "api_error_status":400, "total_cost_usd":0, with is_error: true and subtype: success. Their workflow runs conclude success, and the check rows such as pr-review / claude-review-key-credit read green.
Done when
- A key-credit leg that gets this error is recorded as out of credit in the lane telemetry, not as a passing review, so the morning brief and the capacity map can see it.
- While the key is known to be out of credit, the lanes skip the key-credit leg and go straight to the subscription leg, then try the key again on a set interval, so a topped-up or rotated key is picked up with no edit. Compare the job count per PR before and after.
- Map: run the agent lanes at full Claude capacity #6958 records whether CI's key should be rotated to an account with credit. That is an account and secret change for the owner to make or approve; nothing here depends on it.
This was generated by AI.
Metadata
Metadata
Assignees
Labels
Type
Fields
Work status
Activity
github-actions commentedon Nov 2, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Known
Flags: none
Blocked by: none
#12526 (comment)
This decision settles the open questions: nothing here bills, so the earlier cost hold does not apply. I found no duplicate among the open issues.
Automated triage, applied by the lanes App.
melodic-automation-lanes commentedon Nov 2, 2026
Not admitted to the issue lane: its title or stated work names a charge (subscription), which needs an owner decision. Admission takes the issue once each of those is met, when intake triages it again (
.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Nov 2, 2026
The latest triage verdict holds nothing: admission moved the issue to
status: needs-decision; its comment says why.kyle-sexton commentedon Nov 3, 2026
Decision recorded.
Agent choice, not an owner answer. Issue Triage wrote this record under the owner's standing direction §1 and §3.
Admission held this issue on the charge word "subscription". The word here names the Claude subscription leg of the CI lanes. The standing direction §3 lists "model usage under the Claude subscriptions, and each account's promotional $200 API credits" as not cost, and the body says the same: "This cannot bill anything (the promotional credits are capacity)". Nothing here bills, so the hold is not an owner stop.
Decision: build the body's first two done-when lines as written. The third line stays with #6958, and nothing here waits on it, as the body says.
Credit balance is too lowis recorded as out of credit in the lane telemetry, not as a passing review, so the morning brief and the capacity map can see it.Builder: Automation Lanes, because the change is in the lanes' workflows (and possibly the shared review workflow in ci-workflows). This issue moves from
status: needs-decisiontostatus: confirmed, withoutagent-ready.kyle-sexton commentedon Nov 3, 2026
kyle-sexton commentedon Nov 3, 2026
Measured after #12617 merged (2026-11-03T04:17:28Z): the records are right, the re-probe interval does not hold. From REST job lists, against the baseline in the issue body.
Item 1, records. A key leg that hits the credit stop writes a lane-outcome record with class
credit-stop, kindapi-keyandhandoff: true. A key leg the gate skips writes outcomeskipwith classcredit-stop. The morning brief counts both in itsKEY:line. Gap:pr-review-main.ymlis not in the brief's default lane list (plugins/work-items/skills/morning-brief/scripts/morning-brief.sh:214-222), so its records are read only when--lane-workflowsnames it.Item 2, skip and re-probe. With a 1-hour window, each workflow should try the empty key at most once an hour.
The hosted lanes, with one gate per PR run and few no-op runs, behave about as designed. pr-verify and pr-review-main do not. Every sampled gate log says
no run of the newest 3 settles it. Most of their runs are fast no-op runs with no record step and no model call, so they settle nothing. The gate reads at most 3 completed runs (MAX_JOB_READS), and the run carrying the credit-stop record is usually still in flight on its subscription leg, so the gate cannot list it yet (#12617's residual b). An offline replay of the gate's decision over the stored job lists matched that case for 48 of 56 pr-verify key legs and all 14 pr-review-main key legs; 4 of 56 did not fit the replay's completion-time approximation. The skip held while a record run was among the newest 3: from 05:03 to 05:59Z, 55 pr-verify runs skipped the key. Once that window ended at 06:03Z, 30 key stops followed in a row.Job counts: pr-verify runner jobs per run rose from 1.47 (30 runs on 11-02) to 2.46 (212 runs after), because
check-intent-creditruns on every run; model legs per run fell from 0.93 to 0.50. These are per run, sinceworkflow_runruns carry no pull request.adjudicate-feedback. It ran 6 times since the merge with no disputed thread, so it made no model call. Before the merge both of its model calls ended in a credit stop. It has no subscription leg (
.github/workflows/pr-unblock.yml:1836-1850), so while the key is empty a disputed thread escalatesfeedback-unadjudicatedto theneeds-sessionqueue. Latent, not observed since the merge.What remains, so this stays open:
pr-review-main.ymlto the morning brief's default lanes.Raw data and the replay script are in a session scratch directory and were not committed.
Posted by the overnight session. This was generated by AI.
kyle-sexton commentedon Nov 3, 2026
Measured again after #12799 and #12838 moved pr-verify and pr-review-main to the exact-name marker: the skip holds, but each window opening still lets several calls through. Window: 16:45Z to 18:01Z (1.27 h), from REST job lists, annotations, the marker artifact lists and gate logs.
Records (item 1): done. Credit stops and skips write their lane records. #12851 added
pr-review-main.ymlto the morning brief's default lanes (morning-brief.sh:216-219), and the KEY line counts both record shapes (morning-brief.sh:2127).Skip and re-probe (item 2): partly done.
pr-verify fell from 17.3 key calls an hour to 2.4. It skipped 33 times in a row, so the "newest 3 runs" gap is gone, and its 17:55Z call is the designed re-probe one hour after the 16:52Z marker. Its jobs per run fell from 2.46 to 1.32, because the gate is now a step inside
check-intent.The extra calls have one cause: the marker is uploaded only after the key leg's credit stop, so every gate that reads between the end of the window and that upload also runs the key. In pr-verify two runs 5 s apart both called it. In the hosted lanes the key job starts 6 to 8 minutes after its gate, so one opening let through 6 code-review calls and 4 security calls (the gate at 17:00:55Z logged "no credit-stop marker in the window"; the first marker landed at 17:06:10Z). The hosted callers pass no
key-credit-window-hours, so they use the reusable's default of 1 hour.Both keys are still out of credit: every key call in the window ended
class=credit-stop,api_error_status=400.Decision (agent, on this evidence): a gate that lets the key through writes a short-lived "probing" marker at once, and the other gates skip while one is fresh. That closes the race for every caller in one place, the
check-key-creditaction. Rejected: re-reading the marker at the start of the key job, which shrinks the hosted gap but not the 5 s pr-verify case; and a concurrency group on the key leg, which serializes runs. The window length stays at 1 hour until the burst is gone and re-measured.Separate from credit: hosted security run 41678666577 timed out at "Apply the Claude managed settings" on its key leg, before any model call; no subscription leg ran, so that head got no security review. Seen once.
Item 3 stays with #6958.
Posted by the overnight session. This was generated by AI.
kyle-sexton commentedon Nov 3, 2026
Measured after ci-workflows#980 (hosted lanes, merged 19:55:35Z) and #12935 (pr-verify, pr-review-main, merged 21:59:15Z): every window opening let through exactly 1 key call. Measured to 23:08Z with the method of the last measurement: REST job lists, marker artifact
created_attimes fromactions/artifacts?name=<exact name>, and the gate job logs' gate lines.Per opening, each probe run was also the run that uploaded the next credit-stop marker: its key leg credit-stopped and started the next window.
A second pr-verify opening probed once since: probing marker 23:08:20Z from run 41736651732.
Leaks: none. Neither of #980's known gaps happened: no other gate read inside a 2-27 s probe gap, and no key-leg job was cancelled. Three pr-review-main gates after the window logged "this run probes the key" with no review leg due (41732012355, 41732511119, 41732552590); they wrote no probe and called nothing, and the probe went to the first run with a leg due, as #12935's start-term fix intends.
Key-leg jobs per PR, before and after (done-when item 2). PRs counted as distinct head branches of
pull_requestruns (the runs'pull_requestsfield was empty for 56 of 59 runs before).pr-verify and pr-review-main run on
workflow_runwith no PR: pr-verify made 3 key calls in 136 runs before and 1 in 58 after; pr-review-main 1 run in 136 before and 1 in 58 after.The before burst shows in the credit-stop markers themselves: pr-review-hosted uploaded 3 within 17:08:02-17:08:40Z and 3 within 19:23:31-19:24:57Z; after the fix it uploads one per window (20:29:58Z, 21:38:00Z, 22:40:30Z).
Verdict: each workflow makes at most 1 key call per window opening. pr-verify and pr-review-main rest on one or two openings each. Done-when item 3 stays with #6958 as written. Closing.
Follow-ups recorded elsewhere: probe skips still report as
credit-stopfallbacks (a #980 known gap; 4 of 189 hosted skips here), and #12983 for the body-only pr-verify fail that cost #12935 a commit.