[sxlijin]

Disable the IX CI workflows. They've been hanging since Friday 10am PT, presumably because of capacity exhaustion, and are now just noise for both humans and agents.

Direct answers

1. Last actual Rust workload to acquire ix compute: run 33196059386 / Browser Tests job 98933519350, for PR #4623, acquired runner baml-run-a9c278e0-mtd7weak000d (ix, ix-web) at 2026-08-28 17:44:53 UTC / 10:44:53 PDT. It began the substantive Rust-backed WASM build at 18:02:18 UTC / 11:02:18 PDT, logged Finished release profile [optimized] target(s) in 42.85s, ran the browser test, logged 1 passed, and completed successfully at 18:03:43 UTC / 11:03:43 PDT. API queue time was 3 seconds and execution time was 18m50s. The workflow event was pull_request, branch codex/function-spec-streaming-redesign, run head SHA 02fe831222f70c947ef7c31fabb9ea7f4179a2fb, and checked-out PR merge SHA e655eb0c306a12f4db7eab89c6cdd39bf29c93ab.
2. Last actual Rust ix success: the same Browser Tests job above. Its sibling Unit Tests job 98933519425 also built the Rust/WASM package and passed four test files, finishing earlier at 2026-08-28 17:56:25 UTC / 10:56:25 PDT. The overall workflow was later cancelled by a newer run for the same PR, so this is an individual job success, not a green suite.

For core Rust lanes, the latest job to acquire a runner was Cargo Tests (musl) job 98895925475, assigned at 2026-08-28 17:25:02 UTC / 10:25:02 PDT; it executed Rust compilation and then hit its 45-minute job timeout. The last successful core Cargo test was GNU job 98666282239, completed 2026-08-28 17:38:05 UTC / 10:38:05 PDT after waiting 20h32m52s for a runner and executing for 21m40s. The last Size Gate success was Linux job 98666282151, completed 2026-08-28 17:38:35 UTC / 10:38:35 PDT. The last WASM Pack success was job 97573349611, completed 2026-08-24 20:05:02 UTC / 13:05:02 PDT.

The most recently triggered run in which the intended ix suite was genuinely green was run 32339726862 for PR #4541: all 19 ix jobs acquired runners and succeeded, completing 2026-08-20 10:58:14 UTC / 03:58:14 PDT. An earlier-triggered canary run 32335256169 finished later in wall-clock time at 11:12:57 UTC / 04:12:57 PDT, with all 22 ix jobs successful. These were the only two genuinely all-ix-green attempts in the retained history.

Method and definition

The investigation paginated the complete retained history for .github/workflows/ix-ci.yml: 574 workflow runs, run numbers 1-582, and 597 attempts including 23 reruns. Run numbers 15, 19, 20, 23, 34, and 100-102 were unavailable; because the workflow was introduced on August 18, this is not an age-based retention boundary, but the API does not reveal why those records are absent. Every retained attempt's jobs, steps, timestamps, conclusions, runner name/group/labels, annotations, and available logs were inspected. The current workflow and its four reusable workflows were compared with the relevant historical SHAs.

An actual Rust CI workload means an ix-labelled Cargo Tests, Size Gate, WASM Pack Tests, Webview test, SDK test, or other Rust/build job that had a non-empty runner name, began runner-side steps, and executed substantive Cargo, Rust compilation, build, or test commands. Determine changes alone, skipped job placeholders, and jobs that GitHub created but never assigned to a runner do not count. This distinction matters because queued self-hosted jobs can have an API started_at equal to created_at; runner identity plus step/log evidence was required.

Retained-history findings

Population	Result
Attempts with no ix job	190
Successful no-Rust/path-gated attempts	122
Attempts that scheduled ix Rust work	407
Completed scheduled attempts where no ix job acquired a runner	193
Attempts where at least one ix job acquired a runner	190
Genuine all-ix-success attempts	2
ix jobs	8,084
ix jobs assigned to runners	1,956
ix jobs never assigned	6,128
Unassigned jobs that waited exactly 24 hours	1,327

In the bounded recent window from 2026-08-28 00:15:10 UTC through 2026-09-01 16:31:06 UTC, 244 attempts comprised 107 path-gated/no-ix attempts and 137 Rust-scheduled attempts. Of the scheduled attempts, 130 acquired no ix runner at all: 25 completed after the exact 24-hour runner wait, 81 were cancelled or superseded by concurrency, and 24 remained active/queued at the snapshot. The 137 attempts created 2,737 ix jobs; 414 were still queued at the snapshot and 585 unassigned jobs in the window reached the exact 24-hour limit. Only 19 jobs, across seven attempts, acquired ix runners: nine succeeded individually, six failed after losing runner communication, and four hit execution timeouts. There were zero substantive test/build failures and zero genuinely green suite attempts in this window. Attempt-level categories overlap for partially assigned runs.

Representative exact evidence:

• Capacity wait: job 99509638608 — The job has exceeded the maximum execution time while awaiting a runner for 24h0m0s.
• Concurrency cancellation: job 99445259056 — Canceling since a higher priority waiting request for ix-preview-4557/merge-4557 exists.
• Execution timeout: musl job 98895925475 — The job has exceeded the maximum execution time of 45m0s.
• Runner infrastructure failure: job 98722431364 — The self-hosted runner lost communication with the server.... All six recently assigned jobs whose conclusion was failure had this annotation.
• A genuine test failure did occur outside the bounded recent window: C# SDK job 98346987769, in Run C# sdk tests.

Root cause and onset

The directly proven mechanism is that no eligible communicating ix runner became available after 2026-08-28 17:44:53 UTC. Hundreds of jobs reached GitHub's exact 24-hour self-hosted-runner acquisition limit; the final assigned jobs also included six runner-lost-communication failures. GitHub-hosted Determine changes jobs continued to run, the same compound labels (ix plus family labels) had matched successfully before the cutoff, and a job's timeout-minutes starts after assignment, so workflow execution timeout does not explain the queue. Concurrency explains many superseded runs, but not the canary jobs that waited exactly 24 hours.

The best-supported inference is that the ix control plane stopped supplying healthy matching runners at sufficient capacity, degrading from long queues to complete admission/provisioning failure. Repository evidence cannot distinguish ordinary provider VM/region capacity exhaustion from an ix GitHub App admission/provisioning outage or all matching organization runners being offline. The repository runners endpoint showed zero because these runners are organization-scoped, while the organization runner endpoint requires administration access and returned 403. There was no provider-side admission error in never-assigned job logs. Missing runner labels are unlikely because all label families matched previously; organization concurrency quota is also unlikely as the GitHub-hosted jobs continued while only ix-labelled jobs stalled, but neither can be excluded without organization/ix administrative telemetry.

Degradation was visible before the cutoff: the first assignment queue over one hour appeared on August 20; job 96621761615 was the first over 12 hours, waiting 16h51m9s before assignment on August 21; August 25 assignments waited 12-15 hours; August 26-27 commonly waited 18-24 hours. The final August 28 batch had 29 assignments in the 17:00 UTC hour with a median queue of 16h52m25s, although a few fresh web jobs acquired runners immediately. There were no assignments afterward.

The workflow was introduced on August 19 with runs-on: [ix, family]. Provisioning moved from a repository reconciliation workflow to the ix GitHub App on August 21; shared R2 cache credentials followed on August 24. Runners continued to work for a week after the App switch, so it correlates with the degraded period but is not an immediate causal boundary. The Size Gate, WASM Pack, and Webview reusable workflows were byte-identical between the last green SHA and the latest actual-run SHA; Cargo changes affected package selection, not runner routing. No workflow change coincides with the final cutoff. The current production pool configuration is external to the repository. An abandoned pre-merge prototype showed 32 one-slot members with nine warm members across the runner families, but it is not authoritative evidence for the App-managed pool.

The top-level concurrency key includes the PR number or SHA. New commits cancel old runs for the same PR, which explains many cancellations. On canary, including the SHA means each push has a unique group, so old 19-22-job waves accumulate rather than supersede one another. Nested reusable-workflow concurrency keys independently cancel some WASM/Webview jobs, producing partial attempts.

Follow-up recommendations

1. Restore and instrument the control plane before re-enabling automatic triggers: inspect ix App/webhook/admission logs, per-family runner registrations, cloud VM/region quotas, provisioning failures, and runner-loss telemetry.
2. Size the pool for at least one complete canary/code wave: SDK 9, GNU 4, WASM 3, Web 3, musl 1, MSRV 1, and light 1, or 22 concurrent slots. Two overlapping waves require roughly 44 slots unless workflow fan-out is reduced.
3. Make canary concurrency branch-stable if only the newest canary result matters, and consolidate nested reusable concurrency so cancellation happens once at the suite boundary.
4. Alert on runner queue age at 5-10 minutes, well before GitHub's 24-hour limit, and expose oldest queue age, online/busy runners by label family, admissions, provisioning errors, and runner-loss counts.
5. Add a hosted final summary that reports path-gate outputs, expected ix job count, and a clear classification such as no Rust scheduled, waiting for runner, execution timeout, runner lost, test/build failed, or passed.
6. Make path-gated successes explicitly say No ix workload scheduled so they cannot be confused with Rust CI success.
7. Consider a health-gated fallback to the existing Blacksmith lanes for trusted same-repository events, with a guard that prevents duplicate execution after an ix job starts and preserves the current fork-security boundary.
8. Fail or skip the non-gating preview early when the pool is unhealthy instead of leaving ambiguous 24-hour queues.

Reproduction

gh api --paginate 'repos/BoundaryML/baml/actions/workflows/ix-ci.yml/runs?per_page=100' | jq -s '[.[].workflow_runs[]]'
gh api 'repos/BoundaryML/baml/actions/runs/RUN_ID/attempts/ATTEMPT/jobs?per_page=100'
gh api 'repos/BoundaryML/baml/check-runs/JOB_ID/annotations?per_page=100'
gh run view 33196059386 --repo BoundaryML/baml --job 98933519350 --log
gh api 'repos/BoundaryML/baml/contents/.github/workflows/ix-webview-tests.reusable.yaml?ref=e655eb0c306a12f4db7eab89c6cdd39bf29c93ab' --jq .content | tr -d '\n' | base64 -d
gh api 'repos/BoundaryML/baml/actions/runners?per_page=100'
gh api 'orgs/BoundaryML/actions/runners?per_page=100'

Validation

• actionlint .github/workflows/ix-ci.yml
• git diff --check

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
beps	Ready	Preview	Sep 1, 2026 5:33pm UTC
developer-docs	Error		Sep 1, 2026 5:33pm UTC
promptfiddle2	Ready	Preview	Sep 1, 2026 5:33pm UTC

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-01T17:25:38.764517Z	eae023f	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[github-actions]

⏭️ Performance benchmarks were skipped

Perf benchmarks (CodSpeed) are opt-in on pull requests — they no longer run on every push. They always run automatically after merge to canary/main.

To run them on this PR, do any of the following, then push a commit (or re-run CI):

• Add RUN_CODSPEED=1 to the PR description, or
• Include run-perf or /perf in the PR title or any commit message.