Repository navigation
CI: Miniflare D1 suites flake (fetch-sync assertion, list-values statement budget) #190
Description
Two CI runs on 2026-11-10 failed in D1 (Miniflare) suites that pass on sqlite and DO SQLite, and passed on rerun:
- PR fix(security): prototype pollution, HTML text and ReDoS backlog (CodeQL) #187, Node 24:
packages/core-tstest/list-values.test.ts, "RFC-0042 §4 list values, repairs: Cloudflare D1 (Miniflare)", "the backfill keeps to its statement budget; discovery is two statements". AssertionError: falsy value. Run 42789152142. - Release PR chore(release): version packages #189, Node 26:
packages/apitest/roles.test.ts, "roles and invites: Cloudflare D1 (Miniflare)". The first test failed inside Miniflare itself (miniflare/src/plugins/core/proxy/fetch-sync.ts:147:3, an assert in the sync fetch proxy). The rest of the describe block then failed in 1–3 ms each. Run 42793831919.
Locally, the same suites often drop connections or run out of ports under load (noted in several PRs this week). Every flake blocks a release or a security merge until someone reruns it.
What to look into
- Whether the D1 suites share one Miniflare instance per file across many tests. If so, how
fetch-syncbehaves underturbo --concurrency=2with the CI runner's CPU count. - Whether Miniflare 4.20260730 (pinned, ADR-0012) has a known fetch-sync assertion bug. If a newer 4.x fixes it, bump.
- Making the D1 halves resilient: create a fresh Miniflare per describe block, and dispose of it in
afterAll. Lower the concurrency for the D1 project only. As a last resort, use Vitestretry: 1scoped to the D1 projects. Do not retry globally, because that would hide real failures. - The "statement budget" test: check whether it relies on timing (
ms: 60_000budget versus a slow runner), or on statement counting that a dropped connection can skew.
Done when the D1 halves pass in 20 consecutive CI runs (a scheduled or workflow_dispatch loop is fine), or the root cause is fixed upstream and pinned.
Activity
test: reproduce the Miniflare sync-proxy race in the #190 guard
Madraka commentedon Nov 10, 2026
Root cause. Both flakes are one upstream Miniflare 4.20260730.0 bug in the synchronous binding proxy. The "statement budget" failure is not a timing or counting problem: its stack ends in the same
message?.id === idassertion atfetch-sync.ts:147, reached fromD1Database#prepare/bind.The proxy's handshake waits on a shared 0/1 flag. If the proxy thread is preempted between its
Atomics.storeand itsAtomics.notify, the late notify wakes the next call before that call's reply exists. Every later call then reads the previous reply, which explains the cascade inroles.test.ts. Upstream fixed this in cloudflare/workers-sdk#15552, but only for Miniflare 5. No newer 4.x exists.Fix: #192.
node -rpreload that stalls both sides of the handshake. The unpatched build fails with the CI assertion; the patched build passes.Stress. The patched build passed 20/20 D1-half runs on an oversubscribed runner. The unpatched build also passed 40 such runs, so the race is too rare to trigger without forcing it. The forced reproduction is the proof.
Still open (local only, pre-existing): port exhaustion on macOS. Async D1 calls take one TIME_WAIT port each, and a single core-ts run fills 16,340 of 16,384 ports. The keep-alive backport (#15781) fixes that, but it caused an
ECONNRESETin 1 of 20 localroles.test.tsruns, which can't be retried safely. So that backport was left out of #192. ADR-0012's 2026-11-10 addendum has the details.Merge pull request #192 from cloudhal-dev/test/d1-stabilise-190
Merge pull request #197 from cloudhal-dev/ci/d1-stress-followup