test: fix six recurring CI flakes by codebytere · Pull Request #65780 · nodejs/node

codebytere · 2026-09-04T08:27:08Z

Six CI flakes on main, each root-caused and fixed in its own commit. Four keep showing up in the nodejs/reliability reports over the last two weeks (picked by counting distinct main-targeting PRs they failed in since 2026-08-21, dropping everything since fixed on main or already in review in #65755, #65767, #65770); the last two (test-dns-resolver-max-timeout, the node:bench tests) failed this PR's own CI runs this week and will be in the next reports.

test	where	why it fails
`client-proxy/test-https-proxy-request-invalid-char-in-url`	macOS	only client-proxy test asserting zero proxy socket errors; the client resets the tunnel while the proxy is still relaying the upstream's TLS shutdown, logged as `ECONNRESET`
`parallel/test-external-memory-reasonable-size`	SmartOS	the child aborts with 1.2 GB resident and writes a core file, which outlasts the 300 s timeout; now runs under `ulimit -c 0` like the other abort tests
`test-runner/test-run-watch-emit-restarted`	macOS x64 (marked flaky)	FSEvents reports the fixture setup writes after the first run has started, so the intentional write is the second restart; now only restarts after the write count, flaky mark dropped
`wasi/test-wasi-pthread`	all (marked flaky on three platforms)	the fixture's `thread-spawn` waits on the same value the worker stores on success, so an early notify is lost, and gives the worker only 1 s to instantiate; either way `pthread_create()` fails. Flaky marks dropped
`parallel/test-dns-resolver-max-timeout`	rhel10-ppc64le	compares the wall time of `{ timeout: 500, tries: 3 }` with and without `maxTimeout: 500`, but c-ares only expires a try on cares_wrap's `timeout` ms timer tick (one or two ticks per try) and jitters uncapped retries by 0.5-1x, so the two ranges meet at 3000 ms; now `timeout: 100, tries: 5`, where they are ~600-1000 ms vs ~5000-7500 ms
`parallel/test-bench-*`	debian12-x64	the tests record samples with `b.start(); process.hrtime.bigint(); b.end(1)` and rely on the middle read to move the clock; on that host all three reads return the same value and `end()` throws `insufficient clock precision`. Now a `completeSample()` helper in `test/common/bench.js` spins until hrtime moves, used at every site and in the bench-runner fixtures

Looked at and left alone: parallel/test-runner-run ("should support timeout" reporting uncaughtException on macos15-x64) doesn't reproduce under load on Linux and I couldn't pin it down from the code; sequential/test-debugger-pid and test-run-watch-cwd-isolation-none* need a Windows/macOS box; pummel/test-fs-watch-non-recursive on AIX is the host running out of AHAFS watchers.

Tests: each changed test passes locally (Linux x64), 16-48 repeats at -j16 for the watch and WASI ones, 40 concurrent runs beside 24 busy-loop processes for the dns one, all test-bench-* under a preload that quantizes process.hrtime.bigint() to 100 µs (six of them fail that way before the change), and the parallel/ ones also under tools/test.py --worker.

Fixes: #64226
Refs: #59146
Refs: #54534

Disclosure: the code and this description were written by Claude Code, directed and reviewed by @codebytere.

nodejs-github-bot · 2026-09-04T08:27:14Z

Review requested:

@nodejs/wasi

codecov · 2026-09-04T10:12:28Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.20%. Comparing base (b0d56c7) to head (b26f47e).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #65780   +/-   ##
=======================================
  Coverage   90.19%   90.20%           
=======================================
  Files         770      770           
  Lines      264410   264410           
  Branches    50243    50236    -7     
=======================================
+ Hits       238490   238516   +26     
+ Misses      16924    16907   -17     
+ Partials     8996     8987    -9

see 31 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

panva · 2026-09-04T15:20:18Z

cc @nodejs/build this jenkins run started by @codebytere does not have Resume CI

MikeMcC399 · 2026-09-04T16:23:26Z

It's similar to the problem I also reported in nodejs/build#4453

17:15:25 FATAL: Channel "hudson.remoting.Channel@39c0b532:JNLP4-connect connection from 67.158.54.159/67.158.54.159:54594": Remote call on JNLP4-connect connection from 67.158.54.159/67.158.54.159:54594 failed. The channel is closing down or has closed down

gurgunday

lgtm

lpinca · 2026-09-05T17:40:08Z

There is another PR (#65770) for test/parallel/test-child-process-fork-closed-channel-segfault.js` that was opened before this one.

test-https-proxy-request-invalid-char-in-url is the only client-proxy test that asserts the proxy logged no socket errors at all. Once the last response has been read the client destroys its tunnel, and if the proxy is still relaying the upstream's TLS close_notify at that point the client answers with a reset, which the proxy records as ECONNRESET on the CONNECT socket. That has been failing the test on macOS even though every request was routed to the sanitized URL. Keep asserting on other errors but leave connection resets out. Signed-off-by: Shelley Vohr <[email protected]>

test-external-memory-reasonable-size makes a child allocate 1.2 GB of external memory so that V8's --external-memory-max-reasonable-size check fires and the process aborts. The abort raises SIGABRT with all of that memory resident, and on hosts that write core files (the SmartOS CI machines in particular) the dump takes longer than the test timeout, so the test has been timing out there since it was added. Run the child under `ulimit -c 0` on POSIX, the same way test-abort-fatal-error and common.childShouldThrowAndAbort() handle their aborting children. Refs: nodejs#65589 Signed-off-by: Shelley Vohr <[email protected]>

test-run-watch-emit-restarted expected exactly one test:watch:restarted event, but it starts run({ watch: true }) right after writing the fixtures into the watched directory. Watch backends that deliver events with some latency, FSEvents on macOS most visibly, can still report those setup writes once the first run is under way, which restarts it and makes the later, intentional write the second restart. The test has been marked flaky on macOS x64 for that reason. Wait for the first drain, then require that the write is followed by a restart and a drain, ignoring whatever the setup produced before it, and drop the flaky marker. Refs: nodejs#54534 Signed-off-by: Shelley Vohr <[email protected]>

test-wasi-pthread fails now and then on every platform with "Assertion failed: r == 0 (c/pthread.c: main: 17)", i.e. pthread_create() itself reporting an error. The fixture implements `thread-spawn` by starting a Worker and blocking in Atomics.wait(result, 0, 0, 1000) until the worker signals that it has instantiated the module. Two things go wrong there: the worker signals success by storing 0, the value the main thread is already waiting on, so when the worker is quicker than the main thread its notify is lost and the wait runs into the timeout; and one second is not always enough for a Worker to start and instantiate a threads build on the slower CI hosts (arm debug, Windows, macOS). Either way spawn() returns -6 and wasi-libc turns that into a pthread_create() failure. Wait on a sentinel value that neither outcome writes, and give the worker a platform-scaled 30 seconds. Drop the flaky markers. Fixes: nodejs#64226 Refs: nodejs#59146 Signed-off-by: Shelley Vohr <[email protected]>

test-dns-resolver-max-timeout times a query with `{ timeout: 500, tries: 3 }` against one that also sets `maxTimeout: 500` and asserts the first took longer. c-ares only expires a try when cares_wrap's timer fires, every `timeout` ms, so each try costs one or two ticks depending on sub-millisecond ordering, and uncapped retries also get 0.5-1x jitter. That leaves the capped run anywhere in 1500-3000 ms and the uncapped one in 3000-4500 ms; on a busy rhel10-ppc64le host both came out at 3005 ms. With `timeout: 100, tries: 5` the ranges (about 600-1000 ms and 5000-7500 ms measured under load) cannot meet. Signed-off-by: Shelley Vohr <[email protected]>

codebytere · 2026-09-05T18:01:07Z

@lpinca ah, missed that one, thanks - dropped it here in favor of #65770 and swapped in a fix for test-dns-resolver-max-timeout instead.

panva · 2026-09-05T18:10:57Z

i'm suspecting a number of test/parallel/test-bench-* jobs are going to pop up in the report tomorrow, how about we preemptively deflake those too before starting another full CI?

cc @jasnell

off just these two deflake PRs that ran its CI today:

Most node:bench tests record a sample with `b.start(); process.hrtime.bigint(); b.end(1)`, counting on the extra clock read to make end() see a later timestamp than start(). On a debian12-x64 CI host the monotonic clock is coarse enough that all three reads returned the same value, so end() threw ERR_INVALID_STATE ("insufficient clock precision for benchmark sample") and test-bench-harness-errors failed; every test using the idiom can fail the same way there. Add test/common/bench.js with completeSample(), which spins until process.hrtime.bigint() moves between start() and end(), and use it at all of those sites, bench-runner fixtures included. Signed-off-by: Shelley Vohr <[email protected]>

codebytere · 2026-09-05T18:16:57Z

@panva ah good shout - they all record samples with b.start(); process.hrtime.bigint(); b.end(1) and count on that middle read to move the clock, which it didn't on the debian12 host. added a completeSample() helper in test/common/bench.js that spins until hrtime moves and switched every site (and the bench-runner fixtures) over in b26f47e.

jasnell · 2026-09-05T18:21:58Z

Yeah sounds good. I'll work on deflaking those this next week.

codebytere · 2026-09-05T18:24:08Z

@jasnell fwiw b26f47e in here already covers the zero-duration case across all the test-bench-* files and fixtures, in case that saves you the pass - if you had other flake modes in mind for them i can leave those to you.

jasnell · 2026-09-05T18:26:23Z

Awesome, thank you. I will try to get to any others later today.

nodejs-github-bot · 2026-09-05T19:41:49Z

CI: https://ci.nodejs.org/job/node-test-pull-request/77048/

nodejs-github-bot added needs-ci PRs that need a full CI run. test Issues and PRs related to Node.js core tests and test infrastructure. labels Sep 4, 2026

This comment has been minimized.

Sign in to view

codebytere requested review from panva and trivikr September 4, 2026 08:29

panva added flaky-test Issues and PRs involving tests that fail intermittently in CI. commit-queue-rebase PRs the Commit Queue should land as multiple self-contained commits. labels Sep 4, 2026

panva approved these changes Sep 4, 2026

View reviewed changes

panva added the author ready PRs with CI started, the required approvals, and no outstanding review comments. label Sep 4, 2026

benjamingr approved these changes Sep 4, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

jasnell approved these changes Sep 4, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

gurgunday approved these changes Sep 5, 2026

View reviewed changes

This comment has been minimized.

Sign in to view

codebytere added 5 commits September 5, 2026 17:49

codebytere force-pushed the test-deflake branch from ce80b32 to 8783b15 Compare September 5, 2026 18:01

codebytere changed the title ~~test: fix five recurring CI flakes~~ test: fix six recurring CI flakes Sep 5, 2026

panva approved these changes Sep 5, 2026

View reviewed changes

Uh oh!

Uh oh!

Conversation

codebytere commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nodejs-github-bot commented Sep 4, 2026

Uh oh!

This comment has been minimized.

codecov Bot commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

panva commented Sep 4, 2026

Uh oh!

MikeMcC399 commented Sep 4, 2026

Uh oh!

This comment has been minimized.

This comment has been minimized.

gurgunday left a comment

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

lpinca commented Sep 5, 2026

Uh oh!

codebytere commented Sep 5, 2026

Uh oh!

panva commented Sep 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codebytere commented Sep 5, 2026

Uh oh!

jasnell commented Sep 5, 2026

Uh oh!

codebytere commented Sep 5, 2026

Uh oh!

jasnell commented Sep 5, 2026

Uh oh!

nodejs-github-bot commented Sep 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

codebytere commented Sep 4, 2026 •

edited

Loading

codecov Bot commented Sep 4, 2026 •

edited

Loading

panva commented Sep 5, 2026 •

edited

Loading