Repository navigation
Closed
YodHeVauHe/twd-cli
#1Labels
enhancementNew feature or requestNew feature or requesthelp wantedExtra attention is neededExtra attention is needed
Description
The problem
Flakiness is statistical. One green run tells you little about a test that fails one time in ten. Today the way to measure that is a shell loop, as docs/cpu-throttling.md shows:
for i in $(seq 10); do
npx twd-cli run --cpu-throttle 6 --test "checkout flow" --no-report > /dev/null && echo pass || echo FAIL
doneThat relaunches Chrome and reloads the app on every round, and all it gives back is a run-level pass or FAIL. It can't tell you which test is flaky, or how often it fails.
This was proposed as a follow-up in #37 (--cpu-throttle), where the loop stands in for it.
Proposal
npx twd-cli run --repeat 10 --test "checkout flow"
npx twd-cli run --repeat 20 --cpu-throttle 6 --changed-since mainRun the selected tests n times in one twd-cli run and report per-test pass counts:
--- Run complete ---
Passed: 18 | Failed: 2 | Skipped: 0
Duration: 41.3s
Repeat: 10x
Flaky (1):
~ Checkout > checkout flow (8/10 passed)
Expected element [data-testid="total"] to have text "42.00"
Things to decide (part of the work)
- Same page or a reload between rounds. A reload per round isolates state but costs a page load; reusing the page is faster, but state can leak between rounds and hide or create flakes. The loop above reloads every time. A reasonable start: one
page.goto+#twd-sidebar-rootwait per round, without relaunching the browser. - Exit code. Probably 1 if any round of any test failed, since a flaky test is a failing test.
retryCount. Retries hide exactly what--repeatis looking for. Either force a single attempt while repeating, or keep the user's value and say so. Note the known bug:"retryCount": 0currently crashes the in-pageonFail(it readserr.messageoffnull). See feat(run): add --cpu-throttle <rate> to reproduce CI-only timing flakes #37's caveats.maxFailures. Should it count failed rounds or failed tests? Bailing after 10 failed rounds of the same test is probably not what you want.- Run report.
run.json(schema v3) has one entry per test today. Per-test counts (runs,passes) could be added without breaking readers, plusrepeatin therunheader.outcomemust still agree with the exit code. The HTML andsummary.mdwould list flaky tests under "Needs attention". - What to switch off. Recording, coverage and contract validation across n rounds are probably out of scope for a first version; refusing
--recordtogether with--repeatwith a clear message is fine. - Sharding. Likely not combinable in v1 (
--shardwith--repeatrefused).
Where to look
src/parseArgs.js: add the flag toparseRunArgsandRUN_FLAGS.tests/usage.test.jsfails until it also has a help line insrc/usage.js.src/index.js:runTests(). The chunk loop aroundrunner.runByIds(chunkIds)is where results are collected (partialStatus, deduplicated byseenIds, which a repeat would need to rethink).src/testSummary.js:formatRunComplete, the console block.src/runReport.js(buildRunReport/finalizeReport),src/needsAttention.js,src/reportHtml.js,src/reportMarkdown.jsfor the report.CLAUDE.mddescribes the architecture and the rules tests follow: no test may need a real browser;tests/runTests.test.jsmocks puppeteer.
Done when
--repeat <n>accepts a positive integer (both--repeat 5and--repeat=5) and refuses anything else.- The run-complete block shows per-test pass counts for any test that failed at least one round.
run.jsoncarries the counts, and the HTML and markdown summary show flaky tests.- README documents the flag, and
docs/cpu-throttling.mdswaps the shell loop for it. - Tests cover parsing, counting across rounds, the exit code and the report fields.
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or requesthelp wantedExtra attention is neededExtra attention is needed
Activity