[fuad-daoud]

Summary

In short: When a fuzz search finds a new input, a worker stops searching to shrink it, for up to 10s. This bounds that at 1s, which found more new inputs and more coverage on the measured targets, and makes a search run by hand default to 5s a target instead of 10s; the nightly job's minute a target is unchanged.

Each search bounds minimizing a new input at 1s, not 10s (-fuzzminimizetime). In a target taking a string or []byte, 13 of the 15, the flag governs every input that adds coverage, not only a crash, and a worker that is minimizing runs no search. A shorter bound can leave a crash's reproducer and each new corpus entry less minimal, a cost the measurement weighed: new inputs and coverage bits both rose at 1s. A search run by hand now defaults to 5s a target (fuzz.sh, the Makefile help and FUZZTIME, and the README). The nightly job's FUZZTIME: 1m is unchanged, and so is its timeout estimate: -fuzztime's deadline cancels minimizing too (go1.27 internal/fuzz/fuzz.go: the workers' context is a child of the -fuzztime timeout), so the bound never lengthens a target's search.

The 5s default is a second decision, carried here with the bound from the same commit. Once #1382 lands, the nightly job does the deep search, and a search run by hand is a quick local check before a push, so 5s halves its wait: about 75s of search over 15 targets instead of 150s. By this body's own figures it costs about a fifth of the new inputs such a search finds, 812 at 5s against 1,035 at 10s, both at the 1s bound.

A budget replays a target's corpus first, and the two decisions compound: the 1s bound keeps far more entries and the 5s default leaves less time to replay them. After make fuzz FUZZTIME=5m, the 5s default ended inside FuzzCompile's replay of 1,404 entries at 759, searched nothing and exited 0, where a 10s budget on the same corpus, or the 10s bound's 393 entries at 5s, still reached the search. FuzzCycleDetector did the same in the review's runs. So fuzz.sh now names such a target: fuzz.sh: searched nothing, the budget ended replaying the corpus: FuzzCompile. Its exit stays 0, as no input failed. The nightly job is not reached: all 15 targets reached the search on two simulated nights.

Stacked on #1382, which moves the search to a nightly job and is stacked on #1373 and #1311; the diff here shows only this change. The stack lands bottom-up: once #1382 merges, this is retargeted to main (the repository does not delete merged branches, so GitHub will not retarget it), and Closes #1290 links there. This is split from Omar's 22cd99d0 (#1301), whose other changes are in #1382. Omar's d16302ac carries his author and date and his measurement in its message; a4dd4b41 then names the corpus cost in the fuzz.sh comment, and fe26eda1 cites the Go source it rests on.

Why 1s

Fresh corpus each run, three runs a side, alternating, -fuzztime=1m -parallel 4, coverage read from the corpus through GODEBUG=fuzzdebug=1, which reads 410 bits for FuzzNewBigVal_AcceptedFormsAreJSONValid's seeds alone against 854 after an 8s search:

Target	Coverage bits, 10s	Coverage bits, 1s	New inputs, 10s	New inputs, 1s
FuzzCycleDetector	40,006–41,090	42,402–42,766	33–71	240–269
FuzzNewBigVal_AcceptedFormsAreJSONValid	930–989	997–1,020	124–152	137–151

The cost #1290 names depends on the crash. Over four runs a side on a planted crash a short input reaches, reproducers were 2 or 3 bytes at both bounds, and 10s missed the crash once within 2 minutes. On one that needs a valid document, the shape of any assertOracles failure, seven runs a side in FuzzCompile left 366 to 3,825 bytes at 1s and 52 to 59 at 10s. A reproducer cut at 1s still replays. d16302ac's message still says the cost did not appear, so the squash body should leave that sentence out. At a 10s budget across all 15 targets, 1s also found 1,035 new inputs against 863, and a search run by hand at the new 5s default found 812. This compares one shorter bound on two targets, plus the 15-target totals. The table's runs name no machine: rerunning its method on linux/amd64 at 8c5e7c21 read FuzzCycleDetector at 22,088–22,811 bits at 10s and 24,921–25,503 at 1s, with new inputs 90–139 and 339–400, so its direction holds and its magnitude does not reproduce, and FuzzNewBigVal_AcceptedFormsAreJSONValid at 970–986 and 993–1,016.

A bound below 1s is not taken here. In the same review's runs, 100ms found more new inputs and coverage on three targets, with corpus bytes about 3 times larger, and found a small planted crash in 2 of 5 runs against 5 of 5 at 1s, too few runs to settle.

A second look at FuzzNewBigVal_AcceptedFormsAreJSONValid on this branch, two runs a side, -fuzztime=1m -parallel 4, same method: coverage bits 943 and 950 at 10s, 984 and 973 at 1s. New inputs 120 and 113 at 10s, 126 and 136 at 1s.

Out of scope

A deadline that reports no finding (#466) is left alone, as #1290 puts it.

Before and after

flowchart LR
  subgraph before["Before"]
    direction TB
    b1["a search finds a new input"] --> b2["a worker minimizes it for up to 10s<br/>and searches nothing meanwhile"]
    b2 --> b3["at a 10s budget, as long as the search itself"]
  end
  subgraph after["After"]
    direction TB
    a1["a search finds a new input"] --> a2["minimized for up to 1s"]
    a2 --> a3["more of each budget spent searching:<br/>more new inputs and coverage bits"]
    a2 -. "cost" .-> a4["reproducers and corpus entries<br/>can be less minimal"]
  end
  before ~~~ after

Loading

What it brings: each night's minute a target, and each search run by hand, spends less of its time shrinking inputs and finds more: on the measured targets 1s found more new inputs and more coverage than 10s.

Minimizing still dominates a search: at a 45s or 1m budget it took about 80 to 90% of the workers' time on FuzzCompile, FuzzCycleDetector and FuzzLowerSchema and about 40% on FuzzNewBigVal_AcceptedFormsAreJSONValid, and at the 5s default about 70 to 99% on all four (a minimization cut at its bound logs 0s and is charged at the bound).

Test plan

• A go wrapper on PATH that logs go test arguments, then ./scripts/fuzz.sh with FUZZTIME unset: 15 go test lines, each -fuzztime=5s -fuzzminimizetime=1s.
• FUZZTIME=1m: 15 lines, each -fuzztime=1m -fuzzminimizetime=1s; make -n fuzz prints ./scripts/fuzz.sh 5s.
• Planted minimize_time=10s: the log shows -fuzzminimizetime=10s on all 15 lines. Reverted.
• The replay report, on a corpus a 5m FuzzCompile search left: ./scripts/fuzz.sh prints the line above, runs all 15 targets and exits 0. With its grep planted out, no line. On seeds alone at 2s, no line. With a crash planted in FuzzNewBigVal_AcceptedFormsAreJSONValid: exit 1, fuzz.sh: failed: naming it, all 15 run. A SIGINT to the group: exit 130, one target started, go test's closing lines kept, no temporary file left.
• bash -n and shellcheck scripts/fuzz.sh; make fuzz-check.
• make gate: CI run 42342694556 passed on 3c31b78c (lint, bench and the suite job success, fuzz search skipped as a pull request runs no search; coverage gate: all 20273 statements covered). No Go changed.

At merge, set the squash body by hand: a merge commit on this branch, 494bc457, carries attribution trailers that a default squash would paste onto main.

Closes #1290

Related: #1268, #1273, #1301, #1382