diff --git a/CHANGELOG.md b/CHANGELOG.md index aa4fa364..c7063855 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- **Power Doppler schedule** (`--schedule doppler`, `core/power_doppler.py`): per-seed ensembles of mutant hit counts; mean + SVD wall filter, CFAR χ² flow detection; flow power scales seed energy to `[1, max_mult]`. - **OS / network scheduler ports**: seed arms `mlfq`, `stride`, `eevdf`, `bfq`, `sfq`, `codel`, `aimd`, `p2c` (`--seed--scheduler`) and op arms `op_stride`, `op_p2c` (`--op-stride`, `--op-p2c`); `core/fair_queue.py` gains `Stride` and `EEVDF`. Elo arms, in `--hail-mary`; seed arms also run without `--elo`. Falsification and adversarial tests. No paired benchmark yet. - **Ten op mutators** for in-tree targets and text decoders: `json_mutate` (fuzzgoat), `sql_mutate` (sqlite SQL path), `ecdsa_field_mutate` (secp256k1), `recompress_lz4`, `recompress_png_idat` (format band, sniffer-gated); `encoding_wrap`, `escape_mutate`, `ascii_float` (structural); `utf16_transcode`, `nest_bomb` (radamsa). Tests: `tests/test_{json_mutate,sql_mutate,ecdsa_field_mutate,recompress_roundtrip,text_codec,nest_bomb}.py`. Discovery effect on fuzzgoat unmeasured. - **`covering_array_webp`, `covering_array_isobmff`, `covering_array_zip`** operators (`core/mutations/covering_array_container.py`): pairwise sweep of WebP `VP8X` (riff_size, chunk_size, flags, reserved, width, height; 49 rows), ISO-BMFF `ftyp` (size, major brand, minor; 30 rows) and the first ZIP local file header (version, flags, method, sizes, name/extra length; 50 rows). Each gated on its own magic. Selection share unmeasured; ISO-BMFF `tkhd` not covered (variable depth). diff --git a/README.md b/README.md index dab31a61..16df63f8 100644 --- a/README.md +++ b/README.md @@ -143,7 +143,7 @@ per-sub-operator reward instead of uniformly (`--no-adaptive-havoc` restores uni | Metropolis admission | `--metropolis` | Accept non-improving inputs with P = exp(−ΔE/T) | | Secretary stopping | `--secretary` | Secretary-problem stopping ranks (display only) | | honggfuzz power factors | `--honggfuzz` | Novelty decay, freshness, fertility, density, entropy and timeout penalties | -| AFL++ power schedules | `--schedule` | FAST/COE/RARE/MMOPT/LIN/QUAD/GO/AFLGO/ENTROPIC seed-level energy | +| AFL++ power schedules | `--schedule` | FAST/COE/RARE/MMOPT/LIN/QUAD/GO/AFLGO/ENTROPIC/DOPPLER seed-level energy | | AFLGo directed annealing | `--schedule aflgo` | Exact AFLGo power factor — symmetric 32×/1/32× energy by distance-to-target with time-based cooling (`--t-x`, `--aflgo-cooling`) | | Entropic power schedule | `--schedule entropic` | libFuzzer `-entropic`: energy ∝ log(1 + rare-feature count) from already-tracked rare-edge ownership | | **K-Scheduler Katz centrality** | auto | On trace-pc targets: whole-program ICFG → horizon graph (contracted visited deletion, DAG) → out-degree Katz with β from node-hit counts; Elo-rated `katz` seed arm plus a clamped `--schedule katz` energy | diff --git a/docs/DEEP_DIVE.md b/docs/DEEP_DIVE.md index fefa6f82..95426d5f 100644 --- a/docs/DEEP_DIVE.md +++ b/docs/DEEP_DIVE.md @@ -84,7 +84,8 @@ For production and sensitive binaries using AFL family fuzzers is the best cours - **AFLGo power schedule** (`--schedule aflgo`, `--aflgo-cooling exp|log|lin|quad`, `--t-x MINUTES`): the exact `calculate_score()` distance section from AFLGo's afl-fuzz.c. Temperature `T` follows the chosen cooling over t_x minutes to exploitation (`exp`: 1/20^progress; `log`: 1/(1+2·ln(1+progress·13358.7268297)); `lin`: 1/(1+19·progress); `quad`: 1/(1+19·progress²)); `p = (1−nd)(1−T) + 0.5T` with `nd = (d−min)/(max−min)` normalized over the observed queue (min/max tracked per run, excluding the no-data sentinel); `factor = 2^(2·log2(32)·(p−0.5))` — symmetric around 1.0: early campaigns treat every seed equally, late campaigns give near-target seeds up to 32× energy and far seeds as little as 1/32×. - **AFLGo SHM-tail distance channel**: compiled into EVERY shim-linked target since `__AFL_DISTANCE_MODE` defaulted to 1 (2026-08-24; `-D__AFL_DISTANCE_MODE=0` opts out). Inert unless directed mode uploads a distance table (`DistanceTableShm`/`__AFL_DIST_SHM_ID`): without one, sum/count stay 0 and every reader takes the Python-side path. `build_targets.sh --distance` additionally builds the trace-pc-instrumented `*_dist.so`/`*_dist_asan.so` variants, where the shim accumulates per-block distances in `__sanitizer_cov_trace_pc()` — the PC (relative to the dladdr-derived object base) probes an open-addressing table of `{key, dist}` entries (packed 12-byte layout; the 4-byte header holds the slot *capacity*, a power of two ≥ 2×entries so empty slots exist, and the builder hash-inserts at `key % capacity` with linear probing to mirror the shim's probe — uploaded by the fuzzer at startup via `DistanceTableShm`/`__AFL_DIST_SHM_ID`), accumulating sum/count into the 16-byte SHM **tail** (after the edge table: `u64 dist_sum`, `u64 dist_count`), written at reset, at process exit (subprocess runs never call reset), and per-iteration in in-process modes via `__afl_dist_flush` (direct_lite has no process boundary, so the runner flushes the tail after each `run_one`). Per-execution `avg_distance = sum/count/100` is read straight from the tail and preferred over Python-side computation; blocks without a table entry don't count (AFLGo semantics). The table's PC keys are recovered by scanning text for `call __sanitizer_cov_trace_pc` sites (modern clang emits no `__sancov_pcs` for trace-pc) mapped to valued blocks via the CFGs — `TargetDistance.pc_distance_table()`. The shim's sanitizer-coverage callbacks are hidden-visibility so a libasan LD_PRELOAD cannot interpose over them in PIE builds. `tools/gen_distance_table.py` emits the table as C or text for inspection. Without the table (count==0) everything degrades to the Python-side path. Works in subprocess, direct_lite, and persistent modes (ASAN direct_lite requires libasan preloaded at fuzzer-process start — the `use_direct_lite` gate). The periodic stats line shows live distance when directed mode is active: `dist: avg: min: max:` (or `no-data`). `build_targets.sh --distance` builds both `*_dist.so` (no-ASAN) and `*_dist_asan.so` (ASAN) variants with the cmplog shim linked in, so `--cmplog` keeps them in direct_lite mode. Startup reports `[*] Distance instrumentation: detected` when the target carries the channel (the shim's `__afl_dist_flush` or a defined `__sanitizer_cov_trace_pc`), mirroring the AFL-instrumentation check. With `--elo` in directed mode, `aflgo` joins the Elo-arbitrated seed-strategy pool — a distance-pure arm picking `P(seed) ∝ exp(-2·norm_dist)` (distinct from the generic `weighted` arm, which blends distance with speed/size/entropy). - **AFLGo distance-annealed schedule** (`--schedule go`, requires `--target-functions`): wires the precomputed `avg_distance` (per-seed distance to directed targets) and `_anneal_progress` (exploration/exploitation annealing variable) into `SeedScorer.score()` for mutation budget scaling. During exploration phase (`anneal_progress` ≈ 0): uniform energy. During exploitation phase (`anneal_progress` → 1): `energy *= exp(β · (1 - norm_dist))` where `β = anneal_progress * 5`, capping at 100x. Seeds near the target get exponentially more mutations as the campaign matures. Previously these metrics only influenced seed selection but not mutation intensity. -- **Power schedules** (`--schedule base|fast|coe|rare|mopt|lin|quad|go|aflgo|entropic`): AFL++ power schedules ported to control mutation budget per seed via `SeedScorer`. Each schedule modifies a base score (100) by frequency-based factors. Honggfuzz-style novelty decay, density, fertility, freshness, and entropy factors are applied multiplicatively on top. `entropic` (libFuzzer `-entropic`) scales energy by `1 + log2(1 + rare)`, where `rare` is the larger of `rare_edge_count`/`tc_ref` already collected for RARE/honggfuzz scoring — an approximation of libFuzzer's feature-frequency Shannon entropy using signal the fuzzer already tracks. +- **Power schedules** (`--schedule base|fast|coe|rare|mopt|lin|quad|go|aflgo|entropic|doppler`): AFL++ power schedules ported to control mutation budget per seed via `SeedScorer`. Each schedule modifies a base score (100) by frequency-based factors. Honggfuzz-style novelty decay, density, fertility, freshness, and entropy factors are applied multiplicatively on top. `entropic` (libFuzzer `-entropic`) scales energy by `1 + log2(1 + rare)`, where `rare` is the larger of `rare_edge_count`/`tc_ref` already collected for RARE/honggfuzz scoring — an approximation of libFuzzer's feature-frequency Shannon entropy using signal the fuzzer already tracks. +- **Power Doppler schedule** (`--schedule doppler`, `core/power_doppler.py`): ultrasound power Doppler on coverage. Slow time = successive mutants of one seed (32 per ensemble); pixel = edge; sample = `log2(1 + hits)` (raw SHM counts, not buckets). Wall filter: mean removal (static path), then SVD components whose participation ratio spans ≥ half the seed's edges (and ≥ 4) are dropped as clutter — an early reject moving the whole path at once ("flash"). CFAR: residual power per edge vs `σ² · χ²_{dof}(1 − 10⁻³)`, `σ²` = median residual variance (floor 10⁻³). Seed power = summed flow power / dof; energy = `log1p(p)/log1p(max p)` ∈ [0, 1], scaled to `[1, max_mult]` like `katz`. Unscored or static seeds stay 1×. Gram eigendecomposition (n×n) replaces the full SVD (~6× faster). Bounded: 64 open ensembles × 2048 edges (float32, ≤16 MiB), 4096 scores (LRU). SHM coverage only; cost ~100 µs/exec at 2k live edges (dict→array conversion dominates), zero when off. Unmeasured — see `docs/TODO.md`. - **Favored set / cull_queue** (`core/schedules.py`, `services/fuzzer.py`): AFL-style `top_rated` minimal-set-cover selection. For each edge, the cheapest seed covering it is selected, then a greedy cover builds the favored set. FAST and COE schedules apply energy bonuses to favored seeds (`_fast_factor`, `_coe_factor`, `coe_skip`). `_cull_queue()` runs periodically during fuzzing and updates `self._favored`; the score call site passes `favored=(seed_key in self._favored)` so the scheduler actually uses it. - **Bayesian seed quality** (`--bayesian`): `BayesianSeedQuality` (`core/seed_quality.py`) maintains a Beta-Bernoulli posterior per seed over `P(outcome = new_coverage)`. Thompson sampling naturally balances explore/exploit without a manual temperature knob — unexplored seeds have high posterior variance and get sampled. The `record_outcome()` feedback loop is now wired in `fuzz_one()` (was previously a dead code path with all posteriors stuck at Beta(1,1)). State is persisted to `seed_quality.json` and restored on resume. @@ -587,7 +588,7 @@ fuzzer-tool rank ./target -d corpus -n 10 --dump top_seeds | `--wfc` | Wave Function Collapse structural generation (chunk reordering, pixel generation) | | `--enable-smt-z3` | Z3-based SMT solving for arithmetic constraint solving on cmplog pairs | | `--hw-perf` | Hardware performance counters via perf_event_open (instructions, branches, misses) | -| `--schedule base\|fast\|coe\|rare\|mopt\|lin\|quad\|go\|aflgo\|entropic` | AFL++ power schedule (`aflgo` = exact AFLGo distance annealing, `entropic` = libFuzzer `-entropic`, log-scaled rare-feature energy) | +| `--schedule base\|fast\|coe\|rare\|mopt\|lin\|quad\|go\|aflgo\|entropic\|doppler` | AFL++ power schedule (`aflgo` = exact AFLGo distance annealing, `entropic` = libFuzzer `-entropic`, log-scaled rare-feature energy, `doppler` = power Doppler flow energy) | | `--aflgo-cooling exp\|log\|lin\|quad` | Cooling schedule for the `aflgo` power factor (default exp) | | `--t-x MINUTES` | AFLGo time-to-exploitation in minutes; temperature cools to 1/20 over this window | | `--markov-order N` | Markov chain order(s), comma-separated (e.g. '0,1,2' for ensemble) | diff --git a/docs/TODO.md b/docs/TODO.md index b9b01369..ecbb27de 100644 --- a/docs/TODO.md +++ b/docs/TODO.md @@ -25,6 +25,7 @@ - [ ] **ptrace breakpoints only on dominator-tree leaves** (2026-09-26) — a hit block implies its dominators ran, so `ptrace_coverage.py` could place int3 on leaves only and infer the rest. Edges `(prev, curr)` are not implied the same way; measure breakpoint count and edge-set loss on fuzzgoat first. See `docs/learnings/2026-09-26-dominators-chk-worst-case.md`. ## Scheduling +- [ ] **A/B `--schedule doppler`** (2026-10-01) — power Doppler seed energy (`core/power_doppler.py`) is unit-tested and wired; never measured. Paired `bench_paired.py` vs `fast` on clang-built fuzzgoat (Hard Rule 52). Open: (a) ~100 µs/exec at 2k edges, mostly `get_edge_counts()` dict → numpy; a numpy accessor on `ShmCoverage` (exposes `_active_columns`, Hard Rule 34 — needs approval) removes it; (b) ensembles span picks, so a seed needs 32 picks to score — tune `ensemble` vs pick rate; (c) unused: `flow_edges()` (input-sensitive edge set) could feed position/operator targeting; (d) scores not persisted across `--resume`. - [ ] **A/B the OS / network scheduler ports** (2026-09-30) — `mlfq`, `stride`, `eevdf`, `bfq`, `sfq`, `codel`, `aimd`, `p2c` seed arms and `op_stride`, `op_p2c` are unit- and wiring-tested only. Run `tools/lib/bench_paired.py` per arm vs `seed_round_robin` / `drr` on fuzzgoat (clang, ASAN). Untuned: MLFQ allotment/boost, BFQ budget range, CoDel target/interval, AIMD alpha/beta/loss run. Open: (a) `sfq` groups only direct siblings (`parent_key`), and only under `--lineage`; a lineage-root flow would group whole families; (b) no arm persists state; (c) `seed_round_robin.select_seed` is O(n^2) per pick (`s in seed_ids` on a list): ~100 ms at 5000 seeds. - [ ] **A/B the effector/token/chunk/changed/rare_mask position arms** (2026-09-30) — wired and unit-tested; a 3k-exec fuzzgoat run confirms live signal (`changed`: 673 moved / 96 unmoved / 2231 unmeasured rounds; `rare_mask`: target on 328/3000 rounds) but fuzzgoat cannot rank position arms (see `pos_fibonacci` entry). Run `tools/lib/bench_paired.py` `pos-arena-{token,chunk,changed,rare-mask}` vs `pos-arena-uniform` on png_read (`chunk` needs a container corpus). Open: (a) `effector` had no drained map in 3k execs (SkipDet skipped 111/112 seeds) -- measure its reach on long runs before A/B, it is not subset-testable (needs the det stage); (b) `changed` credits only ~25% of rounds: parents without a recorded path hash (spliced/generated inputs) -- record one at admission or accept; (c) neither `changed` nor `rare_mask` persists state. - [ ] **Pre-existing, found 2026-09-30:** `test_regression_hail_mary_gates.py` fails on base (`cuckoo_seed_filter`, `swap_walk` missing from `_HAIL_MARY_FLAGS` or an exclusion); `tools/build_targets.sh` ASAN variants fail in the cloud container (non-ASAN builds fine). diff --git a/docs/architecture.dot b/docs/architecture.dot index 4a63a267..5d096454 100644 --- a/docs/architecture.dot +++ b/docs/architecture.dot @@ -56,7 +56,7 @@ digraph fuzzer_tool { subgraph cluster_sched { label="1 · Scheduling — what to fuzz next"; color="#bcd0c4"; fillcolor="#f2f8f4"; fontcolor="#33604a"; - picker [label="services/seed_picker.py + core/schedules.py\lweights · Pareto front · crowding · saturation gate\lFAST COE RARE MMOPT LIN QUAD GO AFLGO ENTROPIC KATZ\lKRUSKAL-COUNT (coupling walkers · recombination)\l", + picker [label="services/seed_picker.py + core/schedules.py\lweights · Pareto front · crowding · saturation gate\lFAST COE RARE MMOPT LIN QUAD GO AFLGO ENTROPIC KATZ DOPPLER\lKRUSKAL-COUNT (coupling walkers · recombination)\l", fillcolor="#d8eade", fontcolor="#1f4433"]; bandit [label="core/schedulers/ (16) + elo · shapley\lmonte_carlo mopt exp3 gp_ucb cmaes contextual\lducb swucb kl_ducb kl_swucb cucb hierarchical replicator epsilon_greedy mcts katz\l", fillcolor="#d8eade", fontcolor="#1f4433"]; diff --git a/docs/images/architecture.png b/docs/images/architecture.png index a28f9c26..a60e9f77 100644 Binary files a/docs/images/architecture.png and b/docs/images/architecture.png differ diff --git a/docs/images/architecture.svg b/docs/images/architecture.svg index aad964d9..55a0116d 100644 --- a/docs/images/architecture.svg +++ b/docs/images/architecture.svg @@ -4,20 +4,20 @@ - + fuzzer_tool - + cluster_start - -Startup + +Startup cluster_sched - -1 · Scheduling   —   what to fuzz next + +1 · Scheduling   —   what to fuzz next cluster_mut @@ -52,47 +52,47 @@ cli - -cli/commands.py -fuzz · rank · minimize · sweep · tmin -root-cause · replay · verify · estimate -import · ppmd + +cli/commands.py +fuzz · rank · minimize · sweep · tmin +root-cause · replay · verify · estimate +import · ppmd fuzzer - -services/fuzzer.py   —   fuzz_one() campaign loop + +services/fuzzer.py   —   fuzz_one() campaign loop cli->fuzzer - - + + profiler - -core/target_profiler.py   +   cfg · icfg · dwarf -ELF strings · magic bytes · DIV constants -call graph · CFG · DWARF line tables + +core/target_profiler.py   +   cfg · icfg · dwarf +ELF strings · magic bytes · DIV constants +call graph · CFG · DWARF line tables picker - -services/seed_picker.py  +  core/schedules.py -weights · Pareto front · crowding · saturation gate -FAST COE RARE MMOPT LIN QUAD GO AFLGO ENTROPIC KATZ -KRUSKAL-COUNT (coupling walkers · recombination) + +services/seed_picker.py  +  core/schedules.py +weights · Pareto front · crowding · saturation gate +FAST COE RARE MMOPT LIN QUAD GO AFLGO ENTROPIC KATZ DOPPLER +KRUSKAL-COUNT (coupling walkers · recombination) profiler->picker - - -distance · boundary markers   + + +distance · boundary markers   @@ -104,32 +104,32 @@ profiler->ops - - -dictionary tokens   + + +dictionary tokens   fuzzer->picker - - -  pick seed + + +  pick seed tarena - -services/target_arena.py  (multi-target) -Elo over tgt_ arms: weighted round_robin wrr wfq phi -gale_shapley (core/stable_matching.py) -auction (core/assignment.py) + +services/target_arena.py  (multi-target) +Elo over tgt_ arms: weighted round_robin wrr wfq phi +gale_shapley (core/stable_matching.py) +auction (core/assignment.py) fuzzer->tarena - - -  pick target + + +  pick target @@ -141,31 +141,31 @@ bandit - -core/schedulers/ (16)  +  elo · shapley -monte_carlo mopt exp3 gp_ucb cmaes contextual -ducb swucb kl_ducb kl_swucb cucb hierarchical replicator epsilon_greedy mcts katz + +core/schedulers/ (16)  +  elo · shapley +monte_carlo mopt exp3 gp_ucb cmaes contextual +ducb swucb kl_ducb kl_swucb cucb hierarchical replicator epsilon_greedy mcts katz bandit->ops - - -  operator + + +  operator tarena->picker - - -  matched seed + + +  matched seed ops->bandit - - -reward   + + +reward   @@ -272,8 +272,8 @@ cov->picker - - + + rarity · Chao2   @@ -337,8 +337,8 @@ cmplog->bandit - - + + walls / stall   @@ -371,23 +371,23 @@ ledger->picker - - + + strata   ledger->bandit - - -stratum   + + +stratum   corpus->picker - - -queue   + + +queue   diff --git a/src/fuzzer_tool/cli/commands.py b/src/fuzzer_tool/cli/commands.py index 8c4146a0..437b38ee 100644 --- a/src/fuzzer_tool/cli/commands.py +++ b/src/fuzzer_tool/cli/commands.py @@ -4514,10 +4514,23 @@ def main() -> int: fuzz_parser.add_argument( "--schedule", default="base", - choices=("base", "fast", "coe", "rare", "mopt", "lin", "quad", "go", "aflgo", "entropic"), - help="Power schedule: base|fast|coe|rare|mopt|lin|quad|go|aflgo|entropic " + choices=( + "base", + "fast", + "coe", + "rare", + "mopt", + "lin", + "quad", + "go", + "aflgo", + "entropic", + "doppler", + ), + help="Power schedule: base|fast|coe|rare|mopt|lin|quad|go|aflgo|entropic|doppler " "(aflgo = exact AFLGo distance annealing, see --t-x; " - "entropic = libFuzzer -entropic, log-scaled rare-feature energy)", + "entropic = libFuzzer -entropic, log-scaled rare-feature energy; " + "doppler = power Doppler flow energy, SHM coverage only)", ) fuzz_parser.add_argument( "--aflgo-cooling", diff --git a/src/fuzzer_tool/core/power_doppler.py b/src/fuzzer_tool/core/power_doppler.py new file mode 100644 index 00000000..e47e3cea --- /dev/null +++ b/src/fuzzer_tool/core/power_doppler.py @@ -0,0 +1,292 @@ +"""Power Doppler seed energy: how much of a seed's coverage its mutants move. + +Ultrasound power Doppler fires N pulses at a pixel, filters out the strong +static tissue echo, and integrates what is left: energy of moving +scatterers, direction-blind, no frequency estimate, sensitive to slow flow. +Here a seed is the probe, its mutants are the pulses, edges are the pixels: + + mutants of seed s (slow time) -> X[n, e] = log2(1 + hits_e) + | + mean removal static clutter: edges every mutant hits alike + | + SVD, drop coherent flash: an early reject moves the whole path at once + | + CFAR chi^2 test flow: residual power above the ensemble noise floor + | + power = sum PD_e -> energy in [0, 1] -> SeedScorer 'doppler' + +Seeds whose mutants steer many edges locally get energy; seeds whose +mutants change nothing, or only fail parsing en bloc, do not. +""" + +from __future__ import annotations + +import functools +import math +from collections.abc import Mapping + +import numpy as np + +from fuzzer_tool.core.chi_squared import chi_squared_critical_value +from fuzzer_tool.core.lru import LRUCache + +# Mutants per ensemble (pulses per Doppler frame). +DEFAULT_ENSEMBLE = 32 +# Open ensembles held at once: each is an ENSEMBLE x edges float32 matrix. +DEFAULT_MAX_SEEDS = 64 +# Edge columns per ensemble; 64 x 32 x 2048 x 4 B bounds the matrices at 16 MiB. +DEFAULT_MAX_EDGES = 2048 +# Closed-ensemble scores kept (two floats and a frozenset each). +DEFAULT_MAX_SCORES = 4096 +# CFAR false-alarm probability per edge. +FALSE_ALARM = 1e-3 +# A component spread over at least this share of the seed's edges is a flash. +COHERENT_FRACTION = 0.5 +# ...and over at least this many edges: one toggling edge is never a flash. +MIN_COHERENT_EDGES = 4 +# Ordered-statistic CFAR: noise = this quantile of per-edge residual variance. +NOISE_QUANTILE = 0.5 +# Per-sample variance floor (log2 units) for fully deterministic targets. +NOISE_FLOOR = 1e-3 +# Singular values below this fraction of the largest are numerical zero. +SV_EPS = 1e-9 +# First column allocation; grows by doubling up to max_edges. +INITIAL_COLS = 64 + +_MIN_ENSEMBLE = 3 # mean removal + one clutter rank still leaves a dof + + +def _components(y: np.ndarray) -> tuple[np.ndarray, np.ndarray]: + """Per-component projections P = U^T y and energies s^2 (descending). + + Samples << edges, so the n x n Gram eigenproblem replaces a full SVD: + same U and s^2, O(n^2 m) through one matmul. + """ + w, u = np.linalg.eigh(y @ y.T) + p = u[:, ::-1].T @ y + return p, np.maximum(w[::-1], 0.0) + + +def _clutter_mask(p: np.ndarray, energy: np.ndarray, m: int) -> np.ndarray: + """Mask of coherent components: participation ratio spans the path. + + For unit v = P_i / s_i, PR = 1 / sum(v^4) counts the edges it moves + (one edge -> 1, k equal edges -> k); here PR = s^4 / sum(P_i^4). + """ + live = energy > energy[0] * SV_EPS**2 + p2 = p * p + pr = energy**2 / np.maximum((p2 * p2).sum(axis=1), np.finfo(float).tiny) + wide = pr >= max(COHERENT_FRACTION * m, MIN_COHERENT_EDGES) + return live & wide + + +@functools.cache +def _cfar_gain(dof: int) -> float: + """chi^2 critical value at FALSE_ALARM; dof <= ensemble, so few distinct.""" + return chi_squared_critical_value(dof, FALSE_ALARM) + + +def doppler_power(x: np.ndarray) -> tuple[float, np.ndarray, int]: + """Clutter-filtered flow power of one ensemble. + + Args: + x: Samples x edges matrix of log hit counts. + + Returns: + (power, flow mask per edge, clutter rank removed). + """ + n, m = x.shape + none = np.zeros(m, dtype=bool) + if n < 2 or m == 0: + return 0.0, none, 0 + + # Order-0 wall filter: drop the DC (static path). + y = x - x.mean(axis=0) + if not y.any(): + return 0.0, none, 0 + + # SVD wall filter: drop spatially coherent motion (flash). + p, energy = _components(y) + clutter = _clutter_mask(p, energy, m) + rank = int(clutter.sum()) + dof = n - 1 - rank + if dof < 1: + return 0.0, none, rank + + # Residual power per edge: sum over kept components of (s_i v_ie)^2. + pk = p[~clutter] + pd = (pk * pk).sum(axis=0) + + # CFAR: under H0 (noise only) pd / sigma^2 ~ chi^2(dof). + sigma2 = max(float(np.quantile(pd / dof, NOISE_QUANTILE)), NOISE_FLOOR) + flow = pd > sigma2 * _cfar_gain(dof) + return float(pd[flow].sum()) / dof, flow, rank + + +class _Ensemble: + """One seed's open slow-time matrix with an edge-id -> column map.""" + + __slots__ = ("x", "n", "m", "cap", "ids", "skeys", "sslots", "last") + + def __init__(self, length: int, cap: int) -> None: + cols = min(INITIAL_COLS, cap) + self.x = np.zeros((length, cols), dtype=np.float32) + self.n = 0 + self.m = 0 + self.cap = cap + self.ids = np.zeros(cols, dtype=np.int64) # column -> edge id + self.skeys = np.zeros(0, dtype=np.int64) # sorted edge ids + self.sslots = np.zeros(0, dtype=np.int64) # column of skeys[i] + # Previous sample's (ids, columns, kept): mutants mostly replay the + # seed's path in the same SHM order, so the lookup is usually reused. + self.last: tuple[np.ndarray, np.ndarray, np.ndarray] | None = None + + def _grow(self, need: int) -> None: + cols = self.x.shape[1] + if need <= cols: + return + + cols = min(max(need, cols * 2), self.cap) + x = np.zeros((self.x.shape[0], cols), dtype=np.float32) + x[:, : self.m] = self.x[:, : self.m] + ids = np.zeros(cols, dtype=np.int64) + ids[: self.m] = self.ids[: self.m] + self.x, self.ids = x, ids + + def columns(self, ids: np.ndarray) -> tuple[np.ndarray, np.ndarray]: + """Columns for *ids* (unique), adding new edges while room lasts. + + Returns (columns of kept ids, kept mask over *ids*). + """ + last = self.last + if last is not None and np.array_equal(last[0], ids): + return last[1], last[2] + + cols, kept = self._lookup(ids) + self.last = (ids, cols, kept) + return cols, kept + + def _lookup(self, ids: np.ndarray) -> tuple[np.ndarray, np.ndarray]: + """Uncached :meth:`columns`; sorted needles keep searchsorted cache-friendly.""" + k = self.skeys.size + order = np.argsort(ids) + pos = np.empty(ids.size, dtype=np.int64) + pos[order] = np.searchsorted(self.skeys, ids[order]) + pos = np.minimum(pos, max(k - 1, 0)) + kept = self.skeys[pos] == ids if k else np.zeros(ids.size, dtype=bool) + slots = np.zeros(ids.size, dtype=np.int64) + slots[kept] = self.sslots[pos[kept]] + + # New edges: absent from earlier samples, i.e. zero there already. + fresh = np.flatnonzero(~kept)[: self.cap - self.m] + if fresh.size: + new = np.arange(self.m, self.m + fresh.size) + self._grow(self.m + fresh.size) + self.ids[new] = ids[fresh] + self.m += fresh.size + slots[fresh] = new + kept[fresh] = True + keys = np.concatenate([self.skeys, ids[fresh]]) + order = np.argsort(keys, kind="stable") + self.skeys = keys[order] + self.sslots = np.concatenate([self.sslots, new])[order] + return slots[kept], kept + + +class PowerDoppler: + """Per-seed power Doppler over successive mutant executions. + + Args: + ensemble: Mutants per closed ensemble. + max_seeds: Open ensembles kept (LRU). + max_edges: Edge columns per ensemble; extra edges are dropped. + max_scores: Closed-ensemble scores kept (LRU). + """ + + def __init__( + self, + ensemble: int = DEFAULT_ENSEMBLE, + max_seeds: int = DEFAULT_MAX_SEEDS, + max_edges: int = DEFAULT_MAX_EDGES, + max_scores: int = DEFAULT_MAX_SCORES, + ) -> None: + if ensemble < _MIN_ENSEMBLE: + raise ValueError(f"ensemble must be >= {_MIN_ENSEMBLE}, got {ensemble}") + if max_edges < 1: + raise ValueError(f"max_edges must be >= 1, got {max_edges}") + self._ensemble = ensemble + self._max_edges = max_edges + self._open: LRUCache = LRUCache(max_seeds) + self._scores: LRUCache = LRUCache(max_scores, on_evict=self._evicted) + self._max_power = 0.0 + self._max_stale = False + self._closed = 0 + self._dropped = 0 + self._samples = 0 + + def _evicted(self, _key) -> None: + self._max_stale = True + + def observe(self, seed_key: str, hits: Mapping[int, int]) -> None: + """Add one mutant execution of *seed_key*: ``{edge_id: hit count}``.""" + n = len(hits) + if not n: + return + + ids = np.fromiter(hits.keys(), dtype=np.int64, count=n) + counts = np.fromiter(hits.values(), dtype=np.int64, count=n) + ens = self._open.get(seed_key) + if ens is None: + ens = _Ensemble(self._ensemble, self._max_edges) + self._open[seed_key] = ens + + cols, kept = ens.columns(ids) + self._dropped += int(ids.size - cols.size) + ens.x[ens.n, cols] = np.log2(1.0 + counts[kept]) + ens.n += 1 + self._samples += 1 + if ens.n < self._ensemble: + return + + # Frame complete: score it, start the next one fresh. + del self._open[seed_key] + self._close(seed_key, ens) + + def _close(self, seed_key: str, ens: _Ensemble) -> None: + power, flow, _ = doppler_power(ens.x[:, : ens.m].astype(np.float64)) + old = self._scores.get(seed_key) + if old is not None and old[0] >= self._max_power: + self._max_stale = True + self._scores[seed_key] = (power, frozenset(ens.ids[: ens.m][flow].tolist())) + self._max_power = max(self._max_power, power) + self._closed += 1 + + def _peak(self) -> float: + if self._max_stale: + self._max_power = max((p for p, _ in self._scores.values()), default=0.0) + self._max_stale = False + return self._max_power + + def energy(self, seed_key: str) -> float: + """Seed's flow power normalized to [0, 1] (log scale); 0 if unscored.""" + score = self._scores.get(seed_key) + peak = self._peak() + if score is None or peak <= 0.0: + return 0.0 + return math.log1p(score[0]) / math.log1p(peak) + + def flow_edges(self, seed_key: str) -> frozenset[int]: + """Input-sensitive edges of the seed's last closed ensemble.""" + score = self._scores.get(seed_key) + return frozenset() if score is None else score[1] + + def stats(self) -> dict[str, float]: + """Diagnostics for reports.""" + return { + "samples": self._samples, + "open": len(self._open), + "scored": len(self._scores), + "ensembles": self._closed, + "dropped_edges": self._dropped, + "max_power": self._peak(), + } diff --git a/src/fuzzer_tool/core/schedules.py b/src/fuzzer_tool/core/schedules.py index 4cdce2f8..4a66acad 100644 --- a/src/fuzzer_tool/core/schedules.py +++ b/src/fuzzer_tool/core/schedules.py @@ -24,6 +24,9 @@ scoring. Seeds touching more rare/undersampled features get proportionally more mutation budget; seeds with no rare features get the schedule-neutral 1.0x factor. +- DOPPLER: power Doppler flow energy (core/power_doppler.py) — seeds whose + mutants steer many edges locally get up to max_mult; unscored or + static seeds stay at 1.0x. Honggfuzz factors (applied multiplicatively on top of schedule scoring): - Novelty decay: new-edge bonus that decays over 10 minutes @@ -175,6 +178,7 @@ class SeedScorer: "aflgo", "entropic", "katz", + "doppler", ) COOLING = ("exp", "log", "lin", "quad") @@ -239,6 +243,8 @@ def score( # K-Scheduler centrality (normalized 0-1 from the katz arm) katz_energy: float = 0.0, stack_depth: int = 0, + # Power Doppler flow energy (normalized 0-1 from PowerDoppler) + doppler_energy: float = 0.0, ) -> float: """Compute the energy score for a queue entry. @@ -331,6 +337,10 @@ def score( # queue (same clamp rationale as the paper's AFL integration). norm = min(max(katz_energy, 0.0), 1.0) perf_score *= 1.0 + norm * (self.max_mult - 1) + elif self.schedule == "doppler": + # Same [1, max_mult] clamp as katz. + norm = min(max(doppler_energy, 0.0), 1.0) + perf_score *= 1.0 + norm * (self.max_mult - 1) # ── Honggfuzz power factors (applied on top of schedule) ──────── # Positional: this runs once per seed pick, and a 15-keyword call diff --git a/src/fuzzer_tool/services/fuzzer.py b/src/fuzzer_tool/services/fuzzer.py index 1d144e4a..9a88204e 100644 --- a/src/fuzzer_tool/services/fuzzer.py +++ b/src/fuzzer_tool/services/fuzzer.py @@ -2344,6 +2344,12 @@ def __init__( ) self._power_schedule = schedule self._last_perf_score = 100.0 # default multiplier (1x) + # Power Doppler flow energy: fed per execution, read per pick. + self._doppler = None + if schedule == "doppler": + from fuzzer_tool.core.power_doppler import PowerDoppler + + self._doppler = PowerDoppler() # Seed key: computed on demand from corpus manager. Caching the # full bytes object as a dict key pinned every unique mutation in @@ -6356,6 +6362,16 @@ def fuzz_one(self, data: bytes) -> bool: ): self._exec_perplexity.observe(scanned_shm.get_edge_counts()) + # Power Doppler slow-time sample: this mutant's hit counts, filed + # under its parent seed. Truncated executions skipped as above. + if ( + self._doppler is not None + and scanned_shm is not None + and not is_crash + and not is_timeout + ): + self._doppler.observe(self._seed_key(data), scanned_shm.get_edge_counts()) + # Performance novelty: an edge whose trip count grew substantially # past anything seen before. The hit-count buckets saturate (129 and # 10^6 are the same bucket), so this is the only signal that stays @@ -9226,6 +9242,8 @@ def _print_enabled_features(self) -> None: groups["Seed selection"].append("aflgo") if getattr(self, "_katz_channel", None) is not None: groups["Seed selection"].append("katz") + if getattr(self, "_doppler", None) is not None: + groups["Seed selection"].append("doppler") if getattr(self, "_tang", None) is not None: groups["Seed selection"].append("tang") if getattr(self, "_kruskal_count", None) is not None: @@ -9773,6 +9791,9 @@ def run(self, iterations=0, max_execs=0): if getattr(self, "_katz_channel", None) is not None else 0.0 ), + doppler_energy=( + self._doppler.energy(seed_key) if self._doppler is not None else 0.0 + ), **hf_kwargs, ) else: diff --git a/tests/test_power_doppler.py b/tests/test_power_doppler.py new file mode 100644 index 00000000..1e75d5ab --- /dev/null +++ b/tests/test_power_doppler.py @@ -0,0 +1,263 @@ +"""Power Doppler seed energy (core/power_doppler.py). + +Slow time = successive mutants of one seed; pixel = edge; signal = log hit +count. SVD drops spatially coherent components (clutter: the shared path, +or an early-reject "flash" moving the whole path at once); CFAR keeps edges +whose residual power beats the noise floor (flow: input-sensitive edges). +""" + +from __future__ import annotations + +import shutil +import subprocess +from pathlib import Path + +import numpy as np +import pytest + +from fuzzer_tool.core.power_doppler import PowerDoppler, _components, doppler_power +from fuzzer_tool.core.schedules import SeedScorer + +ROOT = Path(__file__).resolve().parent.parent +SHIM = ROOT / "src" / "fuzzer_tool" / "adapters" / "afl_shim.c" + +N = 16 # ensemble length used by the unit tests +PATH = 20 # edges on the seed's fixed path + + +def _static(n=N, m=PATH, level=3.0): + """Ensemble where no mutant moves anything: pure clutter.""" + return np.full((n, m), level) + + +def _with_flow(x, pattern): + """Append one edge whose value follows *pattern* (local flow).""" + return np.column_stack([x, np.asarray(pattern, dtype=float)]) + + +def _toggle(n=N, period=3): + """Edge hit in 1 of every *period* mutants (slow flow).""" + return [1.0 if i % period == 0 else 0.0 for i in range(n)] + + +def _feed(pd, key, rows, ids): + for row in rows: + pd.observe(key, {e: int(c) for e, c in zip(ids, row, strict=True)}) + + +class TestDopplerPower: + def test_falsify_static_ensemble_has_no_flow(self): + power, flow, rank = doppler_power(_static()) + + assert power == 0.0 + assert not flow.any() + assert rank == 0 + + def test_single_toggling_edge_is_the_only_flow(self): + x = _with_flow(_static(), _toggle()) + + power, flow, _ = doppler_power(x) + + assert power > 0.0 + assert flow.tolist() == [False] * PATH + [True] + + def test_adversarial_flash_is_clutter_not_flow(self): + # Early reject: every path edge drops to 0 together in half the mutants. + x = _static() + x[::2] = 0.0 + + power, flow, rank = doppler_power(x) + + assert rank >= 1 + assert not flow.any() + assert power == 0.0 + + def test_adversarial_flow_survives_flash(self): + # Flash and local flow are not orthogonal: leakage must not hide flow. + x = _static() + x[::2] = 0.0 + x = _with_flow(x, _toggle(period=4)) + + _, flow, rank = doppler_power(x) + + assert rank >= 1 + assert flow[PATH] + assert not flow[:PATH].any() + + def test_gram_components_match_lapack_svd(self): + # Reference: LAPACK SVD energies s^2 and per-edge (s v)^2 per component. + y = np.random.default_rng(7).normal(size=(N, 50)) + y -= y.mean(axis=0) + _, s, vt = np.linalg.svd(y, full_matrices=False) + + p, energy = _components(y) + + np.testing.assert_allclose(energy, s**2, rtol=1e-9, atol=1e-9) + np.testing.assert_allclose(p * p, (s[:, None] * vt) ** 2, rtol=1e-6, atol=1e-9) + + def test_degenerate_shapes(self): + assert doppler_power(np.zeros((1, 5)))[0] == 0.0 + assert doppler_power(np.zeros((N, 0)))[0] == 0.0 + assert doppler_power(np.zeros((N, 3)))[0] == 0.0 + + +class TestPowerDoppler: + def test_energy_zero_until_ensemble_closes(self): + pd = PowerDoppler(ensemble=N) + rows = _with_flow(_static(), _toggle()) + ids = list(range(PATH + 1)) + + _feed(pd, "s", rows[:-1], ids) + assert pd.energy("s") == 0.0 + + _feed(pd, "s", rows[-1:], ids) + assert pd.energy("s") == 1.0 + assert pd.flow_edges("s") == frozenset({PATH}) + + def test_falsify_static_seed_gets_no_energy(self): + pd = PowerDoppler(ensemble=N) + ids = list(range(PATH + 1)) + _feed(pd, "flow", _with_flow(_static(), _toggle()), ids) + _feed(pd, "static", _with_flow(_static(), [0.0] * N), ids) + + assert pd.energy("static") == 0.0 + assert pd.energy("flow") == 1.0 + + def test_adversarial_flash_seed_ranks_below_flow_seed(self): + pd = PowerDoppler(ensemble=N) + ids = list(range(PATH)) + flash = _static(level=200.0) + flash[::2] = 0.0 + _feed(pd, "flash", flash, ids) + _feed(pd, "flow", _with_flow(_static(), _toggle()), ids + [PATH]) + + assert pd.energy("flash") < pd.energy("flow") + + def test_edge_first_seen_late_counts_as_flow(self): + # Absent edges read as 0: an edge only the last mutant reaches is flow. + pd = PowerDoppler(ensemble=N) + big = 10**9 + ids = list(range(PATH)) + _feed(pd, "s", _static()[:-1], ids) + _feed(pd, "s", [[3] * PATH + [1]], ids + [big]) + + assert big in pd.flow_edges("s") + + def test_unknown_seed_is_neutral(self): + pd = PowerDoppler(ensemble=N) + assert pd.energy("nope") == 0.0 + assert pd.flow_edges("nope") == frozenset() + + def test_saturated_counts_do_not_overflow(self): + pd = PowerDoppler(ensemble=N) + rows = [[0xFFFFFF if i % 2 else 0, 1] for i in range(N)] + _feed(pd, "s", rows, [1, 2]) + + assert np.isfinite(pd.energy("s")) + + def test_memory_bounded_by_seed_cap(self): + cap = 4 + pd = PowerDoppler(ensemble=N, max_seeds=cap) + for k in range(cap * 3): + _feed(pd, f"s{k}", _static(n=2), list(range(PATH))) + + assert pd.stats()["open"] <= cap + + def test_memory_bounded_by_edge_cap(self): + cap = 8 + pd = PowerDoppler(ensemble=N, max_edges=cap) + _feed(pd, "s", _static(n=1, m=PATH), list(range(PATH))) + + assert pd.stats()["dropped_edges"] == PATH - cap + + def test_bad_parameters_rejected(self): + with pytest.raises(ValueError): + PowerDoppler(ensemble=2) + with pytest.raises(ValueError): + PowerDoppler(max_seeds=0) + with pytest.raises(ValueError): + PowerDoppler(max_edges=0) + + +class TestDopplerSchedule: + def _score(self, energy): + return SeedScorer("doppler").score( + exec_us=100, + avg_exec_us=100, + bitmap_size=10, + avg_bitmap_size=10, + handicap=0, + depth=0, + fuzz_level=0, + n_fuzz=0, + total_execs=1, + doppler_energy=energy, + ) + + def test_falsify_zero_energy_is_neutral(self): + assert self._score(0.0) == SeedScorer("base").score(100, 100, 10, 10, 0, 0, 0, 0, 1) + + def test_full_energy_hits_max_mult(self): + sc = SeedScorer("doppler") + assert self._score(1.0) == self._score(0.0) * sc.max_mult + + def test_adversarial_energy_out_of_range_is_clamped(self): + assert self._score(-5.0) == self._score(0.0) + assert self._score(50.0) == self._score(1.0) + + +_DRIVER = r""" +#include +#include +int main(int argc, char **argv) { + FILE *f = fopen(argv[1], "rb"); if (!f) return 0; + unsigned char buf[64]; size_t n = fread(buf, 1, sizeof buf, f); fclose(f); + for (size_t i = 0; i < n; i++) { + uint32_t reps = 1 + (buf[i] & 7); + for (uint32_t r = 0; r < reps; r++) { + uint32_t guard = 1 + (buf[i] >> 2); + __sanitizer_cov_trace_pc_guard(&guard); + } + } + return 0; +} +""" + + +@pytest.mark.skipif(shutil.which("clang") is None, reason="no clang") +def test_fuzz_loop_feeds_doppler(tmp_path): + """--schedule doppler wiring: executions reach the ensembles.""" + from fuzzer_tool.services.fuzzer import Fuzzer + + src, exe = tmp_path / "drv.c", tmp_path / "drv" + src.write_text(_DRIVER) + r = subprocess.run( + ["clang", "-O1", "-include", str(SHIM), "-o", str(exe), str(src)], + capture_output=True, + text=True, + ) + if r.returncode != 0: + pytest.skip(f"driver failed to build: {r.stderr[:300]}") + corpus = tmp_path / "corpus" + corpus.mkdir() + (corpus / "a").write_bytes(b"hello world") + f = Fuzzer( + target=str(exe), + corpus_dir=str(corpus), + crashes_dir=str(tmp_path / "crashes"), + max_len=64, + timeout=1, + mutations_per_input=2, + quiet_stats=True, + use_coverage=True, + file_mode=True, + schedule="doppler", + ) + assert f.shm_cov is not None + # Short frames so ensembles close within the run and energy is read back. + f._doppler = PowerDoppler(ensemble=3) + f.run(iterations=100_000, max_execs=600) + + stats = f._doppler.stats() + assert stats["samples"] >= 500 + assert stats["ensembles"] > 0