Skip to content

[Not for merge] Ephyra type-checker harness (#16102) - #16113

Closed
alpaylan wants to merge 1 commit into
elixir-lang:mainfrom
alpaylan:ephyra-type-checker-harness
Closed

alpaylan wants to merge 1 commit into
elixir-lang:mainfrom
alpaylan:ephyra-type-checker-harness

Conversation

@alpaylan

Copy link
Copy Markdown

Not for merge. Shared on request in #16102: the harness that found #16101, #16102 and #16105–#16111, as it was when the run stopped, cleaned up only enough to run outside our machine.

How it found these bugs

Ephyra is a property-based testing library in Rust, and ephyra-agent is a loop of Claude agents around it. The agents get a target (here, Elixir main and its type system docs), write a harness, and run it continuously in 30-minute search windows. They triage each new failure (shrink it, check it on main and v1.20.5, look for an existing issue). When a bug is real they patch it locally so the search can get past it, and they change the generator wherever Erlang cover shows Module.Types* code it never reaches.

Rust is only the driver. It generates small modules (multi-clause defs, patterns, guards, case/fn/with/for, maps, tuples, lists, bitstrings, structs, Map/Tuple/Enum calls) and shrinks failures. The checking happens in worker.exs, a long-lived VM built from the tree under test. For each module it:

  1. compiles it and reads the inferred signatures from the ExCk chunk, runs the calls, and checks every returned value against the inferred return type;
  2. checks warnings that claim code never runs or always fails against probes that record what actually ran;
  3. checks Module.Types.Descr operations (union, intersection, difference, subtype?, map/tuple/list operators) for consistency, and parses printed types back;
  4. records any exception from Module.Types while compiling a valid module.

Over the run (about 9M cases), cover measured 82% of Module.Types* lines. Every issue was re-checked by hand on current main and searched against existing issues before filing.

Running it

From the repo root (needs OTP 27+, stable Rust and rsync; see ephyra-harness/README.md):

bash ephyra-harness/build.sh . /tmp/eb
bash ephyra-harness/run.sh /tmp/eb 1 60 /tmp/out
ls /tmp/out/findings        # each has case.ex and a repro.sh

On today's main (0823e72) a 15-second run still reports a few failures. They haven't been triaged, so they may be harness false positives rather than bugs.

The same harness is in datadog-labs/ephyra#34, with the 9 reports.

Assisted-by: Claude Code:Claude Opus 5.5

Shared on request in elixir-lang#16102: the generated harness that found elixir-lang#16101,
elixir-lang#16102 and elixir-lang#16105-elixir-lang#16111.

Assisted-by: Claude Code:Claude Opus 5.5
@josevalim

Copy link
Copy Markdown
Member

Excellent, thanks for sharing!

@josevalim josevalim closed this Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants