You'll never write evals. So it writes them from your own prompt.
Point it at a system prompt and a handful of inputs. It mines the prompt for the rules you already wrote, turns each into a checkable assertion, runs them against a model, and tells you what broke.
npm install
npm run exampleTeams ship LLM features, then change a prompt or upgrade a model and find out it broke from a customer. The reason isn't laziness about testing — it's that writing eval cases is tedious, unrewarding work nobody is assigned. Building the harness was never the hard part.
So the wedge is: don't ask anyone to write test cases. Your system prompt is already a specification in prose — "always respond in JSON", "never speculate about pricing", "keep it under 400 characters". Those are tests somebody wrote months ago without realizing it. driftwood recovers them.
- Mine — the system prompt is parsed into individually testable criteria, each grounded in a verbatim quote so a generated suite can be audited.
- Generate — each case gets concrete assertions, biased hard toward deterministic checks (format, length, literals, refusal detection) that are free to run and never flake.
- Run — assertions are evaluated against a candidate model. Rubric assertions for a case are graded in a single call, must cite evidence from the output, and pass when uncertain.
- Report — pass/fail with the failing evidence, plus latency and token cost. Non-zero exit code, so it drops into CI as-is.
Judge reliability is the whole ballgame: one loud false positive costs more trust than five true findings. That's why the generator is told to prefer code over judgement, and the grader is told to pass on ambiguity.
# The demo: mine, generate, run, report
npm run driftwood -- report --prompt examples/support-prompt.txt --cases examples/cases.jsonl
# Generate a suite and commit it
npm run driftwood -- suite --prompt ./prompt.txt --cases ./cases.jsonl --out suite.json
# Re-run the committed suite against a different model
npm run driftwood -- run --suite suite.json --model claude-opus-5
# The one that matters: run twice and report only what CHANGED
npm run driftwood -- check --suite suite.json --base-model claude-sonnet-5 --model claude-opus-5
# Same thing from two saved runs, formatted for a PR comment
npm run driftwood -- compare --base before.json --head after.json --markdownrun tells you which assertions fail. That's a linter, and it's only mildly
useful — some assertions were already failing and nobody cares.
check tells you what your change broke. It ignores pre-existing failures
entirely, which is the difference between a check people act on and a check
people mute.
Running this suite against the same model on both sides produced one regression and one improvement — the same shape as a real model comparison. LLM outputs vary between identical calls, so a single observation is not a fact.
--repeat N evaluates each case N times per side. An assertion is only called a
regression when it passed every baseline attempt and failed every candidate
attempt. Anything that flips is reported as unstable instead:
npm run driftwood -- check --suite suite.json --base-model X --model Y --repeat 3Unstable assertions aren't noise to hide — they're a finding. An assertion that can't make up its mind usually means the prompt is under-specified on that point, which is something the author wants to know.
Two exit-code rules follow from that:
- A deterministic assertion that used to pass and now fails exits non-zero. These never flake, so they're safe to block a merge on.
- A model-graded assertion that flips is reported but does not fail the
build unless you pass
--strict, because a rubric verdict can change on identical behaviour. One loud false positive costs more trust than five true findings.
Cases are JSONL, one object per line. Only input is required:
{"id": "refund", "input": "Can I get a refund?", "output": "<known-good response>"}output (optional) is treated as a baseline the new response must agree with
factually — not match verbatim. context (optional) enables grounding checks
for RAG calls. This is the same shape a capture SDK will emit, so the hosted
product is a drop-in upgrade rather than a migration.
Requires Node 24+ (TypeScript runs directly — no build step; tsc is
typecheck-only).
Get an API key from console.anthropic.com → Settings → API keys. This is billed separately from a Claude.ai subscription; the console needs its own credit balance.
cp .env.example .env # then paste your key into .env — it's gitignoredThe npm scripts load .env automatically. An exported ANTHROPIC_API_KEY works
too and takes precedence.
Model tiering is the main cost lever and is set in .env.example:
| Role | Default | Why |
|---|---|---|
| generator | claude-opus-5 |
Once per suite, quality-critical |
| judge | claude-sonnet-5 |
Every case, every run — high volume, low effort |
| target | claude-sonnet-5 |
The model under test |
packages/core/src/
spec-miner.ts system prompt -> criteria
generate.ts criteria + case -> assertions
assertions/
deterministic.ts free, never flaky, catches most real regressions
judge.ts batched, evidence-citing, passes on ambiguity
run.ts execute a suite against a candidate model
report.ts terminal output
cli.ts report | suite | run
Planned next: packages/capture-node and packages/capture-python (thin —
wrap, redact, batch-POST), then sampling and clustering over captured traffic so
the suite comes from real production distribution instead of a hand-written file.
npm run dev # http://localhost:3737Paste a system prompt and a few real questions, watch the suite get generated and run, keep the JSON. No signup — the magic moment has to happen before you ask anyone for an email address.
Results stream as NDJSON so criteria appear the moment they're mined and cases report as they finish. A sixty-second wait with no feedback reads as broken.
The hosted report spends our own API key, so app/api/report/limiter.ts guards
it at two levels: 5 reports per IP per hour and 100 per day globally.
Both are configurable with DRIFTWOOD_LIMIT_PER_IP and DRIFTWOOD_LIMIT_PER_DAY.
Counters live in a shared store, picked by whichever credentials are present:
| Store | Env | Setup |
|---|---|---|
| Supabase (Postgres) | SUPABASE_URL, SUPABASE_SERVICE_ROLE_KEY |
run apps/web/supabase/rate_limits.sql once |
| Upstash (Redis) | UPSTASH_REDIS_REST_URL, UPSTASH_REDIS_REST_TOKEN |
nothing |
| In-memory | — | fallback only |
Configure one before exposing this publicly. On serverless, every cold start gets a fresh Map, so the in-memory limit resets constantly — it's a speed bump, not a control. If the configured store errors, the limiter degrades to in-memory rather than failing open or taking the site down.
The Supabase path does its increment-and-expire inside a single Postgres function, so two requests arriving together can't both read "4" and both decide they're under a limit of 5.
Rejected requests never count against either budget. A request that fails validation spends nothing, and charging it would lock someone out over typos in a tool whose entire loop is "edit the prompt, run it again".
The web app lives in apps/web and imports packages/core directly.
- Root directory:
apps/web - Build command:
npm run build(run from the repo root so workspaces resolve) - Required env:
ANTHROPIC_API_KEY - Strongly recommended:
UPSTASH_REDIS_REST_URL,UPSTASH_REDIS_REST_TOKEN
Use a separate API key with a spend limit for the deployment, not the one you develop against. The limiter caps request volume; a spend cap on the key itself is what caps the actual bill if anything gets past it.
examples/github-actions-evals.yml is a ready-to-copy GitHub Action: it runs
the committed suite on every PR that touches a prompt, posts the diff as a PR
comment, and fails the build on a deterministic regression. Copy it to
.github/workflows/ and set an ANTHROPIC_API_KEY secret.
It updates its comment in place rather than posting a new one per push. A PR carrying nine eval comments gets muted, and a muted check is exactly as useful as no check.
npm testThe deterministic evaluator and the diff logic are pure functions, so they're covered by real tests that need no API key. Everything that calls Anthropic — mining, generation, judging — is currently exercised only by running it.
Verified: typecheck clean, 19 tests passing, CLI runs.
Written but not yet executed against the API: spec mining, assertion generation, and judging. These need a key and a first real run.
Not built: traffic capture SDKs, clustering/sampling, run history, web dashboard, billing.
MIT. Use it, fork it, run it against your own key.