What is your classifier's confidence really worth?

jeval measures how well a classifier's confidence matches reality, and turns what a mistake costs
into the threshold where the machine should stop deciding and a human should start.

Install  ·  See it  ·  Get your data in  ·  Read the report  ·  Drift in CI  ·  Commands

---

Your classifier answers with a label and a confidence. Two questions follow, and jeval answers both:

1. When it says 0.9, how often is it actually right?
2. Given what a mistake costs, where should the line sit between "the machine decides" and "a human decides"?

The answers come out as one HTML file you can open offline and one YAML file your app reads. Everything is measured from labeled decision records on disk: no server, no database, no network call, no account and no token.

jeval works with anything that returns a probability: a hosted API, a gateway, a local model, a logistic regression, a scoring rule. The name comes from one model family, but the tool sits above all of them.

You get	It answers	Who reads it
report.html — one file, no dependencies	Are the confidences trustworthy, where do they break, and what is the current threshold costing you?	you
thresholds.yaml	The threshold to deploy, with an uncertainty range, and whether a segment needs its own line	your app
labels.csv	Which decisions to label next, to learn the most per answer	whoever has the answers
calibration-*.yaml — optional	A correction map your app can apply, written only when the gain is real	your app

---

Install

curl -fsSL https://raw.githubusercontent.com/rlaope/jeval/main/install.sh | sh
jeval demo --out-dir /tmp/jeval-demo    # a synthetic log, through the real report path
# open the report.html path the command prints

That is the whole loop. The installer gives you a jeval command and nothing else: no uv, no pipx, no root and no PyPI. It keeps a private environment in ~/.local/share/jeval, links jeval into ~/.local/bin, and prints the one line to add if that folder is not on your PATH yet — or pass --modify-path and it edits your shell file for you. Run it again to upgrade; sh install.sh uninstall removes everything it created.

Install nothing at all, or pin a version

uvx --from git+https://github.com/rlaope/jeval jeval --version            # current main
uvx --from git+https://github.com/rlaope/[email protected] jeval --version     # pinned tag
pip install \
  https://github.com/rlaope/jeval/releases/download/v0.2.0/jeval_cli-0.2.0-py3-none-any.whl

Every release tag is built by the release workflow, which attaches the wheel and the source archive to the GitHub release and checks the file list afterwards. From a checkout: uv sync --all-groups.

Warning

pip install jeval installs an unrelated project. That name on PyPI belongs to someone else, and this tool is not on PyPI at all. Use one of the lines above.

Python 3.10 or newer. The runtime dependencies are numpy, pydantic, pyyaml and typer.

---

Two ways to use it, one directory between them

	In your service (the library)	On the command line
What you do	collect.track(client) where the client is built; collect.resolve(...) where a human settles a case	jeval status, jeval report, jeval threshold, jeval drift
What happens	every answer and every human answer is appended to $JEVAL_ROOT/.jeval/	the same files are read, joined and measured
What it never does	raise, block, call a model, or send anything off the machine	write configuration it was not asked to, or need a network

from jeval import collect

client = collect.track(TicketClassifier(), source_key=lambda **kw: kw["ticket_id"])
...
collect.resolve(source_key=ticket.id, question="department", answer=ticket.final_department)

JEVAL_ROOT=/var/lib/jeval    # in the service
jeval status --root /var/lib/jeval
jeval report --root /var/lib/jeval

Either half works alone: the library needs no command line to record, and the command line reads a log you already have with jeval ingest. docs/library.md is the library's reference — every argument, every environment variable, the counters to check, and the limits.

---

See it

Everything below is cropped from examples/report-example.html, a report committed to this repository and generated by the command in Read the report. Every number quoted on this page can be checked against that file, and a test fails if the two disagree. The screenshots follow your GitHub theme.

Reliability. What the model claimed, against how often it was right. Points below the dashed diagonal are overconfident, the bars are 95% intervals, dot size follows the sample behind each bin, and the strip underneath shows where the decisions land. The figures a reader quotes sit beside the chart.

Cost. Expected cost per case for every candidate threshold, with the line in use and the recommended line both drawn and named. The shaded band is the flat region: thresholds the sample cannot tell apart.

Impact. What moving the line buys, said as a sentence and set as four figures, each with the one it replaces struck through. Every section works this way: the finding first, drawn at a size you can read across a room, and the table it came from folded underneath for anyone who wants to copy a number. A slider below answers "what if" without writing anything to disk.

Segments. Where the miscalibration actually lives. A grey bar is a segment with too few labels to judge.

Two more surfaces, built from the same code

• One screen with the whole argument — for a demo recording or reading the case in one sitting: the line in use, the line the cost minimum points at, both drawn and named on one chart, and what moving it buys. examples/make-demo-dashboard.py builds it from the same library, and a test rebuilds it and requires a byte-for-byte match.
• The workbench — the one-screen argument first, and the full example report folded underneath it in a native <details>, so one file answers "show me everything". examples/make-workbench.py reads examples/report-example.html at build time; no JavaScript, no external reference.

examples/README.md lists all three surfaces and the command that rebuilds each.

---

Get your data in

Start from the demo

The demo builds a synthetic log whose miscalibration is known on purpose, so there is something real for the report to find:

uv run jeval demo --out-dir examples/report-example --seed 11 --scale 0.5

Point it at a log you already have

Describe the layout instead of writing a wrapper. field_map says which column means what, questions_field names the question, and a flat column can be used as a segment:

jeval init --root ~/myproject
jeval ingest ~/myproject/decisions.jsonl --root ~/myproject
jeval ingest ~/myproject/decisions.jsonl --labels ~/myproject/resolutions.jsonl \
  --label-field final_department --label-source human_override \
  --join-on ticket_id --label-question department --root ~/myproject
jeval report --root ~/myproject --currency KRW

If your product's response shape is already known, one preset may cover it:

$ jeval ingest --preset jev-native api-decisions.jsonl
preset: jev-native (response 'response', answers 'answers', join key 'request_id')
read 4 rows
wrote 8 records to .jeval/records.jsonl

jeval ingest --list-presets shows the presets. --response-field and --source-key-field override where a preset looks.

Or hand the setup to an agent

Nobody wants to learn nine commands. Hand an agent one sentence and take the report back:

Install jeval (uvx --from git+https://github.com/rlaope/jeval jeval --help), find where my classifier's decisions are logged, describe that layout in .jeval/ingest-map.yaml, run jeval report, and show me the report file.

llms.txt is the short entry point for a machine, and docs/agent-setup.md is the longer playbook it follows. Both cover the two places where people get stuck: nothing is logged yet, and nothing is labeled yet.

Six skills ship in this repository as plain markdown, one for each job: hand the whole thing to an agent, audit the calibration, turn costs into a threshold, harvest the labels you already have, instrument a running service, and gate a model change in CI. They are written for the agent, not for you: each one carries the commands, the check that proves it worked, the failure modes that really happen, and what it must not claim.

curl -fsSL https://raw.githubusercontent.com/rlaope/jeval/main/install-skills.sh \
  | sh -s -- --list
curl -fsSL https://raw.githubusercontent.com/rlaope/jeval/main/install-skills.sh \
  | sh -s -- --host claude-code

The same six skills are exported into the layout each host expects, so nothing is hand-copied per host. Eight targets, each verified against its host's own documentation: .agents/skills, which Codex CLI, Hermes, OpenClaw, Pi, Cursor and OpenCode all read, .claude/skills for Claude Code, five host-specific roots (.cursor/skills, .hermes/skills, .opencode/skills, .pi/skills, .openclaw/skills), and one AGENTS.md digest for hosts that read a single instruction file. docs/skills.md lists them with their sources.

Or record from the service you already run

You add two lines: one where the classifier is called, one where the human answer arrives.

from jeval import collect

client = collect.track(
    TypeSafeClient(),  # your SDK, not jeval's
    method_names=("system_one",),  # the method that answers questions
    source_key=lambda **kw: kw["trace_id"],  # what a human answer is joined back on
    segment=lambda **kw: {"lang": kw.get("lang")},  # request fields to compare later
)

# where the case closes
collect.resolve(source_key=trace_id, question="department", answer=final_department)

Every command joins the answers resolve recorded onto the decisions they answer, on read, with no ingest step; jeval status shows what has accumulated and what is still missing.

Then check that collect.stats()["calls"] is not zero after the first request. A wrapper that found no method to patch is counted in no_method_found, because a silent no-op looks exactly like a working setup. Set JEVAL_ROOT=/var/lib/jeval to choose where the records go: it writes the same .jeval/records.jsonl that jeval report --root /var/lib/jeval reads. Set JEVAL_COLLECT=0 to turn collection off.

The wrapper only watches a call your code already makes. It never calls a model, never picks one, never retries, never blocks and never raises — a failed write is counted and dropped. It imports no vendor SDK, and any product-specific field name lives in a preset rather than in the core. The library reference is docs/library.md, and the full walkthrough, with the output of each step, is in docs/instrumenting-a-service.md.

Where labels come from

The usual reason people give up on calibration tools is believing they need a labeling project. You are most likely producing labels already; the work is joining them, not creating them:

You already have	Where the label is
Cases a human reviewed after escalation	the human's final answer — the truth for the model's answer
Auto-handled cases that were later reversed	the reversal — the model was wrong
Refund approvals and rejections	the outcome — a real answer, with a date

label_from:
  field: resolution.final_department   # dotted paths work
  source: human_override               # human_review | human_override | silver
  join_on: ticket_id                   # the key your log and the resolution log share
  question: department                 # which question this column answers

question: is not decoration. Every question of a request shares the same join key, so without it a department answer would also be written as the intent answer. Two rules protect your records: a harvest never overwrites a label unless you pass --overwrite, and it refuses a label the record's own question could not have produced. Refusals are counted and named, because a wrong label is worse than a missing one. The harvest rewrites .jeval/records.jsonl in place and atomically, and touches only the label fields.

Important

With no labels, jeval measures nothing. It says so and stops instead of making up a number.

---

Read the report

It reads top to bottom as one case, always in the same order: the verdict, the reliability chart, whether confidence ranks the errors at all, the cost curve with both lines named, what moving the line changes, the segments that do worst, drift before and after, and the state of the data. Every chart is inline SVG drawn by jeval — no chart library, no web font, no request to anywhere — and the file opens with the network switched off, in light or dark mode.

Each question gets its own curve and its own line, and the report prints which action that line belongs to. A report can therefore carry several thresholds without pooling them into a number that describes none of them.

What the numbers mean

A confidence is a claim. Here are five rows of the reliability table, copied from the example report — generated from synthetic data, and committed so you can check every number on this page against the file:

Confidence bin      n   Stated   Observed   Wilson 95%      Gap
0.60-0.69          26     65%       46%     [29%, 65%]   -0.188
0.74-0.77          26     76%       69%     [50%, 83%]   -0.064
0.82-0.85          26     83%       69%     [50%, 83%]   -0.140
0.91-0.95          25     93%       96%     [80%, 99%]   +0.033
0.97-1.00          26     98%      100%     [87%, 100%]  +0.016

Read the third row: the model said 83% and was right 69% of the time, and the range around that number runs from 50% to 83% — that is all a bin of 26 decisions can support.

ECE is the average gap between the confidence claimed and how often the model was right; 0 means the confidence can be taken at face value. Here is the verdict in that report, for the threshold the demo uses:

Your threshold is too low. Band 0.86-0.91 measures 75.4% accuracy on 69 decisions (of 696 labels); the threshold belongs at 0.75, above the 0.60 in use.

And what acting on it would change:

what changes	in use	recommended	change
confidence threshold	0.60	0.75	+0.15
auto rate	33%	30%	-2.9 pt
accuracy (auto)	85%	91%	+5.4 pt
cost per case	KRW 1,926	KRW 1,738	-9.8%
monthly cost	KRW 38,524,590	KRW 34,754,098	-9.8%

None of these numbers were typed in by hand, because the report itself is in the repository. Open examples/report-example.html in a browser (one 374 KB file, no network, no server), or build it again yourself:

uv run jeval demo --out-dir examples/report-example --seed 11 --scale 0.5
cp examples/report-example/report.html examples/report-example.html

Honest is not the same as useful

ECE says whether confidence is honest on average. It cannot say whether a confident answer is more likely to be right than an unconfident one, and that is the only thing a threshold can use: a model that says 0.8 on every answer and is right 80% of the time has an ECE of zero and ranks nothing, so raising its line escalates a random sample. The Discrimination section asks that second question with one risk-coverage curve per question: automate the most confident answers first and read the error rate among them, against a flat line (confidence ranks nothing) and the best any ranking could do. AUROC is the chance that a right answer carries more confidence than a wrong one — 0.50 is a coin flip — and AURC is the area under the curve. The example report reads, for intent:

AUROC 0.77 (95% CI 0.71-0.84, n=244): a right answer outranks a wrong one 77% of the time. Moderate separation; a higher threshold buys accuracy at a cost in volume.

Its error rate with every answer automated is 17.2%, and AURC is 0.077 where no ranking would leave 0.172. jeval report prints the same AUROC for each question in the terminal.

Top-1 calibration has a blind spot of its own: it averages over whichever class was predicted, so a class that is overconfident only when it is the answer — the class a cost action fires on — can hide behind classes that err the other way. For a choice question the reliability block adds a table that measures every class on its own probability, P(class) against whether the label was that class. In the example, intent has a top-1 ECE of 0.091 and per-class ECEs of 0.057 (check_balance), 0.086 (other) and 0.063 (refund_request, the class auto_refund fires on). The report names other as the worst class and says in the same sentence that its interval (0.051-0.121) overlaps another class's, so 244 labels do not settle the ranking. The demo generator splits the probability left over after the top class at random, so these per-class figures describe that synthetic map, not a real model. A map that is missing or does not sum to 1 within 0.02 — a top-k map — is set aside and counted rather than read as zeros, and a class with fewer than 30 labels is named and not measured.

Money is written the way its currency is

Costs are printed in the currency you name with --currency (default USD) — in the report and in jeval threshold's terminal output alike — using that currency's own decimals from ISO 4217: KRW and JPY have none, so a month reads KRW 38,524,590 rather than KRW 38,524,590.16; BHD and KWD have three; most others have two. A per-case average smaller than one unit keeps two significant figures instead of rounding to zero, and any unit you pass that is not an ISO code, such as credits, is printed as written with two decimals. jeval never converts between currencies and never guesses one from your locale.

You could measure this yourself with 30 lines of pandas

And you probably should, once. Your number will still differ from jeval report, because of one choice: pd.cut makes bins of equal width, while jeval makes bins of equal size. On the demo's intent question that is ECE 0.107 with equal-width bins against ECE 0.091 with jeval's — equal-width binning put 132 of 244 labels in the top bin while the others held 8 to 17, so one bin carried more than half the weight. jeval report --bins-equal-width shows the equal-width number next to the same data. Which number you ship is a decision, not a detail.

---

Catch a model change in CI

A notebook measures once. jeval drift is what you run when the model behind the API changes:

$ jeval drift --root /tmp/jeval-drift --fail-on ece-increase=0.05
costs: /tmp/jeval-drift/costs.yaml
model changed: jev-1.13.0 -> jev-1.14.0 (Sep 16)
  question    ECE before  ECE after   delta
  department       0.028      0.141  +0.113   FAIL
recommended threshold (department): 0.96 -> 0.98
  at the current 0.96: auto-rate 2% -> 5%
failed ece-increase: department: ECE 0.028 -> 0.141 (+0.113), limit 0.050
note: ECE 95% bootstrap intervals per question: department 0.025-0.056 -> 0.110-0.170. cost matrix applied: cost-derived thresholds attached to 2 of 2 slice(s).
exit 1
$ echo $?
1

This is a captured run, not a drawing, and you can rebuild the log it ran on — 1,800 synthetic decisions in which the newer model version is deliberately overconfident:

uv run python examples/make-drift-log.py /tmp/jeval-drift
uv run jeval drift --root /tmp/jeval-drift --fail-on ece-increase=0.05

Calibration is not the only thing a new version can move. --fail-on threshold-shift=0.05 fails the build when the cost-optimal line moves by more than 0.05 in either direction — on this log the line moved 0.96 to 0.98, so a limit of 0.01 fails with failed threshold-shift: department: recommended threshold 0.96 -> 0.98 (+0.02), limit 0.010. The line a reviewer approved is the one the application runs, so a version that moves it needs a new approval even when its calibration looks fine. The four checks are ece-increase, ece-above, auto-rate-drop and threshold-shift.

The threshold line appears only when a cost matrix is present. Without costs the output says recommended threshold: not available (no cost matrix was applied) rather than inventing a number, and threshold-shift is refused with exit 1 rather than passed. --save-baseline .jeval/baseline.json compares against the last measurement you accepted, which matters when the model string never changes but its behaviour does; a snapshot holds measurements, not records, so it is safe to commit. examples/ci/drift.yml is a copy-paste workflow: it runs the check, prints the markdown summary and posts it on the pull request, because jeval never holds a token.

Head to head on the same requests

When shadow traffic sends the same requests to both versions, log both with the same source_key and add --paired. jeval then compares the versions request by request, which takes request difficulty out of the difference, and can say whether the newer one is really less accurate:

$ jeval drift --root /tmp/jeval-paired --fail-on ece-increase=0.05 --paired
model changed: jev-1.13.0 -> jev-1.14.0 (Sep 01)
  question    ECE before  ECE after   delta
  department       0.036      0.035  -0.000   ok
recommended threshold: not available (no cost matrix was applied to this comparison)
paired: jev-1.13.0 -> jev-1.14.0 (the same requests, matched by source_key and question)
  department: 711 pairs
    accuracy  0.796 -> 0.745  -0.051  95% CI -0.093 to -0.010  McNemar p=0.023 (right on baseline only: 137, on current only: 101)
    ECE       0.034 -> 0.036  +0.002  95% CI -0.024 to +0.034
    Brier     0.148 -> 0.179  +0.031  95% CI +0.012 to +0.052
    verdict: current is less accurate beyond noise (McNemar p=0.023); no calibration difference the sample can resolve
  not paired: 0 without source_key, 0 at a key logged twice by one version, 0 with no partner, 189 pair(s) without a gold label on both sides, 0 with conflicting labels, 0 score
note: ECE 95% bootstrap intervals per question: department 0.026-0.068 -> 0.028-0.071.
exit 0

The ECE gate passes, because the newer version is exactly as honest about its confidence as the older one. The paired check shows that it is right less often. The log is synthetic and you can rebuild it: uv run python examples/make-paired-log.py /tmp/jeval-paired.

Every difference is current minus baseline, with a 95% bootstrap interval over pairs. Accuracy is tested with the exact McNemar test on the requests where only one version was right. The pairing rules are strict, and every record left out is counted on the not paired line: records without a source_key, keys logged twice by the same version (jeval refuses to pick one), requests only one version answered, pairs without a gold label on both sides, pairs whose two labels disagree, and score records. Below 30 pairs, a question is refused with no numbers. --paired reports but never changes the exit code; only --fail-on decides that. It cannot be combined with --by-period or --baseline, because a snapshot holds measurements, not records.

---

More from the same records

How many more labels? jeval plan shows how many more labels each question needs to tighten its range, and refuses to guess below 200 labels:

$ jeval plan --root examples/report-example --target-ci 0.05
key                       n     ECE      CI  needed
department              256   0.078   0.075  0.019: 2,442 · 0.037: 533 · 0.050: 214
intent                  244   0.091   0.081  0.020: 8,753 · 0.040: 1,326 · 0.050: 677
is_urgent               196   0.062   0.081  only 196 labels; 200 needed to fit the scaling

Which ones? jeval label ranks what to label instead of asking for a labeling project: it writes a CSV of the decisions that sit on the decision line, and applies your answers back with jeval label --apply labels.csv. It is a queue and a sheet, not a full-screen terminal app.

Can it be corrected? jeval calibrate fits a correction — temperature scaling or isotonic regression — and writes it as a YAML map your app can apply. It is measured on held-out records, never on the ones it was fitted on. If the gain does not clear the noise in your own log, it writes nothing and tells you. jeval never applies the map itself: it writes a file, your app reads it.

Does one segment deserve its own line? jeval threshold --by lang asks the question in money. A split is recommended only when the segment's best threshold moves by more than one step of the sweep and adopting it changes cost per case by more than 2%. Otherwise the output says "splitting does not pay" and names the clause that failed.

---

What jeval does not do

• No labels, no measurement. Without a human's answer on a row, jeval cannot tell whether a confident prediction was right. It stops instead of making up a number.
• The costs are yours. Every recommended threshold follows directly from the figures in costs.yaml. Wrong costs give wrong thresholds, and jeval cannot know that a refund costs more than an hour of support at your company.
• A wide range is a labeling problem, not an analysis problem. jeval plan says how many more labels you need; nothing in the output can rescue a sample that is too small.
• Correctness is only defined for choice and yes/no noul questions. A yes/no answer is measured on the probability that it is right, max(p, 1 - p), the same scale as a choice answer. score questions get MAE, RMSE and rank agreement, are never folded into binary accuracy, and the excluded count is printed.
• Silver labels give you an agreement rate, not an accuracy, and the report says so. A label with no label_source counts as silver, never as gold.
• The threshold in use is only known if you say it. Pass --current, or keep a thresholds.yaml. If nothing is deployed, the report says so instead of comparing the recommendation with itself.
• --bins 1 is refused. One bin averages every decision together, so ECE collapses toward zero and the report reads as "trustworthy" whatever the data says.
• A projection is an estimate. The label projection fits the scaling of the range width from your own subsamples, prints the fit with its residual, and says when it fell back to 1/sqrt(n).
• The demo is synthetic. jeval demo shows what the tool computes, not what a real model does. The numbers on this page come from that demo, and the report says so itself.
• No gateway, no router, no hosting, no prompt tuning, no fine-tuning, no dashboard, no accounts. Adapters and request-path libraries stay out of scope: jeval writes files, your app writes the request path.

---

Commands

tests/test_documented_features.py fails if this table and the real CLI disagree in either direction, and no command here is a stub that only looks implemented.

Command	What it does	Status
jeval init	create .jeval/ config and the ingest map	implemented
jeval ingest	JSONL/CSV logs to decision records; --preset jev-native reads a decision API's own response log; --labels brings in human answers	implemented
jeval report	the report itself: verdict, reliability, discrimination, cost, impact, segments, score questions, labels and correction, drift, data quality — one HTML file, or --format md for a summary you can paste	implemented
jeval threshold	cost matrix to a threshold per action, with an uncertainty range, written to thresholds.yaml; --by <segment> answers whether splitting pays	implemented
jeval drift	compare model versions or periods, save a baseline, fail a build with --fail-on, see where the threshold moved per slice, and compare two versions head to head on shared requests with --paired	implemented
jeval label	a labeling queue with a CSV sheet to fill in	implemented
jeval plan	how many more labels each question needs for a tighter range	implemented
jeval calibrate	a temperature or isotonic correction map, measured on held-out data, exported as YAML	implemented
jeval demo	a synthetic log with a known miscalibration, through the same report path	implemented
jeval status	what the library (or ingest) has collected, per model and question, and how many more gold labels the verdict needs	implemented

jeval report writes one self-contained HTML file plus a short terminal summary. The report never writes configuration, never fetches anything at run time and never posts to a pull request.

---

The data

Everything is a decision record — one question per record — in .jeval/records.jsonl, so it diffs, streams and works with whatever tool you already read files with:

{
  "id": "rec_1f4c2a...",
  "ts": "2026-09-20T10:31:02Z",
  "model": "jev-1.13.0",          // the model string from the response; the drift anchor
  "question_key": "department",
  "question_type": "choice",     // choice | score | noul
  "prediction": "billing",
  "confidence": 0.91,            // top-1 probability; for noul, |P(yes) - 0.5| * 2
  "probabilities": {"billing": 0.91, "technical": 0.06, "other": 0.03},
  "label": "billing",            // null until something labels it
  "label_source": "human_override",
  "segment": {"lang": "ko", "tier": "pro"},
  "state_tokens": 1840,
  "latency_ms": 210,
  "cost_usd": 0.00008
}

A yes/no (noul) record stores its confidence as the distance from a coin flip, so a 50/50 answer sits at zero; jeval measures and thresholds it on max(p, 1 - p), the probability that the answer is right. When you ingest yes/no rows, map a probabilities field holding the probability of yes (and no) rather than a confidence column: a bare confidence for a noul row is read on the distance scale, so 0.9 there means P(yes) = 0.95.

Three fields carry most of the value:

• One record per question. A request that answers "is this a refund?" and "how annoyed is this customer?" can be reliable on one and useless on the other. Pooled metrics hide exactly that.
• model is kept exactly as it came. jev-latest points at different models over time, and a threshold tuned against it is tuned against something that no longer exists.
• state_tokens is kept. Accuracy tends to fall as the state grows, so --by state_tokens is one of the more useful reports you can run.

---

Docs

llms.txt	the short entry point for an agent
docs/agent-setup.md	the setup playbook, including the no-labels path
docs/library.md	the library in your service: collect.track, collect.resolve, the environment variables, the counters, the limits
docs/instrumenting-a-service.md	set it up on a running service, step by step, with the output of every step
docs/skills.md	the agent skill pack, and where each host reads it
examples/report-example.html	a real generated report
examples/demo-dashboard.html	the whole argument on one screen
examples/workbench.html	the screen and the report on one page
examples/README.md	every example surface, its design language, and the command that rebuilds it
examples/ci/drift.yml	a CI starting point
CHANGELOG.md	what changed, and why

Development

uv sync --all-groups
uv run pytest
uv run ruff format --check .
uv run ruff check .
uv run mypy jeval
uv run jeval demo --out-dir /tmp/jeval-demo

The statistics are the product. tests/test_synth.py builds decision logs with a known miscalibration and checks that the code recovers it: a calibrated sample must report ECE near zero, an inflated one must report the inflation, and a sample that is 95% accurate but always claims 0.99 must be caught as overconfident. If a change to the statistics cannot pass those tests, the change is wrong, not the tests. See CONTRIBUTING.md for the commit convention and CLAUDE.md for the rules that apply to automated contributors.

License

Apache-2.0. See LICENSE.