Assay

Confidence-gated invoice extraction. A model reads an invoice into a structured record, every field carries a confidence score, and anything the model is not sure about routes to a human instead of being accepted silently.

The premise is that a 95%-accurate extractor is useless unsupervised, because the bad 5% is indistinguishable from the good 95% at the point of use — someone reads every document anyway. What makes it useful is a trustworthy ordering: rank the output by how likely it is to be wrong, and a threshold splits one accuracy number into two populations, a slice that posts unreviewed and a remainder that goes to a person. Calibration matters more than raw accuracy, and everything in the repository follows from that.

The scores are measured, not asserted — see Results and eval/RESULTS.md.

Results

Measured over all 500 documents of the DocILE val split with Qwen2.5-VL-7B-Instruct, text-layout prompt, greedy decoding. Regenerate with make eval && make report; the full tables are in eval/RESULTS.md.

483 documents extracted, 17 failed. 5,670 scoreable header fields, 57.1% accurate.

The ordering works

Aggregation	+ checks	AUROC	ECE	AURC
min	yes	0.733	0.204	0.258
min	no	0.705	0.228	0.271
geometric mean	no	0.615	0.355	0.315
mean	no	0.610	0.361	0.317
random ordering		0.500		0.429

Two of the four open questions in docs/architecture.md are now answered.

min is the right aggregation, and not marginally: 0.705 AUROC against 0.610 for the mean. The hypothesis was that a field is wrong if any part of it is wrong, so the weakest token should set the score. It holds. Averaging lets a long, mostly-certain value hide the one token where the model guessed.

The arithmetic checks are worth folding in. They improve every aggregation — min goes 0.705 to 0.733 AUROC and its ECE falls from 0.228 to 0.204 — with a hand-set multiplier, no learning involved. The two signals fail differently, as predicted: logprobs catch hesitation, arithmetic catches confident nonsense.

The calibration does not

Confidence	Fields	Accuracy
0.9-1.0	2441	71.9%
0.8-0.9	810	64.0%
0.7-0.8	702	53.7%
0.5-0.6	556	33.8%
0.2-0.3	112	31.2%

Monotone, which is what the gate needs, and badly overconfident, which is what anyone reading a number needs to know. ECE 0.228. A field scoring above 0.9 is right 72% of the time, not 95%. Treat the scores as a ranking, not as probabilities, until something recalibrates them.

What the gate buys, and what it does not

Threshold	Coverage	Accuracy accepted	Errors caught
0.70	69.7%	67.1%	46.4%
0.85	51.0%	70.7%	65.2%
0.99	14.6%	87.8%	95.8%

At the shipped 0.85 the gate holds back 65% of all errors while passing half the volume — the ordering is doing real work. But read the same curve as a business question and the answer is negative:

Error budget	Max coverage
1%	0.2%
5%	0.7%
10%	12.9%

No threshold lets this model post unreviewed at any error rate an accounts payable team would accept. That is a result about the model, not about the gate: selection cannot manufacture accuracy it was not given, and a 57%-accurate extractor has none to spare. The gate is the right architecture and the 7B text-only configuration is the wrong extractor. Raising base accuracy — page images in the prompt, column detection, a larger model — is where the next gain is, and the harness is now in place to tell whether any of it worked.

The dominant error is invention, not misreading

Outcome	Share
correct	35.6%
correct_null	21.6%
hallucinated	24.4%
wrong	15.3%
missed	3.1%

Fabricating a value for a field the document never states is more common than misreading one, by a wide margin — and per field it is worse than the aggregate suggests: of the documents that do not print a net total, the model invents one on 96.4% of them.

This is the failure the nullable schema was designed to make possible to catch, and it confirms that half of the design. The model has a legal null path and declines to take it, despite a prompt that says to. What it does not confirm is that the gate catches these: total_net scores 0.625 AUROC, barely better than coin-flipping. An abstention the model does not want to make is still expensive.

Caveats

• Support is uneven. customer_vat_id is stated on 0% of the val split, so its 85.7% accuracy is the pipeline correctly saying "absent" — and its 14.3% invention rate is the only informative number in that row. Every per-field row in eval/RESULTS.md prints support beside accuracy for this reason.
• Line items are aligned by row position, so their 40.3% accuracy measures alignment as much as extraction. Their AUROC of 0.567 does suggest cell-level confidence is close to useless on tables, which is evidence for the still-open question of whether rows need their own gate.
• Ambiguous dates are scored leniently. 2/8/00 is accepted as either reading, because the page carries no marker saying which.
• 17 documents failed: 15 overflowed a 4,096-token output cap on very long tables, and 2 produced an impossible date — \d{4}-\d{2}-\d{2} is a shape the grammar can enforce and 2019-02-31 is a semantic error only Pydantic catches, which is the division of labour the schema was built around.

Setup

Requires Python 3.12, uv, and Docker.

make dev    # venv, dependencies, .env with generated secrets, pre-commit hooks
make up     # postgres, redis, langfuse
make test

make on its own lists every target.

make dev writes .env if it is missing and tops it up if it already exists, generating any secret that is still empty. It never overwrites a value you already set — .env holds the DocILE token, which is not regenerable. DOCILE_TOKEN is the one thing it cannot fill for you.

Services

make up brings up six containers. Two are the application's:

Service	Port	For
postgres 16	5432	extraction results, review queue
redis 7	6379	job queue (db 1)
langfuse	3000	trace and eval UI

The other three — ClickHouse, MinIO, and a second Langfuse container — exist only because Langfuse v4 needs them: it keeps traces in ClickHouse, event payloads in S3 (MinIO standing in locally), and hands ingestion from its web tier to a worker over Redis. make up-core skips all four Langfuse containers and starts just Postgres and Redis, which is enough for everything except tracing.

Langfuse provisions its org, project, API keys, and login from .env on first boot, so tracing works immediately — log in at http://localhost:3000 with LANGFUSE_INIT_USER_EMAIL and LANGFUSE_INIT_USER_PASSWORD.

Everything binds to localhost only, apart from the Langfuse UI.

The schema

assay.schema.invoice defines the extraction target.

from assay.schema import Invoice, invoice_json_schema

invoice_json_schema()   # -> dict, ready for vLLM's guided_json

Every field is optional and defaults to None. That is the load-bearing property of the whole design. A constrained decoder can only emit what the grammar allows, so a required non-nullable field leaves the model no way to say "this invoice has no due date" — it must invent one. That turns an abstention into a confident error, and a confident error is precisely what a confidence-gated pipeline cannot catch: the fabricated value arrives with the same high logprob as a correctly read one.

invoice_json_schema() is not just model_json_schema(). Four things differ, each because the raw Pydantic output does not survive constrained decoding:

• No regex lookahead. Pydantic types Decimal with the pattern ^(?!^[-+.]*$)[+-]?0*\d*\.?\d*$. vLLM's backends (xgrammar, outlines) compile patterns to finite automata, which cannot express lookahead, so the pattern fails to compile or is silently dropped.
• Money is a string, not a number. A JSON number reaches Decimal through a float, and floats do not round-trip decimal cents — a VAT total of 19.19 comes back as Decimal('19.190000000000001').
• Dates carry a pattern, not format: date. format is advisory; backends may ignore it, leaving an unconstrained string. The grammar enforces the shape and Pydantic rejects impossible dates like 2026-13-45.
• Every key is required, and every type still admits null. The model must emit "due_date": null rather than omitting the key. An omitted key produces no tokens, and no tokens means nothing to score — so a silent omission would be invisible to the gate. Forcing the key gives every field a decision point with a logprob on it.

Text extraction

from assay.ingest import extract

doc = extract("00134dd365a24343b35b78c6")
doc.source            # "pdf" or "ocr"
doc.words[0].text     # "TELETIME"
doc.words[0].bbox     # (x0, y0, x1, y1), normalized to [0, 1], top-left origin
doc.text              # words joined in reading order

About 31% of DocILE PDFs are scans with no text layer. PyMuPDF returns nothing for those, silently, so extract() reads the embedded text layer when there is one and falls back to DocILE's pre-computed OCR when there is not. Both paths emit the same Word records, so callers do not need to branch on the source; doc.source reports which was used, and Word.confidence is None for text-layer words since those characters are exact rather than predicted.

Measured over the full trainval split (5,680 documents, 1.79M words): 69.2% read from the text layer, 30.8% from OCR, no failures, ~14s single-threaded.

One PyMuPDF subtlety worth knowing if you touch the geometry: word boxes come back in unrotated cropbox space while page.rect is rotation-adjusted, so on /Rotate 90 pages the two disagree and dividing by page.rect overflows [0, 1]. _pdf_words maps boxes through page.rotation_matrix first.

Layout

src/assay/
    ingest/       documents in, words out (PDF text layer, OCR fallback)
    extract/      words in, a candidate Invoice out (constrained decoding)
    validate/     arithmetic, format, and cross-field checks
    confidence/   per-field scores, and the gate that routes low-confidence work
    review/       human-in-the-loop correction of what the gate held back
    datasets/     benchmark labels, mapped onto the schema
    eval/         accuracy and calibration measurement
    api/          HTTP surface — not built yet
    schema/       the extraction target
eval/             what the harness produces: runs/ and RESULTS.md
tests/            `make test` skips anything needing the dataset
notebooks/        exploration
docs/             design notes

assay.extract is the extraction stage. Reading words off a page is ingestion, and lives in assay.ingest.

The one stage still unbuilt is api/. review/ has provenance — locating a value back on the page — but no queue.

Dataset

This project uses DocILE (Document Information Localization and Extraction Benchmark).

1. Get a token

Register at https://docile.rossum.ai/ to obtain a secret token, then set DOCILE_TOKEN in .env.

.env is gitignored. Do not commit the token — the download script interpolates it straight into the S3 URL path, so it is a live credential.

2. Download

set -a && source .env && set +a
./scripts/download_dataset.sh "$DOCILE_TOKEN" annotated-trainval data/docile --unzip

Data lands in data/, which is gitignored — it is far too large for git.

Sizes, as reported by the S3 bucket:

Split	Size
annotated-trainval	1.06 GB
synthetic	3.19 GB
unlabeled-annotations	0.30 GB
unlabeled	94 chunks, very large

The script's --help is wrong about the first split. It advertises labeled-trainval, but no such object exists in the bucket — that name 404s. The real key is annotated-trainval, matching the upstream README. The script does not validate split names (it just interpolates them into the object name), so passing the correct name works fine.

Run ./scripts/download_dataset.sh --help for chunked downloads of the large splits, and --without-pdfs to fetch pre-computed OCR only for unlabeled.

scripts/download_dataset.sh is vendored verbatim from rossumai/docile.