Skip to content

Commit 33d6770

Browse files
jasperdewjasperdew
committed
docs(readme): state the held-out benchmark results with their claim IDs
Signed-off-by: jasperdew <[email protected]>
1 parent db16419 commit 33d6770

1 file changed

Lines changed: 11 additions & 15 deletions

File tree

‎README.md‎

Lines changed: 11 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -166,21 +166,17 @@ Extraction runs in one of two tiers:
166166
Balanced/tables: glibc Linux, macOS, Windows; not on musl Linux
167167
(loading returns a typed ModelError).
168168

169-
`balanced` scored higher than Docling on FinTabNet and PubTables-1M, level with
170-
it on ParseBench, and ahead of pymupdf4llm on FinTabNet and ParseBench
171-
[A38, A39, A40, 2026-10-14]. On PubTables-1M against pymupdf4llm our score is
172-
higher, but not above our pre-set threshold of 0.05. The margin is +0.081
173-
[+0.046; +0.127] on 120 pages from 55 documents, from the development (tuning)
174-
split [A39, 2026-10-14]. Docling had more fully perfect ParseBench tables in an
175-
earlier round [A37, 2026-10-08]. All three come from the development (tuning)
176-
split, which we used while building the table work. Our side is an upper bound,
177-
not a held-out result. The table-structure model we ship is Microsoft's Table
178-
Transformer v1.1-all, trained on PubTables-1M and FinTabNet.c. We have not
179-
checked whether our sample pages fall inside that training data, so the
180-
FinTabNet and PubTables-1M results may favour us for that reason too. Table
181-
structure only. Numbers, sample sizes and intervals: [1.1.0 release
182-
notes](https://github.com/pdfluent/pdfluent-sdk/releases/tag/v1.1.0),
183-
<EVIDENCE_BUNDLE_URL>.
169+
On the held-out test set (claim A41, measured 2026-10-17), `balanced` with the
170+
table model has the highest average of the three tools across 19 extraction
171+
measures: 0.721, against 0.693 for pymupdf4llm 1.28.2 and 0.672 for Docling
172+
2.126.0 (claim A41). It scores higher than Docling on 16 of the 19 measures and
173+
higher than pymupdf4llm on 13 (claim A41). On table structure (TEDS) it is ahead
174+
of both on FinTabNet and PubTables-1M, with every 95% confidence interval above
175+
zero (claim A42, measured 2026-10-17). The held-out half was not used while
176+
building the table work. The table-structure model we ship is Microsoft's Table
177+
Transformer v1.1-all, trained on PubTables-1M and FinTabNet.c; we have not
178+
checked whether our sample pages fall inside that training data. Per-measure
179+
numbers, sample sizes and intervals: <https://pdfluent.com/benchmarks/>.
184180

185181
The Python wheel, the Node addon and the CLI archives ship with the model
186182
features (`ort-layout`, `ort-tables`) already built in. The `pdfluent` command

0 commit comments

Comments
 (0)