A hybrid deterministic + Vision-LLM OCR pipeline for confidential documents.
It pairs a deterministic engine (PaddleOCR / PP-DocLayoutV2, or Apple Vision) with a self-hosted VLM so you get the best of both: trustworthy bounding boxes from the deterministic side and clean, structure-aware reading (tables → markdown, translation) from the VLM — fused into one "best of" output with a searchable bbox text overlay.
Everything runs on your own estate — local models via Ollama/MLX now, an in-VPC CUDA endpoint (e.g. vLLM on the transcription GPU) later — with no third-party LLM calls. The most-sensitive tier runs fully air-gapped.
| Concern | Owner |
|---|---|
| Geometry (boxes, tables, reading order) | deterministic — PaddleOCR / Apple Vision; layout via PP-DocLayoutV2 |
| Semantics (reading, structure, translation) | VLM (orchestrator can choose by task e.g. Qwen 3.6 MoE for latin text; Typhoon for Thai) |
| Fusion | align VLM text onto deterministic boxes; the deterministic layer serves as a anti-hallucination gate |
Boxes never come from the VLM. That's what makes the searchable overlay reliable and keeps the VLM from inventing text where there was no ink.
The real stages are in: triage → layout → language → OCR (PaddleOCR / Apple Vision) → VLM read → tables → fusion → render. A dropped PDF emits a provenance-carrying segment index, per-language markdown, and a searchable bbox overlay. On a born-digital recognition eval the OCR path scores ~95% word recall / ~96% precision; the headline case — a handwritten note tesseract recovers 6 characters from — comes through at ~3,185 searchable characters, fully local.
The contract underneath — Document in / Document out, raw and inferred both
retained, serialised between stages for a recipe-fingerprinted resume — is in place and
tested (214 tests). The pre-registered evidence campaign is complete (streams A–G,
manifests in eval_out/manifests/): handwriting recall generalises to 0.955 at n=100
(IAM), fused box placement ≥ deterministic under the band metric, the blank-page
hallucination gate holds (0 gated invented words), and the anti-misalignment gate's value
is measured (+1.6–2.6pt placement for −0.6–0.9pt recall). Word-level overlay subdivision
and a few table refinements are follow-ups (see the roadmap).
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]" # bare plumbing — no model deps
cp config.example.toml config.toml
python -m fusion_ocr.watcher --once # process anything in in/, then exit
# drop a PDF into in/ ; artifacts land in out/<sha256>/
pytest # ~290 tests in ~20 s; engine tests skip where an
# engine is absent. CI runs the same on every push/PR.For a reproducible Linux development environment, open the repository in VS Code
and choose Dev Containers: Reopen in Container. The container uses Python 3.11
on Linux x86_64 and installs the API, PaddleOCR, VLM-client, and test dependencies
against constraints.txt. On Apple Silicon, Docker may emulate x86_64, so PaddleOCR
will be CPU-only and can be slower than a native run.
The first container setup creates config.toml from
.devcontainer/config.toml if one does not already exist. That starter config
points the VLM client at the host's MLX server (host.docker.internal:8080) and
disables the airgap guard so the container can reach it. For Ollama, change the
URL to http://host.docker.internal:11434/v1. If you already have a config.toml,
it is left untouched; update its VLM URL and airgap setting for container use
if needed.
Apple Vision and MLX stay on macOS; they are not installed in the Linux container.
Keep .env on the host/workspace (it is gitignored and is not copied into the
image); the API loads its token from that file. For API access from VS Code,
port 8000 is forwarded automatically.
Each job's results land in a content-addressed folder out/<sha256>/ — the three
deliverables (document.md reading view, overlay.pdf searchable layer,
segment_index.json provenance) plus the resumable per-stage state. What every file is:
Docs/outputs.md.
Add capability behind extras as stages are implemented:
pip install -e ".[ocr,vlm,api]"
# For a byte-for-byte reproducible environment (same OCR output for a given PDF),
# install against the pinned lock — versions are bounded in pyproject, exact in the lock:
pip install -e ".[ocr,vlm,api]" -c constraints.txt| Extra | Brings | For |
|---|---|---|
ocr |
PyMuPDF, PaddleOCR, PaddlePaddle | triage + geometry + structure |
vlm |
httpx, Pillow | the OpenAI-compatible VLM client |
api |
FastAPI, uvicorn, python-multipart | the HTTP job API |
Readers are OpenAI-compatible endpoints, so each tool runs wherever it's fastest and the router points per-script at the right one. Default setup on Apple Silicon:
# generalist reader — Qwen3.6-35B-A3B (MoE) via MLX (~10-40x faster than Ollama on Apple Silicon)
pip install mlx-vlm
python -m mlx_vlm.server --port 8080 # serves mlx-community/Qwen3.6-35B-A3B-4bit
# Thai specialist — Typhoon on Ollama
ollama pull scb10x/typhoon-ocr1.5-3bThe router sends Latin/handwriting/etc. → Qwen3.6/MLX (:8080), Thai → Typhoon
(:11434). One config value moves any reader — Ollama, MLX, or in-VPC vLLM all speak
the same API, so swapping local → transcription-GPU is just a base_url change. See
Docs/routing.md.
Configure via config.toml (copy from config.example.toml), or change the
output-affecting tuning knobs on a running service over HTTP. The job API (the api
extra; run it with fusion-ocr-serve) is the stable contract callers use:
| Method & path | Purpose |
|---|---|
POST /jobs (multipart pdf) |
enqueue a PDF or image → 202 {sha256, status, original_name} |
GET /jobs [?status=] |
queue / completed-job feed |
GET /jobs/{sha256} |
job status + artifact list (poll until done; see Docs/outputs.md) |
GET /jobs/{sha256}/artifacts/{name} |
fetch an artifact's bytes (document.md, overlay.pdf, …) |
GET /config |
surface every setting (secrets masked) + its constraints |
PATCH /config {path: value} |
configure the allowlisted settings at runtime (reaches the worker too) |
POST /config/save |
persist the current config to disk (explicit, opt-in) |
Submit is asynchronous: POST /jobs enqueues and returns immediately. Run the API
(fusion-ocr-serve) alongside a worker (fusion-ocr, the watcher) that drains the queue;
clients poll GET /jobs/{sha256}. JobStore is the queue boundary and artifacts are
content-addressed (storage.py), so a distributed queue / object store can drop in later.
The API is bearer-token auth'd and fails closed: set FUSION_OCR_API_TOKEN (it refuses
to start without one) and send Authorization: Bearer …. Security/identity fields
(airgap, in_dir, out_dir, routes) are surfaced but read-only — the API can't
unseal the airgap tier or repoint paths. A tuning change re-keys the resume cache, so the
next job reprocesses rather than reusing a stale result. To reach it from other machines on
your LAN, set api_host = "0.0.0.0" and serve — see
Docs/configuration.md for the full settings table and endpoint
details, and Docs/deployment.md for a TLS deployment behind nginx
(sample configs in deploy/).
src/fusion_ocr/
models.py the Document/Page/Region/Segment record
pipeline.py Stage protocol + orchestration + resume
config.py config + airgap guard
settings.py settings registry — what's surfaceable vs runtime-configurable
raster.py page-raster cache — render a page once, share across stages
jobs.py SQLite job queue (enqueue / atomic claim / status)
storage.py content-addressed artifact location (object-store swap point)
watcher.py drop-folder entrypoint + the queue worker
api.py HTTP job + config API (stable contract for callers)
stages/ triage · layout · language · ocr_det · vlm_read · fusion · render
vlm/ client protocol + OpenAI-compatible impl + prompts
overlay/ PyMuPDF invisible-text overlay (line- → word-level)
PaddleOCR downloads its models to ~/.paddlex on first use. For the air-gapped
sensitive tier, pre-pull the models once on a connected machine (run any OCR job),
then run sealed — enforce_airgap() sets PADDLE_PDX_DISABLE_MODEL_SOURCE_CHECK so
Paddle won't attempt its connectivity check.
The pipeline triages each region and routes it on two independent axes — geometry
(always a deterministic engine — PaddleOCR, or Apple Vision when preferred — never the
VLM) and reading (VLM, specialist varies). See Docs/routing.md. Thai is the first specialist
route: paddle_lang=th reads Thai forms that the generalist VLMs can't, and a
config [routing.thai] section points the reader at a served Typhoon endpoint when
one's available.
Accuracy is a number, not a claim. A born-digital page carries its own exact text in the layer, so the eval renders it to an image (dropping the text layer), forces the OCR path, and scores the recovered text against the embedded text — no hand-labelling:
python -m fusion_ocr.eval report.pdf --pages 6,9,24 --apple-visionIt reports word recall / precision (order-insensitive: recognition completeness and
the hallucination rate) alongside CER / WER (sequence-based, so reading order on
multi-column pages inflates them — trust CER on single-column text). Caveat: rendered-clean
pages are a floor on difficulty; genuinely degraded scans / handwriting need a small
hand-labelled set. See src/fusion_ocr/eval/.
Why this tool exists and how its architecture serves that is narrated in Docs/dev_notes/motivation_and_strategy.md.
Build-vs-adopt, determinism vs learned models, and auditing model defaults are codified in Docs/principles.md — the short version: orchestrate trusted tools, build only the connective tissue and the journalism-specific guarantees no component provides, and prefer a library's public surface over its internals.
What's shipped is logged in Docs/dev_notes/done.md; what's next — and the scale-triggered items (distributed queue, object store) — is in Docs/dev_notes/roadmap.md.