graph-chunker

Structure-aware chunking for collections of PDFs. Preserve chapters, sections, source pages, tables, definitions and references in an inspectable document graph. No graph database, vector store or LLM is required for the default workflow.

Architecture

flowchart LR
    PDF[Files/*.pdf] --> Parser[PyMuPDF or optional Docling]
    Parser --> Tree[Shared document tree]
    Tree --> Normalize[Normalize headings, tables and boilerplate]
    Normalize --> Chunk[Shared token-bounded chunker]
    Chunk --> Graph[In-memory NetworkX graph]
    Graph --> JSON[graph.json + chunks.jsonl]
    Graph --> HTML[3D / 2D HTML explorer]

Loading

Parsers only extract structure. They do not select their own chunkers. Both feed the same chunking rules, tokenizer, node schema and graph builder. This gives consistent output contracts, not identical text: complex tables, OCR, reading order and heading detection can differ. parity measures those differences.

Install

Python 3.10+:

python -m pip install -e .

Optional capabilities:

python -m pip install -e '.[docling]'
python -m pip install -e '.[ollama]'
python -m pip install -e '.[hf]'
python -m pip install -e '.[ocr]'
python -m pip install -e '.[dev]'

The tokenizer may download its vocabulary on first use. Docling, Hugging Face and EasyOCR may also download model weights. After caching these resources, default chunking needs no model service. Tesseract fallback requires the Tesseract executable on PATH.

Quick Start

Put your PDFs in Files/. Subdirectories and .PDF extensions are supported. Private input files and default output files are excluded by the ignore rules.

graph-chunker run --input Files --out output --viz

Outputs:

• output/graph.json: portable nodes, links, configuration and per-file failures.
• output/chunks.jsonl: one record per chunk, table or figure fragment.
• output/graph.html: browser-based explorer, produced with --viz.

Open the generated HTML directly. No server is started. Graph data is embedded; the pinned JavaScript libraries and font load from CDNs, so the viewer needs an internet connection. It uses WebGL for 3D and canvas for 2D. Treat exports as private: they contain extracted document text.

graph-chunker viz output/graph.json
graph-chunker run --parser docling --max-tokens 512
graph-chunker run --parser auto
graph-chunker run --ocr

pymupdf is the default. It detects headings through bookmark-title matches, font styles and numbered-heading heuristics. auto explicitly opts into Docling with PyMuPDF fallback; fallback is logged and recorded in metadata. LCEL batches use bounded concurrency; direct PyMuPDF access is serialized because its native library is not thread-safe. Use Docling for complex layouts. The lightweight parser is best for simple reading-order PDFs, and cannot reliably reconstruct every multi-column layout.

Boundary Policies

Policy	Behavior
strict	Never merge distinct sections.
sibling (default)	Merge adjacent short nested leaf sections sharing a parent, within the token budget.
soft	Apply sibling rules and require embedding similarity above a threshold.

Top-level chapters never merge. Parent-child boundaries never merge. Empty heading stubs remain structural nodes, with their title inherited through the child's breadcrumb; they do not create empty chunks. Short chunks are allowed when merging would violate structure. min_tokens is a merge target, not a guaranteed floor. max_tokens is enforced on stored text, including semantic output. Oversized tables and figure captions become typed fragments.

Stored text contains source content without added context. embed_text adds the breadcrumb and a bounded previous-sibling tail. It can exceed max_tokens; check your embedding model's input limit before using it. This package exports embedding-ready text but does not create a vector index.

graph-chunker run --policy strict --min-tokens 80 --max-tokens 512
graph-chunker run --splitter semantic --embedding-provider ollama
graph-chunker run --splitter semantic --embedding-provider hf
graph-chunker run --policy soft --embedding-provider hf --soft-similarity-threshold 0.8

Ollama requires a running service and a pulled model:

ollama pull mxbai-embed-large

Defaults: Ollama mxbai-embed-large at http://localhost:11434; Hugging Face sentence-transformers/all-MiniLM-L6-v2. Override with --embedding-model and --ollama-url. Semantic splitting uses LangChain's SemanticChunker, followed by recursive splitting to enforce the hard maximum. LangGraph is not required.

Graph Contract

Schema version: 1.0. Node types: corpus, document, section, chunk, table, figure, term. Links have source, target, key, and type.

Edges: CONTAINS, NEXT, PREV, HAS_TABLE, HAS_FIGURE, REFERS_TO, DEFINES, MENTIONS. Only containment is acyclic; the full graph intentionally includes reading-order cycles. Merged chunks link to every section they cover.

Chunk records include id, doc_id, text, embed_text, pages, section_ids, and metadata containing token count, breadcrumb and order. Document identities use content hashes; identical PDFs are ingested once. Exported source paths are relative to the input folder, not machine paths. IDs are deterministic for the same input and configuration, but can change when parser output or chunk settings change.

References resolve within a document only. Ambiguous and missing targets remain in unresolved_references; no cross-document similarity is invented. Definition and glossary terms are extracted heuristically. These are structural relations, not an LLM-extracted knowledge graph. Figures contain caption text, not inferred visual descriptions. Page attribution is at extracted-block granularity.

The Python API retains the graph in memory:

from graph_chunker import ChunkerConfig
from graph_chunker.pipeline import run_corpus

result = run_corpus("Files", ChunkerConfig(boundary_policy="sibling"))
graph = result.graph
print(graph.number_of_nodes(), graph.number_of_edges())

Parser Parity

graph-chunker parity --input Files --tolerance 0.25

Requires the Docling extra and its models. Reports count differences, token percentiles, heading agreement (title, number, start page) and page coverage. The comparison always uses the shared recursive splitter and sibling policy. Tolerance controls count/token deltas and heading disagreement; page coverage must match. This measures consistency, not accuracy against ground truth.

Exit codes: 0 success; 1 failed PDFs or parity outside tolerance; 2 invalid input/configuration or unavailable parity dependencies. A batch exports successful files and reports failures; it does not silently call a partial run successful. No PDFs is an error.

Development

python -m pip install -e '.[dev]'
python -m pytest -q
python -m ruff check src tests

Tests generate synthetic PDFs in temporary directories. Default tests do not start Ollama, download document models or use proprietary documents. Optional Docling adapter tests can run against docling-core without model conversion. The CI matrix covers Python 3.10, 3.11 and 3.12.

Good evaluation sources include OpenStax, IETF RFCs, and eCFR. Check each document's reuse terms before sharing it.

Not included: retrieval, generated section summaries, vector databases, cross-document similarity or non-PDF loaders.

Licensing

The original code in this project is available under the MIT licence in LICENSE. Dependencies retain their own licences. PyMuPDF is AGPL-3.0/commercial; review its terms and your distribution/use obligations before publishing or embedding this package in another product. Model weights and source documents also have independent licensing terms. This project's licence does not override them.