docs-rag-eval

Retrieval-augmented question answering over documentation, with every part of it measured.

Ask a question about the FastAPI documentation. The system retrieves the relevant passages, answers with numbered citations, and says "I don't know" when the documentation doesn't cover it. A labelled evaluation set measures how well each retriever finds the right page and how well the full pipeline answers, cites and abstains. CI fails if retrieval quality drops.

_{The web UI running fully offline (BM25 retrieval and the extractive answerer, no API key).}

---

Results

48 hand-written questions over 114 pages of FastAPI documentation (1,041 chunks): 42 answerable, each labelled with the page(s) that answer it, and 6 that the documentation cannot answer. Reproduce with docsrag eval.

Retrieval (answerable questions, top 5 documents)

Retriever	Hit@5	Recall@5	MRR	nDCG@5	Misses
BM25	0.952	0.917	0.730	0.763	2
TF-IDF (unigrams + bigrams)	0.929	0.893	0.660	0.708	3
Hybrid: BM25 + TF-IDF (reciprocal rank fusion)	0.929	0.893	0.712	0.745	3

Full pipeline (BM25 retrieval, top 5 chunks)

Answerer	Answer rate	Abstention accuracy	Citation precision	Invalid citations
Extractive baseline (offline)	0.738	0.667 (4 of 6)	0.578	0
LLM (Claude)	run docsrag eval --generators extractive,anthropic with an API key

• Answer rate: answerable questions that received an answer rather than "I don't know".
• Abstention accuracy: unanswerable questions that correctly received "I don't know".
• Citation precision: cited passages that come from a page labelled as relevant.

What the numbers say

• Fusing two lexical retrievers did not help. The hybrid scored below BM25 alone. BM25 and TF-IDF fail on the same questions, the ones where the user's words differ from the docs' words, so fusing them adds noise rather than new signal. Hybrid search pays off when the second retriever sees different evidence. That is why the next experiment is BM25 + dense embeddings (--retrievers bm25,embed,hybrid-embed, already implemented).
• Both BM25 misses are vocabulary mismatches. "How do I split the endpoints across several files?" should find Bigger Applications – Multiple Files, and "show an example request body" should find Declare Request Example Data. Neither page uses the user's key words. These are exactly the cases embeddings are designed to fix.
• The extractive baseline trades coverage for safety. It answers only when a retrieved passage covers at least half of the question's IDF-weighted terms. It never cites a passage that doesn't exist, but it still answered 2 of the 6 off-topic questions by latching onto a single shared word ("PostgreSQL" in a replication question, "last" in a revenue question). That is the gap an LLM answerer with an explicit abstention instruction should close, and the eval measures it.

---

How it works

Markdown docs ──► ingest: strip MkDocs syntax, split by heading, 300-word windows (50 overlap)
                     │                                         chunk id = path#n, keeps heading trail
                     ▼
question ──► retriever ── BM25 (implemented from scratch) │ TF-IDF │ embeddings │ hybrid (RRF)
                     │  top-k chunks
                     ▼
             answerer ── LLM: "answer only from the passages, cite [n], else say I don't know"
                     │   extractive: IDF-weighted sentence selection, abstains on low coverage
                     ▼
             answer + citations ──► API / web UI / CLI
                     │
evals/questions.jsonl ──► retrieval metrics (hit, recall, MRR, nDCG) + answer metrics ──► CI gate

Design choices worth noting:

• Relevance is labelled per page, not per chunk, so the labels stay valid when chunk size changes. Retrieved chunks are collapsed to pages in rank order before scoring.
• BM25 is written out in about 30 lines rather than imported, so the scoring (IDF, term-frequency saturation k1, length normalisation b) is visible and unit-tested.
• Reciprocal Rank Fusion combines retrievers by rank, 1 / (60 + rank), so scores on different scales never need calibrating.
• Citations are validated. A citation number that points to no retrieved passage is counted as invalid, and an answer containing "I don't know" never counts as cited.
• Quality is gated in CI. tests/test_quality_regression.py fails the build if BM25 Hit@5 drops below 0.95 or MRR below 0.70 on the real corpus, and the full evaluation is published to each CI run's summary.

---

Quick start

git clone https://github.com/SAIDUMPALA01/docs-rag-eval.git
cd docs-rag-eval
pip install -e ".[dev,anthropic]"

docsrag ask "How do I return a 404 with my own message?"   # offline: BM25 + extractive
docsrag eval                                                 # tables above; details in results/
docsrag serve                                                # UI at http://127.0.0.1:8000, docs at /docs

export ANTHROPIC_API_KEY=...                                 # optional: LLM answers
docsrag ask "How do I hash passwords?" --generator anthropic
docsrag eval --generators extractive,anthropic

pip install -e ".[embeddings]"                               # optional: dense retrieval
docsrag eval --retrievers bm25,embed,hybrid-embed

Docker:

docker build -t docs-rag-eval .
docker run -p 7860:7860 -e ANTHROPIC_API_KEY=... docs-rag-eval   # without a key it falls back to the extractive answerer

API

Method	Path
POST	/ask	{"question": "...", "k": 5} → answer, abstained, sources with URL, snippet and cited flag, latency, token usage
GET	/search?q=...&k=5	Retrieval only: ranked chunks with scores
GET	/health	Liveness

Settings: DOCSRAG_CORPUS (folder of Markdown), DOCSRAG_RETRIEVER (bm25, tfidf, hybrid, embed, hybrid-embed), DOCSRAG_GENERATOR (extractive or anthropic; defaults to anthropic when a key is set), RAG_MODEL.

Use your own docs: point --corpus at any folder of Markdown files and write your own questions.jsonl ({"id", "question", "relevant": ["path.md"], "answerable"}).

---

Project layout

src/docsrag/
  ingest.py       Markdown cleaning and heading-aware chunking
  retrievers.py   BM25, TF-IDF, embeddings, reciprocal rank fusion
  answer.py       LLM and extractive answerers, citation parsing, pipeline
  evaluation.py   retrieval and answer metrics, report tables
  api.py          FastAPI app and web UI
  cli.py          `docsrag` command
data/fastapi-docs FastAPI documentation subset (MIT), pinned commit in SOURCE.md
evals/            labelled questions
tests/            27 tests, including the quality gate on the real corpus

Limitations and honest notes

• 48 questions written and labelled by one person. That is enough to compare retrievers and catch regressions, but the confidence intervals are wide.
• There is no separate held-out set; the same questions were used while building the system. One label was corrected after error analysis: "How do I add a custom header to a response?" now also counts tutorial/middleware.md, which shows exactly that.
• The LLM answerer's numbers aren't published yet because they need an API key to produce. The harness is ready (--generators anthropic).

Roadmap

• Publish LLM-answerer and dense-retrieval results
• Judge answer faithfulness with an LLM judge validated against human labels, using my sixdim-eval library
• Re-ranking with a cross-encoder, measured against the same set

---

Corpus: FastAPI documentation © Sebastián Ramírez, MIT licence; see data/fastapi-docs/SOURCE.md. Code: MIT. Built by Sai Dumpala — GitHub · LinkedIn