Multi-Document RAG System

An enterprise multi-format RAG architecture capable of ingesting PDF, Markdown, DOCX, and TXT files, providing cross-document synthesis and document-level metadata filtering.

---

Key Features

• Heterogeneous: file format ingestion (PDF, Markdown, DOCX, TXT)
• Metadata-tagged: vector indexing with document type and author attributes
• Cross-document: comparative synthesis and multi-hop question answering
• Hybrid: lexical and dense vector search capabilities
• Full: REST API for document listing and grounded retrieval

---

Architecture

flowchart LR
    PDF[PDF Files] --> Ingest[Multi-Format Ingestion Pipeline]
    MD[Markdown] --> Ingest
    DOCX[Word Docs] --> Ingest
    Ingest --> MetaVector[(Metadata Vector Store)]
    Query([Cross-Doc Query]) --> Engine[Multi-Doc RAG Engine]
    MetaVector --> Engine
    Engine --> Synthesis[Synthesized Multi-Doc Answer]

Loading

---

Tech Stack

Component	Technology	Purpose
Runtime	Python 3.12	Core execution environment
API Framework	FastAPI & Uvicorn	High-performance asynchronous REST endpoints
Data Validation	Pydantic v2	Strict schema validation and serialization
Execution Engine	Dual-Mode (Local + Cloud)	Production-ready logic with offline verification
Testing	Unittest & Pytest	Deterministic automated verification suite

---

Project Structure

multi-document-rag-system/
├── app/
│   ├── __init__.py
│   ├── api.py           # FastAPI routes and server definitions
│   ├── config.py        # Environment variables and application settings
│   ├── models.py        # Pydantic data schemas
│   └── services/        # Core business automation logic
├── tests/
│   ├── __init__.py
│   └── test_multi_doc.py   # Automated test suite
├── .env.example         # Template for environment configuration
├── .gitignore           # Python and runtime exclusions
├── LICENSE              # MIT License
├── README.md            # Comprehensive project documentation
└── requirements.txt     # Python package dependencies

---

Getting Started

Prerequisites

• Python 3.10+ (Python 3.12 recommended)
• pip package manager

Installation

1. Clone the repository:

   git clone https://github.com/erhatechnologiesai/multi-document-rag-system.git
   cd multi-document-rag-system

2. Create and activate a virtual environment:

   python -m venv venv
   # On Windows:
   venv\Scripts\activate
   # On macOS/Linux:
   source venv/bin/activate

3. Install dependencies:

   pip install -r requirements.txt

4. Configure environment variables:

   cp .env.example .env

---

Running the Application

Start the local development server with auto-reload:

python -m uvicorn app.api:app --reload --host 0.0.0.0 --port 8000

Once running, interactive documentation is accessible at:

• Swagger UI: http://127.0.0.1:8000/docs
• ReDoc: http://127.0.0.1:8000/redoc

---

API Endpoints

Method	Endpoint	Description
GET	/documents	List all ingested documents and their metadata
POST	/query	Query across all or specific indexed documents

Example Request

curl -X POST http://127.0.0.1:8000/query -H "Content-Type: application/json" -d '{"query": "Compare the SLA agreements across all vendors"}'

---

Running Tests

Execute the automated test suite:

python -m unittest tests/test_multi_doc.py

Or using pytest:

pytest tests/

All test cases are self-contained and run offline without requiring third-party API credentials.

---

Security & Best Practices

• Zero Credential Leakage: API tokens and secrets are loaded exclusively via environment variables and excluded by .gitignore.
• Strict Validation: All incoming request payloads are strictly validated using Pydantic schemas.
• Fail-Safe Fallbacks: Deterministic offline engines guarantee application continuity even during external provider outages.

---

License

This project is licensed under the terms of the MIT License.