Multi-Source Knowledge Graph (GraphRAG POC)

A proof-of-concept that builds a unified knowledge graph from three heterogeneous data sources — PostgreSQL, MongoDB, and free text — then answers natural-language questions over it using hybrid GraphRAG (vector search + graph traversal).

The sample domain is telecom / hybrid network topology (network nodes, VMs, alerts, incidents), but the extraction is schema-agnostic and works on any data.

---

How it works

The system has two phases, orchestrated from main.py.

1. Ingestion pipeline (LangGraph)

                ┌─> SQL extractor ──┐
   START ──────>├─> Mongo extractor ─┤──> resolver ──> neo4j_loader ──> END
                └─> Text extractor ─┘

• Extractors dump each source and send it to a local LLM (qwen2.5-coder) that returns structured nodes/edges as JSON. The three branches run in parallel and fan in to the resolver.
• Resolver deduplicates and merges entities by ID across sources.
• Neo4j loader wipes the graph, writes the merged nodes/edges, and creates a 768-dim cosine vector index for retrieval.

2. Hybrid GraphRAG QA loop

1. Embed the question (nomic-embed-text) and vector-search for "pivot" nodes.
2. Traverse 1–2 hops out from each pivot to gather connected topology.
3. Assemble a structured context block and synthesize a network-analyst answer with llama3.1:8b.

---

Project layout

File	Purpose
main.py	Entry point; --ingest flag controls graph rebuild
graph.py	LangGraph workflow wiring (parallel extract → resolve → load)
nodes.py	Pipeline steps: extractors, resolver, Neo4j loader
extractor.py	Generic LLM extraction of nodes/edges from raw data
schemas.py	Pydantic extraction models + LangGraph state
prompts.py	Extraction and QA prompt templates
qa.py	Hybrid retrieval + answer synthesis loop
embedder.py	Standalone backfill of embeddings for nodes missing them
config.py	Loads settings from environment / .env
test.py	Neo4j connectivity smoke test
incident_report.txt	Sample text source

---

Prerequisites

• Python 3.10+
• Ollama running locally, with the models pulled:

  ollama pull qwen2.5-coder
  ollama pull nomic-embed-text
  ollama pull llama3.1:8b

• Neo4j (5.x, with vector index support)
• PostgreSQL and MongoDB (optional — extractors skip a source gracefully if it's down)

---

Setup

# 1. Install dependencies
pip install -r requirements.txt

# 2. Configure connections
cp .env.example .env
# edit .env with your DB credentials and Neo4j / Ollama settings

The .env file is git-ignored and holds all secrets/connection settings. config.py falls back to sensible defaults if a value is missing.

---

Usage

# Build/rebuild the graph from all sources, then start the QA loop
python main.py --ingest

# Start the QA loop only (query the existing graph — does NOT wipe it)
python main.py

# Build the graph only, no interactive session
python main.py --ingest --no-qa

# Backfill embeddings for any nodes missing them
python embedder.py

# Verify Neo4j connectivity
python test.py

Note: --ingest wipes and rebuilds the Neo4j graph. Run plain python main.py for day-to-day querying.

Example session

[Question]: What was the impact of the CPU spike on VM-01?

FINAL ANALYST REPORT:
The CPU spike (94.2%) on VM-01 induced processing latency on the edge router
NODE-RTR-099, which dropped control packets. The downstream 5G Macro Cell
(NODE-5G-001) and Public Wi-Fi AP (NODE-WIFI-102) consequently lost
synchronization, resulting in ~12.5% packet loss in the downtown sector...

Type exit or quit to leave the session.

---

Configuration reference

All settings come from environment variables (see .env.example):

Variable	Default	Description
LLM_MODEL	qwen2.5-coder:latest	Extraction / JSON mapping model
EMBEDDING_MODEL	nomic-embed-text	Vector embedding model (768-dim)
QA_LLM_MODEL	llama3.1:8b	Answer synthesis model
PG_HOST / PG_USER / PG_PASS / PG_DB	localhost / postgres / — / knowledge_graph_poc	PostgreSQL source
MONGO_URI / MONGO_DB	localhost:27017 / kg_poc	MongoDB source
TEXT_FILE_PATH	incident_report.txt	Free-text source
NEO4J_URI / NEO4J_USER / NEO4J_PASS	bolt://localhost:7687 / — / —	Neo4j target

---

Status

This is a proof of concept. Known areas for future work: fuzzy/embedding-based entity resolution across sources, ontology-constrained extraction for consistent labels, and chunking large source dumps to stay within the LLM context window.