Mini OpenAI Platform

A self-evaluating LLM platform. It serves OpenAI-style APIs for chat, embeddings and RAG — and unlike most RAG stacks, it can tell you whether its answers are good, what they cost, and when quality regresses.

Why this isn't another RAG demo

	What it does	Why it matters
Semantic caching	Paraphrased queries are answered from cache without touching the LLM	Cache hits return in ~0.1s vs ~9s generated; savings are graphed in LLM-seconds
Smart model routing	A difficulty heuristic routes easy prompts to a small model, hard ones to a large one	Cost-aware inference with the verdict exposed in every response
CI quality gate	A golden dataset grades retrieval (recall@k, MRR) on every push	The build fails when answer quality regresses — regression testing for RAG

Everything is observable: 16 Grafana panels cover traffic, latency, tokens, cache economics, retrieval quality and per-model routing.

Architecture

        browser                          curl / any OpenAI-style client
           │                                          │
           ▼                                          │
 ┌──────────────────┐                                 │
 │  frontend :8080  │                                 │
 │ React SPA, nginx ├──── proxies /api/* ────┐        │
 └──────────────────┘                        ▼        ▼
                                   ┌──────────────────────────┐
                                   │     api-gateway :8000    │
                                   │ auth · rate limit · cache│
                                   │ CORS · upload caps       │
                                   └──────┬────────────┬──────┘
                                          │            │
                  ┌───────────────────────┘            │
                  ▼                                    ▼
       ┌────────────────────┐   grounded     ┌────────────────────┐
       │  rag-service :8003 │   prompt       │  llm-service :8001 │
       │ chunking           ├───────────────►│ difficulty-based   │
       │ retrieval          │                │ model router       │
       │ semantic cache     │                │ small ⇄ large tier │
       └────┬──────────┬────┘                └─────────┬──────────┘
            │          │                               │
            ▼          ▼                               ▼
 ┌──────────────────┐ ┌────────────────┐    ┌─────────────────────┐
 │ embedding-service│ │  qdrant :6333  │    │    Ollama :11434    │
 │ :8002            │ │ doc chunks +   │    │ on host (dev) or    │
 │ MiniLM-L6-v2     │ │ semantic cache │    │ in the stack (prod) │
 └──────────────────┘ └────────────────┘    └─────────────────────┘

 prometheus :9090 scrapes every service ───► grafana :3000 (16 panels)

Service	Responsibility
api-gateway :8000	Auth (multi-key, constant-time), rate limiting, response caching, CORS, upload caps, routing
llm-service :8001	Generation via Ollama with difficulty-based model routing and tier fallback
embedding-service :8002	all-MiniLM-L6-v2 sentence embeddings
rag-service :8003	Ingestion, chunking, retrieval, semantic cache, grounded generation
frontend :8080	React SPA; nginx proxies /api/* to the gateway (single origin, no CORS)
qdrant :6333	Vector store: document chunks + semantic cache collections
prometheus :9090 / grafana :3000	Metrics and dashboards

All ports are loopback-bound — on a public host, nothing is reachable until a reverse proxy fronts the gateway.

RAG request flow

Client ─► API Gateway ─► RAG Service
                           │ 1. embed query          (Embedding Service)
                           │ 2. semantic cache lookup ──► hit? return answer + sources, no LLM call
                           │ 3. retrieve top-k chunks (Qdrant)
                           │ 4. build grounded prompt
                           ▼
                         LLM Service ─► route by difficulty: small / large model (Ollama)
                           │
                           ▼
                         answer + sources + cache/routing metadata ─► Client

Quickstart

Requires Docker and Ollama on the host with the two routing tiers pulled:

ollama pull llama3.2:3b && ollama pull phi3:latest

docker compose -f infrastructure/compose/docker-compose.yml up -d --build

Open http://localhost:8080 (default API key: dev-secret-key) and:

1. Chat mode — ask "What is the capital of France?", then "Explain why HNSW indexes trade recall for speed". Compare the routing badges.
2. RAG mode — upload a document (+ doc), ask about it, expand the sources. Ask the same thing again and watch the ⚡ cached answer land instantly.

Grafana lives at http://localhost:3000 (admin/admin).

The Frontend

The UI surfaces the platform's internals instead of hiding them: every answer carries badges for the serving model, the routing verdict with its difficulty score, cache status, latency, and (in RAG mode) the retrieved chunks with similarity scores.

Semantic Cache

Exact-match caches are useless for natural language — "how does RAG work?" and "explain retrieval augmented generation" never share a key. This platform caches at the meaning level: answered queries are stored in a dedicated Qdrant collection keyed by their embedding, and a new query whose cosine similarity clears the threshold is served the cached answer — skipping retrieval and generation entirely.

Two safeguards keep cached answers honest: entries expire after a TTL, and any document upload invalidates the whole cache, because the corpus the answers were grounded in has changed.

Env (rag-service)	Default
SEMANTIC_CACHE_ENABLED	true
SEMANTIC_CACHE_SIMILARITY_THRESHOLD	0.95	min cosine similarity for a hit
SEMANTIC_CACHE_TTL_SECONDS	600	entry lifetime

Smart Model Routing

Running every prompt through the biggest model wastes compute. The llm-service scores each prompt's difficulty in [0, 1] with a transparent heuristic — length (RAG prompts carry retrieved context and score high), reasoning keywords (explain / compare / why / step by step), multi-part questions, code — and routes below-threshold prompts to the small tier. If the chosen tier fails, the request falls back to the other tier instead of erroring.

The verdict ships in every response, so routing is debuggable from the client:

"routing": {"decision": "routed_small", "difficulty": 0.0468, "fallback_used": false}

"model": "auto" (the default) enables routing; an explicit model name pins the request. The heuristic was chosen over an ML classifier deliberately: zero added latency, no training data needed, and every decision is explainable.

Env (llm-service)	Default
ROUTING_ENABLED	false (true in compose)
OLLAMA_SMALL_MODEL	llama3.2:3b	easy prompts
OLLAMA_LARGE_MODEL	phi3:latest	hard prompts
ROUTING_THRESHOLD	0.3	difficulty cutoff

RAG Evaluation & Quality Gate

Most RAG projects cannot answer the question "is it any good?". This one ships its own harness under evals/:

• golden.jsonl — questions with expected_phrases that must appear in a retrieved chunk, plus reference answers
• corpus/ — a controlled corpus of three deliberately distinct documents, so retrieval mistakes are detectable
• run_eval.py — uploads the corpus, runs every question, and grades hit rate (recall@k), MRR, and optionally faithfulness (an LLM judge scores answers 1–5 against the reference)

# what CI runs — no LLM required
python evals/run_eval.py --retrieval-only --min-hit-rate 0.8 --min-mrr 0.5

# full evaluation including LLM-judged answer quality
python evals/run_eval.py --min-hit-rate 0.8 --min-mrr 0.5 --min-faithfulness 3.5

The rag-quality-gate CI job boots qdrant + embedding-service + rag-service and fails the build if retrieval metrics drop below thresholds. It needs no GPU and no Ollama: the rag-service exposes an LLM-free POST /retrieve endpoint precisely so retrieval quality can be graded in CI. The eval report is uploaded as a build artifact on every run.

Observability

Prometheus scrapes every service; the provisioned Grafana dashboard tells the story top-to-bottom: gateway traffic → token economics → cache hit rate and LLM-seconds saved → retrieval quality → per-model routing.

Metrics worth knowing about:

Metric	Meaning
rag_semantic_cache_latency_saved_seconds_total	LLM generation time skipped by cache hits
rag_retrieval_top_score (histogram)	similarity of the best retrieved chunk per query
rag_insufficient_context_answers_total	answers that admitted the context wasn't enough
llm_generations_total{model, decision}	who served what, and why
llm_routing_fallbacks_total	failovers between model tiers

API

curl examples (gateway, port 8000)

# Chat — "auto" lets the router pick; an explicit model name pins it
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer dev-secret-key" \
  -d '{"model": "auto", "messages": [{"role": "user", "content": "Briefly explain vector databases."}], "max_tokens": 80}'

# Embeddings
curl -X POST http://localhost:8000/v1/embeddings \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer dev-secret-key" \
  -d '{"model": "sentence-transformers/all-MiniLM-L6-v2", "input": ["Embeddings power semantic search."]}'

# Document upload
curl -X POST http://localhost:8000/v1/documents/upload \
  -H "Authorization: Bearer dev-secret-key" \
  -F "file=@sample_rag.txt"

# RAG query — responses include cached status and sources with scores
curl -X POST http://localhost:8000/v1/rag/query \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer dev-secret-key" \
  -d '{"query": "How does retrieval augmented generation work?", "top_k": 3}'

Production Deployment

Target topology: one VPS, Docker Compose, Caddy for TLS. The base compose file binds every port to 127.0.0.1, so nothing is internet-reachable until you deliberately put a reverse proxy in front of the frontend — exposing a service is an explicit act, not a default.

1. Provision. Any VPS with ~8 GB RAM runs both CPU model tiers (llama3.2:3b ≈ 2 GB, phi3 ≈ 4 GB resident). Install Docker with the compose plugin.

2. Configure secrets — create infrastructure/compose/.env (compose picks it up automatically; it never enters the image):

API_GATEWAY_API_KEYS=key-for-frontend,key-for-cli   # one per client, constant-time compared
GRAFANA_ADMIN_PASSWORD=<strong password>
CORS_ALLOWED_ORIGINS=https://demo.example.com

3. Launch with the production overlay. It runs Ollama inside the stack (no host install) and a one-shot job pulls both model tiers into a persistent volume on first boot:

docker compose -f infrastructure/compose/docker-compose.yml \
               -f infrastructure/compose/docker-compose.prod.yml up -d --build

4. Expose with TLS. Point a DNS A record at the VPS and let Caddy terminate HTTPS on the host, forwarding to the loopback-bound frontend:

# /etc/caddy/Caddyfile
demo.example.com {
    reverse_proxy 127.0.0.1:8080
}

That single hostname serves both the UI and the API — the frontend's nginx already proxies /api/* to the gateway internally. To share dashboards, add a second site block for Grafana (reverse_proxy 127.0.0.1:3000); everything else stays private.

5. Operate. Updates are git pull + the same compose command; Qdrant data, Grafana state and pulled models live in named volumes and survive rebuilds. Uploads are capped at MAX_UPLOAD_BYTES (5 MB default) and rate limiting is per API key.

CPU inference on a VPS takes a few seconds per generated answer — which makes the semantic cache demo better: paraphrased questions visibly snap back in milliseconds.

Development

pip install -r requirements-dev.txt
pytest tests -q                        # 27 unit tests: chunking, prompts, cache, router, eval metrics
cd frontend && npm install && npm run dev   # hot-reload UI against localhost:8000

CI (.github/workflows/ci.yml) runs three jobs: unit tests, Docker builds for all services, and the retrieval quality gate.

Project Layout

services/
  api-gateway/        auth, rate limiting, caching, CORS, routing
  llm-service/        Ollama client + difficulty-based model router
  embedding-service/  sentence-transformers embeddings
  rag-service/        chunking, retrieval, semantic cache, generation
frontend/             React SPA + nginx API proxy
evals/                golden dataset, corpus, eval runner (CI quality gate)
infrastructure/       docker compose (base + prod overlay), prometheus config
monitoring/           provisioned Grafana dashboard (16 panels)
tests/                unit tests