A production-ready FastAPI microservice that intercepts LLM queries, vectorizes them using a lightweight local embedding model (BAAI/bge-small-en-v1.5 via FastEmbed), and performs Vector Similarity Search (VSS) in Redis Stack.
If a semantically equivalent query exists in cache (X-Cache: HIT). Otherwise, it executes the LLM request, stores the vector embedding and response into Redis Stack (X-Cache: MISS), and returns the output.
- Framework: Python 3.11+, FastAPI, Pydantic v2
- Package Manager:
uv - Vector Database: Redis Stack (RediSearch VSS with HNSW index)
- Embedding Model:
fastembed(BAAI/bge-small-en-v1.5, 384-dim CPU-optimized embeddings) - Containerization: Docker & Docker Compose
semantic-cache-layer/
├── app/
│ ├── api/
│ │ ├── __init__.py
│ │ └── endpoints.py
│ ├── core/
│ │ ├── config.py
│ │ └── exceptions.py
│ ├── services/
│ │ ├── embedding.py
│ │ └── cache.py
│ ├── main.py
├── scripts/
│ └── demo_benchmark.py
├── docker-compose.yml
├── Dockerfile
├── pyproject.toml
└── README.md
Launch the Redis Stack server and FastAPI microservice:
docker-compose up --buildThe service will start at http://localhost:8000. OpenAPI docs are available at http://localhost:8000/docs.
Run the included terminal benchmark script to verify semantic caching and measure latency:
uv sync
uv run python scripts/demo_benchmark.pypip install httpx rich
python scripts/demo_benchmark.pyThe benchmark sends 5 queries. Query 1 and Query 2 are semantically identical but phrased differently:
- Query 1: "How to configure CORS in FastAPI?"
$\rightarrow$ X-Cache: MISS(~180ms) - Query 2: "FastAPI CORS setup tutorial"
$\rightarrow$ X-Cache: HIT(< 5ms, > 0.85 similarity match)
{
"prompt": "How to configure CORS in FastAPI?"
}X-Cache:HITorMISSX-Latency-Ms: Processing duration in milliseconds
{
"prompt": "How to configure CORS in FastAPI?",
"response": "Synthesized answer for prompt: 'How to configure CORS in FastAPI?'. [Generated via Mock LLM]",
"cached": false,
"similarity": 1.0,
"latency_ms": 184.12
}Health check endpoint verifying Redis Stack connection.