Skip to content

About

High-performance semantic caching microservice for LLM queries powered by FastAPI, FastEmbed (bge-small-en-v1.5), and Redis Vector Search (VSS). Sub-10ms cache hit response times.

Topics

Resources

Stars

2 stars

Watchers

1 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

Repository files navigation

Semantic Cache Layer

A production-ready FastAPI microservice that intercepts LLM queries, vectorizes them using a lightweight local embedding model (BAAI/bge-small-en-v1.5 via FastEmbed), and performs Vector Similarity Search (VSS) in Redis Stack.

If a semantically equivalent query exists in cache ($> 0.85$ similarity threshold), it returns the response instantaneously (X-Cache: HIT). Otherwise, it executes the LLM request, stores the vector embedding and response into Redis Stack (X-Cache: MISS), and returns the output.


Tech Stack

  • Framework: Python 3.11+, FastAPI, Pydantic v2
  • Package Manager: uv
  • Vector Database: Redis Stack (RediSearch VSS with HNSW index)
  • Embedding Model: fastembed (BAAI/bge-small-en-v1.5, 384-dim CPU-optimized embeddings)
  • Containerization: Docker & Docker Compose

Directory Structure

semantic-cache-layer/
├── app/
│   ├── api/
│   │   ├── __init__.py
│   │   └── endpoints.py
│   ├── core/
│   │   ├── config.py
│   │   └── exceptions.py
│   ├── services/
│   │   ├── embedding.py
│   │   └── cache.py
│   ├── main.py
├── scripts/
│   └── demo_benchmark.py
├── docker-compose.yml
├── Dockerfile
├── pyproject.toml
└── README.md

Quickstart

1. Start Services with Docker Compose

Launch the Redis Stack server and FastAPI microservice:

docker-compose up --build

The service will start at http://localhost:8000. OpenAPI docs are available at http://localhost:8000/docs.


Running the Benchmark

Run the included terminal benchmark script to verify semantic caching and measure latency:

Option A: Using uv

uv sync
uv run python scripts/demo_benchmark.py

Option B: Standard Python

pip install httpx rich
python scripts/demo_benchmark.py

Expected Output

The benchmark sends 5 queries. Query 1 and Query 2 are semantically identical but phrased differently:

  • Query 1: "How to configure CORS in FastAPI?" $\rightarrow$ X-Cache: MISS (~180ms)
  • Query 2: "FastAPI CORS setup tutorial" $\rightarrow$ X-Cache: HIT (< 5ms, > 0.85 similarity match)

API Reference

POST /api/v1/generate

Request Body

{
  "prompt": "How to configure CORS in FastAPI?"
}

Response Headers

  • X-Cache: HIT or MISS
  • X-Latency-Ms: Processing duration in milliseconds

Response Body

{
  "prompt": "How to configure CORS in FastAPI?",
  "response": "Synthesized answer for prompt: 'How to configure CORS in FastAPI?'. [Generated via Mock LLM]",
  "cached": false,
  "similarity": 1.0,
  "latency_ms": 184.12
}

GET /health

Health check endpoint verifying Redis Stack connection.

About

High-performance semantic caching microservice for LLM queries powered by FastAPI, FastEmbed (bge-small-en-v1.5), and Redis Vector Search (VSS). Sub-10ms cache hit response times.

Topics

Resources

Stars

2 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages