Tools, libraries, papers, and patterns for reducing the cost of running large language models in production.
-
Updated
Oct 4, 2026
Tools, libraries, papers, and patterns for reducing the cost of running large language models in production.
High-performance LLM query cache with semantic search. Reduce API costs 80% and latency from 8.5s to 1ms using Redis + Qdrant vector DB. Multi-provider support (OpenAI, Anthropic).
Self-hosted LLM control layer for cost optimization, policy enforcement, observability, and governance. OpenAI-compatible, with observe/enforce modes and optional security, privacy, and compliance modules.
Biological nervous systems don't recompute known workflows from scratch. Mnemon gives LLM agents the same primitive — execution memory that caches plans, not responses. 93% token reduction, 2.66ms vs 20s, zero tokens on repeat runs. LangChain, CrewAI, AutoGen.
Verification layer for semantic caches: checks whether a cached answer is right for the question, not just how similar it is, on top of the cache you already run. Randomized audits give a conservative upper bound on the live error rate; CRC and online ACI thresholds; drift detection with auto-recalibration; entries to refresh; local scoring.
Research behind CacheVerifier (cacheverifier.com): verifier gating for semantic cache hits vs. static and adaptive thresholds, on ~210k requests across three public benchmarks plus real Twitter support traffic (4 datasets). Fine-tuning vs. off-the-shelf, label noise, drift, conformal risk control under shift, online ACI from randomized audits.
High-performance semantic caching microservice for LLM queries powered by FastAPI, FastEmbed (bge-small-en-v1.5), and Redis Vector Search (VSS). Sub-10ms cache hit response times.
Cost-aware KV cache lease optimization + nonblocking long-task runtime for Hermes Agent (cache intelligence, provider-aware request replay)
Read once. Brief anywhere. Reusable document briefings from fingerprinted cache.
High-performance Reverse Proxy for LLM Caching and Deduplication (OpenAI, Anthropic, Gemini, Copilot). Saves API costs, encrypts cache at rest, and multicasts concurrent streams.
Self-hosted SemanticGuard deployment modules — bring the AI gateway with self-validating semantic caching into your own GCP / AWS / Azure account. https://www.semanticguard.dev
EDU AI LAB: Local AI is not CHEAP at ALL! - Every AI token carries a COST! How to measure them?
Production-ready Retrieval-Augmented Generation (RAG) system using LangChain and Google Gemini. Features hybrid search (Chroma + BM25), reciprocal rank fusion, idempotent document ingestion, persistent SQLite LLM caching, an interactive CLI shell, and a Streamlit UI.
System 0 for your LLM: learns the fixed-choice answers your LLM gives (intents, routing, moderation) and serves the repeats in microseconds from a small CPU table, at most 5% disagreement, certified. Zero-code proxy for the OpenAI Decisions API, System One and chat models.
Adaptive sliding window query debouncing and micro-batch coalescing engine optimizing LLM and tool call throughput
An autonomous, zero-trust data governance agent leveraging the Model Context Protocol (MCP) to secure high-volume data lakehouses. Intercepts semantic query intent, maps real-time AWS IAM controls, and enforces automated PII masking via dynamic Vector RAG structures.
Evaluates prompt similarity via n-gram shingling, token hashing, and min-hash signatures to serve deterministic cached agent responses and eliminate redundant LLM calls | GenPark: https://www.genpark.ai | LinkedIn: https://www.llinkedin.com/company/92955550
Pre-indexed code knowledge graph for Claude Code, Codex, Gemini, Cursor, OpenCode, AntiGravity, Kiro, and Hermes Agent — fewer tokens, fewer tool calls, 100% local
FreCoS: freshness- and cost-aware extension to GPTCache (BSc course project)
To associate your repository with the llm-caching topic, visit your repo's landing page and select "manage topics."