Tenant-fair LLM inference orchestration on a single GPU. No Kubernetes.
-
Updated
Aug 8, 2026 - Python
#
gpu-serving
Here are 2 public repositories matching this topic...
Real-time AI memory orchestrator for multi-model GPU serving. PERC: provably optimal KV cache eviction via fractional knapsack. 25-79% eviction cost reduction vs LRU.
-
Updated
Apr 22, 2026 - Python
Improve this page
Add a description, image, and links to the gpu-serving topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the gpu-serving topic, visit your repo's landing page and select "manage topics."