LLM inference, optimized for your Mac
Continuous batching and tiered KV caching, managed directly from your menu bar.
Install · Quickstart · Features · Models · CLI Configuration · GitHub
Every LLM server I tried made me choose between convenience and control. I wanted to pin everyday models in memory, auto-swap heavier ones on demand, set context limits - and manage it all from a menu bar.
oMLX persists KV cache across a hot in-memory tier and cold SSD tier - even when context changes mid-conversation, all past context stays cached and reusable across requests, making local LLMs practical for real coding work with tools like Claude Code. That's why I built it.
Download the .dmg from Releases, drag to Applications, done. The app includes in-app auto-update, so future upgrades are just one click.
brew tap jundot/omlx https://github.com/jundot/omlx
brew install omlx
# Upgrade to the latest version
brew update && brew upgrade omlx
# Run as a background service (auto-restarts on crash)
brew services start omlxgit clone https://github.com/jundot/omlx.git
cd omlx
pip install -e .Requires Python 3.10+ and Apple Silicon (M1/M2/M3/M4).
Launch oMLX from your Applications folder. The Welcome screen guides you through three steps - model directory, server start, and first model download. That's it.
omlx serve --model-dir ~/modelsThe server discovers models from subdirectories automatically. Any OpenAI-compatible client can connect to http://localhost:8000/v1. A built-in chat UI is also available at http://localhost:8000/admin/chat.
If you installed via Homebrew, you can run oMLX as a managed background service:
brew services start omlx # Start (auto-restarts on crash)
brew services stop omlx # Stop
brew services restart omlx # Restart
brew services info omlx # Check statusThe service runs omlx serve with zero-config defaults (~/.omlx/models, port 8000). To customize, either set environment variables (OMLX_MODEL_DIR, OMLX_PORT, etc.) or run omlx serve --model-dir /your/path once to persist settings to ~/.omlx/settings.json.
Logs are written to two locations:
- Service log:
$(brew --prefix)/var/log/omlx.log(stdout/stderr) - Server log:
~/.omlx/logs/server.log(structured application log)
oMLX is built on top of vllm-mlx, extending it with tiered KV caching, multi-model serving, an admin dashboard, Claude Code optimization, and Anthropic API support. Currently supports text-based LLMs - VLM and OCR model support is planned for upcoming milestones.
Web UI at /admin for real-time monitoring, model management, chat, benchmark, and per-model settings. All CDN dependencies are vendored for fully offline operation.
Block-based KV cache management inspired by vLLM, with prefix sharing and Copy-on-Write. The cache operates across two tiers:
- Hot tier (RAM): Frequently accessed blocks stay in memory for fast access.
- Cold tier (SSD): When the hot cache fills up, blocks are offloaded to SSD in safetensors format. On the next request with a matching prefix, they're restored from disk instead of recomputed from scratch - even after a server restart.
Handles concurrent requests through mlx-lm's BatchGenerator. Prefill and completion batch sizes are configurable.
Context scaling support for running smaller context models with Claude Code. Scales reported token counts so that auto-compact triggers at the right timing, and SSE keep-alive prevents read timeouts during long prefill.
Load LLMs, embedding models, and rerankers within the same server. Models are managed through a combination of automatic and manual controls:
- LRU eviction: Least-recently-used models are evicted automatically when memory runs low.
- Manual load/unload: Interactive status badges in the admin panel let you load or unload models on demand.
- Model pinning: Pin frequently used models to keep them always loaded.
- Per-model TTL: Set an idle timeout per model to auto-unload after a period of inactivity.
- Process memory enforcement: Total memory limit (default: system RAM - 8GB) prevents system-wide OOM.
Configure sampling parameters, chat template kwargs, TTL, and more per model directly from the admin panel. Changes apply immediately without server restart.
Chat directly with any loaded model from the admin panel. Supports conversation history, model switching, dark mode, and reasoning model output.
Search and download MLX models from HuggingFace directly in the admin dashboard. Browse model cards, check file sizes, and download with one click.
One-click benchmarking from the admin panel. Measures prefill (PP) and text generation (TG) tokens per second, with partial prefix cache hit testing for realistic performance numbers.
Native PyObjC menubar app (not Electron). Start, stop, and monitor the server without opening a terminal. Includes real-time serving stats, auto-restart on crash, and in-app auto-update.
Drop-in replacement for OpenAI and Anthropic APIs. Supports streaming usage stats (stream_options.include_usage) and Anthropic adaptive thinking.
| Endpoint | Description |
|---|---|
POST /v1/chat/completions |
Chat completions (streaming) |
POST /v1/completions |
Text completions (streaming) |
POST /v1/messages |
Anthropic Messages API |
POST /v1/embeddings |
Text embeddings |
POST /v1/rerank |
Document reranking |
GET /v1/models |
List available models |
Supports all function calling formats available in mlx-lm, JSON schema validation, and MCP tool integration. Tool calling requires the model's chat template to support the tools parameter. The following model families are auto-detected via mlx-lm's built-in tool parsers:
| Model Family | Format |
|---|---|
| Llama, Qwen, DeepSeek, etc. | JSON <tool_call> |
| Qwen3 Coder | XML <function=...> |
| Gemma | <start_function_call> |
| GLM (4.7, 5) | <arg_key>/<arg_value> XML |
| MiniMax | Namespaced <minimax:tool_call> |
| Mistral | [TOOL_CALLS] |
| Kimi K2 | <|tool_calls_section_begin|> |
| Longcat | <longcat_tool_call> |
Models not listed above may still work if their chat template accepts tools and their output uses a recognized <tool_call> XML format. Streaming requests with tool calls buffer all content and emit results at completion.
Point --model-dir at a directory containing MLX-format model subdirectories. Two-level organization folders (e.g., mlx-community/model-name/) are also supported.
~/models/
├── Step-3.5-Flash-8bit/
├── Qwen3-Coder-Next-8bit/
├── gpt-oss-120b-MXFP4-Q8/
└── bge-m3/
Models are auto-detected by type. You can also download models directly from the admin dashboard.
| Type | Models |
|---|---|
| LLM | Any model supported by mlx-lm |
| Embedding | BERT, BGE-M3, ModernBERT |
| Reranker | ModernBERT, XLM-RoBERTa |
# Memory limit for loaded models
omlx serve --model-dir ~/models --max-model-memory 32GB
# Process-level memory limit (default: auto = RAM - 8GB)
omlx serve --model-dir ~/models --max-process-memory 80%
# Enable SSD cache for KV blocks
omlx serve --model-dir ~/models --paged-ssd-cache-dir ~/.omlx/cache
# Set in-memory hot cache size
omlx serve --model-dir ~/models --hot-cache-max-size 20%
# Adjust batch sizes
omlx serve --model-dir ~/models --prefill-batch-size 8 --completion-batch-size 32
# With MCP tools
omlx serve --model-dir ~/models --mcp-config mcp.json
# API key authentication
omlx serve --model-dir ~/models --api-key your-secret-keyAll settings can also be configured from the web admin panel at /admin. Settings are persisted to ~/.omlx/settings.json, and CLI flags take precedence.
Architecture
FastAPI Server (OpenAI / Anthropic API)
│
├── EnginePool (multi-model, LRU eviction, TTL, manual load/unload)
│ ├── BatchedEngine (LLMs, continuous batching)
│ ├── EmbeddingEngine
│ └── RerankerEngine
│
├── ProcessMemoryEnforcer (total memory limit, TTL checks)
│
├── Scheduler (FCFS, configurable batch sizes)
│ └── mlx-lm BatchGenerator
│
└── Cache Stack
├── PagedCacheManager (GPU, block-based, CoW, prefix sharing)
├── Hot Cache (in-memory tier, write-back)
└── PagedSSDCacheManager (SSD cold tier, safetensors format)
git clone https://github.com/jundot/omlx.git
cd omlx
pip install -e ".[dev]"
pytest -m "not slow"Requires Python 3.11+ and venvstacks (pip install venvstacks).
cd packaging
# Full build (venvstacks + app bundle + DMG)
python build.py
# Skip venvstacks (code changes only)
python build.py --skip-venv
# DMG only
python build.py --dmg-onlySee packaging/README.md for details on the app bundle structure and layer configuration.
Contributions are welcome! See Contributing Guide for details.
- Bug fixes and improvements
- Performance optimizations
- Documentation improvements
- MLX and mlx-lm by Apple
- vllm-mlx - oMLX originated as a fork of vllm-mlx v0.1.0, since re-architected with multi-model serving, paged SSD caching, an admin panel, and a standalone macOS menu bar app
- venvstacks - Portable Python environment layering for the macOS app bundle
- mlx-embeddings - Embedding model support for Apple Silicon