Local Llama Inference Service

A production-oriented Flask API for running Llama-family instruct models on your own hardware. The service supports llama.cpp GGUF models by default, an optional Hugging Face Transformers backend, lazy model loading, API-key protection, structured errors, health checks, Docker, and model-free API tests.

This repository is a cleaned, reusable version of multilingual LLM service patterns originally developed for AI-enabled media workflows. Model weights and proprietary application data are intentionally not included.

What it demonstrates

• Local LLM inference with llama-cpp-python or Hugging Face Transformers
• OpenAI-style POST /v1/chat/completions requests
• Lazy, thread-safe model loading instead of loading weights at import time
• Environment-driven model, context-window, device and generation settings
• Optional bearer-token authentication
• Backward-compatible endpoints for older integrations
• Dependency-injected tests that run without downloading a model
• Gunicorn, Docker and GitHub Actions delivery paths

Architecture

client
  -> Flask API / validation / optional auth
  -> backend interface
       -> llama.cpp + local GGUF (default)
       -> Hugging Face Transformers (optional)
  -> structured chat-completion response

Quick start: GGUF with llama.cpp

Requirements: Python 3.10+ and a chat/instruct GGUF model you are licensed to use.

python -m venv .venv
source .venv/bin/activate  # Windows: .venv\Scripts\activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

cp .env.example .env
export LLAMA_MODEL_PATH=/models/your-model.gguf
gunicorn -c gunicorn.conf.py "app:app"

PowerShell:

$env:LLAMA_MODEL_PATH = "C:\models\your-model.gguf"
python app.py

The first generation request loads the model. /health stays lightweight and reports whether the backend has loaded.

API

Chat completions

curl http://localhost:5002/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer change-me" \
  -d '{
    "messages": [
      {"role": "system", "content": "Answer concisely."},
      {"role": "user", "content": "Explain retrieval-augmented generation."}
    ],
    "max_tokens": 256,
    "temperature": 0.2,
    "top_p": 0.9
  }'

Simple generation

curl http://localhost:5002/generate \
  -H "Content-Type: application/json" \
  -d '{"prompt":"日本語で自己紹介してください。"}'

Health

curl http://localhost:5002/health

Legacy POST /get_response requests with a question field remain supported.

Configuration

Variable	Default	Purpose
LLAMA_BACKEND	llama_cpp	llama_cpp or transformers
LLAMA_MODEL_PATH	none	Path to a GGUF model; required for llama.cpp
LLAMA_CHAT_FORMAT	unset	Optional llama.cpp chat template name
LLAMA_N_CTX	4096	Context window
LLAMA_N_GPU_LAYERS	0	Layers offloaded to GPU
HF_MODEL_ID	none	Hugging Face model ID or local directory
LLAMA_API_KEY	unset	Enables bearer-token authentication when set
MAX_PROMPT_CHARS	20000	Per-message content limit
PORT	5002	Development server port

For the Transformers backend:

python -m pip install -r requirements-transformers.txt
export LLAMA_BACKEND=transformers
export HF_MODEL_ID=meta-llama/Meta-Llama-3-8B-Instruct

Access to gated models must be obtained from the model publisher. Never commit access tokens or weights.

Tests and quality checks

python -m pip install -r requirements-dev.txt
ruff check .
pytest -q

Tests inject a fake inference backend, so CI validates request handling, authentication and response contracts without a GPU or model download.

Docker

docker build -t local-llama-service .
docker run --rm -p 5002:5002 \
  -e LLAMA_MODEL_PATH=/models/model.gguf \
  -v /absolute/path/to/models:/models:ro \
  local-llama-service

Production notes

• Configure LLAMA_API_KEY before exposing the service beyond a trusted network.
• Put TLS, request-rate limits and request-size limits at a reverse proxy or API gateway.
• Use one worker per loaded model unless memory capacity has been measured.
• Record latency, token usage, model version and quality evaluations in the consuming application.
• Treat prompts and generated text as potentially sensitive data; logging is intentionally conservative.

Repository scope

This is an inference-service reference implementation, not a hosted model and not a claim that every model will fit every machine. Exact latency and memory usage depend on model size, quantisation, context length and hardware.

License and attribution

Application code is MIT licensed. Model weights retain their original licences and acceptable-use terms. Llama is a trademark of Meta Platforms, Inc.; this independent project is not affiliated with or endorsed by Meta.