OctoLlama

A web panel for managing Ollama, the LiteLLM aggregator, and Open WebUI across multiple hosts on your home network — no desktop app, with login, reachable from a browser (including your phone). The web-based counterpart to the Ollama Manager desktop app (PyQt6/KDE).

(Polska wersja tego pliku: README_PL.md)

Features

• Ollama service control — start/stop/autostart, install on demand, every environment variable that affects performance (context size, VRAM, Vulkan/iGPU, KV cache, network availability).
• Model management — list, size, what's currently loaded in memory, pulling new models with a progress bar, deleting.
• LiteLLM aggregator — a single OpenAI-compatible endpoint over models from every host at once; a deliberate choice of which models get exposed.
• Open WebUI — a chat panel wired to LiteLLM (not to a single host), so it sees exactly the models you chose to expose.
• Continue.dev config (VS Code) — generated from the currently exposed models, for manual pasting (the panel never overwrites the user's config).
• Multi-host — add remote hosts (e.g. a mini-PC running Ollama) with an auto-generated installer for the new machine.
• Host power management — Wake-on-LAN to wake a sleeping/off host, plus remote power off/restart/suspend.
• Login — single user, password hash in a local file, no database.
• Multi-language — Polish and English (switcher in the header), easy to extend with more languages.

How it works

Changing systemd service state (start/stop, environment variables) requires root. Instead of giving the web panel root, each host runs two separate processes:

┌─────────────────────────────┐         ┌──────────────────────────────┐
│  octollama-web              │         │  octollama-daemon            │
│  (user, NO root)            │         │  (root, systemd system unit)  │
│                              │         │                                │
│  - web panel + login        │         │  - inotify on the state file   │
│  - model operations         │  state  │  - diff: what changed           │
│    -> straight to /api/...  │  file   │  - override.conf + reload/     │
│    (no root needed)         │ ──────► │    restart/enable/disable       │
│  - writes "what the user    │ (JSON)  │  - writes status.json with      │
│    wants" to the state file │ ◄────── │    the result (OK/error)        │
└─────────────────────────────┘ status  └──────────────────────────────┘
                                 .json

The web panel never calls systemctl directly — it writes the desired state to state.json, and a local root-owned daemon applies it and reports the result in status.json. The only contact root has with the outside world is a file on disk, zero network port. Operations that don't need root (models through /api/..., LiteLLM through systemd --user) go straight from the panel.

Adding a remote host (e.g. a mini-PC) works similarly: the management host is an NFS server, exports a separate directory per host (restricted to its IP), and the panel generates a ready-to-run install script with the daemon's code baked in — run once, manually, on the new machine.

Requirements

• Linux with systemd (Debian/Ubuntu — the installer uses apt).
• Python 3.11+.
• sudo privileges (for installing the daemon, NFS, system dependencies).

Installation

On the host that will be the "master" (the one running the web panel):

git clone [email protected]:cyryllo/OctoLlama.git
cd OctoLlama
./install.sh

The script installs (skipping anything already installed):

1. Ollama (official installer from ollama.com),
2. LiteLLM (uv tool install litellm[proxy], no root),
3. Open WebUI (uv tool install --python 3.11 open-webui, no root — just the binary, starting it is left to the button in the panel's WebUI tab),
4. nfs-kernel-server (to support remote hosts),
5. octollama-daemon — a system systemd service (root),
6. octollama-web — a systemd --user service, the web panel (asks for a port on first install, default 5000 — just press Enter to accept it).

On first install, the script also asks you to set your login username/password right there (password entered twice, to confirm). To change credentials later, run it again by hand:

cd ~/.local/share/octollama-web
./.venv/bin/python3 manage_users.py

The panel will be available at http://<this-host's-address>:<port> (the port you chose, or 5000 by default). Both the port and the credentials are only asked for once — re-running the installer to update won't reset them.

Updating

git pull then re-run ./install.sh. It detects the installed version (from the VERSION file) against the one in your checkout:

• older installed — offers to update (default: yes),
• same version — asks before reinstalling (default: no),
• newer installed — warns before overwriting it with an older one.

Uninstalling

./install.sh --uninstall

Always removes OctoLlama's own daemon and web panel (services, units, installed files), then asks separately — each defaulting to no — whether to also remove: the state directory and NFS exports for remote hosts, LiteLLM, Open WebUI, Ollama (this deletes every downloaded model), and nfs-kernel-server. Nothing beyond OctoLlama itself is removed unless you say yes.

Managing the services

Two independent services get installed — the daemon (system, root) and the web panel (user, no root):

# octollama-daemon (root, system service)
sudo systemctl status octollama-daemon
sudo systemctl restart octollama-daemon
sudo systemctl stop octollama-daemon
sudo systemctl start octollama-daemon
sudo journalctl -u octollama-daemon -f      # live log

# octollama-web (user service, no sudo)
systemctl --user status octollama-web
systemctl --user restart octollama-web
systemctl --user stop octollama-web
systemctl --user start octollama-web
journalctl --user -u octollama-web -f       # live log

Restarting the web panel does not affect the Ollama service itself — the daemon keeps enforcing the last state it received from state.json regardless of whether the panel is running. Stopping the daemon means changes made in the panel (start/stop/env variables) won't be applied until it's running again.

Usage

The panel has four tabs:

• Master — the Ollama service and its environment variables on this host, a link to manage models.
• Slave — add/remove remote Ollama hosts and see each one's status (MAC shown under its IP). After adding a host, the panel generates install-<name>.sh — download and run it (over SSH) on the target machine; it installs Ollama, mounts the state directory over NFS (retrying if the export isn't ready on the master yet), and sets up its own daemon. Each host's row has icon buttons (hover for a label) for its own "Ollama settings" page (same start/stop/autostart/environment-variable controls as Master, just for that host), model management, re-downloading the installer, removal, Wake-on-LAN (auto-detected from ARP when reachable, or entered manually), and remote suspend/restart/power-off — the last four ask for confirmation before acting.
• LLM — start/stop the LiteLLM aggregator, choose which models from which hosts get exposed, generate the Continue.dev config, plus load balancing and reliability settings (see below).
• WebUI — start/stop Open WebUI, wired to LiteLLM (sees the same, deliberately selected models, from every host at once).

Load balancing and reliability (LLM tab)

When the same model (same name and tag, e.g. qwen2.5-coder:14b) is exposed on more than one host, LiteLLM automatically load-balances requests between them — the panel shows this as "balanced across N hosts". On top of that, the LLM tab exposes LiteLLM's built-in reliability features:

• Routing strategy — how LiteLLM picks a host among the balanced ones: simple-shuffle (random, default), least-busy, latency-based-routing, usage-based-routing-v2.
• Host priority — an optional order per host for a balanced model (lower number = higher priority). Leave blank for pure load balancing with no preferred host.
• Fallbacks — an optional backup model for any exposed model, tried when the primary one fails.
• Retry / timeout / cooldown — num_retries, request timeout, cooldown_time, and allowed_fails before a host is temporarily cooled down.
• Context window fallbacks (checkbox) — when enabled, a request that exceeds a small model's context window automatically retries against a larger model you pick.

These settings are stored in web/litellm_ustawienia.json (not committed, same pattern as hosts.json). Saving them immediately restarts the running LiteLLM service with the new config, same as the model selection above. The generator merges its own model_list/router_settings/litellm_settings entries into litellm_config.yaml (tagging its own model entries with model_info.managed_by: octollama) instead of overwriting the file, so manual additions to that file (e.g. another provider's model with its own API key, general_settings) are preserved across restarts.

Model roles for Continue.dev (LLM tab)

Each exposed model can be assigned one or more Continue.dev roles: chat, edit, apply, autocomplete, embed, rerank. Only the roles a model's capabilities actually support are shown as checkboxes at all (checked by default) — determined from what Ollama itself reports for that model (shown next to it) via /api/tags: chat needs completion, autocomplete needs insert (FIM support), edit/apply need tools (function-calling), and embed/rerank need embedding. A model with none of the required capabilities reported (e.g. an older Ollama that doesn't send them at all) isn't restricted. This prevents assigning a role that would just fail with an error in Continue (e.g. edit on a model without tool support). Your choice is stored in role_modele inside web/litellm_ustawienia.json and only affects the generated Continue config (see below), not the running LiteLLM service, so saving it doesn't trigger a restart.

Repo layout

daemon/
  octollama_daemon.py   Root-owned daemon (systemd system unit)
  systemd/                   systemd unit for the daemon
web/
  app.py                     Routes / view logic (Flask)
  ollama_client.py           Ollama REST API client (models)
  litellm_manager.py         LiteLLM control + Continue.dev config
  litellm_ustawienia.py      LiteLLM routing/fallback/retry settings storage
  openwebui_manager.py       Open WebUI control (wired to LiteLLM)
  hosts_store.py             Host list (Slave tab)
  install_generator.py       Installer generator for remote hosts
  wol.py                     Wake-on-LAN (magic packet)
  state_store.py             state.json / status.json read/write
  pobierania.py              Background model-pull progress tracking
  i18n.py                    Translations (_(), language choice in session)
  lang/en.json               English translation dictionary
  templates/                 Jinja templates
  static/style.css           Styles (light/dark theme)
install.sh                   Installer for the management host

Adding another language

Same pattern as Ollama Manager (lang/*.json keyed by the Polish source text): create web/lang/<code>.json with translations, add "<code>": "name" to JEZYKI in web/i18n.py. A missing dictionary entry = the panel shows the original (Polish), so an incomplete translation never leaves a blank spot.

Status

A working skeleton — service control, models, LiteLLM, Open WebUI, Continue.dev config, multi-host support (NFS + installer generator), and host power management (Wake-on-LAN + remote power off/restart/suspend) all work end-to-end. The NFS/remote-daemon path hasn't been verified on real hardware yet. Deliberately left out: TLS (to be handled by a reverse proxy in front of the panel).

Related projects

• Ollama Manager — the PyQt6/KDE desktop app, the source of the service/model/LiteLLM control logic ported here.

License

GNU GPLv3 — this project contains logic ported from Ollama Manager (GPLv3), so it inherits the same license.