Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions config.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
from providers.cline_pass import CLINE_API_BASE_URL
from providers.codex_everywhere import codex_everywhere_base_urls
from providers.image_relays import image_relay_base_urls
from providers.local_inference import local_inference_base_urls
from providers.nanogpt import nanogpt_text_base_url


Expand Down Expand Up @@ -212,6 +213,7 @@ class Config:
'gemma': 'https://generativelanguage.googleapis.com/v1beta',
**codex_everywhere_base_urls(),
**image_relay_base_urls(),
**local_inference_base_urls(),
}

# Provider-specific timeouts (connect_timeout, read_timeout)
Expand Down
76 changes: 76 additions & 0 deletions docs/local-inference.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
# Local inference providers

Configure an OpenAI-compatible server explicitly to use `ollama`, `vllm`,
`lmstudio`, `llamacpp`, or `sglang` model prefixes. All five providers are
disabled when their base URL variables are unset or empty. There is no port
scanning, host discovery, model download, or inference runtime installation.

| Provider | Base URL variable | Optional provider key variable |
| --- | --- | --- |
| `ollama` | `OLLAMA_BASE_URL` | `OLLAMA_API_KEY` |
| `vllm` | `VLLM_BASE_URL` | `VLLM_API_KEY` |
| `lmstudio` | `LM_STUDIO_BASE_URL` | `LM_STUDIO_API_KEY` |
| `llamacpp` | `LLAMA_CPP_BASE_URL` | `LLAMA_CPP_API_KEY` |
| `sglang` | `SGLANG_BASE_URL` | `SGLANG_API_KEY` |

For example, set `OLLAMA_BASE_URL=http://127.0.0.1:11434/v1` before starting
standalone Flask. A bare origin receives `/v1`; a trailing slash is removed.
A service prefix such as `https://inference.example/serve/v1` is preserved.
Configure the server's actual OpenAI-compatible endpoint; Ollama's native
`/api/chat` protocol is not supported here.

URL validation rejects embedded credentials, query strings, fragments,
encoded paths, path traversal, invalid ports, and paths outside `/v1`.
Malformed settings disable that provider and log the variable name once without
its value. Plain HTTP accepts only explicitly configured loopback, RFC 1918,
or IPv6 unique-local addresses, and `localhost` names. Public HTTP, wildcard
bind addresses, multicast, and link-local metadata addresses are rejected.
Private DNS names require HTTPS; validation does not resolve DNS names.

Use a model already loaded on your server:

```json
{
"model": "ollama:example-model:latest",
"messages": [{"role": "user", "content": "Hello"}]
}
```

Send this body to `/v1/chat/completions` with your gateway authorization.
Only the provider prefix is removed from the upstream model ID. Requests go to
the configured `/v1/chat/completions`. Managed `/v1/responses` uses the existing
Chat conversion; native Responses continuation controls such as
`previous_response_id` return a real validation error before dispatch.
Gateway authorization remains required even when the local server needs no key.
Provider keys use the private provider credential loader and are sent only to
the configured upstream, never copied from the caller's gateway authorization.

A provider key is optional. When none is configured, an explicitly configured
local server is called without an `Authorization` header, both for generation
and for catalog refresh. Providers other than these five still refuse a request
when their key is missing.

Catalog refresh queries only configured servers at `/v1/models` using existing
refresh controls and transport timeouts. Results appear in `/v1/models` with
provider prefixes. Explicit model IDs do not require catalog discovery.
Failed refreshes retain the last good catalog and report the real failure;
disabled local providers' saved catalog entries are hidden.
Catalogs retain bounded provider declarations and attach
`provider_catalog:<provider>` provenance. Tools, vision, and other optional
capabilities are not inferred from model names or assumed from OpenAI API
compatibility. The adapter declares Chat and streaming transport support only.
The existing precision policy exposes declared precision when
`MODEL_PRECISION_PREFERENCE` is enabled; missing precision stays `unknown`.
Compatibility does not certify a server's model capabilities.

Local inference behind the Worker runs through Flask in its Container. Loopback
there means the Container, not the operator's workstation. Use a configured
origin reachable from the Container and follow the existing outbound/SSRF
policy; configuring a URL does not create connectivity or bypass that policy.
There is no GPU inference on the Worker and no additional native Worker route.

No monetary price is inferred for local compute. Existing accounting and
retention policies still apply; requests are not free merely because a server
is local. This integration stores model catalog metadata without provider
credentials and adds no prompt or response logging. Upstream errors remain
errors, and ambiguous generation failures do not authorize a new retry.
129 changes: 129 additions & 0 deletions providers/local_inference.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
"""Explicit OpenAI-compatible local endpoints; no host or model discovery."""

from collections.abc import Mapping
import ipaddress
import logging
import os
import re
from typing import Any
from urllib.parse import urlsplit, urlunsplit

from providers.base import ProviderCapabilities
from providers.openai_compatible import OpenAICompatibleAdapter

LOCAL_INFERENCE_ENV = {
"ollama": ("OLLAMA_BASE_URL", "OLLAMA_API_KEY"),
"vllm": ("VLLM_BASE_URL", "VLLM_API_KEY"),
"lmstudio": ("LM_STUDIO_BASE_URL", "LM_STUDIO_API_KEY"),
"llamacpp": ("LLAMA_CPP_BASE_URL", "LLAMA_CPP_API_KEY"),
"sglang": ("SGLANG_BASE_URL", "SGLANG_API_KEY"),
}
LOCAL_INFERENCE_PROVIDERS = frozenset(LOCAL_INFERENCE_ENV)
_PRIVATE_NETWORKS = tuple(ipaddress.ip_network(value) for value in (
"10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", "fc00::/7",
))
_HOST_PATTERN = re.compile(r"[a-z0-9](?:[a-z0-9.-]*[a-z0-9])?", re.IGNORECASE)
_PATH_PATTERN = re.compile(r"(?:/[A-Za-z0-9_-]+)*/v1")
_invalid_settings_logged: set[str] = set()
logger = logging.getLogger(__name__)


def normalize_local_base_url(value: str) -> str:
"""Validate an operator endpoint without resolving names or probing ports."""
if len(value) > 2048 or any(ord(char) < 32 or ord(char) == 127 for char in value):
raise ValueError("Invalid local inference base URL")
candidate = value.strip()
if any(char in candidate for char in "\\%?#"):
raise ValueError("Invalid local inference base URL")
parsed = urlsplit(candidate)
host = (parsed.hostname or "").lower().rstrip(".")
port = parsed.port
if (parsed.scheme not in {"http", "https"} or not host
or parsed.username is not None or parsed.password is not None
or (port is not None and not 1 <= port <= 65535)):
raise ValueError("Invalid local inference origin")
try:
address = ipaddress.ip_address(host)
except ValueError:
address = None
private = bool(address and (address.is_loopback or any(
address.version == network.version and address in network
for network in _PRIVATE_NETWORKS
)))
localhost = host == "localhost" or host.endswith(".localhost")
if address is not None and not (private or address.is_global):
raise ValueError("Invalid local inference address")
if address is None and not _HOST_PATTERN.fullmatch(host):
raise ValueError("Invalid local inference hostname")
if parsed.scheme == "http" and not (private or localhost):
raise ValueError("Local HTTP requires a loopback or private address")
path = parsed.path.rstrip("/") or "/v1"
if not _PATH_PATTERN.fullmatch(path):
raise ValueError("Local inference requires an OpenAI-compatible /v1 service")
authority = f"[{host}]" if ":" in host else host
if port is not None:
authority += f":{port}"
return urlunsplit((parsed.scheme, authority, path, "", ""))


def local_inference_base_urls() -> dict[str, str]:
urls = {}
for provider, (env_name, _) in LOCAL_INFERENCE_ENV.items():
value = os.environ.get(env_name, "")
if not value.strip():
continue
try:
urls[provider] = normalize_local_base_url(value)
except ValueError:
if env_name not in _invalid_settings_logged:
_invalid_settings_logged.add(env_name)
logger.warning("Ignoring invalid %s; local inference provider is off", env_name)
return urls


def local_inference_adapters(base_urls: Mapping[str, str]) -> dict[str, OpenAICompatibleAdapter]:
adapters = {}
for provider in LOCAL_INFERENCE_ENV:
value = base_urls.get(provider)
if not value:
continue
try:
base_url = normalize_local_base_url(value)
except ValueError:
continue
adapters[provider] = OpenAICompatibleAdapter(
name=provider, base_url=base_url, chat_path="chat/completions",
provider_capabilities=ProviderCapabilities(),
)
return adapters


def local_credential_env_names(provider: str) -> tuple[str, ...]:
settings = LOCAL_INFERENCE_ENV.get(provider)
return (settings[1],) if settings else ()


def local_provider_allows_keyless(provider: str, base_urls: Mapping[str, str]) -> bool:
return provider in local_inference_adapters(base_urls)


def local_catalog_metadata(provider: str, metadata: Mapping[str, Any] | None) -> dict[str, Any]:
"""Tag declarations, leaving model-name guesses and monetary cost unknown."""
result = dict(metadata or {})
source = f"provider_catalog:{provider}"
fields = {key for key in result if key.startswith("supports_") or key in {
"input_modalities", "output_modalities", "context_window", "max_output_tokens",
}}
result["metadata_source"] = source
result["metadata_provenance"] = {
**(result.get("metadata_provenance") or {}),
**{key: source for key in fields},
}
# Precision declarations follow the existing operator precision policy.
from services.model_precision import configured_preference

if configured_preference():
result.setdefault("precision", "unknown")
result.setdefault("precision_source", source)
result["metadata_provenance"]["precision"] = result["precision_source"]
return result
2 changes: 2 additions & 0 deletions providers/registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
from providers.base import ProviderCapabilities
from providers.codex_everywhere import CODEX_EVERYWHERE_POOLS, OPENAI_PROTOCOL
from providers.image_relays import image_relay_specs
from providers.local_inference import local_inference_adapters
from providers.openai_compatible import OpenAICompatibleAdapter


Expand Down Expand Up @@ -174,6 +175,7 @@ def build_default_registry(base_urls: Mapping[str, str]) -> dict[str, OpenAIComp
chat_path="v1/chat/completions",
provider_capabilities=spec.capabilities(),
)
registry.update(local_inference_adapters(base_urls))
return registry


Expand Down
6 changes: 5 additions & 1 deletion routes/provider_credentials.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
from services.nanogpt_key_pool import NanoGPTUnifiedKeyPool as NanoGPTKeyPool


def _provider_token(app, auth_service_cls, proxy_service_cls, provider: str, *, model=None, quota_bucket=None) -> str:
def _provider_token(app, auth_service_cls, proxy_service_cls, provider: str, *, model=None, quota_bucket=None) -> str | None:
context = selection_context(provider, model, config=app.config, quota_bucket=quota_bucket)
if provider == "googleai":
token = auth_service_cls.get_google_token()
Expand All @@ -42,6 +42,10 @@ def _provider_token(app, auth_service_cls, proxy_service_cls, provider: str, *,
else:
token = auth_service_cls.get_api_key(provider, **context)
if not token:
# Only an explicitly configured local inference server may be called without a key.
requires_key = getattr(auth_service_cls, "provider_requires_api_key", None)
if requires_key is not None and not requires_key(provider, app.config.get("API_BASE_URLS", {})):
return None
raise AutoRouteCandidateUnavailable(
f"API key not configured for {provider}",
)
Expand Down
18 changes: 18 additions & 0 deletions services/auth_service.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,12 @@
from error_handlers import APIError
from providers.codex_everywhere import CODEX_EVERYWHERE_POOLS
from providers.image_relays import image_relay_api_key, image_relay_specs
from providers.local_inference import (
LOCAL_INFERENCE_PROVIDERS,
local_credential_env_names,
local_inference_base_urls,
local_provider_allows_keyless,
)
from services.credential_pool import CredentialPool
from services.auth_primitives import (
DEFAULT_ADMIN_SCOPES,
Expand Down Expand Up @@ -672,6 +678,11 @@ def initialize(cls) -> None:
@classmethod
def get_api_key(cls, provider: str, *, model: str | None = None, quota_bucket: str | None = None) -> Optional[str]:
"""Get API key for a provider."""
if provider in LOCAL_INFERENCE_PROVIDERS:
if provider not in local_inference_base_urls():
return None
return next((key for name in local_credential_env_names(provider)
if (key := usable_credential(os.environ.get(name), name))), None)
if CredentialPool.pooled(provider):
pooled = CredentialPool.select(provider, model=model, quota_bucket=quota_bucket)
if pooled:
Expand All @@ -688,8 +699,15 @@ def get_api_key(cls, provider: str, *, model: str | None = None, quota_bucket: s
@classmethod
def provider_credential_env_names(cls, provider: str) -> tuple[str, ...]:
"""Return safe environment-variable names for dashboard setup guidance."""
if provider in LOCAL_INFERENCE_PROVIDERS:
return local_credential_env_names(provider)
return provider_credential_env_names(provider)

@staticmethod
def provider_requires_api_key(provider: str, base_urls) -> bool:
"""Local servers may be keyless only for an explicitly configured endpoint."""
return not local_provider_allows_keyless(provider, base_urls)

@classmethod
def get_api_keys(cls, provider: str) -> List[str]:
"""Get every configured key for providers that support a key pool."""
Expand Down
2 changes: 2 additions & 0 deletions services/model_registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
from providers.cline_pass import CLINE_PASS_MODEL_IDS
from providers.codex_everywhere import CODEX_EVERYWHERE_PROVIDERS
from providers.image_relays import image_relay_model_ids
from providers.local_inference import LOCAL_INFERENCE_PROVIDERS
from providers.opencode_go import (
OPENCODE_GO_LEGACY_MODEL_IDS,
OPENCODE_GO_MODEL_IDS,
Expand All @@ -31,6 +32,7 @@
}

DYNAMIC_PROVIDER_MODELS = {
*LOCAL_INFERENCE_PROVIDERS,
"codex-easy",
*CODEX_EVERYWHERE_PROVIDERS,
"linkapi",
Expand Down
26 changes: 26 additions & 0 deletions services/provider_catalog_service.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,12 @@
)
from providers.codex_everywhere import CODEX_EVERYWHERE_POOLS
from providers.image_relays import image_relay_specs
from providers.local_inference import (
LOCAL_INFERENCE_PROVIDERS,
local_catalog_metadata,
local_inference_base_urls,
normalize_local_base_url,
)
from providers.opencode_go import is_opencode_zen_free_model
from services import provider_catalog_d1
from services.provider_capability_discovery import enrich_model_capabilities
Expand Down Expand Up @@ -48,6 +54,8 @@ class ProviderCatalogModel:


PROVIDER_CATALOG_SPECS = {
**{provider: ProviderCatalogSpec("models", f"/{provider}/models")
for provider in sorted(LOCAL_INFERENCE_PROVIDERS)},
"aihubmix": ProviderCatalogSpec("v1/models", "/aihubmix/v1/models"),
"openai": ProviderCatalogSpec("v1/models", "/openai/v1/models"),
"openrouter": ProviderCatalogSpec("models", "/openrouter/models"),
Expand Down Expand Up @@ -175,6 +183,13 @@ def _ensure_storage(cls, connection: sqlite3.Connection) -> None:

@classmethod
def list_models(cls) -> list[ProviderCatalogModel]:
models = cls._stored_models()
enabled = local_inference_base_urls()
return [model for model in models
if model.provider not in LOCAL_INFERENCE_PROVIDERS or model.provider in enabled]

@classmethod
def _stored_models(cls) -> list[ProviderCatalogModel]:
if provider_catalog_d1.using_d1():
return list(cls._durable_models())
cache_key = cls._cache_key()
Expand Down Expand Up @@ -219,6 +234,8 @@ def list_models(cls) -> list[ProviderCatalogModel]:
@classmethod
def has_model(cls, provider: str, model_id: str) -> bool:
"""Return whether a model appears in the last successful live catalog."""
if provider in LOCAL_INFERENCE_PROVIDERS and provider not in local_inference_base_urls():
return False
if provider_catalog_d1.using_d1():
return any(
model.provider == provider and model.model_id == model_id
Expand Down Expand Up @@ -408,6 +425,8 @@ def extract_models(
)
existing = normalized.get(model_id)
metadata = sanitize_provider_metadata(item)
if provider in LOCAL_INFERENCE_PROVIDERS:
metadata = local_catalog_metadata(provider, metadata)
if existing:
context_window = cls._minimum_limit(
existing.context_window,
Expand Down Expand Up @@ -440,6 +459,8 @@ def _credential_candidates(
auth_service_cls,
provider: str,
) -> tuple[str | None, ...]:
if provider in LOCAL_INFERENCE_PROVIDERS:
return (auth_service_cls.get_api_key(provider),)
if provider in PUBLIC_CATALOG_PROVIDERS:
return (None,)
if provider == "nanogpt":
Expand Down Expand Up @@ -599,6 +620,11 @@ def refresh_configured(
base_url = base_urls.get(provider)
if not base_url:
continue
if provider in LOCAL_INFERENCE_PROVIDERS:
try:
base_url = normalize_local_base_url(base_url)
except ValueError:
continue
credentials = cls._credential_candidates(auth_service_cls, provider)
if not credentials:
results.append(
Expand Down
Loading
Loading