diff --git a/CHANGELOG.md b/CHANGELOG.md index 0974a27..ea1998b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -36,6 +36,7 @@ ### Features +* route models LiteLLM reports with `mode: "responses"` through the OpenAI Responses API (per-model runtime package on OpenCode 2; a `-responses` sibling on OpenCode 1) * support the OpenCode 2 API with the OpenCode 1 server entrypoint * add `formatModelNames` option to keep raw model ids in the picker diff --git a/README.md b/README.md index 991f156..a7defee 100644 --- a/README.md +++ b/README.md @@ -77,6 +77,7 @@ also shows how to configure a provider directly in `opencode.json`. | 🧠 **Modality-aware** | Enriches `/v1/models` entries with `/v1/model/info` (`mode`, token limits, capability flags) and hides embedding / image / audio models from the picker. | | 💵 **Real pricing** | Maps `input_cost_per_token` / `output_cost_per_token` (and cache read/write costs) from `/v1/model/info` into OpenCode's `cost` field, so the picker and `/cost` show what the proxy actually bills instead of `$0.00`. Models LiteLLM has no price for are left unpriced, not falsely marked free. | | 🧩 **Reasoning-effort variants** | When LiteLLM reports per-model effort support (`supports_low_reasoning_effort`, …), the plugin surfaces each level as a picker variant automatically. | +| 🧭 **Responses-API routing** | Models LiteLLM reports with `mode: "responses"` are invoked through `/v1/responses` instead of `/v1/chat/completions`, with no extra configuration. OpenCode 2 overrides the runtime package per model; OpenCode 1 gets a `-responses` sibling. | | 🔐 **Auth-aware** | Honours `LITELLM_API_KEY` / `LITELLM_MASTER_KEY`, `providers.litellm.settings.apiKey`, and OpenCode `/connect` credentials on the V1 server path. | | 🌐 **Gateway-friendly** | Supports provider `headers` (and the legacy `customHeaders` option) for proxies behind Cloudflare Access or other API gateways. | | 🧩 **Splittable catalog** | `includeModels` / `excludeModels` (glob patterns) let one LiteLLM proxy be divided into several OpenCode providers — e.g. by naming prefix — without hand-maintaining a model list. | @@ -208,8 +209,11 @@ Here, `openai/gpt-4o` keeps your custom name; every other model from the proxy i ### Reasoning models and effort variants -All discovered models — reasoning-tier included — register under your single -LiteLLM provider and are invoked through `/v1/chat/completions`. +All discovered models register under your single LiteLLM provider, and models +LiteLLM reports with `mode: "responses"` are invoked through the OpenAI +Responses API (`/v1/responses`) instead of `/v1/chat/completions`. See +[Responses API models](#responses-api-models) for how that routing is applied +on each OpenCode version. If LiteLLM reports per-model reasoning-effort support, the plugin automatically surfaces it as OpenCode variants. Explicit `reasoning_effort_levels` (or @@ -236,11 +240,49 @@ between effort levels from the model picker without hand-curating every entry. > **Note**: OpenAI's reasoning-tier models (gpt-5, o1, o3, o4) reject > requests that combine `reasoning_effort` with function tools on -> `/v1/chat/completions`. If you hit that error, fix it on the LiteLLM -> side — e.g. enable the Responses API for that model -> (`use_responses_api: true` in its `litellm_params`) — or leave -> `reasoningEffort` unset for that model in OpenCode. See the FAQ entry -> below. +> `/v1/chat/completions`. Enable the Responses API for such a model on the +> LiteLLM side (`use_responses_api: true` in its `litellm_params`) and the +> plugin routes it automatically; otherwise reach it without +> `reasoningEffort`. See the FAQ entry below. + +### Responses API models + +LiteLLM reports `mode` per model on `/v1/model/info` (the plugin fetches it +alongside `/v1/models`). When a model's `mode` is `responses` — usually after +you enable the Responses API for it in LiteLLM, e.g. +`use_responses_api: true` — the plugin routes that model to `/v1/responses` +instead of `/v1/chat/completions`. Only the reported `mode` is used; the plugin +does not guess from the model id, so an alias that merely looks like a reasoning +model stays on the transport LiteLLM actually serves it from. + +The routing is applied without any extra configuration, and it differs by +OpenCode version because the two versions pick a runtime package differently: + +- **OpenCode 2** chooses the runtime per model. A Responses model keeps its + provider id and simply overrides the package to + `@opencode/ai/providers/openai/responses`. Chat models are untouched, and no + additional provider appears. +- **OpenCode 1** chooses the runtime per provider, so the plugin adds a sibling + provider named `-responses` (for the default `litellm` provider, + `litellm-responses`) using the `@ai-sdk/openai` package, which defaults to the + Responses API. It reuses the same endpoint, credentials, and custom headers, + and is created only when at least one model needs it. + +Because the OpenCode 1 sibling occupies the `-responses` id suffix, the plugin +does not manage any provider whose id ends with `-responses`; pick a different +name if you want the plugin to discover models into a provider of your own. +Selecting a Responses model on OpenCode 1 therefore means picking +`-responses/` rather than `/`. + +> **Hand-curated models on OpenCode 1.** OpenCode 1 chooses the runtime per +> provider, so routing a model to the Responses API means putting it in the +> `-responses` provider. The plugin only relocates models it +> discovered itself; entries you hand-defined under the chat provider keep +> their place and stay on Chat Completions. To route a hand-defined model +> through the Responses API on OpenCode 1, define it under the +> `-responses` provider (or omit it and let discovery add it). +> OpenCode 2 needs no relocation because it overrides the runtime per model, so +> the same model keeps its provider id either way. ### Authentication @@ -519,10 +561,10 @@ The `customHeaders` map works for any gateway that requires extra HTTP headers
I get Function tools with reasoning_effort are not supported … in /v1/chat/completions — what do I do? -This error comes from OpenAI: their reasoning-tier models (gpt-5, o1, o3, o4) refuse function-tool calls on `/v1/chat/completions` when `reasoning_effort` is set. The plugin registers every discovered model through the chat-completions path, so fix this on the LiteLLM side: +This error comes from OpenAI: their reasoning-tier models (gpt-5, o1, o3, o4) refuse function-tool calls on `/v1/chat/completions` when `reasoning_effort` is set. When LiteLLM reports such a model with `mode: "responses"` — for example after setting `use_responses_api: true` in its `litellm_params` — the plugin sends that model to `/v1/responses` and the error goes away. See [Responses API models](#responses-api-models). -- Enable the Responses API for that model in your LiteLLM config (e.g. `use_responses_api: true` in its `litellm_params`), or -- Leave the model's `reasoningEffort` unset in OpenCode (don't pick a reasoning-effort variant for it). +- If the model isn't routed, confirm `/v1/model/info` reports `mode: "responses"` for it. The plugin acts only on that signal and does not guess from the model id. +- If you'd rather stay on Chat Completions, leave the model's `reasoningEffort` unset in OpenCode (don't pick a reasoning-effort variant for it). If your model id doesn't look like a reasoning model to LiteLLM (e.g. you renamed it), also check that its `model_info` in `litellm config.yaml` carries the right `supports_*` flags.
diff --git a/src/plugin/index.ts b/src/plugin/index.ts index acd14e2..ab820c2 100644 --- a/src/plugin/index.ts +++ b/src/plugin/index.ts @@ -11,11 +11,13 @@ import { formatModelName, categorizeModel, } from '../utils/format-model-name' +import { requiresResponsesAPI } from '../utils/transport' import type { LiteLLMModel, LiteLLMModelInfo } from '../types' import { getOpenCodeStoredApiKey } from '../utils/opencode-auth' import { buildCacheKey, readModelCache, + readModelCacheResponsesIds, writeModelCache, readModelCacheSavedAt, } from '../utils/model-cache' @@ -25,6 +27,13 @@ import { applyCapabilityOverrides, parseModelCapabilities } from '../utils/model import type { ModelCapabilities } from '../utils/model-capabilities' const CHAT_PROVIDER_ID = 'litellm' +// The OpenCode 1 config hook can only choose a runtime package per provider, +// so models that need the Responses API get a sibling provider. OpenCode 2 +// selects the runtime per model instead (see plugin/v2.ts) and never creates +// this provider. The suffix is reserved: the config hook skips ids that end +// with it so it never treats its own sibling as a chat provider. +const RESPONSES_PROVIDER_SUFFIX = '-responses' +const RESPONSES_NPM = '@ai-sdk/openai' // Covers the 3 s health check plus the parallel models/model-info fetch // phase, with headroom. Scales with LITELLM_REQUEST_TIMEOUT_MS so slow // proxies aren't cut off by the overall cap either (issue #20). @@ -296,16 +305,29 @@ export function toConfigModel( return entry } +/** + * Result of discovering a LiteLLM proxy's models. + * + * `models` is the `{ id -> entry }` map written into an OpenCode provider + * config (or translated into `Model.Info` on OpenCode 2). `responsesIds` + * lists the subset that must be invoked through the OpenAI Responses API; + * callers route those accordingly without inspecting each entry. + */ +export interface DiscoveredModels { + models: Record + responsesIds: string[] +} + /** * Fetch and build OpenCode model entries from a LiteLLM proxy. * * Pure with respect to plugin config: it performs the network calls, - * classifies + formats each model, and returns a `{ id -> entry }` map. - * The provider's `includeModels`/`excludeModels` filters, - * `modelCapabilities` overrides and `formatModelNames` choice are - * applied here (not at merge time) so every path that persists or - * serves a cache — cold discovery and background refresh — writes the - * same adjusted view. + * classifies + formats each model, and returns the built entries plus the + * ids that require the Responses API. The provider's + * `includeModels`/`excludeModels` filters, `modelCapabilities` overrides and + * `formatModelNames` choice are applied here (not at merge time) so every + * path that persists or serves a cache — cold discovery and background + * refresh — writes the same adjusted view. * * Returns `null` when the proxy is unreachable/unauthorized or exposes * no models, so callers can distinguish "no data" from "empty result". @@ -318,7 +340,7 @@ export async function discoverModels( filters: ModelFilters = {}, capabilities: ModelCapabilities = {}, formatModelNames = true, -): Promise | null> { +): Promise { if (!(await checkLiteLLMHealth(baseURL, apiKey, customHeaders))) { log( 'warn', @@ -368,6 +390,7 @@ export async function discoverModels( } const built: Record = {} + const responsesIds: string[] = [] let skipped = 0 let wildcards = 0 let filtered = 0 @@ -389,16 +412,14 @@ export async function discoverModels( } const info = infoByName?.get(model.id) if (infoByName && !info) unmatched.push(model.id) - const entry = toConfigModel( - enrichModel(model, info, capabilities[model.id]), - info, - formatModelNames, - ) + const enriched = enrichModel(model, info, capabilities[model.id]) + const entry = toConfigModel(enriched, info, formatModelNames) if (!entry) { skipped++ continue } built[model.id] = entry + if (requiresResponsesAPI(enriched)) responsesIds.push(model.id) } if (unmatched.length > 0) { @@ -430,7 +451,7 @@ export async function discoverModels( ')', ) - return built + return { models: built, responsesIds } } /** @@ -455,6 +476,134 @@ function mergeModels( return added } +/** + * Ensure the sibling provider that carries Responses-API models exists in + * the OpenCode 1 config, mirroring the chat provider's endpoint and + * credentials. Returns its `models` map. + * + * The Responses runtime is `@ai-sdk/openai` (AI SDK 5's OpenAI provider + * defaults to `/v1/responses`). The sibling is only materialized when at + * least one discovered model needs it, so chat-only proxies keep a single + * provider and never pay for the extra entry. + */ +function ensureResponsesProvider( + config: { provider: Record }, + baseProviderId: string, + baseProvider: Record, + baseURL: string, + apiKey: string | undefined, + customHeaders: Record | undefined, +): Record { + const id = `${baseProviderId}${RESPONSES_PROVIDER_SUFFIX}` + if (!config.provider[id] || typeof config.provider[id] !== 'object') { + config.provider[id] = {} + } + const provider = config.provider[id] as Record + if (!provider.npm) provider.npm = RESPONSES_NPM + if (typeof provider.name !== 'string' || !provider.name) { + const baseName = + typeof baseProvider.name === 'string' && baseProvider.name + ? baseProvider.name + : 'LiteLLM (proxy)' + provider.name = `${baseName} Responses` + } + if (!provider.options || typeof provider.options !== 'object') { + provider.options = {} + } + const options = provider.options as Record + if (!options.baseURL) options.baseURL = `${baseURL}/v1` + if (!options.apiKey && apiKey) options.apiKey = apiKey + // OpenCode 1 reads provider headers from `options.headers` and hands them to + // the runtime package (`@ai-sdk/openai` here); it has no `customHeaders` key. + // Mirror the chat provider's native headers and promote the plugin's + // `customHeaders` so Responses requests carry the same gateway headers. + const baseOptions = (baseProvider.options ?? {}) as Record + const mergedHeaders = { + ...readHeaderMap(baseOptions.headers), + ...(customHeaders ?? {}), + ...readHeaderMap(options.headers), + } + if (Object.keys(mergedHeaders).length > 0) { + options.headers = mergedHeaders + } + if (!provider.models || typeof provider.models !== 'object') { + provider.models = {} + } + return provider.models as Record +} + +/** + * Extract a `string -> string` header map from an arbitrary provider option + * value, ignoring anything that isn't a string. Returns an empty map for + * missing or malformed input. + */ +function readHeaderMap(value: unknown): Record { + const out: Record = {} + if (value && typeof value === 'object' && !Array.isArray(value)) { + for (const [key, header] of Object.entries(value as Record)) { + if (typeof header === 'string') out[key] = header + } + } + return out +} + +/** + * Split freshly discovered entries into the chat provider and the Responses + * sibling, injecting each side without clobbering curated entries. Returns + * every id this call added, across both providers, so the config hook's + * idempotency check sees a single set. + */ +function injectDiscoveredModels( + config: { provider: Record }, + baseProviderId: string, + baseProvider: Record, + baseModels: Record, + entries: Record, + responsesIds: readonly string[], + baseURL: string, + apiKey: string | undefined, + customHeaders: Record | undefined, +): string[] { + const responsesSet = new Set(responsesIds) + const existingSibling = config.provider[`${baseProviderId}${RESPONSES_PROVIDER_SUFFIX}`] + const siblingModelsExisting: Record = + existingSibling && + typeof existingSibling === 'object' && + (existingSibling as Record).models && + typeof (existingSibling as Record).models === 'object' + ? ((existingSibling as Record).models as Record) + : {} + + const chatEntries: Record = {} + const responsesEntries: Record = {} + for (const [id, entry] of Object.entries(entries)) { + // A model already present under the other provider was curated there (or + // injected by an earlier run); leave it alone instead of duplicating it + // across both providers. + if (responsesSet.has(id)) { + if (Object.hasOwn(baseModels, id)) continue + responsesEntries[id] = entry + } else { + if (Object.hasOwn(siblingModelsExisting, id)) continue + chatEntries[id] = entry + } + } + + const added = mergeModels(baseModels, chatEntries) + if (Object.keys(responsesEntries).length > 0) { + const siblingModels = ensureResponsesProvider( + config, + baseProviderId, + baseProvider, + baseURL, + apiKey, + customHeaders, + ) + added.push(...mergeModels(siblingModels, responsesEntries)) + } + return added +} + /** * Revalidate a provider's model cache off the critical path (SWR). The * refreshed entries land in the on-disk cache and surface on the next @@ -473,7 +622,7 @@ async function backgroundRefresh(cacheKey: string): Promise { } refreshInFlight.add(cacheKey) try { - const built = await withTimeout( + const discovered = await withTimeout( discoverModels( ctx.baseURL, ctx.apiKey, @@ -485,11 +634,11 @@ async function backgroundRefresh(cacheKey: string): Promise { ), DISCOVERY_TIMEOUT_MS, ) - if (built && Object.keys(built).length > 0) { - writeModelCache(cacheKey, built) + if (discovered && Object.keys(discovered.models).length > 0) { + writeModelCache(cacheKey, discovered.models, discovered.responsesIds) log( 'info', - `[opencode-litellm] Background-refreshed model cache for ${ctx.baseURL} (${Object.keys(built).length} models)`, + `[opencode-litellm] Background-refreshed model cache for ${ctx.baseURL} (${Object.keys(discovered.models).length} models)`, ) } } catch { @@ -535,6 +684,9 @@ export const LiteLLMPlugin: Plugin = async (input: PluginInput) => { const liteLLMProviders: Array<{ id: string; provider: Record }> = [] for (const id of providerIds) { + // The Responses sibling is created and populated below; never treat + // it as a chat provider (that would recurse into `…-responses-responses`). + if (id.endsWith(RESPONSES_PROVIDER_SUFFIX)) continue const provider = config.provider[id] if (provider && typeof provider === 'object') { const options = (provider.options ?? {}) as Record @@ -659,12 +811,27 @@ export const LiteLLMPlugin: Plugin = async (input: PluginInput) => { providerId, }) + // The Responses sibling (if any) holds a disjoint set of ids, so the + // idempotency check has to look at both providers. + const responsesProviderId = `${providerId}${RESPONSES_PROVIDER_SUFFIX}` + const responsesModels = (): Record => { + const sibling = config.provider[responsesProviderId] + if (!sibling || typeof sibling !== 'object') return {} + const candidate = (sibling as Record).models + return candidate && typeof candidate === 'object' + ? (candidate as Record) + : {} + } + // Repeat config-hook invocations within a run are a no-op once // we've injected this provider's models. const alreadyInjected = injectedModelIds.get(cacheKey) if ( alreadyInjected && - [...alreadyInjected].every((id) => Object.hasOwn(models, id)) + [...alreadyInjected].every( + (id) => + Object.hasOwn(models, id) || Object.hasOwn(responsesModels(), id), + ) ) { continue } @@ -676,7 +843,17 @@ export const LiteLLMPlugin: Plugin = async (input: PluginInput) => { // the same proxy (with different keys) don't share model lists. const cached = readModelCache(cacheKey) if (cached && Object.keys(cached).length > 0) { - const added = mergeModels(models, cached) + const added = injectDiscoveredModels( + config, + providerId, + actualProvider, + models, + cached, + readModelCacheResponsesIds(cacheKey), + baseURL, + apiKey, + customHeaders, + ) injectedModelIds.set(cacheKey, new Set(added)) log( 'info', @@ -688,7 +865,7 @@ export const LiteLLMPlugin: Plugin = async (input: PluginInput) => { // Cold cache: do a live fetch (slow first run only), inject, and // persist for subsequent startups. Capped by a timeout so a slow // proxy never blocks boot. - const built = await withTimeout( + const discovered = await withTimeout( discoverModels( baseURL, apiKey, @@ -700,10 +877,20 @@ export const LiteLLMPlugin: Plugin = async (input: PluginInput) => { ), DISCOVERY_TIMEOUT_MS, ) - if (built && Object.keys(built).length > 0) { - const added = mergeModels(models, built) + if (discovered && Object.keys(discovered.models).length > 0) { + const added = injectDiscoveredModels( + config, + providerId, + actualProvider, + models, + discovered.models, + discovered.responsesIds, + baseURL, + apiKey, + customHeaders, + ) injectedModelIds.set(cacheKey, new Set(added)) - writeModelCache(cacheKey, built) + writeModelCache(cacheKey, discovered.models, discovered.responsesIds) } } }, diff --git a/src/plugin/v2.ts b/src/plugin/v2.ts index e7e2d24..5a868c5 100644 --- a/src/plugin/v2.ts +++ b/src/plugin/v2.ts @@ -15,6 +15,7 @@ import { import { buildCacheKey, readModelCache, + readModelCacheResponsesIds, readModelCacheSavedAt, writeModelCache, } from '../utils/model-cache' @@ -25,6 +26,14 @@ import type { ModelFilters } from '../utils/model-filter' const CHAT_PROVIDER_ID = 'litellm' const OPENAI_COMPATIBLE_PACKAGE = '@opencode/ai/providers/openai-compatible' +// Runtime package that talks to `/v1/responses`. OpenCode 2 lets one model +// override the provider's package, so Responses models stay under their +// original provider id and no sibling provider is needed here. +// +// `@opencode/ai/providers/openai/responses` is the Responses runtime OpenCode +// bundles today; a plain OpenAI-compatible provider defaults to the Chat +// Completions runtime, so the override is what selects the Responses API. +const RESPONSES_PACKAGE = '@opencode/ai/providers/openai/responses' const REFRESH_MIN_INTERVAL_MS = 5 * 60 * 1000 type ProviderInfo = Awaited>['data'][number] @@ -98,6 +107,7 @@ function numberOrZero(value: unknown): number { function toProviderModels( providerID: string, entries: Record, + responsesIds: ReadonlySet = new Set(), ): Model.Info[] { const provider = Provider.ID.make(providerID) @@ -138,6 +148,10 @@ function toProviderModels( return [ { ...defaults, + // A model that targets `/v1/responses` overrides the provider's + // runtime package for itself, so it keeps the provider id while + // still reaching the Responses API. + ...(responsesIds.has(modelID) ? { package: RESPONSES_PACKAGE } : {}), name: readString(entry.name) ?? modelID, limit: { context: numberOrDefault(limit.context, defaults.limit.context), @@ -270,9 +284,10 @@ async function makeProviderSource( formatModelNames, }) let entries = readModelCache(cacheKey) + let responsesIds = readModelCacheResponsesIds(cacheKey) if (!entries || Object.keys(entries).length === 0) { - entries = await withTimeout( + const discovered = await withTimeout( discoverModels( baseURL, apiKey, @@ -284,7 +299,14 @@ async function makeProviderSource( ), DISCOVERY_TIMEOUT_MS, ) - if (entries && Object.keys(entries).length > 0) writeModelCache(cacheKey, entries) + if (discovered && Object.keys(discovered.models).length > 0) { + entries = discovered.models + responsesIds = discovered.responsesIds + writeModelCache(cacheKey, entries, responsesIds) + } else { + entries = null + responsesIds = [] + } } else { logInfo( `[opencode-litellm] Loaded ${Object.keys(entries).length} models from cache for provider "${id}" (${baseURL}); refresh happens in the background on new sessions.`, @@ -311,7 +333,7 @@ async function makeProviderSource( capabilities, formatModelNames, cacheKey, - models: toProviderModels(id, entries ?? {}), + models: toProviderModels(id, entries ?? {}, new Set(responsesIds)), ownedModelIds: new Set(), } } @@ -345,7 +367,7 @@ async function refreshProviderSource( let refresh!: Promise refresh = (async () => { try { - const entries = await withTimeout( + const discovered = await withTimeout( discoverModels( source.baseURL, apiKey, @@ -357,11 +379,21 @@ async function refreshProviderSource( ), DISCOVERY_TIMEOUT_MS, ) - if (!entries || Object.keys(entries).length === 0 || apiKey !== source.apiKey) return + if ( + !discovered || + Object.keys(discovered.models).length === 0 || + apiKey !== source.apiKey + ) { + return + } - const models = toProviderModels(source.id, entries) + const models = toProviderModels( + source.id, + discovered.models, + new Set(discovered.responsesIds), + ) if (JSON.stringify(models) === JSON.stringify(source.models)) { - writeModelCache(source.cacheKey, entries) + writeModelCache(source.cacheKey, discovered.models, discovered.responsesIds) source.refreshRequired = false return } @@ -375,7 +407,7 @@ async function refreshProviderSource( throw error } - writeModelCache(source.cacheKey, entries) + writeModelCache(source.cacheKey, discovered.models, discovered.responsesIds) source.refreshRequired = false logInfo( `[opencode-litellm] Refreshed ${models.length} models for provider "${source.id}"; the provider registry was reloaded.`, diff --git a/src/utils/index.ts b/src/utils/index.ts index a62fab8..6f21701 100644 --- a/src/utils/index.ts +++ b/src/utils/index.ts @@ -1,4 +1,5 @@ export * from './litellm-api' export * from './format-model-name' +export * from './transport' export * from './model-capabilities' export * from './model-filter' diff --git a/src/utils/model-cache.ts b/src/utils/model-cache.ts index fb59c8c..42f29fb 100644 --- a/src/utils/model-cache.ts +++ b/src/utils/model-cache.ts @@ -8,7 +8,7 @@ import { join } from 'node:path' * from older plugin versions are ignored rather than deserialized into * an incompatible structure. */ -const CACHE_VERSION = 2 +const CACHE_VERSION = 3 /** * Maximum age of a cache entry before it's treated as a miss. Without a @@ -23,6 +23,12 @@ interface ModelCacheFile { version: number savedAt: number models: Record + /** + * Model ids that must be invoked through the OpenAI Responses API. Kept + * alongside `models` because the provider-config entries themselves carry + * no transport marker (their shape is the user-facing OpenCode config). + */ + responsesIds?: string[] } /** @@ -147,6 +153,23 @@ export function readModelCacheSavedAt(cacheKey: string): number | null { } } +/** + * Return the cached Responses-API model ids for a cache key. Returns an + * empty array on any problem (missing file, parse error, version + * mismatch), matching {@link readModelCache}'s miss semantics. + */ +export function readModelCacheResponsesIds(cacheKey: string): string[] { + try { + const raw = readFileSync(cacheFile(cacheKey), 'utf8') + const parsed = JSON.parse(raw) as ModelCacheFile + if (parsed.version !== CACHE_VERSION) return [] + if (!Array.isArray(parsed.responsesIds)) return [] + return parsed.responsesIds.filter((id): id is string => typeof id === 'string') + } catch { + return [] + } +} + /** * Persist model entries for a cache key. Best-effort: never throws, so * a read-only or full filesystem can't break discovery. @@ -154,6 +177,7 @@ export function readModelCacheSavedAt(cacheKey: string): number | null { export function writeModelCache( cacheKey: string, models: Record, + responsesIds: readonly string[] = [], ): void { try { mkdirSync(cacheDir(), { recursive: true }) @@ -161,6 +185,7 @@ export function writeModelCache( version: CACHE_VERSION, savedAt: Date.now(), models, + responsesIds: [...responsesIds], } // Write to a temp file and rename so concurrent OpenCode processes // never observe a partially-written JSON file (rename is atomic on diff --git a/src/utils/transport.ts b/src/utils/transport.ts new file mode 100644 index 0000000..c3d3d68 --- /dev/null +++ b/src/utils/transport.ts @@ -0,0 +1,20 @@ +import type { LiteLLMModel } from '../types' + +/** + * Decide whether a model must be invoked through the OpenAI Responses API + * (`/v1/responses`) instead of `/v1/chat/completions`. + * + * The plugin registers every discovered model through the provider's runtime + * package. That package is chosen once per provider, so a model that targets a + * different API surface needs to be routed explicitly. LiteLLM reports this on + * `/v1/model/info` as `mode: "responses"` for deployments configured with the + * Responses API (for example `use_responses_api: true`). + * + * Only the explicit `mode` signal is used. Model-id heuristics (`gpt-5*`, + * `o1/o3/o4*`) are deliberately not applied: an alias that merely looks like a + * reasoning model may still be served through Chat Completions by the proxy, + * and misrouting it would break a deployment the proxy reports as `chat`. + */ +export function requiresResponsesAPI(model: LiteLLMModel): boolean { + return model.mode?.trim().toLowerCase() === 'responses' +} diff --git a/test/model-cache.test.ts b/test/model-cache.test.ts index eb6110c..f5ebff5 100644 --- a/test/model-cache.test.ts +++ b/test/model-cache.test.ts @@ -3,7 +3,7 @@ import { mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'n import { tmpdir } from 'node:os' import { join } from 'node:path' import { afterAll, beforeAll, describe, expect, it } from 'vitest' -import { readModelCache, readModelCacheSavedAt, writeModelCache, buildCacheKey } from '../src/utils/model-cache' +import { readModelCache, readModelCacheResponsesIds, readModelCacheSavedAt, writeModelCache, buildCacheKey } from '../src/utils/model-cache' const KEY = 'litellm@http://localhost:4000' @@ -57,6 +57,18 @@ describe('model cache', () => { expect(Object.keys(readModelCache('b@http://x') ?? {})).toEqual(['b']) }) + it('round-trips the Responses-API model ids alongside the models', () => { + writeModelCache('r@http://x', { chat: {}, responses: {} }, ['responses']) + expect(readModelCache('r@http://x')).toEqual({ chat: {}, responses: {} }) + expect(readModelCacheResponsesIds('r@http://x')).toEqual(['responses']) + }) + + it('reads an empty Responses id list from a missing or id-less cache', () => { + expect(readModelCacheResponsesIds('missing@http://nowhere')).toEqual([]) + writeModelCache('no-ids@http://x', { m: {} }) + expect(readModelCacheResponsesIds('no-ids@http://x')).toEqual([]) + }) + it('ignores cache files written by a different plugin version', () => { writeModelCache(KEY, { m: {} }) rewriteCacheFile(KEY, (parsed) => { diff --git a/test/plugin-v1.test.ts b/test/plugin-v1.test.ts new file mode 100644 index 0000000..9be41d1 --- /dev/null +++ b/test/plugin-v1.test.ts @@ -0,0 +1,228 @@ +import { mkdtempSync, rmSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { LiteLLMPlugin } from '../src/plugin' +import { __resetOpenCodeAuthCacheForTests } from '../src/utils/opencode-auth' + +interface ProviderConfig { + npm?: string + name?: string + options?: Record + models?: Record +} + +interface OpenCodeConfig { + provider: Record +} + +function restoreEnv(name: string, value: string | undefined): void { + if (value === undefined) delete process.env[name] + else process.env[name] = value +} + +describe('OpenCode 1 config hook', () => { + const originalFetch = globalThis.fetch + let originalCacheHome: string | undefined + let originalHome: string | undefined + let originalLiteLLMApiKey: string | undefined + let originalLiteLLMMasterKey: string | undefined + let cacheDirectory: string + let homeDirectory: string + + beforeEach(() => { + originalCacheHome = process.env.XDG_CACHE_HOME + originalHome = process.env.HOME + originalLiteLLMApiKey = process.env.LITELLM_API_KEY + originalLiteLLMMasterKey = process.env.LITELLM_MASTER_KEY + delete process.env.LITELLM_API_KEY + delete process.env.LITELLM_MASTER_KEY + __resetOpenCodeAuthCacheForTests() + }) + + afterEach(() => { + globalThis.fetch = originalFetch + restoreEnv('XDG_CACHE_HOME', originalCacheHome) + restoreEnv('HOME', originalHome) + restoreEnv('LITELLM_API_KEY', originalLiteLLMApiKey) + restoreEnv('LITELLM_MASTER_KEY', originalLiteLLMMasterKey) + if (cacheDirectory) rmSync(cacheDirectory, { recursive: true, force: true }) + if (homeDirectory) rmSync(homeDirectory, { recursive: true, force: true }) + vi.restoreAllMocks() + }) + + function useIsolatedState(): void { + cacheDirectory = mkdtempSync(join(tmpdir(), 'opencode-litellm-v1-cache-')) + homeDirectory = mkdtempSync(join(tmpdir(), 'opencode-litellm-v1-home-')) + process.env.XDG_CACHE_HOME = cacheDirectory + process.env.HOME = homeDirectory + } + + function mockProxy(models: Array<{ id: string; mode?: string }>): void { + globalThis.fetch = vi.fn(async (input) => { + const url = String(input) + if (url.endsWith('/v1/model/info')) { + return new Response( + JSON.stringify({ + data: models.map((model) => ({ + model_name: model.id, + model_info: { key: model.id, mode: model.mode ?? 'chat' }, + })), + }), + { status: 200 }, + ) + } + return new Response( + JSON.stringify({ + data: models.map((model) => ({ id: model.id, object: 'model' })), + }), + { status: 200 }, + ) + }) + } + + async function runConfigHook(baseURL: string): Promise { + const input = { client: { app: { log: vi.fn(async () => {}) } } } + const plugin = await LiteLLMPlugin(input as never) + const config: OpenCodeConfig = { + provider: { + litellm: { + npm: '@ai-sdk/openai-compatible', + name: 'LiteLLM (proxy)', + options: { baseURL, apiKey: 'example-key' }, + models: {}, + }, + }, + } + await plugin.config!(config as never) + return config + } + + it('routes Responses models to a sibling provider, leaving chat models alone', async () => { + useIsolatedState() + const baseURL = 'http://127.0.0.1:45551/v1' + mockProxy([ + { id: 'chat-model', mode: 'chat' }, + { id: 'responses-model', mode: 'responses' }, + ]) + + const config = await runConfigHook(baseURL) + + expect(Object.keys(config.provider.litellm.models ?? {})).toEqual(['chat-model']) + expect(config.provider.litellm.npm).toBe('@ai-sdk/openai-compatible') + + const sibling = config.provider['litellm-responses'] + expect(sibling).toBeDefined() + expect(sibling.npm).toBe('@ai-sdk/openai') + expect(sibling.options).toMatchObject({ + baseURL, + apiKey: 'example-key', + }) + expect(Object.keys(sibling.models ?? {})).toEqual(['responses-model']) + }) + + it('creates no sibling provider when no model needs the Responses API', async () => { + useIsolatedState() + const baseURL = 'http://127.0.0.1:45552/v1' + mockProxy([ + { id: 'chat-model', mode: 'chat' }, + { id: 'completion-model', mode: 'completion' }, + ]) + + const config = await runConfigHook(baseURL) + + expect(Object.keys(config.provider.litellm.models ?? {}).sort()).toEqual([ + 'chat-model', + 'completion-model', + ]) + expect(config.provider['litellm-responses']).toBeUndefined() + }) + + it('does not treat the Responses sibling as a chat provider on later passes', async () => { + useIsolatedState() + const baseURL = 'http://127.0.0.1:45553/v1' + mockProxy([{ id: 'responses-model', mode: 'responses' }]) + + const input = { client: { app: { log: vi.fn(async () => {}) } } } + const plugin = await LiteLLMPlugin(input as never) + const config: OpenCodeConfig = { + provider: { + litellm: { + npm: '@ai-sdk/openai-compatible', + name: 'LiteLLM (proxy)', + options: { baseURL, apiKey: 'example-key' }, + models: {}, + }, + }, + } + + await plugin.config!(config as never) + await plugin.config!(config as never) + + // A naive re-run would nest `litellm-responses-responses`. + expect(Object.keys(config.provider).sort()).toEqual([ + 'litellm', + 'litellm-responses', + ]) + expect(Object.keys(config.provider['litellm-responses'].models ?? {})).toEqual([ + 'responses-model', + ]) + }) + + it('does not duplicate a Responses model already curated on the chat side', async () => { + useIsolatedState() + const baseURL = 'http://127.0.0.1:45554/v1' + mockProxy([{ id: 'responses-model', mode: 'responses' }]) + + const input = { client: { app: { log: vi.fn(async () => {}) } } } + const plugin = await LiteLLMPlugin(input as never) + const config: OpenCodeConfig = { + provider: { + litellm: { + npm: '@ai-sdk/openai-compatible', + options: { baseURL, apiKey: 'example-key' }, + models: { 'responses-model': { name: 'Curated' } }, + }, + }, + } + + await plugin.config!(config as never) + + // The curated entry wins; the plugin leaves it where it is rather than + // mirroring it into the sibling. + expect(Object.keys(config.provider.litellm.models ?? {})).toContain('responses-model') + expect(config.provider['litellm-responses']).toBeUndefined() + }) + + it('propagates gateway headers to the Responses sibling', async () => { + useIsolatedState() + const baseURL = 'http://127.0.0.1:45555/v1' + mockProxy([{ id: 'responses-model', mode: 'responses' }]) + + const input = { client: { app: { log: vi.fn(async () => {}) } } } + const plugin = await LiteLLMPlugin(input as never) + const config: OpenCodeConfig = { + provider: { + litellm: { + npm: '@ai-sdk/openai-compatible', + options: { + baseURL, + apiKey: 'example-key', + headers: { 'CF-Access-Client-Id': 'id-from-base' }, + customHeaders: { 'X-Legacy': 'legacy' }, + }, + models: {}, + }, + }, + } + + await plugin.config!(config as never) + + // OpenCode 1 reads `options.headers`; both the chat provider's native + // headers and the plugin's `customHeaders` must reach the sibling. + expect(config.provider['litellm-responses'].options?.headers).toMatchObject({ + 'CF-Access-Client-Id': 'id-from-base', + 'X-Legacy': 'legacy', + }) + }) +}) diff --git a/test/plugin-v2.test.ts b/test/plugin-v2.test.ts index f4381d5..542a9a4 100644 --- a/test/plugin-v2.test.ts +++ b/test/plugin-v2.test.ts @@ -229,6 +229,8 @@ describe('OpenCode 2 plugin entrypoint', () => { expect(registered[0].models[3]).toMatchObject({ id: 'company-pro', + // `mode: 'responses'` overrides the runtime package for this model only. + package: '@opencode/ai/providers/openai/responses', variants: [ { id: 'medium', settings: { reasoningEffort: 'medium' } }, { id: 'high', settings: { reasoningEffort: 'high' } }, @@ -236,6 +238,11 @@ describe('OpenCode 2 plugin entrypoint', () => { ], }) + // Chat models keep the provider's default package (no per-model override). + expect(registered[0].models[0].package).toBeUndefined() + expect(registered[0].models[1].package).toBeUndefined() + expect(registered[0].models[2].package).toBeUndefined() + await cleanup?.() expect(disposeTransform).toHaveBeenCalledOnce() }) diff --git a/test/transport.test.ts b/test/transport.test.ts new file mode 100644 index 0000000..0109258 --- /dev/null +++ b/test/transport.test.ts @@ -0,0 +1,28 @@ +import { describe, expect, it } from 'vitest' +import { requiresResponsesAPI } from '../src/utils/transport' +import type { LiteLLMModel } from '../src/types' + +function model(id: string, mode?: string): LiteLLMModel { + return { id, object: 'model', mode } +} + +describe('requiresResponsesAPI', () => { + it('routes models LiteLLM reports with mode "responses"', () => { + expect(requiresResponsesAPI(model('any-model', 'responses'))).toBe(true) + expect(requiresResponsesAPI(model('any-model', 'Responses'))).toBe(true) + expect(requiresResponsesAPI(model('any-model', ' responses '))).toBe(true) + }) + + it('keeps chat and other modes on Chat Completions', () => { + expect(requiresResponsesAPI(model('any-model', 'chat'))).toBe(false) + expect(requiresResponsesAPI(model('any-model', 'completion'))).toBe(false) + expect(requiresResponsesAPI(model('any-model'))).toBe(false) + }) + + it('does not guess from the model id alone', () => { + // An alias that merely looks like a reasoning model may still be served + // through Chat Completions; only the proxy's reported mode decides. + expect(requiresResponsesAPI(model('gpt-5.4-pro'))).toBe(false) + expect(requiresResponsesAPI(model('o3-mini'))).toBe(false) + }) +})