diff --git a/CHANGELOG.md b/CHANGELOG.md index bf68998..9161612 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,122 @@ here. The release version is defined in the workspace root `package.json`. ## [Unreleased] +## [0.5.0] - 2026-09-18 + +### Added + +- Decision models, a new model class that answers typed questions + (boolean, choice, score) about a state and returns probabilities. The first + model is `jev` (TypeSafe AI). It is served by whichever provider has a key: + the TypeSafe API (`TYPESAFE_MODELS_API_KEY`, new), Vercel AI Gateway + (`VERCEL_MODELS_API_KEY`) or OpenRouter (`OPENROUTER_MODELS_API_KEY`), in + that order of precedence. `GET /api/v1/platform/model/list?type=decision` + lists the class. +- `POST /api/v1/decision/create` answers typed questions with a decision + model. It takes `model`, `state` and `questions`, returns `answers` and + `usage`, and is part of the public API specification. It is metered against + the token limit on the model's input tokens, like the other model classes. + A deployment with no decision provider key answers a 400 that says so. + +### Changed + +- The code snippets the application shows for the SDKs, the CLI and Terraform + (on the token, bot, dataset, skillset and secret pages, and in generated + Terraform) now use the `token` option, `api_token` and the + `CHATBOTKIT_API_TOKEN` environment variable. They need the SDK and provider + releases that introduce those names; the former names keep working there. + The SDK guide in `docs/sdks.md` also covers `CHATBOTKIT_API_URL` for pointing + the CLI and Terraform at a deployment. + +### Fixed + +- The image and video create and edit routes failed with an internal error on + a deployment that has no image or video provider key, because an empty model + catalogue accepts any model name. They now answer a 400 that says no such + model is configured. Editing without naming a model also failed when the + preferred edit model (`gpt-image-1`, `grok-imagine-video`) was not served; + it now falls back to the deployment's default model. Deployments that serve + those models are unaffected. +- The chat app showed a generic failure, and reported an unhandled error to + Sentry, when a completion failed after the stream had started, for example + when the account was out of tokens. The failure left the server action + after it had returned, so the framework removed its message and code. The + React SDK stream now sends it as an error chunk, and the app shows the + prompt that matches the code, such as the limits reached one. +- A timeout or protocol error from a user's MCP server while installing its + tools now surfaces as an upstream error with the MCP code, the way a tool + call already did, instead of a raw `McpError`. +- The Call GitHub API ability failed with a JSON syntax error, reported to + Sentry, whenever the endpoint answered with something other than JSON. A + plain text response, such as a job log, is now returned as text and capped + to its last 60000 characters. A binary response, such as the zip of a + workflow run's logs, is refused with a 400 that names the content type and + points the model at the job logs endpoint. +- An HTTP failure from a user's MCP server during a tool call, such as a + 403 or a 502 while the server's container restarts, now surfaces with the + matching error code instead of a generic error. Statuses the platform + treats as expected, such as 401, 403, 404 and 429, no longer reach Sentry. +- Creating or updating a dataset record whose text is only nonprintable + characters, such as a zero-width space, now returns a 400. The text passed + the whitespace check but normalization emptied it, so the vector store + refused the record and the request failed with a 500. +- Writing a file larger than about 768 KiB from a shell skillset action + failed with an opaque 413 from the sandbox service and was reported to + Sentry. The service now accepts write bodies up to 4 MiB, and the exec, + write and rw actions reject contents over 3 MB with a message that tells + the model to write the file in smaller parts. +- The `kimi-k2.5` model advertised a context window of 262114 tokens, a + digit transposition of the 262144 the gateway serves. The catalogue test + that compares configured limits against the live gateway now passes. +- An upstream API refusal inside a skillset ability, such as a GitHub 403, + is no longer reported to Sentry. The error left the handler as a + `FetchError` but was serialized with a generic code, because the bundle + holds several copies of the errors module and `instanceof` does not hold + across them; a `SystemError` now carries a brand the serializer recognises + from any copy. +- Compacting a conversation that contains tool activity now includes the + tool calls and results in the summary. The summary input carried only type + and text, so every activity message was reported as an unexpected state and + dropped before summarization. +- Finishing a dataset import whose sitemap or Notion integration was deleted + while the job ran no longer fails the job with a record-not-found error. +- Onboarding no longer fails at the last step with a byte-length error when + the organization name is written in a non-latin script. The value was + clipped by character count only, while the column is byte-bound. +- Minting or using a JWT secret whose value is not a PEM private key answers + 400 with a config error instead of a 500. +- Signing in with an email code no longer fails when the address is typed + with a capital letter, as phone keyboards do. The code form sent the address + as typed while the code was issued under the lowercased one, so every such + attempt was refused and consumed the code. +- The widget frame no longer fails to render in Firefox when the host page + blocks third-party storage. Opening the trace broadcast channel threw a + `SecurityError` inside a render effect, which the error boundary reported + as a page error on every load; both broadcast channel hooks now treat a + refused channel as unavailable. +- Initiating an email integration with a missing `email`, `subject` or + `text` answers 400. The fields were optional in the request schema but + required by the queue payload, so an empty body failed at enqueue time + with a 500. +- A widget message whose session token cannot be refreshed, such as an embed + of a deleted widget, no longer surfaces as an unhandled rejection. The + dispatched submit handler rethrew into nothing, so every attempt reached + Sentry even for expected refusals. +- A function handler that fails with an expected code, such as a client + function whose channel wait timed out, is no longer reported to Sentry. The + outcome for the model is unchanged. +- DeepSeek V4 Pro on the Vercel AI Gateway bills at the Alibaba backend rate + the gateway added, so no routing decision charges more than the model + configuration. +- Mistral Large and Mistral Small follow the current Mistral catalogue: both + carry the 262,144-token context Mistral now serves, and Mistral Small bills + at Mistral's current list price instead of the retired one. + +### Removed + +- Devstral 2 is no longer offered. Mistral retired it from the Vercel AI + Gateway, so the name now resolves to Mistral Large for existing bots. + ## [0.4.1] - 2026-09-14 ### Fixed diff --git a/docs/sdks.md b/docs/sdks.md index 4b653f7..ee20d1e 100644 --- a/docs/sdks.md +++ b/docs/sdks.md @@ -34,9 +34,12 @@ Sign in and open `/tokens` to create a token. The SDKs send it as a bearer token. The value shown at creation time is the only copy, so store it where the client will read it. -The Node.js SDK and CLI read `CHATBOTKIT_API_SECRET`, and the Terraform -provider reads `CHATBOTKIT_API_KEY`. The Python and Go SDKs take the token as -a constructor argument; pass it from whatever environment variable you prefer. +The CLI and the Terraform provider read the token from `CHATBOTKIT_API_TOKEN`, +or `CBK_API_TOKEN` for short. The older `CHATBOTKIT_API_SECRET` and +`CHATBOTKIT_API_KEY` names, and their `CBK_` forms, are still read. The +Node.js, Python and Go SDKs take the token as the `token` option; pass it from +whatever environment variable you prefer. The former option names (`secret`, +and `api_key` in Terraform) still work and are deprecated. ## Point an SDK at your deployment @@ -45,9 +48,11 @@ replaces it. The SDKs build request paths as `/api/v1/...` and only strip the `/api` prefix for the hosted API host, so the override is the bare origin of your deployment, with no `/api` suffix. -None of the SDKs read the base URL from the environment. Set it in code or -provider configuration, sourcing the value from your own configuration if the -same program has to run against both a local deployment and the hosted API. +The SDK libraries do not read the base URL from the environment; set it in +code. The CLI and the Terraform provider read the deployment origin from +`CHATBOTKIT_API_URL` (or `CBK_API_URL`), so one variable points both at the +same deployment. Plain `http` works for local use, and a path prefix on the +origin is preserved, so a deployment served under a sub-path is reachable. ### Node.js @@ -59,7 +64,7 @@ npm install @chatbotkit/sdk import { BotClient } from '@chatbotkit/sdk/bot/index.js' const bot = new BotClient({ - secret: process.env.CHATBOTKIT_API_SECRET, + token: process.env.CHATBOTKIT_API_TOKEN, baseUrl: 'http://127.0.0.1:8080', }) @@ -82,7 +87,7 @@ import os from chatbotkit import ChatBotKit cbk = ChatBotKit( - secret=os.environ["CHATBOTKIT_API_SECRET"], + token=os.environ["CHATBOTKIT_API_TOKEN"], base_url="http://127.0.0.1:8080", ) @@ -97,7 +102,7 @@ go get github.com/chatbotkit/go-sdk ```go client := sdk.New(sdk.Options{ - Secret: os.Getenv("CHATBOTKIT_API_SECRET"), + Token: os.Getenv("CHATBOTKIT_API_TOKEN"), BaseURL: "http://127.0.0.1:8080", }) @@ -107,12 +112,13 @@ bots, err := client.Bot.List(ctx, nil) ### Terraform The provider speaks GraphQL, so its `base_url` is the full GraphQL endpoint -rather than the origin. +rather than the origin. When `base_url` is not set, the provider derives the +endpoint from the origin in `CHATBOTKIT_API_URL`. ```terraform provider "chatbotkit" { - api_key = var.chatbotkit_api_key # or CHATBOTKIT_API_KEY - base_url = "http://127.0.0.1:8080/api/v1/graphql" + api_token = var.chatbotkit_api_token # or CHATBOTKIT_API_TOKEN + base_url = "http://127.0.0.1:8080/api/v1/graphql" } ``` @@ -142,7 +148,7 @@ describe a local instance. A plain HTTP call confirms the origin and token before involving an SDK: ```bash -curl -H "Authorization: Bearer $CHATBOTKIT_API_SECRET" \ +curl -H "Authorization: Bearer $CHATBOTKIT_API_TOKEN" \ http://127.0.0.1:8080/api/v1/bot/list ``` diff --git a/package.json b/package.json index 9e137ea..eb1d3b6 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "platform", - "version": "0.4.1", + "version": "0.5.0", "private": true, "license": "Apache-2.0", "packageManager": "pnpm@11.24.0", diff --git a/packages/errors/src/index.test.js b/packages/errors/src/index.test.js index eea8d3e..8613eaf 100644 --- a/packages/errors/src/index.test.js +++ b/packages/errors/src/index.test.js @@ -567,6 +567,48 @@ describe('errorToErrorResponse', () => { }) }) + it('should keep the code of a SystemError from another module copy', () => { + const { errorToErrorResponse } = require('./index') + + // @note a second bundled copy of this module has its own SystemError + // class, so only the shared brand identifies it + class ForeignSystemError extends Error { + constructor(message, code) { + super(message) + + this.code = code + + Object.defineProperty( + this, + Symbol.for('@chatbotkit-dev/errors/SystemError'), + { value: true, enumerable: false } + ) + } + } + + const result = errorToErrorResponse( + new ForeignSystemError('Upstream refused', 'NOT_AUTHORIZED') + ) + + expect(result).toEqual({ + code: 'NOT_AUTHORIZED', + message: 'Upstream refused', + }) + }) + + it('should not treat a plain error with a code as a SystemError', () => { + const { errorToErrorResponse } = require('./index') + + const error = Object.assign(new Error('socket hang up'), { + code: 'ECONNRESET', + }) + + expect(errorToErrorResponse(error)).toEqual({ + code: 'GENERIC_ERROR', + message: 'socket hang up', + }) + }) + it('should handle string error', () => { const { errorToErrorResponse } = require('./index') diff --git a/packages/errors/src/index.ts b/packages/errors/src/index.ts index a8d84f4..ebe2417 100644 --- a/packages/errors/src/index.ts +++ b/packages/errors/src/index.ts @@ -26,6 +26,13 @@ export const CONTENT_MODERATION_ERROR_CODE = 'CONTENT_MODERATION' // eslint-disable-next-line @typescript-eslint/no-explicit-any export type Thrown = any +// @note a bundle can carry more than one copy of this module (one per +// runtime), each with its own class, so instanceof alone cannot recognise a +// SystemError thrown by another copy - the brand is shared through the global +// symbol registry + +const SYSTEM_ERROR_BRAND = Symbol.for('@chatbotkit-dev/errors/SystemError') + export class SystemError extends Error { public code: string @@ -38,9 +45,28 @@ export class SystemError extends Error { this.code = code this.data = data + + Object.defineProperty(this, SYSTEM_ERROR_BRAND, { + value: true, + enumerable: false, + }) } } +/** + * Recognises a SystemError from any copy of this module. + */ +export function isSystemError(error: unknown): error is SystemError { + if (error instanceof SystemError) { + return true + } + + return ( + error instanceof Error && + (error as unknown as Record)[SYSTEM_ERROR_BRAND] === true + ) +} + /** * Represents an error that is composed of multiple errors. */ @@ -254,7 +280,7 @@ export function errorIn(error: Error, collection: string[]) { return collection.includes(error.name) || collection.includes(error.message) } -export function isKnownError(error: Error|string): boolean { +export function isKnownError(error: Error | string): boolean { if (typeof error === 'string') { error = new Error(error) } @@ -290,7 +316,7 @@ export function errorToErrorResponse(error: Thrown): { } switch (true) { - case error instanceof SystemError: { + case isSystemError(error): { return { code: error.code, message: error.message.toString() } } @@ -362,7 +388,7 @@ export function errorResponseToError( * */ export function errorToSystemError(error: Thrown, data?: unknown): SystemError { - if (error instanceof SystemError) { + if (isSystemError(error)) { return error } @@ -454,9 +480,7 @@ const MAX_CAUSE_DEPTH = 5 */ export function extractCauseChain( error: Thrown -): - | Array<{ name?: string; message?: string; code?: string }> - | undefined { +): Array<{ name?: string; message?: string; code?: string }> | undefined { /** @type {Array<{name?: string, message?: string, code?: string}>} */ const chain: { name: string | undefined @@ -483,8 +507,8 @@ export function extractCauseChain( typeof current.message === 'string' ? current.message : typeof current === 'string' - ? current - : undefined, + ? current + : undefined, code: current.code !== undefined && current.code !== null @@ -671,7 +695,10 @@ export async function captureException(e: Thrown): Promise { } } -export async function captureInputError(e: Thrown, data: unknown): Promise { +export async function captureInputError( + e: Thrown, + data: unknown +): Promise { // eslint-disable-next-line console.error(e) diff --git a/platform/.env.example b/platform/.env.example index 5d9a4e5..33b6c2a 100644 --- a/platform/.env.example +++ b/platform/.env.example @@ -194,6 +194,7 @@ VERCEL_MODELS_API_KEY= # MISTRAL_MODELS_API_KEY= # GROQ_MODELS_API_KEY= # DEEPSEEK_MODELS_API_KEY= +# TYPESAFE_MODELS_API_KEY= # # ANALYTICS diff --git a/platform/app/apps/(adhoc)/b4d0c8f2/components.jsx b/platform/app/apps/(adhoc)/b4d0c8f2/components.jsx index 6cb89ce..54de7ee 100644 --- a/platform/app/apps/(adhoc)/b4d0c8f2/components.jsx +++ b/platform/app/apps/(adhoc)/b4d0c8f2/components.jsx @@ -673,7 +673,7 @@ function RequestCodeBlock({ requestSchema, method, path }) { "import { ChatBotKit } from '@chatbotkit/sdk'", '', 'const cbk = new ChatBotKit({', - ' secret: process.env.CHATBOTKIT_API_KEY!,', + ' token: process.env.CHATBOTKIT_API_TOKEN!,', ` baseUrl: ${JSON.stringify(new URL(url).origin)},`, '})', '', @@ -709,7 +709,7 @@ function RequestCodeBlock({ requestSchema, method, path }) { ')', '', 'client := sdk.New(sdk.Options{', - ' Secret: os.Getenv("CHATBOTKIT_API_KEY"),', + ' Token: os.Getenv("CHATBOTKIT_API_TOKEN"),', ` BaseURL: ${JSON.stringify(new URL(url).origin)},`, '})', '', diff --git a/platform/app/apps/chat/server.tsx b/platform/app/apps/chat/server.tsx index 3a382e4..eab1653 100644 --- a/platform/app/apps/chat/server.tsx +++ b/platform/app/apps/chat/server.tsx @@ -38,7 +38,6 @@ import { runTasks } from '@/lib/job' import { getBaseLanguageModelTokenCount } from '@/lib/model.utils' import { nameToIcon } from '@/lib/name.icon' import { execPrompt } from '@/lib/prompt' -import { parse as parseStructStr } from '@/lib/structstr' import { NOT_AUTHORIZED_CODE, NOT_FOUND_CODE, @@ -48,10 +47,12 @@ import { } from '@/lib/response' import type { Session } from '@/lib/session.get' import { byteSlice, toCamelCase, toSlug } from '@/lib/string' +import { parse as parseStructStr } from '@/lib/structstr' import { Usage } from '@/lib/usage.model' import { stringify as stringifyYaml } from '@/lib/yaml' import type { ZodSchemaFor } from '@/lib/zod.schema' import { z } from '@/lib/zod.schema' + import type { InlineAbility } from '@/schemas/inlineExtensions' import autoAgentPrompt from '@/prompts/auto_agent_v1.yaml' @@ -626,7 +627,8 @@ const listInternalSources = appMethodHandler( name: edge.node.name, description: edge.node.description || '', // @note technically the description cannot be null instruction: edge.node.instruction, - linkedSecretId: edge.node.linkedSecret?.id || undefined, + linkedSecretId: + edge.node.linkedSecret?.id || undefined, }, ] }), diff --git a/platform/components/Auth.jsx b/platform/components/Auth.jsx index 9c8c506..a0175f0 100644 --- a/platform/components/Auth.jsx +++ b/platform/components/Auth.jsx @@ -295,10 +295,14 @@ export default function Auth({ toast.success('Signing you in...') + // @note NextAuth stores the identifier lowercased at issuance and rejects + // the callback on a strict compare, and codes are lowercase hex, so a + // capitalised address or code must be normalized before it is sent + const url = new URL('/api/auth/callback/email', window.location.origin) - url.searchParams.append('email', formRef.current.email.value) - url.searchParams.append('token', formRef.current.token.value) + url.searchParams.append('email', email.normalize('NFKC').toLowerCase()) + url.searchParams.append('token', token.toLowerCase()) url.searchParams.append('callbackUrl', nextUrl) // @note email codes are single-use too, so skip client routing's diff --git a/platform/components/Auth.utest.js b/platform/components/Auth.utest.js index c957dcc..45100ff 100644 --- a/platform/components/Auth.utest.js +++ b/platform/components/Auth.utest.js @@ -276,6 +276,39 @@ describe('Auth sign-in callbacks', () => { } ) + it('lowercases a capitalised address and code before verifying', async () => { + const signin = jest.fn().mockResolvedValue({ ok: true }) + const push = jest.fn() + + require('@/hooks/useSignin').mockReturnValue({ signin }) + require('@/hooks/useRouter').mockReturnValue({ query: {}, push }) + + const { container, findByLabelText } = render() + const input = container.querySelector('input[name="email"]') + + Object.defineProperties(input.form, { + email: { get: () => input.form.elements.namedItem('email') }, + token: { get: () => input.form.elements.namedItem('token') }, + }) + + // @note a phone keyboard capitalises the first letter of the address; + // NextAuth issued the code under the lowercased identifier + fireEvent.change(input, { target: { value: 'Luma@Example.com' } }) + fireEvent.keyDown(input, { key: 'Enter' }) + + const pin = await findByLabelText('PIN field 1 of 6') + + fireEvent.change(pin, { target: { value: 'BD6E92' } }) + + await waitFor(() => expect(window.location.assign).toHaveBeenCalledTimes(1)) + + const callback = new URL(window.location.assign.mock.calls[0][0]) + + expect(callback.pathname).toBe('/api/auth/callback/email') + expect(callback.searchParams.get('email')).toBe('luma@example.com') + expect(callback.searchParams.get('token')).toBe('bd6e92') + }) + it('normalizes the email and uses a fresh token for each attempt', async () => { const signin = jest.fn().mockResolvedValue({ ok: true }) const push = jest.fn() diff --git a/platform/config/models.ts b/platform/config/models.ts index da91448..3703024 100644 --- a/platform/config/models.ts +++ b/platform/config/models.ts @@ -1,4 +1,5 @@ import type { + AnyDecisionModel, AnyImageModel, AnyLanguageModel, AnyRerankModel, @@ -56,6 +57,8 @@ const WITH_GROQ_MODELS = IS_BROWSER || !!process.env.GROQ_MODELS_API_KEY const WITH_DEEPSEEK_MODELS = IS_BROWSER || !!process.env.DEEPSEEK_MODELS_API_KEY +const WITH_TYPESAFE_MODELS = IS_BROWSER || !!process.env.TYPESAFE_MODELS_API_KEY + // @note these aliases carry no constraint of their own; they exist so a // catalogue's keys read as what they are. They came across from the JSDoc // typedefs this file used before it was TypeScript. @@ -4176,48 +4179,6 @@ export const vercelLanguageModels: Record< // mistral - 'devstral-2': { - description: `Devstral 2 is Mistral AI's coding-focused model for agentic software engineering workflows.`, - - provider: 'vercel', - - providerModel: 'mistral/devstral-2', - - family: 'devstral', - - features: ['chat', 'functions'], - - region: 'us', - availableRegions: ['us'], - - maxTokens: 256_000, - maxInputTokens: 256_000 - 64_000, - maxOutputTokens: 64_000, - - pricing: { - tokenRatio: 0.1111, - inputTokenRatio: 0.0286, - outputTokenRatio: 0.1111, - inputPrice: 0.4, - outputPrice: 2, - }, - - interactionMaxMessages: DEFAULT_INTERACTION_MAX_MESSAGES, - - thresholdStrategy: 'truncate', - - visible: true, - deprecated: false, - - temperature: DEFAULT_TEMPERATURE, - - frequencyPenalty: 0, - presencePenalty: 0, - - tags: [], - - addedDate: '2026-06-14', - }, 'mistral-large-latest': { description: `Top-tier reasoning for high-complexity tasks. The most powerful model of the Mistral AI family.`, @@ -4233,8 +4194,8 @@ export const vercelLanguageModels: Record< region: 'us', availableRegions: ['us'], - maxTokens: 256_000, - maxInputTokens: 256_000 - 64_000, + maxTokens: 262_144, + maxInputTokens: 262_144 - 64_000, maxOutputTokens: 64_000, pricing: { @@ -4276,16 +4237,16 @@ export const vercelLanguageModels: Record< region: 'us', availableRegions: ['us'], - maxTokens: 32_000, - maxInputTokens: 28_000, + maxTokens: 262_144, + maxInputTokens: 262_144 - 4_000, maxOutputTokens: 4_000, pricing: { - tokenRatio: 0.0167, - inputTokenRatio: 0.0071, - outputTokenRatio: 0.0167, - inputPrice: 0.1, - outputPrice: 0.3, + tokenRatio: 0.0333, + inputTokenRatio: 0.0107, + outputTokenRatio: 0.0333, + inputPrice: 0.15, + outputPrice: 0.6, }, interactionMaxMessages: DEFAULT_INTERACTION_MAX_MESSAGES, @@ -4334,11 +4295,11 @@ export const vercelLanguageModels: Record< maxOutputTokens: Math.ceil(1_000_000 * MAX_OUTPUT_TOKENS_RATIO), pricing: { - tokenRatio: 0.2444, - inputTokenRatio: 0.15, - outputTokenRatio: 0.2444, - inputPrice: 2.1, - outputPrice: 4.4, + tokenRatio: 0.2667, + inputTokenRatio: 0.1714, + outputTokenRatio: 0.2667, + inputPrice: 2.4, + outputPrice: 4.8, }, interactionMaxMessages: DEFAULT_INTERACTION_MAX_MESSAGES, @@ -5789,8 +5750,8 @@ export const vercelLanguageModels: Record< region: 'us', availableRegions: ['us'], - maxTokens: 262_114, - maxInputTokens: 262_114 - 65_535, + maxTokens: 262_144, + maxInputTokens: 262_144 - 65_535, maxOutputTokens: 65_535, pricing: { @@ -6432,46 +6393,6 @@ export const mistralLanguageModels: Record< MistralLanguageModel > = WITH_MISTRAL_MODELS ? { - 'devstral-2': { - description: `Devstral 2 is Mistral AI's coding-focused model for agentic software engineering workflows.`, - - provider: 'mistral', - - family: 'devstral', - - features: ['chat', 'functions'], - - region: 'us', - availableRegions: ['us'], - - maxTokens: 256_000, - maxInputTokens: 256_000 - 64_000, - maxOutputTokens: 64_000, - - pricing: { - tokenRatio: 0.1111, - inputTokenRatio: 0.0286, - outputTokenRatio: 0.1111, - inputPrice: 0.4, - outputPrice: 2, - }, - - interactionMaxMessages: DEFAULT_INTERACTION_MAX_MESSAGES, - - thresholdStrategy: 'truncate', - - visible: true, - deprecated: false, - - temperature: DEFAULT_TEMPERATURE, - - frequencyPenalty: 0, - presencePenalty: 0, - - tags: [], - - addedDate: '2026-06-14', - }, 'mistral-large-latest': { description: `Top-tier reasoning for high-complexity tasks. The most powerful model of the Mistral AI family.`, @@ -6485,8 +6406,8 @@ export const mistralLanguageModels: Record< region: 'us', availableRegions: ['us'], - maxTokens: 256_000, - maxInputTokens: 256_000 - 64_000, + maxTokens: 262_144, + maxInputTokens: 262_144 - 64_000, maxOutputTokens: 64_000, pricing: { @@ -6526,16 +6447,16 @@ export const mistralLanguageModels: Record< region: 'us', availableRegions: ['us'], - maxTokens: 32_000, - maxInputTokens: 28_000, + maxTokens: 262_144, + maxInputTokens: 262_144 - 4_000, maxOutputTokens: 4_000, pricing: { - tokenRatio: 0.0167, - inputTokenRatio: 0.0071, - outputTokenRatio: 0.0167, - inputPrice: 0.1, - outputPrice: 0.3, + tokenRatio: 0.0333, + inputTokenRatio: 0.0107, + outputTokenRatio: 0.0333, + inputPrice: 0.15, + outputPrice: 0.6, }, interactionMaxMessages: DEFAULT_INTERACTION_MAX_MESSAGES, @@ -7386,6 +7307,9 @@ const deprecatedLanguageModelProxyMapping: Record = { 'claude-v2': 'claude-3.5-sonnet', 'claude-instant-v1': 'claude-3.5-haiku', + // Mistral + 'devstral-2': 'mistral-large-latest', + // Deepseek 'deepseek-chat': 'deepseek-v3.2', @@ -8449,6 +8373,95 @@ export const visibleRerankModels: Record = // --- // --- +const jevDecisionModel = { + description: `Jev is TypeSafe AI's System One decision model. It answers typed questions (boolean, choice, score) about a shared state and returns probabilities, suited to classification, routing, rubric-based assessment and automated verification.`, + + family: 'jev', + + features: [], + + pricing: { + // @note input-only; every provider bills jev at $0.042 per 1M input tokens + // and nothing for output, so the default ratio carries the input ratio. + tokenRatio: 0.003, + inputTokenRatio: 0.003, + outputTokenRatio: 0, + inputPrice: 0.042, + outputPrice: 0, + }, + + region: 'us', + availableRegions: ['us'], + + visible: true, + deprecated: false, + + tags: [], + + addedDate: '2026-09-18', +} satisfies Omit + +export const openrouterDecisionModels: Record = + WITH_OPENROUTER_MODELS + ? { + jev: { + ...jevDecisionModel, + + provider: 'openrouter', + + providerModel: '~typesafe/jev-latest', + }, + } + : {} + +export const vercelDecisionModels: Record = + WITH_VERCEL_MODELS + ? { + jev: { + ...jevDecisionModel, + + provider: 'vercel', + + providerModel: 'typesafe-ai/jev', + }, + } + : {} + +export const typesafeDecisionModels: Record = + WITH_TYPESAFE_MODELS + ? { + jev: { + ...jevDecisionModel, + + provider: 'typesafe', + + providerModel: 'jev-latest', + }, + } + : {} + +// @note one name, served by whichever provider is configured; a later spread +// wins, so the official TypeSafe API takes precedence over the gateways. +export const decisionModels: Record = { + ...openrouterDecisionModels, + ...vercelDecisionModels, + ...typesafeDecisionModels, +} + +export const defaultDecisionModel: string = pickDefaultModel( + 'jev', + decisionModels +) + +export const visibleDecisionModels: Record = + Object.fromEntries( + Object.entries(decisionModels).filter(([, { visible }]) => visible) + ) + +// --- +// --- +// --- + export const speechToTextModels: Record = { 'gpt-4o-transcribe': { description: `GPT-4o Transcribe is OpenAI's speech-to-text model for audio transcription.`, @@ -8544,6 +8557,10 @@ const models = { defaultRerankModel, visibleRerankModels, + decisionModels, + defaultDecisionModel, + visibleDecisionModels, + speechToTextModels, defaultSpeechToTextModel, visibleSpeechToTextModels, diff --git a/platform/examples/catalogue/projects.yaml b/platform/examples/catalogue/projects.yaml index e54c4a1..062081d 100644 --- a/platform/examples/catalogue/projects.yaml +++ b/platform/examples/catalogue/projects.yaml @@ -374,7 +374,7 @@ datasets, conversations, and settings. Isolation comes from the provider's run_as attribute. You hold one - parent User token (CHATBOTKIT_API_KEY) and configure one provider alias + parent User token (CHATBOTKIT_API_TOKEN) and configure one provider alias per customer, each with run_as set to that customer's child User ID, which sends the X-RunAs-UserId header. There are no per-customer tokens - account IDs are not secret. This is the standard Terraform multi-account pattern, @@ -421,7 +421,7 @@ might be a support bot, another a research agent with a sandbox. A single shared root composes them all and deploys every customer in one apply, with one shared state. Isolation comes from the provider's run_as attribute: you - hold one parent User token (CHATBOTKIT_API_KEY) and each module is wired to + hold one parent User token (CHATBOTKIT_API_TOKEN) and each module is wired to its customer's child User through a provider alias whose run_as sends the X-RunAs-UserId header. No per-customer tokens, no for_each. diff --git a/platform/hooks/useAvailableModels.ts b/platform/hooks/useAvailableModels.ts index c54819b..6be6bc6 100644 --- a/platform/hooks/useAvailableModels.ts +++ b/platform/hooks/useAvailableModels.ts @@ -2,7 +2,12 @@ import { useEffect, useState } from 'react' import fetch from '@/lib/fetch' -export type ModelType = 'language' | 'image' | 'video' | 'rerank' +export type ModelType = + | 'language' + | 'image' + | 'video' + | 'rerank' + | 'decision' type AvailableModels = { ids: string[] diff --git a/platform/hooks/useBroadcastChannel.tsx b/platform/hooks/useBroadcastChannel.tsx index 29c7df6..9310230 100644 --- a/platform/hooks/useBroadcastChannel.tsx +++ b/platform/hooks/useBroadcastChannel.tsx @@ -6,14 +6,25 @@ export default function useBroadcastChannel( const [channel, setChannel] = useState(null) useEffect(() => { - if (typeof window !== 'undefined' && window.BroadcastChannel) { - const bc = new BroadcastChannel(channelName) + if (typeof window === 'undefined' || !window.BroadcastChannel) { + return + } + + let bc: BroadcastChannel + + try { + bc = new BroadcastChannel(channelName) + } catch { + // @note Firefox throws SecurityError when storage is blocked, as in a + // third-party iframe under strict tracking protection + + return + } - setChannel(bc) + setChannel(bc) - return () => { - bc.close() - } + return () => { + bc.close() } }, [channelName]) diff --git a/platform/hooks/useBroadcastChannel.utest.js b/platform/hooks/useBroadcastChannel.utest.js index d5a16ae..7e3ad8b 100644 --- a/platform/hooks/useBroadcastChannel.utest.js +++ b/platform/hooks/useBroadcastChannel.utest.js @@ -48,6 +48,21 @@ describe('useBroadcastChannel', () => { expect(mockBroadcastChannel).toHaveBeenCalledWith('custom-name') expect(result.current.name).toBe('custom-name') }) + + it('should stay null when the browser refuses the channel', () => { + global.BroadcastChannel = jest.fn(() => { + throw new DOMException('The operation is insecure.', 'SecurityError') + }) + + const { result, unmount } = renderHook(() => + useBroadcastChannel('test-channel') + ) + + expect(global.BroadcastChannel).toHaveBeenCalledWith('test-channel') + expect(result.current).toBeNull() + + expect(() => unmount()).not.toThrow() + }) }) describe('channel lifecycle', () => { diff --git a/platform/hooks/useBroadcastChannelState.ts b/platform/hooks/useBroadcastChannelState.ts index 515cdf6..fd7fc44 100644 --- a/platform/hooks/useBroadcastChannelState.ts +++ b/platform/hooks/useBroadcastChannelState.ts @@ -16,7 +16,20 @@ export default function useBroadcastChannelState( const valueRef = useRef(value) useEffect(() => { - const channel = new BroadcastChannel(`${uniquePrefix}-${channelName}`) + if (typeof BroadcastChannel === 'undefined') { + return + } + + let channel: BroadcastChannel + + try { + channel = new BroadcastChannel(`${uniquePrefix}-${channelName}`) + } catch { + // @note Firefox throws SecurityError when storage is blocked, as in a + // third-party iframe under strict tracking protection + + return + } setChannel(channel) diff --git a/platform/hooks/useBroadcastChannelState.utest.js b/platform/hooks/useBroadcastChannelState.utest.js index fcdd482..f5a8be9 100644 --- a/platform/hooks/useBroadcastChannelState.utest.js +++ b/platform/hooks/useBroadcastChannelState.utest.js @@ -335,6 +335,39 @@ describe('useBroadcastChannelState', () => { }) }) + describe('refused channel', () => { + let originalBroadcastChannel + + beforeEach(() => { + originalBroadcastChannel = global.BroadcastChannel + + global.BroadcastChannel = jest.fn(() => { + throw new DOMException('The operation is insecure.', 'SecurityError') + }) + }) + + afterEach(() => { + global.BroadcastChannel = originalBroadcastChannel + }) + + it('should keep the initial value and ignore sends', () => { + const { result, unmount } = renderHook(() => + useBroadcastChannelState('test-channel', 'initial') + ) + + expect(global.BroadcastChannel).toHaveBeenCalledTimes(1) + expect(result.current[0]).toBe('initial') + + act(() => { + result.current[1]('updated') + }) + + expect(result.current[0]).toBe('initial') + + expect(() => unmount()).not.toThrow() + }) + }) + describe('channel isolation', () => { it('should isolate messages between different channel names', async () => { const { result: result1 } = renderHook(() => diff --git a/platform/lib/action.exec.shell.ts b/platform/lib/action.exec.shell.ts index d0cdc79..c3441f6 100644 --- a/platform/lib/action.exec.shell.ts +++ b/platform/lib/action.exec.shell.ts @@ -38,7 +38,7 @@ import { exec, readFile, runCode, writeFile } from '@/lib/sandbox.shell' import { getTemporaryUserToken } from '@/lib/session.temp' import { getActiveSkillsetAbilities } from '@/lib/skillset.abilities' import { canUseSkillset } from '@/lib/skillset.access' -import { toKebabCase } from '@/lib/string' +import { byteLength, toKebabCase } from '@/lib/string' import { Usage } from '@/lib/usage.model' import { fastGetUserById } from '@/lib/user.get' import { revealUserPlan } from '@/lib/user.plan' @@ -174,6 +174,20 @@ function getShellExecutionSession(options: ActionOptions): string { // @see data/abilities/catalogue/cbk.shell.ts for ability definitions related // to these schemas +// @note the sandbox service caps a write-file request body at 4 MiB and the +// contents travel base64-encoded (4/3 inflation), so 3 MB of contents is the +// most that fits; the model gets a message it can act on instead of a 413 +export const MAX_FILE_CONTENTS_BYTES = 3_000_000 + +function boundedContents(description: string) { + return z + .string() + .refine((value) => byteLength(value) <= MAX_FILE_CONTENTS_BYTES, { + message: `contents must be at most ${MAX_FILE_CONTENTS_BYTES} bytes; write the file in smaller parts`, + }) + .describe(description) +} + /** * Shell exec schema defines the parameters for executing shell commands. */ @@ -183,7 +197,10 @@ export const shellExecSchema = z.object({ .array( z.object({ path: z.string().min(1).describe('The file path'), - contents: z.string().min(1).describe('The file contents'), + contents: boundedContents('The file contents').refine( + (value) => value.length > 0, + { message: 'contents must not be empty' } + ), }) ) .optional() @@ -288,7 +305,7 @@ export type ShellReadSchema = z.infer */ export const shellWriteSchema = z.object({ file: z.string().min(1).describe('The file path to write'), - contents: z.string().describe('The contents to write to the file'), + contents: boundedContents('The contents to write to the file'), startLine: z.coerce .number() .int() @@ -316,10 +333,9 @@ export type ShellWriteSchema = z.infer export const shellRwSchema = z.object({ file: z.string().min(1).describe('The file path to read from or write to'), mode: z.enum(['read', 'write']).describe('The operation mode: read or write'), - contents: z - .string() - .optional() - .describe('The contents to write to the file (required for write mode)'), + contents: boundedContents( + 'The contents to write to the file (required for write mode)' + ).optional(), startLine: z.coerce .number() .int() diff --git a/platform/lib/action.exec.shell.utest.js b/platform/lib/action.exec.shell.utest.js index 5f50aea..5b6385e 100644 --- a/platform/lib/action.exec.shell.utest.js +++ b/platform/lib/action.exec.shell.utest.js @@ -1,4 +1,5 @@ import { + MAX_FILE_CONTENTS_BYTES, doShellExec, doShellSkillsetInstall, doShellScript, @@ -213,6 +214,29 @@ describe('action.exec.shell', () => { }) }) + it('should reject a file whose contents exceed the byte cap', async () => { + const { exec } = await import('@/lib/sandbox.shell') + + await expect( + doShellExec({ + session: 'test-namespace', + input: 'cat big.txt', + params: { + cmd: 'cat big.txt', + files: [ + { + path: 'big.txt', + contents: 'a'.repeat(MAX_FILE_CONTENTS_BYTES + 1), + }, + ], + }, + options: { userId: 'user-123' }, + }) + ).rejects.toThrow(/at most \d+ bytes/) + + expect(exec).not.toHaveBeenCalled() + }) + it('should throw error when cmd is missing', async () => { await expect( doShellExec({ @@ -991,6 +1015,38 @@ describe('action.exec.shell', () => { ).rejects.toThrow() }) + it('should reject contents that exceed the byte cap', async () => { + const { writeFile } = await import('@/lib/sandbox.shell') + + await expect( + doShellWrite({ + session: 'test-namespace', + input: '', + params: { + file: 'big.txt', + contents: 'a'.repeat(MAX_FILE_CONTENTS_BYTES + 1), + }, + options: { userId: 'user-123' }, + }) + ).rejects.toThrow(/at most \d+ bytes/) + + expect(writeFile).not.toHaveBeenCalled() + + // @note the cap counts bytes, so a multi-byte string under the character + // limit can still exceed it + await expect( + doShellWrite({ + session: 'test-namespace', + input: '', + params: { + file: 'big.txt', + contents: 'é'.repeat(MAX_FILE_CONTENTS_BYTES / 2 + 1), + }, + options: { userId: 'user-123' }, + }) + ).rejects.toThrow(/at most \d+ bytes/) + }) + it('should throw error when contents parameter is missing', async () => { await expect( doShellWrite({ diff --git a/platform/lib/blueprint.terraform.js b/platform/lib/blueprint.terraform.js index 6e585af..056a3e1 100644 --- a/platform/lib/blueprint.terraform.js +++ b/platform/lib/blueprint.terraform.js @@ -539,7 +539,7 @@ export function blueprintToTerraform(blueprint) { } provider "chatbotkit" { - # api_key = "..." # Or set CHATBOTKIT_API_KEY env var + # api_token = "..." # Or set CHATBOTKIT_API_TOKEN env var }`) // Convert each resource type in order diff --git a/platform/lib/conversation.engine.js b/platform/lib/conversation.engine.js index 0625c97..dda2709 100644 --- a/platform/lib/conversation.engine.js +++ b/platform/lib/conversation.engine.js @@ -154,6 +154,7 @@ import { detectPiiEntities, getSafeTextAndEntities } from '@/lib/pii' import { fallbackOnFailure, neitherTrue, wait } from '@/lib/promise' import { computePrompt } from '@/lib/prompt' import { + isUnknownError, throwBadRequest, throwConflict, throwNoSubscription, @@ -5114,8 +5115,15 @@ ${getCombinedDescription(inlineSkillset.description)}` let success = false try { + // @note meta must travel along - an activity message carries its tool + // call and result there and has no text of its own + const { summary, usage: compactUsage } = await compactMessages( - messagesToSummarize.map(({ type, text }) => ({ type, text })), + messagesToSummarize.map(({ type, text, meta }) => ({ + type, + text, + meta, + })), { user: { id: this.userId }, usageReferences: this.usageReferences, @@ -5421,7 +5429,12 @@ export class BasicFunctionEngine extends CoreEngine { { newMessages: [] } ) } catch (e) { - await captureException(e) + // @note an expected code, such as a channel wait that timed + // out, is normal operation and stays out of Sentry + + if (isUnknownError(e)) { + await captureException(e) + } // @note we are deliberately hiding the error from the user // because this is an internal issue diff --git a/platform/lib/conversation.engine.utest.js b/platform/lib/conversation.engine.utest.js index 08ecd3f..9022e6c 100644 --- a/platform/lib/conversation.engine.utest.js +++ b/platform/lib/conversation.engine.utest.js @@ -2606,6 +2606,56 @@ The weather in London is rainy. }) }) + it('reports only unexpected function handler errors to Sentry', async () => { + const observability = (await import('@chatbotkit-dev/observability')) + .default + + const captureSpy = jest + .spyOn(observability, 'captureException') + .mockResolvedValue(undefined) + + const expected = new SystemError( + 'No message received: channel wait was aborted (likely timeout)', + 'no_message_received_aborted' + ) + + const unexpected = new Error('handler crashed') + + mockChatResponses([ + { finishReason: 'toolCalls', toolCalls: [makeToolCall('_expected')] }, + { finishReason: 'toolCalls', toolCalls: [makeToolCall('_unexpected')] }, + { finishReason: 'stop', completion: 'Done' }, + ]) + + const { engine } = makeEngine({ + maxCalls: 10, + maxCycles: 10, + internalFunctions: [ + { + name: '_expected', + description: 'Fails with an expected code', + parameters: {}, + handler: jest.fn().mockRejectedValue(expected), + }, + { + name: '_unexpected', + description: 'Fails unexpectedly', + parameters: {}, + handler: jest.fn().mockRejectedValue(unexpected), + }, + ], + }) + + const response = await engine.complete() + + expect(response.reason).toBe('stop') + expect(captureSpy.mock.calls.map(([error]) => error)).toEqual([ + unexpected, + ]) + + captureSpy.mockRestore() + }) + it('limits tool-call recursion with maxIterations', async () => { const handler = jest.fn().mockResolvedValue({ ok: true }) @@ -6127,6 +6177,63 @@ describe('CoreEngine.addMessages', () => { }) }) +describe('CoreEngine.definitelyCompact', () => { + it('summarizes with activity meta and without backstory and checkpoints', async () => { + extractData.mockResolvedValueOnce({ + data: { summary: 'Summary' }, + usage: { token: 3 }, + }) + + const engine = new CoreEngine({ + userId: '123', + model: 'gpt-4o', + backstory: 'Backstory', + messages: [ + { type: MessageType.checkpoint, text: 'Old summary' }, + { type: MessageType.user, text: 'Hello' }, + { + type: MessageType.activity, + text: '{}', + meta: { activity: { type: 'request', function: { name: 'fn' } } }, + }, + { + type: MessageType.activity, + text: '{}', + meta: { activity: { type: 'response', function: { name: 'fn' } } }, + }, + { type: MessageType.bot, text: 'Hi' }, + ], + }) + + const { message, usage } = await engine.definitelyCompact() + + expect(extractData).toHaveBeenCalledWith( + [ + { type: MessageType.user, text: 'Hello', meta: undefined }, + { + type: MessageType.activity, + text: '{}', + meta: { activity: { type: 'request', function: { name: 'fn' } } }, + }, + { + type: MessageType.activity, + text: '{}', + meta: { activity: { type: 'response', function: { name: 'fn' } } }, + }, + { type: MessageType.bot, text: 'Hi', meta: undefined }, + ], + expect.anything(), + expect.anything() + ) + + expect(message).toMatchObject({ + type: MessageType.checkpoint, + text: 'Summary', + }) + expect(usage).toEqual({ token: 3 }) + }) +}) + describe('CoreEngine.stream', () => { // Helper to create a controlled async iterable from a list of items async function* makeStream(items) { diff --git a/platform/lib/debounce.utest.js b/platform/lib/debounce.utest.js index 4509e16..f7abfa0 100644 --- a/platform/lib/debounce.utest.js +++ b/platform/lib/debounce.utest.js @@ -5,6 +5,16 @@ function sleep(ms) { } describe('createDebouncedAction (leading throttle)', () => { + // @note the window is measured with Date.now(), so a real clock lets a + // loaded CI runner push the second trigger past it; fake timers pin it + beforeEach(() => { + jest.useFakeTimers() + }) + + afterEach(() => { + jest.useRealTimers() + }) + test('immediate first trigger', async () => { let count = 0 @@ -48,7 +58,7 @@ describe('createDebouncedAction (leading throttle)', () => { }) await d.trigger() - await sleep(45) + await jest.advanceTimersByTimeAsync(45) await d.trigger() expect(count).toBe(2) @@ -105,10 +115,19 @@ describe('createDebouncedAction (leading throttle)', () => { intervalMs: 50, }) - await d.trigger() - await d.trigger() - await sleep(55) - await d.trigger() + // the action sleeps on the fake clock, so each trigger is driven by hand + async function triggerAndSettle() { + const pending = d.trigger() + + await jest.advanceTimersByTimeAsync(10) + + await pending + } + + await triggerAndSettle() + await triggerAndSettle() + await jest.advanceTimersByTimeAsync(55) + await triggerAndSettle() expect(events).toEqual(['done', 'done']) }) @@ -136,7 +155,7 @@ describe('createDebouncedAction (leading throttle)', () => { await d1.trigger() await d2.trigger() // suppressed - await sleep(35) + await jest.advanceTimersByTimeAsync(35) await d1.trigger() await d2.trigger() diff --git a/platform/lib/decision.core.ts b/platform/lib/decision.core.ts new file mode 100644 index 0000000..b027539 --- /dev/null +++ b/platform/lib/decision.core.ts @@ -0,0 +1,94 @@ +import { assertUnreachable } from '@chatbotkit-dev/typescript-utils/unreachable' + +import { decisionModels, defaultDecisionModel } from '@/config/models' + +import debug from '@/lib/debug' +import type { + CreateDecisionResult, + DecisionInput, + DecisionQuestion, +} from '@/lib/decision.types' +import { decide as decideOpenRouter } from '@/lib/model.provider.openrouter' +import { decide as decideTypeSafe } from '@/lib/model.provider.typesafe' +import { decide as decideVercel } from '@/lib/model.provider.vercel' +import { parseAndRevealDecisionModel } from '@/lib/model.utils' +import { throwBadRequest } from '@/lib/response' + +interface CreateDecisionOptions { + model?: string + signal?: AbortSignal +} + +/** + * Answers typed questions about a state using the configured decision model. + * + * @note the usage is returned (not recorded here) so the caller can record it + * against the usage log, consistent with the image/video/rerank modules. + */ +export async function createDecision( + state: DecisionInput, + questions: Record, + options?: CreateDecisionOptions +): Promise { + debug(`creating decision`, { + questionCount: Object.keys(questions).length, + options, + }) + + const { model = defaultDecisionModel, signal } = options || {} + + // @note a deployment serves decision models only when a provider key is set; + // without one the catalogue is empty and any name would pass validation + + if (!Object.keys(decisionModels).length) { + throwBadRequest('No decision model is configured on this deployment') + } + + const { name, config } = parseAndRevealDecisionModel(model) + + const provider = config.provider + + let decide: typeof decideVercel + + switch (provider) { + case 'typesafe': { + decide = decideTypeSafe + + break + } + + case 'vercel': { + decide = decideVercel + + break + } + + case 'openrouter': { + decide = decideOpenRouter + + break + } + + default: { + assertUnreachable(provider) + } + } + + // @note transient failures are retried by each provider's fetch instance; + // retrying again here would multiply the attempts against a failing provider + + const { answers, usage } = await decide({ + model: config.providerModel || name, + modelOptions: config.providerOptions, + + state, + questions, + + signal, + }) + + return { + answers, + usage: { ...usage, model: name }, + } +} diff --git a/platform/lib/decision.core.unconfigured.utest.js b/platform/lib/decision.core.unconfigured.utest.js new file mode 100644 index 0000000..c02022e --- /dev/null +++ b/platform/lib/decision.core.unconfigured.utest.js @@ -0,0 +1,33 @@ +import { createDecision } from '@/lib/decision.core' +import { decide as decideTypeSafe } from '@/lib/model.provider.typesafe' + +jest.mock('@/config/models', () => ({ + ...jest.requireActual('@/config/models'), + __esModule: true, + decisionModels: {}, + defaultDecisionModel: 'jev', +})) + +jest.mock('@/lib/model.provider.typesafe', () => ({ decide: jest.fn() })) +jest.mock('@/lib/model.provider.vercel', () => ({ decide: jest.fn() })) +jest.mock('@/lib/model.provider.openrouter', () => ({ decide: jest.fn() })) + +const questions = { q: { type: 'boolean', instructions: 'Is it urgent?' } } + +// @note a deployment serves decision models only when a provider key is set, +// so an empty catalogue is the normal state of a fresh install +describe('createDecision on a deployment that serves no decision model', () => { + it.each([[undefined], ['jev'], ['anything']])( + 'answers a bad request rather than an internal error for model %s', + async (model) => { + await expect( + createDecision('state', questions, { model }) + ).rejects.toMatchObject({ + message: 'No decision model is configured on this deployment', + code: 'BAD_REQUEST', + }) + + expect(decideTypeSafe).not.toHaveBeenCalled() + } + ) +}) diff --git a/platform/lib/decision.core.utest.js b/platform/lib/decision.core.utest.js new file mode 100644 index 0000000..cf72ab3 --- /dev/null +++ b/platform/lib/decision.core.utest.js @@ -0,0 +1,145 @@ +import { createDecision } from '@/lib/decision.core' +import { decide as decideOpenRouter } from '@/lib/model.provider.openrouter' +import { decide as decideTypeSafe } from '@/lib/model.provider.typesafe' +import { decide as decideVercel } from '@/lib/model.provider.vercel' + +jest.mock('@/config/models', () => { + const actual = jest.requireActual('@/config/models') + + const pricing = { tokenRatio: 0.003 } + + return { + ...actual, + __esModule: true, + decisionModels: { + jev: { provider: 'typesafe', providerModel: 'jev-latest', pricing }, + 'jev-vercel': { + provider: 'vercel', + providerModel: 'typesafe-ai/jev', + providerOptions: { gateway: { only: ['typesafe-ai'] } }, + pricing, + }, + 'jev-openrouter': { + provider: 'openrouter', + providerModel: '~typesafe/jev-latest', + pricing, + }, + }, + defaultDecisionModel: 'jev', + } +}) + +jest.mock('@/lib/model.provider.typesafe', () => ({ decide: jest.fn() })) +jest.mock('@/lib/model.provider.vercel', () => ({ decide: jest.fn() })) +jest.mock('@/lib/model.provider.openrouter', () => ({ decide: jest.fn() })) + +const questions = { + refunded: { type: 'boolean', instructions: 'Was a refund issued?' }, +} + +const answers = { refunded: { type: 'boolean', probability: 0.99 } } + +const result = { + answers, + usage: { model: 'provider-side-id', inputTokens: 283, outputTokens: 21 }, +} + +describe('createDecision', () => { + beforeEach(() => { + jest.clearAllMocks() + + // @ts-ignore + decideTypeSafe.mockResolvedValue(result) + // @ts-ignore + decideVercel.mockResolvedValue(result) + // @ts-ignore + decideOpenRouter.mockResolvedValue(result) + }) + + it.each([ + ['jev', decideTypeSafe, 'jev-latest', undefined], + [ + 'jev-vercel', + decideVercel, + 'typesafe-ai/jev', + { gateway: { only: ['typesafe-ai'] } }, + ], + ['jev-openrouter', decideOpenRouter, '~typesafe/jev-latest', undefined], + ])( + 'dispatches %s to its provider with the provider model id', + async (model, decide, providerModel, modelOptions) => { + await createDecision('a refund was issued', questions, { model }) + + expect(decide).toHaveBeenCalledTimes(1) + expect(decide).toHaveBeenCalledWith( + expect.objectContaining({ + model: providerModel, + modelOptions, + state: 'a refund was issued', + questions, + }) + ) + + for (const other of [ + decideTypeSafe, + decideVercel, + decideOpenRouter, + ]) { + if (other !== decide) { + expect(other).not.toHaveBeenCalled() + } + } + } + ) + + it('falls back to the default decision model', async () => { + await createDecision('state', questions) + + expect(decideTypeSafe).toHaveBeenCalledWith( + expect.objectContaining({ model: 'jev-latest' }) + ) + }) + + it('returns the answers and stamps the platform model name onto usage', async () => { + expect( + await createDecision('state', questions, { model: 'jev-vercel' }) + ).toEqual({ + answers, + usage: { model: 'jev-vercel', inputTokens: 283, outputTokens: 21 }, + }) + }) + + it('rejects a model the catalogue does not define', async () => { + await expect( + createDecision('state', questions, { model: 'gpt-image-2' }) + ).rejects.toThrow() + + expect(decideTypeSafe).not.toHaveBeenCalled() + }) + + it('leaves retrying to the provider, so a failure is not multiplied here', async () => { + // @ts-ignore + decideTypeSafe.mockRejectedValueOnce( + Object.assign(new Error('bad gateway'), { status: 502 }) + ) + + await expect(createDecision('state', questions)).rejects.toThrow( + 'bad gateway' + ) + + expect(decideTypeSafe).toHaveBeenCalledTimes(1) + }) + + it('does not retry a request the provider rejected', async () => { + // @ts-ignore + decideTypeSafe.mockRejectedValueOnce( + Object.assign(new Error('unprocessable'), { status: 422 }) + ) + + await expect(createDecision('state', questions)).rejects.toThrow( + 'unprocessable' + ) + + expect(decideTypeSafe).toHaveBeenCalledTimes(1) + }) +}) diff --git a/platform/lib/decision.types.ts b/platform/lib/decision.types.ts new file mode 100644 index 0000000..95e8a30 --- /dev/null +++ b/platform/lib/decision.types.ts @@ -0,0 +1,48 @@ +export type DecisionInput = string | Record | unknown[] + +export type DecisionQuestion = + | { + type: 'boolean' + instructions: DecisionInput + criteria?: { + true: DecisionInput | null + false: DecisionInput | null + } + } + | { + type: 'choice' + instructions: DecisionInput + criteria: Record + } + | { + type: 'score' + instructions: DecisionInput + criteria: (DecisionInput | null)[] + } + +export type DecisionAnswer = + | { type: 'boolean'; probability: number } + | { type: 'choice'; choice: string; probabilities?: Record } + | { type: 'score'; score: number; probabilities?: Record } + +export interface DecisionUsage { + model: string + inputTokens: number + outputTokens: number +} + +export interface CreateDecisionOptions { + state: DecisionInput + + questions: Record + + model: string + modelOptions?: Record + + signal?: AbortSignal +} + +export interface CreateDecisionResult { + answers: Record + usage: DecisionUsage +} diff --git a/platform/lib/extract.data.ts b/platform/lib/extract.data.ts index f132976..ff5a282 100644 --- a/platform/lib/extract.data.ts +++ b/platform/lib/extract.data.ts @@ -20,6 +20,7 @@ import zodToJsonSchema from 'zod-to-json-schema' export interface Message { type: MessageType text: string + meta?: Record } // --- Core Extraction Functionality --- diff --git a/platform/lib/github.app.ts b/platform/lib/github.app.ts index 2d36ad0..e3890db 100644 --- a/platform/lib/github.app.ts +++ b/platform/lib/github.app.ts @@ -13,6 +13,8 @@ const INSTALLATION_TOKEN_TTL_SECONDS = 50 * 60 // @note the App slug is stable; cache for a day const APP_SLUG_TTL_SECONDS = 24 * 60 * 60 +const MAX_TEXT_CHARS = 60_000 + interface GithubRequestOptions { method?: string body?: unknown @@ -62,7 +64,30 @@ export async function githubRequest( return null } - return await response.json() + const contentType = response.headers?.get('content-type') || '' + + if (!contentType || /json/i.test(contentType)) { + return await response.json() + } + + // @note some endpoints answer with plain text instead of JSON, e.g. the job + // logs at /repos/{owner}/{repo}/actions/jobs/{job_id}/logs + if (/^text\//i.test(contentType)) { + const text = await response.text() + + // @note the tail is kept because the end of a log is where failures are + return text.length > MAX_TEXT_CHARS + ? '[text truncated]\n' + text.slice(-MAX_TEXT_CHARS) + : text + } + + await response.body?.cancel() + + throw new FetchError( + `GitHub API ${method} ${path} returned ${contentType} content, which cannot be returned here; only JSON and text responses are supported (for workflow logs use /repos/{owner}/{repo}/actions/jobs/{job_id}/logs, which is plain text)`, + statusToCodeMap[400], + { method, path, status: response.status, contentType } + ) } // --- App JWT + installation token minting (per-integration GitHub App) --- diff --git a/platform/lib/github.app.utest.js b/platform/lib/github.app.utest.js index 06a1214..999a8bb 100644 --- a/platform/lib/github.app.utest.js +++ b/platform/lib/github.app.utest.js @@ -28,6 +28,7 @@ jest.mock('@/lib/fetch', () => ({ jest.mock('@/lib/response', () => ({ statusToCodeMap: { + 400: 'BAD_REQUEST', 401: 'NOT_AUTHORIZED', 404: 'NOT_FOUND', 500: 'INTERNAL_SERVER_ERROR', @@ -113,6 +114,93 @@ describe('github.app', () => { meta: { method: 'GET', path: '/repos/acme/missing', status: 404 }, }) }) + + it('parses JSON when the content type is a JSON media type', async () => { + mockFetch.mockResolvedValue({ + ok: true, + status: 200, + headers: new Headers({ + 'content-type': 'application/vnd.github+json; charset=utf-8', + }), + json: jest.fn().mockResolvedValue({ id: 1 }), + }) + + const data = await githubRequest('/repos/acme/demo', { token: 'token-1' }) + + expect(data).toEqual({ id: 1 }) + }) + + it('returns plain text responses as text instead of parsing JSON', async () => { + const json = jest.fn().mockRejectedValue(new SyntaxError('not JSON')) + + mockFetch.mockResolvedValue({ + ok: true, + status: 200, + headers: new Headers({ 'content-type': 'text/plain; charset=utf-8' }), + json, + text: jest.fn().mockResolvedValue('2026-09-18T09:51:00Z build ok'), + }) + + const data = await githubRequest( + '/repos/acme/demo/actions/jobs/1/logs', + { token: 'token-1' } + ) + + expect(data).toBe('2026-09-18T09:51:00Z build ok') + expect(json).not.toHaveBeenCalled() + }) + + it('keeps the tail of oversized text responses', async () => { + const text = 'a'.repeat(100_000) + 'THE-END' + + mockFetch.mockResolvedValue({ + ok: true, + status: 200, + headers: new Headers({ 'content-type': 'text/plain' }), + text: jest.fn().mockResolvedValue(text), + }) + + const data = await githubRequest( + '/repos/acme/demo/actions/jobs/1/logs', + { token: 'token-1' } + ) + + expect(data.length).toBeLessThan(text.length) + expect(data.startsWith('[text truncated]')).toBe(true) + expect(data.endsWith('THE-END')).toBe(true) + }) + + it('rejects binary responses with an expected code without reading the body', async () => { + const json = jest.fn().mockRejectedValue(new SyntaxError('not JSON')) + const cancel = jest.fn().mockResolvedValue(undefined) + + mockFetch.mockResolvedValue({ + ok: true, + status: 200, + headers: new Headers({ 'content-type': 'application/zip' }), + body: { cancel }, + json, + }) + + await expect( + githubRequest('/repos/acme/demo/actions/runs/1/logs', { + token: 'token-1', + }) + ).rejects.toMatchObject({ + name: 'FetchError', + message: expect.stringContaining('application/zip'), + code: 'BAD_REQUEST', + meta: { + method: 'GET', + path: '/repos/acme/demo/actions/runs/1/logs', + status: 200, + contentType: 'application/zip', + }, + }) + + expect(json).not.toHaveBeenCalled() + expect(cancel).toHaveBeenCalled() + }) }) describe('credentials assertion', () => { diff --git a/platform/lib/image.edit.default.utest.js b/platform/lib/image.edit.default.utest.js new file mode 100644 index 0000000..3284fb1 --- /dev/null +++ b/platform/lib/image.edit.default.utest.js @@ -0,0 +1,46 @@ +import { editImage } from '@/lib/image' +import { editImage as editOpenAIImage } from '@/lib/model.provider.openai' +import { editImage as editVercelImage } from '@/lib/model.provider.vercel.adaptor' + +// @note a deployment with only a Vercel key: gpt-image-1, the preferred edit +// model, is not served +jest.mock('@/config/models', () => ({ + ...jest.requireActual('@/config/models'), + __esModule: true, + imageModels: { + 'gateway-image': { provider: 'vercel', pricing: { tokenRatio: 1 } }, + }, + defaultImageModel: 'gateway-image', +})) + +jest.mock('@/lib/storage', () => ({ getObject: jest.fn(), putObject: jest.fn() })) + +jest.mock('@/lib/host', () => ({ + getExternalHostURL: () => 'https://example.com', +})) + +jest.mock('@/lib/model.provider.openai', () => ({ + createImage: jest.fn(), + editImage: jest.fn(), +})) + +jest.mock('@/lib/model.provider.vercel.adaptor', () => ({ + createImage: jest.fn(), + editImage: jest.fn(), +})) + +describe('editImage default model', () => { + it('falls back to the catalogue default when the preferred edit model is not served', async () => { + editVercelImage.mockResolvedValue({ + urls: [], + usage: { model: 'gateway-image', inputTokens: 1, outputTokens: 1 }, + }) + + await editImage('a cat', [new Blob(['x'])], {}) + + expect(editVercelImage).toHaveBeenCalledWith( + expect.objectContaining({ model: 'gateway-image' }) + ) + expect(editOpenAIImage).not.toHaveBeenCalled() + }) +}) diff --git a/platform/lib/image.ts b/platform/lib/image.ts index 9d4e5f6..44ce47d 100644 --- a/platform/lib/image.ts +++ b/platform/lib/image.ts @@ -1,6 +1,6 @@ import { assertUnreachable } from '@chatbotkit-dev/typescript-utils/unreachable' -import { defaultImageModel } from '@/config/models' +import { defaultImageModel, imageModels } from '@/config/models' import { parseDataURL } from '@/lib/dataurl.parse' import debug from '@/lib/debug' @@ -23,6 +23,7 @@ import { editImage as editVercelImage, } from '@/lib/model.provider.vercel.adaptor' import { parseAndRevealImageModel } from '@/lib/model.utils' +import { throwBadRequest } from '@/lib/response' import { getObject, putObject } from '@/lib/storage' import { v1 as uuidv1 } from 'uuid' @@ -135,6 +136,13 @@ export async function createImage( const { model = defaultImageModel, user, signal } = options || {} + // @note a deployment serves image models only when a provider key is set; + // without one the catalogue is empty and any name would pass validation + + if (!Object.keys(imageModels).length) { + throwBadRequest('No image model is configured on this deployment') + } + const { name, config } = parseAndRevealImageModel(model) // @note use providerModel if set on the model config - this holds the exact @@ -258,7 +266,22 @@ export async function editImage( ): Promise { debug(`edit image`, { prompt, images, options }) - const { model = 'gpt-image-1', user, mask, signal } = options || {} + // @note the preferred edit model holds only when the deployment serves it; + // otherwise the catalogue default stands in rather than a name that cannot + // resolve + const { + model = imageModels['gpt-image-1'] ? 'gpt-image-1' : defaultImageModel, + user, + mask, + signal, + } = options || {} + + // @note a deployment serves image models only when a provider key is set; + // without one the catalogue is empty and any name would pass validation + + if (!Object.keys(imageModels).length) { + throwBadRequest('No image model is configured on this deployment') + } const { name, config } = parseAndRevealImageModel(model) diff --git a/platform/lib/image.unconfigured.utest.js b/platform/lib/image.unconfigured.utest.js new file mode 100644 index 0000000..21e8d0e --- /dev/null +++ b/platform/lib/image.unconfigured.utest.js @@ -0,0 +1,46 @@ +import { createImage, editImage } from '@/lib/image' +import { createImage as createOpenAIImage } from '@/lib/model.provider.openai' + +jest.mock('@/config/models', () => ({ + ...jest.requireActual('@/config/models'), + __esModule: true, + imageModels: {}, + defaultImageModel: 'gpt-image-2', +})) + +jest.mock('@/lib/storage', () => ({ getObject: jest.fn(), putObject: jest.fn() })) + +jest.mock('@/lib/host', () => ({ + getExternalHostURL: () => 'https://example.com', +})) + +jest.mock('@/lib/model.provider.openai', () => ({ + createImage: jest.fn(), + editImage: jest.fn(), +})) + +// @note a deployment serves image models only when a provider key is set, so +// an empty catalogue is the normal state of a fresh install +describe('image on a deployment that serves no image model', () => { + const expected = { + message: 'No image model is configured on this deployment', + code: 'BAD_REQUEST', + } + + it.each([[undefined], ['gpt-image-2'], ['anything']])( + 'createImage answers a bad request rather than an internal error for model %s', + async (model) => { + await expect(createImage('a cat', { model })).rejects.toMatchObject( + expected + ) + + expect(createOpenAIImage).not.toHaveBeenCalled() + } + ) + + it('editImage answers a bad request rather than an internal error', async () => { + await expect( + editImage('a cat', [new Blob(['x'])], {}) + ).rejects.toMatchObject(expected) + }) +}) diff --git a/platform/lib/image.utest.js b/platform/lib/image.utest.js index 195fbc4..8790b04 100644 --- a/platform/lib/image.utest.js +++ b/platform/lib/image.utest.js @@ -22,6 +22,22 @@ jest.mock('@/lib/model.provider.openrouter', () => ({ editImage: jest.fn(), })) +// @note the module resolves models through the mocked parseAndRevealImageModel +// below; the catalogue only has to be non-empty, as it is on any deployment +// that serves image models +jest.mock('@/config/models', () => { + const actual = jest.requireActual('@/config/models') + + return { + ...actual, + __esModule: true, + imageModels: { + ...actual.imageModels, + 'gpt-image-1': { provider: 'openai' }, + }, + } +}) + jest.mock('@/lib/model.utils', () => ({ parseAndRevealImageModel: jest.fn(), })) diff --git a/platform/lib/mcp.error.ts b/platform/lib/mcp.error.ts index c4202d0..6d2d964 100644 --- a/platform/lib/mcp.error.ts +++ b/platform/lib/mcp.error.ts @@ -1,5 +1,8 @@ +import { statusToCodeMap } from '@chatbotkit-dev/http-codes' + import { FetchError } from '@/lib/fetch' +import { StreamableHTTPError } from '@modelcontextprotocol/sdk/client/streamableHttp.js' import { McpError } from '@modelcontextprotocol/sdk/types.js' /** @@ -28,5 +31,19 @@ export function rethrowMcpError(e: unknown): never { ) } + if (e instanceof StreamableHTTPError) { + // @note the transport carries the remote server's HTTP status as `code`; + // mapping it the way getFetchError does gives a 401/403/404/429 from the + // user's server its expected code instead of landing as a generic error + + const status = Number(e.code) + + throw new FetchError( + e.message, + (statusToCodeMap as Record)[status] ?? String(e.code), + { status } + ) + } + throw e } diff --git a/platform/lib/mcp.error.utest.js b/platform/lib/mcp.error.utest.js index 5d84397..adc37a2 100644 --- a/platform/lib/mcp.error.utest.js +++ b/platform/lib/mcp.error.utest.js @@ -1,6 +1,7 @@ import { FetchError } from '@/lib/fetch' import { rethrowMcpError } from '@/lib/mcp.error' +import { StreamableHTTPError } from '@modelcontextprotocol/sdk/client/streamableHttp.js' import { McpError } from '@modelcontextprotocol/sdk/types.js' describe('mcp.error', () => { @@ -66,6 +67,52 @@ describe('mcp.error', () => { } }) + it('should convert StreamableHTTPError to FetchError with the status code mapped', () => { + const error = new StreamableHTTPError( + 403, + 'Error POSTing to endpoint: forbidden' + ) + + try { + rethrowMcpError(error) + } catch (e) { + expect(e).toBeInstanceOf(FetchError) + expect(e.message).toBe( + 'Streamable HTTP error: Error POSTing to endpoint: forbidden' + ) + expect(e.code).toBe('NOT_AUTHORIZED') + expect(e.name).toBe('FetchError({"status":403})') + } + }) + + it('should map a 5xx StreamableHTTPError to its gateway code', () => { + const error = new StreamableHTTPError( + 502, + 'Error POSTing to endpoint: Container suddenly disconnected, try again' + ) + + try { + rethrowMcpError(error) + } catch (e) { + expect(e).toBeInstanceOf(FetchError) + expect(e.code).toBe('BAD_GATEWAY') + } + }) + + it('should keep an unmapped StreamableHTTPError status as the code', () => { + const error = new StreamableHTTPError( + 418, + 'Error POSTing to endpoint: teapot' + ) + + try { + rethrowMcpError(error) + } catch (e) { + expect(e).toBeInstanceOf(FetchError) + expect(e.code).toBe('418') + } + }) + it('should rethrow non-McpError as-is', () => { const regularError = new Error('regular error') diff --git a/platform/lib/model.provider.openai.conv.ts b/platform/lib/model.provider.openai.conv.ts index 327a173..28db2a9 100644 --- a/platform/lib/model.provider.openai.conv.ts +++ b/platform/lib/model.provider.openai.conv.ts @@ -94,7 +94,7 @@ import { } from '@/lib/namespace.attachment' import { clone } from '@/lib/object' import { awaitWithAbortGrace } from '@/lib/promise' -import { throwConflict } from '@/lib/response' +import { isUnknownError, throwConflict } from '@/lib/response' import { Result } from '@/lib/result' import { byteSlice, getRandomId } from '@/lib/string' import { @@ -4505,7 +4505,12 @@ async function* completeChatConversationRound( () => ({ error: HANDLER_DEADLINE_BYPASS_ERROR }) ) } catch (e) { - await captureException(e) + // @note an expected code, such as a channel wait that timed out, + // is normal operation and stays out of Sentry + + if (isUnknownError(e)) { + await captureException(e) + } // @note we are deliberately hiding the error from the user // because this is an internal issue @@ -5010,7 +5015,12 @@ async function* completeChatConversationRound( () => ({ error: HANDLER_DEADLINE_BYPASS_ERROR }) ) } catch (e) { - await captureException(e) + // @note an expected code, such as a channel wait that timed out, + // is normal operation and stays out of Sentry + + if (isUnknownError(e)) { + await captureException(e) + } if (e instanceof SafeError) { result = { error: e.message } @@ -6561,7 +6571,12 @@ async function* completeResponseConversationRound( () => ({ error: HANDLER_DEADLINE_BYPASS_ERROR }) ) } catch (e) { - await captureException(e) + // @note an expected code, such as a channel wait that timed out, + // is normal operation and stays out of Sentry + + if (isUnknownError(e)) { + await captureException(e) + } if (e instanceof SafeError) { result = { error: e.message } @@ -7635,7 +7650,12 @@ export async function* completeRealtimeConversationStream( newMessages, }) } catch (e) { - await captureException(e) + // @note an expected code, such as a channel wait that timed out, + // is normal operation and stays out of Sentry + + if (isUnknownError(e)) { + await captureException(e) + } if (e instanceof SafeError) { result = { error: e.message } diff --git a/platform/lib/model.provider.openrouter.decision.utest.js b/platform/lib/model.provider.openrouter.decision.utest.js new file mode 100644 index 0000000..7cda94f --- /dev/null +++ b/platform/lib/model.provider.openrouter.decision.utest.js @@ -0,0 +1,157 @@ +import _fetch from '@/lib/fetch' +import { throwOpenAIError } from '@/lib/model.provider.openai' +import { decide } from '@/lib/model.provider.openrouter' + +jest.mock('@/lib/fetch', () => { + const actual = jest.fn() + + return { + __esModule: true, + default: actual, + withRetry: jest.fn((fn) => fn), + withTimeout: jest.fn((fn) => fn), + } +}) + +jest.mock('@/config/site', () => ({ + siteUrl: 'https://site.example', + siteHostname: 'site.example', +})) + +jest.mock('@/lib/model.context', () => ({ + getSafeModelStore: () => ({}), +})) + +jest.mock('@/lib/model.provider.openai', () => ({ + createChatCompletion: jest.fn(), + createChatCompletionStream: jest.fn(), + throwOpenAIError: jest.fn(), +})) + +const questions = { + urgent: { type: 'boolean', instructions: 'Does this convey urgency?' }, + department: { + type: 'choice', + instructions: 'Which team should handle this', + criteria: { billing: 'payments', technical: null }, + }, +} + +describe('decide', () => { + const original = process.env.OPENROUTER_MODELS_API_KEY + + beforeEach(() => { + jest.clearAllMocks() + + process.env.OPENROUTER_MODELS_API_KEY = 'test-openrouter-key' + + // @ts-ignore + _fetch.mockResolvedValue({ + ok: true, + json: async () => ({ + model: 'typesafe/jev-1.13', + answers: { + urgent: { type: 'noul', noul: 0.93 }, + department: { + type: 'choice', + choice: 'billing', + confidence: 0.6, + probabilities: { billing: 0.84, technical: 0.16 }, + }, + }, + usage: { input_tokens: 312, output_tokens: 48, cost: 0.00001 }, + }), + }) + }) + + afterAll(() => { + if (original === undefined) { + delete process.env.OPENROUTER_MODELS_API_KEY + } else { + process.env.OPENROUTER_MODELS_API_KEY = original + } + }) + + it('must post noul questions to the alpha decisions endpoint', async () => { + await decide({ + state: [{ role: 'user', content: 'help asap' }], + questions, + model: '~typesafe/jev-latest', + }) + + // @ts-ignore + const [url, init] = _fetch.mock.calls[0] + + expect(url).toBe('https://openrouter.ai/api/alpha/decisions') + expect(init.headers.Authorization).toBe('Bearer test-openrouter-key') + expect(JSON.parse(init.body)).toEqual({ + model: '~typesafe/jev-latest', + state: [{ role: 'user', content: 'help asap' }], + questions: { + urgent: { type: 'noul', instructions: 'Does this convey urgency?' }, + department: questions.department, + }, + }) + }) + + it('must not request zero data retention, which the only provider cannot route', async () => { + await decide({ state: 's', questions, model: '~typesafe/jev-latest' }) + + // @ts-ignore + const body = JSON.parse(_fetch.mock.calls[0][1].body) + + expect(body.provider).toBeUndefined() + expect(body.zdr).toBeUndefined() + }) + + it('must carry catalogue provider options as routing preferences', async () => { + await decide({ + state: 's', + questions, + model: '~typesafe/jev-latest', + modelOptions: { data_collection: 'deny' }, + }) + + // @ts-ignore + expect(JSON.parse(_fetch.mock.calls[0][1].body).provider).toEqual({ + data_collection: 'deny', + }) + }) + + it('must return platform answers and camel-cased usage', async () => { + const result = await decide({ + state: 's', + questions, + model: '~typesafe/jev-latest', + }) + + expect(result).toEqual({ + answers: { + urgent: { type: 'boolean', probability: 0.93 }, + department: { + type: 'choice', + choice: 'billing', + probabilities: { billing: 0.84, technical: 0.16 }, + }, + }, + usage: { + model: '~typesafe/jev-latest', + inputTokens: 312, + outputTokens: 48, + }, + }) + }) + + it('must surface provider failures through the provider error path', async () => { + const response = { ok: false, status: 402 } + + // @ts-ignore + _fetch.mockResolvedValue(response) + + await decide({ state: 's', questions, model: '~typesafe/jev-latest' }) + + expect(throwOpenAIError).toHaveBeenCalledWith(response, { + errorPrefix: 'OR_', + }) + }) +}) diff --git a/platform/lib/model.provider.openrouter.ts b/platform/lib/model.provider.openrouter.ts index 831af17..fc596da 100644 --- a/platform/lib/model.provider.openrouter.ts +++ b/platform/lib/model.provider.openrouter.ts @@ -1,6 +1,10 @@ import { siteHostname, siteUrl } from '@/config/site' import debug from '@/lib/debug' +import type { + CreateDecisionOptions, + CreateDecisionResult, +} from '@/lib/decision.types' import _fetch, { withRetry, withTimeout } from '@/lib/fetch' import { getSafeModelStore } from '@/lib/model.context' import { resolveProviderCredential } from '@/lib/model.credentials' @@ -9,6 +13,10 @@ import { createChatCompletionStream as createOpenAICompatibleChatCompletionStream, throwOpenAIError, } from '@/lib/model.provider.openai' +import { + fromSystemOneAnswers, + toSystemOneQuestions, +} from '@/lib/model.provider.typesafe' /** * fetch instance dedicated for image creation (no timeout) @@ -22,6 +30,16 @@ const fetchForImage = withRetry(withTimeout(_fetch, { timeout: 0 }), { retryTimeout: true, }) +/** + * fetch instance dedicated for decisions, bounded so a slow provider fails + * within the response budget. + */ +const fetchForDecision = withRetry(withTimeout(_fetch, { timeout: 15_000 }), { + retries: 5, + retryDelay: 250, + retryTimeout: false, +}) + /** * Gets the OpenRouter API key from model store or environment. * @@ -373,3 +391,69 @@ export async function editImage( }, } } + +// --- Decision --- + +/** + * Answers typed questions about a state using OpenRouter's decision models + * (e.g. typesafe/jev). + * + * @note decisions live on an alpha endpoint outside /api/v1 and speak the + * System One wire format. zdr is not requested: the only provider is TypeSafe, + * which offers it to enterprise accounts only, so the request would not route. + */ +export async function decide( + options: CreateDecisionOptions +): Promise { + const { state, questions, model, modelOptions, signal } = options + + debug(`decide using`, { + model, + modelOptions, + questionCount: Object.keys(questions).length, + }).log('openrouter.decide') + + const body = { + model, + + state, + questions: toSystemOneQuestions(questions), + + ...(modelOptions && { provider: modelOptions }), + } + + const response = await fetchForDecision( + 'https://openrouter.ai/api/alpha/decisions', + { + method: 'POST', + + headers: { + Authorization: `Bearer ${getOpenRouterAPIKey()}`, + 'Content-Type': 'application/json', + 'HTTP-Referer': siteUrl, + 'X-Title': siteHostname, + }, + + body: JSON.stringify(body), + + signal, + } + ) + + if (!response.ok) { + return await throwOpenAIError(response, { errorPrefix: 'OR_' }) + } + + const data = await response.json() + + debug(`received data`, { data }).log('openrouter.decide.received') + + return { + answers: fromSystemOneAnswers(data.answers), + usage: { + model, + inputTokens: data.usage?.input_tokens ?? 0, + outputTokens: data.usage?.output_tokens ?? 0, + }, + } +} diff --git a/platform/lib/model.provider.typesafe.ts b/platform/lib/model.provider.typesafe.ts new file mode 100644 index 0000000..5bc3c22 --- /dev/null +++ b/platform/lib/model.provider.typesafe.ts @@ -0,0 +1,203 @@ +import debug from '@/lib/debug' +import type { + CreateDecisionOptions, + CreateDecisionResult, + DecisionAnswer, + DecisionQuestion, +} from '@/lib/decision.types' +import _fetch, { withRetry, withTimeout } from '@/lib/fetch' +import { resolveProviderCredential } from '@/lib/model.credentials' +import { getOpenAIError } from '@/lib/model.provider.openai' + +/** + * fetch instance dedicated for decisions, bounded so a slow provider fails + * within the response budget. + */ +const fetchForDecision = withRetry(withTimeout(_fetch, { timeout: 15_000 }), { + retries: 5, + retryDelay: 250, + retryTimeout: false, +}) + +/** + * Gets the TypeSafe API key from the environment. + * + * @throws {UserConfigError} if no key is configured + */ +export function getTypeSafeAPIKey(): string { + return resolveProviderCredential({ + label: 'TypeSafe', + storeKey: undefined, + storeUrl: undefined, + envKey: process.env.TYPESAFE_MODELS_API_KEY, + }) +} + +// --- Decision --- + +/** + * Converts questions to the System One wire format. + * + * @note System One names the boolean question type `noul`; everything else is + * carried as is. OpenRouter's decisions endpoint speaks the same format. + */ +export function toSystemOneQuestions( + questions: Record +): Record { + return Object.fromEntries( + Object.entries(questions).map(([key, question]) => [ + key, + question.type === 'boolean' ? { ...question, type: 'noul' } : question, + ]) + ) +} + +interface SystemOneAnswer { + type: string + noul?: number + choice?: string + score?: number + probabilities?: Record +} + +/** + * Converts System One answers to the platform answer format. + */ +export function fromSystemOneAnswers( + answers: Record | undefined +): Record { + return Object.fromEntries( + Object.entries(answers || {}).map(([key, answer]) => { + switch (answer.type) { + case 'noul': { + return [key, { type: 'boolean', probability: answer.noul }] + } + + case 'choice': { + return [ + key, + { + type: 'choice', + choice: answer.choice, + probabilities: answer.probabilities, + }, + ] + } + + case 'score': { + return [ + key, + { + type: 'score', + score: answer.score, + probabilities: answer.probabilities, + }, + ] + } + + default: { + throw new Error(`Unrecognized answer type ${answer.type}`) + } + } + }) + ) +} + +/** + * Reads the reason out of an error response. + * + * @note the API documents that a validation error names the offending field + * but not the shape of the body, so the common shapes are all tried. An + * unrecognised body yields no message and the status text stands in. + */ +async function getErrorMessage(response: Response): Promise { + let data: Record | null + + try { + data = await response.json() + } catch { + return undefined + } + + const error = data?.error + + const reason = + (typeof error === 'object' && error !== null + ? (error as Record).message + : undefined) ?? + error ?? + data?.message ?? + data?.detail + + if (typeof reason === 'string') { + return reason + } + + if (reason !== undefined && reason !== null) { + return JSON.stringify(reason) + } + + return undefined +} + +/** + * Answers typed questions about a state using TypeSafe's System One API. + */ +export async function decide( + options: CreateDecisionOptions +): Promise { + const { state, questions, model, signal } = options + + debug(`decide using`, { + model, + questionCount: Object.keys(questions).length, + }).log('typesafe.decide') + + const body = { + model, + + state, + questions: toSystemOneQuestions(questions), + } + + const response = await fetchForDecision( + 'https://api.typesafe.ai/v1/systemone', + { + method: 'POST', + + headers: { + Authorization: `Bearer ${getTypeSafeAPIKey()}`, + 'Content-Type': 'application/json', + }, + + body: JSON.stringify(body), + + signal, + } + ) + + if (!response.ok) { + throw getOpenAIError( + { + response: { + status: response.status, + data: { error: { message: await getErrorMessage(response) } }, + }, + }, + { errorPrefix: 'TS_' } + ) + } + + const data = await response.json() + + debug(`received data`, { data }).log('typesafe.decide.received') + + return { + answers: fromSystemOneAnswers(data.answers), + usage: { + model, + inputTokens: data.usage?.input_tokens ?? 0, + outputTokens: data.usage?.output_tokens ?? 0, + }, + } +} diff --git a/platform/lib/model.provider.typesafe.utest.js b/platform/lib/model.provider.typesafe.utest.js new file mode 100644 index 0000000..0428178 --- /dev/null +++ b/platform/lib/model.provider.typesafe.utest.js @@ -0,0 +1,212 @@ +import _fetch from '@/lib/fetch' +import { getOpenAIError } from '@/lib/model.provider.openai' +import { + decide, + fromSystemOneAnswers, + toSystemOneQuestions, +} from '@/lib/model.provider.typesafe' + +jest.mock('@/lib/fetch', () => { + const actual = jest.fn() + + return { + __esModule: true, + default: actual, + withRetry: jest.fn((fn) => fn), + withTimeout: jest.fn((fn) => fn), + } +}) + +jest.mock('@/lib/model.provider.openai', () => ({ + getOpenAIError: jest.fn( + (error) => new Error(String(error.response.data.error.message)) + ), +})) + +const questions = { + urgent: { + type: 'boolean', + instructions: 'Does this convey urgency?', + criteria: { true: 'time-sensitive', false: null }, + }, + department: { + type: 'choice', + instructions: 'Which team should handle this', + criteria: { billing: 'payments', technical: null }, + }, + frustration: { + type: 'score', + instructions: 'How frustrated the customer appears', + criteria: ['calm', 'frustrated', 'angry'], + }, +} + +const systemOneAnswers = { + urgent: { type: 'noul', noul: 0.999 }, + department: { + type: 'choice', + choice: 'billing', + probabilities: { billing: 0.84, technical: 0.16 }, + confidence: 0.596, + }, + frustration: { + type: 'score', + score: 1.035, + legend: { 0: 'calm', 1: 'frustrated', 2: 'angry' }, + probabilities: { 0: 0.1, 1: 0.7, 2: 0.2 }, + confidence: 0.842, + }, +} + +describe('toSystemOneQuestions', () => { + it('renames the boolean type to noul and carries everything else as is', () => { + expect(toSystemOneQuestions(questions)).toEqual({ + urgent: { ...questions.urgent, type: 'noul' }, + department: questions.department, + frustration: questions.frustration, + }) + }) +}) + +describe('fromSystemOneAnswers', () => { + it('maps noul to a boolean probability and drops provider-only fields', () => { + expect(fromSystemOneAnswers(systemOneAnswers)).toEqual({ + urgent: { type: 'boolean', probability: 0.999 }, + department: { + type: 'choice', + choice: 'billing', + probabilities: { billing: 0.84, technical: 0.16 }, + }, + frustration: { + type: 'score', + score: 1.035, + probabilities: { 0: 0.1, 1: 0.7, 2: 0.2 }, + }, + }) + }) + + it('returns no answers when the provider sends none', () => { + expect(fromSystemOneAnswers(undefined)).toEqual({}) + }) + + it('refuses an answer type it does not know', () => { + expect(() => fromSystemOneAnswers({ q: { type: 'text' } })).toThrow( + 'Unrecognized answer type text' + ) + }) +}) + +describe('decide', () => { + const original = process.env.TYPESAFE_MODELS_API_KEY + + beforeEach(() => { + jest.clearAllMocks() + + process.env.TYPESAFE_MODELS_API_KEY = 'test-typesafe-key' + + // @ts-ignore + _fetch.mockResolvedValue({ + ok: true, + json: async () => ({ + model: 'jev-1.13.0', + answers: systemOneAnswers, + usage: { input_tokens: 312, output_tokens: 48 }, + }), + }) + }) + + afterAll(() => { + if (original === undefined) { + delete process.env.TYPESAFE_MODELS_API_KEY + } else { + process.env.TYPESAFE_MODELS_API_KEY = original + } + }) + + it('must post the model, state and System One questions to the systemone endpoint', async () => { + await decide({ state: 'help asap', questions, model: 'jev-latest' }) + + // @ts-ignore + const [url, init] = _fetch.mock.calls[0] + + expect(url).toBe('https://api.typesafe.ai/v1/systemone') + expect(init.method).toBe('POST') + expect(init.headers.Authorization).toBe('Bearer test-typesafe-key') + expect(JSON.parse(init.body)).toEqual({ + model: 'jev-latest', + state: 'help asap', + questions: toSystemOneQuestions(questions), + }) + }) + + it('must return platform answers and camel-cased usage', async () => { + const result = await decide({ + state: 'help asap', + questions, + model: 'jev-latest', + }) + + expect(result.answers.urgent).toEqual({ + type: 'boolean', + probability: 0.999, + }) + expect(result.usage).toEqual({ + model: 'jev-latest', + inputTokens: 312, + outputTokens: 48, + }) + }) + + it('must fail with a configuration error when no key is set', async () => { + delete process.env.TYPESAFE_MODELS_API_KEY + + await expect( + decide({ state: 'help asap', questions, model: 'jev-latest' }) + ).rejects.toThrow() + + expect(_fetch).not.toHaveBeenCalled() + }) + + it.each([ + ['an OpenAI-style body', { error: { message: 'criteria is required' } }, 'criteria is required'], + ['a plain message', { message: 'state is too long' }, 'state is too long'], + ['a string detail', { detail: 'unknown model' }, 'unknown model'], + [ + 'a structured detail', + { detail: [{ loc: ['questions', 'q', 'criteria'], msg: 'too few levels' }] }, + '[{"loc":["questions","q","criteria"],"msg":"too few levels"}]', + ], + ])('must keep the reason the API gives in %s', async (_name, body, message) => { + // @ts-ignore + _fetch.mockResolvedValue({ ok: false, status: 422, json: async () => body }) + + await expect( + decide({ state: 'help asap', questions, model: 'jev-latest' }) + ).rejects.toThrow(message) + + expect(getOpenAIError).toHaveBeenCalledWith( + { response: { status: 422, data: { error: { message } } } }, + { errorPrefix: 'TS_' } + ) + }) + + it('must fall back to the status when the body explains nothing', async () => { + // @ts-ignore + _fetch.mockResolvedValue({ + ok: false, + status: 529, + json: async () => { + throw new Error('not json') + }, + }) + + await expect( + decide({ state: 'help asap', questions, model: 'jev-latest' }) + ).rejects.toThrow() + + expect(getOpenAIError).toHaveBeenCalledWith( + { response: { status: 529, data: { error: { message: undefined } } } }, + { errorPrefix: 'TS_' } + ) + }) +}) diff --git a/platform/lib/model.provider.vercel.decision.utest.js b/platform/lib/model.provider.vercel.decision.utest.js new file mode 100644 index 0000000..cbc1e8c --- /dev/null +++ b/platform/lib/model.provider.vercel.decision.utest.js @@ -0,0 +1,148 @@ +import _fetch from '@/lib/fetch' +import { decide } from '@/lib/model.provider.vercel' +import { throwOpenAIError } from '@/lib/model.provider.openai' + +jest.mock('@/lib/fetch', () => { + const actual = jest.fn() + + // @ts-ignore + actual.withRetry = jest.fn((fn) => fn) + // @ts-ignore + actual.withTimeout = jest.fn((fn) => fn) + // @ts-ignore + actual.withBodyTimeout = jest.fn((fn) => fn) + + return { + __esModule: true, + default: actual, + // @ts-ignore + withRetry: actual.withRetry, + // @ts-ignore + withTimeout: actual.withTimeout, + // @ts-ignore + withBodyTimeout: actual.withBodyTimeout, + } +}) + +jest.mock('@/lib/model.context', () => ({ + getSafeModelStore: () => ({}), +})) + +jest.mock('@/lib/model.provider.openai', () => ({ + createChatCompletion: jest.fn(), + createChatCompletionStream: jest.fn(), + throwOpenAIError: jest.fn(), +})) + +const state = [{ role: 'user', content: 'I was charged twice' }] + +const questions = { + refunded: { type: 'boolean', instructions: 'Was a refund issued?' }, + topic: { + type: 'choice', + instructions: 'What is the topic?', + criteria: { billing: 'payments', technical: 'bugs' }, + }, +} + +const answers = { + refunded: { type: 'boolean', probability: 0.02 }, + topic: { + type: 'choice', + choice: 'billing', + probabilities: { billing: 0.99, technical: 0.01 }, + }, +} + +describe('decide', () => { + beforeEach(() => { + jest.clearAllMocks() + + process.env.VERCEL_MODELS_API_KEY = 'test-vercel-key' + + // @ts-ignore + _fetch.mockResolvedValue({ + ok: true, + json: async () => ({ + answers, + usage: { inputTokens: 283, outputTokens: 21 }, + }), + }) + }) + + it('must target the gateway decision-model protocol with the model id header', async () => { + await decide({ state, questions, model: 'typesafe-ai/jev' }) + + // @ts-ignore + const [url, init] = _fetch.mock.calls[0] + + expect(url).toBe('https://ai-gateway.vercel.sh/v4/ai/evaluation-model') + expect(init.method).toBe('POST') + expect(init.headers).toEqual( + expect.objectContaining({ + Authorization: 'Bearer test-vercel-key', + 'ai-evaluation-model-specification-version': '4', + 'ai-model-id': 'typesafe-ai/jev', + }) + ) + }) + + it('must send the state and questions unserialised, and provider options only when set', async () => { + await decide({ state, questions, model: 'typesafe-ai/jev' }) + + // @ts-ignore + expect(JSON.parse(_fetch.mock.calls[0][1].body)).toEqual({ + state, + questions, + }) + + await decide({ + state, + questions, + model: 'typesafe-ai/jev', + modelOptions: { gateway: { only: ['typesafe-ai'] } }, + }) + + // @ts-ignore + expect(JSON.parse(_fetch.mock.calls[1][1].body)).toEqual({ + state, + questions, + providerOptions: { gateway: { only: ['typesafe-ai'] } }, + }) + }) + + it('must return the answers and the reported token usage', async () => { + const result = await decide({ state, questions, model: 'typesafe-ai/jev' }) + + expect(result).toEqual({ + answers, + usage: { model: 'typesafe-ai/jev', inputTokens: 283, outputTokens: 21 }, + }) + }) + + it('must default the usage to zero when the gateway reports none', async () => { + // @ts-ignore + _fetch.mockResolvedValue({ ok: true, json: async () => ({ answers }) }) + + const result = await decide({ state, questions, model: 'typesafe-ai/jev' }) + + expect(result.usage).toEqual({ + model: 'typesafe-ai/jev', + inputTokens: 0, + outputTokens: 0, + }) + }) + + it('must surface gateway failures through the provider error path', async () => { + const response = { ok: false, status: 400 } + + // @ts-ignore + _fetch.mockResolvedValue(response) + + await decide({ state, questions, model: 'typesafe-ai/jev' }) + + expect(throwOpenAIError).toHaveBeenCalledWith(response, { + errorPrefix: 'VR_', + }) + }) +}) diff --git a/platform/lib/model.provider.vercel.ts b/platform/lib/model.provider.vercel.ts index fa0b523..b53c523 100644 --- a/platform/lib/model.provider.vercel.ts +++ b/platform/lib/model.provider.vercel.ts @@ -1,6 +1,10 @@ import { blobToDataUrl } from '@/lib/dataurl.blob' import debug from '@/lib/debug' import { SystemError, UserInputError } from '@/lib/error' +import type { + CreateDecisionOptions, + CreateDecisionResult, +} from '@/lib/decision.types' import _fetch, { withRetry, withTimeout } from '@/lib/fetch' import { getSafeModelStore } from '@/lib/model.context' import { resolveProviderCredential } from '@/lib/model.credentials' @@ -42,6 +46,16 @@ const fetchForRerank = withRetry(withTimeout(_fetch, { timeout: 15_000 }), { retryTimeout: false, }) +/** + * fetch instance dedicated for decisions, bounded like fetchForRerank so a + * slow gateway fails within the response budget. + */ +const fetchForDecision = withRetry(withTimeout(_fetch, { timeout: 15_000 }), { + retries: 5, + retryDelay: 250, + retryTimeout: false, +}) + /** * Gets the Vercel API key from model store or environment. * @@ -1060,3 +1074,69 @@ export async function rerank( }, } } + +// --- Decision --- + +/** + * Answers typed questions about a state using Vercel AI Gateway's evaluation + * models (e.g. typesafe-ai/jev), the gateway's name for decision models. + * + * @note these are not exposed through the OpenAI-compatible endpoint, so this + * targets the gateway model protocol directly (POST /v4/ai/evaluation-model + * with the ai-model-id header), mirroring the reranking model contract above. + */ +export async function decide( + options: CreateDecisionOptions +): Promise { + const { state, questions, model, modelOptions, signal } = options + + debug(`decide using`, { + model, + modelOptions, + questionCount: Object.keys(questions).length, + }).log('vercel.decide') + + const body = { + state, + questions, + + ...(modelOptions && { providerOptions: modelOptions }), + } + + const response = await fetchForDecision( + 'https://ai-gateway.vercel.sh/v4/ai/evaluation-model', + { + method: 'POST', + + headers: { + Authorization: `Bearer ${getVercelAPIKey()}`, + 'Content-Type': 'application/json', + 'ai-gateway-protocol-version': '0.0.1', + 'ai-gateway-auth-method': 'api-key', + 'ai-evaluation-model-specification-version': '4', + 'ai-model-id': model, + }, + + body: JSON.stringify(body), + + signal, + } + ) + + if (!response.ok) { + return await throwOpenAIError(response, { errorPrefix: 'VR_' }) + } + + const data = await response.json() + + debug(`received data`, { data }).log('vercel.decide.received') + + return { + answers: data.answers || {}, + usage: { + model, + inputTokens: data.usage?.inputTokens ?? 0, + outputTokens: data.usage?.outputTokens ?? 0, + }, + } +} diff --git a/platform/lib/model.types.ts b/platform/lib/model.types.ts index 546e73e..93c3b7f 100644 --- a/platform/lib/model.types.ts +++ b/platform/lib/model.types.ts @@ -389,6 +389,37 @@ export type VercelRerankModel = RerankModel & { export type AnyRerankModel = VercelRerankModel +/** + * DECISION + */ + +export type DecisionModel = Model & { + pricing: { + tokenRatio: number + inputTokenRatio?: number + outputTokenRatio?: number + inputPrice?: number + outputPrice?: number + } +} + +export type VercelDecisionModel = DecisionModel & { + provider: 'vercel' +} + +export type OpenRouterDecisionModel = DecisionModel & { + provider: 'openrouter' +} + +export type TypeSafeDecisionModel = DecisionModel & { + provider: 'typesafe' +} + +export type AnyDecisionModel = + | VercelDecisionModel + | OpenRouterDecisionModel + | TypeSafeDecisionModel + /** * SPEACH TO TEXT */ diff --git a/platform/lib/model.utils.js b/platform/lib/model.utils.js index bb83deb..b98f0aa 100644 --- a/platform/lib/model.utils.js +++ b/platform/lib/model.utils.js @@ -3,10 +3,12 @@ import { assertUnreachable } from '@chatbotkit-dev/typescript-utils/unreachable' import { baseLanguageModel, + defaultDecisionModel, defaultImageModel, defaultLanguageModel, defaultRerankModel, defaultVideoModel, + decisionModels, imageModels, languageModels, rerankModels, @@ -28,6 +30,7 @@ import externalUrlSchema from '@/schemas/externalUrl' * @typedef {import('@/lib/model.types').AnyImageModel} AnyImageModel * @typedef {import('@/lib/model.types').AnyVideoModel} AnyVideoModel * @typedef {import('@/lib/model.types').AnyRerankModel} AnyRerankModel + * @typedef {import('@/lib/model.types').AnyDecisionModel} AnyDecisionModel * @typedef {import('@/lib/model.types').AnySpeechToTextModel} AnySpeechToTextModel * @typedef {import('@/lib/model.types').AnyTextToSpeechModel} AnyTextToSpeechModel */ @@ -94,6 +97,21 @@ export const rerankModelToUseTypeMapping = Object.fromEntries( ]) ) +/** + * Maps the decision models to their corresponding use types. + * + * @type {Object.} + */ +export const decisionModelToUseTypeMapping = Object.fromEntries( + Object.entries(decisionModels).map(([key, { provider }]) => [ + key, + provider.toUpperCase() + + '_' + + key.toUpperCase().replace(/[-.]/g, '_') + + '_TOKEN', + ]) +) + /** * Maps the speech-to-text models to their corresponding use types. * @@ -176,6 +194,18 @@ export const useTypeToRerankModelMapping = Object.fromEntries( ]) ) +/** + * Maps the use types to their corresponding decision models. + * + * @type {Object.} + */ +export const useTypeToDecisionModelMapping = Object.fromEntries( + Object.entries(decisionModelToUseTypeMapping).map(([key, value]) => [ + value, + key, + ]) +) + /** * Maps the use types to their corresponding speech-to-text models. * @@ -954,6 +984,153 @@ export function rerankModelToUseType(model) { return type } +// --- Decision Models --- + +/** + * @param {string} model + * @returns {number} + */ +export function getDecisionModelDefaultTokenRatio(model) { + const { name } = parseDecisionModel(model) + + const config = decisionModels[name] + + assert(config, `Model ${name} is not recognized`) + + const { + pricing: { tokenRatio }, + } = config + + assert(tokenRatio, `Model ${name} does not have a token ratio`) + + return tokenRatio +} + +/** + * @param {string} model + * @returns {number} + */ +export function getDecisionModelInputTokenRatio(model) { + const { name } = parseDecisionModel(model) + + const config = decisionModels[name] + + assert(config, `Model ${name} is not recognized`) + + const { + pricing: { tokenRatio, inputTokenRatio = tokenRatio }, + } = config + + return inputTokenRatio +} + +/** + * @param {string} model + * @returns {number} + */ +export function getDecisionModelOutputTokenRatio(model) { + const { name } = parseDecisionModel(model) + + const config = decisionModels[name] + + assert(config, `Model ${name} is not recognized`) + + const { + pricing: { tokenRatio, outputTokenRatio = tokenRatio }, + } = config + + return outputTokenRatio +} + +/** + * @param {string} model + * @param {'default'|'output'|'input'} [type='default'] + * @returns {number} + */ +export function getDecisionModelTokenRatio(model, type = 'default') { + let tokenRatio = 1 + + switch (type) { + case 'default': { + tokenRatio = getDecisionModelDefaultTokenRatio(model) + + break + } + + case 'output': { + tokenRatio = getDecisionModelOutputTokenRatio(model) + + break + } + + case 'input': { + tokenRatio = getDecisionModelInputTokenRatio(model) + + break + } + + default: { + assertUnreachable(type) + } + } + + return tokenRatio +} + +/** + * @param {string} sourceModel + * @param {number} sourceCount + * @param {'default'|'output'|'input'} [type='default'] + * @returns {number} + * @throws {Error} + */ +export function getBaseDecisionModelTokenCount( + sourceModel, + sourceCount, + type = 'default' +) { + debug(`getting base model token count`, { + sourceModel, + sourceCount, + type, + }).log('model.getBaseModelTokenCount') + + if (sourceCount === 0) { + return 0 + } + + const tokenRatio = getDecisionModelTokenRatio(sourceModel, type) + + const count = Math.max(1, Math.round(sourceCount * tokenRatio)) + + debug(`base model token count`, { + sourceModel, + sourceCount, + type, + tokenRatio, + count, + }).log('model.getBaseModelTokenCount') + + return count +} + +/** + * @param {string} model + * @returns {string} + * @throws {Error} + */ +export function decisionModelToUseType(model) { + const { name } = parseDecisionModel(model) + + const type = decisionModelToUseTypeMapping[name] + + if (!type) { + throw new Error(`Unrecognized model ${name}`) + } + + return type +} + // --- Audio Models --- /** @@ -1663,6 +1840,115 @@ export function parseAndRevealRerankModel(model) { return revealRerankModel(parseRerankModel(model)) } +/** + * @typedef {{ + * region?: 'us'|'eu' + * }} DecisionModelConfig + */ + +const decisionModelValidationSchema = schema.object().keys({ + name: schema + .string() + .valid(...Object.keys(decisionModels)) + .required(), + config: schema + .object() + .keys({ + region: schema.string().valid('us', 'eu'), + }) + .required(), +}) + +/** + * @param {string} model + * @returns {{name: string, config: DecisionModelConfig}} + * @throws {Error} + */ +export function parseDecisionModel(model) { + model = model || defaultDecisionModel + + let { name, config } = parse(model, defaultDecisionModel) + + const { error, value } = decisionModelValidationSchema.validate({ + name, + config, + }) + + if (error) { + throw error + } + + name = value.name + config = value.config + + return { name, config } +} + +/** + * @param {string} name + * @param {DecisionModelConfig} config + * @returns {string} + * @throws {Error} + */ +export function buildDecisionModel(name, config) { + const { error, value } = decisionModelValidationSchema.validate({ + name, + config, + }) + + if (error) { + throw error + } + + name = value.name + config = value.config + + const details = build(name, config, decisionModels[name]) + + return details +} + +/** + * @param {{name: string, config: DecisionModelConfig}} parsedModel + * @param {AnyDecisionModel} [decisionModel] + * @returns {{name: string, config: AnyDecisionModel & DecisionModelConfig, originalName: string, originalConfig: DecisionModelConfig}} + */ +export function revealDecisionModel({ name, config }, decisionModel) { + const originalName = name + const originalConfig = config + + let newName = name + + let newConfig = { + ...(decisionModel + ? (({ proxyToModel: _, ...o }) => o)(decisionModel) + : undefined), + + ...decisionModels[name], + + ...config, + } + + if (newConfig.proxyToModel) { + name = newConfig.proxyToModel + + const result = revealDecisionModel({ name, config }, newConfig) + + newName = result.name + newConfig = result.config + } + + return { name: newName, config: newConfig, originalName, originalConfig } +} + +/** + * @param {string} model + * @returns {ReturnType} + */ +export function parseAndRevealDecisionModel(model) { + return revealDecisionModel(parseDecisionModel(model)) +} + /** * @typedef {{}} SpeechToTextModelConfig */ diff --git a/platform/lib/model.utils.utest.js b/platform/lib/model.utils.utest.js index 26a3165..7b2a99c 100644 --- a/platform/lib/model.utils.utest.js +++ b/platform/lib/model.utils.utest.js @@ -23,7 +23,10 @@ import { import { audioModelToUseType, audioModelToUseTypeMapping, + buildDecisionModel, buildLanguageModel, + decisionModelToUseType, + decisionModelToUseTypeMapping, convertLanguageModelTokenCount, getBaseImageModelTokenCount, getBaseLanguageModelTokenCount, @@ -31,6 +34,8 @@ import { getImageModelTokenRatio, getVideoModelTokenRatio, hasLanguageModelsByProvider, + getBaseDecisionModelTokenCount, + getDecisionModelTokenRatio, imageModelToUseType, imageModelToUseTypeMapping, languageModelToUseType, @@ -42,6 +47,8 @@ import { modelSupportsImageInput, modelSupportsRealtime, modelSupportsResponses, + parseAndRevealDecisionModel, + parseDecisionModel, parseImageModel, parseLanguageModel, redactLanguageModel, @@ -50,6 +57,7 @@ import { speechToTextModelToUseTypeMapping, textToSpeechModelToUseType, textToSpeechModelToUseTypeMapping, + useTypeToDecisionModelMapping, } from '@/lib/model.utils' jest.mock('@/config/models', () => { @@ -216,6 +224,18 @@ jest.mock('@/config/models', () => { }, ...actual.videoModels, }, + decisionModels: { + jev: { + provider: 'typesafe', + providerModel: 'jev-latest', + pricing: { + tokenRatio: 0.003, + inputTokenRatio: 0.003, + outputTokenRatio: 0, + }, + }, + }, + defaultDecisionModel: 'jev', speechToTextModels: { 'gpt-4o-transcribe': { provider: 'openai' }, ...actual.speechToTextModels, @@ -635,6 +655,51 @@ describe('imageModelToUseType', () => { }) }) +describe('decision models', () => { + it('must correctly return the correct use type', () => { + const type = decisionModelToUseType('jev') + + expect(type).toBe('TYPESAFE_JEV_TOKEN') + expect(type).toBe(decisionModelToUseTypeMapping.jev) + expect(useTypeToDecisionModelMapping[type]).toBe('jev') + }) + + it('must throw for a model the catalogue does not define', () => { + expect(() => decisionModelToUseType('gpt-image-1.5')).toThrow() + }) + + it('must return the ratio of each side', () => { + expect(getDecisionModelTokenRatio('jev')).toBe(0.003) + expect(getDecisionModelTokenRatio('jev', 'input')).toBe(0.003) + expect(getDecisionModelTokenRatio('jev', 'output')).toBe(0) + }) + + it('must calibrate to at least one base token, and to none for no tokens', () => { + expect(getBaseDecisionModelTokenCount('jev', 0, 'input')).toBe(0) + expect(getBaseDecisionModelTokenCount('jev', 100, 'input')).toBe(1) + expect(getBaseDecisionModelTokenCount('jev', 100000, 'input')).toBe(300) + }) + + it('must parse, build and reveal a model round trip', () => { + const built = buildDecisionModel('jev', { region: 'us' }) + + expect(parseDecisionModel(built)).toEqual({ + name: 'jev', + config: { region: 'us' }, + }) + + const { name, config } = parseAndRevealDecisionModel(built) + + expect(name).toBe('jev') + expect(config.provider).toBe('typesafe') + expect(config.providerModel).toBe('jev-latest') + }) + + it('must refuse to build a model the catalogue does not define', () => { + expect(() => buildDecisionModel('gpt-image-1.5', {})).toThrow() + }) +}) + describe('speechToTextModelToUseType', () => { it('must correctly return the correct use type', () => { const model = 'gpt-4o-transcribe' diff --git a/platform/lib/secret.value.ts b/platform/lib/secret.value.ts index f85bdc7..781ad82 100644 --- a/platform/lib/secret.value.ts +++ b/platform/lib/secret.value.ts @@ -13,7 +13,7 @@ import { } from '@/lib/context.store' import { parseBasicCredentials } from '@/lib/creds.basic.parse' import debug, { assert } from '@/lib/debug' -import { UserAuthError } from '@/lib/error' +import { UserAuthError, UserConfigError } from '@/lib/error' import { toHeaders } from '@/lib/header' import { getAccessToken } from '@/lib/oauth.token' import { canUseSecret } from '@/lib/secret.access' @@ -483,10 +483,18 @@ export async function getSecretValueAndType( const { importPKCS8, SignJWT } = await import('jose') - const privateKey = await importPKCS8( - normalizePrivateKeyPemToPKCS8(trimmedValue), - algorithm - ) + let privateKey + + try { + privateKey = await importPKCS8( + normalizePrivateKeyPemToPKCS8(trimmedValue), + algorithm + ) + } catch { + throw new UserConfigError( + `The JWT secret value is not a valid ${algorithm} private key in PEM (PKCS#8) form` + ) + } const token = await new SignJWT(claims) .setProtectedHeader({ alg: algorithm }) diff --git a/platform/lib/secret.value.utest.js b/platform/lib/secret.value.utest.js index 826940c..5ed01d0 100644 --- a/platform/lib/secret.value.utest.js +++ b/platform/lib/secret.value.utest.js @@ -6,6 +6,7 @@ import { mockDeep, mockReset } from 'jest-mock-extended' import prisma from '@/prisma/client' import { encode as encodeB64 } from '@/lib/b64' +import { UserConfigError } from '@/lib/error' import { runInContext, setContextContact, @@ -3607,6 +3608,22 @@ describe('getSecretValueAndType - jwt secret type', () => { })() }) + it('should reject a value that is not a private key as a config error', async () => { + await runInContext(async () => { + setContextNamespace('test') + + const secret = { + kind: 'shared', + type: 'jwt', + value: 'not-a-pem-key', + } + + await expect(getSecretValueAndType(secret)).rejects.toBeInstanceOf( + UserConfigError + ) + })() + }) + it('should sign a JWT from a PKCS#1 RSA private key', async () => { const { jwtVerify } = await import('jose') diff --git a/platform/lib/usage.model.ts b/platform/lib/usage.model.ts index d773f5c..85e9ddf 100644 --- a/platform/lib/usage.model.ts +++ b/platform/lib/usage.model.ts @@ -6,12 +6,15 @@ import debug from '@/lib/debug' import { captureObservation } from '@/lib/error' import { convertLanguageModelTokenCount, + getBaseDecisionModelTokenCount, getBaseImageModelTokenCount, getBaseLanguageModelTokenCount, getBaseVideoModelTokenCount, + getDecisionModelTokenRatio, getImageModelTokenRatio, getLanguageModelTokenRatio, getVideoModelTokenRatio, + parseDecisionModel, parseImageModel, parseLanguageModel, parseVideoModel, @@ -211,6 +214,54 @@ export class Usage { }).log('usage.Usage.addImageTokens') } + /** + * Add decision tokens to the usage. Mirrors addImageTokens but uses the + * decision model calibration. + */ + addDecisionTokens( + tokens: number, + model: string, + type: OperationType = 'default' + ) { + debug(`adding decision tokens`, { + '#baseToken': this.#baseToken, + '#lineItems': this.#lineItems, + + tokens, + model, + type, + }).log('usage.Usage.addDecisionTokens') + + const tokenRatio = getDecisionModelTokenRatio(model, type) + + // @note a side the model does not charge for (zero ratio) is not usage + + if (tokens <= 0 || tokenRatio === 0) { + debug(`no tokens to add`, { tokens }).log('usage.Usage.addDecisionTokens') + + return + } + + const { name: modelName } = parseDecisionModel(model) + + const baseToken = getBaseDecisionModelTokenCount(model, tokens, type) + + this.#baseToken += baseToken + + this.#lineItems.push({ + tokens: tokens, + model: modelName, + type: type, + debit: baseToken, + ratio: tokenRatio, + }) + + debug(`added decision tokens`, { + '#baseToken': this.#baseToken, + '#lineItems': this.#lineItems, + }).log('usage.Usage.addDecisionTokens') + } + /** * Records the usage of the tokens. If a model is provided, we will attempt to * re-calibrate the tokens based on the model. diff --git a/platform/lib/usage.model.utest.js b/platform/lib/usage.model.utest.js index c2cca01..7f7be06 100644 --- a/platform/lib/usage.model.utest.js +++ b/platform/lib/usage.model.utest.js @@ -7,9 +7,11 @@ import { import { convertLanguageModelTokenCount, + getBaseDecisionModelTokenCount, getBaseImageModelTokenCount, getBaseLanguageModelTokenCount, getBaseVideoModelTokenCount, + getDecisionModelTokenRatio, getImageModelTokenRatio, getVideoModelTokenRatio, } from '@/lib/model.utils' @@ -53,6 +55,17 @@ jest.mock('@/config/models', () => { }, }, }, + decisionModels: { + jev: { + provider: 'typesafe', + pricing: { + tokenRatio: 0.003, + inputTokenRatio: 0.003, + outputTokenRatio: 0, + }, + }, + }, + defaultDecisionModel: 'jev', videoModels: { ...actual.videoModels, 'veo-3.1': { @@ -473,6 +486,85 @@ describe('Usage', () => { }) }) + describe('addDecisionTokens', () => { + it('calibrates the tokens with the ratio of their side', () => { + const usage = new Usage() + + usage.addDecisionTokens(100000, 'jev', 'input') + + expect(usage.token).toBe(300) + expect(usage.items).toEqual([ + { tokens: 100000, model: 'jev', type: 'input', debit: 300, ratio: 0.003 }, + ]) + }) + + it('debits every call at least one base token, like the other model classes', () => { + const usage = new Usage() + + usage.addDecisionTokens(100, 'jev', 'input') // worth 0.3 + usage.addDecisionTokens(300, 'jev', 'input') // worth 0.9 + + expect(usage.token).toBe(2) + expect(usage.items.map(({ debit }) => debit)).toEqual([1, 1]) + }) + + it('agrees with getBaseDecisionModelTokenCount / getDecisionModelTokenRatio', () => { + const usage = new Usage() + + usage.addDecisionTokens(4321, 'jev', 'input') + + expect(usage.token).toBe( + getBaseDecisionModelTokenCount('jev', 4321, 'input') + ) + expect(usage.items[0].ratio).toBe( + getDecisionModelTokenRatio('jev', 'input') + ) + }) + + it('debits nothing for a free side and gives it no line item', () => { + const usage = new Usage() + + usage.addDecisionTokens(5000, 'jev', 'output') + + expect(usage.token).toBe(0) + expect(usage.items).toEqual([]) + }) + + it('is a no-op for zero or negative tokens', () => { + const usage = new Usage() + + usage.addDecisionTokens(0, 'jev', 'input') + usage.addDecisionTokens(-5, 'jev', 'input') + + expect(usage.token).toBe(0) + expect(usage.items).toEqual([]) + }) + + it('records as base tokens with the decision line items', async () => { + const usage = new Usage() + + usage.addDecisionTokens(1000, 'jev', 'input') + + await usage.recordBaseTokens({ + user: { id: 'user-1' }, + meta: { reason: 'decision/create' }, + }) + + expect(recordLanguageTokenUsage).toHaveBeenCalledWith( + expect.objectContaining({ + count: 3, + model: baseLanguageModel, + meta: { + reason: 'decision/create', + lineItems: [ + { tokens: 1000, model: 'jev', type: 'input', debit: 3, ratio: 0.003 }, + ], + }, + }) + ) + }) + }) + describe('addUsage', () => { it('should add another usage object tokens', () => { const otherUsage = new Usage() diff --git a/platform/lib/usage.record.ts b/platform/lib/usage.record.ts index bd39f95..8f0a05a 100644 --- a/platform/lib/usage.record.ts +++ b/platform/lib/usage.record.ts @@ -17,6 +17,8 @@ import { import memcache from '@/lib/memcache' import { audioModelToUseType, + decisionModelToUseType, + getBaseDecisionModelTokenCount, getBaseImageModelTokenCount, getBaseLanguageModelTokenCount, getBaseRerankModelTokenCount, @@ -24,6 +26,7 @@ import { imageModelToUseType, languageModelToUseType, rerankModelToUseType, + useTypeToDecisionModelMapping, useTypeToImageModelMapping, useTypeToLanguageModelMapping, useTypeToRerankModelMapping, @@ -99,6 +102,13 @@ export function getCalibratedBaseCount(type: string, count: number): number { ) } + if (useTypeToDecisionModelMapping[type]) { + return getBaseDecisionModelTokenCount( + useTypeToDecisionModelMapping[type], + count + ) + } + return count } @@ -792,6 +802,43 @@ export async function recordRerankTokenUsage({ await recordUsage({ user, type, count, meta, references }) } +/** + * Options for recordDecisionTokenUsage. + */ +interface RecordDecisionTokenUsageOptions { + user: Pick + count: number + model: string + meta?: Record + references?: UsageReferences +} + +/** + * Records decision model token usage. + */ +export async function recordDecisionTokenUsage({ + user, + count, + model, + meta, + references, +}: RecordDecisionTokenUsageOptions): Promise { + debug(`recording decision token usage`, { + user, + count, + model, + meta, + references, + }).log('usage.record.recordDecisionTokenUsage') + + const type = decisionModelToUseType(model) + + // @note the reason we do not calibrate here is because this operation is + // handled by the usage queue + + await recordUsage({ user, type, count, meta, references }) +} + /** * Options for recordAudioTokenUsage. */ diff --git a/platform/lib/usage.record.utest.js b/platform/lib/usage.record.utest.js index 68bc50b..790ba23 100644 --- a/platform/lib/usage.record.utest.js +++ b/platform/lib/usage.record.utest.js @@ -8,6 +8,8 @@ import prisma from '@/prisma/client' import memcache from '@/lib/memcache' import { audioModelToUseType, + decisionModelToUseType, + getBaseDecisionModelTokenCount, getBaseImageModelTokenCount, getBaseLanguageModelTokenCount, getBaseRerankModelTokenCount, @@ -33,6 +35,7 @@ import { recordImageUsage, recordLanguageTokenUsage, recordMessageUsage, + recordDecisionTokenUsage, recordRerankTokenUsage, recordUsage, recordVideoTokenUsage, @@ -78,6 +81,8 @@ jest.mock('@/lib/model.utils', () => ({ getBaseRerankModelTokenCount: jest.fn((model, count) => count * 4), + getBaseDecisionModelTokenCount: jest.fn((model, count) => count * 5), + languageModelToUseType: jest.fn((model) => { const mapping = { 'gpt-4': 'OPENAI_GPT_4_TOKEN', @@ -93,6 +98,8 @@ jest.mock('@/lib/model.utils', () => ({ rerankModelToUseType: jest.fn(() => 'VERCEL_RERANK_V4_FAST_TOKEN'), + decisionModelToUseType: jest.fn(() => 'TYPESAFE_JEV_TOKEN'), + audioModelToUseType: jest.fn((model) => { const mapping = { 'gpt-4o-transcribe': 'OPENAI_GPT_4O_TRANSCRIBE_TOKEN', @@ -118,6 +125,10 @@ jest.mock('@/lib/model.utils', () => ({ useTypeToRerankModelMapping: { VERCEL_RERANK_V4_FAST_TOKEN: 'rerank-v4-fast', }, + + useTypeToDecisionModelMapping: { + TYPESAFE_JEV_TOKEN: 'jev', + }, })) jest.mock('@/lib/user.get', () => ({ @@ -251,6 +262,11 @@ describe('usage.record', () => { expect(getCalibratedBaseCount('VERCEL_RERANK_V4_FAST_TOKEN', 3)).toBe(12) }) + it('should calibrate decision model token counts', () => { + // mock getBaseDecisionModelTokenCount returns count * 5 + expect(getCalibratedBaseCount('TYPESAFE_JEV_TOKEN', 3)).toBe(15) + }) + it('should pass through counts for non-calibrated types', () => { expect(getCalibratedBaseCount('CHATBOTKIT_CONVERSATION', 7)).toBe(7) expect(getCalibratedBaseCount('CHATBOTKIT_EMAIL', 3)).toBe(3) @@ -1497,6 +1513,51 @@ describe('usage.record', () => { }) }) + describe('recordDecisionTokenUsage', () => { + beforeEach(() => { + fastGetUserById.mockResolvedValue({ id: 'user123', parentId: null }) + + prisma.usage.create.mockResolvedValue({}) + + memcache.incrementInWindow.mockResolvedValue(1) + }) + + it('should convert model to use type and calibrate base token usage', async () => { + await recordDecisionTokenUsage({ + user: { id: 'user123' }, + count: 1, + model: 'jev', + }) + + expect(decisionModelToUseType).toHaveBeenCalledWith('jev') + expect(getBaseDecisionModelTokenCount).toHaveBeenCalledWith( + 'jev', + 1 + ) + expect(prisma.usage.create).toHaveBeenCalledWith({ + data: { + userId: 'user123', + type: 'TYPESAFE_JEV_TOKEN', + count: 1, + conversationId: 'conv123', + messageId: undefined, + contactId: 'contact456', + botId: 'bot789', + datasetId: undefined, + skillsetId: undefined, + meta: { + ipAddress: '192.168.1.1', + }, + }, + }) + expect(memcache.incrementInWindow).toHaveBeenCalledWith( + 'usage-user123-token', + 5, + 2678400 + ) + }) + }) + describe('recordVideoUsage', () => { beforeEach(() => { fastGetUserById.mockResolvedValue({ id: 'user123', parentId: null }) diff --git a/platform/lib/video.edit.default.utest.js b/platform/lib/video.edit.default.utest.js new file mode 100644 index 0000000..f0a3a28 --- /dev/null +++ b/platform/lib/video.edit.default.utest.js @@ -0,0 +1,45 @@ +import { editVideo as editVercelVideo } from '@/lib/model.provider.vercel.adaptor' +import { editVideo } from '@/lib/video' + +// @note a deployment that does not serve grok-imagine-video, the preferred +// edit model +jest.mock('@/config/models', () => ({ + ...jest.requireActual('@/config/models'), + __esModule: true, + videoModels: { + 'gateway-video': { + provider: 'vercel', + pricing: { tokenRatio: 1 }, + duration: 8, + availableDurations: [8], + availableAspectRatios: ['16:9'], + }, + }, + defaultVideoModel: 'gateway-video', +})) + +jest.mock('@/lib/storage', () => ({ getObject: jest.fn(), putObject: jest.fn() })) + +jest.mock('@/lib/host', () => ({ + getExternalHostURL: () => 'https://example.com', +})) + +jest.mock('@/lib/model.provider.vercel.adaptor', () => ({ + createVideo: jest.fn(), + editVideo: jest.fn(), +})) + +describe('editVideo default model', () => { + it('falls back to the catalogue default when the preferred edit model is not served', async () => { + editVercelVideo.mockResolvedValue({ + urls: [], + usage: { model: 'gateway-video', inputTokens: 1, outputTokens: 1 }, + }) + + await editVideo('a cat', ['https://example.com/source.mp4'], {}) + + expect(editVercelVideo).toHaveBeenCalledWith( + expect.objectContaining({ model: 'gateway-video' }) + ) + }) +}) diff --git a/platform/lib/video.ts b/platform/lib/video.ts index 8015556..54fabfd 100644 --- a/platform/lib/video.ts +++ b/platform/lib/video.ts @@ -1,6 +1,6 @@ import { assertUnreachable } from '@chatbotkit-dev/typescript-utils/unreachable' -import { defaultVideoModel } from '@/config/models' +import { defaultVideoModel, videoModels } from '@/config/models' import { parseDataURL } from '@/lib/dataurl.parse' import debug from '@/lib/debug' @@ -16,6 +16,7 @@ import { editVideo as editVercelVideo, } from '@/lib/model.provider.vercel.adaptor' import { parseAndRevealVideoModel } from '@/lib/model.utils' +import { throwBadRequest } from '@/lib/response' import { getObject, putObject } from '@/lib/storage' import { v1 as uuidv1 } from 'uuid' @@ -226,6 +227,13 @@ export async function createVideo( const { model = defaultVideoModel, user, signal } = options || {} + // @note a deployment serves video models only when a provider key is set; + // without one the catalogue is empty and any name would pass validation + + if (!Object.keys(videoModels).length) { + throwBadRequest('No video model is configured on this deployment') + } + const { name, config } = parseAndRevealVideoModel(model) const provider = config.provider @@ -314,7 +322,21 @@ export async function editVideo( throw new Error('At least one video, frame, or audio is required') } - const { model = 'grok-imagine-video', user, signal } = options || {} + // @note the preferred edit model holds only when the deployment serves it; + // otherwise the catalogue default stands in rather than a name that cannot + // resolve + const { + model = videoModels['grok-imagine-video'] ? 'grok-imagine-video' : defaultVideoModel, + user, + signal, + } = options || {} + + // @note a deployment serves video models only when a provider key is set; + // without one the catalogue is empty and any name would pass validation + + if (!Object.keys(videoModels).length) { + throwBadRequest('No video model is configured on this deployment') + } const { name, config } = parseAndRevealVideoModel(model) diff --git a/platform/lib/video.unconfigured.utest.js b/platform/lib/video.unconfigured.utest.js new file mode 100644 index 0000000..993ddba --- /dev/null +++ b/platform/lib/video.unconfigured.utest.js @@ -0,0 +1,46 @@ +import { createVideo as createVercelVideo } from '@/lib/model.provider.vercel.adaptor' +import { createVideo, editVideo } from '@/lib/video' + +jest.mock('@/config/models', () => ({ + ...jest.requireActual('@/config/models'), + __esModule: true, + videoModels: {}, + defaultVideoModel: 'veo-3.1', +})) + +jest.mock('@/lib/storage', () => ({ getObject: jest.fn(), putObject: jest.fn() })) + +jest.mock('@/lib/host', () => ({ + getExternalHostURL: () => 'https://example.com', +})) + +jest.mock('@/lib/model.provider.vercel.adaptor', () => ({ + createVideo: jest.fn(), + editVideo: jest.fn(), +})) + +// @note a deployment serves video models only when a provider key is set, so +// an empty catalogue is the normal state of a fresh install +describe('video on a deployment that serves no video model', () => { + const expected = { + message: 'No video model is configured on this deployment', + code: 'BAD_REQUEST', + } + + it.each([[undefined], ['veo-3.1'], ['anything']])( + 'createVideo answers a bad request rather than an internal error for model %s', + async (model) => { + await expect(createVideo('a cat', { model })).rejects.toMatchObject( + expected + ) + + expect(createVercelVideo).not.toHaveBeenCalled() + } + ) + + it('editVideo answers a bad request rather than an internal error', async () => { + await expect( + editVideo('a cat', ['https://example.com/source.mp4'], {}) + ).rejects.toMatchObject(expected) + }) +}) diff --git a/platform/pages/api/auxiliary/skillset/ability/chatbotkit/integration/_github.utest.js b/platform/pages/api/auxiliary/skillset/ability/chatbotkit/integration/_github.utest.js new file mode 100644 index 0000000..ebacf53 --- /dev/null +++ b/platform/pages/api/auxiliary/skillset/ability/chatbotkit/integration/_github.utest.js @@ -0,0 +1,94 @@ +/** + * @jest-environment node + */ + +/* eslint-disable @typescript-eslint/no-require-imports */ + +let capturedHandlers = null + +jest.mock('@/lib/auxiliary.handler', () => ({ + authenticatedMultiHandler: jest.fn((handlers) => { + capturedHandlers = handlers + + return jest.fn() + }), +})) + +jest.mock('@/prisma/client', () => ({ + __esModule: true, + default: { + githubIntegration: { + findUniqueByIdentifier: jest.fn(), + }, + }, +})) + +jest.mock('@/lib/github.app', () => ({ + getInstallationTokenForOwner: jest.fn(), + githubRequest: jest.fn(), +})) + +jest.mock('@/lib/debug', () => jest.fn(() => ({ log: jest.fn() }))) + +// Import after mocks are set up so capturedHandlers is populated +require('@/pages/api/auxiliary/skillset/ability/chatbotkit/integration/github') + +const prisma = require('@/prisma/client').default +const { + getInstallationTokenForOwner, + githubRequest, +} = require('@/lib/github.app') + +describe('auxiliary/skillset/ability/chatbotkit/integration/github', () => { + const session = { user: { id: 'user-1' } } + + const parameters = { + githubIntegrationId: 'integration-1', + method: 'GET', + path: '/repos/acme/demo/actions/jobs/1/logs', + } + + beforeEach(() => { + jest.clearAllMocks() + + prisma.githubIntegration.findUniqueByIdentifier.mockResolvedValue({ + id: 'integration-1', + userId: 'user-1', + appId: 'app-1', + privateKey: 'key-1', + }) + + getInstallationTokenForOwner.mockResolvedValue('token-1') + }) + + describe('apiCall', () => { + it('returns JSON results as they are', async () => { + githubRequest.mockResolvedValue({ id: 1 }) + + const result = await capturedHandlers.apiCall.fn(session, parameters) + + expect(result).toEqual({ id: 1 }) + expect(githubRequest).toHaveBeenCalledWith(parameters.path, { + method: 'GET', + body: undefined, + token: 'token-1', + }) + }) + + it('normalizes no content to a success object', async () => { + githubRequest.mockResolvedValue(null) + + const result = await capturedHandlers.apiCall.fn(session, parameters) + + expect(result).toEqual({ ok: true }) + }) + + it('wraps text results in an object', async () => { + githubRequest.mockResolvedValue('build ok') + + const result = await capturedHandlers.apiCall.fn(session, parameters) + + expect(result).toEqual({ text: 'build ok' }) + }) + }) +}) diff --git a/platform/pages/api/auxiliary/skillset/ability/chatbotkit/integration/github.ts b/platform/pages/api/auxiliary/skillset/ability/chatbotkit/integration/github.ts index fdc9a66..4df71ba 100644 --- a/platform/pages/api/auxiliary/skillset/ability/chatbotkit/integration/github.ts +++ b/platform/pages/api/auxiliary/skillset/ability/chatbotkit/integration/github.ts @@ -419,6 +419,10 @@ async function apiCall( const result = await githubRequest(path, { method, body: parsedBody, token }) + if (typeof result === 'string') { + return { text: result } + } + // @note normalize 204 No Content (e.g. DELETE) to a success object return result ?? { ok: true } } diff --git a/platform/pages/api/auxiliary/skillset/ability/chatbotkit/mcp/tool/_install.utest.js b/platform/pages/api/auxiliary/skillset/ability/chatbotkit/mcp/tool/_install.utest.js index 9c47b8e..9ad856a 100644 --- a/platform/pages/api/auxiliary/skillset/ability/chatbotkit/mcp/tool/_install.utest.js +++ b/platform/pages/api/auxiliary/skillset/ability/chatbotkit/mcp/tool/_install.utest.js @@ -267,6 +267,25 @@ describe('auxiliary/skillset/ability/chatbotkit/mcp/tool/install', () => { capturedHandlerFn(mockSession, baseParameters, mockHeaders) ).rejects.toThrow('Connection refused') }) + + it('should convert an McpError from installMcpTools to a FetchError', async () => { + const { McpError } = require('@modelcontextprotocol/sdk/types.js') + const { FetchError } = require('@/lib/fetch') + + installMcpTools.mockRejectedValue( + new McpError(-32001, 'Request timed out', { timeout: 60000 }) + ) + + const error = await capturedHandlerFn( + mockSession, + baseParameters, + mockHeaders + ).catch((e) => e) + + expect(error).toBeInstanceOf(FetchError) + expect(error.message).toBe('MCP error -32001: Request timed out') + expect(error.code).toBe('-32001') + }) }) describe('combined context', () => { diff --git a/platform/pages/api/auxiliary/skillset/ability/chatbotkit/mcp/tool/install.ts b/platform/pages/api/auxiliary/skillset/ability/chatbotkit/mcp/tool/install.ts index 588d84c..25c30ac 100644 --- a/platform/pages/api/auxiliary/skillset/ability/chatbotkit/mcp/tool/install.ts +++ b/platform/pages/api/auxiliary/skillset/ability/chatbotkit/mcp/tool/install.ts @@ -8,6 +8,7 @@ import { } from '@/lib/context.store' import debug from '@/lib/debug' import { installMcpTools } from '@/lib/mcp.direct' +import { rethrowMcpError } from '@/lib/mcp.error' import { throwNotAuthorized } from '@/lib/response' import type { ZodSchemaFor } from '@/lib/zod.schema' import z from '@/lib/zod.schema' @@ -104,23 +105,27 @@ export default authenticatedHandler( setContextNamespace(namespace) } - const result = await installMcpTools(session.user, { - sessionId, + try { + const result = await installMcpTools(session.user, { + sessionId, - url: mcpUrl, - headers: mcpHeaders, + url: mcpUrl, + headers: mcpHeaders, - headerSource, + headerSource, - tools, + tools, - prefix, - }) + prefix, + }) - debug('installed tools', { result }).log( - 'auxiliary.skillset.ability.chatbotkit.mcp.tool.install.handler' - ) + debug('installed tools', { result }).log( + 'auxiliary.skillset.ability.chatbotkit.mcp.tool.install.handler' + ) - return result + return result + } catch (e) { + rethrowMcpError(e) + } } ) diff --git a/platform/pages/api/v1/dataset/[datasetId]/_queue.utest.js b/platform/pages/api/v1/dataset/[datasetId]/_queue.utest.js index de96f8a..41c10e7 100644 --- a/platform/pages/api/v1/dataset/[datasetId]/_queue.utest.js +++ b/platform/pages/api/v1/dataset/[datasetId]/_queue.utest.js @@ -1,7 +1,13 @@ +import prisma from '@/prisma/client' + import { chunkFile } from '@/lib/chunk' +import { logEvent } from '@/lib/log' import { upsertRecord } from '@/lib/record' -import { splitImportBlob } from '@/pages/api/v1/dataset/[datasetId]/queue' +import { + handleImportJobEndEvent, + splitImportBlob, +} from '@/pages/api/v1/dataset/[datasetId]/queue' jest.mock('@/prisma/client', () => ({ __esModule: true, @@ -22,6 +28,18 @@ jest.mock('@/lib/chunk', () => ({ chunkUrl: jest.fn(), })) +jest.mock('@/lib/limit.core', () => ({ + databaseLimitsOk: jest.fn(() => Promise.resolve(true)), +})) + +jest.mock('@/lib/log', () => ({ + logEvent: jest.fn(), +})) + +jest.mock('@/lib/notify', () => ({ + notifyDatasetSyncCompleted: jest.fn(), +})) + const capturedErrors = [] jest.mock('@/lib/error', () => ({ @@ -44,6 +62,34 @@ describe('dataset queue', () => { capturedErrors.length = 0 }) + describe('handleImportJobEndEvent', () => { + it('marks the source integration synced without failing when it is gone', async () => { + prisma.dataset = { + findUnique: jest.fn().mockResolvedValue(mockDataset), + } + + // @note updateMany resolves with a zero count for a deleted integration + // where update would throw a not found error + prisma.sitemapIntegration = { + updateMany: jest.fn().mockResolvedValue({ count: 0 }), + } + + await handleImportJobEndEvent('dataset-123', { + context: { sitemapIntegrationId: 'sitemap-1' }, + urls: [], + }) + + expect(prisma.sitemapIntegration.updateMany).toHaveBeenCalledWith({ + where: { id: 'sitemap-1' }, + data: expect.objectContaining({ syncStatus: 'synced' }), + }) + + expect(logEvent).toHaveBeenCalledWith( + expect.objectContaining({ type: 'dataset.import.job.finish' }) + ) + }) + }) + describe('splitImportBlob', () => { it('should skip items with empty or whitespace-only text without errors', async () => { // Simulate chunkFile returning items with empty/whitespace text diff --git a/platform/pages/api/v1/dataset/[datasetId]/queue.js b/platform/pages/api/v1/dataset/[datasetId]/queue.js index bd54c45..328bfc2 100644 --- a/platform/pages/api/v1/dataset/[datasetId]/queue.js +++ b/platform/pages/api/v1/dataset/[datasetId]/queue.js @@ -626,16 +626,18 @@ export const handleImportJobEndEvent = withDatasetAndLimits( // @note update integration sync status to synced when job completes // @todo generalize this so that each integration can handle this itself + // @note updateMany because the integration may have been deleted while + // the import job was running { if (context?.sitemapIntegrationId) { - await prisma.sitemapIntegration.update({ + await prisma.sitemapIntegration.updateMany({ where: { id: context.sitemapIntegrationId }, data: { syncStatus: SyncStatus.synced, lastSyncedAt: new Date() }, }) } if (context?.notionIntegrationId) { - await prisma.notionIntegration.update({ + await prisma.notionIntegration.updateMany({ where: { id: context.notionIntegrationId }, data: { syncStatus: SyncStatus.synced, lastSyncedAt: new Date() }, }) diff --git a/platform/pages/api/v1/dataset/[datasetId]/record/[recordId]/_update.utest.js b/platform/pages/api/v1/dataset/[datasetId]/record/[recordId]/_update.utest.js index ebe630f..cd5634e 100644 --- a/platform/pages/api/v1/dataset/[datasetId]/record/[recordId]/_update.utest.js +++ b/platform/pages/api/v1/dataset/[datasetId]/record/[recordId]/_update.utest.js @@ -226,5 +226,13 @@ describe('POST /api/v1/dataset/{datasetId}/record/{recordId}/update', () => { '"text"' ) }) + + it('should reject text that is blank once normalized', () => { + // @note zero-width space is not whitespace, but normalization drops it + expect(bodySchema.validate({ text: '\u200b' }).error.message).toContain( + 'printable' + ) + expect(bodySchema.validate({ text: 'a\u200b' }).error).toBeUndefined() + }) }) }) diff --git a/platform/pages/api/v1/dataset/[datasetId]/record/[recordId]/update.js b/platform/pages/api/v1/dataset/[datasetId]/record/[recordId]/update.js index cc72678..4255114 100644 --- a/platform/pages/api/v1/dataset/[datasetId]/record/[recordId]/update.js +++ b/platform/pages/api/v1/dataset/[datasetId]/record/[recordId]/update.js @@ -18,13 +18,13 @@ import { withSession } from '@/lib/session.handler' import { getStore } from '@/lib/store.types' import metaSchema from '@/schemas/meta' -import recordTextSchema from '@/schemas/recordText' +import { nonBlankRecordTextSchema } from '@/schemas/recordText' import sourceSchema from '@/schemas/source' export const bodySchema = schema.object({ // @note optional, but the store keeps the stored text only when the field // is absent - an empty string would replace it and the store refuses that - text: recordTextSchema.invalid('').pattern(/\S/, 'non-blank'), + text: nonBlankRecordTextSchema, source: sourceSchema, diff --git a/platform/pages/api/v1/dataset/[datasetId]/record/_create.utest.js b/platform/pages/api/v1/dataset/[datasetId]/record/_create.utest.js index fff5720..0f6e82a 100644 --- a/platform/pages/api/v1/dataset/[datasetId]/record/_create.utest.js +++ b/platform/pages/api/v1/dataset/[datasetId]/record/_create.utest.js @@ -223,5 +223,15 @@ describe('POST /api/v1/dataset/[datasetId]/record/create', () => { '"text"' ) }) + + it('should reject text that is blank once normalized', () => { + // @note zero-width space is not whitespace, but normalization drops it + expect(bodySchema.validate({ text: '\u200b' }).error.message).toContain( + 'printable' + ) + expect( + bodySchema.validate({ text: 'a\u200b' }).error + ).toBeUndefined() + }) }) }) diff --git a/platform/pages/api/v1/dataset/[datasetId]/record/create.js b/platform/pages/api/v1/dataset/[datasetId]/record/create.js index b6e3b1f..d4ae82b 100644 --- a/platform/pages/api/v1/dataset/[datasetId]/record/create.js +++ b/platform/pages/api/v1/dataset/[datasetId]/record/create.js @@ -17,13 +17,13 @@ import { import { getStore } from '@/lib/store.types' import metaSchema from '@/schemas/meta' -import recordTextSchema from '@/schemas/recordText' +import { nonBlankRecordTextSchema } from '@/schemas/recordText' import sourceSchema from '@/schemas/source' export const bodySchema = schema.object({ // @note the vector store refuses a record without text - reject blank text // here so the caller gets a 400 instead of a store error - text: recordTextSchema.invalid('').pattern(/\S/, 'non-blank').required(), + text: nonBlankRecordTextSchema.required(), source: sourceSchema, diff --git a/platform/pages/api/v1/decision/_create.utest.js b/platform/pages/api/v1/decision/_create.utest.js new file mode 100644 index 0000000..00c0434 --- /dev/null +++ b/platform/pages/api/v1/decision/_create.utest.js @@ -0,0 +1,225 @@ +/* eslint-disable @typescript-eslint/no-require-imports */ +import { baseLanguageModel, decisionModels } from '@/config/models' + +import handler, { bodySchema } from './create' + +jest.mock('@/config/models', () => { + const actual = jest.requireActual('@/config/models') + + return { + ...actual, + __esModule: true, + decisionModels: { + jev: { + provider: 'typesafe', + pricing: { tokenRatio: 0.003, inputTokenRatio: 0.003, outputTokenRatio: 0 }, + }, + }, + defaultDecisionModel: 'jev', + } +}) + +jest.mock('@/lib/method', () => ({ + withPost: (fn) => fn, +})) + +jest.mock('@/lib/limit.handler', () => ({ + withSessionLimits: (_limits, fn) => fn, +})) + +jest.mock('@/lib/joi.handler', () => ({ + __esModule: true, + default: jest.requireActual('@/lib/joi.schema').default, + withSchema: (_schema, fn) => fn, +})) + +jest.mock('@/lib/stream', () => ({ + withStream: (fn) => fn, +})) + +jest.mock('@/lib/decision.core', () => ({ + createDecision: jest.fn(), +})) + +// @note the real Usage class runs end-to-end; only the downstream recorder is +// mocked so the final payload can be asserted. +jest.mock('@/lib/usage.record', () => ({ + recordLanguageTokenUsage: jest.fn(), +})) + +const { createDecision } = require('@/lib/decision.core') +const { recordLanguageTokenUsage } = require('@/lib/usage.record') + +const questions = { + refunded: { type: 'boolean', instructions: 'Was a refund issued?' }, +} + +const answers = { refunded: { type: 'boolean', probability: 0.99 } } + +describe('POST /api/v1/decision/create', () => { + beforeEach(() => { + jest.clearAllMocks() + }) + + it('returns the answers and debits only the priced input side', async () => { + const inputRatio = decisionModels.jev.pricing.inputTokenRatio + const expectedDebit = Math.round(100000 * inputRatio) + + createDecision.mockResolvedValue({ + answers, + usage: { model: 'jev', inputTokens: 100000, outputTokens: 21 }, + }) + + const stream = { abortSignal: { aborted: false }, result: jest.fn() } + const session = { user: { id: 'user-1' } } + const body = { model: 'jev', state: 'a refund was issued', questions } + + await handler({}, stream, session, body) + + expect(createDecision).toHaveBeenCalledWith( + 'a refund was issued', + questions, + { model: 'jev', signal: stream.abortSignal } + ) + + expect(recordLanguageTokenUsage).toHaveBeenCalledWith({ + user: session.user, + count: expectedDebit, + model: baseLanguageModel, + meta: { + reason: 'decision/create', + lineItems: [ + { + tokens: 100000, + model: 'jev', + type: 'input', + debit: expectedDebit, + ratio: inputRatio, + }, + ], + }, + references: undefined, + }) + + expect(stream.result).toHaveBeenCalledWith({ + answers, + usage: { model: 'jev', inputTokens: 100000, outputTokens: 21 }, + }) + }) + + it('accounts for a call worth less than one base token', async () => { + createDecision.mockResolvedValue({ + answers, + usage: { model: 'jev', inputTokens: 120, outputTokens: 21 }, + }) + + const stream = { abortSignal: { aborted: false }, result: jest.fn() } + const session = { user: { id: 'user-1' } } + + await handler({}, stream, session, { state: 'text', questions }) + + expect(recordLanguageTokenUsage).toHaveBeenCalledWith( + expect.objectContaining({ + count: 1, + meta: expect.objectContaining({ + lineItems: [ + expect.objectContaining({ tokens: 120, type: 'input', debit: 1 }), + ], + }), + }) + ) + }) + + describe('bodySchema', () => { + it.each([ + ['a string state', { state: 'text', questions }], + ['an object state', { state: { status: 'open' }, questions }], + [ + 'a message array state with choice and score questions', + { + model: 'jev', + state: [{ role: 'user', content: 'hi' }], + questions: { + topic: { + type: 'choice', + instructions: 'Topic?', + criteria: { billing: 'payments', technical: null }, + }, + quality: { + type: 'score', + instructions: 'Quality?', + criteria: ['poor', 'fair', 'good'], + }, + refunded: { + type: 'boolean', + instructions: 'Refunded?', + criteria: { true: 'money was returned', false: null }, + }, + }, + }, + ], + ])('accepts %s', (_name, body) => { + expect(bodySchema.validate(body).error).toBeUndefined() + }) + + it.each([ + ['a missing state', { questions }], + ['no questions', { state: 'text', questions: {} }], + [ + 'an unknown question type', + { state: 'text', questions: { q: { type: 'text', instructions: 'x' } } }, + ], + [ + 'a question without instructions', + { state: 'text', questions: { q: { type: 'boolean' } } }, + ], + [ + 'a choice question without criteria', + { state: 'text', questions: { q: { type: 'choice', instructions: 'x' } } }, + ], + [ + 'a score question with a single label', + { + state: 'text', + questions: { + q: { type: 'score', instructions: 'x', criteria: ['only'] }, + }, + }, + ], + [ + 'boolean criteria that describe only one side', + { + state: 'text', + questions: { + q: { type: 'boolean', instructions: 'x', criteria: { true: 'yes' } }, + }, + }, + ], + [ + 'a score question with more than ten levels', + { + state: 'text', + questions: { + q: { + type: 'score', + instructions: 'x', + criteria: Array.from({ length: 11 }, (_, i) => `level ${i}`), + }, + }, + }, + ], + [ + 'array criteria on a boolean question', + { + state: 'text', + questions: { + q: { type: 'boolean', instructions: 'x', criteria: ['a', 'b'] }, + }, + }, + ], + ['a model outside the decision catalogue', { model: 'gpt-image-2', state: 'text', questions }], + ])('rejects %s', (_name, body) => { + expect(bodySchema.validate(body).error).toBeDefined() + }) + }) +}) diff --git a/platform/pages/api/v1/decision/create.js b/platform/pages/api/v1/decision/create.js new file mode 100644 index 0000000..4ccd274 --- /dev/null +++ b/platform/pages/api/v1/decision/create.js @@ -0,0 +1,269 @@ +// @ts-check +import { createDecision } from '@/lib/decision.core' +import schema, { withSchema } from '@/lib/joi.handler' +import { withSessionLimits } from '@/lib/limit.handler' +import { withPost } from '@/lib/method' +import { withStream } from '@/lib/stream' +import { Usage } from '@/lib/usage.model' + +import decisionModelSchema from '@/schemas/decisionModel' + +const inputSchema = schema.alternatives( + schema.string(), + schema.object(), + schema.array() +) + +const criterionSchema = inputSchema.allow(null) + +const questionSchema = schema.object({ + type: schema.string().valid('boolean', 'choice', 'score').required(), + + instructions: inputSchema.required(), + + criteria: schema.when('type', { + switch: [ + { + is: 'boolean', + then: schema.object({ + true: criterionSchema.required(), + false: criterionSchema.required(), + }), + }, + { + is: 'choice', + then: schema + .object() + .pattern(schema.string(), criterionSchema) + .min(2) + .max(255) + .required(), + }, + { + is: 'score', + then: schema.array().items(criterionSchema).min(2).max(10).required(), + }, + ], + }), +}) + +export const bodySchema = schema.object({ + model: decisionModelSchema, + + state: inputSchema.required(), + + questions: schema + .object() + .pattern(schema.string(), questionSchema) + .min(1) + .required(), +}) + +/** + * @swagger + * + * /decision/create: + * post: + * operationId: createDecision + * summary: Answer typed questions about a state + * tags: + * - Decision + * requestBody: + * required: true + * content: + * application/json: + * schema: + * type: object + * properties: + * model: + * description: The decision model to use + * type: string + * state: + * description: The content to decide about, as text or a JSON object or array of related context + * oneOf: + * - type: string + * - type: object + * additionalProperties: true + * - type: array + * items: {} + * questions: + * description: The questions to answer keyed by a name of your choice + * type: object + * minProperties: 1 + * additionalProperties: + * $ref: '#/components/schemas/DecisionQuestion' + * required: + * - state + * - questions + * responses: + * 200: + * description: The decision was created successfully + * content: + * application/json: + * schema: + * type: object + * properties: + * answers: + * description: The answers keyed by the question names + * type: object + * additionalProperties: + * $ref: '#/components/schemas/DecisionAnswer' + * usage: + * type: object + * properties: + * model: + * description: The model that answered + * type: string + * inputTokens: + * type: number + * outputTokens: + * type: number + * required: + * - model + * - inputTokens + * - outputTokens + * required: + * - answers + * - usage + * default: + * $ref: '#/components/responses/ErrorResponse' + */ +export default withPost( + withSessionLimits( + ['token'], + withSchema( + bodySchema, + withStream(async function (_req, stream, session, body) { + const { model, state, questions } = body + + const { answers, usage } = await createDecision(state, questions, { + model, + signal: stream.abortSignal, + }) + + const usageRecorder = new Usage() + + usageRecorder.addDecisionTokens(usage.inputTokens, usage.model, 'input') + usageRecorder.addDecisionTokens( + usage.outputTokens, + usage.model, + 'output' + ) + + await usageRecorder.recordBaseTokens({ + user: session.user, + meta: { + reason: 'decision/create', + }, + }) + + await stream.result({ + answers, + usage, + }) + }) + ) + ) +) + +/** + * @manual Decisions + * @description Decisions answer typed questions about a piece of state and return probabilities, for classification, routing, rubric scoring and verification. + * @category Platform + * @tags decisions, models, classification, routing + * @index 14 + * + * A decision model reads a state and answers one or more typed questions about + * it. Unlike a language model it does not generate text: every answer is a + * structured value with probabilities, so code can act on it directly. Use it + * to classify a message, route a conversation, score something against a + * rubric, or verify that an outcome happened. + * + * ## Creating a Decision + * + * ```http + * POST /api/v1/decision/create + * Content-Type: application/json + * + * { + * "state": "I was charged twice and need this fixed today", + * "questions": { + * "urgent": { + * "type": "boolean", + * "instructions": "Does this convey urgency?" + * }, + * "topic": { + * "type": "choice", + * "instructions": "Which team should handle this?", + * "criteria": { + * "billing": "Payment or subscription issues", + * "technical": "Bugs or integration problems" + * } + * }, + * "frustration": { + * "type": "score", + * "instructions": "How frustrated is the customer?", + * "criteria": ["Calm", "Frustrated but civil", "Very angry"] + * } + * } + * } + * ``` + * + * The `state` is the content to decide about. It can be text, or a JSON object + * or array of related context, such as a record or a message history. The + * `questions` are keyed by a name of your choice and the answers come back + * under the same names. All questions are answered in one request. + * + * ## Question Types + * + * - **boolean**: a yes or no question. `criteria` is optional and, when given, + * describes what `true` and `false` mean. + * - **choice**: one of several named options. `criteria` maps each option name + * to a description, or to `null` when the name says enough. It takes between + * 2 and 255 options. + * - **score**: a level on an ordered scale. `criteria` lists the levels from + * lowest to highest and takes between 2 and 10 of them. + * + * `instructions`, and each criterion, accept text or a JSON object or array. + * + * ## Reading the Answers + * + * ```json + * { + * "answers": { + * "urgent": { "type": "boolean", "probability": 0.97 }, + * "topic": { + * "type": "choice", + * "choice": "billing", + * "probabilities": { "billing": 0.9, "technical": 0.1 } + * }, + * "frustration": { + * "type": "score", + * "score": 1.2, + * "probabilities": { "0": 0.1, "1": 0.6, "2": 0.3 } + * } + * }, + * "usage": { "model": "jev", "inputTokens": 120, "outputTokens": 8 } + * } + * ``` + * + * A boolean answer is the probability, from 0 to 1, that the answer is true. + * Compare it against a threshold that suits the cost of a wrong decision. A + * choice answer names the most likely option and the probability of each. A + * score answer is the probability-weighted level index, starting at 0, so it + * can fall between two levels. + * + * ## Choosing a Model + * + * The `model` field is optional and defaults to the deployment's default + * decision model. List the decision models a deployment serves with: + * + * ```http + * GET /api/v1/platform/model/list?type=decision + * ``` + * + * ## Usage + * + * Decisions are metered against the token limit on the tokens the model + * reports, using the same calibration as other model classes. + */ diff --git a/platform/pages/api/v1/integration/email/[emailIntegrationId]/_initiate.utest.js b/platform/pages/api/v1/integration/email/[emailIntegrationId]/_initiate.utest.js index d0119da..f28dd95 100644 --- a/platform/pages/api/v1/integration/email/[emailIntegrationId]/_initiate.utest.js +++ b/platform/pages/api/v1/integration/email/[emailIntegrationId]/_initiate.utest.js @@ -90,6 +90,10 @@ describe('POST /api/v1/integration/email/{emailIntegrationId}/initiate', () => { subject: 'Hello', text: ' ', }, + // @note the queue payload requires every field, so a missing one must + // fail here as a 400 rather than at enqueue time + {}, + { email: 'recipient@example.com', subject: 'Hello' }, ] for (const body of invalidBodies) { diff --git a/platform/pages/api/v1/integration/email/[emailIntegrationId]/initiate.ts b/platform/pages/api/v1/integration/email/[emailIntegrationId]/initiate.ts index 5eb6c90..8acacf5 100644 --- a/platform/pages/api/v1/integration/email/[emailIntegrationId]/initiate.ts +++ b/platform/pages/api/v1/integration/email/[emailIntegrationId]/initiate.ts @@ -14,9 +14,9 @@ import { import { INITIATE_EVENT_TYPE, sendEvent } from './queue' export const bodySchema = schema.object({ - email: schema.string().trim().email(), - subject: schema.string().trim().min(1), - text: schema.string().trim().min(1), + email: schema.string().trim().email().required(), + subject: schema.string().trim().min(1).required(), + text: schema.string().trim().min(1).required(), }) /** diff --git a/platform/pages/api/v1/platform/model/_list.utest.js b/platform/pages/api/v1/platform/model/_list.utest.js index 78ec782..5049ac1 100644 --- a/platform/pages/api/v1/platform/model/_list.utest.js +++ b/platform/pages/api/v1/platform/model/_list.utest.js @@ -2,6 +2,17 @@ import { imageModels, languageModels } from '@/config/models' import handler from './list' +jest.mock('@/config/models', () => { + const actual = jest.requireActual('@/config/models') + + return { + ...actual, + __esModule: true, + decisionModels: { jev: { provider: 'typesafe', visible: true } }, + defaultDecisionModel: 'jev', + } +}) + jest.mock('@/lib/method', () => ({ withGet: (fn) => fn, })) @@ -69,6 +80,14 @@ describe('/api/v1/platform/model/list', () => { ).toBe(true) }) + it('lists the decision models and marks the default', async () => { + const response = await handler({ query: { type: 'decision' } }) + + expect(response.items).toEqual([ + expect.objectContaining({ id: 'jev', type: 'decision', default: true }), + ]) + }) + it('rejects an unknown model type', async () => { await expect(handler({ query: { type: 'nope' } })).rejects.toThrow() }) diff --git a/platform/pages/api/v1/platform/model/list.js b/platform/pages/api/v1/platform/model/list.js index bdf912f..ea939a0 100644 --- a/platform/pages/api/v1/platform/model/list.js +++ b/platform/pages/api/v1/platform/model/list.js @@ -1,9 +1,11 @@ // @ts-check import { + defaultDecisionModel, defaultImageModel, defaultLanguageModel, defaultRerankModel, defaultVideoModel, + decisionModels, imageModels, languageModels, rerankModels, @@ -21,6 +23,7 @@ const modelCatalogues = { image: imageModels, video: videoModels, rerank: rerankModels, + decision: decisionModels, } // @note evaluated server-side, so these reflect the deployment's real, @@ -30,6 +33,7 @@ const modelDefaults = { image: defaultImageModel, video: defaultVideoModel, rerank: defaultRerankModel, + decision: defaultDecisionModel, } /** @@ -52,6 +56,7 @@ const modelDefaults = { * - image * - video * - rerank + * - decision * default: language * - in: query * name: cursor @@ -104,6 +109,7 @@ const modelDefaults = { * - image * - video * - rerank + * - decision * default: * description: Whether this model is the deployment's default for its type * type: boolean diff --git a/platform/pages/bots/[botId]/index.jsx b/platform/pages/bots/[botId]/index.jsx index 704b9df..10e32cc 100644 --- a/platform/pages/bots/[botId]/index.jsx +++ b/platform/pages/bots/[botId]/index.jsx @@ -461,7 +461,7 @@ function getBotExecutionSections( content: `import { ChatBotKit } from '@chatbotkit/sdk' const client = new ChatBotKit({ - secret: process.env.CHATBOTKIT_API_SECRET, + token: process.env.CHATBOTKIT_API_TOKEN, }) const botId = '${botId}' @@ -526,7 +526,7 @@ func main() { ctx := context.Background() client := sdk.New(sdk.Options{ - Secret: os.Getenv("CHATBOTKIT_API_SECRET"), + Token: os.Getenv("CHATBOTKIT_API_TOKEN"), }) botID := "${botId}" @@ -592,11 +592,11 @@ func main() { code: { language: 'bash', content: `# Required env vars: -# export CHATBOTKIT_API_SECRET="..." +# export CHATBOTKIT_API_TOKEN="..." # export BOT_ID="${botId}" API_BASE="${apiBase}" -AUTH_HEADER="Authorization: Bearer $CHATBOTKIT_API_SECRET" +AUTH_HEADER="Authorization: Bearer $CHATBOTKIT_API_TOKEN" JSON_HEADER="Content-Type: application/json" # 1) Completion stream (foreground) diff --git a/platform/pages/datasets/[datasetId]/index.jsx b/platform/pages/datasets/[datasetId]/index.jsx index 0b1b25b..300076a 100644 --- a/platform/pages/datasets/[datasetId]/index.jsx +++ b/platform/pages/datasets/[datasetId]/index.jsx @@ -786,7 +786,7 @@ function getDatasetSetupSections(datasetId, apiBase = getExternalAPIHostURL('/v1 content: `import { ChatBotKit } from '@chatbotkit/sdk' const client = new ChatBotKit({ - secret: process.env.CHATBOTKIT_API_SECRET, + token: process.env.CHATBOTKIT_API_TOKEN, }) const datasetId = '${datasetId}' @@ -845,7 +845,7 @@ func main() { ctx := context.Background() client := sdk.New(sdk.Options{ - Secret: os.Getenv("CHATBOTKIT_API_SECRET"), + Token: os.Getenv("CHATBOTKIT_API_TOKEN"), }) datasetID := "${datasetId}" @@ -898,11 +898,11 @@ func main() { code: { language: 'bash', content: `# Required env vars: -# export CHATBOTKIT_API_SECRET="..." +# export CHATBOTKIT_API_TOKEN="..." # export DATASET_ID="${datasetId}" API_BASE="${apiBase}" -AUTH_HEADER="Authorization: Bearer $CHATBOTKIT_API_SECRET" +AUTH_HEADER="Authorization: Bearer $CHATBOTKIT_API_TOKEN" JSON_HEADER="Content-Type: application/json" # Add a record directly diff --git a/platform/pages/integrations/widget/[widgetIntegrationId]/frame.jsx b/platform/pages/integrations/widget/[widgetIntegrationId]/frame.jsx index 70782b8..b8de1a9 100644 --- a/platform/pages/integrations/widget/[widgetIntegrationId]/frame.jsx +++ b/platform/pages/integrations/widget/[widgetIntegrationId]/frame.jsx @@ -70,8 +70,8 @@ import { getAccept } from '@/lib/mime' import { equal, merge, pick } from '@/lib/object' import { sleep } from '@/lib/promise' import { isComponent } from '@/lib/react' -import { captureUnknownError, isUnknownError } from '@/lib/response' import { textToEmojiSpans, wordsToSpans } from '@/lib/rehype.plugins' +import { captureUnknownError, isUnknownError } from '@/lib/response' import { saveBlob, saveUrl } from '@/lib/save' import { buildOriginRestrictedCsp } from '@/lib/security.headers' import { anyString, byteSlice, getRandomId, toPascalCase } from '@/lib/string' @@ -564,7 +564,6 @@ export function Form({ className, children, isLast, ...props }) { return [method, url, fields] }, [children]) - // eslint-disable-next-line react-hooks/rules-of-hooks const [target] = useDOMQuerySelector('#mainInputArea', { waitForElements: true, }) @@ -3728,8 +3727,8 @@ export function ReceivedMessages({ type === 'user' ? index === lastUserMessageIndex : type === 'bot' - ? index === lastBotMessageIndex - : false + ? index === lastBotMessageIndex + : false return (

diff --git a/platform/schemas/decisionModel.js b/platform/schemas/decisionModel.js new file mode 100644 index 0000000..cf51155 --- /dev/null +++ b/platform/schemas/decisionModel.js @@ -0,0 +1,14 @@ +// @ts-check +import schema from '@/lib/joi.schema' +import { parseDecisionModel } from '@/lib/model.utils' + +export default schema + .string() + .allow(null, '') + .custom((value) => { + if (value) { + parseDecisionModel(value) + } + + return value + }, 'model') diff --git a/platform/schemas/recordText.js b/platform/schemas/recordText.js index 7b9c98f..7eaaf11 100644 --- a/platform/schemas/recordText.js +++ b/platform/schemas/recordText.js @@ -2,5 +2,25 @@ import { MAX_DB_TEXT_BYTES_LENGTH } from '@/prisma/constraints' import schema from '@/lib/joi.schema' +import { normalizeText } from '@/lib/string' -export default schema.string().allow('').maxByteLength(MAX_DB_TEXT_BYTES_LENGTH) +const recordTextSchema = schema + .string() + .allow('') + .maxByteLength(MAX_DB_TEXT_BYTES_LENGTH) + +export default recordTextSchema + +// @note the vector store refuses a record without text, and normalization +// strips nonprintable characters, so text like "\u200b" passes a whitespace +// check yet arrives empty - validate what will be stored, not what was sent + +export const nonBlankRecordTextSchema = recordTextSchema.invalid('').custom( + (value, helpers) => + /\S/.test(normalizeText(value)) + ? value + : helpers.message({ + custom: '"text" must contain printable characters', + }), + 'non-blank' +) diff --git a/platform/schemas/recordText.utest.js b/platform/schemas/recordText.utest.js index b31b1d4..d998f1e 100644 --- a/platform/schemas/recordText.utest.js +++ b/platform/schemas/recordText.utest.js @@ -1,6 +1,8 @@ import { MAX_DB_TEXT_BYTES_LENGTH } from '@/prisma/constraints' -import recordTextSchema from '@/schemas/recordText' +import recordTextSchema, { + nonBlankRecordTextSchema, +} from '@/schemas/recordText' const itIfTextLengthIsConstrained = MAX_DB_TEXT_BYTES_LENGTH <= 1000000 ? it : it.skip @@ -104,4 +106,18 @@ describe('recordTextSchema', () => { expect(result.error).toBeDefined() expect(result.error.message).toContain('string') }) + + describe('nonBlankRecordTextSchema', () => { + it('should reject empty, whitespace and nonprintable-only text', () => { + expect(nonBlankRecordTextSchema.validate('').error).toBeDefined() + expect(nonBlankRecordTextSchema.validate(' \n ').error).toBeDefined() + expect( + nonBlankRecordTextSchema.validate('\u200b\u200c').error.message + ).toContain('printable') + }) + + it('should accept text with printable characters', () => { + expect(nonBlankRecordTextSchema.validate('hello').error).toBeUndefined() + }) + }) }) diff --git a/platform/scripts/build-api-spec.ts b/platform/scripts/build-api-spec.ts index 2be8c9a..1d627a0 100644 --- a/platform/scripts/build-api-spec.ts +++ b/platform/scripts/build-api-spec.ts @@ -34,6 +34,24 @@ import { import fs from 'node:fs' import { zodToJsonSchema } from 'zod-to-json-schema' +// @note inlined rather than declared as components: a component that is a +// union of primitives and objects has no name in the generated Go types +const decisionInputSchema = { + description: 'Text, or a JSON object or array of related context', + oneOf: [ + { type: 'string' }, + { type: 'object', additionalProperties: true }, + { type: 'array', items: {} }, + ], +} + +const decisionCriterionSchema = { + ...decisionInputSchema, + description: + 'A description of an option or level, or null when its name says enough', + nullable: true, +} + export const swaggerDefinitionV1 = { failOnErrors: true, @@ -1069,6 +1087,112 @@ export const swaggerDefinitionV1 = { }, ], }, + + DecisionQuestion: { + description: 'A typed question to answer about the state', + oneOf: [ + { + type: 'object', + properties: { + type: { type: 'string', enum: ['boolean'] }, + instructions: decisionInputSchema, + criteria: { + description: 'What a true and a false answer mean', + type: 'object', + properties: { + true: decisionCriterionSchema, + false: decisionCriterionSchema, + }, + required: ['true', 'false'], + }, + }, + required: ['type', 'instructions'], + }, + { + type: 'object', + properties: { + type: { type: 'string', enum: ['choice'] }, + instructions: decisionInputSchema, + criteria: { + description: + 'The options keyed by name, each with a description (2 to 255)', + type: 'object', + minProperties: 2, + maxProperties: 255, + additionalProperties: decisionCriterionSchema, + }, + }, + required: ['type', 'instructions', 'criteria'], + }, + { + type: 'object', + properties: { + type: { type: 'string', enum: ['score'] }, + instructions: decisionInputSchema, + criteria: { + description: + 'The levels ordered from lowest to highest (2 to 10)', + type: 'array', + minItems: 2, + maxItems: 10, + items: decisionCriterionSchema, + }, + }, + required: ['type', 'instructions', 'criteria'], + }, + ], + }, + + DecisionAnswer: { + description: 'The answer to a typed question', + oneOf: [ + { + type: 'object', + properties: { + type: { type: 'string', enum: ['boolean'] }, + probability: { + description: + 'The probability from 0 to 1 that the answer is true', + type: 'number', + }, + }, + required: ['type', 'probability'], + }, + { + type: 'object', + properties: { + type: { type: 'string', enum: ['choice'] }, + choice: { + description: 'The name of the most likely option', + type: 'string', + }, + probabilities: { + description: 'The probability of each option', + type: 'object', + additionalProperties: { type: 'number' }, + }, + }, + required: ['type', 'choice'], + }, + { + type: 'object', + properties: { + type: { type: 'string', enum: ['score'] }, + score: { + description: + 'The probability-weighted level index, starting at 0', + type: 'number', + }, + probabilities: { + description: 'The probability of each level keyed by its index', + type: 'object', + additionalProperties: { type: 'number' }, + }, + }, + required: ['type', 'score'], + }, + ], + }, }, responses: { diff --git a/platform/scripts/nuke-account.js b/platform/scripts/nuke-account.js index bcda6d4..5a6809a 100644 --- a/platform/scripts/nuke-account.js +++ b/platform/scripts/nuke-account.js @@ -322,6 +322,7 @@ async function nukeAccount(userId, skipSet = new Set(), dryRun = true) { }) // Initialize SDK client + // @note `secret` until the pinned @chatbotkit/sdk is one that accepts `token` const client = new ChatBotKit({ secret: token, }) diff --git a/platform/templates/onboarding.js b/platform/templates/onboarding.js index 6cda0b9..5afa7f5 100644 --- a/platform/templates/onboarding.js +++ b/platform/templates/onboarding.js @@ -1,6 +1,9 @@ // @ts-check +import { MAX_DB_STRING_BYTES_LENGTH } from '@/prisma/constraints' + import { resolveBuilderExperience } from '@/lib/experience' import { hostToHostname } from '@/lib/host.parse' +import { trimToByteLength } from '@/lib/string' import { getDocumentHost } from '@/hooks/useHost' import { getPartnerFromDocument } from '@/hooks/usePartner' @@ -13,6 +16,19 @@ const allSteps = [ '/new/channel', ] +/** + * @param {string|undefined} value + * @param {number} maxLength + * @returns {string|undefined} + */ +function clip(value, maxLength) { + if (typeof value !== 'string') { + return value + } + + return trimToByteLength(value.slice(0, maxLength), MAX_DB_STRING_BYTES_LENGTH) +} + /** * @type {import('./index').Template} */ @@ -56,13 +72,14 @@ export const template = { const { channel, organization, industry, role, goal, intent } = values // @note the values are clipped to the same limits /api/v1/me/update - // enforces so pre-seeded or stale values can never fail the save + // enforces so pre-seeded or stale values can never fail the save - the + // string columns are also byte-bound, which matters for non-latin names const { error: userUpdateError } = await fetch(`/api/v1/me/update`, { data: { - channel: channel?.slice(0, 64), - organization: organization?.slice(0, 128), - industry: industry?.slice(0, 64), - role: role?.slice(0, 64), + channel: clip(channel, 64), + organization: clip(organization, 128), + industry: clip(industry, 64), + role: clip(role, 64), goal: goal?.slice(0, 2048), }, loadingMessage: `Saving your information and preferences...`, diff --git a/platform/templates/onboarding.utest.js b/platform/templates/onboarding.utest.js index 60e8c95..04ad7f1 100644 --- a/platform/templates/onboarding.utest.js +++ b/platform/templates/onboarding.utest.js @@ -5,6 +5,10 @@ import { getPartnerFromDocument } from '@/hooks/usePartner' import template from '@/templates/onboarding' +jest.mock('@/prisma/constraints', () => ({ + MAX_DB_STRING_BYTES_LENGTH: 12, +})) + jest.mock('@/lib/experience', () => ({ resolveBuilderExperience: jest.fn(), })) @@ -107,6 +111,28 @@ describe('onboarding template', () => { ) }) + it('clips the customer details to the column byte limit', async () => { + await template.task({ + values: { + channel: 'website', + organization: 'Организация', + industry: 'Software', + role: 'Founder', + goal: 'Support', + }, + fetch: fetchMock, + }) + + expect(fetchMock).toHaveBeenCalledWith( + '/api/v1/me/update', + expect.objectContaining({ + data: expect.objectContaining({ + organization: 'Органи', + }), + }) + ) + }) + it('continues to template creation when an intent is selected', async () => { const result = await template.task({ values: { diff --git a/platform/tests/config/models.utest.js b/platform/tests/config/models.utest.js index bbfbba5..82d2de5 100644 --- a/platform/tests/config/models.utest.js +++ b/platform/tests/config/models.utest.js @@ -3,12 +3,14 @@ */ import { baseLanguageModel, + defaultDecisionModel, defaultImageModel, defaultLanguageModel, defaultRerankModel, defaultSpeechToTextModel, defaultTextToSpeechModel, defaultVideoModel, + decisionModels, imageModels, languageModels, rerankModels, @@ -28,6 +30,10 @@ const itIfImageModelsConfigured = Object.keys(imageModels).length ? it : it.skip const itIfVideoModelsConfigured = Object.keys(videoModels).length ? it : it.skip +const itIfDecisionModelsConfigured = Object.keys(decisionModels).length + ? it + : it.skip + // @note these tests assert properties of the model catalogue itself rather // than behaviour of the platform that reads it: the data is the thing under // test. @@ -79,6 +85,75 @@ describe('model catalogue', () => { } ) + itIfDecisionModelsConfigured( + 'defaultDecisionModel names a model the catalogue defines', + () => { + expect(decisionModels[defaultDecisionModel]).toBeDefined() + } + ) + + describe('decision model providers', () => { + const keys = [ + 'OPENROUTER_MODELS_API_KEY', + 'VERCEL_MODELS_API_KEY', + 'TYPESAFE_MODELS_API_KEY', + ] + + const original = Object.fromEntries(keys.map((k) => [k, process.env[k]])) + + afterEach(() => { + for (const key of keys) { + if (original[key] === undefined) { + delete process.env[key] + } else { + process.env[key] = original[key] + } + } + }) + + async function loadWith(configured) { + for (const key of keys) { + delete process.env[key] + } + + for (const key of configured) { + process.env[key] = 'test-key' + } + + jest.resetModules() + + const { decisionModels } = await import('@/config/models') + + return decisionModels + } + + it.each([ + [[], undefined, undefined], + [['OPENROUTER_MODELS_API_KEY'], 'openrouter', '~typesafe/jev-latest'], + [['VERCEL_MODELS_API_KEY'], 'vercel', 'typesafe-ai/jev'], + [['TYPESAFE_MODELS_API_KEY'], 'typesafe', 'jev-latest'], + [ + ['OPENROUTER_MODELS_API_KEY', 'VERCEL_MODELS_API_KEY'], + 'vercel', + 'typesafe-ai/jev', + ], + [ + [ + 'OPENROUTER_MODELS_API_KEY', + 'VERCEL_MODELS_API_KEY', + 'TYPESAFE_MODELS_API_KEY', + ], + 'typesafe', + 'jev-latest', + ], + ])('serves jev with %j through %s', async (configured, provider, providerModel) => { + const { jev } = await loadWith(configured) + + expect(jev?.provider).toBe(provider) + expect(jev?.providerModel).toBe(providerModel) + }) + }) + it.each([ ['baseLanguageModel', baseLanguageModel, languageModels], ['defaultRerankModel', defaultRerankModel, rerankModels],