diff --git a/AGENTS.md b/AGENTS.md index ca5aeab..ac8081b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -12,6 +12,7 @@ These instructions apply to maintaining this repository, not consuming apps. - `references/api_reference.md`: official endpoint links, not copied field tables. - `references/models.md`: selection defaults and current account-source pointers. - `references/examples.md`: ECNU-specific integration recipes. +- `references/workbuddy_setup.md`: WorkBuddy desktop custom-model configuration recipe. - `references/workflows.md`: symptom-driven, opt-in diagnostics. - `references/known_deviations.md`: dated live evidence; preserve dates and scope. - `references/agent_development.md`: historical guidance cited by old observations. diff --git a/README.md b/README.md index 58563cd..4820a3b 100644 --- a/README.md +++ b/README.md @@ -38,6 +38,7 @@ credentials should not prevent the Agent from writing or reviewing code. | Start a normal integration | [SKILL.md](SKILL.md) | | Find the relevant official endpoint contract | [Endpoint map](references/api_reference.md) | | Image edits, multimodal retrieval, LangChain embeddings, thinking/tool history, Anthropic SDK | [Integration recipes](references/examples.md) | +| WorkBuddy desktop custom-model setup | [WorkBuddy recipe](references/workbuddy_setup.md) | | Model choice or current account facts | [Model/account pointers](references/models.md) | | Diagnose a failure or select a live probe | [Targeted diagnosis](references/workflows.md) | | Inspect dated evidence | [Known deviations](references/known_deviations.md) | diff --git a/SKILL.md b/SKILL.md index 253efbf..8b92dd1 100644 --- a/SKILL.md +++ b/SKILL.md @@ -59,6 +59,7 @@ Its full messages URL ends in `/open/api/anthropic/v1/messages`. | Image/text embeddings or rerank | [Multimodal retrieval](references/examples.md#multimodal-retrieval): choose the model and preserve vector-space compatibility | | Anthropic SDK setup | [Anthropic recipe](references/examples.md#anthropic-sdk) | | Thinking and tool continuation | [Tool-history recipe](references/examples.md#thinking-and-tool-history) plus the linked ECNU contract | +| WorkBuddy desktop custom-model setup | [WorkBuddy recipe](references/workbuddy_setup.md) | | Model choice, prices, quotas, or deployment questions | [Model and account pointers](references/models.md) | | An error, unexpected result, or requested live check | [Targeted diagnosis](references/workflows.md); consult only the matching dated deviation | diff --git a/references/examples.md b/references/examples.md index c26cb08..2bb2a46 100644 --- a/references/examples.md +++ b/references/examples.md @@ -172,6 +172,15 @@ when thinking is enabled; thinking defaults to off. Pass ECNU extensions through `extra_body` with the OpenAI SDK. Do not substitute upstream template switches or silently map an unsupported tier between models. +For clients that cannot send `thinking`, the documented compatibility alias +`ecnu-reasoner` defaults to enabled thinking on `ecnu-max`; see the +[WorkBuddy recipe](workbuddy_setup.md) for one client-specific setup. + +The 2026-09-12 observations of [effort-only activation](known_deviations.md#ecnu-max-reasoning-effort-as-thinking-trigger) +and [extra max effort tiers](known_deviations.md#unavailable-reasoning-effort-tiers) +remain historical evidence. They do not override the current contract above or +justify rejecting `medium` on `ecnu-plus`. + For a tool exchange, append the complete actual assistant message before the matching tool results. ECNU documents preserving `reasoning_content` for thinking-mode tool calls through subsequent user turns. Keep it only in process diff --git a/references/known_deviations.md b/references/known_deviations.md index 9306a61..7fe01d0 100644 --- a/references/known_deviations.md +++ b/references/known_deviations.md @@ -23,6 +23,17 @@ langchain-openai 0.3.35, and httpx 0.28.1. Requests used synthetic text, 60-second timeouts, no retries, and the same cumulative 50-credit ceiling. Only response structure, field-preservation checks, and usage were retained. +Test environment `live-2026-09-12-c` used the account owner's personal tokens +(two distinct keys), Windows x64, Node.js 24.14.1 `fetch` direct HTTP, and +Asia/Shanghai local dates 2026-09-12 through 2026-09-14, during an +owner-authorized interactive integration session rather than the smoke runner. +Reliability runs were serial with roughly four-second spacing; unspaced bursts +are called out per entry. Repeated attempts in those runs were deliberate +characterization samples, not uncertainty retries, and credit consumption was +not metered against the runner's ceiling. Only synthetic arithmetic and +vision-fixture inputs were used. Prompts, generated content, reasoning text, +and credentials were not retained. + ## Invalid bearer on model discovery - **Tested at:** 2026-08-23; reproduced 2026-09-12 @@ -333,6 +344,101 @@ estimated cumulative consumption was 3.92822 credits. Neither is verified account debit. This targeted run does not revalidate other historical cases, other SDK versions, streaming, or arbitrary framework adapters. +## ecnu-reasoner alias default thinking + +- **Tested at:** 2026-09-12 +- **Environment:** live-2026-09-12-c, Node.js `fetch` direct HTTP +- **Protocol and endpoint:** OpenAI-compatible `POST /chat/completions` +- **Documented expectation:** The alias `ecnu-reasoner` equals `ecnu-max` with `thinking: {"type": "enabled"}`; a client must send the `thinking` parameter to activate thinking on `ecnu-max` itself. +- **Observed behavior:** A bare `ecnu-reasoner` request with no `thinking` field returned `200` and a non-zero `usage.completion_tokens_details.reasoning_tokens` count (range 18-198 across samples). A matching bare `ecnu-max` control returned `reasoning_tokens: 0`. The alias activates thinking server-side without any client-side `thinking` parameter. +- **Reproduction conditions:** Send two otherwise identical minimal requests, one with `model: "ecnu-reasoner"` and one with `model: "ecnu-max"`, both omitting `thinking`; compare `usage.completion_tokens_details.reasoning_tokens`. +- **Impact:** A client that relies on sending `thinking` to detect whether reasoning is active will misclassify `ecnu-reasoner` responses. The absence of `message.reasoning_content` is not a reliable thinking indicator either (see the prior "Max thinking response fields" entry). +- **Recommended fallback:** Detect active thinking via `usage.completion_tokens_details.reasoning_tokens`, not via `message.reasoning_content` or the presence of a client-side `thinking` parameter. Treat `ecnu-reasoner` as always-thinking for routing decisions. +- **Status:** active + +## ecnu-max reasoning_effort as thinking trigger + +**Current-contract note (2026-10-02):** The [thinking page](https://developer.ecnu.edu.cn/vitepress/llm/thinking.html) +still requires explicit `thinking` on `ecnu-max`. The observation below has +not been rerun against the current deployment; use the documented switch for +new integrations. Its original test date and status are retained. + +- **Tested at:** 2026-09-12 +- **Environment:** live-2026-09-12-c, Node.js `fetch` direct HTTP +- **Protocol and endpoint:** OpenAI-compatible `POST /chat/completions` +- **Documented expectation:** The thinking page states that `reasoning_effort` only takes effect when thinking mode is enabled via `thinking: {"type": "enabled"}`; without it, `reasoning_effort` is ignored. +- **Observed behavior:** `ecnu-max` requests with `reasoning_effort` set but no `thinking` field returned `200` with non-zero `usage.completion_tokens_details.reasoning_tokens`. Dose-response was clean across `low` (median ~117), `high` (~155), and `max` (~343). Adding `thinking: {"type": "enabled"}` alongside `reasoning_effort` did not increase the token count beyond the same-effort baseline. +- **Reproduction conditions:** Send `ecnu-max` requests with `reasoning_effort` set to `low`, `high`, and `max` but omit `thinking`; record `usage.completion_tokens_details.reasoning_tokens` for each. +- **Impact:** Clients following the documented precondition may unnecessarily send a `thinking` parameter, or may wrongly conclude thinking is off when only `reasoning_effort` is sent. The documented gating did not match the sampled deployment. +- **Recommended fallback:** Send the documented explicit `thinking` switch for `ecnu-max`; do not rely on this dated effort-only behavior. For a client that cannot send the switch, the documented `ecnu-reasoner` alias defaults to enabled thinking. +- **Status:** active + +## Unavailable reasoning effort tiers + +**Current-contract note (2026-10-02):** Current documented tiers are +`low` / `high` / `max` for `ecnu-max` and `low` / `medium` / `xhigh` for +`ecnu-plus`. The September observation below covers max only and has not +been rerun; it does not make `medium` invalid on plus or `xhigh` a supported +max default. Its original test date and status are retained. + +- **Tested at:** 2026-09-12 +- **Environment:** live-2026-09-12-c, Node.js `fetch` direct HTTP +- **Protocol and endpoint:** OpenAI-compatible `POST /chat/completions` +- **Documented expectation:** The thinking page lists `low`, `high`, and `max` as the three supported `reasoning_effort` values for `ecnu-max`. +- **Observed behavior:** `minimal` returned HTTP 500 on 5 of 7 attempts; `medium` returned HTTP 500 on 7 of 9 attempts. Both occasionally succeeded but cannot be relied on. `xhigh`, although absent from the documentation, returned `200` on all attempts with a reasoning-token median (~178) between `high` and `max`. `low`, `high`, and `max` were stable across all samples. +- **Reproduction conditions:** Send `ecnu-max` requests with each of `minimal`, `medium`, `xhigh`, `low`, `high`, and `max` as `reasoning_effort`; use serial requests with roughly four-second spacing to separate this check from the rapid-request error observation below. +- **Impact:** A client that sends `minimal` or `medium` (for example, via an SDK default or a UI selector that does not filter values) will hit intermittent 500 errors. `xhigh` is a usable but undocumented tier. +- **Recommended fallback:** Use each model's current documented tiers, not the extra tiers accepted by this historical deployment. Reject an unsupported value locally rather than silently remapping it to a different effort. +- **Status:** active + +## Rapid-request 401 metadata failure + +- **Tested at:** 2026-09-12 +- **Environment:** live-2026-09-12-c, Node.js `fetch` direct HTTP +- **Protocol and endpoint:** OpenAI-compatible `POST /chat/completions` and `GET /models` +- **Documented expectation:** `401` indicates an authentication failure; the official guidance only advises avoiding parallel calls. +- **Observed behavior:** Rapid sequential requests (under roughly one second apart) to either endpoint returned `401` with `{"detail": "获取第三方元数据失败"}` even though the same bearer passed spaced requests. Spacing requests by roughly four seconds eliminated the failures. The error shape is identical to the unsupported-model case above, making them indistinguishable without a working control. +- **Reproduction conditions:** Send the same valid bearer to the same documented model in a tight loop (sub-second spacing); then repeat with four-second spacing. +- **Impact:** A client may mistake this spacing-correlated error for a credential failure and trigger unnecessary key rotation. These observations do not establish a server-side rate-limit cause or whether rejected requests were billed. +- **Recommended fallback:** Inspect the complete error and chosen model first. If the request was explicitly rejected and a live check is authorized, one spaced control request can test whether the error persists. Do not automatically repeat a possibly accepted POST after a timeout or dropped connection, or rotate keys from this error shape alone. +- **Status:** active + +## Verified coverage on 2026-09-12 through 2026-09-14 + +**Current-contract note (2026-10-02):** This is historical coverage, not a +current capability test. The model page now specifies max as text-only with +512K context; use plus for image understanding and the current per-model +thinking tiers. No live revalidation was performed for this note. + +Environment `live-2026-09-12-c` made characterization requests across two +personal tokens on Windows x64, Node.js 24.14.1, over Asia/Shanghai dates +2026-09-12 through 2026-09-14. Requests were serial with roughly four-second +spacing except where unspaced bursts are noted. No credit metering was applied; +this was an owner-authorized interactive session, not a smoke-runner batch. + +Successful coverage included: + +- Both `ecnu-max` and `ecnu-reasoner` returned `200` for `reasoning_effort` + values `low`, `high`, `xhigh`, and `max`, with a clean dose-response curve + (median reasoning tokens: low ~117, high ~155, xhigh ~178, max ~343). +- `ecnu-reasoner` bare requests returned non-zero reasoning tokens; `ecnu-max` + bare requests returned zero. The alias activates thinking server-side. +- `ecnu-max` with `reasoning_effort` but no `thinking` parameter activated + thinking, contradicting the documented gating. +- `minimal` and `medium` efforts returned intermittent HTTP 500 and are + unreliable; `xhigh` is undocumented but stable. +- Both models correctly recognized a synthetic 96x96 red-circle PNG and + returned the expected "circle red" description, with and without thinking. +- Function calling with thinking coexisted normally on both models; a + two-turn tool exchange without preserved `reasoning_content` returned `200`. +- Rapid sub-second request bursts returned `401` + `{"detail": "获取第三方元数据失败"}`; four-second spacing eliminated it. + +No SDK, streaming, long-context, embedding, rerank, TTS, or image-generation +revalidation was performed in this run. The dose-response medians are +characterization samples, not quality benchmarks. Older observations retain +their original dates and statuses. + ## Update rules 1. Record the date, non-secret environment, protocol, endpoint, expectation, diff --git a/references/models.md b/references/models.md index a24c5dd..400c043 100644 --- a/references/models.md +++ b/references/models.md @@ -55,6 +55,12 @@ Upstream model cards can explain model design, but do not establish ECNU's request fields, thinking defaults, context units, output limits, performance, or deployment path. Do not copy upstream serving flags into requests. +For thinking controls, follow the [current recipe](examples.md#thinking-and-tool-history). +The [2026-09-12 effort observations](known_deviations.md#ecnu-max-reasoning-effort-as-thinking-trigger) +are scoped to that deployment; they are not current cross-model defaults. +The [WorkBuddy recipe](workbuddy_setup.md) covers clients that cannot send the +explicit thinking switch. + For a cost calculation, fetch current prices and show the input/output and cache assumptions. Do not assume a cache-hit ratio or treat estimated usage as verified account debit. A code-only task does not require a cost calculation. diff --git a/references/workbuddy_setup.md b/references/workbuddy_setup.md new file mode 100644 index 0000000..ec5303d --- /dev/null +++ b/references/workbuddy_setup.md @@ -0,0 +1,161 @@ +# WorkBuddy desktop integration + +Connect ECNU to WorkBuddy desktop as a custom OpenAI-compatible model source. +The desktop implementation and startup behavior below were inspected on +2026-09-12; they have not been revalidated on current WorkBuddy builds. +The configuration follows the [ECNU model contract](https://developer.ecnu.edu.cn/vitepress/llm/model.html) +checked on 2026-10-02, but this revised configuration has not been live-tested. + +## Configuration file + +The inspected implementation reads custom models from `%USERPROFILE%\.workbuddy\models.json` on +Windows (or `~/.workbuddy/models.json` on macOS/Linux). The path can be +redirected via the `WORKBUDDY_CONFIG_DIR` environment variable. + +The file **must be a top-level JSON array** of model objects, not the +`{"models": [...]}` object shape shown in the bundled CodeBuddy CLI +documentation. Two consumers share this file: + +- The CLI custom-models provider reads it via `extractConfig`, which accepts + either shape. +- The desktop local-model service reads it via `ModelsJsonStorage.update`, + which treats the file as an array: `Array.isArray(current) ? [...current] : []`. + On hardware-gate failure (CPU/GPU not in the whitelist), it rewrites the file + unconditionally. An object-shaped file is collapsed to `[]` on every startup. + +A bare array satisfies both: `extractConfig` returns `{models: parsed}` for an +array, and the purge's `filter(m => m?.local !== true)` preserves entries +without a `local` field. The `availableModels` field cannot be expressed in +array form; omit it to show all models. + +Do not save from WorkBuddy's Settings → Custom Models editor: its save path +uses `parseModelsJson`, which does not accept arrays and will overwrite the +file with an object shape, which is then purged to `[]` on the next restart. +Edit the JSON file directly. + +## Configuration example + +Use `ecnu-reasoner` for thinking and `ecnu-max` for ordinary text requests. +Both are currently text-only. The input/output limits below are conservative +client budgets: together they stay below the documented 512K context, without +assuming whether K means 1000 or 1024. They are not backend maximums. + +```json +[ + { + "id": "ecnu-reasoner", + "name": "ECNU Max (thinking)", + "vendor": "ECNU", + "url": "https://chat.ecnu.edu.cn/open/api/v1/chat/completions", + "apiKey": "sk-REDACTED", + "maxInputTokens": 440000, + "maxOutputTokens": 65536, + "supportsToolCall": true, + "supportsImages": false, + "supportsReasoning": true, + "reasoning": { + "defaultEffort": "max", + "supportedEfforts": ["low", "high", "max"], + "canDisableThinking": false + }, + "relatedModels": { + "lite": "ecnu-max", + "reasoning": "ecnu-reasoner" + }, + "onlyReasoning": true + }, + { + "id": "ecnu-max", + "name": "ECNU Max", + "vendor": "ECNU", + "url": "https://chat.ecnu.edu.cn/open/api/v1/chat/completions", + "apiKey": "sk-REDACTED", + "maxInputTokens": 440000, + "maxOutputTokens": 65536, + "supportsToolCall": true, + "supportsImages": false, + "supportsReasoning": false, + "relatedModels": { + "lite": "ecnu-max", + "reasoning": "ecnu-reasoner" + } + } +] +``` + +Never commit a real key. Store it in the file locally or reference an +environment variable via `"apiKey": "${ECNU_API_KEY}"`. + +## Field notes + +| Field | Value | Notes | +|---|---|---| +| `id` | `ecnu-max` / `ecnu-reasoner` | The id is also the `model` field sent to the API; do not change it to an upstream name like `deepseek-v4-flash`—ECNU rejects unknown ids with `{"detail":"获取第三方元数据失败"}`. | +| `url` | `…/v1/chat/completions` | Must end with `/chat/completions`. WorkBuddy auto-appends it unless `useCustomProtocol: true` is set. | +| `vendor` | `ECNU` or `Custom` | Cosmetic; does not affect routing. | +| `supportsReasoning` | `true` on reasoner, `false` on max | Enables the effort controls only for the thinking route in this recipe; it is not a statement that max lacks thinking support. | +| `reasoning.defaultEffort` | `max` on reasoner only | The effort sent when no session or global override is set. | +| `reasoning.supportedEfforts` | `["low","high","max"]` | Current documented max/reasoner tiers; the September 12 `xhigh` result is [historical](known_deviations.md#unavailable-reasoning-effort-tiers). | +| `reasoning.canDisableThinking` / `onlyReasoning` | `false` / `true` on reasoner only | Hide the "Off" entry for this default-thinking route. Select the separate max entry for ordinary requests; these UI settings do not establish that the server cannot disable thinking. | +| `relatedModels.lite` | `ecnu-max` | Routes background work to the ordinary text entry. Verify that session/global overrides do not inject thinking or effort. | +| `relatedModels.reasoning` | `ecnu-reasoner` | Model used when WorkBuddy internally needs a reasoning model. | +| `temperature` | omit | ECNU documents that sampling parameters may be ignored under thinking mode. Omitting it lets the server use its default. | +| `useCustomProtocol` | `false` (or omit) | When `true`, WorkBuddy uses the URL as-is without appending `/chat/completions`. Leave `false` for ECNU. | + +## How effort reaches the API + +The September 12 implementation resolved effort in this priority order: + +1. Session override (UI "Deep Thinking" selector or `/effort ` command) +2. Global `reasoningEffort` setting (persisted by `/effort`) +3. Model entry `reasoning.defaultEffort` +4. Model entry `reasoning.effort` +5. Fallback `high` if thinking is enabled and no effort is found + +The value is passed through `withSupportedEffortsFallback`, which builds an +identity `thinkingLevelMap` from `supportedEfforts` (each value maps to itself). +Values not in the map are passed through unchanged, so the map is not request +validation. + +The `/effort` command accepts all six WorkBuddy tiers +(`minimal`/`low`/`medium`/`high`/`xhigh`/`max`) without filtering by +`supportedEfforts`. Only the UI selector respects the filter. For these +max/reasoner entries, use only `low`, `high`, or `max`; other tiers are outside +the current documented contract. Clear session/global overrides when checking +the ordinary max route, and inspect the actual request. + +## Why `thinking` is not in the request + +In the inspected build, WorkBuddy's `thinking-format-translator` was the code +path that injected `thinking: {"type": "enabled"}`. It triggered only +when the model is in WorkBuddy's built-in capability catalog (a models.dev +snapshot bundled in `codebuddy.js`, ~1061 entries) and has a `thinkingFormat` +value. `ecnu-max` and `ecnu-reasoner` were absent from that catalog, so this +custom-model route did not send the `thinking` parameter. + +This recipe uses the documented default-thinking alias `ecnu-reasoner` for +that route. Current ECNU guidance requires `thinking` to enable thinking on +`ecnu-max`; do not rely on the [September 12 effort-only observation](known_deviations.md#ecnu-max-reasoning-effort-as-thinking-trigger). +If a newer WorkBuddy build adds custom thinking controls, recheck its outgoing +request before changing this routing. + +## Verification + +The inspected build hot-reloaded `models.json` within ~1 second. Check the +daemon log for `Loaded custom models config from user: …models.json` with no +error. The model selector should show both entries with a `custom` tag. + +To confirm thinking is active at runtime, inspect the response's +`usage.completion_tokens_details.reasoning_tokens`. Do not rely on +`message.reasoning_content` alone—it was sometimes `null` with thinking active +in the [September 12 observations](known_deviations.md#max-thinking-response-fields). + +## Historical rapid-request errors + +On September 12, sub-second sequential requests returned `401` with +`{"detail": "获取第三方元数据失败"}`, while requests spaced roughly four seconds +apart succeeded ([observation](known_deviations.md#rapid-request-401-metadata-failure)). +This association does not establish a rate limit or prove the credential is +valid. After an explicitly rejected request, a spaced control request may help +diagnose it. Keep requests serial; do not automatically retry a possibly +accepted POST after a timeout or dropped connection. diff --git a/references/workflows.md b/references/workflows.md index 5dd043a..1d82068 100644 --- a/references/workflows.md +++ b/references/workflows.md @@ -18,8 +18,12 @@ These links are dated observations, not claims that the issue still reproduces. | Missing auth produces an unexpected error | Preserve actual status and body type: [missing authorization](known_deviations.md#missing-authorization-on-model-discovery) | | A new model ID appears but fails | Discovery is not endpoint capability: [runtime-only model](known_deviations.md#undocumented-model-visible-at-runtime) | | `401` only for an unsupported model or `[1m]` | Compare a documented working control; do not rotate keys blindly: [unsupported model](known_deviations.md#unsupported-chat-model-error), [suffix conditions](known_deviations.md#anthropic-long-context-suffix-metadata) | +| `401` with `获取第三方元数据失败` under rapid requests | Compare the dated spacing observation; the error alone does not identify the cause: [rapid-request 401](known_deviations.md#rapid-request-401-metadata-failure) | | SDK works for chat but embeddings fail | Check raw strings and local validation: [embedding recipe](examples.md#langchain-embeddings) | | Tool continuation loses state or reasoning fields | Preserve the actual message and inspect serialization: [recipe](examples.md#thinking-and-tool-history), [field variation](known_deviations.md#max-thinking-response-fields) | +| `reasoning_content` is absent but thinking seems active | Check `usage.completion_tokens_details.reasoning_tokens`, not `message.reasoning_content`: [field variation](known_deviations.md#max-thinking-response-fields), [alias default thinking](known_deviations.md#ecnu-reasoner-alias-default-thinking) | +| `ecnu-max` thinking will not activate | Send the documented explicit switch; [effort-only activation](known_deviations.md#ecnu-max-reasoning-effort-as-thinking-trigger) was observed in September, not revalidated for the current backend | +| An effort tier fails | Use the chosen model's [documented tiers](examples.md#thinking-and-tool-history); the [September max observation](known_deviations.md#unavailable-reasoning-effort-tiers) does not apply to plus | | TTS error parsing crashes | Tolerate non-JSON errors: [invalid voice](known_deviations.md#invalid-tts-voice-error-shape) | | PCM bytes arrive without format metadata | Configure the format explicitly: [missing headers](known_deviations.md#successful-tts-response-headers) | | Old max-vision success conflicts with current docs | Current max is text-only; the [September 12 fixture](known_deviations.md#direct-ecnu-max-image-input) predates the September 18 contract change |