diff --git a/README.md b/README.md index 330a2de..d818d1a 100644 --- a/README.md +++ b/README.md @@ -114,6 +114,14 @@ Use $deepseek-coding-delegation to delegate this implementation and review the r Codex selects the relevant repository instructions, passes bounded `scope_paths`, task-specific `required_reads`, and compact `critical_constraints`, then independently reviews the resulting diff and tests. Large rule documents can be limited to named sections instead of being copied into the prompt. +## Token and latency expectations + +Delegation is a quality and independent-implementation tool, not a token-saving shortcut. In a controlled 27-case synthetic benchmark, the optimized routing policy consumed more combined Codex + DeepSeek tokens than direct Codex execution at every task size. It was cheaper than forced delegation for small and medium tasks, but not for large tasks. + +Use direct execution when minimizing total tokens or latency is the primary goal. Delegate when an independent implementation pass, unfamiliar multi-file discovery, or a bounded second perspective is worth the controller and review overhead. + +See [the benchmark report](./docs/token-routing-benchmark-2026-09-14.md) for methodology, medians, failures, and limitations. + ## Safety model - Codex owns scope, authorization, business decisions, review, verification, and goal completion. diff --git a/README.zh-CN.md b/README.zh-CN.md index c89fdd5..10def8a 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -114,6 +114,14 @@ npm --prefix plugins/deepseek-code-agent run test:live Codex 会选择相关仓库规则，传入明确的 `scope_paths`、任务相关 `required_reads` 和精简的 `critical_constraints`，然后独立审核 diff 与测试。较大的规则文档可以只指定相关章节，不需要整份复制进提示词。 +## Token 与耗时预期 + +委派的主要价值是独立实现与质量复核，不是节省 Token。在一轮包含 27 个最终可比较结果的受控合成基准中，优化后的 policy 在小型、中型和大型任务上，Codex + DeepSeek 总路径 Token 都高于 Codex 直接完成。policy 在小型和中型任务上低于强制委派，但大型任务反而更高。 + +如果首要目标是降低总 Token 或缩短耗时，应优先直接执行。只有当跨文件探索、独立实现视角或第二轮工程判断的价值足以覆盖控制器与复核开销时，才建议委派。 + +方法、各档中位数、失败尝试和适用限制见[基准报告](./docs/token-routing-benchmark-2026-09-14.md)。 + ## 安全边界 - Codex 负责范围、授权、业务决定、审核、验证和目标完成状态。 diff --git a/docs/token-routing-benchmark-2026-09-14.md b/docs/token-routing-benchmark-2026-09-14.md new file mode 100644 index 0000000..bef7424 --- /dev/null +++ b/docs/token-routing-benchmark-2026-09-14.md @@ -0,0 +1,46 @@ +# Token Routing Benchmark — 2026-09-14 + +This report measures whether routing Codex implementation work through DeepSeek reduces total model tokens. It is a project-maintainer benchmark, not an independent evaluation or a universal model ranking. + +## Setup + +- Nine synthetic coding tasks: three small, three medium, and three large. +- Three fresh-task variants per task: `direct`, optimized `policy`, and `forced` DeepSeek delegation. +- The same Codex model and reasoning effort for every run. +- Every task started from the same pinned Git commit in a disposable isolated worktree. +- No database, external business service, Docker service, browser, worker, or scheduler was used. +- A run passed only when its complete prescribed validation sequence passed. Failed attempts were recorded separately and never counted as token savings. +- `route_total = codex_total + deepseek_total`. Cached tokens remain part of the processed-token total and are also recorded separately. + +After retrying failed attempts in fresh tasks, all 27 task/variant slots had a successful comparable result. There were 38 attempts in total: 27 successful attempts and 11 failures. + +## Median successful result by task size + +| Size | Direct route tokens | Policy route tokens | Policy savings vs direct | Forced route tokens | Forced savings vs direct | +|---|---:|---:|---:|---:|---:| +| Small | 329,752 | 735,032 | -122.90% | 1,758,139 | -433.17% | +| Medium | 509,666 | 2,634,784 | -416.96% | 3,900,068 | -665.22% | +| Large | 1,058,360 | 5,549,401 | -424.34% | 3,695,768 | -249.20% | + +A negative saving means the route used more tokens than direct execution. Expressed as multiples, policy used approximately 2.23x, 5.17x, and 5.24x the direct route tokens for small, medium, and large tasks. + +## Median wall time + +| Size | Direct | Policy | Forced | +|---|---:|---:|---:| +| Small | 115.7 s | 188.9 s | 335.7 s | +| Medium | 137.0 s | 465.8 s | 524.4 s | +| Large | 206.6 s | 549.9 s | 443.2 s | + +Raw attempt success rates, including failed attempts, were 64.29% for small, 90.00% for medium, and 64.29% for large tasks. Failures included environment-harness issues, validation failures, and worker/controller rework; they are part of the operational overhead rather than savings. + +## Interpretation + +- Direct execution was the lowest-token and lowest-latency route in every size bucket. +- Policy routing improved on forced delegation for small and medium tasks by 58.19% and 32.44% respectively, but used 50.16% more route tokens than forced delegation for large tasks. +- Policy therefore reduces unnecessary delegation relative to `forced` in some workloads, but it did not make delegation cheaper than direct execution in this benchmark. +- Use delegation for the value of an independent implementation pass, bounded repository discovery, or a second engineering perspective. Do not advertise or select it primarily as a token-saving feature. + +## Limitations + +The sample contains nine synthetic tasks from one repository and one fixed tool/model configuration. Results can change with task shape, prompts, models, caching, controller behavior, and provider pricing. Future claims should be based on repeated benchmarks across more repositories and should continue to report failures and retries rather than filtering them out. diff --git a/plugins/deepseek-code-agent/README.md b/plugins/deepseek-code-agent/README.md index adf4f54..9899b2b 100644 --- a/plugins/deepseek-code-agent/README.md +++ b/plugins/deepseek-code-agent/README.md @@ -32,3 +32,7 @@ npm run test:live ``` The live test makes one small DeepSeek request in a temporary Git repository and removes the temporary worktree afterward. + +## Routing expectation + +Delegation is intended for independent implementation and review value, not token savings. A controlled benchmark found higher combined Codex + DeepSeek token use than direct Codex execution across small, medium, and large tasks. Prefer direct execution when minimizing total tokens or latency is the primary goal; delegate when the extra implementation perspective justifies the coordination overhead. See the repository-level benchmark report for the measured results and limitations. diff --git a/plugins/deepseek-code-agent/skills/deepseek-coding-delegation/SKILL.md b/plugins/deepseek-code-agent/skills/deepseek-coding-delegation/SKILL.md index 6474e91..1ffadf2 100644 --- a/plugins/deepseek-code-agent/skills/deepseek-coding-delegation/SKILL.md +++ b/plugins/deepseek-code-agent/skills/deepseek-coding-delegation/SKILL.md @@ -1,6 +1,6 @@ --- name: deepseek-coding-delegation -description: "Delegate implementation work to the stateful DeepSeek coding agent while Codex remains responsible for scope, review, corrections, and verification. Use for coding tasks and active /goal coding objectives that contain a bounded implementation subtask. Do not use for explanation-only requests or when the user explicitly asks Codex to implement without delegation." +description: "Delegate non-trivial bounded implementation work to the stateful DeepSeek coding agent while Codex remains responsible for scope, review, corrections, and verification. Use when a coding task or active /goal has enough implementation work to outweigh delegation overhead. Route tiny edits directly unless the user explicitly requires a DeepSeek model call." --- # DeepSeek Coding Delegation @@ -9,9 +9,11 @@ Use the `ds_*` MCP tools to treat DeepSeek V4.1 Flash Max as the implementation ## When to delegate -- For an active `/goal` whose objective includes coding, delegate a bounded implementation unit when the DeepSeek tools are available. +- Treat delegation as a quality and independent-implementation mechanism, not as a token-saving mechanism. Controlled benchmark results show higher combined Codex + DeepSeek token use than direct execution across small, medium, and large tasks. +- Work directly when minimizing total tokens or latency is the primary goal. +- Work directly when the expected change is one self-contained function or at most about 30 lines in one file, needs no unfamiliar repository discovery, and has a trivial focused check. +- Delegate a bounded implementation unit when it spans multiple behaviors or files, needs unfamiliar repository discovery, benefits from an independent implementation pass, or is an active coding `/goal` with enough work to amortize controller and review round trips. - Prefer one DeepSeek worker. Do not also create a Codex model subagent unless the user explicitly asks for another model. -- Handle tiny edits directly when delegation would require more context than the implementation itself. - Do not delegate analysis-only, review-only, status, or explanation requests unless implementation is also requested. ## Control loop