Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,12 @@ Codex selects the relevant repository instructions, passes bounded `scope_paths`

## Queen token efficiency

### Route by task complexity

Choose the route from reasoning, risk, and exploration/verification burden, not line or file counts. **Small** known edits that take a few minutes stay with Codex. **Medium** mechanical work with clear acceptance criteria prefers one persistent worker. **Large or high-risk** work — state machines, concurrency, permissions, transactions, migrations — may still delegate the labor, but Codex must first fix the design, interfaces, invariants, failure semantics and acceptance cases, then review the actual diff and run independent verification; a worker's self-report or passing self-tests are never acceptance. An explicit user request to delegate or to stay direct overrides these defaults.

A 2026-09-15 pilot paired one direct and one delegated execution per size from the same commit: [task routing pilot](./docs/task-routing-pilot-2026-09-15.md). It ran one execution per cell in parallel without alternating order, so it is not the repeated benchmark the queen protocol asks for, and its numbers neither generalize nor promise a fixed saving. In that pilot, delegated worker non-cached input plus output was 13.78% below the three direct runs, while the controller's own interval was 55.41% below the direct proxy total; the controller interval also covers managing the native control arm and reviewing both runs, so it is not a pure delegation bill. Worker totals including cached reads were higher than direct totals, so combined tokens and latency can still increase; medium and large worker usage is marked incomplete.

### Design, implement, verify, review

Codex reads the critical call chain and decides interfaces, invariants and acceptance cases before delegation. An optional `task_spec` gives the worker those decisions and the exact check commands. The worker completes implementation, self-tests and routine repairs in one session. Codex reviews the patch and batches feedback; after two unsuccessful correction rounds it can take over after confirming the worker and verification have stopped.
Expand Down
6 changes: 6 additions & 0 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,12 @@ Codex 会选择相关仓库规则,传入明确的 `scope_paths`、任务相关

## Queen Token 优化

### 按任务复杂度路由

路由依据判断量、风险以及探索与验证负担,而不是行数或文件数。**小型**已知改动几分钟可完成,由 Codex 直接处理;**中型**机械实现且验收标准清晰,优先交给一个持久执行者;**大型或高风险**任务(状态机、并发、权限、事务、迁移等)可以把实现交给执行者,但 Codex 必须先确定设计、接口、不变量、失败语义和验收案例,再审查实际 diff 并独立验证,绝不按 worker 自报或自测通过来接受。用户明确要求委派或直接处理时,以用户要求为准。

2026-09-15 的 pilot 在同一基线 commit 上为每个规模各做了一次 direct 与委派执行:[任务分级 pilot](./docs/task-routing-pilot-2026-09-15.md)。每格仅一次、双臂并行、顺序未交替,因此它不是 queen 协议要求的三次重复正式基准,结果不能推广,也不承诺固定节省。该 pilot 中,worker 非缓存输入加输出合计比三次 direct 低 13.78%,controller 自身区间的 proxy 比 direct proxy 合计低 55.41%;controller 区间也包含原生对照管理和两边审查,不能当作纯委派账单。计入缓存后 worker total 高于 direct total,总 Token 和耗时仍可能增加;medium 和 large 的 worker 用量标记为不完整。

### Codex 设计与审核,GLM 实现

Codex 先阅读关键调用链,明确接口、必须保持的行为和验收案例,通过可选 `task_spec` 一次交付设计决定和检查命令。执行模型在同一会话中完成实现、自测和常规修复,Codex 集中审核并批量反馈;两轮修正仍未通过时,确认 worker 和验收进程停止后再接管。
Expand Down
202 changes: 202 additions & 0 deletions docs/task-routing-pilot-2026-09-15.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,202 @@
{
"schema_version": 1,
"pilot_id": "task-routing-pilot-2026-09-15",
"kind": "transparent_single_pair_pilot",
"date": "2026-09-15",
"baseline_commit": "90d84be225fe2751ed0583917e7391ff29245559",
"task_sizes": ["small", "medium", "large"],
"tasks": {
"small": {
"task_id": "notice-export-content-disposition-ascii-fallback",
"definition": "Content-Disposition fallback ASCII whitelist hardening",
"scope": ["notice_export_service", "service test"]
},
"medium": {
"task_id": "notification-priority-dirty-input-degradation",
"definition": "Notification priority dirty-input stable degradation and enrich pure-function extraction",
"scope": ["service", "new test"]
},
"large": {
"task_id": "export-lease-thread-safe-success-cooldown",
"definition": "Thread-safe export lease with success-commit cooldown, wired into three routes and covering failure rollback",
"scope": ["service", "2 routes", "tests"]
}
},
"design": {
"arms": ["direct_codex", "delegated_deepseek"],
"repetitions_per_cell": 1,
"execution": "parallel",
"order_alternated": false,
"matches_queen_protocol": false,
"queen_protocol_requirement": "The queen protocol asks for at least three alternated repetitions per variant; this pilot must not be presented as that formal benchmark.",
"wall_time_used_for_token_reduction": false
},
"unit_conventions": {
"codex_total": "input + output, where input is inclusive of cached_input",
"codex_proxy": "(input - cached_input) + output",
"worker_total": "input + cache_read + output + reasoning",
"worker_proxy": "input + output",
"reduction_percent": "1 - compared_proxy / direct_proxy_total, rounded to two decimals",
"wall_time_seconds": "observed duration from the recorded wall_time_source; never used in token reduction calculations"
},
"direct_codex": {
"small": {
"commit": "90d84be225fe2751ed0583917e7391ff29245559",
"input": 554251,
"cached_input": 508032,
"output": 3508,
"reasoning": 1034,
"total": 557759,
"acceptance": true,
"corrections": 0,
"wall_time_seconds": 108.921,
"wall_time_source": "Codex session first-to-last timestamp",
"controller_acceptance": {
"tests_total": 32,
"ruff": "passed",
"format": "passed",
"diff": "passed"
}
},
"medium": {
"commit": "90d84be225fe2751ed0583917e7391ff29245559",
"input": 812011,
"cached_input": 751872,
"output": 7019,
"reasoning": 2209,
"total": 819030,
"acceptance": true,
"corrections": 0,
"wall_time_seconds": 178.325,
"wall_time_source": "Codex session first-to-last timestamp",
"controller_acceptance": {
"tests_total": 51,
"ruff": "passed",
"format": "passed",
"diff": "passed"
}
},
"large": {
"commit": "90d84be225fe2751ed0583917e7391ff29245559",
"input": 1696112,
"cached_input": 1626240,
"output": 15444,
"reasoning": 4958,
"total": 1711556,
"acceptance": true,
"corrections": 0,
"wall_time_seconds": 379.631,
"wall_time_source": "Codex session first-to-last timestamp",
"controller_acceptance": {
"tests_service": 21,
"tests_api": 49,
"ruff": "passed",
"format": "passed",
"diff": "passed"
}
}
},
"delegated_deepseek": {
"small": {
"commit": "90d84be225fe2751ed0583917e7391ff29245559",
"input": 26939,
"cache_read": 333952,
"output": 3475,
"reasoning": 1842,
"total": 366208,
"cost_usd": 0.008232906,
"acceptance": true,
"corrections": 0,
"usage_complete": true,
"wall_time_seconds": 210.517,
"wall_time_source": "worker creation through controller verification",
"controller_acceptance": {
"tests_total": 41,
"ruff": "passed",
"format": "passed",
"diff": "passed"
}
},
"medium": {
"commit": "90d84be225fe2751ed0583917e7391ff29245559",
"input": 27079,
"cache_read": 554240,
"output": 8597,
"reasoning": 3316,
"total": 593232,
"cost_usd": 0.01287237,
"acceptance": true,
"corrections": 0,
"usage_complete": false,
"wall_time_seconds": 414.657,
"wall_time_source": "worker creation through controller verification",
"controller_acceptance": {
"tests_total": 120,
"ruff": "passed",
"format": "passed",
"diff": "passed"
}
},
"large": {
"commit": "90d84be225fe2751ed0583917e7391ff29245559",
"input": 84304,
"cache_read": 5069568,
"output": 23944,
"reasoning": 18216,
"total": 5196032,
"cost_usd": 0.053150304,
"acceptance": true,
"corrections": 1,
"usage_complete": false,
"wall_time_seconds": 811.445,
"wall_time_source": "worker creation through controller verification",
"wall_time_includes_correction_rounds": 1,
"controller_acceptance": {
"tests_service": 25,
"tests_api": 53,
"ruff": "passed",
"format": "passed",
"diff": "passed"
}
}
},
"controller_interval": {
"description": "Codex controller increment from launching the six executions through acceptance of both arms",
"input": 6358712,
"cached_input": 6287488,
"output": 18933,
"reasoning": 6389,
"total": 6377645,
"noncached_input": 71224,
"proxy": 90157,
"includes": ["native control-arm management", "review of both arms"],
"pure_delegation_bill": false
},
"derived": {
"direct_proxy_total": 202201,
"worker_proxy_total": 174338,
"worker_proxy_reduction_vs_direct_percent": 13.78,
"controller_proxy_total": 90157,
"controller_proxy_reduction_vs_direct_percent": 55.41,
"direct_total_sum": 3088345,
"worker_total_sum": 6155472
},
"review_corrections": [
{
"arm": "delegated_deepseek",
"task_size": "large",
"rounds": 1,
"finding": "Codex review found the first revision let a stale lease break a new lease and let empty results enter cooldown.",
"resolution": "The worker corrected both defects; the revised patch passed review and independent verification and was accepted."
}
],
"limitations": [
"One execution per size and arm; the arms ran in parallel and the order was not alternated, so this is not the repeated benchmark the queen protocol requests.",
"One repository, one task set, one model configuration and one date; results do not generalize and promise no fixed or billing-grade saving.",
"The controller interval covers launch of all six executions through acceptance of both arms, including native control-arm management and review of both arms, so it is not a pure delegation bill.",
"Worker usage for medium and large is marked usage_complete=false; those worker totals and costs may understate actual usage.",
"Worker total including cached reads (6155472) exceeds direct total (3088345), so combined tokens and latency can increase.",
"Both arms ran in parallel, so each wall time can include contention for the same machine and provider capacity; the durations are single observations, not a stable performance baseline.",
"This pilot is separate from the 2026-09-14 combined-token benchmark and must not be merged with it."
]
}
83 changes: 83 additions & 0 deletions docs/task-routing-pilot-2026-09-15.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# Task Routing Pilot — 2026-09-15

A transparent single-pair pilot of the task routing policy. It records one direct Codex execution and one delegated DeepSeek worker execution for each of a small, medium and large task, all from the same baseline commit `90d84be225fe2751ed0583917e7391ff29245559`. The machine-readable source is [task-routing-pilot-2026-09-15.json](./task-routing-pilot-2026-09-15.json); this page summarizes it and adds no new measurements.

This is not a formal benchmark. There is exactly one execution per cell, both arms ran in parallel, and the order was not alternated. The queen protocol asks for at least three alternated repetitions per variant, so this pilot does not satisfy it. Do not generalize these numbers, promise a fixed saving, or merge them with the [2026-09-14 combined-token report](./token-routing-benchmark-2026-09-14.md). Combined tokens and latency can still increase.

## Tasks

| Size | Task ID | Definition | Scope |
|---|---|---|---|
| small | notice-export-content-disposition-ascii-fallback | Content-Disposition fallback ASCII whitelist hardening | notice_export_service, service test |
| medium | notification-priority-dirty-input-degradation | Notification priority dirty-input stable degradation and enrich pure-function extraction | service, new test |
| large | export-lease-thread-safe-success-cooldown | Thread-safe export lease with success-commit cooldown, wired into three routes and covering failure rollback | service, 2 routes, tests |

The JSON records these IDs and definitions under `tasks`, matched to the three sizes.

## Raw results

### Direct Codex

| Size | Input | Cached input | Output | Reasoning | Total | Acceptance | Corrections |
|---|---:|---:|---:|---:|---:|---|---:|
| small | 554251 | 508032 | 3508 | 1034 | 557759 | true | 0 |
| medium | 812011 | 751872 | 7019 | 2209 | 819030 | true | 0 |
| large | 1696112 | 1626240 | 15444 | 4958 | 1711556 | true | 0 |

Direct total = input + output, where input includes cached input. These are model usage counters, not a billing statement.

### Delegated DeepSeek worker

| Size | Input | Cache read | Output | Reasoning | Total | Cost (USD) | Acceptance | Corrections | Usage complete |
|---|---:|---:|---:|---:|---:|---:|---|---:|---|
| small | 26939 | 333952 | 3475 | 1842 | 366208 | 0.008232906 | true | 0 | true |
| medium | 27079 | 554240 | 8597 | 3316 | 593232 | 0.01287237 | true | 0 | false |
| large | 84304 | 5069568 | 23944 | 18216 | 5196032 | 0.053150304 | true | 1 | false |

Worker total = input + cache_read + output + reasoning. Medium and large are marked `usage_complete=false`; their totals and costs may understate actual usage. The provider-reported cost is retained as raw data and is not converted into a saving claim.

### Codex controller interval

From launching the six executions through accepting both arms, the controller increment was: input 6358712, cached input 6287488, output 18933, reasoning 6389, total 6377645, non-cached input 71224, proxy 90157.

This interval covers the whole two-arm experiment, including managing the native control arm and reviewing both arms. It is not a pure delegation bill and must not be presented as one.

### Wall time and independent acceptance

| Size | Direct wall time (s) | Delegated wall time (s) | Direct acceptance | Delegated acceptance |
|---|---:|---:|---|---|
| small | 108.921 | 210.517 | 32 tests | 41 tests |
| medium | 178.325 | 414.657 | 51 tests | 120 tests |
| large | 379.631 | 811.445 | 21 service + 49 API tests | 25 service + 53 API tests |

Wall time sources are recorded per execution in the JSON: direct durations span the Codex session's first-to-last timestamp; delegated durations run from worker creation through controller verification. The delegated large duration includes its one correction round. Independent controller verification passed for all six executions, with ruff, format and diff checks passing for each. Wall time values are single observations from arms that ran in parallel, so they can reflect contention for the same machine and provider capacity; they are not a stable performance baseline and are never used in token reduction calculations.

## Derived comparisons

Definitions: direct proxy = (input − cached input) + output; worker proxy = input + output; reduction = 1 − compared proxy / direct proxy total.

| Quantity | Value |
|---|---:|
| Direct proxy total (three tasks) | 202201 |
| Worker proxy total (three tasks) | 174338 |
| Worker proxy reduction vs direct proxy total | 13.78% |
| Controller proxy | 90157 |
| Controller proxy reduction vs direct proxy total | 55.41% |
| Direct total sum | 3088345 |
| Worker total sum (cache included) | 6155472 |

The 55.41% is a conservative controller-side comparison inside this pilot: the controller interval also absorbs native control-arm management and review of both arms, so it is neither a pure delegation bill nor a promised saving. Worker proxy was 13.78% below the direct proxy total, but worker total including cached reads was 6155472 against 3088345 direct, so combined tokens can increase. No fixed saving is claimed, here or elsewhere.

## Corrections and failures

All six executions were accepted. The three direct executions each needed 0 corrections, and the delegated small and medium executions were accepted with 0 corrections. The delegated large execution was rejected on first review: the first revision let a stale lease break a new lease and let empty results enter cooldown. The worker corrected both defects, and the revised patch passed review and independent verification. Nothing was discarded, and `usage_complete=false` for medium and large is retained as reported.

## Limitations

- One execution per size and arm, parallel, non-alternating. This is not the repeated formal benchmark required by the queen protocol and supports no statistical claim.
- One repository, one task set, one model configuration and one date. Nothing here generalizes, and no billing-grade conclusion follows.
- The controller interval includes native control-arm management and both-arm review, so 55.41% must not be treated as pure delegated savings.
- Worker usage completeness is false for two of three tasks, so the worker-side comparison may understate worker cost.
- Cache-inclusive worker total exceeded direct total, so combined tokens and latency may increase even when the worker proxy looks lower.
- Both arms ran in parallel, so each wall time can include contention for the same machine and provider capacity; the durations are single observations, not a stable performance baseline, and they feed no token reduction calculation.
- This pilot is independent of the earlier combined-token benchmark, and the two must not be mixed.
Loading
Loading