Skip to content

Fix r3 strict deduction-trace schema compatibility / 修复 r3 严格扣分依据 Schema 兼容性 - #65

Merged
yyuan308 merged 1 commit into
mainfrom
codex/r3-strict-schema-compat
Aug 25, 2026
Merged

yyuan308 merged 1 commit into
mainfrom
codex/r3-strict-schema-compat

Conversation

@yyuan308

Copy link
Copy Markdown
Collaborator

Summary / 摘要

  • Make the r3 deduction_trace_v1 output schema compatible with Codex strict structured output: nullable trace and attention fields are required structurally, while remaining semantically optional when not applicable.
  • 使 r3 deduction_trace_v1 输出 schema 兼容 Codex 严格结构化输出:扣分依据和注意事项字段在结构上必填,但在不适用时可为 null,其评分语义不变。
  • Normalize only the provider spelling deduction_trace: [] to null for full-credit leaves. Non-full leaves with an empty trace still fail validation.
  • 仅将满分叶子项中的 deduction_trace: [] 规范化为 null;非满分叶子项的空 trace 仍会严格校验失败。

Validation / 验证

  • python -m unittest discover -s tests/benchmark/core — 477 passed.
  • Real zero-data r3 contract probes passed for both Codex and official stable DeepSeek V4Flash. The probes used only synthetic content and were automatically deleted.
  • 真实零数据 r3 契约探针已分别通过 Codex 和官方稳定版 DeepSeek V4Flash;探针仅使用合成内容并自动删除。

Privacy and reproducibility / 隐私与可复现性

  • This PR contains only public runner, schema, and synthetic-test changes. No Data/, submissions, identities, transcripts, gold scores, raw model output, or metrics are included.
  • 本 PR 仅包含公共运行器、schema 和合成测试改动;不包含 Data/、作答、身份信息、转录、gold、原始模型输出或指标。
  • Existing interrupted r3 attempts are private audit artifacts only and do not count as experimental results.
  • 已中断的 r3 尝试仅保留为私有审计材料,不计入实验结果。

Limitations and next step / 局限与下一步

  • This fixes output-contract compatibility only; it does not change the frozen rubric, human review decisions, or make any accuracy claim.
  • 本修复仅解决输出契约兼容性;不改动冻结 rubric、人工复核决定,也不作任何准确率主张。
  • After review and merge, rebuild the private r3 development packets with new hashes, then run M1-Codex, G1-Codex, and G1-DeepSeek V4Flash with aggregate-only progress. Heldout remains untouched.
  • 审核合并后,将用新 hash 重建私有 r3 开发集 packet,再运行 M1-Codex、G1-Codex 与 G1-DeepSeek V4Flash,仅报告聚合进度;heldout 保持不访问。

@yyuan308
yyuan308 merged commit 9a618c7 into main Aug 25, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant