Skip to content

fix: restore extracted benchmark campaign execution and evidence paths - #8

Merged
lalalune merged 11 commits into
mainfrom
fix/benchmark-campaign-contracts-30684
Sep 15, 2026
Merged

lalalune merged 11 commits into
mainfrom
fix/benchmark-campaign-contracts-30684

Conversation

@lalalune

Copy link
Copy Markdown
Member

Repairs the extracted benchmark campaign so its CLI resolves the checkout correctly, explicit mock-command smoke runs do not require unused OpenCode, campaign inventory describes the current suite layout, and producers, resume, publication, and the viewer share the repository's benchmark_results store. Nonempty legacy result stores fail with an actionable migration error instead of being silently mixed or ignored. Lifecycle scoring now attests the installed contract and scenario files under suites/.

Closes #4. Closes #5. Closes #6. Closes #7.

Built on #3; preserve its runtime-sensitive resume identity and keyless LifeOps workflow repair. This PR contains no live benchmark scores and does not relax publication requirements.

Validation:

  • PYTHONPATH=<checkout-parent> ELIZA_TRAINING_ROOT=<restored-training-root> python3 -m pytest tests suites/orchestrator/tests -q: 873 passed, 152.72 seconds on 9cb9b16. The subsequent merge of Bind benchmark resume to runtime and source identity #3 changes only its previously tested LifeOps workflow.
  • Actual CLI tests execute outside the repository working directory. Mock-command smoke and missing-live-provider cases exercise the real subprocess boundary.
  • Producer-to-viewer coverage retains the full 125k-character trajectory prompt through SQLite and the viewer. Legacy-store rejection leaves existing data untouched.
  • Action-calling corpus was reconstructed from immutable public Hermes snapshot dae3e1d28cfbcf4b915c04ea1e072030529b4bda using the historical converter. All 11,578 rows reproduce the pinned SHA-256 78dd3bc0b8cc98e0637231ae96dee7a63dbe1b5ae3e1b4fd7d862733ff5e162b; no expected hash was changed to accept a different dataset.
  • Registry adapter tests: 155 passed. The original source fails its real installed lifecycle-contract attestation; the repaired path passes.
  • Ruff passes affected Python sources. Hosted checks for this PR are still pending.

The ledger now records 72 entries (58 adapters and 14 direct suites). Full comparative campaign counts remain inventory, not completed evaluation; unknown corpus sizes and manual/non-agent lanes remain explicit.

@github-actions

Copy link
Copy Markdown

LifeOpsBench (Python) — smoke

Run ID: 34917542768
Result file: lifeops_conformance_20260915_030643.json

metric value
pass@1 0.000
pass@k 0.000
agent_cost_usd $0.0000
eval_cost_usd $0.0000
total_cost_usd $0.0000
total_latency_ms 0
scenarios_run 5
scenarios_skipped (cost / timeout) 0

Full artifacts: lifeops-smoke-34917542768 upload on this run.

@lalalune
lalalune merged commit cecd345 into main Sep 15, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant