Skip to content

Bind benchmark resume to runtime and source identity - #3

Merged
lalalune merged 4 commits into
mainfrom
fix/runtime-bound-benchmark-resume
Sep 15, 2026
Merged

lalalune merged 4 commits into
mainfrom
fix/runtime-bound-benchmark-resume

Conversation

@lalalune

@lalalune lalalune commented Sep 14, 2026 •

Copy link
Copy Markdown
Member

Changing the runtime previously left successful-run signatures and subscription checkpoint namespaces unchanged. Bind both reuse paths to one execution identity containing repository revisions, runtime and harness source, benchmark/corpus source, and root lockfiles. Keep comparison grouping separate. An uninitialized Eliza submodule now reports no runtime revision, and resumed cohorts read the same SQLite location that records their runs.

Fixes #2. Needed for elizaOS/eliza#30684 context acceptance.

Validation: 36 focused filesystem/SQLite/identity/cohort tests pass; Ruff checks pass. The broader orchestrator and root suite produced 812 passes, 41 failures and 10 skips before the two additional source-identity cases. All 41 failures independently reproduce on unchanged main 7eb18762cb6b6d2c0f2d32eee0ab41e3bfc9e428; no complete-suite green claim. Corrected the adjacent cohort fixtures' old repository paths so they exercise the real database lifecycle. No live-model scores or published benchmark results are included.

The first complete hosted LifeOps run failed only on an obsolete test requiring --import tsx in the manifest command, which now uses Bun. Removed that launcher-string assertion; the Bun CI lane now actually launches lifeops-bench:manifest --help. All five remaining manifest checks and the executable CLI smoke pass locally. This smoke verifies CLI startup, not regeneration against a runtime checkout. The separate LifeOps smoke job now explicitly uses offline conformance with the deliberately wrong static agent. The prior command selected the semantic live grader in a keyless job and failed for missing credentials. The corrected actual CLI completed all five smoke domains locally with expected zero scores and a non-publishable conformance artifact. Hosted CI is running on 895bf09; earlier green runs do not qualify this revision.

Final hosted validation at 895bf095ebf1d672d1b5e8737207daa0d06afb5b: all required jobs passed. Main CI 34913512755 passed Bun, Python syntax, root pytest and the LifeOps unit suite (1h6m44s). Separate LifeOps CI 34913512902 passed unit tests (53m3s) and the keyless smoke benchmark (19s); paid live benchmarking was intentionally not run.

@github-actions

Copy link
Copy Markdown

LifeOpsBench (Python) — smoke

Run ID: 34913512902
Result file: lifeops_conformance_20260915_012608.json

metric value
pass@1 0.000
pass@k 0.000
agent_cost_usd $0.0000
eval_cost_usd $0.0000
total_cost_usd $0.0000
total_latency_ms 0
scenarios_run 5
scenarios_skipped (cost / timeout) 0

Full artifacts: lifeops-smoke-34913512902 upload on this run.

@lalalune
lalalune merged commit 432f7c4 into main Sep 15, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bind benchmark resume and checkpoints to the evaluated runtime and source inputs

1 participant