Repository navigation
Bind benchmark resume to runtime and source identity - #3
Merged
Merged
Conversation
LifeOpsBench (Python) — smokeRun ID:
Full artifacts: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changing the runtime previously left successful-run signatures and subscription checkpoint namespaces unchanged. Bind both reuse paths to one execution identity containing repository revisions, runtime and harness source, benchmark/corpus source, and root lockfiles. Keep comparison grouping separate. An uninitialized Eliza submodule now reports no runtime revision, and resumed cohorts read the same SQLite location that records their runs.
Fixes #2. Needed for elizaOS/eliza#30684 context acceptance.
Validation: 36 focused filesystem/SQLite/identity/cohort tests pass; Ruff checks pass. The broader orchestrator and root suite produced 812 passes, 41 failures and 10 skips before the two additional source-identity cases. All 41 failures independently reproduce on unchanged main
7eb18762cb6b6d2c0f2d32eee0ab41e3bfc9e428; no complete-suite green claim. Corrected the adjacent cohort fixtures' old repository paths so they exercise the real database lifecycle. No live-model scores or published benchmark results are included.The first complete hosted LifeOps run failed only on an obsolete test requiring
--import tsxin the manifest command, which now uses Bun. Removed that launcher-string assertion; the Bun CI lane now actually launcheslifeops-bench:manifest --help. All five remaining manifest checks and the executable CLI smoke pass locally. This smoke verifies CLI startup, not regeneration against a runtime checkout. The separate LifeOps smoke job now explicitly uses offline conformance with the deliberately wrong static agent. The prior command selected the semantic live grader in a keyless job and failed for missing credentials. The corrected actual CLI completed all five smoke domains locally with expected zero scores and a non-publishable conformance artifact. Hosted CI is running on895bf09; earlier green runs do not qualify this revision.Final hosted validation at
895bf095ebf1d672d1b5e8737207daa0d06afb5b: all required jobs passed. Main CI 34913512755 passed Bun, Python syntax, root pytest and the LifeOps unit suite (1h6m44s). Separate LifeOps CI 34913512902 passed unit tests (53m3s) and the keyless smoke benchmark (19s); paid live benchmarking was intentionally not run.