Repository navigation
Publish reproducible performance results - #875
Merged
Merged
Conversation
scripts/benchmark.py drives hosted Sessions through the official SDK, times E2B with its SDK, and builds the table from the Session records and the Core log lines it keeps. docs/performance.md describes each measurement, the setup and how to reproduce it; the records it was built from are in docs/performance. The README shows the table beside E2B and the limits that apply. The backend comparison listed E2B as not suspending idle sandboxes; E2B pauses them in place.
…resume Measure every row with the final script, add the full-node and npm install rows and a run that resumes from disk, describe how the E2B comparison differs, and put the Docker and workspace limits in their owning docs.
…ecords The E2B run now records each sandbox's CPUs and memory, and build-template.py takes --memory-mb so the 4 GiB comparison template can be built from the repository. report fails when a measurement has no samples or the Core log yields no lines, and a hygiene check compares the published tables with report's output. The docs state the run conditions, the full list of E2B differences and the suspension triggers by link.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
scripts/benchmark.pydrives hosted Sessions through the official OpenAI SDK, times E2B with its SDK, and prints the results table from the Session records and the Core log lines it keeps.docs/performance/;python3 scripts/benchmark.py report docs/performanceprints the table again, andreport docs/performance/coldthe run that resumed from disk after the node's page cache was dropped.Measured on 2026-10-25 on the mx test installation with the final script: Core at c5d4205, node and Runtime at 170a498, 10 samples of each phase; E2B with a 2 vCPU, 4 GiB template like the microsandbox sandboxes.
Need help on this PR? Tag
@codesmith-botwith what you need.