Interactive report · Completed notebook · Results and limitations
A comparison of Jev 1.13.0 and 11 classical classification pipelines across eight datasets, using three training seeds and a shared test set. The published run uses protocol 3.0.1 (V3).
The V4 additional tests are in development: new policy, temporal, and iterative-decision experiments responding to Reddit feedback, with separate Jev/local notebooks. V4 does not repeat the completed benchmark. The results below and the public report remain V3.
Jev's strongest result is IMDb sentiment classification: 96.3% raw balanced accuracy, compared with 88.4% for the best classical pipeline in this run. Results elsewhere are mixed; classical pipelines lead on all four tabular datasets. This is a bounded-budget comparison.
| If you want to… | Open |
|---|---|
| Explore the scores | Interactive report |
| Read the findings and caveats | Results and interpretation |
| Inspect the executed run | V3 notebook |
| Download the tables | Raw CSV · Adjusted CSV |
| Understand the methodology | V3 protocol · Common V2 protocol |
| Check provenance | Manifest and hashes · Validation record |
| Download the frozen source | V3 release bundle |
| Work on the repository | Development guide |
Threshold-adjusted Jev uses labeled policy data for binary tasks, so those scores are not zero-shot end to end. Seed standard deviations are not confidence intervals. The full Kaggle result archive and per-example diagnostics are unavailable in this release; see the full limitations.
- Upload the V3 notebook to a fresh session. Enable Internet and two T4 GPUs; select the latest GPU environment.
- Add and enable the Kaggle secret
TYPESAFE_API_KEY. - Choose a new output
ROOTif changing the configuration. The notebook contains the exact source used for the published run. - Run extraction, setup, preparation, and classical ML cells first. The separate Jev cell makes paid API calls.
- Save the generated results ZIP before ending the session. It includes diagnostics and predictions beyond the displayed tables.
The saved run reports 38,922 request attempts. Its approximately $4.19 input-cost value is a configured estimate, not an invoice or total billed cost. Counts, retries, cache reuse, prices, and snapshots can change on a new run; use the notebook's budget controls.
jevbench/ Benchmark implementation used by V2 and V3
jevbench_v4/ V4 additional-task adapters, draft policy pairs, and comparators
notebooks/ Published V3 notebook and historical V1/V2 notebooks
bundles/ Original downloadable source and notebook archives
published_results/ Tables extracted from the completed run and provenance
scripts/ Notebook builders, result export, report build, diagnostics
tests/ Artifact, backend, process, and integration checks
docs/ Report website, protocols, results, and historical notes
legacy/ Original V1 benchmark implementation
examples/ Standalone API smoke test and sample request
From the repository root:
python -m pip install -r requirements-dev.txt
python -m tests.validate_v3_artifact
python -m tests.validate_v3_backends
python -m tests.validate_gpu_processThese checks need no API key or GPU. Python 3.12 is recorded in the completed notebook. See the development guide for build commands, additional checks, and the source-module map.
The notebook index explains the earlier experiments. All historical artifacts are retained. Fresh builds go into ignored generated/; local results and caches stay in ignored results/.