A small, transparent runner for comparing current models through the Raytone OpenAI-compatible API. The repository publishes the prompts, request defaults, and artifact schema used by the benchmark page at raytone.ai/benchmarks/token-routing.
This baseline records request status, latency, token usage, the returned model identifier, and a SHA-256 digest of the output. It does not claim a quality score. Inspect the manifest and add task-specific validation before using it for a model-selection decision.
- Node.js 22 or newer
- A Raytone API key
- One or more model identifiers from the current model catalog
export RAYTONE_API_KEY="your-key"
export RAYTONE_MODELS="model-a,model-b"
node benchmark.mjsTo target another OpenAI-compatible endpoint:
export RAYTONE_BASE_URL="https://api.raytone.ai/v1"The command writes a uniquely named JSON artifact in the current directory and returns a non-zero exit code when any request fails. API keys and raw response text are never written to the artifact.
The initial manifest contains three deterministic prompts:
- Structured extraction
- Code repair
- Instruction following
manifest.json is the source of truth for the published workload and output
fields. The runner embeds the same manifest so a downloaded single file remains
reproducible.
- Compare results from the same region and similar time window.
- Run multiple repetitions before interpreting latency.
- Treat provider-reported token fields as part of the measured response.
- Add validators for your own quality requirements.
- Record the model catalog and runner commit used for each shared result.
The runner is available under the MIT License. The published benchmark manifest is available under CC BY 4.0.