Skip to content

OpenRouter benchmarks: results and scripts - #15

Merged
JohnnyTriggerDev merged 3 commits into
mainfrom
openrouter-bench
Oct 18, 2026
Merged

JohnnyTriggerDev merged 3 commits into
mainfrom
openrouter-bench

Conversation

@JohnnyTriggerDev

Copy link
Copy Markdown
Owner

Results of the OpenRouter benchmarks for version 7 against no skill, plus the scripts to reproduce them.

  • RESEARCH: new section "OpenRouter benchmarks"
  • README: one line in Evidence
  • scripts/bench: bench.py (40 synthetic ADHD scenarios) and bench_ihad.py (the i-have-adhd eval suite, MIT, credited); no personal paths, key read at run time
  • .gitignore for .env and run outputs

Generated by Claude Code

- RESEARCH: version 7 vs no skill on 40 synthetic ADHD scenarios and on the
  i-have-adhd eval suite, DeepSeek V4.1 Flash, Haiku and GLM judges
- scripts/bench: resumable, logged, budget-capped runners; key read at run time
- .gitignore: .env, run outputs and logs

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_014hA3pyEv4h4ntXdaCfeZ3R
Copilot AI balanced review requested due to automatic review settings October 18, 2026 22:06
- --skill NAME=PATH (repeatable), --no-baseline, --gen, --judge, --judge-only,
  --budget, --inject system|wrap; conditions keyed by name plus text hash
- shared plumbing in common.py; results and logs in results/ (git-ignored)
- judge outputs validated and retried; no launchers, run from a shell (WSL)

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_014hA3pyEv4h4ntXdaCfeZ3R

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Reproduction failures, ineffective budget enforcement, and judgement-handling defects undermine reliable benchmark runs.

5 open issues
What changed in this PR

Adds OpenRouter benchmark results and reproduction scripts for evaluating the skill against no-skill responses.

Changes:

  • Adds two benchmark runners with blind model judging.
  • Includes scenario datasets, scoring rubrics, and usage instructions.
  • Publishes results and ignores credentials and generated outputs.
File Description
scripts/​bench/​usefulness.md Defines usefulness scoring criteria.
scripts/​bench/​README.md Documents benchmark usage and attribution.
scripts/​bench/​ihad_rubric_block.md Adds weighted evaluation criteria.
scripts/​bench/​ihad_cases.jsonl Includes 14 upstream evaluation cases.
scripts/​bench/​cases_all.json Adds 40 synthetic scenarios.
scripts/​bench/​bench.py Implements generation and pairwise judging.
scripts/​bench/​bench_ihad.py Implements repeated weighted evaluations.
README.md Summarizes benchmark evidence.
docs/​RESEARCH.md Reports results and limitations.
.gitignore Excludes credentials and run artifacts.

Review effort: Balanced

Comment thread scripts/bench/bench_ihad.py Outdated
# 2e: this repo's skill; ihad: a local copy of ayghri/i-have-adhd skills/i-have-adhd/SKILL.md (not shipped here)
SKILLS = {"ihad": "skill_ihad.md",
"2e": os.path.join("..", "..", "skills", "twice-exceptional-skill", "SKILL.md")}
CONDS = ["baseline", "ihad", "2e"]
Comment thread scripts/bench/bench_ihad.py Outdated
Comment on lines +27 to +30
if len(sys.argv) >= 3 and sys.argv[1] == "judge":
JUDGE_MODEL = sys.argv[2]
OUT_J = "ihad_judgements_" + JUDGE_MODEL.split("/")[-1] + ".jsonl"
JUDGE_MAX_TOKENS = 12000 # reasoning judges need room before the JSON
Comment thread scripts/bench/bench.py Outdated
"gen_model": "deepseek/deepseek-v4.1-flash-1017",
"judge_model": "anthropic/claude-haiku-5.5",
# the skill text is stored in the OpenRouter presets; the model is forced to gen_model for every condition
"conditions": {"none": "", "v6": "@preset/twice-v6", "v7": "@preset/twice-v7"},
Comment thread scripts/bench/bench.py Outdated
Comment on lines +119 to +120
if budget_check() >= 1:
raise RuntimeError("budget reached")
Comment thread scripts/bench/bench_ihad.py Outdated

def judge(cases):
rubric = open(B.path("ihad_rubric_block.md"), encoding="utf-8").read()
reps = {r["job"]: r["text"] for r in B.load_jsonl(OUT_R) if r.get("ok")}
@JohnnyTriggerDev
JohnnyTriggerDev merged commit b0cd16e into main Oct 18, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants