Repository navigation
OpenRouter benchmarks: results and scripts - #15
Merged
Merged
Conversation
- RESEARCH: version 7 vs no skill on 40 synthetic ADHD scenarios and on the i-have-adhd eval suite, DeepSeek V4.1 Flash, Haiku and GLM judges - scripts/bench: resumable, logged, budget-capped runners; key read at run time - .gitignore: .env, run outputs and logs Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_014hA3pyEv4h4ntXdaCfeZ3R
Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_014hA3pyEv4h4ntXdaCfeZ3R
- --skill NAME=PATH (repeatable), --no-baseline, --gen, --judge, --judge-only, --budget, --inject system|wrap; conditions keyed by name plus text hash - shared plumbing in common.py; results and logs in results/ (git-ignored) - judge outputs validated and retried; no launchers, run from a shell (WSL) Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_014hA3pyEv4h4ntXdaCfeZ3R
There was a problem hiding this comment.
🟡 Changes recommended
Reproduction failures, ineffective budget enforcement, and judgement-handling defects undermine reliable benchmark runs.
5 open issues
What changed in this PR
Adds OpenRouter benchmark results and reproduction scripts for evaluating the skill against no-skill responses.
Changes:
- Adds two benchmark runners with blind model judging.
- Includes scenario datasets, scoring rubrics, and usage instructions.
- Publishes results and ignores credentials and generated outputs.
| File | Description |
|---|---|
scripts/bench/usefulness.md |
Defines usefulness scoring criteria. |
scripts/bench/README.md |
Documents benchmark usage and attribution. |
scripts/bench/ihad_rubric_block.md |
Adds weighted evaluation criteria. |
scripts/bench/ihad_cases.jsonl |
Includes 14 upstream evaluation cases. |
scripts/bench/cases_all.json |
Adds 40 synthetic scenarios. |
scripts/bench/bench.py |
Implements generation and pairwise judging. |
scripts/bench/bench_ihad.py |
Implements repeated weighted evaluations. |
README.md |
Summarizes benchmark evidence. |
docs/RESEARCH.md |
Reports results and limitations. |
.gitignore |
Excludes credentials and run artifacts. |
Review effort: Balanced
| # 2e: this repo's skill; ihad: a local copy of ayghri/i-have-adhd skills/i-have-adhd/SKILL.md (not shipped here) | ||
| SKILLS = {"ihad": "skill_ihad.md", | ||
| "2e": os.path.join("..", "..", "skills", "twice-exceptional-skill", "SKILL.md")} | ||
| CONDS = ["baseline", "ihad", "2e"] |
Comment on lines
+27
to
+30
| if len(sys.argv) >= 3 and sys.argv[1] == "judge": | ||
| JUDGE_MODEL = sys.argv[2] | ||
| OUT_J = "ihad_judgements_" + JUDGE_MODEL.split("/")[-1] + ".jsonl" | ||
| JUDGE_MAX_TOKENS = 12000 # reasoning judges need room before the JSON |
| "gen_model": "deepseek/deepseek-v4.1-flash-1017", | ||
| "judge_model": "anthropic/claude-haiku-5.5", | ||
| # the skill text is stored in the OpenRouter presets; the model is forced to gen_model for every condition | ||
| "conditions": {"none": "", "v6": "@preset/twice-v6", "v7": "@preset/twice-v7"}, |
Comment on lines
+119
to
+120
| if budget_check() >= 1: | ||
| raise RuntimeError("budget reached") |
|
|
||
| def judge(cases): | ||
| rubric = open(B.path("ihad_rubric_block.md"), encoding="utf-8").read() | ||
| reps = {r["job"]: r["text"] for r in B.load_jsonl(OUT_R) if r.get("ok")} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Results of the OpenRouter benchmarks for version 7 against no skill, plus the scripts to reproduce them.
Generated by Claude Code