Add Convex Decision Evals

[mikecann]

Project: https://www.convex.dev/evals/decision (source: https://github.com/get-convex/convex-evals, Apache-2.0)

Where Jev fits: Jev answers 108 four-option multiple-choice questions about the Convex backend platform through OpenRouter's /api/alpha/decisions with native typed Choice questions, and is ranked alongside 14 LLMs on accuracy, latency, and cost.

Details:

• The questions are verified and drawn from 90 of the 112 Convex coding evals. Random guessing is 25%.
• No guidelines in the prompt and no tools. LLMs run at low reasoning effort and answer via structured output limited to A-D. Each run asks every question 3 times with shuffled options. The leaderboard score is the mean of each model's last 3 full runs.
• Latest results (23 Sept 2026, 15 models): typesafe/jev-1.13 scores 84.6% ±0.7 at 199 ms median per question and $0.0088 per full run. The top model, GPT-6 Astra, scores 98.0% at 2.12 s and $1.59 per run.
• Per-answer explorer with every question, each model's choice, Jev's probabilities, and the raw request/response: https://convex-evals.netlify.app
• Decision runner code: runner/decisions/ in the repo.

Disclosure: I'm Mike Cann. I work at Convex and built the decision evals. Parts were built with help from AI coding agents.

• I added one object to data/projects.json and ran npm run validate
• I did not commit README.md or site/index.html (CI regenerates them)
• I am the author, or I linked where the project was announced

🤖 Generated with Claude Code

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

Summary by CodeRabbit

• New Features
  • Added Convex Decision Evals to the project listings, with a leaderboard comparing Jev against 14 LLMs across 108 Convex questions.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 0700221e-68dd-4251-a39a-d687a00f07ef

📥 Commits

Reviewing files that changed from the base of the PR and between 1149aac and c1505e1.

📒 Files selected for processing (1)

• data/projects.json

Included review availability: Your plan provides up to 2 included reviews per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The project catalog adds Convex Decision Evals, a TypeScript research project with a listed site, zero stars, and an added date of 2026-09-24.

Changes

Project catalog entry

Layer / File(s)	Summary
Add Convex Decision Evals data/projects.json	Adds the project’s repository, site, category, language, star count, and added date to the projects array.

Priority: ⬇️ Low

Estimated code review effort: 1 (Trivial) | ~3 minutes

Change: Feature

Suggested reviewers: scald

Merge Risk: ⚪ Minimal · up to c1505

This change adds a project listing, and no material merge risk is established. It appears ready under normal checks.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the main change: adding the Convex Decision Evals project.
Description check	✅ Passed	The description includes the required Project and Where Jev fits sections, confirms the required validation and file-change checks, and identifies the author or project source. It provides relevant im…
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[scald]

Landed on main as 97b7510 with you as the commit author, now live on https://awesomejev.com. Closing rather than merging because every PR appends to the same data file and they conflict with each other; landing directly keeps your authorship. Thanks!