inter-rater-reliability

Measure whether your model judge agrees with human raters. Chance-corrected agreement statistics, bootstrap confidence intervals, and calibration gates for Swift Testing and CI. Zero dependencies.

A self-hosted multi-rater data labeling platform for structured annotation workflows.

Labeling queue library for managing human labeling workflows

A study in "Agent Trajectory Evaluation": judging an AI agent's full Execution Path, Not just its final Answer. A small ReAct Agent generates Trajectories; ~40 are hand-labelled as Ground truth; 'Rule-based and LLM-judge scorers' are measured against 'those labels'. The Deliverable is the agreement analysis, and where Automated Evaluation breaks.

Statistical validity checks for human-graded AI evaluations

Inter-rater agreement study: 40 legal questions with gold references, an anchored rubric, four LLM judges, and the statistics that say whether they measure the same thing

Design-aware temporal reliability auditing for human annotation studies

Cross-model LLM-as-judge eval harness: validate AI judges with Fleiss' kappa / Krippendorff's alpha, not accuracy. Ships a real 7-model panel (Claude, GPT, Gemini, Grok, Qwen, DeepSeek, GLM) you can replay in ~30s, no API key. MIT.

Agent skills for decisions under uncertainty, plus the evaluation harness that measures them: pre-registered predictions enforced by git ancestry, blind LLM-as-a-judge relabeling with chance-corrected agreement, and every run published with raw transcripts.

Do LLM non-determinism and deployment-stack variation alter the conclusions of environmental-health meta-analyses? 36,000 LLM calls across 6 deployment stacks, with a pre-registered dual-human validation of the gold standard.

Statistical analysis of inter-rater reliability and quality patterns in LLM evaluation systems using R

⚖️ Dual-Judge: 让AI测试结果真正有说服力 | 双LLM交叉验证消除单模型偏见 | 独立于具体Agent的通用评估框架 | Making AI Evaluation Trustworthy

Blind pairwise evaluation — seeded blinding, position-bias detection, and inter-rater agreement. The win rate is the number that means least. Zero dependencies; MIT.

Measure whether your QA graders agree with each other. Exact agreement, mean absolute difference and weighted Cohen kappa per scorecard item, plus a calibration pack and a blind re-grade queue.

Python package implementing measure from my working paper "Kappa-IoU: Inter-Rater Reliability for Spatial Annotation"

Tool-agnostic inter-coder reliability (Krippendorff alpha, Cohen/Fleiss kappa) and disagreement adjudication for qualitative coding

How reliable is LLM-as-judge? Inter-model agreement and failure modes across 5 local and frontier models on 200 claim-verification items.

Score job descriptions against a 44-domain life-science ontology, calibrated against a clinical expert panel and validated by inter-rater agreement (quadratic weighted kappa) between two independent runs.

Expert review operations for AI-training data: versioned weighted rubrics, exact rubric scoring, and inter-rater agreement (Cohen, Fleiss, Krippendorff) with seeded bootstrap CIs.

An evaluation harness that refuses to print a rate without a confidence interval, or a model grader's score without its calibration against human labels