Group-safe research prototype for prioritising potentially inconsistent nucleus annotations for expert review.
Statistical validity checks for human-graded AI evaluations
Emotion architecture from Reddit comments: rater behavior, semantic clusters, and contradiction mapping in GoEmotions.
LabelBench by Bayanat Labs, a reproducible audit of automated Arabic data labeling. Measures how much of a labeling queue can be accepted at a declared quality floor, by taxonomy, task and Arabic dialect region. Code only; datasets download from their original sources.
Code, logs and evaluation artifacts for "Shortcut Learning in a Public Grape Disease Dataset: Annotation Granularity as a Modulator, Not a Cause" — inconsistent annotation granularity induces a quantifiable shortcut, established by counterfactual retraining and a pre-registered negative result.
B.Sc. seminar project: annotation agreement analysis (Bennett's S, kappa, AC1), an annotation experiment on 216 tweets, and a 76-criterion COCO vs SA-1B dataset evaluation.
Modular visual evaluation and computer-vision framework for image QA, structured annotation, reliability analysis, and human pose data.
Production IAA engine — Krippendorff's Alpha, SBERT semantic agreement, Fleiss' Kappa, Shannon entropy, 3-layer collusion detection. Ran behind RawEval's 9-annotator workbench.
Reliability, rogue-rater, drift and leakage diagnostics for labelled data — on the raw GoEmotions ratings, 27 of 28 emotions fall below the 0.667 agreement floor.
To associate your repository with the annotation-quality topic, visit your repo's landing page and select "manage topics."