Eight decision models, measured as the candidate list grows, the option order changes, and the wrong answers stop being obvious.
[NeurIPS'2026] Position Bias in Mamba and Hybrid Language Models | Evidence Position Bias Across Sequence Mixers in Long-Context Question Answering
Do LLM judges know when they're wrong? Three open-weight judges (Qwen2.5-7B, kev-8b, auto-j-13b) graded against MT-Bench human votes: calibration, position and padding attacks, and what auto-accepting confident verdicts would cost. Interactive site included.
ACL 2026 | CapCal: content-agnostic probability calibration for de-biasing listwise rerankers.
Flight-metasearch SERP analytics lab: ranking, position bias, partner monetisation and A/B readout
Learning-to-rank lab: listwise models, position bias, calibration, and NDCG evaluation
position bias in LLM judges is worse than people think: ask a small instruct model to choose 1 of 2 items. 81% of the answer will be based on which slot the item was in, not which it was. swap the order and the answer flips 78% of the time. an inconclusive cognitive dissonance experiment
Position bias, length bias and self-uncertainty of a small LLM judge on MT-Bench (Qwen2.5-7B)
Independent research on AI evaluation validity, LLM-as-a-judge reliability, benchmark comparability and experimental assurance.
Measure position/verbosity/assertiveness bias in an LLM-as-judge (Claude) by judging pairs in both orders. Finding: no position bias, but 75% verbosity bias and 100% assertiveness bias — judge scores gameable by length + tone.
Two kinds of saturation: why LLM-judge order bias is hard to measure — essay, Lean proofs (0 sorry), and a reproducible dispersion measurement.
Blind side-by-side human eval of two model responses on a weighted rubric — randomized panes to kill position bias, bootstrap CIs on the margin, inter-rater reliability. An eval you can't audit is a vote, not a measurement.
Official reproduction code for 'Efficient LLM Preference Classification Through Position Bias Mitigation and Architectural Symmetry' (Applied Intelligence APIN-D-26-05359)
Open-source implementation of the typed-decision pattern popularised by TypeSafe's Jev: read a decision out of a small language model's logits, in the browser. Library, benchmarks and paper. Not affiliated with TypeSafe.
Qwen3.5-0.8B fine-tuned into a decision model: one forward pass, a probability per option. An open take on Microsoft's Decision-1, trained in 14 min on a free Kaggle T4.
Do LLM-judge biases compose? A label-free, scale-aware test (SSIT) for position×length interaction in LLM-as-a-judge pipelines, with proofs, tie-coding and curvature diagnostics, 7-judge/6-family data, and the TMLR manuscript.
Position-bias-aware ranking: estimating and correcting position bias to optimize Earnings Per Visitor (EPV) · simulation study · IPW & propensity modeling · Python
Audit of presentation effects and biases in Jev, Clef Flash, and Perplexity Decider across humor, helpfulness, and summarization. Includes 702,000 recorded judgments and offline reproduction.
Counterfactual learning-to-rank for marketplace search logs in PySpark: position-bias estimation, IPS-weighted training, NDCG evaluation against known ground truth
A frozen probe set and a harness for auditing LLM judges for bias. No audit published yet.
To associate your repository with the position-bias topic, visit your repo's landing page and select "manage topics."