expected-calibration-error

Open reproduction of TypeSafe Jev: a 150M typed decision engine (noul/choice/score in one non-autoregressive pass, calibrated confidence). 0.697 vs Jev's 0.727, 2.5x better calibrated, 4x faster, free. Trains on a Colab T4 in 30 min.

Decision-safe evaluation + Streamlit dashboard for AI vs Human vs Post-Edited AI text detection. Generates a reliability report card (Accuracy, Macro F1, ECE, Brier), calibration plots, confidence histograms, and a coverage-vs-performance abstention curve. Recommends an operating threshold for human-review routing.

Calibration and uncertainty quantification for ranking systems.

Research framework for analyzing calibration collapse under class imbalance in tabular machine learning, with adaptive calibration methods, stress-test benchmarks, and minority reliability evaluation.

How Chain-of-Thought Budgets Induce Overconfidence in LLMs. Investigating Calibration Drift Under Reasoning (CDUR), hypothesis lock-in mechanisms, and dynamic token budgeting via the CABStop optimal stopping algorithm.

MSc thesis: evaluating & improving uncertainty calibration in retrieval-augmented LLMs. Compares 5 embedding models (BGE beats OpenAI) and cuts ECE ~32% via isotonic regression on real financial filings.

Fine-tuning and calibrating the Chronos (T5) transformer for financial time series forecasting. Achieves up to 15% better accuracy (MASE) and reliable confidence estimates using perturbation-based consistency calibration (C3).

Metric-only reproduction tools for LLM-as-a-Judge calibration experiments

Measuring and fixing DNN overconfidence on MNIST with ECE and temperature scaling.

Pre-registered study of how MLX quantization (bf16 → 2-bit, mxfp4, mixed recipes) breaks LLM confidence calibration — not just accuracy.

Measure whether a decision model's probabilities are trustworthy on your own labeled data. Not a leaderboard.

¿Jev entiende tu acento? Pre-registered audit of TypeSafe AI's Jev on Spanish — accuracy, calibration and token cost — plus a CLI to run the same comparison on your own labelled data.

Semantic ORDER BY for DuckDB, backed by TypeSafe AI's Jev model. Ships with independent calibration numbers.