[dependabot]

Updates the requirements on lm-eval to permit the latest version.

Release notes

Sourced from lm-eval's releases.

v0.4.13 Release Notes

A fix-focused release. The main fixes are for few-shot leakage, a multiple-choice filter bug, and group stderr, alongside two new ONNX backends and eight new benchmark suites. Also updated most configs for datasets>=4, which accounts for much of the diff by volume.

Highlights

Bug Fixes

Fixes that may shift previously reported numbers:

• Eval documents leaked into few-shot prompts. The sampler could draw the document under test into its own demonstrations by @​SiavashShams in #3978, and gen_prefix was resolved against the eval doc rather than the few-shot doc — splicing the evaluated question into every shot for RULER niah_single_1, humaneval_instruct, and humaneval_64_instruct by @​adityasingh2400 in #3979
• MultiChoiceRegexFilter prefix-shadowing. Regex alternation is leftmost-wins, so a choice that prefixed a longer choice ("Guilty" vs "Guilty of Romance") matched inside it and scored correct answers as wrong — visible on BBH movie_recommendation by @​iamsharduld in #3884
• Group stderr with weight_by_size: false. Groups reported the size-weighted pooled stderr even when the point estimate was an unweighted mean, giving error bars up to ~3x too narrow. Only unequal-sized subtasks change by @​iamsharduld in #3882
• minerva_math answer normalization. sqrt shorthand no longer corrupts indexed roots (#4037), the thousands-separator strip no longer fuses digit tuples (0,1 → 01, #4039), an answer identical to the gold now scores correct (#4034), and the few-shot prompt LaTeX is corrected (#4045) by @​feiiiiii5 and @​nata2627. putnam_axiom shares these helpers and picks up the same fixes.

New Model Backends

• onnxruntime — raw onnxruntime backend for Model Builder ONNX exports by @​amd-sourjya in #3984
• onnxruntime-genai — cross-platform ONNX Runtime GenAI backend, with winml refactored on top of it by @​thiagocrepaldi in #3960
• Megatron-LM v0.18 compatibility by @​oberpierre in #3991

Install with pip install lm_eval[onnxruntime] or lm_eval[onnxruntime-genai]. For non-CPU execution providers install the matching wheel instead — onnxruntime-gpu / onnxruntime-rocm, or onnxruntime-genai-cuda / onnxruntime-genai-directml — these are mutually exclusive.

New Tasks

• LongProc — long-context procedural reasoning, 6 task types across 16 configs by @​xiye17 in #3544
• Uncheatable Eval — contamination-resistant evaluation: rolling log-likelihood over recently-published documents from Wikipedia, GitHub, BBC News, arXiv, bioRxiv, and AO3, as 15 category tasks plus a size-weighted group by @​ziqing-huang in #3442
• TyDiQA Gold Passage — 9-language multilingual QA by @​bongho in #4044
• LegalBench — new suite, 19 tasks: the HELM-lite LegalBench subset (5 tasks, #3860) and the Contract NLI suite (14 NDA entailment tasks, #3954) by @​bongho
• IndicParam — MCQ benchmark covering 12 low-resource Indic languages by @​hjoshifonteva in #3826
• IndicXNLI Gujarati (indicxnli_gu) by @​bhaumik611 in #4056
• GreekMMLU — official native-sourced configuration by @​mersinkonomi in #3581
• Physics GRE — InflectionAI multiple-choice physics benchmark by @​bongho in #3853
• Putnam Axiom — competition-level mathematical reasoning by @​baberabb in #3998
• Portuguese Bench — ASSIN2 RTE and STS by @​bongho in #3812
• Catalanbench — cieaCOVA (#4043) and Terretaqa (#3784) by @​baberabb and @​ivanmartinezmurillo
• MLQA — a tag to run all variants at once by @​ivanbaldo in #2977

Task Changes

Correctness & Prompts

• global_piqa restructured into parallel/non-parallel × cloze/generation variants; the global_piqa_completions and global_piqa_prompted groups are now global_piqa_cloze and global_piqa_generation by @​baberabb in #3816
• IrokoBench — repaired afrimmlu / afrimgsm / afrixnli task registration by @​discobot in #3841, fixed the broken afrisenti / mafand prompt_2 group references by @​DaoyuanLi2816 in #3847, and switched afrixnli prompt_1 doc_to_text to Jinja braces by @​Solaris-star in #3944
• KorMedMCQA — answer-choice extraction now follows the paper's Appendix B by @​discobot in #3842
• JSONSchema Bench — per-sample validation timeout so a pathological schema no longer hangs the whole eval by @​hancheolcho in #3923
• RULER — every task crashed with TypeError: unhashable type: 'dict' before inference when the tokenizer arrived as anything but a plain string (e.g. under local-chat-completions); the name is now resolved before the cached lookup by @​nata2627 in #4048
• NIAH — no longer loops forever when max_seq_lengths < 4096 by @​vnayakde in #3372
• LongBench — code_sim_score skips a blank leading line before extraction by @​cameronshinn in #3921
• FDA / SWDE / SQuAD_completion — whitespace stripped from input and target by @​EphraiemSarabamoun in #3795

... (truncated)

Commits

• ddd6722 chore(version): 0.4.13 (#4071)
• 2fca61d fix: make --use_cache keys hashable for multimodal image/byte requests (#4040)
• 3d019dc Fix/indicparam citation (#4066)
• 8c1291c fix(evaluator): report a sample count that does not depend on metric order (#...
• 6bfdbcb feat(physics_gre): add InflectionAI Physics GRE multiple-choice task (#3853)
• c1b3b3a Add IndicXNLI Gujarati task (indicxnli_gu) (#4056)
• a966806 Add Uncheatable Eval (#3442)
• d1193e3 fix(scripts): repair requests_caching.py entrypoint (bad kwargs + helper defi...
• 9fd734a chore(pre-commit) (#4060)
• 0fdda00 fix(ruler): resolve the tokenizer name before the cached lookup (#4048)
• Additional commits viewable in compare view