Skip to content

Pull requests: EleutherAI/lm-evaluation-harness

New pull request New
157 Open 1,637 Closed
157 Open 1,637 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

[fix] add math and longbench to test dependencies
#3321 opened Sep 30, 2025 by jannalulu Loading…
Add support for Titulm Bangla MMLU dataset
#3317 opened Sep 29, 2025 by Ismail-Hossain-1 Loading…
1
unpin datasets; update pre-commit
#3316 opened Sep 27, 2025 by baberabb Loading…
1
feat: Add support for accelerate-wrapped models in simple_evaluate()
#3313 opened Sep 26, 2025 by DhruvaKashyap Loading…
2
Add MATH500
#3311 opened Sep 26, 2025 by jannalulu Loading…
Support empty response for Completions and ChatCompletions API
#3309 opened Sep 22, 2025 by tboerstad Loading…
1
Adding New Task SLR-Bench : Scalable Logical Reasoning Benchmark
#3305 opened Sep 20, 2025 by Ahmad21Omar Loading…
1
Support torchrun vllm DP
#3304 opened Sep 19, 2025 by luccafong Loading…
1
Fix: VLLM model when data_parallel_size>1
#3303 opened Sep 18, 2025 by Dornavineeth Loading…
2
Gemini evaluation support
#3300 opened Sep 15, 2025 by IsraelAbebe Loading…
Fix lambada_multilingual_stablelm
#3294 opened Sep 11, 2025 by jmichaelov Loading…
1
[feature] add support for Moore Threads GPU family
#3290 opened Sep 10, 2025 by houchen-li Loading…
2
Adding SPaRC to lm eval harness
#3262 opened Aug 25, 2025 by lkaesberg Loading…
2
Add long-context evaluation benchmarks (LongBench v2, Babilong, InfiniteBench, Phonebook)
#3256 opened Aug 21, 2025 by Mariani-code Loading…
1
11
fix gsm8k normalization
#3254 opened Aug 20, 2025 by huaanrui Loading…
1
Main
#3250 opened Aug 20, 2025 by seongtaehong Loading…
3
Adding 3LM to lm eval harness
#3241 opened Aug 14, 2025 by GeorgeSherif Loading…
1
Trim thinking content from model output in IFEval
#3240 opened Aug 14, 2025 by davideguidobene Loading…
Remove gen_prefix space and add warning
#3239 opened Aug 14, 2025 by bendboaz Loading…
1
5
Adding support for evaluating with Mistral and Pixtral models
#3235 opened Aug 13, 2025 by LearnerSXH Loading…
1
2
Adding support for Structured Generation with XGrammar
#3232 opened Aug 12, 2025 by ceferisbarov Loading…
5 tasks
4
Pass dataset_kwargs for Unitxt tasks
#3230 opened Aug 11, 2025 by mprahl Loading…
Fewshot refactor
#3227 opened Aug 8, 2025 by baberabb Loading…
1 task
Fix the Unitxt init method to set the task name
#3225 opened Aug 8, 2025 by mprahl Loading…
1
Fix: respect target_delimiter when using a gen_prefix on multiple-choice tasks
#3220 opened Aug 7, 2025 by karanikolopoulos Loading…
1
ProTip! Exclude everything labeled bug with -label:bug.