Skip to content

Issues: EleutherAI/lm-evaluation-harness

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

283 Open 807 Closed
283 Open 807 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

regex filter strips whitespace
#2371 opened Oct 1, 2024 by jkamalu
set trust_remote_code when loading tokenizer in api_model
#2370 opened Oct 1, 2024 by ggbetz
How can we connect to azure openai deployed models?
#2367 opened Sep 30, 2024 by patilpriyadarshini
Extracting vLLM metrics
#2365 opened Sep 29, 2024 by vsmolyakov
Which filter value should be used among the accuracy test results?
#2362 opened Sep 27, 2024 by KKwanhee
[multimodal] llava-1.5-7b-hf doesn't work on mmmu_val bug Something isn't working.
#2360 opened Sep 26, 2024 by BabyChouSr
4
Improve docs/model_guide.md with skeleton template code + description of utils like Collator and Reorderer documentation Improvements or additions to documentation. feature request A feature that isn't implemented yet.
#2358 opened Sep 26, 2024 by haileyschoelkopf
Add a test for scripts/write_out.py and other scripts/ utils
#2356 opened Sep 26, 2024 by haileyschoelkopf
Evaluation of MMLU tasks using a fined tuned Gemma 2 model
#2354 opened Sep 26, 2024 by chamath-eka
Setting limit_mm_per_prompt for vllm_vlm fails argument parser bug Something isn't working.
#2352 opened Sep 25, 2024 by mgoin
The base model and chat model have no difference when using generate_until, loglikelihood, loglikelihood_rolling,right? asking questions For asking for clarification / support on library usage.
#2347 opened Sep 25, 2024 by belle9217
1
Unexpected space character
#2346 opened Sep 25, 2024 by eldarkurtic
1
3
tasks RACE only high not "middle"
#2345 opened Sep 25, 2024 by Choi-jun9803
Reproduce QWen 2.5-14B-Instruct and LLaMa-3.1-8B-Instruct Results
#2344 opened Sep 25, 2024 by ruleGreen
1
gpt2 evaluation
#2343 opened Sep 24, 2024 by sorobedio
Locally reproducible HF-Leaderboard evals asking questions For asking for clarification / support on library usage.
#2338 opened Sep 24, 2024 by eldarkurtic
2
Dynamical prompt with extremely promising results #RIPrompt
#2335 opened Sep 23, 2024 by anthonyrisinger
1
Confusion over the model outputs
#2331 opened Sep 23, 2024 by tranlm
Failed to add a new metric
#2330 opened Sep 23, 2024 by Ofir408
Bug in the float limit handling feature request A feature that isn't implemented yet.
#2324 opened Sep 19, 2024 by cjluo-omniml
6
Error for AGIEval when using fewshot bug Something isn't working. validation For validation of task implementations.
#2323 opened Sep 19, 2024 by BaohaoLiao
1
Which version to use validation For validation of task implementations.
#2322 opened Sep 19, 2024 by sorobedio
9
Multiple generations (sequential) per question asking questions For asking for clarification / support on library usage. feature request A feature that isn't implemented yet.
#2317 opened Sep 17, 2024 by IntrepidEnki
1
Running multiple processes on a shared outlines cache database
#2306 opened Sep 16, 2024 by e-tornike
2
New Task: openai_mmmlu professionaly translated by OpenAI as part of o1 release feature request A feature that isn't implemented yet.
#2305 opened Sep 16, 2024 by giuliolovisotto
1
ProTip! Find all open issues with in progress development work with linked:pr.