Skip to content

Issues: EvolvingLMMs-Lab/lmms-eval

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

164 Open 73 Closed
164 Open 73 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

no PerceptionTest tasks eval
#483 opened Dec 29, 2024 by zef1611
eval on multi-gpus
#482 opened Dec 29, 2024 by masn1310
Firstly, thank you very much for your contribution to the codebase. I am currently working on evaluating LLaVA and have encountered a few issues. I look forward to your guidance.
#476 opened Dec 27, 2024 by LiZhangMing
3
LLaVA-Video-72B-Qwen2
#473 opened Dec 26, 2024 by HulkZh
1
Why is temperature set to 0.2 in some task YAML configurations?
#465 opened Dec 19, 2024 by wswaq
Test with mme always error: NCCL Error 1: unhandled cuda error (run with NCCL_DEBUG=INFO for details)
#464 opened Dec 18, 2024 by ZhaoyangLi-nju
The results of InstructBLIP got zero on TextVQA
#463 opened Dec 16, 2024 by kinredon
1
Problem in Reproducing llava-onevision-72b-ov-chat
#459 opened Dec 13, 2024 by Estrella142857
The results of LLaVA-Video-7B-Qwen2's captioned test on VideoMME were quite different
#452 opened Dec 10, 2024 by ghost
4
I can not reproduce the result JMMMU evaluation result of SakanaAI/Llama-3-EvoVLM-JP-v2
#449 opened Dec 9, 2024 by CHENSSR
1
[Info] Models that natively support audio and video
#446 opened Dec 5, 2024 by gzhihongwei
2
A bug in lmms-eval/api/task.py and a suggestion
#445 opened Dec 5, 2024 by real-lurEn
1
[New Model] Any plan to support PaliGemma?
#443 opened Dec 5, 2024 by siahuat0727
1
AttributeError: 'dict' object has no attribute 'vocab_size' occurs when evaluating lava-onevision-qwen2-0.5b-ov-hf
#442 opened Dec 4, 2024 by huayicong23
1
load_pretrained_model() missing 1 required positional argument: 'model_name' when evaluating VILA-7b
#441 opened Dec 4, 2024 by huayicong23
3
error occurs when runing scienceqa task
#439 opened Dec 4, 2024 by WeiweiZhang1
Error running evals with llava_hf models
#438 opened Dec 3, 2024 by ElliotStein
1
Discussion how can i use vllm as backend
#437 opened Dec 3, 2024 by tenacioustommy
Mistake in MLVU dataset calculation
#435 opened Dec 2, 2024 by lwpyh
1
Problem in Reproducing MMMU benchmark on llava-v1.5-7b
#433 opened Nov 29, 2024 by lyklly
8
image bug for the latest version transformers
#430 opened Nov 28, 2024 by n1ck-guo
Any plan to intergrate the hourvideo evaluation dataset
#429 opened Nov 27, 2024 by zhhongzhi
A bug when processing result in ScienceQA task
#427 opened Nov 27, 2024 by GoGoJoestar
Add Support for the ScreenQA Dataset
#425 opened Nov 26, 2024 by chenghuaWang
How to evaluate a subset of a certain benchmark?
#424 opened Nov 26, 2024 by ZihaoZheng98
1
ProTip! Type g i on any issue or pull request to go back to the issue listing page.