v0.2.0
What's Changed
- pip package by @pufanyi in #1
- Fix mmbench dataset submission format by @pufanyi in #7
- [Feat] add correct tensor parallelism for larger size model. by @Luodian in #4
- update version to 0.1.1 by @pufanyi in #9
- [Tasks] Fix MMBench by @pufanyi in #13
- [Fix] Fix llava reproduce error by @kcz358 in #24
- add_ocrbench by @echo840 in #28
- Joshua/olympiadbench by @JvThunder in #37
- [WIP] adding mmbench dev evaluation (#75) by @Luodian in #46
- Add
llava
model for 🤗 Transformers by @lewtun in #47 - Fix types to allow nullables in
llava_hf.py
by @lewtun in #55 - Add REC tasks for testing model ability to locally ground objects, given a description. This adds REC for all RefCOCO datasets. by @hunterheiden in #52
- [Benchmarks] RealWorldQA by @pufanyi in #57
- add Llava-SGlang by @jzhang38 in #54
- Add MathVerse by @CaraJ7 in #60
- Fix typo in Qwen-VL that was causing "reference before assignment" by @tupini07 in #61
- New Task: ScreenSpot - Grounding (REC) and instruction generation (REG) on screens by @hunterheiden in #63
- [New Task] WebSRC (multimodal Q&A on web screenshots) by @hunterheiden in #69
- Bugfix: WebSRC should be token-level F1 NOT character-level by @hunterheiden in #70
- Multilingual LLava bench by @gagan3012 in #56
- [Fix] repr llava doc by @cocoshe in #36
- add idefics2 by @jzhang38 in #59
- [Feat] Add qwen vl api by @kcz358 in #73
- Adding microsoft/Phi-3-vision-128k-instruct model. by @vfragoso in #87
- Add MathVerse in README.md by @CaraJ7 in #97
- add MM-UPD by @AtsuMiyai in #95
- add Conbench by @Gumpest in #100
- Update conbench in README by @Gumpest in #101
- update gpt-3.5-turbo version by @AtsuMiyai in #107
- [Upgrade to v0.2] Embracing Video Evaluations with LMMs-Eval by @Luodian in #108
New Contributors
- @pufanyi made their first contribution in #1
- @Luodian made their first contribution in #4
- @kcz358 made their first contribution in #24
- @echo840 made their first contribution in #28
- @JvThunder made their first contribution in #37
- @lewtun made their first contribution in #47
- @hunterheiden made their first contribution in #52
- @jzhang38 made their first contribution in #54
- @CaraJ7 made their first contribution in #60
- @tupini07 made their first contribution in #61
- @gagan3012 made their first contribution in #56
- @cocoshe made their first contribution in #36
- @vfragoso made their first contribution in #87
- @AtsuMiyai made their first contribution in #95
- @Gumpest made their first contribution in #100
Full Changelog: v0.1.0...v0.2.0