[Question] Can not reproduce LLaVA 1.5 performance on ScienceQA

[yiwei-chenn]

Question

I use the command to test LLaVA 1.5 7B model's performance on ScienceQA datset, the command is like

CUDA_VISIBLE_DEVICES=0 bash scripts/v1_5/eval/sqa.sh

And the .sh is the same as the github repo offers:

python -m llava.eval.model_vqa_science \
    --model-path liuhaotian/llava-v1.5-13b \
    --question-file dataset/scienceqa/llava_test_CQM-A.json \
    --image-folder dataset/scienceqa/images/test \
    --answers-file dataset/scienceqa/answers/llava-v1.5-13b.jsonl \
    --single-pred-prompt \
    --conv-mode vicuna_v1

python llava/eval/eval_science_qa.py \
    --base-dir dataset/scienceqa \
    --result-file dataset/scienceqa/answers/llava-v1.5-7b.jsonl \
    --output-file dataset/scienceqa/answers/llava-v1.5-7b_output.jsonl \
    --output-result dataset/scienceqa/answers/llava-v1.5-7b_result.json

After running the evaluation, I got 69.51% for 7B version, and 72.73% for 13B version.

I have double checked the images, pid_splits.json, problems.json are both downloaded from ScienceQA repo. And question-file is from the eval.zip.

The results is a little bit higher than the model zoo reported, but it is not consisten. Does anyone could help me solve the inconsistent problem?