Error running evals with llava_hf models

[ElliotStein]

Description

Running the following command results in a ValueError due to an invalid device_map being passed to from_pretrained:

Example Command

accelerate launch --num_processes=8 --main_process_port 12345 -m lmms_eval
--model llava_hf
--model_args pretrained=llava-hf/llava-1.5-7b-hf
--tasks seedbench
--batch_size 1
--output_path ./logs/
--log_samples

Error message:
File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 3551, in from_pretrained
device_map = {"": torch.device(device_map)}
RuntimeError: Device string must not be empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
...
File "/root/lmms-eval/lmms_eval/models/llava_hf.py", line 97, in init
self._model = model_type.from_pretrained(pretrained, revision=revision, torch_dtype=dtype, device_map=self.device_map, trust_remote_code=trust_remote_code, attn_implementation=attn_implementation)
File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 3553, in from_pretrained
raise ValueError(
ValueError: When passing device_map as a string, the value needs to be a device name (e.g. cpu, cuda:0) or 'auto', 'balanced', 'balanced_low_0', 'sequential' but found .

Proposed Fix

In lmms_eval/models/llava_hf.py, update the default value of device_map to "cuda:0", consistent with the default in llava.py.

I'm still getting familiar with the code base so its possible I've missed a better solution, particularly as it seems others haven't had this issue. Either way, it seems like the default settings being consistent between llava.py and llava_hf.py would be helpful.

Thanks!

[kcz358]

You can pass device_map=cuda:0 in model_args if you are using single device to avoid this issue. The default settings is for multi-processing evaluation config