[BUG] how to eval large scale model use 1dp+8pp?

[mxjmtxrm]

Describe the bug

I tired to eval a large scale model use1dp+8pp with accelerate. I use the command like the following:

accelerate launch --multi_gpu --num_processes=1 run_evals_accelerate.py \
    --model_args="pretrained=<path to model on the hub>" \
    --model_parallel \
    --tasks <task parameters> \
    --output_dir output_dir

The error is ValueError: You need to use at least 2 processes to use --multi_gpu

How to solve this problem?

Version info

lighteval-0.3.0

[NathanHB]

Hi ! to use DP = 1 you need to run wihtout accelerate. In your case the command would be:

lighteval accelerate "pretrained=gpt2,model_parallel=True" "leaderboard|truthfulqa:mc|0|0"

The command you described above is deprecated, Where did you find it ?

[mxjmtxrm]

https://github.com/huggingface/lighteval/blob/v0.3.0/README.md#evaluating-a-large-model-with-pipeline-parallelism

I use an old version. Does it support DP = 1 + PP=K?

[mxjmtxrm]

I updated lighteval to V0.6.0 and tried again.
I used the following command:

export CUDA_VISIBLE_DEVICES=0,1,2,3 
lighteval accelerate "pretrained=<model_path>,model_parallel=True" "leaderboard|truthfulqa:mc|0|0"

It failed to load the model with MP, and only GPU 0 was occupied.

[2024-12-25 03:29:43,045] [    INFO]: Model parallel was set to False, max memory set to None and device map to None (base_model.py:389)

I found 'LOCAL_WORLD_SIZE' and 'WORLD_SIZE' was none.
How to solve this problem?

[NathanHB]

Hi ! You are right, I opened a PR you can checkout it should solve the issue.