How to use multi GPU to finetune Llama2

[Qwtdgh]

Hi, I have a question about how to finetune Llama2 by using multi-GPU.

env: 4*A100 40G
yaml: llm/vaseline/exp_yaml/dolly_lda/dolly_federate.yaml

this yaml likes as follow

use_gpu: True
device: 0
early_stop:
  patience: 0
federate:
  mode: standalone
  client_num: 3
  total_round_num: 500

only one A100 is not enough, how can I use other three GPUS to finetune my model.

I try to modify train.data_para_dids=[0, 1, 2, 3], but it is not work, i think the reason is cfg.device only specify one GPU.

Wish your reply!

[rayrayraykk]

data_para_dids is for data-parallel. You should use deepspeed:
Please use the following configs to setup Deepspeed (for other usage, please refer to https://github.com/alibaba/FederatedScope/blob/llm/federatedscope/core/configs/cfg_llm.py):

# ---------------------------------------------------------------------- #
# Deepspeed related options
# ---------------------------------------------------------------------- #
cfg.llm.deepspeed = CN()
cfg.llm.deepspeed.use = False
cfg.llm.deepspeed.ds_config = ''  # compatible with deepspeed config

[Qwtdgh]

Thanks for your reply!
So, I'm not need to set cfg.device? Or device is not work when deepspeed=True?

[cqycqy1999]

Why is it that after configuring DeepSpeed, only one GPU is still being used? Here is my YAML file.
deepspeed: use: True ds_config: 'federatedscope/llm/baseline/deepspeed/ds_config.json'
I look forward to your response.