有办法用 4090 跑 13b 模型吗

[Darmody]

有办法用 4090 跑 13b 模型吗

[tonysy]

Take the baichuan-13b as example.

1 24GB card for 13B model is difficult.

• Method-1:
  You can change the num_gpus=2 or num_gpus=4 in this https://github.com/InternLM/opencompass/blob/60c2d3d76b09686921206b34aa66a37aa90136a0/configs/models/hf_baichuan_13b_chat.py#L18

• Method-2:
  Choose fp-16 or int-4 model.

[Darmody]

Take the baichuan-13b as example.

1 24GB card for 13B model is difficult.

• Method-1:
  You can change the num_gpus=2 or num_gpus=4 in this https://github.com/InternLM/opencompass/blob/60c2d3d76b09686921206b34aa66a37aa90136a0/configs/models/hf_baichuan_13b_chat.py#L18
• Method-2:
  Choose fp-16 or int-4 model.

感谢回复，根据我个人经验，设置 max_memory_mapping 低于 24GB，用 fp-16 是可以跑的。如果不限制，还是跑不了