昇腾910B3 两机16卡 lora sft Qwen2-72b报OOM

[hangxu124]

Reminder

• I have read the README and searched the existing issues.

System Info

[2024-11-19 08:07:30,524] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to npu (auto detect)

• llamafactory version: 0.9.1.dev0
• Platform: Linux-5.10.0-60.18.0.50.oe2203.aarch64-aarch64-with-glibc2.27
• Python version: 3.10.13
• PyTorch version: 2.1.0 (NPU)
• Transformers version: 4.45.2
• Datasets version: 2.19.1
• Accelerate version: 0.34.2
• PEFT version: 0.12.0
• TRL version: 0.8.6
• NPU type: Ascend910B3
• CANN version: 8.0.RC2.alpha001
• DeepSpeed version: 0.13.2

Reproduction

### model
model_name_or_path: /workspace/model/llama-model/model/Qwen2-72B-Instruct

### method
stage: sft
do_train: true
finetuning_type: lora
lora_target: all
deepspeed: examples/deepspeed/ds_z3_config.json

### dataset
dataset: identity,alpaca_en_demo
template: llama3
cutoff_len: 1024
max_samples: 1000
overwrite_cache: true
preprocessing_num_workers: 16

### output
output_dir: saves/qwen2-72b-multi/lora/sft
logging_steps: 10
save_steps: 500
plot_loss: true
overwrite_output_dir: true

### train
per_device_train_batch_size: 1
gradient_accumulation_steps: 8
learning_rate: 1.0e-4
num_train_epochs: 3.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000

### eval
val_size: 0.1
per_device_eval_batch_size: 1
eval_strategy: steps
eval_steps: 500

FORCE_TORCHRUN=1 NNODES=2 RANK=0 MASTER_ADDR=xxxx MASTER_PORT=29500 llamafactory-cli train examples/train_lora/llama3_lora_sft.yaml

FORCE_TORCHRUN=1 NNODES=2 RANK=1 MASTER_ADDR=xxxx MASTER_PORT=29500 llamafactory-cli train examples/train_lora/llama3_lora_sft.yaml

在加载模型阶段就报oom了，
File "/usr/local/python3.10.13/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 240, in wrapped_fn
tensor.data = tensor.data.to(target_fp_dtype)
RuntimeError: NPU out of memory. Tried to allocate 464.00 MiB (NPU 4; 60.97 GiB total capacity; 8.51 GiB already allocated; 8.51 GiB current active; 14.24 MiB free; 59.94 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.

尝试过zero3_cpu_offload, 可以正常训练，但实在太慢。

使用单机8卡zero3是可以训练的，反而两机16卡会报oom。

Expected behavior

使用单机8卡zero3是可以训练的，反而两机16卡会报oom，不明白为什么会这样

Others

No response