[Usage] Training process get stuck in the last iteration of instruction finetuning phrase.

[fmy7834]

Describe the issue

Issue:

When I try to training LLaVA-1.5(with modifications) using Lora in instruction finetuning phrase with vision encoder trainable, it stucks in the last iteration without any outputs. However, when I keep vision encoder frozen, the training process could carry on normally and successfully save ckpts. And the memory is not used up. I was training with zero2. Does anyone meet this problem before?