OOM error when i tried to save model in q_8 gguf

[Mracobes9]

Hi. I'm facing the following problem. After training a model, I get an OOM error when I try to save the result in GGUF Q_8 format. Could you please advise how this can be avoided ?

[shimmyshimmer]

You can followthis and see if it works: https://docs.unsloth.ai/basics/errors#saving-to-gguf-vllm-16bit-crashes

"You can try reducing the maximum GPU usage during saving by changing maximum_memory_usage.

The default is model.save_pretrained(..., maximum_memory_usage = 0.75). Reduce it to say 0.5 to use 50% of GPU peak memory or lower. This can reduce OOM crashes during saving."