import_ckpt hangs when converting DeepSeek-v3 Hugging Face model to Nemo format

[zirui]

Describe the bug

When using import_ckpt to convert the DeepSeek-v3 model (Hugging Face format) to Nemo format, the process hangs indefinitely without errors.

Steps/Code to reproduce bug

    model_path = "/models/DeepSeek-V3-Base-bf16"
    output_path = "/models/DeepSeek-V3-Base-bf16-nemo"

    imported_path = llm.import_ckpt(model=llm.DeepSeekModel(llm.DeepSeekV3Config()), \
            source=f"hf://{model_path}",
            output_path=output_path,
            overwrite=True
    )

The output directory(/models/DeepSeek-V3-Base-bf16-nemo) looks incomplete, containing only 235GB of data, which seems smaller than expected

.
|____weights
| |______0_1.distcp
| |______0_0.distcp
| |____common.pt

Expected behavior
The model should be successfully converted to Nomo format without hanging.

Environment details

• Docker image: nvcr.io/nvidia/nemo:25.02.rc4

[TexasRangers86]

Same problem!
I got a common.pt file with 4K.
Docker image: nvcr.io/nvidia/nemo:25.02.rc5

[cuichenx]

Hi, couple of things to try:

1. The conversion takes around 72 minutes in my environment. How long did you wait?
2. Did you create your BF16 checkpoint with the steps outlined here?
3. If you put the command in a python script directly, can you try wrapping it under a if __name__ == "__main__" block
4. If the above don't work, can you post the output of py-spy dump -p <pid> to see where the process is hanging

[zirui]

Hi, couple of things to try:

1. The conversion takes around 72 minutes in my environment. How long did you wait?
2. Did you create your BF16 checkpoint with the steps outlined here?
3. If you put the command in a python script directly, can you try wrapping it under a if __name__ == "__main__" block
4. If the above don't work, can you post the output of py-spy dump -p <pid> to see where the process is hanging

1. After a few hours, I could see that the weights directory contained 235GB of files. However, after more than ten hours, the process remained in this state, and it seemed to be stuck (hang).
2. Yes, I followed the instructions in this document to convert the checkpoint to BF16.
   one issue I noticed is that the import_ckpt function in the documentation refers to llm.DeepSeekV3Model, but this is not implemented in the code. I used llm.DeepSeekModel as a replacement.
3. My code is already executed inside the if __name__ == "__main__" block.

I will try points 4 later and get back to you with an update.