How to convert the torch_dist ckpt to the nemo file?

[Kamizato-Ayaka]

Description:
I have trained a MegatronGPTSFT model using NeMo. After the training, I only obtained the torch_dist-organized checkpoint directory but did not get a .nemo file. I now need the .nemo file to convert the model to the Hugging Face format. However, I’m unable to figure out how to convert the torch_dist checkpoint into a .nemo file.

I have tried using the following scripts provided by NeMo:

NeMo/scripts/checkpoint_converters/convert_zarr_to_torch_dist.py
NeMo/examples/nlp/language_modeling/megatron_ckpt_to_nemo.py
Both methods fail at the load_checkpoints step. The error log is as follows:

[rank2]: Traceback (most recent call last):
[rank2]:   File "/home/aiscuser/dengjie/NeMo/examples/nlp/language_modeling/megatron_ckpt_to_nemo.py", line 252, in <module>
[rank2]:     convert(local_rank, rank, world_size, args)
[rank2]:   File "/home/aiscuser/dengjie/NeMo/examples/nlp/language_modeling/megatron_ckpt_to_nemo.py", line 203, in convert
[rank2]:     model = MegatronGPTModel.load_from_checkpoint(
[rank2]:   File "/home/aiscuser/dengjie/NeMo/nemo/collections/nlp/models/nlp_model.py", line 385, in load_from_checkpoint
[rank2]:     model = ptl_load_state(cls, checkpoint, strict=strict, cfg=cfg, **kwargs)
[rank2]:   File "/opt/conda/envs/ptca/lib/python3.10/site-packages/lightning/pytorch/core/saving.py", line 165, in _load_state
[rank2]:     obj = instantiator(cls, _cls_kwargs) if instantiator else cls(**_cls_kwargs)
[rank2]:   File "/home/aiscuser/dengjie/NeMo/nemo/collections/nlp/models/language_modeling/megatron_gpt_model.py", line 331, in __init__
[rank2]:     super().__init__(cfg, trainer=trainer, no_lm_init=True)
[rank2]:   File "/home/aiscuser/dengjie/NeMo/nemo/collections/nlp/models/language_modeling/megatron_base_model.py", line 222, in __init__
[rank2]:     self._build_tokenizer()
[rank2]:   File "/home/aiscuser/dengjie/NeMo/nemo/collections/nlp/models/language_modeling/megatron_base_model.py", line 442, in _build_tokenizer
[rank2]:     tokenizer_model=self.register_artifact("tokenizer.model", self._cfg.tokenizer.get('model', None)),
[rank2]:   File "/home/aiscuser/dengjie/NeMo/nemo/collections/nlp/models/nlp_model.py", line 159, in register_artifact
[rank2]:     return super().register_artifact(config_path, src, verify_src_exists=verify_src_exists)
[rank2]:   File "/home/aiscuser/dengjie/NeMo/nemo/core/classes/modelPT.py", line 286, in register_artifact
[rank2]:     return self._save_restore_connector.register_artifact(self, config_path, src, verify_src_exists)
[rank2]:   File "/home/aiscuser/dengjie/NeMo/nemo/core/connectors/save_restore_connector.py", line 406, in register_artifact
[rank2]:     return_path = os.path.abspath(os.path.join(app_state.nemo_file_folder, src[5:]))
[rank2]:   File "/opt/conda/envs/ptca/lib/python3.10/posixpath.py", line 76, in join
[rank2]:     a = os.fspath(a)
[rank2]: TypeError: expected str, bytes or os.PathLike object, not NoneType

Steps I Tried:
I verified the checkpoints in multiple Docker environments, but the issue persists.
I reviewed the NeMo documentation and examples but could not find a resolution.
Question:
Is there a better or more robust way to convert torch_dist checkpoints into a .nemo file? Any suggestions or best practices would be greatly appreciated!