[Question] Pipeline Parallel for Mamba Megatron

[zixianwang2022]

Hi,

I am wondering if the scripts implemented in Mamba Megatron pipeline parallel will be possibly used inside Nemo? Specifically Mamba Megatron supports converting Mamba both to Tensor Parallel & Pipeline Parallel at here. The pipeline parallel code has been using Megatron's without interleaved implementation. Will we be able to utilize any of that Megatron implementation for SFT on Mamba in Nemo?

Thank you!

[akoumpa]

@JRD971000