Feature Request: Support for QwQ-32B Model Integration

[CaptainRui1000]

Background

QwQ-32B is a recently developed model that offers impressive performance while maintaining efficiency, making it a promising candidate for various AI tasks. As the model architecture evolves, integrating support for QwQ-32B would be valuable for those looking to use NeMo for training and deployment.

Request

I would like to request the addition of support for the QwQ-32B model within the NeMo framework. This would allow users to leverage NeMo's capabilities for training, fine-tuning, and deployment with QwQ-32B.

Motivation

Integrating QwQ-32B into NeMo could help users to:

• Utilize cutting-edge models without needing to switch frameworks.
• Benefit from NeMo's optimizations and features (e.g., multi-GPU support, mixed precision).
• Enable faster model training and inference by leveraging QwQ-32B’s efficiency.

Resources

• QwQ-32B: Small Model with Huge Potential
• QwQ-32B - Intelligence, Performance & Price Analysis

Expected Outcome

It would be great to have QwQ-32B supported in the next release or as part of an experimental feature, allowing users to integrate it seamlessly into their workflows.

[euronymous-aithal]

@akoumpa can we cover in Nemo AutoModel ?

[akoumpa]

Hi @CaptainRui1000 ,

Thank you for your request. We recently introduced a HuggingFace-native workflow in NeMo that supports HuggingFace models and provides multigpu scaling via FSDP2. I was able to confirm that QwQ-32B is runnable on a single 8xH100 node.

I would recommend trying the PEFT or SFT notebooks with the QwQ-32B model.

Please feel free to let me know if you have any other questions. Thank you.