[Model] NemotronH support

[vegaluisjose]

This PR adds support to NemotronH family models:

• https://huggingface.co/nvidia/Nemotron-H-8B-Base-8K
• https://huggingface.co/nvidia/Nemotron-H-47B-Base-8K
• https://huggingface.co/nvidia/Nemotron-H-56B-Base-8K

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[hmellor]

Could this model be run with --model-impl transformers without this PR?

If yes (or if it would be easy to fix), I'd propose supporting this model via the transformers backend instead of duplicating the modelling code in vLLM.

[vegaluisjose]

Hi @hmellor ,

This vLLM implementation is way faster than transformers. Also, features such as chunked-prefill, cuda-graphs, FP8, TP>1, etc support is not available in transformers.

could you elaborate more on what do you mean by "duplicating the modelling code in vLLM."?

[hmellor]

I'm not suggesting that you don't use vLLM. I'm referring to the Transformers backend for vLLM https://docs.vllm.ai/en/latest/models/supported_models.html#transformers

could you elaborate more on what do you mean by "duplicating the modelling code in VLLM."?

This PR adds 800 lines of modelling code that already exists in Transformers.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @vegaluisjose.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[vegaluisjose]

Thanks for the pointers @hmellor, I went through them. However, I don't think this will work well given that this is a hybrid model, based on mamba and attention layers, that requires more than just replacing an attention backend to work well in vLLM. Also, the transformers implementation of NemotronH is based on external kernel libraries for mamba layers that conflict with vLLM.

I re-used as much vLLM code as I could including mamba-mixer2 layer and other components used in other hybrid models such as Bamba.

Let me know what you think.

[hmellor]

Ah I see, I have not tried with mamba style hybrid models (sliding window hybrid models do work though).

In that case this model is likely not a good candidate for the transformers backend in its current state.

Thanks for taking the time to look into the transformers backend though 🤗

[vegaluisjose]

Great @hmellor , I will start working on getting pre-commit to work and rebasing.

I will keep you posted.