[Model] Add support for normalized Transformer (nGPT) from NVIDIA

[shan18]

Adds a new architecture called normalized Transformer (nGPT) (Paper).

FIX #18797

We are planning to release a model based on a new architecture called normalized transformer (nGPT) and would like to have vllm support for it.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[DarkLight1337]

cc @hmellor could transformers fallback support this model if only the rope scaling is implemented in vLLM?

[hmellor]

Looking at the transformers PR it looks like it would probably be compatible with the transformers backend. If there are no public checkpoints I cannot test though.

The rope scaling is also implemented in the transformers code, so there should be no need for anything specific to this model to exist in vLLM.

The model could still be "officially" supported if it were added to the registry as:

    "NGPTForCausalLM": ("transformers", "TransformersForCausalLM"),

(there may be other changes necessary as we've not done this before)

[shan18]

@DarkLight1337 @hmellor are there any drawbacks to have the code for the model directly in native vLLM? If it is merged via the transformers backend will it still give the same inference speedup benefits that we see with vLLM?

[hmellor]

are there any drawbacks to have the code for the model directly in native vLLM?

Mainly:

• Code duplication
• Maintenance burden in vLLM

If it is merged via the transformers backend will it still give the same inference speedup benefits that we see with vLLM?

Yeah, the transformers backend uses vLLM's attention module so still gets all the benefits of doing so (paged attention, tp/pp, FA/FI, etc).

Depending on how performant your Transformers modelling code is you may see some performance difference though. You can try it with vllm serve your-checkpoint --model-impl transformers

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @shan18.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork