Support mistral interleaved attn

[patrickvonplaten]

Allow mistral models with interleaved attention patterns to be run-able in vLLM. For now we'll have to cap the max model length to the minimum of the allowed attention pattern until we figure out how to cleanly support interleaved attention pattern (similar to Gemma)

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[DarkLight1337]

Overall looks good, just some nits

[yananchen1989]

May I know what is the name used in vllm for Ministral ?
I only find mistralai/Ministral-8B-Instruct-2410 on the huggingface mistral collection https://huggingface.co/mistralai.

Also, when I try to load it,

OSError: Can't load tokenizer for 'mistralai/Ministral-8B-Instruct-2410'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'mistralai/Ministral-8B-Instruct-2410' is the correct path to a directory containing all relevant files for a LlamaTokenizerFast tokenizer.

[mgoin]

@yananchen1989 make sure to load that model with tokenizer_mode="mistral" since it is a mistral format checkpoint

[yananchen1989]

@yananchen1989 make sure to load that model with tokenizer_mode="mistral" since it is a mistral format checkpoint

shoule I set tokenizer_mode="mistral" each time when I am using mistral series ？ thanks.

[yananchen1989]

tokenizer_mode= "mistral" if args.llm_name.startswith('mistralai') else 'auto' not sure if this writting is fine .

[patrickvonplaten]

@yananchen1989 make sure to load that model with tokenizer_mode="mistral" since it is a mistral format checkpoint

shoule I set tokenizer_mode="mistral" each time when I am using mistral series ？ thanks.

Yes that would be nice! We could consider making it the default actually for the mistral organization

[heheda12345]

I find the huggingface config of mistralai/Ministral-8B-Instruct-2410 is like:

"sliding_window": 32768,

So why do we need to support sliding window as a list?