Feature Request: Add support for Pixtral and other Vision models (llama 3.2 11b/90b etc)

[YorkieDev]

Pixtral works great in mlx-vlm (Blaizzy/mlx-vlm#67) would be great to see support land in LM Studio.

[youcefs21]

mlx-vlm version in lm studio is 0.0.13, pixtral is supported in 0.0.15, don't we just need to upgrade the mlx-vlm version?

[YorkieDev]

That would probably work. Seems fairly straightforward to do @yagil @neilmehta24

[Blaizzy]

Wait for v0.1.0 (Blaizzy/mlx-vlm#41)

It will be released later today or tomorrow.

[Blaizzy]

It has some fixes and many new features.

[julien-blanchon]

Awesome v0.1.0 is now merged

[mattjcly]

Pixtral is now supported thanks to @Blaizzy !

[julien-blanchon]

I hope this will get bundled to LM Studio soon

[yagil]

I hope this will get bundled to LM Studio soon

Not there yet, but keep an eye on https://lmstudio.ai/beta-releases

[neilmehta24]

As of #22, mlx-engine has Pixtral and Llama 3.2 vision support. We expect to roll this out to LM Studio soon.

[orcinus]

Any updates? Vision support in general in LM Studio is abysmal (UI and UX seem more like an MVP than actual usable thing, and recent vision enabled models like Qwen VL are extremely buggy / bordering on unusable). Combined with being months late with support for major vision-enabled LLMs makes using LM Studio a tough sell (and yes, i know it's free - i'd gladly pay for it if it meant faster rollout of features and architecture support).

[YorkieDev]

Marking this issue as solved as 0.3.5 Build 9 has support for Pixtral and Llama 3.2 Vision in the MLX Engine.

https://lmstudio.ai/beta-releases

[orcinus]

Still doesn't work.
Getting Unknown ArrayValue filter: trim when trying to load MLX Llama 3.2 Vision Instruct.

[orcinus]

Nevermind, i'm dumb. Apparently, vision llama 3.2 does not have a System role (but regular 3.2 does). Weird.

[orcinus]

Unfortunately, it still doesn't work. It leaks RAM badly.
Deleting messages from context doesn't reduce RAM usage either, it just keeps ballooning the moment you start first inference.

[yagil]

@orcinus worth it to open an issue for performance. We'd track it separately. Also cc @Blaizzy

[orcinus]

I'm too slow - someone already did in here: #63
I've added my own case in that one.

[Blaizzy]

Thanks!

I'm working on it from tomorrow.