Handle vision language model chat templates

[DePasqualeOrg]

Now that mlx-libraries supports vision models (thanks to @davidkoski's Herculean efforts), we should try to support multimodal chat templates. I'll list some models here for reference.

• Qwen2-VL
• Llama 3.2
• Llava v1.6
• Molmo
• Phi 3.5 Vision

These models have no chat template in tokenizer_config.json:

• Paligemma
• Pixtral

[johnmai-dev]

Thank you for your suggestion. I will follow up on this issue next month, as I've been a bit busy recently.

[davidkoski]

For reference here is the code from mlx-vlm that generates the messages:

• https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/prompt_utils.py#L45

and the matching code in the swift VLM for Qwen2-VL:

• https://github.com/ml-explore/mlx-swift-examples/blob/main/Libraries/MLXVLM/Models/Qwen2VL.swift#L689

The Qwen2-VL chat template expects structured content like this:

[{'role': 'user', 'content': [{'type': 'text', 'text': 'What are these?'}, {'type': 'image'}, {'type': 'image'}, {'type': 'image'}]}]

rather than [[String:String]].