Using OpenAI API compliant to support vision models

[madroidmaq]

curl

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer OPENAI_API_KEY" \
  -d '{
    "model": "microsoft/Phi-3.5-vision-instruct",
    "messages": [
      {
        "role": "user",
        "content": [
          {
            "type": "text",
            "text": "What’s in this image?"
          },
          {
            "type": "image_url",
            "image_url": {
              "url": "http://images.cocodataset.org/val2017/000000039769.jpg"
            }
          }
        ]
      }
    ],
    "max_tokens": 300
  }'

response:

{
	"id": "chatcmpl-6a138c4c",
	"object": "chat.completion",
	"created": 1724780023,
	"model": "microsoft/Phi-3.5-vision-instruct",
	"choices": [{
		"index": 0,
		"message": {
			"role": "assistant",
			"content": "The image shows two cats lying on a pink couch. There are two remote controls placed on the couch next to the cats.<|end|>"
		},
		"finish_reason": "stop"
	}],
	"tool_calls": []
}

[Blaizzy]

Hey @madroidmaq

Sorry for the late response, I was unavaible these past couple weeks but I'm back.

Thank you very much for the PR!

[Blaizzy]

@madroidmaq the style checks are failling, please run:

 pre-commit run --all

[Blaizzy]

After that, if all tests pass we can merge :)

[madroidmaq]

@Blaizzy I have fixed some issues in pre-commit.

[Blaizzy]

Thanks @madroidmaq!

This is an awesome change. Will simplify multiple images as well.

[Blaizzy]

LGTM!