How to use "response_format": { "type": "json_object" } on azure ai studio serverless endpoint

[SaahilClaypool]

When deploying Phi-3-mini-128k-instruct serverless endpoint on azure AI studio and setting the response_format field, the endpoint returns only a stream of newline characters.

I assume this is because the model doesn't support the response_format parameter, but according to the documentation I would think the API would return 422 in that case.

This issue is for a: (mark with an x)

- [x] bug report -> please search issues before submitting
- [x] feature request
- [x] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

POST https://Phi-3-mini-128k-instruct-serverless.eastus2.inference.ai.azure.com/v1/chat/completions
Content-Type: application/json
Authorization: <key>

{
  "messages": [
    {
      "role": "user",
      "content": "What are the top places in Paris? Respond in JSON with the fields 'location_name', 'location_description'"
    }
  ],
  "response_format": {
    "type": "json_object"
  },
  "temperature": 0.7,
  "max_tokens": 100
}

Response:

{
  "id": "cmpl-efb28e8085204ec3a1d2597c639083f4",
  "object": "chat.completion",
  "created": 1719928871,
  "model": "phi3-mini-128k",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": " \n\n{\n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n ",
        "tool_calls": []
      },
      "finish_reason": "length",
      "logprobs": null,
      "stop_reason": null
    }
  ],
  "usage": {
    "prompt_tokens": 28,
    "total_tokens": 128,
    "completion_tokens": 100
  }
}

[leestott]

Hi @SaahilClaypool see https://learn.microsoft.com/en-us/azure/ai-studio/how-to/deploy-models-phi-3?tabs=phi-3-mini

Consume Phi-3 models as a service
Models deployed as serverless APIs can be consumed using the chat API, depending on the type of model you deployed.

From your Project overview page, go to the left sidebar and select Components > Deployments.

Find and select the deployment you created.

Copy the Target URL and the Key value.

Make an API request using the /v1/chat/completions API using <target_url>/v1/chat/completions. For more information on using the APIs, see the Reference: Chat Completions.