Rendered prompt does not show equivalent JSON payload when calling OpenAI API directly

[vicperdana]

Hi,
When using the handlebar template, it seems that the rendered prompt is different than what's being accepted in OpenAI API directly. This is our handlebar template and rendered prompt.

Is the rendered prompt being 'translated' to the acceptable JSON payload before passing to OpenAI?
If, so how do we view the raw request made to OpenAI?

Handlebar template and resulting rendered prompt

name: transcribeScreenshot
description: Extract text from image
template: |
  <message role='user'>
   <text>what is this image</text>
   <image>https://i.imgur.com/1RP78PA.jpeg</image>
  </message>
template_format: handlebars

Output:

trce: Microsoft.SemanticKernel.KernelFunctionFactory[0]
      Rendered prompt: <message role='user'>
       <text>what is this image</text>
       <image>https://i.imgur.com/1RP78PA.jpeg</image>
      </message>

OpenAI API direct call with Postman

Request

{
  "model": "gpt-4o",
  "messages": [
    {
      "role": "user",
      "content": "<message role='user'><text>what is this image</text><image>https://i.imgur.com/1RP78PA.jpeg</image></message>"
    }
  ]
}

Output

{
  "id": "chatcmpl-xxxxx",
  "object": "chat.completion",
  "created": 1718774484,
  "model": "gpt-4o-2024-05-13",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "I'm currently unable to view images directly. However, you can describe the image to me, and I can help provide information or context based on your description!"
      },
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 40,
    "completion_tokens": 31,
    "total_tokens": 71
  },
  "system_fingerprint": "fxxxxx"
}

[dmytrostruk]

@vicperdana

Is the rendered prompt being 'translated' to the acceptable JSON payload before passing to OpenAI?

That's correct.

If, so how do we view the raw request made to OpenAI?

You can define DelegatingHandler, which you can inject in HttpClient and pass that HttpClient when you register Azure/OpenAI chat completion service.

Examples of logging handler:

• semantic-kernel/dotnet/src/InternalUtilities/samples/InternalUtilities/BaseTest.cs

  Line 103 in 745e64a

  protected sealed class LoggingHandler(HttpMessageHandler innerHandler, ITestOutputHelper output) : DelegatingHandler(innerHandler)

• semantic-kernel/dotnet/src/Plugins/Plugins.MsGraph/Connectors/Client/MsGraphClientLoggingHandler.cs

  Line 20 in 745e64a

  public class MsGraphClientLoggingHandler : DelegatingHandler

Example of logging handler injection to see raw request made to OpenAI:

semantic-kernel/dotnet/samples/Concepts/ChatCompletion/OpenAI_FunctionCalling.cs

Lines 100 to 108 in 745e64a

	var handler = new LoggingHandler(new HttpClientHandler(), this.Output);
	HttpClient httpClient = new(handler);
	// Create a kernel with OpenAI chat completion and WeatherPlugin
	IKernelBuilder kernelBuilder = Kernel.CreateBuilder();
	kernelBuilder.AddOpenAIChatCompletion(
	modelId: TestConfiguration.OpenAI.ChatModelId!,
	apiKey: TestConfiguration.OpenAI.ApiKey!,
	httpClient: httpClient);