Add image and audio prompting API

[domenic]

Closes #40. Somewhat helps with #70.

[michaelwasserman]

Some models may only accept a single image or single audio input with each request. Consider describing that edge-case detail behavior (e.g. throw a "NotSupportedError" DOMException) when an unsupported number or combination of image/audio prompt pieces are passed. Maybe also give a single-image example here for wider compatibility.

[domenic]

I think we should encapsulate that away from the user so they don't have to worry about it, by sending two requests to the backend.

[sushraja-msft]

I think number of images aspect of this can be handled through the contextoverflow event. https://github.com/webmachinelearning/prompt-api#tokenization-context-window-length-limits-and-overflow.

Phi 3.5 vision https://huggingface.co/microsoft/Phi-3.5-vision-instruct, recommends at the most 16 images but passing more (say 17) will only run into context length limitations. The context length limitation can be reached earlier as well with large image.

Sending two requests to the backend may not work though, the assistant is going to add its response in between.

Are there other limitations in terms of no mixing of images/audio or limitations in number of images/audio. Ill have to check what the server-side models do for their API.

[michaelwasserman]

Is it worthwhile to have separate role enums and dictionaries for initial/not lines?
Would simply throwing exceptions for invalid 'system' role use in the impl suffice?

[michaelwasserman]

I realize you've explored this thoroughly, but I wonder why the idl couldn't be simpler, like:

dictionary AILanguageModelPromptDict {
  required AILanguageModelPromptRole role = "user";
  required AILanguageModelPromptType type = "text";
  required AILanguageModelPromptData data;
}

typedef (DOMString or AILanguageModelPromptDict) AILanguageModelPrompt;
typedef (AILanguageModelPrompt or sequence<AILanguageModelPrompt>) AILanguageModelPromptInput;

[domenic]

Is it worthwhile to have separate role enums and dictionaries for initial/not lines?
Would simply throwing exceptions for invalid 'system' role use in the impl suffice?

Yeah that's totally valid. If you as an implementer think that's simpler, we can do it.

I realize you've explored this thoroughly, but I wonder why the idl couldn't be simpler, like:

(Note that required and default values are not compatible.)

So, your version is mainly about unnesting the content, right?

I think it mostly works, it just departs from what's idiomatic with other chat completion APIs. Hmm.

[michaelwasserman]

Yeah, I imagined unnesting might slightly simplify IDL, impl, and atypical uses specifying more fields.
create({initialPrompts: [{role: "system", content: {type: "image", data: pixels}}]})
vs
create({initialPrompts: [{role: "system", type: "image", data: pixels}]})

I'm unfamiliar with rationale behind other APIs' idiomatic designs, so I'd readily defer to folks that know more, and maybe even opt for the more familiar pattern.

[domenic]

I think it's worth considering, since we're not matching the APIs exactly in other ways. I will try to outline the two possibilities in more detail to get developer feedback in #40, on Monday.