Add support for Gemini Structured Outputs

[willbakst]

Description

Recently Google released Structured Outputs for Gemini (much like OpenAI).

Should be fairly simple to enable this by updating setup_call for gemini and updating the warning to include gemini as a module that supports strict in the response model schema (although it seems like structured outputs for tools is not quite supported).

See: https://ai.google.dev/gemini-api/docs/json-mode?lang=python#supply-schema-in-config

Only difficulty I can see right now is that the only two supported inputs are either raw type annotations (so no BaseModel) or their genai.protos.Schema type (which will require conversion from BaseModel). Should have most of this implemented already I believe, so still shouldn't be terribly difficult.

[AbhishekRP2002]

hi @willbakst if this issue's still open, I would love to take this up!

[willbakst]

@AbhishekRP2002 it's open! would love for you to take this

if you have any questions let me know! our implementation of OpenAI structured outputs should act as a good reference

[AbhishekRP2002]

sure, will let you know if there's any blocker in any way.

[AbhishekRP2002]

hi @willbakst apologies for no updates since past 1 month, been piled with lots of eng work, so couldn't give this much time.
i was going through the codebase last night and noticed some form of implementation for structured_output in gemini, however i don't see the response_schema param in the config:

i also took a look at the openai implementation,
here is what i am thinking:
adding something like

if response_schema:
        # Convert response_schema to compatible format
        converted_schema = _convert_schema(response_schema)
        
        call_kwargs['generation_config'] = {
            'response_mime_type': 'application/json',
            'response_schema': converted_schema

i also noticed we can extract structured outputs via function calling in gemini which seemed more close to the openai implementation under _setup_call.py

found this example also: https://github.com/Mirascope/mirascope/blob/f4474e505a7bd67741faa44065fd7ba772681c50/examples/learn/response_models/json_mode/gemini/messages.py

seems like already implemented ?

would be really grateful if u can share your thoughts or help me out here in any way

[willbakst]

Ah I can see the confusion here. Hopefully this explanation will clear things up.

There are three types of structured outputs:

1. Tools (which we attempt to force to be called and then extract / convert into the originally provided model)
2. JSON mode (same thing as tools except we use JSON to power the structure)
3. Provider-specific structured outputs (i.e. OpenAI).

We've implemented (1) and (2) across all currently supported providers, but we've only implemented (3) for OpenAI (see here), but Gemini now also supports strict structured outputs.

The idea with this issue would be to add the same support for Gemini that we have for OpenAI (by setting model_config = ResponseModelConfigDict(strict=True)).