Specifying a response_model that is nested fails for both gemini and vertex, even when setting json_mode=True

[barapa]

Description

Mirascope appears to be using tools for structured output from gemini and vertex instead of model_json_schema when setting json_mode=True.

When running this

from typing import Annotated
from pydantic import BaseModel, Field
from datetime import date
from mirascope.core import BaseMessageParam, Messages, anthropic, openai, vertex, gemini  # noqa: F401

class Book(BaseModel):
    title: Annotated[str, Field(description="The title of the book")]
    published_date: Annotated[date, Field(description="The date the book was published")]

class Author(BaseModel):
    name: Annotated[str, Field(description="The name of the author")]
    books: Annotated[list[Book], Field(description="A list of books written by the author")]

@vertex.call("gemini-1.5-pro-002", response_model=Author, json_mode=True)
async def generate_random_author(genre: str) -> str:
    return f"Generate a random author with 4 books written in the genre {genre}"

author = await generate_random_author("Science Fiction")

We get this error

Traceback (most recent call last):
  File "/Users/br/development/projects/legaide-api/.venv/lib/python3.12/site-packages/marimo/_runtime/executor.py", line 131, in execute_cell_async
    await eval(cell.body, glbls)
  Cell marimo://testing_marimo.py#cell=cell-2, line 18, in <module>
    author = await generate_random_author("Science Fiction")
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/br/development/projects/legaide-api/.venv/lib/python3.12/site-packages/mirascope/core/base/_extract.py", line 132, in inner_async
    call_response = await create_decorator(
                    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/br/development/projects/legaide-api/.venv/lib/python3.12/site-packages/mirascope/core/base/_create.py", line 168, in inner_async
    create, prompt_template, messages, tool_types, call_kwargs = setup_call(  # pyright: ignore [reportCallIssue]
                                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/br/development/projects/legaide-api/.venv/lib/python3.12/site-packages/mirascope/core/vertex/_utils/_setup_call.py", line 98, in setup_call
    prompt_template, messages, tool_types, base_call_kwargs = _utils.setup_call(
                                                              ^^^^^^^^^^^^^^^^^^
  File "/Users/br/development/projects/legaide-api/.venv/lib/python3.12/site-packages/mirascope/core/base/_utils/_setup_call.py", line 77, in setup_call
    call_kwargs["tools"] = [tool_type.tool_schema() for tool_type in tool_types]
                            ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/br/development/projects/legaide-api/.venv/lib/python3.12/site-packages/mirascope/core/vertex/tool.py", line 71, in tool_schema
    raise ValueError(
ValueError: Unfortunately Google's Vertex API cannot handle nested structures with $defs.

Python, Mirascope & OS Versions, related packages (not required)

No response

[willbakst]

We should be doing this correctly here, but in base._utils.setup_call we always add "tools" to call_kwargs and then pop it off if we're in JSON mode.

In doing this digging, I'm realizing a few things:

1. We should not be popping tools out of call_kwargs when json_mode=True unless extract is also true since the user may want to use tools with JSON mode.
2. Since setting json_mode with response_model would indicate extraction, we should just skip the tool creation entirely in this case.
3. We can probably implement Is it possible to Combined Response Model Structure Output and Function Calling? #709 by setting response_model with json_mode=True and tools=[...] where the return type would be BaseCallResponseT or ResponseModelT (where if tools are present we would return BaseCallResponseT and otherwise we would try to structure the call as ResponseModelT using the final JSON output).

[willbakst]

For gemini models it appears like we should also remove any references in the schema before passing it to the model.