Add `builtin_tools` to `Agent` #2102

mattbrandman · 2025-06-30T23:53:02Z

Fixes test and merge conflicts for #1722

Closes #840

Co-authored-by: Marcelo Trylesinski <[email protected]>

…dantic#1752) Co-authored-by: Marcelo Trylesinski <[email protected]>

- Added builtin_tools field to ModelRequestParameters - Merged new output_mode and output_object fields from main - Updated test snapshots to include all fields - Resolved import conflicts to include both builtin tools and profiles

…test_google.py

Co-authored-by: David Montague <[email protected]>

mattbrandman · 2025-07-21T17:40:33Z

@Kludex unless we want to break these out into individual classes this feels like its in a pretty good spot for a first pass and to prevent it already feels like a giant PR

pydantic_ai_slim/pydantic_ai/agent.py

pydantic_ai_slim/pydantic_ai/builtin_tools.py

pydantic_ai_slim/pydantic_ai/messages.py

pydantic_ai_slim/pydantic_ai/models/anthropic.py

DouweM · 2025-07-23T14:55:10Z

pydantic_ai_slim/pydantic_ai/agent.py

@@ -307,6 +311,8 @@ def __init__(
            output_retries: The maximum number of retries to allow for output validation, defaults to `retries`.
            tools: Tools to register with the agent, you can also register tools via the decorators
                [`@agent.tool`][pydantic_ai.Agent.tool] and [`@agent.tool_plain`][pydantic_ai.Agent.tool_plain].
+            builtin_tools: The builtin tools that the agent will use. This depends on the model, as some models may not
+                support certain tools. On models that don't support certain tools, the tool will be ignored.


I feel like silently ignoring built-in tools is unexpected, because the model is not going to behave as intended if it can't search the web or execute code. I'd prefer to raise an error from the model class if it sees an unsupported built-in tool, similarly to how we do for unsupported file/binary content types.

I'm not sure it's the same case. I expect builtin tools to fail far more than the different content inputs.

@Kludex I agree, but wouldn't you want the user to realize that they asked for something the model can't actually do?

@mattbrandman Please make it a hard error if an unsupported built-in tool was provided -- I talked about it with Marcelo and he's fine with it.

That means every model class will need to check model_request_parameters.builtin_tools and error if it doesn't implement any at all

pydantic_ai_slim/pydantic_ai/models/groq.py

DouweM · 2025-07-23T15:03:17Z

pydantic_ai_slim/pydantic_ai/models/openai.py

@@ -734,6 +754,7 @@ async def _responses_create(
    ) -> responses.Response | AsyncStream[responses.ResponseStreamEvent]:
        tools = self._get_tools(model_request_parameters)
        tools = list(model_settings.get('openai_builtin_tools', [])) + tools


Should we deprecate this setting and push people to use the new builtin_tools?

This setting currently supports FileSearchToolParam and ComputerToolParam as well, should we implement those as AbstractBuiltinTools?

Should we have a way to build an AbstractBuiltinTool with arbitrary built-in tool JSON supported by a given model, so users can start using them without having to wait for us to add support to the model class?

Should we deprecate this setting and push people to use the new builtin_tools?

Yes.

This setting currently supports FileSearchToolParam and ComputerToolParam as well, should we implement those as AbstractBuiltinTools?

I don't think they work properly right now, but the idea is to support them yes.

Should we have a way to build an AbstractBuiltinTool with arbitrary built-in tool JSON supported by a given model, so users can start using them without having to wait for us to add support to the model class?

Yes, but that needs to be model specific, so I left it out of this PR. I think we should have that possibility tho.

Let's leave openai_builtin_tools for now, as I don't want to further delay this PR by having to support FileSearch and ComputerTool. We'll add ComputerUse later as Anthropic also supports it: https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/computer-use-tool, but FileSearch is currently OpenAI-specific so it seems too early to define a generic interface.

pyproject.toml

Co-authored-by: Douwe Maan <[email protected]>

DouweM

@mattbrandman Thanks for your patience, he's another review! 22 comments sounds like like a lot but it should all be pretty straightforward...

pydantic_ai_slim/pydantic_ai/messages.py

DouweM · 2025-07-29T18:07:47Z

pydantic_ai_slim/pydantic_ai/messages.py

+
+
+@dataclass(repr=False)
+class ServerToolCallPart(BaseToolCallPart):


This (plus docstring and part_kind) also still needs to be renamed to Builtin!

DouweM · 2025-07-29T18:08:20Z

pydantic_ai_slim/pydantic_ai/messages.py

@@ -1145,6 +1181,29 @@ def tool_call_id(self) -> str:
    __repr__ = _utils.dataclasses_no_defaults_repr


+@dataclass(repr=False)
+class ServerToolCallEvent:


This and the next event should also be renamed

DouweM · 2025-07-29T18:10:23Z

pydantic_ai_slim/pydantic_ai/models/huggingface.py

+                        # This is currently never returned from huggingface
+                        pass
+                    elif isinstance(item, BuiltinToolReturnPart):  # pragma: no cover
+                        # BuiltinToolReturnPart represents a tool return from a remote server


This is ambiguous as "remote server" could mean e.g. MCP. Let's make it clear this is specifically about built-in tools and drop "server" everywhere

I think we can just drop this line FYI

DouweM · 2025-07-29T18:11:41Z

pydantic_ai_slim/pydantic_ai/models/cohere.py

+                        # This is currently never returned from cohere
+                        pass
+                    elif isinstance(item, BuiltinToolReturnPart):  # pragma: no cover
+                        # BuiltinToolReturnPart represents a tool return from a remote server


Like I mentioned elsewhere, let's drop this line as the "remote server" bit is potentially confusing and the class name is clear enough

DouweM · 2025-07-29T18:37:04Z

pydantic_ai_slim/pydantic_ai/models/openai.py

+                        type='approximate', **tool.user_location
+                    )
+                tools.append(web_search_tool)
+        return tools


We can support code execution here as well: https://platform.openai.com/docs/guides/tools-code-interpreter

DouweM · 2025-07-29T18:38:31Z

pydantic_ai_slim/pydantic_ai/models/openai.py

@@ -893,6 +928,9 @@ async def _map_messages(
                        openai_messages.append(responses.EasyInputMessageParam(role='assistant', content=item.content))
                    elif isinstance(item, ToolCallPart):
                        openai_messages.append(self._map_tool_call(item))
+                    # OpenAI doesn't return server tool calls
+                    elif isinstance(item, (ServerToolCallPart, BuiltinToolReturnPart)):
+                        continue


Suggested change

continue

pass

DouweM · 2025-07-29T18:39:43Z

pydantic_ai_slim/pydantic_ai/builtin_tools.py

+    """
+
+    city: str
+    country: str


On OpenAI, these needs to be a 2-letter country code: https://platform.openai.com/docs/guides/tools-web-search?api-mode=responses#user-location

DouweM · 2025-07-29T18:43:37Z

pydantic_ai_slim/pydantic_ai/models/openai.py

@@ -298,6 +298,9 @@ async def _completions_create(
        model_request_parameters: ModelRequestParameters,
    ) -> chat.ChatCompletion | AsyncStream[ChatCompletionChunk]:
        tools = self._get_tools(model_request_parameters)
+        web_search_options = self._get_web_search_options(model_request_parameters)
+
+        # standalone function to make it easier to override


Suggested change

# standalone function to make it easier to override

DouweM · 2025-07-29T18:47:12Z

pydantic_ai_slim/pydantic_ai/models/openai.py

@@ -734,6 +754,7 @@ async def _responses_create(
    ) -> responses.Response | AsyncStream[responses.ResponseStreamEvent]:
        tools = self._get_tools(model_request_parameters)
        tools = list(model_settings.get('openai_builtin_tools', [])) + tools


Let's leave openai_builtin_tools for now, as I don't want to further delay this PR by having to support FileSearch and ComputerTool. We'll add ComputerUse later as Anthropic also supports it: https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/computer-use-tool, but FileSearch is currently OpenAI-specific so it seems too early to define a generic interface.

DouweM

@mattbrandman Thanks for your patience, he's another review! 22 comments sounds like like a lot but it should all be pretty straightforward...

Kludex and others added 30 commits May 14, 2025 11:00

Add builtin_tools to Agent

57e568b

make AbstractBuiltinTool serializable

97ab44b

Add more work on it

e3dda9d

Merge remote-tracking branch 'origin/main' into add-builtin-tools

3ad6d38

Add builtin tools

0b43f65

merge

fa7fd11

add more built-in-tools

32324fa

Fix test

f33e568

Add support on Groq

13d7433

Add support for Google

ac85205

Add support for MCP's Streamable HTTP transport (pydantic#1716)

c93633f

Timeout for initializing MCP client (pydantic#1833)

3a8b640

Require mcp 1.9.0+ (pydantic#1840)

360de87

Don't send empty messages to Anthropic (pydantic#1027)

cb4e539

Co-authored-by: Marcelo Trylesinski <[email protected]>

Add vendor_id and finish_reason to Gemini/Google model responses (p…

4e3769a

…ydantic#1800)

Fix units of sse_read_timeout timedelta (pydantic#1843)

ebb536f

Support functions as output_type, as well as lists of functions and o…

c8bb611

…ther types (pydantic#1785)

Enhance Gemini usage tracking to collect comprehensive token data (py…

6bcc1a8

…dantic#1752) Co-authored-by: Marcelo Trylesinski <[email protected]>

more

97ff651

merge

1d47e1e

merge

5f89444

merge

9512987

Pass tests

800a71a

Fix remaining merge conflict markers in openai.py, anthropic.py, and …

bc298d6

…test_google.py

add extra google

46c06c2

fix formatting

3496567

fix codespell

c193059

fixing types

427dec2

fixing types in gemini

866ad21

DouweM mentioned this pull request Jul 16, 2025

feat: Add Google Search grounding to GoogleModel - open issue #840 #2161

Closed

Apply suggestions from code review

4376fc2

Co-authored-by: David Montague <[email protected]>

mattbrandman force-pushed the builtin-tool branch from c4aa1ef to 4376fc2 Compare July 21, 2025 01:45

mattbrandman and others added 8 commits July 20, 2025 22:05

Merge branch 'main' into builtin-tool

c029d4b

updates types

87f59f1

updating types

3fca6d4

fixing event

37a1845

Merge branch 'main' into builtin-tool

5b031d6

updates

e1085fa

Merge branch 'main' into builtin-tool

ae1d4d9

Merge branch 'main' into builtin-tool

3ee4aa9

mattbrandman added 3 commits July 21, 2025 18:43

Merge branch 'main' into builtin-tool

9d24f5b

Merge branch 'main' into builtin-tool

07f7892

Merge branch 'main' into builtin-tool

7d867ee

DouweM requested changes Jul 23, 2025

View reviewed changes

Kludex and others added 10 commits July 23, 2025 17:52

Update pydantic_ai_slim/pydantic_ai/agent.py

ba3f6b2

Co-authored-by: Douwe Maan <[email protected]>

addressing comments

1ae6155

Merge branch 'main' into builtin-tool

3f8cc2d

updates

ba962a7

fixing lint

bfd9b21

fixing lint

e2fef42

fix misspellings

b1ef2f8

rename servertool to builtin tool everywhere

e9cacc3

fixing ordering

aa91640

Merge branch 'main' into builtin-tool

2d451a7

DouweM requested changes Jul 29, 2025

View reviewed changes

DouweM self-assigned this Jul 29, 2025

DouweM added the awaiting author revision label Jul 29, 2025



		@dataclass(repr=False)
		class ServerToolCallPart(BaseToolCallPart):

Add builtin_tools to Agent #2102

Are you sure you want to change the base?

Add builtin_tools to Agent #2102

Conversation

mattbrandman commented Jun 30, 2025 • edited by DouweM Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mattbrandman commented Jul 21, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

DouweM left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DouweM left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Add `builtin_tools` to `Agent` #2102

Add `builtin_tools` to `Agent` #2102

mattbrandman commented Jun 30, 2025 •

edited by DouweM

Loading