Consider caching JSON schema generation in function calling #3403
Description
Current Performance Issue
Function calling applications regenerate identical JSON schemas on every request, causing significant CPU overhead.
Root cause analysis:
// User code pattern
chatClient.prompt("What's the weather?")
.tools(new WeatherTools()) // New instance every time
.call();
// Execution chain:
// 1. ToolCallbacks.from(new WeatherTools())
// 2. MethodToolCallbackProvider.getToolCallbacks() - no caching
// 3. ToolDefinitions.from(toolMethod) - called for each @Tool method
// 4. JsonSchemaGenerator.generateForMethodInput(method) - CPU intensiveEvidence from source code:
MethodToolCallbackProvider.getToolCallbacks() creates new tool definitions every time:
public ToolCallback[] getToolCallbacks() {
return this.toolObjects.stream()
.map(toolMethod -> MethodToolCallback.builder()
.toolDefinition(ToolDefinitions.from(toolMethod)) // No caching here
.build())
.toArray(ToolCallback[]::new);
}ToolDefinitions.from() calls schema generation without caching:
public static ToolDefinition from(Method method) {
return builder(method).build(); // Always creates new
}
public static DefaultToolDefinition.Builder builder(Method method) {
return DefaultToolDefinition.builder()
.inputSchema(JsonSchemaGenerator.generateForMethodInput(method)); // Expensive call
}JsonSchemaGenerator.generateForMethodInput() performs expensive operations every time:
public static String generateForMethodInput(Method method, SchemaOption... schemaOptions) {
// Complex schema generation with reflection
for (int i = 0; i < method.getParameterCount(); i++) {
ObjectNode parameterNode = SUBTYPE_SCHEMA_GENERATOR.generateSchema(parameterType); // CPU intensive
}
return schema.toPrettyString(); // JSON serialization
}Performance Impact
Typical scenario:
@RestController
public class ChatController {
@PostMapping("/chat")
public String chat(@RequestBody String message) {
// Same schemas regenerated on every HTTP request
return chatClient.prompt(message)
.tools(new WeatherTools()) // 2 @Tool methods = 2 schema generations
.call().content();
}
}Current behavior creates repeated work:
- Same method signatures generate identical schemas multiple times
- No reuse of schema generation results across requests
- Optimization opportunity exists for applications with repeated tool usage