.Net: Function name policy to control function fqn creation and parsing #10206
+771
−61
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
markwallace-microsoft
added
.NET
SergeyMenshykh
added
ai connector
SergeyMenshykh
had a problem deploying
to
integration
GitHub Actions
Failure
markwallace-microsoft
added
kernel
SergeyMenshykh
had a problem deploying
to
integration
GitHub Actions
Failure
…te compatibility sepressions.
SergeyMenshykh
temporarily deployed
to
integration
GitHub Actions
Inactive
SergeyMenshykh
temporarily deployed
to
integration
GitHub Actions
Inactive
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation, Context
Today, each AI connector uses a hyphen ("-") as a plugin-function name separator when creating a fully qualified name (FQN) for all functions to advertise them to an AI model. On the way back, they also use the hyphen when splitting FQNs into plugin and function names while handling function calls from the AI model.
From time to time, AI models hallucinate/replace the FQN separator with a different one. For example, if a function is advertised with the
TimePlugin-GetCurrentUtcDateTimeFQN, the AI model may call it with theTimePlugin.GetCurrentUtcDateTimeorTimePlugin_GetCurrentUtcDateTimeFQNs. As a result, when the AI model calls the functions with one of the unexpected separators, SK cannot find the function during the lookup process to invoke it and sends this information back to the AI model, which fails the request with the error: "Invalid 'messages[6].tool_calls[0].function.name': string does not match pattern. Expected a string that matches the pattern '^[a-zA-Z0-9-]+$'"_.Description
This PR enables a few ways for the consuming code to tackle the issue described above via the new
FunctionNamePolicyclass:Use a custom separator instead of the default hyphen. The idea here is that the custom separator might be more natural as part of a function name for the AI model, so it stops changing it.
Provide an FQN parsing delegate to customize the process of splitting the function FQN from the function call by trying different separators. The idea here is to handle the hallucinated name on the client side without unnecessary roundtrips (which may fail anyway if the hallucinated function name contains a dot) to the AI model, only to tell it that it called the function with the wrong separator.
Advertise functions using their names only as FQNs. The idea here is that if the plugin name is not included in the FQN, then the separator is not needed, and as a result, there's nothing to hallucinate. Of course, it's possible to hallucinate the function names in a few other ways, but that has not been the case so far, or at least we have not heard about it.