chore(tools): POC to consolidate immutable ddl tools while preserving the accuracy #354
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
himanshusinghs
force-pushed
the
poc/tool-consolidation
branch
from
e20104b to
e846520
Compare
LangChain's ToolCalling agent was not providing a structured tool call response and different model providers were providing entirely different tool calls for the same tool definition which was too turbulent for us to have any accuracy baseline at all. Vercel's AI SDK pushes us forward on that problem and the tool call responses so far have always been well structured. This commit replaces LangChain based implementation with Vercel's AI SDK based implementation.
When writing test cases, I realized that it is too much duplicated effort to write and maintain mocks. So instead of having only a mocked mcp client, this commit introduces a real mcp client that talks to our mcp server and is still mockable. We are now setting up real MCP client with test data in mongodb database spun up for test suites. Mocking is still an option but we likely never feel the need for that.
introduces the following necessary env variables: - MDB_ACCURACY_RUN_ID: The accuracy run id - MDB_ACCURACY_MDB_URL: The connection string to mongodb instance where the snapshots will be stored - MDB_ACCURACY_MDB_DB: The database for snapshots - MDB_ACCURACY_MDB_COLLECTION: The collection for snapshots
The new field `accuracyRunStatus` is supposed to help guard against cases where jest might fail in between, maybe due to LLM rate limit errors or something else, and we then have a partially saved state of an accuracy run. With the new field `accuracyRunStatus` we should be able to safely look for last runs where `accuracyRunStatus` is done and have complete state of accuracy snapshot.
…accuracyRunStatus
…racy-status script
…nts. 1. Removes unnecessary suite description from tests 2. Removes the test suite name from the storage as well 3. Centralize the constants used everywhere in the SDK 4. Adds clarifying comments and docs wherever necessary 5. Write tests for accuracy-scorer
Co-authored-by: Nikola Irinchev <[email protected]>
Co-authored-by: Nikola Irinchev <[email protected]>
Co-authored-by: Nikola Irinchev <[email protected]>
Co-authored-by: Nikola Irinchev <[email protected]>
Instead of storing multiple documents per accuracy test run(one for each prompt+model response), we will now be storing one document for accuracy result and under that, all the prompt+model responses will be nested.
himanshusinghs
force-pushed
the
chore/issue-307-proposal-2
branch
from
f666014 to
1cc93f2
Compare
himanshusinghs
force-pushed
the
poc/tool-consolidation
branch
from
e846520 to
2062d6f
Compare
1. use commit sha for github actions 2. run workflow also on pushes to main 3. use ai-sdk/google instead of privately published package
himanshusinghs
force-pushed
the
poc/tool-consolidation
branch
from
2062d6f to
cf32564
Compare
Consolidates list-databases, list-collections, collection-indexes and collection-schema. The tool calling accuracy went from 100 to 75. The LLM was always mistaking the command name to be listDatabases instead of list-databases and then course correcting when shown the error.
In the previous commit, our schema description was not clear enough for LLM to correctly provide the name of command and it was hallucinating by providing listDatabases for example instead of list-databases. This commit modifies the descriptions a bit to achieve 100% accuracy for the same prompts.
himanshusinghs
force-pushed
the
poc/tool-consolidation
branch
from
cf32564 to
be45e70
Compare
himanshusinghs
force-pushed
the
chore/issue-307-proposal-2
branch
from
34dd207 to
562a2cb
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Checklist