WIP Mitigate hallucinations with codeblocks containing Typescript/Web Components/Javascript

[nelsonwittwer]

Why

There is nothing more costly to productivity than LLM hallucinations within codeblocks that accepted into a codebase.

We have mitigated that risk with GraphQL. We need to do the same with Typescript, Web components, and Javascript codeblocks using Shopify's APIs.

What

Even if users aren't using Typescript (web components with polaris app home), we can still use the types of the Typescript APIs that are directly or indirectly used.

This introduces a new validate_typescript_codeblocks MCP tool that accepts a TS package name that will be used to detect hallucinations / errors in codeblocks LLMs generated/modified.

In the case of app home, it will accept the Typescript library @shopify/app-bridge-ui-types that app home uses indirectly as our TS source of truth. We'll configure learn_shopify_api's response to require all the LLM to use this tool to validate any codeblocks they would offer return to a user.

WIP

What's still in question is how we will use TS .

This uses Typia to validate TS without having a user's machine. I need to stress test the implementation with more than the 3 weak sauce evals I've been testing this with.

This previous idea added a temporary TS file and using the user's CLI to typescript compile. I worry about how many more things can go wrong with this case: the user doesn't have typescript installed because they prefer JS/Web components. This could send an agent spiraling and modify the user's machine in a way that isn't necessary. It also has the risk of not cleaning up after itself (it temporariliy creates a TS file to run the typescript compile command.