feat(ai): llm provider

[kallebysantos]

What kind of change does this PR introduce?

feature, refactor

What is the current behaviour?

Current the Session only supports self-hosted Ollama or some OpenAI like provider - with no way to specify the API key.

What is the new behaviour?

This PR applies some refactors in the ai module to support an unified LLM provider API, this way it can be easily extended to new providers as well exporting a more standardised output format.

Improved typescript support

The ai module was huge refactored to provide better ts hints that dynamically changes based on the selected type:

examples

using type gte-small:

using type ollama:

using type openaicompatible:

Automatically infer AsyncGenerator type when stream: true

All providers, ollama & openai

Improved error handling support

In order to ensure error checking, the ai module was been refactored to follow Result pattern - Go like. It means that while Session.run() the returned value will be a tuple array of [success, error], this result is compatible with TS pattern matching, so it provides a completely LSP feedback.

examples

Non stream

Result type def

Checking error automatically validates the success part

Stream

When stream: true the first result will handle errors that may occur before create the AsyncGenerator.
Then the incoming message will be a result as well, this way users can apply error handling while streaming.

Result type def

Streaming type def

Common response and Usage metrics

Since all LLM providers must implement a common interface they now also shared a unified response object.

response definitions

Success part

// This object represents a success LLM output, where inner is the raw value returned by the provider
export interface ILLMProviderOutput<T = object> {
  value?: string; // An unified way to get the generated value
  usage: {
    inputTokens: number;
    outputTokens: number;
    totalTokens: number;
  };
  inner: T;
}

Error part

// This object represents an error output, where inner is the original error value
export type SessionError<T = object | string> = {
  message: string;
  inner: T;
};

Tested OpenAI compatible providers

• OpenAI
• TogetherAI

missing

• Anthropic: I don't have free account for it
• Grok: I don't have free account for it
• Amazon Bedrock: I don't have free account for it

ideas

• Gemini: maybe implement a custom provider for this one