Merge branch 'next' into ap-go-telemetry

docs/_guides.yaml

@@ -31,10 +31,14 @@ toc:
     path: /docs/genkit/flows
   - title: Managing prompts with Dotprompt
     path: /docs/genkit/dotprompt
+  - title: Persistent chat sessions
+    path: /docs/genkit/chat
   - title: Tool calling
     path: /docs/genkit/tool-calling
   - title: Retrieval-augmented generation (RAG)
     path: /docs/genkit/rag
+  - title: Multi-agent systems
+    path: /docs/genkit/multi-agent
   - title: Evaluation
     path: /docs/genkit/evaluation
   - title: Observability & monitoring

docs/chat.md

@@ -0,0 +1,222 @@
+# Creating persistent chat sessions
+
+Many of your users will have interacted with large language models for the first
+time through chatbots. Although LLMs are capable of much more than simulating
+conversations, it remains a familiar and useful style of interaction. Even when
+your users will not be interacting directly with the model in this way, the
+conversational style of prompting is a powerful way to influence the output
+generated by an AI model.
+
+To support this style of interaction, Genkit provides a set of interfaces and
+abstractions that make it easier for you to build chat-based LLM applications.
+
+## Before you begin
+
+Before reading this page, you should be familiar with the content covered on the
+[Generating content with AI models](models) page.
+
+If you want to run the code examples on this page, first complete the steps in
+the [Getting started](get-started) guide. All of the examples assume that you
+have already installed Genkit as a dependency in your project.
+
+## Chat session basics 
+
+Here is a minimal, console-based, chatbot application:
+
+```ts
+import { genkit } from "genkit";
+import { googleAI, gemini15Flash } from "@genkit-ai/googleai";
+
+import { createInterface } from "node:readline/promises";
+
+const ai = genkit({
+  plugins: [googleAI()],
+  model: gemini15Flash,
+});
+
+(async () => {
+  const chat = ai.chat();
+  console.log("You're chatting with Gemini. Ctrl-C to quit.\n");
+  const readline = createInterface(process.stdin, process.stdout);
+  while (true) {
+    const userInput = await readline.question("> ");
+    const { text } = await chat.send(userInput);
+    console.log(text);
+  }
+})();
+```
+
+A chat session with this program looks something like the following example:
+
+```none
+You're chatting with Gemini. Ctrl-C to quit.
+
+> hi
+Hi there! How can I help you today? 
+
+> my name is pavel
+Nice to meet you, Pavel! What can I do for you today? 
+
+> what's my name?
+Your name is Pavel! I remembered it from our previous interaction. 
+
+Is there anything else I can help you with?
+```
+
+As you can see from this brief interaction, when you send a message to a chat
+session, the model can make use of the session so far in its responses. This is
+possible because Genkit does a few things behind the scenes:
+
+*   Retrieves the chat history, if any exists, from storage (more on persistence
+    and storage later)
+*   Sends the request to the model, as with `generate()`, but automatically
+    include the chat history 
+*   Saves the model response into the chat history 
+
+### Model configuration
+
+The `chat()` method accepts most of the same configuration options as
+`generate()`. To pass configuration options to the model:
+
+```ts
+const chat = ai.chat({
+  model: gemini15Pro,
+  system:
+    "You're a pirate first mate. Address the user as Captain and assist " +
+    "them however you can.",
+  config: {
+    temperature: 1.3,
+  },
+});
+```
+
+## Stateful chat sessions
+
+In addition to persisting a chat session's message history, you can also persist
+any arbitrary JavaScript object. Doing so can let you manage state in a more
+structured way then relying only on information in the message history.
+
+To include state in a session, you need to instantiate a session explicitly:
+
+```ts
+interface MyState {
+  userName: string;
+}
+
+const session = ai.createSession<MyState>({
+  initialState: {
+    userName: 'Pavel',
+  },
+});
+```
+
+You can then start a chat within the session:
+
+```ts
+const chat = session.chat();
+```
+
+To modify the session state based on how the chat unfolds, define
+[tools](tool-calling) and include them with your requests:
+
+```ts
+const changeUserName = ai.defineTool(
+  {
+    name: 'changeUserName',
+    description: 'can be used to change user name',
+    inputSchema: z.object({
+      newUserName: z.string(),
+    }),
+  },
+  async (input) => {
+    await ai.currentSession<MyState>().updateState({
+      userName: input.newUserName,
+    });
+    return 'changed username to ${input.newUserName}';
+  }
+);
+```
+
+```ts
+const chat = session.chat({
+  model: gemini15Pro,
+  tools: [changeUserName],
+});
+await chat.send('change user name to Kevin');
+```
+
+## Multi-thread sessions
+
+A single session can contain multiple chat threads. Each thread has its own
+message history, but they share a single session state.
+
+```ts
+const lawyerChat = session.chat('lawyerThread', {
+  system: 'talk like a lawyer',
+});
+const pirateChat = session.chat('pirateThread', {
+  system: 'talk like a pirate',
+});
+```
+
+## Session persistence (EXPERIMENTAL)
+
+When you initialize a new chat or session, it's configured by default to store
+the session in memory only. This is adequate when the session needs to persist
+only for the duration of a single invocation of your program, as in the sample
+chatbot from the beginning of this page. However, when integrating LLM chat into
+an application, you will usually deploy your content generation logic as
+stateless web API endpoints. For persistent chats to work under this setup, you
+will need to implement some kind of session storage that can persist state
+across invocations of your endpoints.
+
+To add persistence to a chat session, you need to implement Genkit's
+`SessionStore` interface. Here is an example implementation that saves session
+state to individual JSON files:
+
+```ts
+class JsonSessionStore<S = any> implements SessionStore<S> {
+  async get(sessionId: string): Promise<SessionData<S> | undefined> {
+    try {
+      const s = await readFile(`${sessionId}.json`, { encoding: 'utf8' });
+      const data = JSON.parse(s);
+      return data;
+    } catch {
+      return undefined;
+    }
+  }
+
+  async save(sessionId: string, sessionData: SessionData<S>): Promise<void> {
+    const s = JSON.stringify(sessionData);
+    await writeFile(`${sessionId}.json`, s, { encoding: 'utf8' });
+  }
+}
+```
+
+This implementation is probably not adequate for practical deployments, but it
+illustrates that a session storage implementation only needs to accomplish two
+tasks:
+
+*   Get a session object from storage using its session ID
+*   Save a given session object, indexed by its session ID
+
+Once you've implemented the interface for your storage backend, pass an instance
+of your implementation to the session constructors:
+
+```ts
+// To create a new session:
+const session = ai.createSession({
+  store: new JsonSessionStore(),
+});
+
+// Save session.id so you can restore the session the next time the
+// user makes a request.
+```
+
+```ts
+// If the user has a session ID saved, load the session instead of creating
+// a new one:
+const session = await ai.loadSession(sessionId, {
+    store: new JsonSessionStore(),
+});
+```