Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update chat memory example and separate ChatMemoryStore from ChatMemory usage #107

Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

This file was deleted.

Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
package io.quarkiverse.langchain4j.samples;

import jakarta.enterprise.context.ApplicationScoped;
import jakarta.inject.Inject;

import dev.langchain4j.memory.ChatMemory;
import dev.langchain4j.memory.chat.ChatMemoryProvider;
import dev.langchain4j.memory.chat.MessageWindowChatMemory;
import dev.langchain4j.store.memory.chat.ChatMemoryStore;

@ApplicationScoped
public class ChatMemoryProviderBean implements ChatMemoryProvider {

@Inject
ChatMemoryStore store;

@Override
public ChatMemory get(Object memoryId) {
return MessageWindowChatMemory.builder()
.id(memoryId)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

without eviction or4 cleanup, this will lead to OOM.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to see exactly how this is meant to work.

Copy link
Contributor Author

@andreas-eberle andreas-eberle Dec 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, there is currently no eviction mechanism. However, for a simple example, that would be hard to do. The current idea of always evicting when the request ends makes a chat with API calls impossible.
I added a note to the documentation clarifying this.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The documentation should mention or have a more complex example with eviction.
I'm still unsure about the request scope part. I think, we would need to introduce a "conversation" concept soon.

.maxMessages(20)
.chatMemoryStore(store)
.build();
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
package io.quarkiverse.langchain4j.samples;

import jakarta.enterprise.context.ApplicationScoped;

import dev.langchain4j.store.memory.chat.ChatMemoryStore;
import dev.langchain4j.store.memory.chat.InMemoryChatMemoryStore;

public class ChatMemoryStoreProducer {

@ApplicationScoped
ChatMemoryStore produceChatMemoryStore() {
return new InMemoryChatMemoryStore();
}
}
Original file line number Diff line number Diff line change
@@ -1,32 +1,24 @@
package io.quarkiverse.langchain4j.samples;

import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.function.Supplier;

import dev.langchain4j.memory.ChatMemory;
import jakarta.inject.Inject;

import dev.langchain4j.memory.chat.ChatMemoryProvider;
import dev.langchain4j.memory.chat.MessageWindowChatMemory;
import io.quarkiverse.langchain4j.RemovableChatMemoryProvider;
import dev.langchain4j.store.memory.chat.ChatMemoryStore;

public class MySmallMemoryProvider implements Supplier<ChatMemoryProvider> {
@Override
public ChatMemoryProvider get() {
return new RemovableChatMemoryProvider() {
private final Map<Object, ChatMemory> memories = new ConcurrentHashMap<>();

@Override
public ChatMemory get(Object memoryId) {
return memories.computeIfAbsent(memoryId, id -> MessageWindowChatMemory.builder()
.maxMessages(20)
.id(memoryId)
.build());
}
@Inject
ChatMemoryStore store;

@Override
public void remove(Object id) {
memories.remove(id);
}
};
@Override
public ChatMemoryProvider get() {
return memoryId -> MessageWindowChatMemory.builder()
.id(memoryId)
.maxMessages(20)
.chatMemoryStore(store)
.build();
}
}
12 changes: 10 additions & 2 deletions docs/modules/ROOT/pages/ai-services.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -181,10 +181,18 @@ An example of such a bean is:

[source,java]
----
include::{examples-dir}/io/quarkiverse/langchain4j/samples/ChatMemoryBean.java[]
include::{examples-dir}/io/quarkiverse/langchain4j/samples/ChatMemoryProviderBean.java[]
----

Notice that the messages are deleted when the scope terminates (as it will call the `close` method).
Additionally, it is important to configure a `ChatMemoryStore` which is responsible for actually storing the chat memory.
The following example shows how an application wide `InMemoryChatMemoryStore` can be used for this.

[source,java]
----
include::{examples-dir}/io/quarkiverse/langchain4j/samples/ChatMemoryStoreProducer.java[]
----

Notice that with this example, conversations are not deleted and thus accumulate over time.

NOTE: It is recommended to have your chat memory beans implement `RemovableChatMemoryProvider` because the objects used as memory IDs are removed from the memory when the service goes out of scope.

Expand Down
Loading