Investigate chunking strategies

[orpiske]

We need to investigate chunking strategies that can help the assistant provide better answers:

• Investigate whether chunking libraries already exist
• Investigate whether chunking features are available on some of the libraries/frameworks that we use
• Create a Java library/project that implements typical strategies for chunking
• Use chunking features from a different language (i.e.: Python) and externalize chunking to a different component.

[lburgazzoli]

IMHO, this work should end up being part of lanchain4j and we can eventually use is as one of the tokenize strategy in Apache Camel

[oscerd]

I don't think it's something that should go in Camel. Camel is an integration framework, tokenizing is a feature related to something else.

[orpiske]

IMHO, this work should end up being part of lanchain4j and we can eventually use is as one of the tokenize strategy in Apache Camel

Yeah.

I also don't see it as being part of camel, as rightly pointed by @oscerd. It could be used by it, though.

So, I think a reasonable approach would be to create a Java library and then work to include support for it on langchain4j and Quarkus.

[lburgazzoli]

I would then move this discussion to the langchain4j issue tacker so they may provide some additional info/suggestion as they may have had the chance to think about it already

[orpiske]

For reference, here's a discussion with the Langchain4j project. Their suggestion is to look at the DocumentSplitter interface and work on top of that.

langchain4j/langchain4j#1081