[FEAT] Add an ability to use OpenAI / VoyageAI / Cohere embeddings with SDPMChunker via LiteLLM

[rodion-m]

We can make something line EmbeddingsProvider abstraction with many implementations like SentenceTransformerEmbeddingsProvider, LiteLlmEmbeddingsProvider, FastEmbedEmbeddingsProvider and so on.

[rodion-m]

@bhavnicksm could you please try to add this project into Copilot Workspace and try to implement this issue with it? I believe that it should solve it.

[bhavnicksm]

Hey @rodion-m !

Thanks for opening an issue 😊

Yes! That's the plan, we have a BaseEmbeddings class, that we built to make initial embeddings support for Chonkie 🙌

Some things remaining to plan out is the integration and refactor for Chunkers to use BaseEmbeddings and default Embeddings initialisation scheme...

Planning to continue support for SentenceTransformers as the default but also make it so that a str input inside chunkers could be supported for the users to avoid having too bulky inits on the API side...

Happy to accept comments and PRs for this!

Thanks

[rodion-m]

Yeah, thanks a lot for your work and for a great library. It's great that it's universal and not linked to LangChain / LlamaIndex.

[bhavnicksm]

Yeah, thanks a lot for your work and for a great library. It's great that it's universal and not linked to LangChain / LlamaIndex.

Thank you so much! Appreciate the kind words :)

[bhavnicksm]

@bhavnicksm could you please try to add this project into Copilot Workspace and try to implement this issue with it? I believe that it should solve it.

Yeah, I'll try with Copilot workspace... Till now, I've had a slightly negative experience where it would introduce some bugs that I would need to fix later on. So I'm a bit careful with it, taking only suggestions from copilot and not full PRs.