[FEAT] Support Cohere Embeddings for SemanticChunker and SDPMChunker #118

[Udayk02]

• added every model support for Cohere embeddings.
• added the same to the registry to support AutoEmbeddings.
• added tests for the same.

key things to notice:

1. some models do not have tokenizers available. used the default "embed-english-light-v3.0" for those. (please review and conclude)
2. there is a limit of 96 on the number of documents in a single API call.
3. the model context length mentioned in the Cohere documentation is 512. but, they did not mention anywhere that it will be truncated in the case of larger texts, rather mentioned that it is not optimal. reflected the same in the code.
4. used tokenizers for the complete tokenizing via Cohere official urls as I faced an issue while loading the same through hf via autotiktoken.
5. there isn't a way to check whether the client is initialized with api_key or not. check_api_key is a deprecated function. along with that, there isn't a retry functionality provided by Cohere. handled it in chonkie end.

[bhavnicksm]

Hey @Udayk02!

Thanks for opening a PR 🚀

I am curious about point 4, you mentioned,

used tokenizers for the complete tokenizing via Cohere official urls as I faced an issue while loading the same through hf via autotiktoken.

Could you describe the issue you faced with autotiktokenizer?

Thanks ☺️

[Udayk02]

As I delved through autotiktokenizer, I found the bug. It is a very minor bug. vocab is always assumed to be a dict. But, in the cohere tokenizer.json, it is presented as a list of lists. I raised an issue there. Please check!

[bhavnicksm]

Hey @Udayk02,
Could you check the docstring once? You've mentioned organization here, but the signature says client_name.
Thanks!

[Udayk02]

That is a mistake as I was directly referencing to your implementation for OpenAI Embeddings. Will fix and rebase the branch. Thanks!!

[bhavnicksm]

Hey @Udayk02!

Brilliant work on this PR 🚀 Just a really minor point but is perfect otherwise~

Thanks! 😊

[Udayk02]

Hi @bhavnicksm , thank you! 😊

Please state whether I have to change the tokenizer from tokenizers to autotiktokenizer.
After the fix in autotiktokenizer, it will work for every model except the version 2 models. Keep in mind that the v2.0 models neither have a tokenizer in the cohere official models list nor in huggingface. Only embed-multilingual-v2.0 is listed in the Cohere's end.

Thanks!

[bhavnicksm]

Hey @Udayk02!

Please state whether I have to change the tokenizer from tokenizers to autotiktokenizer.

Yes, it's preferable to keep autotiktokenizer here~ I'll have a look at the PR you've raised and merge that in soon! We can update this PR once that's been merged.

Thanks! 😊