Adding API endpoint to the TextEncoder?

[Vincent-Maladiere]

Problem Description

A lot of people are using embedding models behind API, like OpenAI. I think enabling this through the TextEncoder makes sense.

Advantages

• Access to more powerful and diverse embedding models
• No need to install pytorch, download a heavy model and run it locally
• This results in a faster fit, notwithstanding the IO overhead.
• We already have a token mechanism to stay consistent with HuggingFace Hub, so we don't need extra engineering for this.

Limitations

• Having a token becomes mandatory
• Storing embedding results is important to be cost-efficient. This could trigger us to think about caching results with e.g. joblib, even for the current version of the TextEncoder.

WDYT?

Feature Description

We could detect that the passed path is an API URL –or introduce a new parameter, but I guess checking for an API existence is fairly straightforward.

Alternative Solutions

No response

Additional Context

No response