[Feature]: Elasticsearch Inference API support

[stevapple]

The Feature

Let’s support Elasticsearch as an LLM provider. This includes utilizing Elasticsearch Inference APIs to perform embedding, reranking and chat completion requests.

Motivation, pitch

Elasticsearch has provided a set of Inference APIs for performing inferences with self-hosted or external models. This includes:

• Embedding models, both dense and sparse vectors are supported;
• Completion models, with streaming support;
• Reranking models.

For LiteLLM, it would be great to support using Elasticsearch as an LLM backend. From some perspective, Elasticsearch serves as an LLM proxy gateway in this scenario.

Alternatively, Elasticsearch has provided a set of Trained Models APIs which is dedicated to self-hosted models and not restricted to LLM scenario. While it should provide extra capabilities like NER, classification and mask-filling, these are mostly not related to LiteLLM, so we can focus on the new Inference APIs.

Twitter / LinkedIn details

No response