Also support embedding APIs?

[AbrJA]

Hi there, I hope you are doing well

Thank you for this package you are working on, it's great!

Do you have on mind the implementation for interacting with embedding models? Or maybe you have already implemented this but I didn't find it,

Greetings,

Abraham JA

[hadley]

I think that's currently out of scope for elmer, but we might end up implementing in a separate package. What do you need it for?

[cboettig]

I would love to see this as well! Embedding models are very helpful for part of RAG, allowing users to query text-based documents (say, a shiny app that only answers questions based on the elmer documentation, and provides precise citations back to the URLs of doc pages from where it is drawing the answers). nice walkthrough in https://python.langchain.com/docs/tutorials/rag/.

embedding models can of course be used alone but are often implemented in concert with a chat (completions) model.

[JamesHWade]

For RAG applications, the approach elmer uses for tools might be best. chat$register_context() or maybe chat$register_rag().

I've tried to put RAG into a package (gpttools). From that, I can appreciate the care you'd need to get the abstractions right, and I certainly haven't figured those out yet. A dedicated package makes sense. Too bad tidymodels already had the pkg name embed. Speaking of, I bet the tidymodels crew would have great ideas for how to approach it.

[jpmarindiaz]

Embeddings are very useful for RAG applications.
You can call local embeddings in ollama like this:

curl http://localhost:11434/api/embeddings -d '{
  "model": "mxbai-embed-large",
  "prompt": "Llamas are members of the camelid family"
}'

See https://ollama.com/blog/embedding-models

I have also used this for openai:

openai::create_embedding(model = "text-embedding-3-small",input = txt)

I think it would make a lot of sense keep the same interface in elmer:

chat <- chat_ollama(model = "nomic-embed-text")
chat$chat("hello nomic embedding") or chat$embedding("hello nomic embedding")

chat <- chat_openai(model = "text-embedding-3-small")
chat$chat("hello openai embedding") or chat$embedding("hello openai embedding")

[JBGruber]

If you're interested, rollama already supports this (R package with a focus on Ollama): https://jbgruber.github.io/rollama/articles/text-embedding.html

[hadley]

@JBGruber did you consider returning all the embeddings in a single matrix-column? I would suspect that might be faster to work with.

[hadley]

One reason to do this in elmer is that 90% of HTTP request (e.g. error handling, rate-limiting, ...) will be the same. Implementing it in another package would require either extracting a lot of existing code or duplicating it (neither of which is particularly appealing given how young elmer is). The main reason I'm not sure about implementing in it elmer is that it would require a new API since jamming it into the chat object would be rather forced. The infrastructure is already there (i.e. this would become a new S7 method on the provider generic), but would require exporting a bunch more stuff that's currently internal (making it hard to change in the future). OTOH #202 and #167 already suggest that we need to export this stuff.