[Frontend] Support Tool and RAG

[esmeetu]

This is another function tool implement compared to #3237. This PR is simple and flexible. And this idea was inspired by https://github.com/huggingface/transformers/blob/main/src/transformers/models/cohere/tokenization_cohere_fast.py
That leveraged transformers rich chat template support.

Discussions

1. Should tools parameters follow openai' design strictly, which have a fixed json format? although this server is named openai_server.
2. tool_calls in openai response sometimes is unnecessary. For example, if prompt: 'Send email to Roy', after model returns tool call, my task finished. And i don't need tool_calls for summary anymore.
3. For documents parameter, whether we will use this information for future features? like RAG Speculative decoding and something else that accelerates RAG inference.
4. Sometimes user using RAG api endpoint as chat, which will lost some precisions. They assemble conversations and documents without using tokenizer's special tokens, which doesn't follow model's training prompt.

TODO

• Test and example
• Document for how to use

Further ideas

1. I think we need [Server] Support openai prefix cache #2515 when using tool call feature. Since tool call list is always fixed, users can get max performance by setting prefix position instead of unstable automatic prefix caching.
   For RAG applications, it's also useful to have a indicator(prefix_stop) to adapt prefix caching.
2. As we have more custom parameters in api_server. I think it's time to consider a vllm-client. This might be not related to this PR.

[leiwen83]

Hi,

I wonder which version of openai library support this "documents" parameter?
If only for spec infer reason, maybe we could directly reuse "tools", and make its type as "rag_spec_infer"?

And we may need add additional code to tokenizer those rag text?

[esmeetu]

@leiwen83 Thanks for your feedback.

I wonder which version of openai library support this "documents" parameter?

No, it just a custom parameter in vllm, as we have already a few custom parameter in openai_server.

If only for spec infer reason, maybe we could directly reuse "tools", and make its type as "rag_spec_infer"?

It seems that reusing tools in rag will mislead users to use this api.

And we may need add additional code to tokenizer those rag text?

We leverage the transformers chat template and apply_chat_template function, and we don't need to write custom parser to handle user's RAG documents.

[bohea]

Hi esmeetu
idea of rich chat template support is great

maybe offer a cohere client compatible api server, cohere_server, is a better choice？

[esmeetu]

@bohea Thanks! I think it's a bit early to consider adding another api_server. As we know, Llama3 model's tool use and rag performance is also good.