feat: APIServer supports embeddings

[fangyinc]

Description

API Server adds embedding API compatible with OpenAI embeddings

Others:

• Support load documents with multi-threadeds
• New OpenAPIEmbeddings

How Has This Been Tested?

Test embeddings API

1. Start DB-GPT's API Server according OpenAI SDK Calls Local Multi-model
   

2. Test with cURL

curl http://127.0.0.1:8100/api/v1/embeddings \
-H "Authorization: Bearer EMPTY" \
-H "Content-Type: application/json" \
-d '{
    "model": "text2vec",
    "input": "Hello world!"
}'

Test OpenAPIEmbeddings

  from dbgpt.rag.embedding import OpenAPIEmbeddings

  openai_embeddings = OpenAPIEmbeddings(
      api_url="http://localhost:8100/api/v1/embeddings",
      api_key="EMPTY",
      model_name="text2vec",
  )
  texts = ["Hello, world!", "How are you?"]
  openai_embeddings.embed_documents(texts)

Snapshots:

Include snapshots for easier review.

Checklist:

• My code follows the style guidelines of this project
• I have already rebased the commits and make the commit message conform to the project standard.
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• Any dependent changes have been merged and published in downstream modules

[Aries-ckt]

Test embeddings API

How to usage?

# 1. at first start controller 
dbgpt start controller

# 2. second start embedding model worker
dbgpt start worker --model_name text2vec \
--model_path /app/models/text2vec-large-chinese \
--worker_type text2vec \
--port 8003 \
--controller_addr http://127.0.0.1:8000

# 3. start apiserver 
dbgpt start apiserver --controller_addr http://127.0.0.1:8000 --api_keys EMPTY

# 4. test
curl http://127.0.0.1:8100/api/v1/embeddings \
-H "Authorization: Bearer EMPTY" \
-H "Content-Type: application/json" \
-d '{
    "model": "text2vec",
    "input": "Hello world!"
}'

[csunny]

r+, test good

[Aries-ckt]

LGTM!