streaming api produce repeated character

[zqWu]

1. description:
   [1] fastchat.serve.vllm_worker
   [2] stream = true
   the fastchat.serve.openai_api_server may produce repeat word output

2. how to re-preduce
   [1] setup, use vicuna_13b_v1.5

python -m fastchat.serve.controller --host 0.0.0.0 --port 21001

python -m fastchat.serve.vllm_worker --host 0.0.0.0 --model-path /mnt/data/vicuna/13b_v1.5 --gpu-memory-utilization 0.45 --model-name vicuna-13b-v1.5

python -m fastchat.serve.openai_api_server --host 0.0.0.0

[2] trigger

curl \
-H "Content-Type: application/json" \
--data-binary "{\"messages\": [{\"role\": \"user\", \"content\": \"\n    As a carbon emissions expert,  Use the following pieces of context to summarize and answer the user's question. If you are unsure, state that you are unsure.\n    Please be specific and descriptive in your response, including any relevant data or statistics.\n    The output should be in a clear and concise format, with a focus on providing actionable information.\n    Example:\n    Material Question: What is the carbon footprint of our manufacturing process?\n    Response: Our manufacturing process has a carbon footprint of X, which is below the industry average. We have implemented several initiatives to reduce our carbon emissions, including...\n    User Question: How can I reduce my carbon footprint at home?\n    Response: There are several ways to reduce your carbon footprint at home, such as using energy-efficient appliances, reducing water usage, and using public transportation or carpooling. Here are some specific tips for each category...\n\nQuestion: 中国碳达峰\n    Answer and explain in Chinese:\"}], \"model\": \"vicuna-13b-v1.5\", \"max_tokens\": null, \"stream\": true, \"n\": 1, \"temperature\": 0.0, \"top_p\": 1.0}" \
"http://localhost:8000/v1/chat/completions"

[3] part of the output from 2[2], you can see some repeated chinese character, which's wrong

data: {"id": "chatcmpl-3Xf9eXBQoM5pk3eSAUR2DW", "model": "vicuna-13b-v1.5", "choices": [{"index": 0, "delta": {"content": "而"}, "finish_reason": null}]}

data: {"id": "chatcmpl-3Xf9eXBQoM5pk3eSAUR2DW", "model": "vicuna-13b-v1.5", "choices": [{"index": 0, "delta": {"content": "且"}, "finish_reason": null}]}

data: {"id": "chatcmpl-3Xf9eXBQoM5pk3eSAUR2DW", "model": "vicuna-13b-v1.5", "choices": [{"index": 0, "delta": {"content": "且随"}, "finish_reason": null}]}

data: {"id": "chatcmpl-3Xf9eXBQoM5pk3eSAUR2DW", "model": "vicuna-13b-v1.5", "choices": [{"index": 0, "delta": {"content": "且随着"}, "finish_reason": null}]}

# 而
# 而且 => should be 且
# 且随着 => should be 随着

[4] reason

• file openai_api_server.py, line 437, add a debug log

• some of the output

2, delta_text=None, previous_text=中国的, decode_unicode=中国的

2, delta_text=碳, previous_text=中国的碳, decode_unicode=中国的碳
2, delta_text=None, previous_text=中国的, decode_unicode=中国的  <===== notice this previous_text, it omit a "碳"

2, delta_text=None, previous_text=中国的, decode_unicode=中国的
2, delta_text=碳排, previous_text=中国的碳排, decode_unicode=中国的碳排

• it seems that when delta_text=Noone, previous_text has been set incorrectly

• suggest solution

[merrymercy]

Fixed by #2431?

[zqWu]

looks the same problem. since you fixed, i'll retry it later