[Bug] Can't inference with stream=false 0.0.8

[hatrexltd]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.

Describe the bug

Traceback (most recent call last):
File "/root/miniconda3/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 408, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in call
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/lib/python3.11/site-packages/fastapi/applications.py", line 276, in call
await super().call(scope, receive, send)
File "/root/miniconda3/lib/python3.11/site-packages/starlette/applications.py", line 122, in call
await self.middleware_stack(scope, receive, send)
File "/root/miniconda3/lib/python3.11/site-packages/starlette/middleware/errors.py", line 184, in call
raise exc
File "/root/miniconda3/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in call
await self.app(scope, receive, _send)
File "/root/miniconda3/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in call
raise exc
File "/root/miniconda3/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in call
await self.app(scope, receive, sender)
File "/root/miniconda3/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in call
raise e
File "/root/miniconda3/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in call
await self.app(scope, receive, send)
File "/root/miniconda3/lib/python3.11/site-packages/starlette/routing.py", line 718, in call
await route.handle(scope, receive, send)
File "/root/miniconda3/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
await self.app(scope, receive, send)
File "/root/miniconda3/lib/python3.11/site-packages/starlette/routing.py", line 69, in app
await response(scope, receive, send)
File "/root/miniconda3/lib/python3.11/site-packages/starlette/responses.py", line 280, in call
await self.background()
File "/root/miniconda3/lib/python3.11/site-packages/starlette/background.py", line 43, in call
await task()
File "/root/miniconda3/lib/python3.11/site-packages/starlette/background.py", line 26, in call
await self.func(*self.args, **self.kwargs)
File "/root/miniconda3/lib/python3.11/site-packages/lmdeploy/serve/openai/api_server.py", line 130, in abort_request
async for _ in VariableInterface.async_engine.generate_openai(
File "/root/miniconda3/lib/python3.11/site-packages/lmdeploy/serve/async_engine.py", line 216, in generate_openai
async for outputs in generator.async_stream_infer(
File "/root/miniconda3/lib/python3.11/site-packages/lmdeploy/turbomind/turbomind.py", line 213, in async_stream_infer
for output in self.stream_infer(*args, **kwargs):
File "/root/miniconda3/lib/python3.11/site-packages/lmdeploy/turbomind/turbomind.py", line 347, in stream_infer
if output[-1].item() == self.eos_id:
~~~~~~^^^^
IndexError: index -1 is out of bounds for dimension 0 with size 0

Reproduction

{
"model": "llama2",
"messages": [
{"role": "user", "content": "Tell me about France"}
],
"max_tokens": 372,
"temperature": 0.7,
"top_p": 1,
"n": 1,
"stream": false
}

Error traceback

No response

Request with stream=true works

[lvhan028]

#378 brought side effects.

[wangruohui]

Hi @hatrexltd

Can you try change File "/root/miniconda3/lib/python3.11/site-packages/lmdeploy/turbomind/turbomind.py", line 347, in stream_infer as below, and see if it solve your problem?

if len(output)>0 and output[-1].item() == self.eos_id:

[hatrexltd]

Hi @hatrexltd

Can you try change File "/root/miniconda3/lib/python3.11/site-packages/lmdeploy/turbomind/turbomind.py", line 347, in stream_infer as below, and see if it solve your problem?

if len(output)>0 and output[-1].item() == self.eos_id:

Works. Thanks.

[wangruohui]

To be fixed by #405