[Bug] Unable to run chatQnA successfully

[KMC-intel]

Priority

P2-High

OS type

Ubuntu

Hardware type

Xeon-GNR

Installation method

• Pull docker images from hub.docker.com
• Build docker images from source

Deploy method

• Docker compose
• Docker
• Kubernetes
• Helm

Running nodes

Single Node

What's the version?

f8b4269bdbc303c0e8e0fa8210bc7e874da7d43be62cb6d79a8c7671fa4a7955

Description

unable to start inferencing through the megaservice. I was able to run a query on its own using curl to the service directly. So there just seems to be an issue with the megaservice.

Reproduce steps

Clone chatQnA repo
Switch to xeon dir
export host_ip=<>, export no_proxy=<>
source set_env.sh
docker compose up -d
Send chat query using UI. Receive "Internal Server Error"

Raw log

docker logs chatqna-xeon-backend-server
[2024-12-12 03:09:54,645] [    INFO] - Base service - CORS is enabled.
[2024-12-12 03:09:54,646] [    INFO] - Base service - Setting up HTTP server
[2024-12-12 03:09:54,647] [    INFO] - Base service - Uvicorn server setup on port 8888
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8888 (Press CTRL+C to quit)
[2024-12-12 03:09:54,657] [    INFO] - Base service - HTTP server setup successful
/home/user/chatqna.py:334: PydanticDeprecatedSince20: The `parse_obj` method is deprecated; use `model_validate` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.10/migration/
  chat_request = ChatCompletionRequest.parse_obj(data)
INFO:     172.18.0.10:40486 - "POST /v1/chatqna HTTP/1.0" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.11/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 174, in __call__
    raise exc
  File "/usr/local/lib/python3.11/site-packages/prometheus_fastapi_instrumentator/middleware.py", line 172, in __call__
    await self.app(scope, receive, send_wrapper)
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 93, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 144, in simple_response
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/chatqna.py", line 358, in handle_request
    result_dict, runtime_graph = await self.megaservice.schedule(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/GenAIComps/comps/cores/mega/orchestrator.py", line 106, in schedule
    response, node = await done_task
                     ^^^^^^^^^^^^^^^
  File "/home/user/GenAIComps/comps/cores/mega/orchestrator.py", line 267, in execute
    data = await response.json()
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1277, in json
    raise ContentTypeError(
aiohttp.client_exceptions.ContentTypeError: 504, message='Attempt to decode JSON with unexpected mimetype: text/html', url='http://tei-embedding-service/embed'

[ZailiWang]

Hi , what do you mean by "version ID" f8b4269bdbc303c0e8e0fa8210bc7e874da7d43be62cb6d79a8c7671fa4a7955?
I searched GenAIExamples commits but did not find it.

[daniel-de-leon-user293]

I'm also getting this same error via CLI on Gaudi following the Quick Start steps in the README.

[KMC-intel]

f8b4269bdbc303c0e8e0fa8210bc7e874da7d43be62cb6d79a8c7671fa4a7955
https://hub.docker.com/layers/opea/chatqna/latest/images/sha256-f8b4269bdbc303c0e8e0fa8210bc7e874da7d43be62cb6d79a8c7671fa4a7955

This was version of the opea/chatqna i pulled from docker hub

[ZailiWang]

Hi @KMC-intel @daniel-de-leon-user293, I need a few days since no test server at hand now.
Would you help try if this fix works:
Change this line specified in the error traceback:

  File "/home/user/GenAIComps/comps/cores/mega/orchestrator.py", line 267, in execute
    data = await response.json()
           ^^^^^^^^^^^^^^^^^^^^^

change

data = await response.json()

to

data = await response.json(content_type=None)

Please try and tell if this change could fix the problem. If it works I'll draft a PR including this fix. Thanks!