Connecting to VLLM Instance #4602

ksingh-scogo · 2024-07-08T12:04:10Z

ksingh-scogo
Jul 8, 2024

What happened?

Trying to access LLMLite endpoint to route chat/completion requests to models hosted by VLLM all in the same kubernetes cluster.

User --> (Curl) --> LiteLLM --> (Vllm phi-3 model)

Getting error litellm.NotFoundError: NotFoundError: OpenAIException - Error code: 404

Litellm configmap

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: litellm-config
  namespace: litellm
data:
  proxy_config.yaml: |
    model_list:
      - model_name: azure-gpt-4
        litellm_params:
          model: azure/gpt-4
          api_base: "https://aoi-xxxxxx.openai.azure.com"
          api_key: "xxxxxxxxxxx"
          api_version: "2023-03-15-preview"
      - model_name: microsoft/Phi-3-mini-4k-instruct
        litellm_params:
          model: "openai/microsoft/Phi-3-mini-4k-instruct"
          api_base: "http://vllm-service.vllm.svc.cluster.local"
          api_key: "sk-211111"
          stream: True
          rpm: 20
          timeout: 90
          stream_timeout: 50
    litellm_settings:
      drop_params: True
      num_retries: 3
      request_timeout:
      allowed_fails: 3
        #  master_key: test
      cache: False
    general_settings:
      master_key: sk-211111

LiteLLM deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: litellm-deployment
  namespace: litellm
spec:
  replicas: 1
  selector:
    matchLabels:
      app: litellm
  template:
    metadata:
      labels:
        app: litellm
    spec:
      containers:
        - name: litellm-containerå
          image: ghcr.io/berriai/litellm:main-latest
          imagePullPolicy: Always
          args:
            - "--config"
            - "/app/proxy_config.yaml"  
          volumeMounts:       
            - name: config-volume
              mountPath: /app
              readOnly: true
          livenessProbe:
            httpGet:
              path: /health/liveliness
              port: 4000
            initialDelaySeconds: 120
            periodSeconds: 15
            successThreshold: 1
            failureThreshold: 3
            timeoutSeconds: 10
          readinessProbe:
            httpGet:
              path: /health/readiness
              port: 4000
            initialDelaySeconds: 120
            periodSeconds: 15
            successThreshold: 1
            failureThreshold: 3
            timeoutSeconds: 10
      volumes: 
        - name: config-volume
          configMap:
            name: litellm-config  
---
apiVersion: v1
kind: Service
metadata:
  name: litellm-service
  namespace: litellm
spec:
  selector:
    app: litellm
  ports:
    - protocol: TCP
      port: 4000
      targetPort: 4000
  type: LoadBalancer

Error Screenshot
Direct curl to VLLM endpoint which is hosting the OpenAI compatible phi-3 model works fine

litellm.NotFoundError: NotFoundError: OpenAIException - Error code: 404

Relevant log output

File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 645, in acompletion
    response = await self.async_function_with_fallbacks(**kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 2329, in async_function_with_fallbacks
    raise original_exception
  File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 2174, in async_function_with_fallbacks
    response = await self.async_function_with_retries(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 2429, in async_function_with_retries
    raise original_exception
  File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 2351, in async_function_with_retries
    response = await original_function(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 787, in _acompletion
    raise e
  File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 756, in _acompletion
    response = await _response
               ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/litellm/utils.py", line 1500, in wrapper_async
    raise e
  File "/usr/local/lib/python3.11/site-packages/litellm/utils.py", line 1315, in wrapper_async
    result = await original_function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/litellm/main.py", line 410, in acompletion
    raise exception_type(
          ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/litellm/utils.py", line 7665, in exception_type
    raise e
  File "/usr/local/lib/python3.11/site-packages/litellm/utils.py", line 6036, in exception_type
    raise NotFoundError(
litellm.exceptions.NotFoundError: litellm.NotFoundError: NotFoundError: OpenAIException - Error code: 404 - {'detail': 'Not Found'} LiteLLM Retried: 2 times, LiteLLM Max Retries: 3

10:48:16 - LiteLLM Proxy:ERROR: _common.py:120 - Giving up chat_completion(...) after 1 tries (litellm.proxy._types.ProxyException)
INFO:     127.0.0.1:40330 - "POST /chat/completions HTTP/1.1" 404 Not Found
INFO:     10.244.17.1:55474 - "GET /health/liveliness HTTP/1.1" 200 OK



### Twitter / LinkedIn details

_No response_

Answered by krrishdholakia

Jul 8, 2024

can you modify your api_base to be

http://vllm-service.vllm.svc.cluster.local/v1

and let me know if that works @ksingh-scogo

View full answer

ksingh-scogo · 2024-07-08T12:05:59Z

ksingh-scogo
Jul 8, 2024
Author

Can anyone guide me what am i missing ?

Is the following right way to add vllm hosted models which are

      - model_name: microsoft/Phi-3-mini-4k-instruct
        litellm_params:
          model: "openai/microsoft/Phi-3-mini-4k-instruct"
          api_base: "http://vllm-service.vllm.svc.cluster.local"
          api_key: "sk-211111"
          stream: True
          rpm: 20
          timeout: 90
          stream_timeout: 50

curl --location 'http://127.0.0.1:4000/chat/completions' \
    --header 'Authorization: Bearer sk-219191' \
    --header 'Content-Type: application/json' \
    --data '{
    "model": "microsoft/Phi-3-mini-4k-instruct",
    "messages": [
        {
        "role": "user",
        "content": "what llm are you"
        }
    ],
}'

0 replies

ksingh-scogo · 2024-07-08T12:06:51Z

ksingh-scogo
Jul 8, 2024
Author

FWIW here is the kubernets config file of Vllm model deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-deployment
  namespace: vllm
spec:
  replicas: 1  # Specify the number of replicas
  selector:
    matchLabels:
      app: vllm
  template:
    metadata:
      labels:
        app: vllm
    spec:
      containers:
      - args:
        - --model
        - microsoft/Phi-3-mini-4k-instruct
        - --gpu-memory-utilization
        - "0.95"
        - --enforce-eager
        - --dtype=half
        env:
        - name: HUGGING_FACE_HUB_TOKEN
          valueFrom:
            secretKeyRef:
              key: HUGGINGFACE_TOKEN
              name: huggingface-token
        image: vllm/vllm-openai:latest
        name: vllm
        ports:
        - containerPort: 8000
          protocol: TCP
        resources:
          limits:
            nvidia.com/gpu: "1"
      nodeSelector:
        accelerator: nvidia
      tolerations:
      - key: "sku"
        operator: "Equal"
        value: "gpu"
        effect: "NoSchedule"
      - key: "kubernetes.azure.com/scalesetpriority"
        operator: "Equal"
        value: "spot"
        effect: "NoSchedule"
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-service
  namespace: vllm
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8000
  selector:
    app: vllm
  sessionAffinity: None
  type: ClusterIP

0 replies

krrishdholakia · 2024-07-08T16:17:34Z

krrishdholakia
Jul 8, 2024
Maintainer

the model looks like it was added correctly

can you share the complete stacktrace.

It looks like the same issue you had with ollama where the containers aren't able to speak with each other.

Moving to discussion, as this doesn't look like a litellm error.

5 replies

ksingh-scogo Jul 8, 2024
Author

Glad that you remember my ollam issue @krrishdholakia i made peace with that error and moved on to execute our next plan of using VLLM on GPU k8s nodes and now i am here.

Full stacktrace output

INFO:     10.244.17.1:50150 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.244.17.1:50152 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.244.17.1:55546 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.244.17.1:55548 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.244.17.1:33186 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.244.17.1:33170 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.244.17.1:53982 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.244.17.1:53984 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.244.17.1:53860 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.244.17.1:53862 - "GET /health/readiness HTTP/1.1" 200 OK
INFO:     10.244.17.1:41694 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.244.17.1:41692 - "GET /health/readiness HTTP/1.1" 200 OK
�[92m16:25:25 - LiteLLM:ERROR�[0m: main.py:403 - litellm.acompletion(): Exception occured - Error code: 404 - {'detail': 'Not Found'}
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/litellm/main.py", line 382, in acompletion
    response = await init_response
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/litellm/llms/openai.py", line 920, in acompletion
    raise e
  File "/usr/local/lib/python3.11/site-packages/litellm/llms/openai.py", line 903, in acompletion
    headers, response = await self.make_openai_chat_completion_request(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/litellm/llms/openai.py", line 684, in make_openai_chat_completion_request
    raise e
  File "/usr/local/lib/python3.11/site-packages/litellm/llms/openai.py", line 679, in make_openai_chat_completion_request
    response = await openai_aclient.chat.completions.create(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/openai/resources/chat/completions.py", line 1181, in create
    return await self._post(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/openai/_base_client.py", line 1790, in post
    return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/openai/_base_client.py", line 1493, in request
    return await self._request(
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/openai/_base_client.py", line 1584, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.NotFoundError: Error code: 404 - {'detail': 'Not Found'}

�[92m16:25:29 - LiteLLM Proxy:ERROR�[0m: proxy_server.py:2945 - litellm.proxy.proxy_server.chat_completion(): Exception occured - litellm.NotFoundError: NotFoundError: OpenAIException - Error code: 404 - {'detail': 'Not Found'} LiteLLM Retried: 2 times, LiteLLM Max Retries: 3
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/litellm/main.py", line 382, in acompletion
    response = await init_response
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/litellm/llms/openai.py", line 920, in acompletion
    raise e
  File "/usr/local/lib/python3.11/site-packages/litellm/llms/openai.py", line 903, in acompletion
    headers, response = await self.make_openai_chat_completion_request(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/litellm/llms/openai.py", line 684, in make_openai_chat_completion_request
    raise e
  File "/usr/local/lib/python3.11/site-packages/litellm/llms/openai.py", line 679, in make_openai_chat_completion_request
    response = await openai_aclient.chat.completions.create(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/openai/resources/chat/completions.py", line 1181, in create
    return await self._post(
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/openai/_base_client.py", line 1790, in post
    return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/openai/_base_client.py", line 1493, in request
    return await self._request(
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/openai/_base_client.py", line 1584, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.NotFoundError: Error code: 404 - {'detail': 'Not Found'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/litellm/proxy/proxy_server.py", line 2843, in chat_completion
    responses = await llm_responses
                ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 657, in acompletion
    raise e
  File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 645, in acompletion
    response = await self.async_function_with_fallbacks(**kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 2329, in async_function_with_fallbacks
    raise original_exception
  File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 2174, in async_function_with_fallbacks
    response = await self.async_function_with_retries(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 2429, in async_function_with_retries
    raise original_exception
  File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 2351, in async_function_with_retries
    response = await original_function(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 787, in _acompletion
    raise e
  File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 756, in _acompletion
    response = await _response
               ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/litellm/utils.py", line 1500, in wrapper_async
    raise e
  File "/usr/local/lib/python3.11/site-packages/litellm/utils.py", line 1315, in wrapper_async
    result = await original_function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/litellm/main.py", line 410, in acompletion
    raise exception_type(
          ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/litellm/utils.py", line 7665, in exception_type
    raise e
  File "/usr/local/lib/python3.11/site-packages/litellm/utils.py", line 6036, in exception_type
    raise NotFoundError(
litellm.exceptions.NotFoundError: litellm.NotFoundError: NotFoundError: OpenAIException - Error code: 404 - {'detail': 'Not Found'} LiteLLM Retried: 2 times, LiteLLM Max Retries: 3

�[92m16:25:29 - LiteLLM Proxy:ERROR�[0m: _common.py:120 - Giving up chat_completion(...) after 1 tries (litellm.proxy._types.ProxyException)
INFO:     127.0.0.1:43592 - "POST /chat/completions HTTP/1.1" 404 Not Found
INFO:     10.244.17.1:33106 - "GET /health/liveliness HTTP/1.1" 200 OK
INFO:     10.244.17.1:33104 - "GET /health/readiness HTTP/1.1" 200 OK

krrishdholakia Jul 8, 2024
Maintainer

this is being raised by the OpenAI client

File "/usr/local/lib/python3.11/site-packages/openai/resources/chat/completions.py", line 1181, in create
return await self._post(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/openai/_base_client.py", line 1790, in post
return await self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/openai/_base_client.py", line 1493, in request
return await self._request(
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/openai/_base_client.py", line 1584, in _request
raise self._make_status_error_from_response(err.response) from None
openai.NotFoundError: Error code: 404 - {'detail': 'Not Found'}

It looks like an issue connecting between containers

krrishdholakia Jul 8, 2024
Maintainer

Actually i'm mistaken

a Not Found error typically means the url is not found

the base is probably fine,

krrishdholakia Jul 8, 2024
Maintainer

can you modify your api_base to be

http://vllm-service.vllm.svc.cluster.local/v1

and let me know if that works @ksingh-scogo

Answer selected by ishaan-jaff

krrishdholakia Jul 8, 2024
Maintainer

Happy to setup a 1:1 support channel and help more here

LinkedIn
Discord (just 👋 wave on #general, and i'll setup a channel)

ksingh-scogo · 2024-07-08T19:05:09Z

ksingh-scogo
Jul 8, 2024
Author

@krrishdholakia you caught that right adding /v1 made it work so http://vllm-endpoint.xyz/v1

Would be great to fix it here https://docs.litellm.ai/docs/providers/openai_compatible#usage-with-litellm-proxy-server ?

1 reply

ishaan-jaff Jul 8, 2024
Maintainer

Added here @ksingh-scogo

https://docs.litellm.ai/docs/providers/openai_compatible#usage-with-litellm-proxy-server

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Connecting to VLLM Instance #4602

{{title}}

Replies: 4 comments 6 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Connecting to VLLM Instance #4602

ksingh-scogo Jul 8, 2024

What happened?

Relevant log output

Replies: 4 comments · 6 replies

ksingh-scogo Jul 8, 2024 Author

ksingh-scogo Jul 8, 2024 Author

krrishdholakia Jul 8, 2024 Maintainer

ksingh-scogo Jul 8, 2024 Author

krrishdholakia Jul 8, 2024 Maintainer

krrishdholakia Jul 8, 2024 Maintainer

krrishdholakia Jul 8, 2024 Maintainer

krrishdholakia Jul 8, 2024 Maintainer

ksingh-scogo Jul 8, 2024 Author

ishaan-jaff Jul 8, 2024 Maintainer

ksingh-scogo
Jul 8, 2024

Replies: 4 comments 6 replies

ksingh-scogo
Jul 8, 2024
Author

ksingh-scogo
Jul 8, 2024
Author

krrishdholakia
Jul 8, 2024
Maintainer

ksingh-scogo Jul 8, 2024
Author

krrishdholakia Jul 8, 2024
Maintainer

krrishdholakia Jul 8, 2024
Maintainer

krrishdholakia Jul 8, 2024
Maintainer

krrishdholakia Jul 8, 2024
Maintainer

ksingh-scogo
Jul 8, 2024
Author

ishaan-jaff Jul 8, 2024
Maintainer