Offline inference over multiple vLLM instances

[nivibilla]

Hi,

Is it possible to have multiple vLLM instances on a single machine that has multiple gpus. And then have code to inference instead of a server?

My reason being, currently I manually start 4 different notebooks to batch inference over data using the same model. And I have to partition my dataset and run 4 instances of the same code.

If I could have a router to do this at an individual request level and distribute this to the 4 different model instances throughput would increase by 4x. Btw the reason for this is because doing tensor- paralell 4 does not increase throughput at all. Whereas having 4 individual vLLM instances of a model can increase 4x throughput.

[ishaan-jaff]

Hi @nivibilla yes absolutely this is the perfect use case for the LiteLLM router. If you want python code to run inference on the server here's how you can do so :

https://docs.litellm.ai/docs/routing

from litellm import Router

# list of model deployments 
model_list = [
{ 
    "model_name": "llama2", # model alias 
    "litellm_params": { # params for litellm completion/embedding call 
        "model": "llama2", # actual model name
        "custom_llm_provider": "openai", # tell litellm to route this as an openai endpoint
        "api_base": "http://192.168.1.23:8000/v1"
    }
}, 
{ 
    "model_name": "llama2", 
    "litellm_params": { 
        "model": "llama2", 
        "custom_llm_provider": "openai", 
        "api_base": "http://192.168.1.23:8010/v1"
    }
},
{ 
    "model_name": "llama2", 
    "litellm_params": { 
        "model": "llama2",
        "custom_llm_provider": "openai",
        "api_base": "http://192.168.1.23:8001/v1"
    }
},
]

router = Router(model_list=model_list)

# openai.ChatCompletion.create replacement
response = router.completion(model="gpt-3.5-turbo", 
                messages=[{"role": "user", "content": "Hey, how's it going?"}])

print(response)

[ishaan-jaff]

Please let me know if you run into any issues - I want to make sure it works for your use case

Here's our discord https://discord.com/invite/wuPM9dRgDw + I reached out on Linkedin

[nivibilla]

Hi @ishaan-jaff . Thanks for the code, I understand the routing part. But I see they are endpoints. Is it possible to point them to variables instead?

As in if I had
llm1 = vLLM(model1)
llm2 = vLLM(model1)
llm3 = vLLM(model1)

How can I route these/do paralell inference over a dataset? I'm clear on how to do route for live Inference but im trying to find a solution for batch inference

[krrishdholakia]

@nivibilla check this out - https://docs.litellm.ai/docs/providers/vllm

i believe this would just require changing model name to vllm/

[andakai]

Does this support the mode like llm1=vLLM(model) without api? How to support multi instances offline?

[krrishdholakia]

let me know if this solves your problem

[andakai]

Hi @ishaan-jaff . Thanks for the code, I understand the routing part. But I see they are endpoints. Is it possible to point them to variables instead?

As in if I had llm1 = vLLM(model1) llm2 = vLLM(model1) llm3 = vLLM(model1)

How can I route these/do paralell inference over a dataset? I'm clear on how to do route for live Inference but im trying to find a solution for batch inference

Hi, @nivibilla , do you have solutions now? I am meeting the same need. I also want to use llm=vLLM(model1), not the api mode.

[nivibilla]

@darrenglow I've migrated to using Ray. You can instantiate N number of workers which load a vllm instance. And distribute the data over all available workers.

[andakai]

Is there any good way to achieve load balance?

[nivibilla]

I'm using it for offline batch Inference, it will distribute accordingly.

See this
https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_distributed.py

[nivibilla]

For offline Inference, the solution above should work. There is also ray serve and rayllm but it's a little more complicated to set up.

[andakai]

Thanks. But I realize that I have to do inference on multi nodes, so maybe online inference is a must. :(

[nivibilla]

Then either the solution above. Where you host multiple servers and load balance using litellm or use ray serve/ray llm