added gemma2 9b and 27b vllm with streaming #318
Conversation
|
vllm's next release will add support for gemma2 9/27B. Until then you'd have to build from source on top of a pytorch image which takes 30+ minutes to deploy. vllm-project/vllm#5806 |
dsingal0
changed the title
| logger.info(f"tensor parallelism: {model_metadata['tensor_parallel']}") | ||
| logger.info(f"max num seqs: {model_metadata['max_num_seqs']}") | ||
|
|
||
| self.model_args = AsyncEngineArgs( |
There was a problem hiding this comment.
potential improvement is to move everything to config, e.g. as in this example:
https://github.com/vshulman/truss-examples/tree/main/ultravox-vllm
There was a problem hiding this comment.
we can merge without this change as other vLLM examples also support a partial list of arguments
There was a problem hiding this comment.
Looking through it, it looks like that example uses the vllm openai server instead of explicitly instantiating the vllm AsyncLLMEngine for the model.
There was a problem hiding this comment.
100% -- I just think the same kwargs pattern can apply here. the benefit I see is that going forward all it would take to pass a new argument into vLLM, either the standalone OpenAI server or the Python API above, is adding it to the config.yaml.
| logger.info(f"tensor parallelism: {model_metadata['tensor_parallel']}") | ||
| logger.info(f"max num seqs: {model_metadata['max_num_seqs']}") | ||
|
|
||
| self.model_args = AsyncEngineArgs( |
There was a problem hiding this comment.
100% -- I just think the same kwargs pattern can apply here. the benefit I see is that going forward all it would take to pass a new argument into vLLM, either the standalone OpenAI server or the Python API above, is adding it to the config.yaml.
I believe this is because the vllm implementation didn't use the chat template. I'm not familiar with that, but will give it a try. |
dsingal0
changed the title
… template. todo: confirm 27B working
dsingal0
changed the title
No description provided.