tests_gaudi: Added L2 vllm workload #329
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: vbedida79 <[email protected]>
uMartinXu
reviewed
Nov 6, 2024
| @@ -74,4 +74,83 @@ Welcome to HCCL demo | |||
| [BENCHMARK] NW Bandwidth : 258.209121 GB/s | |||
| [BENCHMARK] Algo Bandwidth : 147.548069 GB/s | |||
| #################################################################################################### | |||
| ``` | |||
|
|
|||
| ## VLLM | |||
| ``` | ||
|
|
||
| ## VLLM | ||
| VLLM is a serving engine for LLM's. The following workloads deploys a VLLM server with an LLM using Intel Gaudi. Refer to [Intel Gaudi VLLM fork](https://github.com/HabanaAI/vllm-fork.git) for more details. |
| Build the workload container image: | ||
| ``` | ||
| $ oc apply -f https://raw.githubusercontent.com/intel/intel-technology-enabling-for-openshift/main/tests/gaudi/l2/vllm_buildconfig.yaml | ||
| ``` |
There was a problem hiding this comment.
Could we add the instruction to let user know whether the building is success. :-)
| ``` | ||
| Deploy the workload: | ||
| * Update the hugging face token and the pvc according to your cluster setup | ||
| ``` |
There was a problem hiding this comment.
Could we have some detail about setting the hugging face token? and also give some brief introduction about what model we are using. :-)
uMartinXu
reviewed
Nov 7, 2024
| runPolicy: "Serial" | ||
| source: | ||
| git: | ||
| uri: https://github.com/opendatahub-io/vllm.git |
There was a problem hiding this comment.
After comparing
- https://github.com/opendatahub-io/vllm.git - ODH fork vllM
- https://github.com/vllm-project/vllm - vLLM upstream
3.https://github.com/HabanaAI/vllm-fork - Habana fork vLLM
I think currently we should start from use the 3. with the change in 1 (adding the ubi based docker file for RH OpenShift), and obviously the Intel are upstreaming from 3 to 2. So in the long run we will using 2.
So I think we need to 1). submit a PR to adding the ubi based docker file for RH, and also add the RH 9.4 support into the documents, and then 2). using repo 3 3) I think the owner of 3 will also help to upstream the ubi based docker file and doc to 2. 4) after that we can switch to use 2 the upstream vLLM.
@vbedida79 any comments? :-)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR includes gaudi l2 vllm workload
Signed-off-by: vbedida79 [email protected]