Skip to content

Commit

Permalink
[Doc] Fix the lora adapter path in server startup script (#6230)
Browse files Browse the repository at this point in the history
  • Loading branch information
Jeffwan authored Jul 16, 2024
1 parent c467dff commit 94162be
Showing 1 changed file with 4 additions and 1 deletion.
5 changes: 4 additions & 1 deletion docs/source/models/lora.rst
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,10 @@ LoRA adapted models can also be served with the Open-AI compatible vLLM server.
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-2-7b-hf \
--enable-lora \
--lora-modules sql-lora=~/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/
--lora-modules sql-lora=$HOME/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/snapshots/0dfa347e8877a4d4ed19ee56c140fa518470028c/
.. note::
The commit ID `0dfa347e8877a4d4ed19ee56c140fa518470028c` may change over time. Please check the latest commit ID in your environment to ensure you are using the correct one.

The server entrypoint accepts all other LoRA configuration parameters (``max_loras``, ``max_lora_rank``, ``max_cpu_loras``,
etc.), which will apply to all forthcoming requests. Upon querying the ``/models`` endpoint, we should see our LoRA along
Expand Down

0 comments on commit 94162be

Please sign in to comment.