Skip to content

Latest commit

 

History

History
117 lines (83 loc) · 2.7 KB

serving.md

File metadata and controls

117 lines (83 loc) · 2.7 KB
Raw

Serving a model

Serving LLaMA-2

You can download llama-2 models from huggingface and serve them like below:

7B 
python3 -m lmdeploy.serve.turbomind.deploy llama2 /path/to/llama-2-7b-chat-hf
bash workspace/service_docker_up.sh
13B 
python3 -m lmdeploy.serve.turbomind.deploy llama2 /path/to/llama-2-13b-chat-hf --tp 2
bash workspace/service_docker_up.sh
70B 
python3 -m lmdeploy.serve.turbomind.deploy llama2 /path/to/llama-2-70b-chat-hf --tp 8
bash workspace/service_docker_up.sh

Serving LLaMA

Weights for the LLaMA models can be obtained from by filling out this form

7B 
python3 -m lmdeploy.serve.turbomind.deploy llama /path/to/llama-7b llama \
    --tokenizer_path /path/to/tokenizer/model
bash workspace/service_docker_up.sh
13B 
python3 -m lmdeploy.serve.turbomind.deploy llama /path/to/llama-13b llama \
    --tokenizer_path /path/to/tokenizer/model --tp 2
bash workspace/service_docker_up.sh
30B 
python3 -m lmdeploy.serve.turbomind.deploy llama /path/to/llama-30b llama \
    --tokenizer_path /path/to/tokenizer/model --tp 4
bash workspace/service_docker_up.sh
65B 
python3 -m lmdeploy.serve.turbomind.deploy llama /path/to/llama-65b llama \
    --tokenizer_path /path/to/tokenizer/model --tp 8
bash workspace/service_docker_up.sh

Serving Vicuna

7B 
python3 -m pip install fschat
python3 -m fastchat.model.apply_delta \
  --base-model-path /path/to/llama-7b \
  --target-model-path /path/to/vicuna-7b \
  --delta-path lmsys/vicuna-7b-delta-v1.1

python3 -m lmdeploy.serve.turbomind.deploy vicuna /path/to/vicuna-7b
bash workspace/service_docker_up.sh
13B 
python3 -m pip install fschat
python3 -m fastchat.model.apply_delta \
  --base-model-path /path/to/llama-13b \
  --target-model-path /path/to/vicuna-13b \
  --delta-path lmsys/vicuna-13b-delta-v1.1

python3 -m lmdeploy.serve.turbomind.deploy vicuna /path/to/vicuna-13b
bash workspace/service_docker_up.sh