[Feature] support multiple lora adaptor

[comeby]

Motivation

S-LoRA: Serving Thousands of Concurrent LoRA Adapters [paper]
The paper claims that “S-LoRA can improve the throughput by up to 4 times and increase the number of served adapters by several orders of magnitude.”

Support multiple lora adaptor could be transcendental to cost effective LoRA model severing.

will you support this feature?

Related resources

https://github.com/S-LoRA/S-LoRA

Additional context

No response

[lvhan028]

Hi, @comeby
Thanks for your attention to lmdeploy
We'll investigate it

[grimoire]

Hi, @comeby
I am working on support S-LoRA on pytorch-poc branch. Where can I get multiple unmerged Adapters?

[Cloopen-ReLiNK]

The effects of reasoning vary widely between the following two strategies:
(1) merged adapter, then load the new model file saved. Obtained the expected accuracy
(2) Load adapter through s-lora ，the accuracy is much lower

adapters = {'default':'xx/lora_test'}
engine_config = PytorchEngineConfig(adapters=adapters)
model_path = 'xxx/chatglm2-6b'

engine = Engine.from_pretrained(model_path,
engine_config=engine_config,
trust_remote_code=True)
gen_config = GenerationConfig(top_p=0.8,
top_k=40,
temperature=0,
max_new_tokens=1024 )

[Cloopen-ReLiNK]

https://modelscope.cn/models/walker31350430a/test_lora/files
Here is an adapter weight and test data (test.json), prediction code (test.py)
@grimoire

[Cloopen-ReLiNK]

solved ，#1042
But why is using pytorch’s inference engine slower than using transformers to predict directly?