WIP: LoRA Adapters

[nstogner]

Addresses #132

[alpe]

Nice drawing. This is very helpful! 🌻
I am not super familiar with LoRa adapters but they can have significant size from what I saw. Caching seems a good idea. For the non-cache scenario, I would suggest to have no-cache or container-managed profile so that it does not look like the default to skip this.
With on-demand LoRa, the disk size may become a problem at some point. This is off scope but purge job or retention time can be things that need to be configured at some point in the profile.

[samos123]

Can you show an example that has the url field? I'm assuming the url field must be used to specify the base model?

[nstogner]

@samos123 I currently have all examples in the diagrams

[samos123]

That's where I looked but none of them have the base model URL set?

[nstogner]

Model .spec.url would be the same as normal.

[nstogner]

Note, it looks like vLLM supports loading adapters from huggingface: vllm-project/vllm#6234

[nstogner]

Note, vLLM has an endpoint to support dynamic loading/unloading of adapters: vllm-project/vllm#6566

[samos123]

does vLLM support directly loading this adapter from HF or is it a hard requirement to download the lora adapter first?

[nstogner]

vLLM can load it from HF but not S3

[nstogner]

See: https://github.com/vllm-project/vllm/blob/d1c6799b8870e513bf4f2305cbf6cda9fc3d773b/vllm/lora/utils.py#L178