Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Apply lora by model patching #3583

Merged
merged 3 commits into from
Jun 28, 2023
Merged

Apply lora by model patching #3583

merged 3 commits into from
Jun 28, 2023

Conversation

StAlKeR7779
Copy link
Contributor

@StAlKeR7779 StAlKeR7779 commented Jun 26, 2023

Rewrite lora to be applied by model patching as it gives us benefits:

  1. On model execution calculates result only on model weight, while with hooks we need to calculate on model and each lora
  2. As lora now patched in model weights, there no need to store lora in vram

Results:
Speed:

loras count hook patch
0 ~4.92 it/s ~4.92 it/s
1 ~3.51 it/s ~4.89 it/s
2 ~2.76 it/s ~4.92 it/s

VRAM:

loras count hook patch
0 ~3.6 gb ~3.6 gb
1 ~4.0 gb ~3.6 gb
2 ~4.4 gb ~3.7 gb

As based on #3547 wait to merge.

@StAlKeR7779 StAlKeR7779 marked this pull request as ready for review June 28, 2023 19:43
Copy link
Collaborator

@lstein lstein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like it. Much more straightforward than the previous solution.

@lstein lstein enabled auto-merge June 28, 2023 19:48
@lstein lstein merged commit 8a90e51 into main Jun 28, 2023
6 checks passed
@lstein lstein deleted the feat/lora_model_patch branch June 28, 2023 19:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants