Warning: While the example images are always valid, it is better to check the Guidelines for more up-to-date information on algorithm comparison
Check our paper for more examples
lora for transformer only vs lora for whole model
Yog-Sothoth LoRA/LoCon: https://civitai.com/models/14878/loconlora-yog-sothoth-depersonalization
LoRA rank=1:
LoCon rank=1
xy grid:
With some experiments from community, finetuning whole model can learn "More". But in some case this means it will overfit heavily.
We introduced LoRA finetuning with Hadamard Product representation from FedPara. And based on this experiments, LoHa with same size (and dim>2, or rank>4) can get better result in some situation.
This is some comments from the experiments:
Why LoHa?
The following comparison should be quite convincing.
(Left 4 LoHa with different number of training steps; Right 4 same thing but for LoCon; same seed same training parameters for the two training)
In addition to the five characters I also train a bunch of style into the model (what I have always been doing actually).
However, LoRa and LoCon do not combine styles with characters that well (characters are only trained with anime images) and this capacity gets largely improved in LoHa.
Note that the two files have almost the same size (around 30mb). For this I set (linear_dim, conv_dim) to (16,8) for LoCon and (8,4) for LoHa. However with Hadamard product the resulting matrix could now be of rank 8x8=64 for linear layers and 4x4=16 for convolutional layers.
And there is more example about LoHA vs LoCon in same file size. (diff < 200KB)
