Network Alpha

[djigneo]

Hello, could someone explain Network Alpha a bit? If there's an article that explains it clearly, please send it my way. I'm seeing a lot of conflicting information and am at this point not sure what to trust.

As I understand:

1. It applies a dampening effect to learning by affecting how much network weight is stored.
2. It ranges in value from 1 to [Network dim / rank].

As I try to understand the article written here: https://github.com/bmaltais/kohya_ss/wiki/LoRA-training-parameters#network-alpha I am unclear which values are higher dampening and which are lower.

If [Network alpha] is set to the value of [Network dim], is it the most dampening possible or the least? (Or am I misunderstanding things entirely?)

Any clarification would be greatly appreciated.

[KW-0330]

From my years of lora training experience, dim = alpha yield the best result. Any alpha value less than dim will cause the learning rate reducing efficiency. To overcome the reducing LR efficiency, you need to increase your training steps. Too many steps will cause over training.

[person4268]

For example, with an Alpha of 16 and a Rank of 32, the strength of the weight used is 16/32 = 0.5, meaning that the learning rate is only half as powerful as the Learning Rate setting.

If Alpha and Rank are the same number, the strength used will be 1 and will have no effect on the learning rate.

makes me think that less alpha == more dampening