is D-Adaptation available in kohya-ss branch too?

[runner22k]

I recently saw a video from Kohya_ss GUI project. In it, he was using some dynamic way of learning without specifying training steps, called "DAdaptation". How can we use dadaptation with sd-scripts? did anyone tried to use such technique?

https://pypi.org/project/dadaptation/

[thojmr]

Yea it is now. With fine_tune.py I added this:

 --optimizer_type "DAdaptation" \
 --learning_rate=1

Remove --8bit_adam since its not compatible.
Weight decay is optional if you are over fitting according to Dadaptation paper --optimizer_args "weight_decay=0.04"

[runner22k]

Thank you so much. so, only learning rate is enough? where can I find documentation? will it work with 8GB VRAM?
What about Text Encoder learning rate, and Unet learning rate? (all three values are needed in kohya_ss GUI, so asking.)

[thojmr]

The only details I have on settings are from here https://github.com/facebookresearch/dadaptation
No Idea on min VRAM usage. Mines usually higher than that.
For LR I if Im using 1e-4 Unet, ill do between 1e-5 and 5e-5 text encoder with LoRA. For Finetune, I don't think the text encoder LR has any effect.

[swfsql]

Just to add more info, I could not run it under 6GB.