-
|
Hello, this blog post describes a supposedly universal gradient accumulation bug, and I was wondering if someone was knowledgeable enough to know whether kohya training scripts (including this repo) suffers from this or it doesn't apply: https://unsloth.ai/blog/gradient |
Beta Was this translation helpful? Give feedback.
Answered by
kohya-ss
Oct 21, 2024
Replies: 1 comment 1 reply
-
|
From my understanding, this issue only occurs with cross entropy loss. Currently, in this repository, we use MSE loss, so this issue should not occur. |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
thanks for reviewing! |
Beta Was this translation helpful? Give feedback.
Answer selected by
ilazarte
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
From my understanding, this issue only occurs with cross entropy loss. Currently, in this repository, we use MSE loss, so this issue should not occur.