GPT training problems after DVAE finetuning

[NikitaKononov]

Hello! I have a clean dataset with ~430 hours of data
I have finetuned DVAE for 5 epochs
Then I try to train GPT with this dvae
But it fails - all outputs are like robotic mechanic screams "eeeeeEEEeeEEEaaa"
What the problem can be? Training for 6 epochs already, situation doesn't change

PS traning with default DVAE is all good on this data

[NikitaKononov]

My DVAE losses

[NikitaKononov]

GPT Loses with new DVAE

[NikitaKononov]

And GPT losses with default DVAE

[a7mdkhald]

hello @NikitaKononov i have the same problem did you manage to solve it ?

[NikitaKononov]

hello @NikitaKononov i have the same problem did you manage to solve it ?

hello
unfortunately I haven't found a solution yet

[NikitaKononov]

tried to fine-tune Hifigan with gpt trained with tuned DVAE, still no good results