TTS - FastPitch+HifiGAN - Increasing validation loss while decreasing training loss

[gedefet]

Hi all.

I'm currently experiencing weird results regarding training. I'm trying to generate a voice, having 26min of audio, and I'm training mixing this dataset with LJSpeech as the tutorial suggest.

I've tried changing the amount of data of one dataset between the other, adding a 3rd dataset (HiFiTTS), but always with the same result. Here are some captures from Tensorboard while training FastPitch:

Training loss:

Validation loss:

As you can see, validation loss is always increasing. That being said, the final audio is not always bad. But certainly can be improved for sure.

On the other hand, HiFiGAN is training and validating properly, and it works great.

Thank you in advance.

[opalrose510]

I have this problem as well, also training FastPitch + HiFiTTS + HiFiGAN, and am also wondering what this signifies? My audio is also generally very good, but it seems like it can be improved.

[gedefet]

@Oktai15 @okuchaiev any ideas?