Mixed precision training?

[ghost]

Pytorch 1.6 has native support for automatic mixed precision (AMP) training: https://pytorch.org/blog/pytorch-1.6-released/

Should we take advantage of this? In particular I think the larger batches would be nice for encoder and synthesizer training.

[ghost]

Mozilla TTS tried it and encountered a bug with RNNs to be fixed in a future Pytorch release. mozilla/TTS#486 (comment)

[ghost]

Pytorch 1.7 just released, so it is time to try again. If we add AMP, the implementation must be clean while preserving support for lower versions of PT.

[ghost]

See fatchord/WaveRNN#229 for an example of how to do this.

[ghost]

Closing due to lack of developer interest at this time.
Please comment and reopen if you would like to work on this.

[ghost]

I made a branch that supports mixed precision training. It is not recommended for use at this time.
https://github.com/blue-fish/Real-Time-Voice-Cloning/tree/487_mixed_precision_training

For me, mixed precision training is much slower than without it, and loss is occasionally nan. I also had to set up my Python environment with Anaconda due to a problem with matrix multiplication. pytorch/pytorch#56747 (comment)

Pytorch AMP enabled (Python 3.9.7 with Anaconda, pytorch==1.10.0):

{| Epoch: 1/8 (20/2564) | Loss: nan | 0.24 steps/s | Step: 0k | }
Average execution time over 10 steps:
  Blocking, waiting for batch (threaded) (10/10):  mean:    0ms   std:    0ms
  Data to cuda (10/10):                            mean:    1ms   std:    0ms
  Forward pass (10/10):                            mean:  956ms   std:  151ms
  Loss (10/10):                                    mean:   14ms   std:    3ms
  Backward pass (10/10):                           mean: 3013ms   std:  456ms
  Parameter update (10/10):                        mean:   71ms   std:    5ms
  Extras (visualizations, saving) (10/10):         mean:    0ms   std:    0ms

Same setup without AMP:

{| Epoch: 1/8 (20/2564) | Loss: 5.778 | 0.76 steps/s | Step: 0k | }
Average execution time over 10 steps:
  Blocking, waiting for batch (threaded) (10/10):  mean:    0ms   std:    0ms
  Data to cuda (10/10):                            mean:    1ms   std:    0ms
  Forward pass (10/10):                            mean:  451ms   std:   52ms
  Loss (10/10):                                    mean:   10ms   std:    2ms
  Backward pass (10/10):                           mean:  753ms   std:   73ms
  Parameter update (10/10):                        mean:   31ms   std:    3ms
  Extras (visualizations, saving) (10/10):         mean:    5ms   std:    0ms

[ghost]

Dropping this due to poor performance and lack of interest.