Losses don't reduce and after some epochs all values become NaN

[prabathbr]

I am trying "CycleGAN" model using unaligned dataloader with float data from NPZ as I need more than 8 bits per channel.
Input and output number of channels is 1. Is there any other place to be changed to suit float data other than the dataloader and visualize (save checkpoint images in NPZ) ?

After training some epochs all values becomes "NaN" and it looks like losses don't reduce either.

What would be the parameters to be adjusted ? Any idea is much appreciated.

learning rate 0.0002000 -> 0.0002000
(epoch: 30, iters: 512, time: 0.029, data: 0.001) D_A: 0.012 G_A: 1.031 cycle_A: 10.047 idt_A: 8.217 D_B: 0.071 G_B: 0.943 cycle_B: 16.442 idt_B: 5.009
(epoch: 30, iters: 1312, time: 0.037, data: 0.001) D_A: 0.035 G_A: 0.951 cycle_A: 10.739 idt_A: 8.698 D_B: 0.041 G_B: 0.598 cycle_B: 17.375 idt_B: 5.392
saving the model at the end of epoch 30, iters 44160
End of epoch 30 / 200    Time Taken: 32 sec
learning rate 0.0002000 -> 0.0002000
(epoch: 31, iters: 640, time: 0.028, data: 0.001) D_A: 0.041 G_A: 1.256 cycle_A: 11.521 idt_A: 8.020 D_B: 0.054 G_B: 0.740 cycle_B: 16.043 idt_B: 5.768
(epoch: 31, iters: 1440, time: 0.025, data: 0.000) D_A: 0.006 G_A: 0.932 cycle_A: 8.065 idt_A: 9.653 D_B: 0.036 G_B: 0.818 cycle_B: 19.315 idt_B: 4.019
End of epoch 31 / 200    Time Taken: 28 sec
learning rate 0.0002000 -> 0.0002000
(epoch: 32, iters: 768, time: 0.029, data: 0.001) D_A: 0.008 G_A: 1.045 cycle_A: 10.575 idt_A: 8.976 D_B: 0.030 G_B: 0.873 cycle_B: 17.940 idt_B: 5.309
End of epoch 32 / 200    Time Taken: 27 sec
learning rate 0.0002000 -> 0.0002000
(epoch: 33, iters: 96, time: 0.028, data: 0.001) D_A: 0.005 G_A: 0.977 cycle_A: 8.332 idt_A: 8.994 D_B: 0.030 G_B: 0.964 cycle_B: 17.992 idt_B: 4.179
(epoch: 33, iters: 896, time: 0.026, data: 0.000) D_A: nan G_A: nan cycle_A: nan idt_A: nan D_B: nan G_B: nan cycle_B: nan idt_B: nan
End of epoch 33 / 200    Time Taken: 27 sec
learning rate 0.0002000 -> 0.0002000
(epoch: 34, iters: 224, time: 0.029, data: 0.000) D_A: nan G_A: nan cycle_A: nan idt_A: nan D_B: nan G_B: nan cycle_B: nan idt_B: nan
(epoch: 34, iters: 1024, time: 0.024, data: 0.000) D_A: nan G_A: nan cycle_A: nan idt_A: nan D_B: nan G_B: nan cycle_B: nan idt_B: nan
End of epoch 34 / 200    Time Taken: 27 sec
learning rate 0.0002000 -> 0.0002000
(epoch: 35, iters: 352, time: 0.028, data: 0.001) D_A: nan G_A: nan cycle_A: nan idt_A: nan D_B: nan G_B: nan cycle_B: nan idt_B: nan
(epoch: 35, iters: 1152, time: 0.024, data: 0.001) D_A: nan G_A: nan cycle_A: nan idt_A: nan D_B: nan G_B: nan cycle_B: nan idt_B: nan
saving the model at the end of epoch 35, iters 51520
End of epoch 35 / 200    Time Taken: 27 sec