training issue

[Koid217]

Hello, I started training again from 0 and iterated 500,000 iters to get the checkpoint and tested it. The average PSNR I got was only about 21, which is different from the checkpoint result you gave. What went wrong?

[Koid217]

How can I adjust to solve this problem.

[wyhlovecpp]

I think you can download ITK-SNAP to check the visualization results. In my experience, maybe IRM training fails. Instead of predicting residuals, IRM output noisy or zero images. The main reason is that the two-stage network of this paper (CPM&IRM) sharing the same gradient has the problem of training instability.
The solution is to check visualisation results as soon as possible. In my experience, training problems are often spotted early in the training phase with checkpoints 40,000 iters. If there is a model collapse, it is sufficient to train from scratch in time.

[Koid217]

I didn't change any parameters during training, why does this happen? How can I adjust to solve this problem? Thanks for your replay.

[wyhlovecpp]

Our experimental code does not have a uniform seed, so the results of multiple training are quite different. And I have explained in detail in the previous reply why the problem occurs and how to solve it.

[Koid217]

Thank you very much for your answer, I will try it.