What makes nnUNet have such a great performance?

[PedroMartelleto]

Thanks for the amazing work on nnUNet!

After reading the discussion at this link, I have a few questions about the inner workings and default settings of nnUNet. While I understand that many of these choices can be dataset-dependent, I’m curious about what generally works best on average based on any testing that has been done.

Normalization Layers: Has anyone tested the performance of BatchNorm compared to InstanceNorm in nnUNet? What were the results?

Optimizer Choice: The original nnUNet paper mentions the use of Adam (https://arxiv.org/pdf/1809.10486), but the current default appears to be SGD with Nesterov. Is there a significant difference in performance between these two optimizers?

Patch Sampling: In the linked discussion, it's mentioned that the location of patch sampling is important for final performance. Why this is the case and how nnUNet addresses this issue? Is it only due to class imbalance?

Automatic Architecture Definition: How does nnUNet automatically define a CNN architecture that fits well within GPU memory? Specifically, how are the channel counts and other parameters determined?

Basically I was curious behind the rationale for some of these inner workings, and which ones are essential for establishing a solid 3D CNN baseline. Any insights or shared experiences would be greatly appreciated! Also feel free to link me to any discussion/detailing of these points that I may have missed.

[MelindaDong]

Same question on Optimizer choice:
'Optimizer Choice: The original nnUNet paper mentions the use of Adam (https://arxiv.org/pdf/1809.10486), but the current default appears to be SGD with Nesterov. Is there a significant difference in performance between these two optimizers?'