Downsampling segmentation labels vs Upsampling predictions

[aicaffeinelife]

Hi guys,

Thanks for having this awesome repo. I'm writing some code for semantic segmentation and noted that you are downsampling labels instead of upsampling predictions. This issue is not a code issue, but rather an open ended question:

1. Downsampling segmentation labels may lead to loss of some spatially correlated pixels, which the NLLLoss may not penalize, so is downsampling justified simply because it leads to less memory consumption?

2. How would we go about implementing auxiliary losses like SELoss1 or patchwise pixel loss if we downsample segmentation labels?

[hangzhaomit]

This is a good question. Your points in 1. are exactly the trade-offs in this problem.
A recent paper tries to solve this problem by sampling more patches around edges, which is effective.
https://arxiv.org/abs/1912.08193

[aicaffeinelife]

Thanks for bringing this very interesting paper to my attention. I've not read the paper in depth, but from what I can understand, sampling points drawn from a uniform distribution can lead to better semantic boundaries, without having to upsample the predictions. I believe that this opens up a very new space for exploration in highly efficient semantic segmentation architectures.

What are your thoughts on this?