Backbone with FPN on top of Swin Transformers

[vsuryamurthy]

🚀 The feature

Adding a backbone with FPN on top of swin transformers similar to the already available resnet_fpn and mobilenet_fpn backbones (https://github.com/pytorch/vision/blob/main/torchvision/models/detection/backbone_utils.py). I do have a working version (need to be confirmed by the authors if it is what was used in the paper for comparisons) in a separate branch. However, as per protocol, I am opening an issue first to see if it is a worthwhile addition.

Motivation, pitch

It lets us use swin transformer based backbones for the object detection frameworks (FasterRCNN, MaskRCNN..).

Alternatives

No response

Additional context

No response

cc @datumbox

[datumbox]

Thanks for offering to help @vsuryamurthy. There are certainly lots of interesting newer backbones that can be combined with FasterRCNN/MaskRCNN. Because the number of such combinations can be easily explode, we currently offer variants that were included on the original paper. This is because the effort of configuring and training new variants is non trivial and can quickly add up. Nevertheless we acknowledge that offer new variants for good detection models can be useful to users and that's why we offer examples on our documentation on how to do this. I think what we can do is leave this issue open and if we decide to offer more variants, we can reach out to you for help. What do you think?

[vsuryamurthy]

Yes understood and I completely agree with you. My only motivation was the fact that swin transformer is quite popular and has high score with MaskRCNN but I can imagine that being surpassed in a short duration and we are left with another state of the art. Lets do according to what you proposed. Thank you :)

[mikeklm]

I'm interested in this @vsuryamurthy, if you wouldn't mind sharing. I wrote an fpn neck on top of swin-t backbone in the manner outlined above as well. I took. me a minute to realize that the output of the swin block was NHWC, but the FPN expects NCHW as input.

Things run fine, but when I create a MaskRCNN model using my swin-t backbone with FPN neck, I get some index out of range error when I try to run (coming from ops/poolers.py)