implementation of "Bottleneck Transformers for Visual Recognition"

[ydhongHIT]

Thanks for your great work. Do you have the plan to implement the models in this paper (https://arxiv.org/pdf/2101.11605.pdf)?

[rwightman]

@ydhongHIT
Yes, but it's in a rather early 'draft' stage. Have bits and pieces working, but slow and not sure it's 100% correct.

I have a related idea LambdaNetworks - https://arxiv.org/abs/2102.08602 a little further along. But again, current impl is quite slow in PyTorch so not very practical yet. I'm having issues with AMP where it's unstable to train (but is fine in float32, but even slower). It'll probably take a bit of time to sort those issues out, and might need to consult some NVidia or PyTorch people to figure out why it's slower than the numbers reported in the papers (which are likely JAX or TF impl).

The plan is to have a ResNet backbone, either separate or part of the current one that allows defining models where specific blocks use a MHA/MHA-like attention module in place of the 3x3.
So far the targets are:

• Bottleneck Transformers - https://arxiv.org/abs/2101.11605
• Lambda Networks - https://arxiv.org/abs/2102.08602
  and possibly,
• GSA - https://arxiv.org/abs/2010.03019

and there are some other proposed attn modules that'd work in that manner...

[ydhongHIT]

Thanks for your reply. Looking forward to your updates.