Scanning theory

[enesdoruk]

Hi, What is the background of scanning half of channel features instead of all channels? The model splits channels by 2, and half of the channels pass through scanning, and the other half use just conv. What if we scan all channels and recalibrate with before scanning features?

[ahatamiz]

Hi @enesdoruk the idea is to have the network learn diverse set of features coming from both SSM and non-SSM branches. The SSM branch encodes an implicit inductive bias for pixel dependency where the network does not have access to the entire tokens. However the non-SSM branch removes all such dependencies. This allows the network to not quickly overfit (e.g. some features are easy guesses for example the head or tail of a bird) and learn more robust feature representations.

Hope it helped