Sdnn skip connections in Lava Processes

[andychoi98]

Does lava support exporting an sdnn with skip connections into a lava process model?

[rileywhite-noblis]

I haven't tried this, but I wonder if this is possible by manually adjusting the delays. So layer 1 connects to its next layer and a skip layer that has been configured with a delay equal to the number of skipped layers. As far as I know there is no "add" layer right now so you may have to be creative.
For example the 2 inputs could be combined using the prodneuron models. Alternatively you could use a concat port feeding into a learned layer. The layer after a concat port could also be manually created to implement a variety of different combination algorithms (see the literature about spiking resnets).

[andychoi98]

Thanks for the reply!
But I'm curious what role the delay plays and why it might be an issue here. Even if there aren't delays for skip connections, wouldn't the computation of adding the skip connection output and the residual block output
What if I'm using an Sdnn without delays, but only convolutional layers of those similar to the pilotnet example? Would this still be a problem?
Also, I know it's possible to connect multiple input ports to a single output port or vice versa, so I was wondering if I export the sdnn of the structure with skip connections, it will successfully convert to a corresponding lava process model.

[rileywhite-noblis]

Delays are needed to account for the propagation delay with the LoihiProtocol (I am not thinking about the asynchronous one here). Every layer adds a time step of propagation delay between the network input and its output. So to account for this your skip layer needs to delay its output to time its output with the layers it skips. Unlike SNNs, SDNNs propagation delay is not accounted for by default in Slayer (shift_delay is False), which I consider a good thing. But that means you need to account for this delay when constructing your lava processes.

Right now delays are not supported with convolutional layers in lava, so you will not be able to do this with conv layers yet.

Something like this will not be supported by NetX and is something you would have to configure manually in Lava. NetX is fairly limited in what it can convert and does not cover the full feature set in Lava or Lava-dl. You can use the hdf5.py file in NetX as a guide for writing your custom loading code.

[andychoi98]

Hi Riley, thanks again for the reply.
I'm still not sure why exactly Delays would be needed or how it is used for synchronizing in the sdnn case. Conceptually, it does make sense, but in SDNNs, I saw that the data as it is passed through the network, it contains all the spiking info for discrete timesteps (for the number of time dimensions specified). In this case, I don't quite understand why a skip connection would be needed because both the input and output from the residual block will have matrices in the same dimensions with the last dimension being the Time axis.

I haven't looked much into the snn compared to the sdnn, so maybe the delays define the temporal synchronization in the snn spikes?
Also, is there an email or contact that I could discuss with you further?
Thanks a lot!

[rileywhite-noblis]

When I talk about delay I am speaking about when you convert it to a Lava process - not in Lava-dl for training because this is just PyTorch. If you want to simulate the propagation in Lava-dl/pytorch delay you can use delay_shift=True to see the results. SDNNs have this false by default.

You can reach me at [email protected], but I would suggest keeping any questions here so others can benefit from our conversation.