why delay/latency of the neural net is considered in yolo_kp example and not considered in netx/mnist example

[gwgknudayanga]

Hi Experts,

• In the netx/yolo_kp example , with the following code they used two buffers saying that "There is a latency in the prediction equal to the number of layers the network has and the encoding step. Two FIFO buffers are used to synchronize the input frame and target annotation with the predicted output."

frame_buffer = netx.modules.FIFO(depth=len(net) + 1)
annotation_buffer = netx.modules.FIFO(depth=len(net) + 1)

• But in netx/mnist there is no such consideration for such a latency due to the layers of the neural network.

• Actually when we consider one time step, the input to the network should be completely propagated to the output of the neural network at the end of that time step. If this is the case, why we worry about additional offset/delay which equals to the len(net) in yolo_kp? This is very confusing.

Thanks and Rgds,
Udayanga

[waltergallegog]

Hi @gwgknudayanga
I'm not an expert of Lava, just an user, and I don't know too much about the specifics of your questions (as I have not used the lava-dl library yet or looked at the examples), but from my own experience with lava, this:

There is a latency in the prediction equal to the number of layers the network has and the encoding step

Is expected when using connection models (like dense or sparse).
For example if you look at the code of the dense connection you will see there is a delay of 1 timestep (by default, that you can customize if you use the DelayDense model).
So the input to the network is not immediately propagated to all the layers in the network. If you have a network like:

L1 -> Dense12 -> L2 -> Dense23 -> L3

For the first 2 timesteps, L3 will process garbage (Dense23 will produce all zeros as output even if there is no input, to avoid deadlocks). Only after the input has propagated to all Layers (timestep 3) you can consider the output of L3 valid.

Take all the above with a grain of salt, but it might be helpful.

BR
Walter

[gwgknudayanga]

@waltergallegog thank you !!.

• So, in addition do i need to set delay_shift = True , in slayer (slayer.cuba.conv(...delay_shift = True)) during training?
• What i did was during training i set the delay_shift = False, and during inference in loihi board , i get the outputs after len(network) time steps. And I reset the network every 32 timesteps with a phase_shift of 1 time step since during training i use 31 time steps for inference. I use mean only bactnormalization. The output logits of the network is obtained by reading the output layer's neuron voltages. In this case i set the threshold of the output layer's neurons to very high value(2048) so that it will never spike during desired 32 time steps. The problem is , while in slayer it shows 96% accuracy while in loihi inference it is 55% , a huge drop. If you have any suggestions for help, really appreciate.

Thanks and Rgds,
Udayanga

[waltergallegog]

Hi @gwgknudayanga
Sorry but I can not help you on this, as I have never used lava-dl / Slayer. I hope someone more experienced can chime in.
BR,
Walter