SNN converted to lava has different outputs

[fangwei123456]

Hi, I have trained a SNN to classify the MNIST dataset. The SNN is built by lava-dl and works well. But when I convert the SNN to lava, I gets different outputs:

import torch
import torch.nn as nn
import torchvision
from torchvision import transforms

from lava.lib.dl import netx, slayer
from lava.proc import io
from lava.magma.core.run_configs import Loihi1SimCfg
from lava.magma.core.run_conditions import RunSteps


def encoder(x: torch.Tensor):
    return (torch.rand_like(x) < x).to(x)


def main():
    T = 16
    data_dir = '/datasets/MNIST/'
    transform_test = transforms.Compose([
        transforms.ToTensor(),
        # transforms.Normalize(0.1307, 0.3081),
    ])

    test_set = torchvision.datasets.MNIST(root=data_dir, train=False, transform=transform_test, download=True)

    neuron_params = {'threshold': 1.0, 'current_decay': 1.0, 'voltage_decay': 0.0, 'scale': 64}
    net_ladl = nn.Sequential(
        slayer.block.cuba.Conv(neuron_params, 1, 16, kernel_size=3, stride=1, padding=1, delay_shift=False),
        slayer.block.cuba.Conv(neuron_params, 16, 16, kernel_size=2, stride=2, delay_shift=False),
        slayer.block.cuba.Conv(neuron_params, 16, 16, kernel_size=3, stride=1, padding=1, delay_shift=False),
        slayer.block.cuba.Conv(neuron_params, 16, 16, kernel_size=2, stride=2, delay_shift=False),
        slayer.block.cuba.Flatten(),
        slayer.block.cuba.Dense(neuron_params, 16 * 7 * 7, 128, delay_shift=False),
        slayer.block.cuba.Dense(neuron_params, 128, 10, delay_shift=False),
    )

    net_ladl.load_state_dict(torch.load('./net_ladl.pt'))

    net_lava = netx.hdf5.Network(net_config='./net_lava_dl.net', input_shape=(28, 28, 1))



    for img, label in test_set:
        print(f'label = {label}')
        x = img.unsqueeze(-1).repeat(1, 1, 1, T)
        # [C, H, W, T]
        x = encoder(x)


        with torch.no_grad():
            y = net_ladl(x.unsqueeze(0))
            print(y)
            print('y(ladl)=', y.sum(-1).argmax().item())

        x = x.permute(1, 2, 0, 3)
        x = x.numpy()
        # x = slayer.io.tensor_to_event(x)
        source = io.source.RingBuffer(data=x)
        sink = io.sink.RingBuffer(shape=(10, ), buffer=T)
        source.s_out.connect(net_lava.inp)
        net_lava.out.connect(sink.a_in)
        run_condition = RunSteps(num_steps=T)
        run_config = Loihi1SimCfg(select_tag='fixed_pt')
        net_lava.run(condition=run_condition, run_cfg=run_config)
        output = sink.data.get()
        net_lava.stop()
        print(output)
        print('y(lava)=', output.sum(-1).argmax())

if __name__ == '__main__':
    main()

The outputs are:

label = 7
tensor([[[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.],
         [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
         [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]]])
y(ladl)= 7
[[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 1. 0. 0. 1.]
 [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0. 0. 0. 0. 1. 1. 0. 1. 0. 0. 1. 0.]
 [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 1. 0. 0. 0. 0. 1.]
 [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]
y(lava)= 4

'./net_ladl.pt' and './net_lava_dl.net' are attached:

weights.zip

# Name                    Version                   Build  Channel
lava                      0.4.0              pyhd8ed1ab_0    conda-forge

I have installed lava 0.4.0. To use the latest lava-dl, I firstly install lava-dl 0.2.0 by anaconda, and overwirtes codes in ~/anaconda3/envs/lava-env/lib/python3.10/site-packages/lava/lib/dl by the latest codes from github.

[bamsumit]

Can you try these two things:

• Change x = x.permute(1, 2, 0, 3) to x = x.permute(2, 1, 0, 3). lava-dl expects data in order CHWT whereas lava expects data in order WHCT (XYZT)
• It seems you are not using delays between layers during training. You can't avoid delays in lava. So you will need to reset the layers at different timesteps to get the same behavior if you have a bias in the network (which I am guessing you don't have).

[fangwei123456]

Thanks for your reply! I get the correct results when I use x = x.permute(2, 1, 0, 3).

I have two more questions:

1. I set delay_shift=False during training with lava, and set delay_shift = True when inference with lava and num_steps = T + n_delay, where n_delay is the number of blocks with delay_shift = True, and I use the last T time-steps as the inference output. Is this the correct way to deal with delays in lava?

2. How can I use bias with lava-dl? I do not find this param in lava.lib.dl.slayer.block.cuba.Conv or lava.lib.dl.slayer.block.cuba.Dense.

[bamsumit]

Right it does not change the outcome because you have no bias. If you use normalization on the neuron, the normalization value comes out as neuron bias. Synaptic bias is not enabled in lava-dl.

[gwgknudayanga]

@bamsumit,
in above discussion you mentioned that when there is bias in the network(which comes due to MeanOnlyBatchNorm) the network should be trained as delay_shift=True. But in the yolo_kp example ("") they trained the network with delay_shift = False though they used mean only batch normalization as follows. After that they exported this trained network to lava (hdf5) and when inferencing in loihi they just read the network outputs with a offset that is equal to number of layers of the (network + 1). So is this way is wrong since the network is trained without setting the delay_shift = True?

synapse_kwargs = dict(weight_norm=False, pre_hook_fx=quantize_8bit)
block_5_kwargs = dict(weight_norm=True, delay_shift=False, pre_hook_fx=quantize_5bit)
block_8_kwargs = dict(weight_norm=True, delay_shift=False, pre_hook_fx=quantize_8bit)
neuron_kwargs = {**sdnn_params, 'norm': slayer.neuron.norm.MeanOnlyBatchNorm}

Thanks and Rgds,
Udayanga

[bamsumit]

Its not wrong to use delay_shift = False. But it does mean that you should be aware that the final inference in lava to different than what you are seeing in lava-dl. With delay_shift = True, you should see very similar outputs.

[gwgknudayanga]

@bamsumit so when i trained the SNN for classification of nmnist setting delay_shift = False, and then i do the inference in loihi i can see the outputs are shifted len(network) - 1, time steps. However, in slayer i got 96% accuracy but in loihi it is dropped to 60%. It should not be , isn't it? . Actually In this case I used mean-only batchnorm for conv layers. And i used the last layer's accumulated neuronal voltage values as the prediction logits(after setting the voltage threshold of neurons to very high value = 2048). And the network is reset every 32 time steps as i trained the network in slayer so that for one inference it will need 31 time steps(and hence in loihi at time step 1,33,65,97... the network is periodically reset ). So why is this big accuracy drop? Do you suggest any tips to debug? This is the network i used for your reference.
neuron_conv_params = {
'threshold' : 1.0,
'current_decay' : 1.0,
'voltage_decay' : 0.03,
'requires_grad' : False,
'persistent_state' : False,
}

neuron_output_layer_params = {
'threshold' : 2048.0,
'current_decay' : 1.0,
'voltage_decay' : 0.03,
'requires_grad' : False,
'persistent_state' : True,
}

neuron_conv_kwargs = {**neuron_conv_params, 'norm': slayer.neuron.norm.MeanOnlyBatchNorm}

quantizer = _quantize_8bit #netx_quantizer #_quantize_8bit

block_kwargs = dict(weight_norm=False, delay_shift=False ,pre_hook_fx=quantizer)
synapse_kwargs = dict(weight_norm=False,pre_hook_fx=quantizer)

net_ladl = nn.Sequential(
slayer.block.cuba.Conv(neuron_conv_kwargs,2,16,kernel_size=3, stride=2, padding=1,**block_kwargs),
slayer.block.cuba.Conv(neuron_conv_kwargs,16,32,kernel_size=3, stride=2, padding=1,**block_kwargs),
slayer.block.cuba.Conv(neuron_conv_kwargs,32,64,kernel_size=3, stride=2, padding=1,**block_kwargs),
slayer.block.cuba.Conv(neuron_conv_kwargs,64,16,kernel_size=3, stride=1, padding=1,**block_kwargs),
slayer.block.cuba.Conv(neuron_output_layer_params,16,num_output,kernel_size=1, stride=1, padding=0,**block_kwargs),
)

net_ladl = net_ladl.to(device)
H, W = (34,34)
N, C, T = 32, 2, 31
input = torch.rand(N, C, H, W, T)
net_ladl(input.to(device=device))

Thanks and Rgds,
Niranjan Udayang

[gwgknudayanga]

@bamsumit

Its not wrong to use delay_shift = False. But it does mean that you should be aware that the final inference in lava to different than what you are seeing in lava-dl. With delay_shift = True, you should see very similar outputs.

** is this mean when delay_shift = False during lava-dl training, the the outputs of the lava network is shifted with T=num_of_blocks in network, but that shifted outputs are similar to the lava-dl outputs, and hence the classification accuracy is almost similar in lava-dl and lava?
** Or otherwise when the delay_shift = False, the classification accuracy of lava and lava-dl are also significantly different?

[gwgknudayanga]

@bamsumit could you please tell me how to dequantize the voltage state that is read from CUBA LIF neurons voltage state ? during training i set threshold of those ouput layer neurons as 2048. but i did not apply any kind of quantization (like quanyize_8bit) for those voltages during training in slayer.

Thanks and Rgds,
Udayanga

[bamsumit]

Please refer to PilotNet SNN inference example to see how the voltage can be decoded during lava inference.

[gwgknudayanga]

@bamsumit yes i did inference referring to that (at there voltage is divided by (num_of_time_steps_per_inference * 32 * 64).
Since I used number_of_time_steps as 32 , i divided the voltage from loihi simulation by (32 * 32 * 64). The output values got during inference in loihi is different from the values got in slayer and hence it shows a accuracy drop from 95% to 60% even though I used the quantize_8bit to make the weights and mean only batchnorm parameters quantized. Do you suggest any debug tips for performance drop?

[bamsumit]

Have you checked your input scaling? That's likely where things could go wrong. Verify the states of the first layer are what you expect compared to lava-dl with the proper scale down similar to the voltage state for the output layer.

[gwgknudayanga]

@bamsumit as the inputs, i used direct binary spikes (ones and zeros). when the fed inputs are identifical for both slayer and loihi.

*In loihi, since, i set the reset interval of the network as 2 and 32 interval , at t=3, t=35 and so on first layer voltages is read as zero.
so at t = 4, i fed input to the network as tensor 'np.ones(34,34,2)'. For that, the min and max of the first layer's conv synapse output is -524.0 930.0. The min and max outputs of the neurons' voltage state are -33536 and 3328

• though the two minimum values of synapse and following neurons voltage states are related with factor 64, the maximum values are not related with factor 64.
• So the problem i can see is, for a given layer, how the voltage state of the neurons and output of the preceding synapses are related?
• Once that relationship is clear , then the synapse output of slayer can be obtained by applying netx.modules.Dequantize(exp=6).
• In the same scenario,In slayer, the min and max values of first layer's conv synapse and subsequent neurons' voltage stages are identical and they are -8.1875 and 14.5312 respectively.
• (i did not use any batchnorm)
• In slayer, i obtained the output as follows since then i can see the intermediate outputs also. I suspect how the voltage states are reset when it generate outputs.
  for block in net_ladl:
  if i == 4:
  input = block.synapse(input)
  current,voltage = block.neuron.dynamics(input)
  output = (voltage > block.neuron.threshold).float()
  else:
  #input = block(input)
  input = block.synapse(input)
  current,voltage = block.neuron.dynamics(input)
  input = (voltage > block.neuron.threshold).float()

[gwgknudayanga]

Have you checked your input scaling? That's likely where things could go wrong. Verify the states of the first layer are what you expect compared to lava-dl with the proper scale down similar to the voltage state for the output layer.

@bamsumit , However in loihi when i decoded as " first dividing the output layer voltage by 64 and then the resultant is dequantized with netx.modules.Dequantize(exp=6)", the max and min values of the output voltage tensor are 11.59375 , -45.5 while in slayer those are 12.2800 and -51.81811. Negetive values shows a significant different. Do in need to quantize the voltages which are read during training in slayer to avoid this or is this normal? (since network is reset every 32 steps with an initial offset of 2, i trained the network in slayer with 31 time bins spike data)