SNN (Self-Normalizing Networks)

[mdraw]

https://arxiv.org/abs/1706.02515 proposes a new activation function SELU, which can replace explicit batch normalization and could improve overall performance and accuracy of our neural networks.
To do:

• Implement SELU activation function
• Implement special "alpha dropout" for SNNs because regular dropout is broken with SELU (see p. 6 of paper). I think we can do this transparently by automatically switching to the alpha dropout algorithm in Perceptron constructors if they detect activation_func='selu'
• Weight initialisation method suggested at the bottom of p.3 of the paper (→ elektronn2.neuromancer.variables.initweights())

Links:

• General discussion
• Practical discussion of issues with an SNN U-Net variant
• https://github.com/bioinf-jku/SNNs
• add new Activation function: selu keras-team/keras#6969