Variational parameter noise

[albertz]

E.g. Lingvo has this, and they say it's important for bigger models or smaller datasets.

We also have it in RETURNN as param_variational_noise option for a layer.
However, here it makes sense to reimplement it.

Related: Weight dropout (#100), weight norm (#91)

The implementation in RETURNN:

      # Only apply this if we get a variable. Otherwise, maybe variational noise was already applied
      # (by some parent var scope), and we don't want to apply it twice.
      if param_variational_noise and param.dtype.is_floating and isinstance(param, tf.Variable):
        with default_control_flow_ctx():  # make independent from loop/cond
          with reuse_name_scope_of_tensor(param, postfix="_variational_noise", add_tensor_name=True):
            param = self.network.cond_on_train(
              fn_train=lambda: param + tf_compat.v1.random_normal(
                tf.shape(param), dtype=param.dtype.base_dtype,
                stddev=param_variational_noise,
                seed=self.network.random.randint(2 ** 31)),
              fn_eval=lambda: param)

In Lingvo:
https://github.com/tensorflow/lingvo/blob/65699192ba14521f330a1cae16141433453f1cbf/lingvo/tasks/asr/params/librispeech.py#L139
https://github.com/tensorflow/lingvo/blob/65699192ba14521f330a1cae16141433453f1cbf/lingvo/core/py_utils.py#L3691

[albertz]

This is implemented now. See nn.variational_weight_noise.