10/23 Conundrum conversation with Brian

[beckynevin]

Notes from the conversation and the conundrum are here - donut slides

The just of the conundrums is how to design a self-consistent dataset and experiments that can inject error in a way that makes sense and then use propagation to retrieve a ground truth prediction of error on a measured quantity that we can then compare to the expectation from the models.

The main confusion at present is:

• what is the difference between inference and prediction? Do NNs still do inference internally with latent parameters (z)? If we have a NN that predicts a parameter is this fundamentally misaligned with inference that infers p(theta|X)?
• parameters versus random variables - is there a cannon for neural networks on this?
• error injection - do we inject error on the network output (y) directly? The network input (x) and then propagate to y? If we inject error on the parameters (theta) and propagate to y this doesn't really make sense if the parameters are the output of inference methods like Bayesian sampling inference.

At current, we're leaning towards not injecting error on the thetas because they are the inference target so this seems a little circular. Need to talk to @Jasonpoh about posterior predictive checks and what they are doing statistically - are they considered to be getting two for the price of one? Are they a valid way for us to do an error comparison with our expectation of error on y?