What is the "standard" for DeepEnsembles in the literature, what is randomized (initial network parameter values, training set shuffling, etc)

[beckynevin]

I'm starting this discussion to brainstorm for our DE approach; basically what have past papers done, is there a standard, and is there something beyond the standard that we might be interested in implementing?

In the original DeepEnsembles paper Lakshminarayanan+2017, they do the following:

• Randomize the weights when initializing
• Use adversarial augmentation to smooth the predicted distributions (this involves adding data to the training set)
• Use the entire training set and randomize the batches
• its recommended to use 5ish ensembles
• no bagging
• loss is Gaussian negative log-likelihood
• epistemic uncertainty:

In DeeplyUncertain (aka Caldeira & Nord 2020) the DeepEnsemble has the following properties:

• Randomize the weights
• idk if the training set is randomized
• I don't think y'all did adversarial augmentation?
• 10 ensembles
• 40 epochs
• no bagging
• loss is from Kendall & Gal 2017 - also negative Gaussian LL
• epistemic uncertainty:

[beckynevin]

@bnord I'm thinking it would be interesting to test the randomization fo the training set separately from the randomization of the weights.

[bnord]

definitely.

[beckynevin]

Also, to comment on my above comment. I did the math to show that the uncertainty in Lakshminarayanan+2017 is the same as total uncertainty (aleatoric and epistemic)

[bnord]

which uncertainty in Lakshminarayanan+2017?

[bnord]

oh sorry, I should review your screenshot above for that definition.

[bnord]

I don't think y'all did adversarial augmentation?

Lol, I don't even know what that is, so I doubt we did it.

[bnord]

is this a good reference?
https://openreview.net/forum?id=ByxdUySKvS

[beckynevin]

Not sure. The type of adversarial training I'm referring to is referenced in Lakshminarayanan+2017. It points to "Explaining and harnessing adversarial examples" - https://arxiv.org/abs/1412.6572. I haven't had time to read through it yet.