Bayesiains Reading Group

Time and place: 2–3:30pm on roughly alternate Mondays, IF 1.16.

Rota

Date	Person	Paper(s)
20 Jan	James	Fast and Accurate Least-Mean-Squares Solvers
3 Feb	Iain	HNSW for approximate nearest neighbours
17 Feb	Tiffany	Partitioned integrators for thermodynamic parameterization of neural networks
9 Mar	Conor	MC Gradient Estimation in ML (Sections 4, 5, 7), Concrete Distribution & Gumbel-Softmax (reparameterization trick for discrete variables), REBAR (control variates for score function estimator using concrete/GS)
31 Mar	Artur	Contrastive methods for representation learning. A Simple Framework for Contrastive Learning of Visual Representations, Representation Learning with Contrastive Predictive Coding, Data-Efficient Image Recognition with Contrastive Predictive Coding
15 Apr	Asa	Calibration under dataset shift Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift
29 Apr	James	SUMO: Unbiased Estimation of Log Marginal Probability for Latent Variable Models
4 May
18 May
1 June
15 June
29 June

Topics/papers

Reset the list for 2020. Brainstorm of ideas, not a schedule:

Approximate Bayesian inference, calibration, ...

• ...

Transfer learning, self-supervision, and friends

• ...

Similarity and nearest neighbours

• HNSW for approximate nearest neighbours does well in some benchmarks. (Although FAISS offers some discussion of the choices for different regimes.)

• nmslib implements HNSW, and has a reading list.

• Any NN-graph methods? Classics like isomap and LLE. Discuss how nearest neighbours can fit into more recent methods?

Optimization

Anything that helps making practical choices, or gives insight into what choices matter?

Batch size:

• Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model

• Measuring the Effects of Data Parallelism on Neural Network Training

Learning rates:

• ...

Discrete models

(Not continuous-like data, such as quantized images.)

Fast generation of high-dim patterns?

• Non-Autoregressive Neural Machine Translation
• Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posterior
• RBMs?

Methods for latent variable models. Concrete distribution, REBAR/RELAX...?

Unbiased estimation in learning

• More on exact estimation with MCMC?

• Unbiased scalable softmax optimization (although critical review

• Efficient Optimization of Loops and Limits with Randomized Telescoping Sums

• ...?

Reinforcement learning?

• ...?

Boosted trees

• A look at what's inside widely-used tools like XGBoost, LightGBM, or CatBoost?

Embedding different data types

• Graph Neural Networks?