assignment 5 (RBM)

Assignment 5 (2018-2019)

Restricted Boltzmann Machines (RBMs)

Training methods

• Contrastive Divergence
• Persistent Contrastive Divergence
• Parallel Tempering

Applications

• Feature extraction (leverage the availability of unlabeled data)
• Data generation
• Data compression
• Image restoration

RBMs are most often used as feature extractors.
VAEs and GANs are better generative models than RBMs.

Visualizing weights during the training phase

• The learned features correspond to the weights of RBM hidden units.
• Evolution of the learned features during the training: random -> global -> local.
• The final weights are shown below.

Results

---

---

References

1. Training Restricted Boltzmann Machines using Approximations to the Likelihood Gradient (Tieleman, 2008)
2. Parallel Tempering for Training of Restricted Boltzmann Machines (Desjardins et al., 2010)
3. Lecture12: RBMs (Vineeth N Balasubramanian, 2016)
4. Training Restricted Boltzmann Machines: An Introduction
5. A Practical Guide to Training Restricted Boltzmann Machines (Hinton, 2010)
6. Introduction to RBMs
7. RBM tutorial (deeplearning.net)
8. (Coursera) Lecture 12.3 — Restricted Boltzmann Machines [Neural Networks for Machine Learning, Hinton]
9. (Coursera) Lecture 12.4 — An example of RBM learning [Neural Networks for Machine Learning, Hinton]
10. Neural networks [5.1]: RBM - definition
11. Neural networks [5.2]: RBM - inference
12. Neural networks [5.3]: RBM - free energy
13. Neural networks [5.4]: RBM - contrastive divergence
14. Neural networks [5.5]: RBM - contrastive divergence (parameter update)
15. Neural networks [5.6]: RBM - persistent contrastive divergence
16. Neural networks [5.7]: RBM - example
17. Neural networks [5.8]: RBM - extensions
18. RBMs for beginners