Checkpoints for Neural Networks

[ayambabu23]

A model had been training for 15 days, but a power cut caused it to stop running. I suggest setting up checkpointing for neural networks.

It is an integral part of training large and complex machine learning models to keep a copy of the model after a certain number of iterations. This gives two benefits: the program has a model it could use to recover the progress, and it helps with picking an optimal model.

Neural nets, after a long time, start overfitting on the data they are given. Having checkpoints allows us to recover past models that perform better. Using the performance on the testing and validation dataset, the engineers can determine which is the ideal model to deploy.