Understanding how neural networks work and learn can be quite challenging, to say the least. I find that the best way to learn and truly understand something is to make it from scratch so that is exactly what I set out to do. I decided to use python and numpy to create a dense neural network(including gradient descent and backpropagation) and train it on the mnist dataset. Making one from scratch was challenging but in a good way. It was actually easier than I thought it would be.
create a virtual environment in the project folder
python -m venv venv
activate it
source venv/bin/activate
install numpy
pip install numpy
and run the main python file
python main.py
- load your data as a python list
- create a network
net = network.Network([784, 30, 10])
The list describes how many neurons should each layer have starting from the input layer
- use stochastic gradient descent to train the network
net.SGD(train_data, 30, 20, 0.1, test_data=test_data)
Parameters are training_data, epochs, batch size, learning rate, and test_data
With 30 hidden neurons and 10 output neurons, I was able to get an accuracy of 94.6% on the mnist dataset. Without modern optimizers, the network often gets stuck in a local minimum so this is the best accuracy out of three runs.