In this project, I attempt to implement deep learning algorithms from scratch. The purpose of this is to make sure I understand the theory behind deep learning. And personally, it's very rewarding to build things from the ground up.
The structure of the library is inspired by PyTorch and Keras. Although I tried to use vectorization wherever possible, my code is not the most efficient in the world (surprise!) but it still gets the job done (at least if you're training on MNIST). Also, there's no GPU support or use of threading, at least for the moment.
Here's an example of how to create and train a model:
from models import Sequential
from layers import Linear, ReLU, Softmax
from losses import CategoricalCrossentropy
from metrics import categorical_accuracy
from optimizers import SGD
# create the model
model = Sequential()
model.add(Linear(128, input_shape=(784,)))
model.add(ReLU())
model.add(Linear(64))
model.add(ReLU())
model.add(Linear(10))
model.add(Softmax())
# configure the model for training
model.configure(loss=CategoricalCrossentropy(),
optimizer=SGD(learning_rate=0.01),
metrics={"acc": categorical_accuracy})
# train the model
history = model.train(X_train, y_train, X_test, y_test, epochs=100)
Epoch 01 loss=1.083168 val_loss=1.117989 acc=0.753333 val_acc=0.720000
...
Epoch 100 loss=0.017159 val_loss=0.237937 acc=0.999000 val_acc=0.939000
You can find more examples in the demos folder.
- Sequential model
- Layers:
- Linear (fully connected) layer
- ReLU
- Tanh
- Sigmoid
- Softmax
- Dropout
- Batch Normalization
- Loss Functions:
- Mean Squared Error (MSE)
- Binary cross-entropy
- Categorical cross-entropy
- Metrics:
- Binary Accuracy
- Categorical Accuracy
- Regularizers:
- L1
- L2
- L1-L2
- Initializers:
- Random Normal
- He Normal
- Optimizers:
- SGD
- SGD with Momentum
- RMSprop
- Adam
The only real dependency is NumPy. However, I use other libraries such as Scikit-Learn in the demos because they are convenient. You can view or install all the dependencies using the requirements.txt file. For example in pip:
pip install -r requirements.txt
MIT License, check the LICENSE file.