Code accompanying article What if Neural Networks had SVDs? accepted for spotlight presentation at NeurIPS 2020.
UPDATE: We currently recommend using the newer fasth++ algorithm from fasthpp.py which only uses PyTorch (no need to compile CUDA code)!
If, for some reason, you want to use our CUDA code, please see this Google Colab.
First, check out how to run the code in Google Colab.
To install locally, run
pip install -r requirements.txt
Check installation by running test cases.
python test_case.py
See test_case.py for expected output.
import torch
from fasth_wrapper import Orthogonal
class LinearSVD(torch.nn.Module):
def __init__(self, d, m=32):
super(LinearSVD, self).__init__()
self.d = d
self.U = Orthogonal(d, m)
self.D = torch.empty(d, 1).uniform_(0.99, 1.01)
self.V = Orthogonal(d, m)
def forward(self, X):
X = self.U(X)
X = self.D * X
X = self.V(X)
return X
bs = 32
d = 512
neuralSVD = LinearSVD(d=d)
neuralSVD.forward(torch.zeros(d, bs).normal_())
If you use this code, please cite
@inproceedings{fasth,
title={What If Neural Networks had SVDs?,
author={Mathiasen, Alexander and Hvilsh{\o}j, Frederik and J{\o}rgensen, Jakob R{\o}dsgaard and Nasery, Anshul and Mottin, Davide},
booktitle={NeurIPS},
year={2020}
}
A previous version of the article was presented at the ICML workshop on Invertible Neural Networks and Normalizing Flows. This does not constitute a dual submission because the workshop does not qualify as an archival peer reviewed venue.