release library

.gitignore

@@ -0,0 +1,5 @@
+
+
+.idea
+
+profiling/

LICENSE

@@ -0,0 +1,34 @@
+
+Copyright (c) 2019 Qualcomm Innovation Center, Inc. 
+All rights reserved. 
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted (subject to the limitations in the 
+disclaimer below) provided that the following conditions are met: 
+
+* Redistributions of source code must retain the above copyright 
+notice, this list of conditions and the following disclaimer.
+
+* Redistributions in binary form must reproduce the above copyright 
+notice, this list of conditions and the following disclaimer in the
+documentation and/or other materials provided with the distribution. 
+
+* Neither the name of Qualcomm Innovation Center, Inc. nor the names of its
+contributors may be used to endorse or promote products derived from
+this software without specific prior written permission. 
+
+NO EXPRESS OR IMPLIED LICENSES TO ANY PARTY'S PATENT RIGHTS ARE
+GRANTED BY THIS LICENSE. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT
+HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED
+WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
+MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
+BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
+WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE
+OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN
+IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 
+
+

README.md

@@ -1 +1,184 @@
-# Coming Soon!
+
+General E(2)-Equivariant Steerable CNNs
+--------------------------------------------------------------------------------
+**[Documentation](https://documentation.coming.soon)** | **[Paper](https://arxiv.org/abs/1911.08251)**
+
+*e2cnn* is a [PyTorch](https://pytorch.org/) extension for equivariant deep learning.
+
+*Equivariant neural networks* guarantee a specified transformation behavior of their feature spaces under transformations of their input.
+For instance, classical convolutional neural networks (*CNN*s) are by design equivariant to translations of their input.
+This means that a translation of an image leads to a corresponding translation of the network's feature maps.
+This package provides implementations of neural network modules which are equivariant under all *isometries* E(2) of the image plane 
+![my equation](https://chart.apis.google.com/chart?cht=tx&chs=19&chl=\mathbb{R}^2)
+, that is, under *translations*, *rotations* and *reflections*.
+In contrast to conventional CNNs, E(2)-equivariant models are guaranteed to generalize over such transformations, and are therefore more data efficient.
+
+The feature spaces of E(2)-Equivariant Steerable CNNs are defined as spaces of *feature fields*, being characterized by their transformation law under rotations and reflections.
+Typical examples are scalar fields (e.g. gray-scale images or temperature fields) or vector fields (e.g. optical flow or electromagnetic fields).
+
+![feature field examples](./visualizations/feature_fields.png)
+
+Instead of a number of channels, the user has to specify the field *types* and their *multiplicities* in order to define a feature space.
+Given a specified input- and output feature space, our ``R2conv`` module instantiates the *most general* convolutional mapping between them.
+Our library provides many other equivariant operations to process feature fields, including nonlinearities, mappings to produce invariant features, batch normalization and dropout.
+Feature fields are represented by ``GeometricTensor`` objects, which wrap a ``torch.Tensor`` with the corresponding transformation law.
+All equivariant operations perform a dynamic type-checking in order to guarantee a geometrically sound processing of the feature fields.
+
+E(2)-Equivariant Steerable CNNs unify and generalize a wide range of isometry equivariant CNNs in one single framework.
+Examples include:
+- [Group Equivariant Convolutional Networks](https://arxiv.org/abs/1602.07576)
+- [Harmonic Networks: Deep Translation and Rotation Equivariance](https://arxiv.org/abs/1612.04642)
+- [Steerable CNNs](https://arxiv.org/abs/1612.08498)
+- [Rotation equivariant vector field networks](https://arxiv.org/abs/1612.09346)
+- [Learning Steerable Filters for Rotation Equivariant CNNs](https://arxiv.org/abs/1711.07289)
+- [HexaConv](https://arxiv.org/abs/1803.02108)
+- [Roto-Translation Covariant Convolutional Networks for Medical Image Analysis](https://arxiv.org/abs/1804.03393)
+
+For more details we refer to our NeurIPS 2019 paper [General E(2)-Equivariant Steerable CNNs](https://arxiv.org/abs/1911.08251).
+
+--------------------------------------------------------------------------------
+
+The library is structured into four subpackages with different high-level features:
+
+| Component                                        | Description                                                      |
+| ------------------------------------------------ | ---------------------------------------------------------------- |
+| [**e2cnn.group**](./e2cnn/group/)                | implements basic concepts of *group* and *representation* theory |
+| [**e2cnn.kernels**](./e2cnn/kernels/)            | solves for spaces of equivariant convolution kernels             |
+| [**e2cnn.gspaces**](e2cnn/gspaces/)              | defines the image plane and its symmetries                       |
+| [**e2cnn.nn**](./e2cnn/nn/)                      | contains equivariant modules to build deep neural networks       |
+
+--------------------------------------------------------------------------------
+
+## Demo
+
+Since E(2)-steerable CNNs are equivariant under rotations and reflections, their inference is independent from the choice of image orientation.
+The visualization below demonstrates this claim by feeding rotated images into a randomly initialized E(2)-steerable CNN (left).
+The middle plot shows the equivariant transformation of a feature space, consisting of one scalar field (color-coded) and one vector field (arrows), after a few layers.
+In the right plot we transform the feature space into a comoving reference frame by rotating the response fields back (stabilized view).
+
+![Equivariant CNN output](visualizations/vectorfield.gif)
+
+The invariance of the features in the comoving frame validates the rotational equivariance of E(2)-steerable CNNs empirically.
+Note that the fluctuations of responses are discretization artifacts due to the sampling of the image on a pixel grid, which does not allow for exact continuous rotations.
+<!-- Note that the fluctuations of responses are due to discretization artifacts coming from the  -->
+
+For comparison, we show a feature map response of a conventional CNN for different image orientations below.
+
+![Conventional CNN output](visualizations/conventional_cnn.gif)
+
+Since conventional CNNs are not equivariant under rotations, the response varies randomly with the image orientation.
+This prevents CNNs from automatically generalizing learned patterns between different reference frames.
+
+
+## Experimental results
+
+E(2)-steerable convolutions can be used as a drop in replacement for the conventional convolutions used in CNNs.
+Keeping the same training setup and *without performing hyperparameter tuning*, this leads to significant performance boosts compared to CNN baselines (values are test errors in percent):
+
+ model        | CIFAR-10                | CIFAR-100                | STL-10             |
+ ------------ | ----------------------- | ------------------------ | ------------------ |
+ CNN baseline | 2.6 &nbsp; ± 0.1 &nbsp; | 17.1 &nbsp; ± 0.3 &nbsp; |       12.74 ± 0.23 |
+ E(2)-CNN *   | 2.39       ± 0.11       | 15.55       ± 0.13       |       10.57 ± 0.70 |
+ E(2)-CNN     | 2.05       ± 0.03       | 14.30       ± 0.09       | &nbsp; 9.80 ± 0.40 |
+
+The models without * are for a fair comparison designed such that the number of parameters of the baseline is approximately preserved while models with * preserve the number of channels, and hence compute.
+For more details we refer to our [paper](https://arxiv.org/abs/1911.08251).
+
+## Getting Started
+
+*e2cnn* is easy to use since it provides a high level user interface which abstracts most intricacies of group and representation theory away.
+The following code snippet shows how to perform an equivariant convolution from an RGB-image to 10 *regular* feature fields (corresponding to a
+[group convolution](https://arxiv.org/abs/1602.07576)).
+
+```python3
+ 1|  from e2cnn import gspaces
+ 2|  from e2cnn import nn
+ 3|  import torch
+ 4|
+ 5|  r2_act = gspaces.Rot2dOnR2(N=8)
+ 6|  feat_type_in  = nn.FieldType(r2_act,  3*[r2_act.trivial_repr])
+ 7|  feat_type_out = nn.FieldType(r2_act, 10*[r2_act.regular_repr])
+ 8|
+ 9|  conv = nn.R2Conv(feat_type_in, feat_type_out, kernel_size=5)
+10|  relu = nn.ReLU(feat_type_out)
+11|
+12|  x = torch.randn(16, 3, 32, 32)
+13|  x = nn.GeometricTensor(x, feat_type_in)
+14|
+15|  y = relu(conv(x))
+```
+
+Line 5 specifies the symmetry group action on the image plane
+![my equation](https://chart.apis.google.com/chart?cht=tx&chs=19&chl=\mathbb{R}^2)
+under which the network should be equivariant.
+We choose the 
+[*cyclic group*](https://en.wikipedia.org/wiki/Cyclic_group)
+ C<sub>8</sub>, which describes discrete rotations by multiples of 2π/8.
+Line 6 specifies the input feature field types.
+The three color channels of an RGB image are thereby to be identified as three independent scalar fields, which transform under the
+[*trivial representation*](https://en.wikipedia.org/wiki/Trivial_representation)
+ of C<sub>8</sub>.
+Similarly, the output feature space is in line 7 specified to consist of 10 feature fields which transform under the
+[*regular representation*](https://en.wikipedia.org/wiki/Regular_representation)
+of C<sub>8</sub>.
+The C<sub>8</sub>-equivariant convolution is then instantiated by passing the input and output type as well as the kernel size to the constructor (line 9).
+Line 10 instantiates an equivariant ReLU nonlinearity which will operate on the output field and is therefore passed the output field type.
+
+Lines 12 and 13 generate a random minibatch of RGB images and wrap them into a `nn.GeometricTensor` to associate them with their correct field type.
+The equivariant modules process the geometric tensor in line 15.
+Each module is thereby checking whether the geometric tensor passed to them satisfies the expected transformation law.
+
+A hands-on tutorial, introducing the basic functionality of *e2cnn*, is provided in [introduction.ipynb](./examples/introduction.ipynb).
+Code for training and evaluating a simple model on the [*rotated MNIST*](https://sites.google.com/a/lisa.iro.umontreal.ca/public_static_twiki/variations-on-the-mnist-digits) dataset is given in [model.ipynb](./examples/model.ipynb).
+
+More complex equivariant *Wide Resnet* models are implemented in [e2wrn.py](./examples/e2wrn.py).
+To try a model which is equivariant under reflections call:
+```
+cd examples
+python e2wrn.py
+```
+A version of the same model which is simultaneously equivariant under reflections and rotations of angles multiple of 90 degrees can be run via:
+```
+python e2wrn.py --rot90
+```
+
+
+## Dependencies
+```
+torch>=1.1
+numpy
+scipy
+```
+Optional:
+```
+pymanopt
+autograd
+```
+
+## Installation
+
+```
+pip install .
+```
+
+
+## Cite
+
+The development of this library was part of the work done for our paper
+[General E(2)-Equivariant Steerable CNNs](https://arxiv.org/abs/1911.08251). 
+Please cite this work if you use our code:
+
+```
+@inproceedings{e2cnn,
+    title={{General E(2)-Equivariant Steerable CNNs}},
+    author={Weiler, Maurice and Cesa, Gabriele},
+    booktitle={Conference on Neural Information Processing Systems (NeurIPS)},
+    year={2019},
+}
+```
+
+Feel free to [contact us](mailto:[email protected],[email protected]).
+
+## License
+
+*e2cnn* is distributed under BSD Clear license. See LICENSE file.

docs/.gitignore

@@ -0,0 +1 @@
+build/doctrees

docs/Makefile

@@ -0,0 +1,30 @@
+
+
+# You can set these variables from the command line.
+SPHINXOPTS    =
+SPHINXBUILD   = sphinx-build
+SPHINXPROJ    = e2cnn
+SOURCEDIR     = source
+BUILDDIR      = build
+
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+api:
+	sphinx-apidoc -fMET -d 1 ../e2cnn -o source/api
+
+github:
+	@make html
+	@cp -a build/html/. ../docs
+
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+

docs/build/html/.buildinfo

@@ -0,0 +1,4 @@
+# Sphinx build info version 1
+# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
+config: a2a4ac7732dd5e58df878ed167fc45b7
+tags: 645f666f9bcd5a90fca523b33c5a78b7