Add support for Neural Networks

[slcott]

It would be cool to support neural nets.

Context of the issue

The library could search different NN designs using different types of layers (fully connected, dropout, convolutions), activation functions (relu, sigmoid, tanh), LSTMs, RNNs etc. There are a lot of options but I think it could be started on something simple like building simple convolutional neural nets for analyzing images. Can also add support for pretrained nets like the caffe model zoo.

This is actually already being explored by other teams, like the researchers working on TensorFlow.
There are several NN libraries: torch, MXNet, Caffe, TensorFlow, and Theano. Theano and TensorFlow are the only ones built for python. Could also build on top of a wrapper like Keras. Or if brave, could code it in straight numpy.

[rhiever]

Great idea @slcott! We've explored this a bit with a summer student on a project called DELFT, but the project has stalled a bit since the student had to go back to school. I agree that it's quite promising to build a TPOT-like tool for designing deep neural networks, though. The primary challenge is that training deep neural networks on any reasonably sized benchmark data set takes a very long time.

[slcott]

Cool!

I've thought a little about the long training time already. It looks like the research community has already been working on possible solutions. There's one that I think that is promising for the slowness issue.

I'm not too familiar with it yet, but there is a technique called transfer learning that uses pre-trained models as a starting point. Caffe has something called a model zoo that offers models.

This is a 2014 paper that explores this topic and it looks like there's more recent papers continuing to explore applications

After my quick literature search it looks like this technique may offer at least a partial solution to the slowness problem, though there were some caveats. Of course to build a system that exploits this would entail users downloading pre-trained models and then having the search algorithm fine tune them (perhaps ambitiously with an option to upload new models, thus offering a growing library of pre-trained models).

[mfeurer]

Hi, don't you rather want something like these:

• https://arxiv.org/abs/1605.07079
• https://scholar.google.com/citations?view_op=view_citation&hl=en&user=WrbJw1wAAAAJ&citation_for_view=WrbJw1wAAAAJ:d1gkVwhDpl0C

?

(Disclaimer: these are my colleagues)

[sashml]

New package in addition, hyperas for tuning keras models - https://github.com/maxpumperla/hyperas

[slavakurilyak]

For inspiration, check out AutoNet by auto-sklearn (Paper, Source Code)

[karan1276]

This looks like a really interesting project. I someone i willing to mentor me i can take it up.

[namisaghaei]

I've worked on something similar, this might be helpful!
https://arxiv.org/abs/1603.06560