README.md

Fastai v1 & PyTorch v1 Course in Vienna

This is the repo for the Fastai v1 & PyTorch v1 Course in Vienna.

• Fast.ai MOOC Details
• Fast.ai MOOC Material (this should be your first address if you are searching for something)
• Fast.ai MOOC - Part 1 Notebooks
• Fastai docs (this should be your second address if you are searching for something)
• fast.ai forum (this should be your third address if you are searching for something)
  • Study group in Austria thread on the Fast.ai forum

Dates

• 04.03.2019 - Stockwerk Coworking - Pater-Schwartz-Gasse 11A, 1150 Wien
• 18.03.2019 - Nic.at - Karlsplatz 1, 1010 Wien
• 01.04.2019 - EBCONT - Millennium Tower, Handelskai 94-96, 1200 Wien
• 15.04.2019 - EBCONT - Millennium Tower, Handelskai 94-96, 1200 Wien
• -- BREAK --
• 13.05.2019 - Nic.at - Karlsplatz 1, 1010 Wien
• 27.05.2019 - Wirtschaftskammer Österreich - Wiedner Hauptstraße 63, 1040 Wien
• 11.06.2019 - Nic.at - Karlsplatz 1, 1010 Wien

--> Sign up for the upcoming fastai PyTorch course v2 in Vienna starting in October! <--

Preparation

Before the course

• Install PyTorch and fastai on your machine (see also fastai developer installation).
  • And/or set up external machine with GPU (see "Server setup") (without GPU the training will take much longer, so this is highly recommended).
• Install a Python IDE, e.g. VS Code, for going through the fastai library in detail.
• Familiarize yourself with Jupyter notebooks, terminal, and the Python debugger (pdb.set_trace(), l, ll, u, n, c, etc.)

During the course

• Fast.ai Lesson Notes A and Notes B
• Basic intro to deep learning
• Python learning resources (for Beginners and advanced)
• Collection of Python tips
• Basic matrix calculus:
  • Matrix multiplication on German Wikipedia (the German version has better visualisations)
  • Animated matrix multiplication
• Broadcasting visualisation

Communication

There are several communication options:

• Fast.ai forums - https://forums.fast.ai/t/study-group-in-austria/26119 (preferred option)
• Slack - either on Keep-Current (#mlcourse) or Vienna Data Science Group
• Facebook groups: Keep-Current or Facebook Developer Circles

Lesson 1 - Intro to fastai and PyTorch

• To dos before the lesson:
  • watch the fastai lesson 1 (hiromis notes lesson 1)
  • run the lesson 1 notebook
• fastai lesson 1 discussion
• Deep learning in a nutshell:
  • data
  • neural network
  • loss function
  • optimizer
• PyTorch intro and building blocks:
  • PyTorch Blitz intro and another intro to PyTorch (tensors and autograd picture)
  • PyTorch workflow: data array -> torch tensor -> torch dataset -> torch dataloader -> network training
  • torch.Tensor & Co. (notebook, based on Part 1 and Part 2, Broadcasting)

Lesson 2 - torch.nn & Co.

• To dos before the lesson:
  • watch the fastai lesson 2 (hiromis notes lesson 2)
  • run the lesson 2 notebook about regression and SGD
  • try the Udacity DL course exercise on PyTorch tensor
  • have a look at the torch.nn tutorial to familiarize yourself with the concepts and to have it easier when we go through it in the meetup
• PyTorch building blocks:
  • (PyTorch Blitz intro)
  • EE-559 – Deep learning 1.6. Tensor internals
  • torch.nn tutorial
  • torch.nn notebook
  • torch.nn docs, incl. nn.Parameter, weights, biases, gradients, etc.
  • torch.nn.functional
  • torch.optim
  • PyTorch and NumPy comparison notebook
• A presentation about Cross entropy loss by Christoph and Pascal
  • Visual information theory

Lesson 3 - Debugging, visualisation, and the fastai workflow

• To dos before the lesson:
  • watch the fastai lesson 3 (hiromis notes lesson 3)
  • run/have a look at the lesson 3 notebook about multi-label prediction with Planet Amazon dataset
  • try the Udacity DL course exercise on Neural networks with PyTorch (solutions)
  • try the Udacity DL course exercise on Neural network weight initialization with PyTorch (solutions)
• PyTorch debugging:
  • CMU DL debugging and visualisation presentation (optional notebook)
  • Debugging Intro Notebook
  • More meaningful error messages on CUDA
• Lesson 3 notebook about multi-label prediction with Planet Amazon dataset
• fastai workflow & building blocks:
  • PyTorch Dataset, PyTorch DataLoader, and fastai DataBunch.
  • DataBlock API docs and DataBlock API sample
  • Looking into the fastai library with your IDE
  • layers

Lesson 4 - NLP

• To dos before the lesson:
  • watch the fastai lesson 4 (hiromis notes lesson 4)
  • Run the lesson 3 imdb notebook
  • Run the lesson 4 Movie recommendation notebook
  • Have a look at the Convolution arithmetic animations and the notebooks from the CNN building blocks section below.
• Convolution Neural Network building blocks:
  • Animation of a Convolution Neural Network at work
  • Convolution arithmetic animations
  • Udacity Notebook - Convolutional Layer
  • Udacity Notebook - CNN Filters
  • Udacity Notebook - Maxpooling Layer
  • CS230 DL: C4M2: Deep Convolutional Models (CNN architectures, 1x1 conv., etc.)
  • CS231n Convolutional Neural Networks for Visual Recognition
• Natural Language Processing:
  • NLP notebook - Building NLP architecture & pipeline
  • Presentation - ABC NLP
  • Writing code for NLP Research
  • NLP code implementations in python - NLP Tutorial
• Learning tips

Lesson 5 - SGD, embeddings & Co.

• To dos before the lesson:
  • watch the fastai lesson 5 (hiromis notes lesson 5)
  • Run the lesson 4 tabular notebook
  • Run the lesson 5 MNIST SGD notebook
  • Run the lesson 7 MNIST ResNet notebook
  • Have a look at the lesson 6 Pets revisited notebook
• Recap basic training & Co.:
  • MNIST SGD notebook
  • An overview of gradient descent optimization algorithms (see "Visualization of algorithms" section for animations)
  • torch.optim docs
  • MNIST ResNet notebook (see Deep Residual Learning for Image Recognition and Bag of Tricks for Image Classification with CNNs)
  • 3D loss surface with and without ResNet blocks (from Visualizing the Loss Landscape of Neural Nets)
  • (Optional: Pets revisited notebook)
• Optional: Understanding SGD, RMSProp, and Adam:
  • Understanding Exponentially Weighted Averages
  • Bias Correction of Exponentially Weighted Averages
  • Gradient Descent With Momentum
  • RMSProp
  • Adam (Momentum + RMSprop)
  • Link to the slides from the videos above
• Learning tips

Lesson 6 - RNN & Co.

• To dos before the lesson:
  • watch the fastai lesson 6 (hiromis notes lesson 6)
  • have a look at the Understanding-LSTMs blog post
  • run the fastai lesson 3 IMDB notebook
• RNN & Co.
  • Understanding-LSTMs
  • nn.RNN & Co. (a simple RNN illustrated)
  • CMU - 11-785 - Recitation 7 - Recurrent Neural Networks
  • PyTorch simple RNN
  • nn.Embedding (see also Entity Embeddings of Categorical Variables)
  • Language model pretraining shown in the Human Numbers notebook from Hiromis notes
  • fastai lesson 7 Human Numbers notebook
• Self-study material:
  • PyTorch Character-Level LSTM
  • PyTorch Sentiment Analysis RNN
  • word2vec
  • Attention notebook 1 & Attention notebook 2 (see also Attention and Augmented Recurrent Neural Networks and CS224n slides)

Lesson 7 - Practical final lesson

• To dos before the lesson:
  • watch the fastai lesson 7 (hiromis notes lesson 7)
  • Prepare your data and your notebooks so we can use the last 2 h as efficiently as possible.
• Datasets ideas:
  • Tabular data:
    • Heart disease classification: https://www.kaggle.com/ronitf/heart-disease-uci
  • Recommendation System:
    • Create an Artificial Sommelier: https://www.kaggle.com/zynicide/wine-reviews
  • Time series:
    • Suicide prediction: https://www.kaggle.com/russellyates88/suicide-rates-overview-1985-to-2016
  • Computer Vision:
    • Malaria infection classification: https://www.kaggle.com/iarunava/cell-images-for-detecting-malaria
    • Dog breed classification: https://www.kaggle.com/jessicali9530/stanford-dogs-dataset
    • Image segmentation - lunar - https://www.kaggle.com/romainpessia/artificial-lunar-rocky-landscape-dataset
    • Fruits classification - https://www.kaggle.com/moltean/fruits
    • Detect the artist, based on the image: https://www.kaggle.com/ikarus777/best-artworks-of-all-time
  • NLP:
    • Sarcasm detection: https://www.kaggle.com/rmisra/news-headlines-dataset-for-sarcasm-detection
    • Open analysis: Predict startup success by media mentions & comments - https://www.kaggle.com/hacker-news/hacker-news
  • GAN:
    • Create realistic images of the moon: https://www.kaggle.com/romainpessia/artificial-lunar-rocky-landscape-dataset
    • Image restoration model (using GAN and fastai 'crapify' idea)
    • Style transfer
• Open lesson for going through practical applications with the fastai library.
• What next, how to keep going, and keep learning!

Stuff that did not fit into our schedule:

• GANs
  • NIPS 2016 Tutorial: Generative Adversarial Networks
  • PyTorch MNIST GAN
  • PyTorch DCGAN
  • fastai WGAN notebook
  • PyTorch Cycle GAN
  • And for the very curious see DFL Wasserstein GAN (contact Michael if you want to tackle this together)

General PyTorch Deep Learning ressources

• PyTorch Tutorials
• PyTorch Cheat Sheet
• PyTorch Docs
• Udacity Deep Learning PyTorch Notebooks
• CMU Deep Learning Course
• CMU Deep Learning Course Recitation Repo
• Deep Lizard PyTorch Tutorials
• EE-559 – EPFL – Deep Learning
• Pytorch torch.einsum (= the best way to get familiar with matrix calculus and einsum)
• PyTorch under the hood
• Advanced PyTorch concepts with code

Deep Learning

• The deep learning book (Ian Goodfellow and Yoshua Bengio and Aaron Courville)
• Neural Networks and Deep Learning (Michael Nielson)
• ML yearning (Andrew Ng) (About how to structure Machine Learning projects.)
• CS 230 Deep Learning Cheatsheets:
  • Convolutional Neural Networks cheatsheet
  • Recurrent Neural Networks cheatsheet
  • Deep Learning Tips and Tricks cheatsheet
• AI Transformation Playbook (Andrew Ng) (A playbook to become a strong AI company.)

Mathematics

• The Matrix Calculus You Need For Deep Learning
• Computational Linear Algebra for Coders
• https://www.3blue1brown.com

Selected publications

• Very Deep Convolutional Networks for Large-Scale Image Recognition (VGG network)
• Deep Residual Learning for Image Recognition (ResNet network)
• Network In Network (1x1 convolutions)
• Going deeper with convolutions (Inception network)
• Everything on https://distill.pub and https://colah.github.io.

Selected newsletters

• Import AI

Possible presentation topics

Present one topic with a general introduction and PyTorch code in a Jupyter notebook in approx. 10-20 min. Feel free to add the notebooks to this repo.

• Weight decay, L1-, and L2 regularization (see weight decay vs. L2 regularization)
• Drop out (see chapter 7.12)
• (fastai) Data augmentation
• CNN (Conv Nets: A Modular Perspective and Understanding Convolutions, Convolution arithmetic animations, and CS231n Convolutional Neural Networks for Visual Recognition), or Advanced CNN Architectures (Advanced Deep Learning for Computer vision - Munich University)
• ResNets & DenseNets (network architecture, PyTorch model code, loss function shape, etc.)
• 1x1 convolutions (Network In Network)
• Batch norm (Udacity Notebook, Batch Normalization, How Does Batch Normalization Help Optimization?, and Group Normalization)
• LSTM unit (Understanding LSTM Networks)
• Attention (Notebook)
• Cross entropy loss (based on this introduction and information theory).
• Mixed precision training, floating point arithmetics, and the fastai callback.

Possible projects

• Tensorboard visualisation with fastai callback using TensorboardX, including 2D visualisations for CNNs (see starter notebook and fastai forum thread)
• Histopathologic Cancer Detection on Kaggle

Deployment

• fast.ai deployment options
• Pythonanywhere - free to start
• Render - free for static (HTML) sites, 5$ for python hosting
• Heroku - free basic cloud account for up to 3 projects

Learning tips

• 10 Top Ideas to Help Your Learning & 10 Pitfalls to Avoid in Your Learning (from the Learning how to learn course)
• Use spaced repetition to memorize important concepts, APIs, and everything else:
  • Short intro to the spacing effect
  • (More detailed) Introduction to augmenting Long-term Memory (concept outlined for a mathematic example)
  • Spaced repitition in detail
  • Anki spaced repitition flashcard app
• Remember, a little bit everyday makes the difference!