lecture_05_07_2022_natural_language_processing.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Natural Language Processing?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/nlp1/drake_reaction_nlp.jpg\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Many times there is a confusion with the \"NLP\" abbreviation. Neural Linguistic Programming is also well known scientific field connected with human communication and body language. But anyway it doesn't have any connection with Natural Language Processing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# What is Natural Language Processing?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![title](https://mildaintrainings.com/wp-content/uploads/2019/09/natural-language-processing-training.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "[Natural Language Processing](https://en.wikipedia.org/wiki/Natural_language_processing)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Natural Language Processing actually started very early in 1950s. Alan Turing published an article titled \"Computing Machinery and Intelligence\" which proposed what is now called the Turing test as a criterion of intelligence. But it took long to achieve some promising results. But Turing Test is still remaining unbeatable by AI models and hence there is a huge space for improvement :)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# How does it look like?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"https://2s7gjr373w3x22jf92z99mgm5w-wpengine.netdna-ssl.com/wp-content/uploads/2014/06/text-analytics.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "The main data source for natural language processing is text. Text is unstructured information which needs to be structured (classified, extracted some data, etc) and most of the NLP problems are trying to do that. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"https://machinelearning.technicacuriosa.com/wp-content/uploads/sites/7/2019/03/Speech_Recognition.jpeg\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Another data source for NLP is sound which is again unstructured data and is being converted into textual format. It's actually called Speech recognition. Recorded sound is transformed into text (or vice versa). Finally text is further processed to get get useful insights from it (user commands for google assistant, controlling devices, etc)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Applications"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"https://i.pinimg.com/originals/45/da/2d/45da2d4d7333a5ccb9598cf08b61cd80.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Here are the wide range of NLP applications and but incomplete (you may find more). Important applications are:\n",
    "- **Information retrieval**: searching through huge dataset and retrieving relevant information (search engines)\n",
    "- **Keyword Extraction**: extracting important words from text which can be used for indexing documents (search engines)\n",
    "- **Text Summarization**: to get short summary of huge text content. Used in Google search results and Google assistant uses it for answering non-abstract questions (extractive and abstractive text summarizations are present)\n",
    "- **Named entity recognition**: recognizing entities like Person, Company, Date, Money, Email, Phone, etc.\n",
    "- **Sentiment Analysis**: analyzing sentiment of text. (Used in social media analysis, comments on products, etc)\n",
    "- **Text Similarity**: good text representation is the best way to make NLP work actually, it's more fundamental problem.\n",
    "- **Machine Translation**: translation is one of the most popular and important feature of NLP. \n",
    "- **Question Answering**: useful in some cases, can be widely used in chatbot like systems for answering simple (and sometimes hard) questions.\n",
    "- **Spell Checking**: very useful in text editors for correcting spelling mistakes and also detect typos.  \n",
    "- etc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![title](https://cdn.setapp.com/blog/images/prizmo-text-from-image.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Language models are used for OCR (Optical character recognition) for checking typos created by OCR errors (incorrect prediction of character because of noise). Having language model means to have understanding the language generally. Simple example to understand what it means is to consider text generation. Given some sentences, model have to generate next sentences as good as person could do/imagine."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "The best OCR comes from Google. It has multilingual and handwritten text recognition capabilities on a very low price per page of document.\n",
    "[OCR APIs](https://source.opennews.org/articles/so-many-ocr-options/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![title](https://thumbs.gfycat.com/ImperfectDarkKarakul-size_restricted.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Google assistant is another good example of real life NLP application. User's speech is converted into text, then text is processed to get insights/commands from it, then model generates answer (search result, command, etc) and that answer is converted back to speech to for good user experience (play music, ask something, ask about weather, etc).  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "[chatbots, assistants, etc](https://medium.com/@takashimokobe/the-evolution-of-conversation-design-chatbots-voice-user-interfaces-and-google-duplex-815d7bee2233)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![title](https://prishtinainsight.com/wp-content/uploads/2017/12/Google-Translate-GIF-highres.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Google translate is also the best translation engine. It converts text from one language into another. It's superior performance is achieved by huge amount of static (wikipedia contents in different language) and dymanic (user actions, searches, etc) contents available in Google. In order to achieve good translation you need to have huge amount of sentence pairs in two different languages. As new terms and words are created Google automatically updates it's engine to be able to track them and to be consistent with translations.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "[best machine translations for 2019](https://analyticsprofile.com/machine-learning/best-language-translation-apis-available-2019/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![title](https://media1.giphy.com/media/bdtCfNg4RK8oM/giphy.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "SEO (Search Engine Optimization) is very popular and important structural part for building websites in an optimized way. The idea is very simple. New website has to be easily reachable and must appear on the first page when user searches in Google. In order to achieve that you need to know important keywords and terms that are used for searching relevant information in search engine. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "[google semantic search ans SEO](https://blog.alexa.com/semantic-search/)\n",
    "\n",
    "[semantic search engine explained](https://www.xenonstack.com/blog/semantic-search-engine/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![title](https://hips.hearstapps.com/digitalspyuk.cdnds.net/16/32/1470838492-donald-trump-gif.gif?resize=480:*)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "[Sentiment Analysis](https://monkeylearn.com/sentiment-analysis/) is actually very useful \"feature\" of NLP. New state of the art Deep learning models are very good at classifying sentiments. Social media is full of comments, posts, actions, etc. It's very important to analyze client reactions, find problems with your product/service and optimize it for maximum profit.  \n",
    "\n",
    "Recently it was amazingly used for Trump Elections. See more [here](https://aisel.aisnet.org/pacis2017/48/)\n",
    "\n",
    "\n",
    "There is very interesting extension of sentiment analysis: Aspect Based Sentiment Analysis. Using that approach it's possible to detect sentiments in more granular way which is very important. see more [here](https://monkeylearn.com/blog/aspect-based-sentiment-analysis/) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Short History"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Here are some highlights from history of NLP. (You may find much more :), History is my weakness :D )\n",
    "\n",
    "- [chomsky-syntactic-structures (1957)](https://doubleoperative.files.wordpress.com/2009/12/chomsky-syntactic-structures-2ed.pdf)\n",
    "\n",
    "- [rule based systems (until 1980)]()\n",
    "\n",
    "- [machine learning approaches (like decision trees) replaced handwritten rules (after 1980)]()\n",
    "\n",
    "- [n-grams]()\n",
    "\n",
    "- [recurrent networks (started in 1997 and become successful in 2007)]()\n",
    "\n",
    "- [feed forward network by yoshua bengio (2001)]()\n",
    "\n",
    "- [siri: first successful voice assistant 2011]()\n",
    "\n",
    "\n",
    "References taken from [here](https://www.dataversity.net/a-brief-history-of-natural-language-processing-nlp/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Datasets & Tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "[squad 2.0](https://rajpurkar.github.io/SQuAD-explorer/explore/v2.0/dev/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "[Yelp Reviews](https://www.yelp.com/dataset)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "[WordNet](https://wordnet.princeton.edu/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "[LibriSpeech](http://www.openslr.org/12/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "[Spam](http://www.dt.fee.unicamp.br/~tiago/smsspamcollection/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "[NLP Progress](http://nlpprogress.com/) is very good source for getting an idea of nlp tasks and datasets.\n",
    "\n",
    "[top 25 datasets](https://lionbridge.ai/datasets/the-best-25-datasets-for-natural-language-processing/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Part-of-speech Tagging (POS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"https://blog.aaronccwong.com/assets/images/bigram-hmm/pos-title.jpg\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Dependency Parsing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"https://achyutjoshi.github.io/aspect_extraction/assets/images/dependency.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Named Entity Recognition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"https://miro.medium.com/max/1000/1*0BuSAIQOLNQGVWJIxZhFuA.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Representation Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"https://mymightyvoice.com/wp-content/uploads/2020/02/what-does-it-mean-to-know-a-word_-1024x1024.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Learning text representations is a goal of natural language processing. Representation learning techniques are improving through time. The idea is to have numerical representation of words and that representation should encode it's meaning. You see the questions above that could arise while trying to explain/describe the meaning of word.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Wordnets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"https://www.researchgate.net/profile/Jaap_Kamps/publication/228729013/figure/fig4/AS:301958960304141@1449004034445/The-WordNet-database-from-the-vista-point-of-verb-be-and-maximal-MPL-of-3-and-polysemy.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "It's the most intuitive representation of words. It's an intuitive try to encode similarity of words. But there are many problems with using WordNet. Similarities of words aren't fine-grained. Two neighbours of the same word can be considered equally similar. If you want to measure similarity of between two terms you may calculate graph-wise distance between them but it's just natural number not float. Similarity between words can be different in different contexts and that can cause errors in analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "[wordnet](https://moz.com/blog/what-can-wordnet-be-used-for?fbclid=IwAR0OfJ_nD8RxX0Je1i5myp2zCksTdbngrt98Zmrsl0SUTD3jsDueIb7phv8)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# One-Hot Encoded Vectors "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"https://miro.medium.com/max/674/1*YEJf9BQQh0ma1ECs6x_7yQ.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "There is no chance to use vectors for comparing different words (calculate similarity) since vectors are orthogonal. Also we have 0 or 1 values instead of counting actual number of occurrence of each word inside document. It's kind of a Bag of Word (BOW) representation which doesn't preserve order of words hence loosing context.\n",
    "\n",
    "BOW representation still works fine in information retrieval cases when you represent document by keywords only. Loosing context isn't a big problem here since speed and simplicity of model is more important."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Better Text Representations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"https://media.makeameme.org/created/wow-come-on.jpg\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# TF-IDF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"https://miro.medium.com/max/2514/1*vskqlnEhoNooBTpkoFobYg.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"https://miro.medium.com/max/3604/0*56JnM18OAx1lhIQb.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "[tf-idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf#:~:text=In%20information%20retrieval%2C%20tf%E2%80%93idf,in%20a%20collection%20or%20corpus.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Word2Vec (Mikolov et al. 2013)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**Word2vec is a framework for learning word vectors.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- We have a large corpus of text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Every word in a fixed vocabulary is represented by a vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Go through each position t in the text, which has a center word c and context (“outside”) words o"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Use the similarity of the word vectors for c and o to calculate the probability of o given c (or vice versa)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Keep adjusting the word vectors to maximize this probability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"https://cdn-images-1.medium.com/max/1000/1*628Ck-nQpr2WP3G3KxQKNg.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"https://www.researchgate.net/profile/Wang_Ling/publication/281812760/figure/fig1/AS:613966665486361@1523392468791/Illustration-of-the-Skip-gram-and-Continuous-Bag-of-Word-CBOW-models.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"https://i.stack.imgur.com/nz4CI.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"https://sangminwoo.github.io/img/cs224n/lec1/3.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"https://aegis4048.github.io/images/featured_images/negative_sampling.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"https://miro.medium.com/max/1200/1*2r1yj0zPAuaSGZeQfG6Wtw.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/add1/wv_2.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/add1/wv_3.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/add1/wv_4.png\" height=\"800\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "[word2vec explained](https://israelg99.github.io/2017-03-23-Word2Vec-Explained/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Glove (Improvement over word2vec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"https://www.erogol.com/wp-content/uploads/2016/08/cooccurance_matrix.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Idea of improvement word2vec performance is to use global information of word co occurences (context). Since word2vec is trained on only local context information it has bad performance on small datasets (the smaller is the dataset the lower is the chance to approximate global context using many local context information). \n",
    "\n",
    "Co-occurence matrix X has counts of occuring i and j words from vocabulary in content through full dataset and hence it contains global information of word co-occurences of word pairs. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"https://theworldofdatascience.files.wordpress.com/2017/04/word2vec-vs-svd.png?w=1108\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "We didn't much consider many approaches of vectorization therefore it's little bit hard to explain what is shown on table above. Simply, I would say that we have two approaches: local and global context. Local context is something similar to word2vec approach which directly is trained for doing predictions of local context. Global context is more like co-occurence matrix analysis which generally uses global information to compare words. \n",
    "Local approach is slower to train and doesn't that much use of global statistics but it better encodes the information of complex local context patterns. In contrast, global approach is faster to train, efficiently uses global statistics about words and their similarity but lacks of local context information and doesn't perform well in complex scenarios since it knows more about words in general but not in concrete cases."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"https://sangminwoo.github.io/img/cs224n/lec2/20.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Idea of GloVe algorithm is to use both global and local context information effectively. It tries to encode similarity of words by their vector multiple. It has more specific details and would be good to consider it in cs224N course to understand the intuition better."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"https://adriancolyer.files.wordpress.com/2016/04/glove-vs-word2vec.png?w=480\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Here we see that GloVe outperforms both word2vec models and trains faster. It's because it uses global contextual information all the time plus encodes word similarity with their scalar multiple."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "[glove explained](https://towardsdatascience.com/light-on-math-ml-intuitive-guide-to-understanding-glove-embeddings-b13b4f19c010)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# PCA & Vector Space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"https://ruder.io/content/images/size/w2000/2016/09/merge_from_ofoct--2-.jpg\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "We explore vector space of word embeddings. It's actually very intuitive and encodes conceptual information like word analogies, relationship between similar word pairs, clusters similar words, etc. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Weaknesses of Non-Contextual Embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Embedding of the word is static"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Averages all contexual embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Words with different meanings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- Fixed vocabulary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- etc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Standard Libraries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"https://spacy.io/static/social_default-1d3b50b1eba4c2b06244425ff0c49570.jpg\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"https://data-flair.training/blogs/wp-content/uploads/sites/2/2018/08/NLTK-NLP-with-Python.jpg\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"https://pbs.twimg.com/profile_images/1232743999837888512/QI2OmtDZ.jpg\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"https://miro.medium.com/max/4000/0*Hm61mNBHsmocinMA.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Georgian NLP ?!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"https://memegenerator.net/img/instances/80371197.jpg\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Conclusions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "There are many libraries for doing very interesting stuff in NLP. Classical algorithms are useful in many cases  (recognizing entities, analyzing sentiments, searching similar content, extracting keywords for indexing documents, correct typos, classify texts, text similarity, etc) but Deep Learning models have more promising results that we will see in the next lecture."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Useful Links"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "[spacy advanced course](https://course.spacy.io/en/)\n",
    "\n",
    "[NLP Papers with code](https://paperswithcode.com/area/natural-language-processing)\n",
    "\n",
    "[mit nlp](https://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-864-advanced-natural-language-processing-fall-2005/)\n",
    "\n",
    "[10 free nlp datasets](https://analyticsindiamag.com/10-nlp-open-source-datasets-to-start-your-first-nlp-project/)\n",
    "\n",
    "[spacy sample projects](https://github.com/explosion/projects)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "[cs224N course materials](http://web.stanford.edu/class/cs224n/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Deep Learning for NLP & Sequence Modelling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<br /><br />\n",
    "\n",
    "### Motivation\n",
    "\n",
    "In this class we are going to cover popular Deep Learning models for **sequential data**, i.e. where each data point $X_i$ represents a sequence of some tokens $X_i={s_0, s_1, ... s_n}$. Note that sequence lengths can **vary** in a a given dataset, i.e $|{X_i}| \\neq |{X_j}|$.\n",
    "\n",
    "NLP is a classical example of sequence modeling, where a dataset represents a set of sentences/paragraphs/documents (sequence of words). In NLP usually we assume that set of tokens (words) come from a predefined fixed **vocabulary**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "### Learning Objectives\n",
    "\n",
    "The goal of this class is a high-level overview/summary of recent trends in Deep Learning for NLP and sequence modeling. The material is huge, and, unfortunately, most details are beyond this class, however, lot of extra reading links are provided to gain more insights. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-30T10:29:57.643260Z",
     "start_time": "2020-06-30T10:29:57.641328Z"
    }
   },
   "source": [
    "## Recurrent Neural Networks (RNN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-30T12:20:37.085084Z",
     "start_time": "2020-06-30T12:20:37.083013Z"
    }
   },
   "source": [
    "### Motivation\n",
    "\n",
    "When we work with sequences, we can simply calculate embeddings for individual tokens (word2vec) of a sequence and then sum them up (average/concatenate). This simple aggregation might work on simple tasks but in real world more complex and smart aggregation mechanism is required. RNNs are natural fit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-30T10:30:12.299263Z",
     "start_time": "2020-06-30T10:30:12.297291Z"
    }
   },
   "source": [
    "### Vanilla RNNs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-30T12:05:46.021718Z",
     "start_time": "2020-06-30T12:05:46.018855Z"
    }
   },
   "source": [
    "<img src='images/seq_models/rnn.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$X={x_0, x_1, ... x_t}$ is our input data point (individual $x_i$ can be bag-of-words or word2vec representations or something else you come up with), represented as blue circles; A green box is a **RNN cell** which contains a computation and weight; $h_0, h_1, ..., h_t$ is a sequence of **hidden states**, which, intuitively, are information carriers from cell to cell."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The RNN models are used to deal with sequences remembering the previous states information:\n",
    "<img src=\"images/detr/rnn_1.gif\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Visualization of the recurrent neural network\n",
    "<img src=\"images/detr/rnn_anim_1.gif\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Each cell is the neural network with recurrent connections, connection of the hidden layer:\n",
    "<img src=\"images/detr/rnn_anim_2.gif\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-30T12:13:27.112209Z",
     "start_time": "2020-06-30T12:13:27.107695Z"
    }
   },
   "source": [
    "Computation in a RNN cell is the follows:  \n",
    "<br />\n",
    "$h_0 = 0$  \n",
    "$h_t = \\sigma(W x_t + U h_{t-1} + b)$, where $\\sigma$ is element-wise sigmoid    \n",
    "<br />\n",
    "$W, U, b$ - are weights  \n",
    "\n",
    "<br />\n",
    "\n",
    "**Q:** How to do sequence classification/regression with RNN?  \n",
    "**A:** Option 1. Just attach a classifier/regressor on the last hidden state. Option 2. Take average/max pooling of hidden layers and pass to classifier/regressor.\n",
    "\n",
    "**Q:** How to do sequence batching?  \n",
    "**A:** You will have a 3D tensor [B, T, E] or [T, B, E], where B - batch size, T - max sequence length in a batch, E - feature vector dim. Due to the fact that sequences are variable length, you take max length in a batch and append rest with zeros or some special token. Refer also https://pytorch.org/docs/master/generated/torch.nn.utils.rnn.pack_padded_sequence.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Long Short-Term Memory Networks (LSTM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Vanilla RNNs suffer much from so called **vanishing gradient** problem. If you try to write down gradient equation for it, you will bump into a part where lot of small numbers (linear to input sequence size) are multiplied together, which causes the result to go to zero. (Details here https://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture07-fancy-rnn.pdf) For long enough sequences this becomes a serious problem and different architecture is required."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-30T13:15:17.920819Z",
     "start_time": "2020-06-30T13:15:17.917489Z"
    }
   },
   "source": [
    "<img src='images/seq_models/lstm.png' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Sequence2sequence models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Sequence to sequence model was developed for machine translation with RNN (LSTM or GRU) encoder and decoder\n",
    "<br>\n",
    "The architecture was first RNN, the encoder to encode the sentence in the one language, for instance French into the fixed size vector: $300$, $400$, $1024$, $2048$, etc\n",
    "<br>\n",
    "and second RNN, the decoder, to decode this vector to the sentence in other language, for instance in English "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Example of sequence2sequence:\n",
    "<img src=\"images/detr/seq2seq_1.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sequence-to-Sequence Framework\n",
    "\n",
    "Sequence-to-Sequence (seq2seq) is a well know method in Deep Learning for transforming one sequence into another (original paper https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf). Classical example is machine translation, where sentence in one language gets transformed in the corresponding sentence in another language. All state-of-the-art translation systems as well as many other systems use seq-to-seq.  \n",
    "\n",
    "The idea is very similar to Autoencoders. We have two networks (LSTM, for instance), encoder and decoder. Encoder accept input sequence and decoder predicts transformed sequence. Decoder's initial hidden state is the last hidden state of the encoder - that's how the information is shared between encoder and decoder. Intuitively, last hidden state consists of compressed representation of input sentence which a decoder should use for transformation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Example of translation:\n",
    "<img src=\"images/detr/seq2seq_2.gif\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Turns out, that this model works quite well if we choose \"good\" pair of sentences and train it for long time\n",
    "<br>\n",
    "As I remember in the original paper, they sad that, when we reverse the input sentences it significantly improved the accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This model works not only for machine translation, but for other tasks like image captioning:\n",
    "<img src=\"images/detr/imcap_1.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The main idea here, instead of using RNN as encoder, use ConvNet and use RNN only as decoder\n",
    "<br>\n",
    "Use pre-trained model without last layer to encoder image to the vector and feed this vector to the decoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/detr/imcap_3.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src='images/seq_models/seq2seq_trans.jpg' />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-04T07:31:58.711694Z",
     "start_time": "2020-07-04T07:31:58.709683Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Attention Mechanism "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-07-04T07:33:45.178185Z",
     "start_time": "2020-07-04T07:33:45.172872Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Q:** When doing classification/regression using RNN, can we use smarter way than just simply using last hidden state or average/max pooling of hidden states for input to final classifier/regressor layer?  \n",
    "  \n",
    "**A:** Attention Mechanism: We can do weighted average of hidden states where weight indicates the importance of the particular hidden state. Weights are learnt jointly along with the mainstream task.\n",
    "<br />\n",
    "<img src=\"images/seq_models/attn.png\" height=\"400\" width=\"400\" />\n",
    "<br /><br />\n",
    "Y's are hidden layers, Z is final aggregated output, TANH - is computation for similarity measurement (can be one layer fully-connected network with tanh activation, or, more popularly, just a scalar product). C - depending on a task, can be a learnable parameter or some other thing. **NOTE:** The implementations of attention mechanism can vary from paper to paper.  \n",
    "\n",
    "Examples: https://distill.pub/2016/augmented-rnns/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-06-30T16:08:05.297561Z",
     "start_time": "2020-06-30T16:08:05.287760Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Suggested Readings about RNNs\n",
    "\n",
    "https://karpathy.github.io/2015/05/21/rnn-effectiveness/\n",
    "\n",
    "\n",
    "### PyTorch Examples\n",
    "\n",
    "https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html  \n",
    "https://pytorch.org/tutorials/intermediate/char_rnn_generation_tutorial.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Questions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![title](https://www.memecreator.org/static/images/memes/4589248.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Transformers for NLP (<a href=\"http://jalammar.github.io/illustrated-transformer/\">source</a>)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<a href=\"https://arxiv.org/abs/1706.03762\"> Attention is all you need</a>\n",
    "<br>\n",
    "\n",
    "<a href=\"http://jalammar.github.io/illustrated-transformer/\">The Illustrated Transformer</a>\n",
    "<br>\n",
    "\n",
    "<a href=\"https://kazemnejad.com/blog/transformer_architecture_positional_encoding/\">Transformer Architecture: The Positional Encoding</a>\n",
    "<br>\n",
    "\n",
    "<a href=\"https://medium.com/lsc-psd/introduction-of-self-attention-layer-in-transformer-fc7bff63f3bc\">Introduction of Self-Attention Layer in Transformer</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Transformer architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Transformer consists with encoder and decoder and first was introduced as an alternative of sequence2sequence models for machine translation.\n",
    "<br>\n",
    "\n",
    "The key concept is that input sentence is fed as matrix with fixed dimensional word embedding at each row:\n",
    "\n",
    "$$\n",
    "\\begin{pmatrix}\n",
    "x_{1 1} & x_{1 2} & \\cdots & x_{1 d} \\\\\n",
    "x_{2 1} & x_{2 2} & \\cdots & x_{2 d} \\\\\n",
    "\\vdots  & \\vdots  & \\ddots & \\vdots  \\\\\n",
    "x_{m 1} & x_{m 2} & \\cdots & x_{m d} \\\\\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "<br>\n",
    "\n",
    "and is multiplied on weights matrix with fixed dimensional columns (the same as word embeddings) and rows (hyper-parameter, $64$ in original paper):\n",
    "<br>\n",
    "\n",
    "$$\n",
    "\\begin{pmatrix}\n",
    "w_{1 1} & w_{1 2} & \\cdots & w_{1 s} \\\\\n",
    "w_{2 1} & w_{2 2} & \\cdots & w_{2 s} \\\\\n",
    "\\vdots  & \\vdots  & \\ddots & \\vdots  \\\\\n",
    "w_{d 1} & w_{d 2} & \\cdots & w_{d s} \\\\\n",
    "\\end{pmatrix}\n",
    "$$\n",
    "<br>\n",
    "\n",
    "So output is always fixed size $\\mathbb{R}^{d \\times s}$ and theoretically model can consume any length sentence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Model architecture:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The encoder and decoder networks:\n",
    "<img src=\"images/add2/transf_1.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Encoder consists with several different networks stacked together as well as decoder ($6$ in paper):\n",
    "<img src=\"images/add2/transf_2.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The architecture of each layer are similar: (multi-head) self-attention and then feed-forward layers for encoder and decoder, plus decoder has encoder-decoder attention layer as sequence2sequence models:\n",
    "<img src=\"images/add2/transf_3.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Self-attention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "First step: create three different vectors from each input vector (word embedding) using three different weight matrices: query, key and value\n",
    "\n",
    "Notice that these new vectors are smaller in dimension than the embedding vector. Their dimensionality is 64, while the embedding and encoder input/output vectors have dimensionality of 512. They don’t HAVE to be smaller, this is an architecture choice to make the computation of multiheaded attention (mostly) constant\n",
    "<img src=\"images/add2/selfatt_1.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The second step: calculate the score for each query with different keys:\n",
    "\n",
    "<br>\n",
    "The score is calculated by taking the dot product of the query vector with the key vector of the respective word we’re scoring. So if we’re processing the self-attention for the word in position #1, the first score would be the dot product of q1 and k1. The second score would be the dot product of q1 and k2\n",
    "<img src=\"images/add2/selfatt_2.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Third step: Divide the scores by 8 (the square root of the dimension of the key vectors (in the paper 64 and divide on 8 respectively)\n",
    "<br>\n",
    "Fourth step: SoftMax the the results:\n",
    "<img src=\"images/add2/selfatt_3.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Fifth step: Multiply each value vector on this scores, this will generate probability masked values as before\n",
    "<br>\n",
    "Sixth step: Sum all value vectors as output for first embedding:\n",
    "<img src=\"images/add2/selfatt_4.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The resulting vector is one we can send along to the feed-forward neural network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "All above steps might be done in matrix calculation:\n",
    "<img src=\"images/add2/selfatt_5.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Then calculate attention outputs\n",
    "<img src=\"images/add2/selfatt_6.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Instead of single self attention, multi-head attention is applied with different weights ($8$ in original paper):\n",
    "<img src=\"images/add2/selfatt_7.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Per embedding, different outputs are generated:\n",
    "<img src=\"images/add2/selfatt_8.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Then outputs are concatenated horizontally and additional weights matrix is used to produce single matrix:\n",
    "<img src=\"images/add2/selfatt_9.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here is the big picture, performance is improved with multi-head attention (compare to features map):\n",
    "<img src=\"images/add2/selfatt_10.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Positional encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "There is no notion of word order (1st word, 2nd word, ..) in the transformer architecture, thus positional encoding is applied, for $d$ dimensional embeddings:\n",
    "$$\n",
    "\\text{PE}(pos,2i)=sin\\left(\\frac{pos}{10000^{2i/d_{model}}}\\right),\n",
    "$$\n",
    "<br>\n",
    "and\n",
    "$$\n",
    "\\text{PE}(pos,2i+1)=cos\\left(\\frac{pos}{10000^{2i/d_{model}}}\\right).\n",
    "$$\n",
    "generate the $d$ dimensional $\\mathbb{R}^d$ vectors with encoded positional information\n",
    "<br>\n",
    "$d_{model}=512$ model $i \\in [0, 255]$ in paper\n",
    "\n",
    "<br>\n",
    "This is not the only possible method for positional encoding. It, however, gives the advantage of being able to scale to unseen lengths of sequences (e.g. if our trained model is asked to translate a sentence longer than any of those in our training set).\n",
    "<br>\n",
    "\n",
    "The main thing here is, positional encoders should be distinguishable and periodic in order to encode sentences with length which was not seen during the training.\n",
    "<br>\n",
    "\n",
    "From the paper: \"We chose this function because we hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset k, PEpos+k can be represented as a linear function of PEpos.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "These vectors follow a specific pattern that the model learns, which helps it determine the position of each word, or the distance between different words in the sequence. The intuition here is that adding these values to the embeddings provides meaningful distances between the embedding vectors once they’re projected into Q/K/V vectors and during dot-product attention:\n",
    "<img src=\"images/add2/selfatt_11.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For example:\n",
    "<img src=\"images/add2/selfatt_12.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In the following figure, each row corresponds the a positional encoding of a vector. So the first row would be the vector we’d add to the embedding of the first word in an input sequence. Each row contains 512 values – each with a value between 1 and -1. We’ve color-coded them so the pattern is visible:\n",
    "<img src=\"images/add2/selfatt_13.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Residual connections:\n",
    "<img src=\"images/add2/selfatt_14.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Different type of normalizations (not only batch normalization exists)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Instead of batch normalization, lets learn mean and standard deviation for instance, layer, group, etc:\n",
    "<img src=\"images/add2/norm_1.png\" height=\"1000\" width=\"1000\">\n",
    "<br>\n",
    "For images anything beside batch normalization does not give any improvement and sometimes deteriorates performance, because of channel structure, but for transformer architecture, according to the inter-text context and (multi-head) attention it significantly improves performance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "More detailed illustration of layer normalization:\n",
    "<img src=\"images/add2/norm_2.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Layer normalization in transformer associated with self attention, here input embeddings $X$ and output of the layer are summed to preserve the original information:\n",
    "<img src=\"images/add2/selfatt_15.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This goes for the sub-layers of the decoder as well. If we’re to think of a Transformer of 2 stacked encoders and decoders, it would look something like this:\n",
    "<img src=\"images/add2/selfatt_16.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Decoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The encoder start by processing the input sequence. The output of the top encoder is then transformed into a set of attention vectors K and V. These are to be used by each decoder in its “encoder-decoder attention” layer which helps the decoder focus on appropriate places in the input sequence:\n",
    "<img src=\"images/add2/selfatt_17.gif\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The following steps repeat the process until a special symbol is reached indicating the transformer decoder has completed its output. The output of each step is fed to the bottom decoder in the next time step, and the decoders bubble up their decoding results just like the encoders did. And just like we did with the encoder inputs, we embed and add positional encoding to those decoder inputs to indicate the position of each word.\n",
    "\n",
    "<img src=\"images/add2/selfatt_18.gif\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The “Encoder-Decoder Attention” layer works just like multiheaded self-attention, except it creates its Queries matrix from the layer below it, and takes the Keys and Values matrix from the output of the encoder stack."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## BERT (Bi-directional encoder representation from transformers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Training the transformer and use encoder for representation embedding generator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The additional layers for SimCLR, MoCo2, etc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Output of transformer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Transformers has fully connected last layers with SoftMax activation of the vocabulary length vector, encoding by the probability, one-hot encoded word:\n",
    "<img src=\"images/add2/transf_4.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Transformers for images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<a href=\"https://arxiv.org/abs/1802.05751\">Image Transformer</a>\n",
    "<br>\n",
    "\n",
    "<a href=\"https://arxiv.org/abs/1904.09925\">Attention Augmented Convolutional Networks</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Multi-head self-attention technique was used for image decoding as well. The experiments show that mixed approach along with ConvNet layers increases accuracy and performs better than fully attention based model:\n",
    "<br>\n",
    "\"We test our method on the CIFAR-100 and ImageNet classification [22, 9] and the COCO object detection [27] tasks, across a wide range of architectures at different com- putational budgets, including a state-of-the art resource constrained architecture [42]. Attention Augmentation yields systematic improvements with minimal additional computational burden and notably outperforms the popu- lar Squeeze-and-Excitation [17] channelwise attention ap- proach in all experiments. In particular, Attention Augmen- tation achieves a 1.3% top-1 accuracy ImageNet on top of a ResNet50 baseline and 1.4 mAP increase in COCO ob- ject detection on top of a RetinaNet baseline. Suprisingly, experiments also reveal that fully self-attentional models, a special case of Attention Augmentation, only perform slightly worse than their fully convolutional counterparts on ImageNet, indicating that self-attention is a powerful stand- alone computational primitive for image classification.\"\n",
    "<br>\n",
    "\n",
    "In my opinion ConvNet layer have the property to forget inactive features, remove noise and extract word level features from continuous data where multi-head self attention shines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Given the input tensor of shape\n",
    "$$\n",
    "(H, W, F_{in})\n",
    "$$\n",
    "multi-head self-attention is defined as:\n",
    "<br>\n",
    "\n",
    "$$\n",
    "O_h=\\text{SoftMax}(\\frac{(X \\cdot W_q)(X \\cdot W_k)}{\\sqrt{d_k^h}}) \\cdot (X \\cdot W_v)\n",
    "$$\n",
    "<br>\n",
    "\n",
    "Where $W_q, W_k \\in \\mathbb{R}^{F_{in} \\times d_k^h}$ and $W_v \\in \\mathbb{R}^{F_{in} \\times d_v^h}$ are learned liner transformations which map $X$ to queries $Q = X \\cdot W_q$, keys $K = X \\cdot W_k$ and values $X \\cdot W_v$\n",
    "<br>\n",
    "\n",
    "Output of multi-head self-attention layer is \\:\n",
    "$$\n",
    "\\text{MHA}(X) = \\text{concat}(Q_1, O_2, \\dots, O_{N_{h}}) \\cdot W^O\n",
    "$$\n",
    "<br>\n",
    "\n",
    "$$\n",
    "W^O = \\mathbb{R}^{d_u \\times d_u}\n",
    "$$\n",
    "<br>\n",
    "\n",
    "Then the output is reshaped in $(H, W, d_u)$ to match the original input dimension"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "At the end attention augmented convolution is:\n",
    "$$\n",
    "\\text{AAConv}(X) = \\text{concat}( \\text{Conv}(X),\\text{MHA}(X))\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Performance of attention augmented models:\n",
    "<img src=\"images/detr/aaconv_1.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Transformer based object detection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<a href=\"https://ai.facebook.com/research/publications/end-to-end-object-detection-with-transformers\">End-to-end Object Detection with Transformers</a>\n",
    "<br>\n",
    "\n",
    "<a href=\"https://medium.com/lsc-psd/detr-object-detection-with-transformer-a97104ea1723\">DETR, Object detection with Transformer</a>\n",
    "<br>\n",
    "\n",
    "<a href=\"https://www.youtube.com/watch?v=T35ba_VXkMY\">Paper explained in video</a>\n",
    "<br>\n",
    "\n",
    "<a href=\"https://www.youtube.com/watch?v=LfUsGv-ESbc\">[Code] How to use Facebook's DETR object detection algorithm in Python (Full Tutorial)</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The attention layers, the transformer architecture will encode spatial information:\n",
    "<img src=\"images/detr/detr_1.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/detr/detr_2.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/detr/detr_7.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/detr/detr_8.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/detr/detr_9.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/detr/detr_10.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/detr/detr_11.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<img src=\"images/detr/detr_12.png\" height=\"1000\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Questions\n",
    "\n",
    "<img src=\"images/add2/questions_1.png\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Thank you"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<img src=\"https://66.media.tumblr.com/c921909d1c1b24b64f99c59e9793c5c3/tumblr_inline_p7bl0hstyg1vi5k5n_1280.jpg\" height=\"600\" width=\"600\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}