t81_558_class_14_05_new_tech.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 14: Other Neural Network Techniques**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Module 14 Video Material\n",
    "\n",
    "* Part 14.1: What is AutoML [[Video]](https://www.youtube.com/watch?v=TFUysIR5AB0&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_14_01_automl.ipynb)\n",
    "* Part 14.2: Using Denoising AutoEncoders in Keras [[Video]](https://www.youtube.com/watch?v=4bTSu6_fucc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_14_02_auto_encode.ipynb)\n",
    "* Part 14.3: Training an Intrusion Detection System with KDD99 [[Video]](https://www.youtube.com/watch?v=1ySn6h2A68I&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_14_03_anomaly.ipynb)\n",
    "* Part 14.4: Anomaly Detection in Keras [[Video]](https://www.youtube.com/watch?v=VgyKQ5MTDFc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_14_04_ids_kdd99.ipynb)\n",
    "* **Part 14.5: The Deep Learning Technologies I am Excited About** [[Video]]() [[Notebook]](t81_558_class_14_05_new_tech.ipynb)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 14.5: New Technologies\n",
    "\n",
    "This course changes often to keep up with the rapidly evolving landscape that is deep learning.  If you would like to continue to monitor this class, I suggest following me on the following:\n",
    "\n",
    "* [GitHub](https://github.com/jeffheaton) - I post all changes to GitHub.\n",
    "* [Jeff Heaton's YouTube Channel](https://www.youtube.com/user/HeatonResearch) - I add new videos for this class at my channel.\n",
    "\n",
    "Currently, four technologies are mainly on my radar for possible future inclusion in this course:\n",
    "\n",
    "* Neural Structured Learning (NSL)\n",
    "* Bert, AlBert, and Other NLP Technologies\n",
    "* Explainability Frameworks\n",
    "\n",
    "This section seeks only to provide a high-level overview of these emerging technologies. I provide links to supplemental material and code in each subsection. I describe these technologies in the following sections.\n",
    "\n",
    "# Neural Structured Learning (NSL)\n",
    "\n",
    "[Neural Structured Learning (NSL)](https://www.tensorflow.org/neural_structured_learning) provides additional training information to the neural network. [[Cite:bui2018neural]](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46568.pdf) This training information is in the form of a graph that relates individual training cases (rows in your training set) among each other. This technique allows the neural networks to train with greater accuracy with a smaller number of labeled data.  When you eventually use the neural network for scoring and prediction, once training completes, the neural network no longer uses the graph.\n",
    "\n",
    "There are two primary sources that this graph data comes from:\n",
    "\n",
    "* Existing Graph Relationships in Data\n",
    "* Automatic Adversarial Modification of Images\n",
    "\n",
    "Often existing graph relationships may exist in data beyond just the labels that describe what individual data items are.  Consider many photo collections.  There may be collections of images placed into specific albums.  This album placement can form additional training information beyond the actual image data and labels.\n",
    "\n",
    "Sometimes graph data cannot be directly obtained for a data set.  Just because you do not have a graph NSL still might be an option.  In such cases, you can make adversarial-like modifications to the data.  You can introduce additional examples and linked them to the original images in the training set.  This technique might make the final trained neural network more resilient to adversarial example attacks.\n",
    "\n",
    "Built into TF 2.0, supports any type of ANN.\n",
    "\n",
    "```\n",
    "pip install neural_structured_learning\n",
    "```\n",
    "\n",
    "Figure 14.NSL is from the original NSL paper. [[Cite:bui2018neural]](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46568.pdf) \n",
    "\n",
    "**Figure 14.NSL: Neural Structured Learning (NSL)**\n",
    "![Neural Structured Learning (NSL)](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/nn_graph_input.png)\n",
    "\n",
    "A: An example of a graph and feature inputs. In this case, there are two labeled nodes ($x_i$, $x_j$) and one unlabeled\n",
    "node ($x_k$), and two edges. The feature vectors, one for each node, are used as neural network inputs. \n",
    "\n",
    "B, C, and D: Illustration of Neural Graph Machine for feed-forward, convolution, and recurrent networks respectively: the training flow ensures the neural net to make accurate node-level predictions and biases the hidden representations/embeddings of neighboring nodes to be similar. In this example, we force $h_i$ and $h_j$ to be similar as there is an edge connecting $x_i$ and $x_j$ nodes. \n",
    "\n",
    "E: Illustration of how we can construct inputs to the neural network using the adjacency matrix. In this example, we have three nodes and two edges. The feature vector created for each node (shown on the right) has 1's at its index and indices of nodes adjacent.\n",
    "\n",
    "Figure 14.NSL shows that NSL can help when there are fewer training elements, it is from a TensorFlow [presentation by Google](https://www.youtube.com/watch?v=2Ucq7a8CY94). This video is a great starting point if you wish to do more with NSL.\n",
    "\n",
    "**Figure 14.NSL: Neural Structured Learning (NSL) Results**\n",
    "![Neural Structured Learning (NSL) Results](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/neural-graph-results.png \"Neural Structured Learning (NSL) Results\")\n",
    "\n",
    "The x-axis shows the amount of training data available, and the y-axis shows the accuracy attained by the model.  The two graphs show NSL applied to different neural network architectures.  You can use NSL with most supervised neural network architectures.  As the number of training elements decreases, there is a period where NSL helps keep the accuracy higher.\n",
    "\n",
    "\n",
    "# Bert, AlBert, and Other NLP Technologies\n",
    "\n",
    "Natural Language Processing (NLP) has seen a tremendous number of advances in the past few years.  One recent technology is Bidirectional Encoder Representations from\n",
    "Transformers (BERT). [[Cite:devlin2018bert]](https://arxiv.org/pdf/1810.04805.pdf) BERT achieved \"state of the art\" results in the following key NLP benchmarks:\n",
    "\n",
    "* [GLUE](https://gluebenchmark.com/) [[Cite:wang2018glue]](https://arxiv.org/abs/1804.07461)\n",
    "* [MultiNLI](https://www.nyu.edu/projects/bowman/multinli/) [[Cite:williams2017broad]](https://www.nyu.edu/projects/bowman/multinli/paper.pdf)\n",
    "* [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/)  [[Cite:rajpurkar2016squad]](https://nlp.stanford.edu/pubs/rajpurkar2018squad.pdf)\n",
    "\n",
    "When a framework, such as BERT claims \"state of the art\" results, it is important to understand what is meant by that.  The GLUE benchmark is composed of the following parts:\n",
    "\n",
    "* Corpus of Linguistic Acceptability (CoLA)\n",
    "* Stanford Sentiment Treebank (SST-2)\n",
    "* Microsoft Research Paraphrase Corpus (MRPC)    \n",
    "* Semantic Textual Similarity Benchmark (STS-B)\n",
    "* Quora Question Pairs (QQP)    \n",
    "* Multi-Genre Natural Language Inference Corpus matched/mismatched (MNLI-m/MNLI-mm)\n",
    "* Stanford Question Answering Dataset (QNLI)\n",
    "* Recognizing Textual Entailment (RTE)\n",
    "* Winograd Schema Challenge (WNLI)\n",
    "\n",
    "**Single Sentence Tasks**\n",
    "\n",
    "CoLA and SST-2 are both single sentence tasks.  CoLA is made up of sample sentences from English grammar textbooks where the authors demonstrated acceptable and unacceptable English grammar usage.  The task in CoLA is to classify a sentence as acceptable or unacceptable.  Examples include:\n",
    "\n",
    "* Acceptable: The angrier Sue gets, the more Fred admires her.\n",
    "* Unacceptable: The most you want, the least you eat.\n",
    "\n",
    "The task SST-2 is used to analyze sentiment.  Sentences are classified by their degree of positivity vs negativity.  For example:\n",
    "\n",
    "* Positive: the greatest musicians\n",
    "* Negative: lend some dignity to a dumb story\n",
    "\n",
    "**Multi-Sentence Similarity Tasks**\n",
    "\n",
    "The MRPC, QQP, and STS-B tasks look at similarity and paraphrase tasks.  The MRPC tests the AIs ability to paraphrase.  Each row contains two sentences and a target that indicates whether each pair captures a paraphrase/semantic equivalence relationship. For example, the following two sentences are considered to be equivalent:\n",
    "\n",
    "* He told The Sun newspaper that Mr. Hussein's daughters had British schools and hospitals in mind when they decided to ask for asylum .    \n",
    "* \"Saddam's daughters had British schools and hospitals in mind when they decided to ask for asylum -- especially the schools,\" he told The Sun.\n",
    "\n",
    "Conversely, though the following two sentences look similar, they are not considered equivalent:\n",
    "\n",
    "* Gyorgy Heizler, head of the local disaster unit, said the coach was carrying 38 passengers.    \n",
    "* The head of the local disaster unit, Gyorgy Heizler, said the coach driver had failed to heed red stoplights.\n",
    "\n",
    "The QQP tasks look at if two questions are asking the same thing.  The Quora website provided this data.  Examples of sentences that are considered to ask the same question:\n",
    "\n",
    "* What are the coolest Android hacks and tricks you know?    \n",
    "* What are some cool hacks for Android phones?\n",
    "\n",
    "Similarly, the following two questions are considered to be different in the QQP dataset.\n",
    "\n",
    "* If you received a check from Donald Knuth, what did you do, and why did you get it?\n",
    "* How can I contact Donald Knuth?\n",
    "\n",
    "The STS-B dataset evaluates how similar two sentences are. If the target/label it 0, then the two sentences are completely dissimilar.  A target value of 5 indicates that the two sentences are completely equivalent, as they\n",
    "mean the same thing.  For example:\n",
    "\n",
    "Two sentences with a label of 0:\n",
    "\n",
    "* A woman is dancing.    \n",
    "* A man is talking.\n",
    "\n",
    "Two sentences with a label of 5:\n",
    "\n",
    "* A plane is taking off.    \n",
    "* An airplane is taking off.\n",
    "\n",
    "**Inference Tasks**\n",
    "\n",
    "The tasks MNLI, QNLI, RTE, and WNLI are all inference tasks.  The MNLI task provides two sentences that must be labeled neutral, contradiction, or entailment.  For example, the following two sentences are a contradiction:\n",
    "\n",
    "* At the end of Rue des Francs-Bourgeois is what many consider to be the city's most handsome residential square, the Place des Vosges, with its stone and red brick facades.\n",
    "* Place des Vosges is constructed entirely of gray marble.\n",
    "\n",
    "These two sentences are an entailment:\n",
    "\n",
    "* I burst through a set of cabin doors, and fell to the ground-    \n",
    "* I burst through the doors and fell down.    \n",
    "\n",
    "These two sentences are neutral:\n",
    "\n",
    "* It's not that the questions they asked weren't exciting or legitimate (though most did fall under the already asked and answered).    \n",
    "* All of the questions were interesting according to a focus group consulted on the subject.    \n",
    "\n",
    "The QNLI task poses a question and supporting sentence.  The label states if the supporting information can answer the question.  For example, the following two are labeled as \"not_entailment\":\n",
    "\n",
    "* Question: Which missile batteries often have individual launchers several kilometers from one another?    \n",
    "* Answer: When MANPADS is operated by specialists, batteries may have several dozen teams deploying separately in small sections; self-propelled air defense guns may deploy in pairs.    \n",
    "\n",
    "Similarly, the following two sentences are labeled as \"entailment\":\n",
    "\n",
    "* Question: What two things does Popper argue Tarski's theory involves in an evaluation of truth?    \n",
    "* Answer: He bases this interpretation on the fact that examples such as the one described above refer to two things: assertions and the facts to which they refer.    \n",
    "\n",
    "The RTE task is similar and looks at whether one sentence entails another.  For example, the following two sentences are labeled as \"entailment\":\n",
    "\n",
    "* Lin Piao, after all, was the creator of Mao's \"Little Red Book\" of quotations.    \n",
    "* Lin Piao wrote the \"Little Red Book\".\n",
    "\n",
    "Similarly, the following two sentences are labeled as \"not_entailment\".\n",
    "\n",
    "* Oil prices fall back as Yukos oil threat lifted    \n",
    "* Oil prices rise.\n",
    "\n",
    "The WNLI task checks to see if two sentences agree to a third.  For example, the following two agree:\n",
    "\n",
    "* The foxes are getting in at night and attacking the chickens. They have gotten very bold.    \n",
    "* The foxes have gotten very bold.\n",
    "\n",
    "Similarly, the following two do not agree:\n",
    "\n",
    "* Sam pulled up a chair to the piano, but it was broken, so he had to stand instead.    \n",
    "* The piano was broken, so he had to stand instead.\n",
    "\n",
    "\n",
    "** BERT High-Level Overview **\n",
    "\n",
    "BERT makes use of both pretraining and fine-tuning before it is ready to be used to evaluate data.  It is important to understand the different roles of these two functions.\n",
    "\n",
    "* **Pretraining** - Ideally, this is done once per language.  Pretraining is the portion of BERT that most will simply obtain from the original BERT model.  These can be [downloaded here](https://github.com/google-research/bert).\n",
    "* **Fine-Tuning** - This is where additional layers are added to the base BERT models to adapt it to the intended task.\n",
    "\n",
    "Figure 14.BERT-1 summarizes the pretraining and fine-tuning phases of BERT.\n",
    "\n",
    "**Figure 14.BERT-1: Pretraining and Fine-Tuning Phases of BERT**\n",
    "![NSL Results](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/bert-1.png)\n",
    "\n",
    "You present sentences to BERT in the method demonstrated in Figure 14.BERT-2.\n",
    "\n",
    "**Figure 14.BERT-2: Sentences Presented to BERT**\n",
    "![NSL Results](https://raw.githubusercontent.com/jeffheaton/t81_558_deep_learning/master/images/bert-2.png)\n",
    "\n",
    "\n",
    "\n",
    "# Explainability Frameworks\n",
    "\n",
    "Neural networks are notorious as black-box models.  Such a model may make accurate predictions; however, explanations of why the black box model chose what it did can be elusive. There are two explainability libraries that I occasionally use:\n",
    "\n",
    "* [Lime](https://github.com/marcotcr/lime)\n",
    "* [Explain it to Me Like I'm 5 (ELI5)](https://eli5.readthedocs.io/en/latest/)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3.7 (tensorflow)",
   "language": "python",
   "name": "tensorflow"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}