2_images_and_audio_solns.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "2_images_and_audio_solns.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyNwkfSE+IKHaUOwHc8XOJoG",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/vanderbilt-data-science/ai_summer/blob/main/2_images_and_audio_solns.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Images and Audio\n",
        "> Learning more about transformers for image and audio processing\n",
        "\n",
        "Today we'll leverage what we've learned about the general structure of transformers, expectations around training models, and expected differences in model training and apply this to two new modalities: audio and images.\n",
        "\n",
        "* Quick reference for breakout room document: https://docs.google.com/document/d/15deDo3TBlgue_7ueoHake-O3HoEqCZKBZOHWfmfUlFQ/edit?usp=sharing\n",
        "\n",
        "# Learning more about standard vision and audio models\n",
        "\n",
        "In this section, we'll explore a little bit more about the most standard models for audio and image transformers currently implemented in HuggingFace. This section is structured to give you insights into the models themselves a bit as well to inform the HF API that you'll see."
      ],
      "metadata": {
        "id": "yPQOpI7w8DnW"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Image models - Vision Transformer (VIT)\n",
        "* [Paper reference](https://arxiv.org/abs/2010.11929)\n",
        "* [Pdf shortcut](https://arxiv.org/pdf/2010.11929.pdf)\n",
        "\n",
        "## Questions to Answer (Breakout Rooms)\n",
        "\n",
        "1. Identify the blocks in the VIT diagram that correspond to the blocks in the Text Pipeline documentation. Starting with the transformer block may help to inform the other pieces.\n",
        "2. What do you expect to go into the transformer portion of ViT?\n",
        "3. What do you expect the outputs of the transformer portion of the transformer to be?\n",
        "\n",
        "<center>\n",
        "<img src=\"https://github.com/vanderbilt-data-science/ai_summer/blob/main/img/vit-relationships.png?raw=true\" width=\"1000\"/>\n",
        "</center>\n",
        "\n",
        "## Looking into the HF API for VIT\n",
        "\n",
        "Let's now look at the HF API for some insights into the model. Using the [HF API for VIT](https://huggingface.co/docs/transformers/model_doc/vit), answer the following questions:\n",
        "1. What's the class name of the element that is functionally equivalent to a tokenizer in text models?\n",
        "2. What tasks are currently supported for VIT?"
      ],
      "metadata": {
        "id": "7pFFzfWR-MbV"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### HuggingFace Implementation\n",
        "**Guided Explanation**\n",
        "* [How-to Guide](https://huggingface.co/docs/transformers/tasks/image_classification)\n",
        "\n",
        "**Breakout Room Exploration**\n",
        "\n",
        "* [HuggingFace Exploratory Notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/image_classification.ipynb)\n",
        "\n",
        "During your exploration of VIT, note the following observations in the breakout room document as compared to our exploration of training text transformers:\n",
        "* What differences do you see in the *preprocessing* of the data? Will this be relevant to your data?\n",
        "* What differences do you see in the *feature extraction* portion of the code? Before, we saw `input_values` as one of the fields produced by tokenizers. How does this differ with VIT? Will this be relevant to your data?\n",
        "* What differences do you see in *data collation* in the process? Do you think this will be relevant to your process?\n",
        "* What differences do you see in training, if any? Do you think these differences will be relevant to your process?\n",
        "* What differences do you see in post-processing, if any? Do you think these differences will be relevant to your process?\n",
        "\n",
        "Make sure to include the answers to these questions in the breakout room document for your room!"
      ],
      "metadata": {
        "id": "jCRql_AaKDyB"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "---\n",
        "\n"
      ],
      "metadata": {
        "id": "hrLu4sesWOqK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Audio models - Wav2Vec2\n",
        "## References\n",
        "* [Paper reference](https://arxiv.org/abs/2006.11477)\n",
        "* [Pdf shortcut](https://arxiv.org/pdf/2006.11477.pdf)\n",
        "\n",
        "## Questions to Answer (Breakout Rooms)\n",
        "\n",
        "1. Identify the blocks in the wav2vec2 diagram that correspond to the blocks in the Text Pipeline documentation. Starting with the transformer block may help to inform the other pieces.\n",
        "2. What do you expect to go into the transformer portion of the wav2vec2 transformer?\n",
        "3. [A little harder] What do you expect the outputs of the transformer portion of the wav2vec2 transformer to be?\n",
        "4. [Much harder] From the paper:\n",
        "    * There were two types of training tasks performed to train this model. One for the context representations and the other was an application. What are these two tasks?\n",
        "    * Because of the fine-tuning application, what elements might you expect to see as a part of the API for wav2vec2?\n",
        "\n",
        "<center>\n",
        "<img src=\"https://github.com/vanderbilt-data-science/ai_summer/blob/main/img/wav2vec-relationships.png?raw=true\" width=\"1000\"/>\n",
        "</center>\n",
        "\n",
        "## Looking into the HF API for wav2vec2\n",
        "\n",
        "Let's now look at the HF API for some insights into the model. Using the [HF API for wav2vec2](https://huggingface.co/docs/transformers/model_doc/wav2vec2), answer the following questions:\n",
        "1. What's the class name of the element that is functionally equivalent to a tokenizer in text models?\n",
        "2. [Hard] Given the fine-tuning application of wav2vec2, what functionality do you think the CTC _Tokenizer_ and Wav2Vec2 _Processor_ relate to? Are those geared more towards text, or towards \"textless\" NLP capabilities?\n",
        "3. [Very hard] How do you think you would do domain adaptation for this model?\n",
        "\n"
      ],
      "metadata": {
        "id": "44Pm_m1D_6T4"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### HuggingFace Implementation\n",
        "**Guided Exploration**\n",
        "\n",
        "* [How-to Guide](https://huggingface.co/docs/transformers/tasks/audio_classification)\n",
        "\n",
        "**Breakout Room Exploration**\n",
        "* [HuggingFace Exploratory Notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/audio_classification.ipynb)\n",
        "\n",
        "During your exploration of wav2vec2, note the following observations in the breakout room document as compared to our exploration of training text transformers:\n",
        "* What differences do you see in the *preprocessing* of the data? Will this be relevant to your data?\n",
        "* What differences do you see in the *feature extraction* portion of the code? Will this be relevant to your data?\n",
        "* What differences do you see in *data collation* in the process? Do you think this will be relevant to your process?\n",
        "* What differences do you see in training, if any? Do you think these differences will be relevant to your process?\n",
        "* What differences do you see in post-processing, if any? Do you think these differences will be relevant to your process?\n",
        "\n",
        "Make sure to include the answers to these questions in the breakout room document for your room!"
      ],
      "metadata": {
        "id": "i9pttGrAJlmA"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Other HuggingFace Tasks and Models for Audio and Images\n",
        "\n",
        "What support is offered for which models? The [Auto Classes](https://huggingface.co/docs/transformers/model_doc/auto) documentation can help us out with this!"
      ],
      "metadata": {
        "id": "NqzdIlTeKg-2"
      }
    }
  ]
}