transformers.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyOqKSd8fVWzyA+vivXtdPMh",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/goyalpramod/transformer_from_scratch/blob/main/transformers.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "This is my implementation of the original [Attention is all you need paper](https://arxiv.org/pdf/1706.03762). The subsequent code in the notebook has been heavily inspired by the following two link:\n",
        "* [gordicaleksa\n",
        "/\n",
        "pytorch-original-transformer](https://github.com/gordicaleksa/pytorch-original-transformer/blob/main/The%20Annotated%20Transformer%20%2B%2B.ipynb)\n",
        "* [Harvard Annotated Transformer](https://nlp.seas.harvard.edu/annotated-transformer/)"
      ],
      "metadata": {
        "id": "hoSDzdymyyLq"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Note to reader\n",
        "The aim of this notebook is to enable a new practioner in ML field to enter into the depths of NLP. I will follow with the assumption you have been fidling with ml for the past few months (1-3) and are now ready for the big leagues\n",
        "\n",
        "* I will give link to resources/libraries/functions I use that I believe any new MLE (machine learning engineer) would face in their journey. If you understand the terms, feel free to skip them.\n",
        "* I will follow a bottom-up approach. Talking first about what we are implementing. Why we are implementing it, along with a simple visualization to help you understand. Subsequently followed by the code implementation.\n",
        "* Transformers in my opinion were a leap in human history, and they were not made overnight. It takes time and effort to understand these. So do not be disheartened and continue on my brave soldier with the same energy you started with.\n",
        "\n",
        "Enough chit chat, as Linus says\n",
        "> \"Talk is trash, show me the code\"\n",
        "\n",
        " let us begin.\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "Ho9LUCwI8u45"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#Table Of Contents\n",
        "* Requirements\n",
        "*"
      ],
      "metadata": {
        "id": "f_hijizd94k9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install torch torchtext\n",
        "!pip install spacy\n",
        "!python -m spacy download en_core_web_sm\n",
        "!python -m spacy download de_core_news_sm"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nSPViFYF9x2R",
        "outputId": "ba60a898-9c8c-4212-9917-059372a047f8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: torch in /usr/local/lib/python3.10/dist-packages (2.5.1+cu121)\n",
            "Requirement already satisfied: torchtext in /usr/local/lib/python3.10/dist-packages (0.18.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch) (3.16.1)\n",
            "Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch) (4.12.2)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch) (3.4.2)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch) (3.1.4)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch) (2024.10.0)\n",
            "Requirement already satisfied: sympy==1.13.1 in /usr/local/lib/python3.10/dist-packages (from torch) (1.13.1)\n",
            "Requirement already satisfied: mpmath<1.4,>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from sympy==1.13.1->torch) (1.3.0)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/dist-packages (from torchtext) (4.66.6)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from torchtext) (2.32.3)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from torchtext) (1.26.4)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch) (3.0.2)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->torchtext) (3.4.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->torchtext) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->torchtext) (2.2.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->torchtext) (2024.8.30)\n",
            "Requirement already satisfied: spacy in /usr/local/lib/python3.10/dist-packages (3.7.5)\n",
            "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /usr/local/lib/python3.10/dist-packages (from spacy) (3.0.12)\n",
            "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (1.0.5)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (1.0.11)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.10/dist-packages (from spacy) (2.0.10)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.10/dist-packages (from spacy) (3.0.9)\n",
            "Requirement already satisfied: thinc<8.3.0,>=8.2.2 in /usr/local/lib/python3.10/dist-packages (from spacy) (8.2.5)\n",
            "Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /usr/local/lib/python3.10/dist-packages (from spacy) (1.1.3)\n",
            "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /usr/local/lib/python3.10/dist-packages (from spacy) (2.4.8)\n",
            "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.10/dist-packages (from spacy) (2.0.10)\n",
            "Requirement already satisfied: weasel<0.5.0,>=0.1.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (0.4.1)\n",
            "Requirement already satisfied: typer<1.0.0,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (0.15.0)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (4.66.6)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (2.32.3)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4 in /usr/local/lib/python3.10/dist-packages (from spacy) (2.10.3)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from spacy) (3.1.4)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from spacy) (75.1.0)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (24.2)\n",
            "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (3.5.0)\n",
            "Requirement already satisfied: numpy>=1.19.0 in /usr/local/lib/python3.10/dist-packages (from spacy) (1.26.4)\n",
            "Requirement already satisfied: language-data>=1.2 in /usr/local/lib/python3.10/dist-packages (from langcodes<4.0.0,>=3.2.0->spacy) (1.3.0)\n",
            "Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy) (0.7.0)\n",
            "Requirement already satisfied: pydantic-core==2.27.1 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy) (2.27.1)\n",
            "Requirement already satisfied: typing-extensions>=4.12.2 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy) (4.12.2)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy) (3.4.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy) (2.2.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy) (2024.8.30)\n",
            "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /usr/local/lib/python3.10/dist-packages (from thinc<8.3.0,>=8.2.2->spacy) (0.7.11)\n",
            "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /usr/local/lib/python3.10/dist-packages (from thinc<8.3.0,>=8.2.2->spacy) (0.1.5)\n",
            "Requirement already satisfied: click>=8.0.0 in /usr/local/lib/python3.10/dist-packages (from typer<1.0.0,>=0.3.0->spacy) (8.1.7)\n",
            "Requirement already satisfied: shellingham>=1.3.0 in /usr/local/lib/python3.10/dist-packages (from typer<1.0.0,>=0.3.0->spacy) (1.5.4)\n",
            "Requirement already satisfied: rich>=10.11.0 in /usr/local/lib/python3.10/dist-packages (from typer<1.0.0,>=0.3.0->spacy) (13.9.4)\n",
            "Requirement already satisfied: cloudpathlib<1.0.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from weasel<0.5.0,>=0.1.0->spacy) (0.20.0)\n",
            "Requirement already satisfied: smart-open<8.0.0,>=5.2.1 in /usr/local/lib/python3.10/dist-packages (from weasel<0.5.0,>=0.1.0->spacy) (7.0.5)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->spacy) (3.0.2)\n",
            "Requirement already satisfied: marisa-trie>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from language-data>=1.2->langcodes<4.0.0,>=3.2.0->spacy) (1.2.1)\n",
            "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich>=10.11.0->typer<1.0.0,>=0.3.0->spacy) (3.0.0)\n",
            "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich>=10.11.0->typer<1.0.0,>=0.3.0->spacy) (2.18.0)\n",
            "Requirement already satisfied: wrapt in /usr/local/lib/python3.10/dist-packages (from smart-open<8.0.0,>=5.2.1->weasel<0.5.0,>=0.1.0->spacy) (1.17.0)\n",
            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich>=10.11.0->typer<1.0.0,>=0.3.0->spacy) (0.1.2)\n",
            "Collecting en-core-web-sm==3.7.1\n",
            "  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.7.1/en_core_web_sm-3.7.1-py3-none-any.whl (12.8 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.8/12.8 MB\u001b[0m \u001b[31m61.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: spacy<3.8.0,>=3.7.2 in /usr/local/lib/python3.10/dist-packages (from en-core-web-sm==3.7.1) (3.7.5)\n",
            "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (3.0.12)\n",
            "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (1.0.5)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (1.0.11)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.0.10)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (3.0.9)\n",
            "Requirement already satisfied: thinc<8.3.0,>=8.2.2 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (8.2.5)\n",
            "Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (1.1.3)\n",
            "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.4.8)\n",
            "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.0.10)\n",
            "Requirement already satisfied: weasel<0.5.0,>=0.1.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (0.4.1)\n",
            "Requirement already satisfied: typer<1.0.0,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (0.15.0)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (4.66.6)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.32.3)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.10.3)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (3.1.4)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (75.1.0)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (24.2)\n",
            "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (3.5.0)\n",
            "Requirement already satisfied: numpy>=1.19.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (1.26.4)\n",
            "Requirement already satisfied: language-data>=1.2 in /usr/local/lib/python3.10/dist-packages (from langcodes<4.0.0,>=3.2.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (1.3.0)\n",
            "Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (0.7.0)\n",
            "Requirement already satisfied: pydantic-core==2.27.1 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.27.1)\n",
            "Requirement already satisfied: typing-extensions>=4.12.2 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (4.12.2)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (3.4.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.2.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2024.8.30)\n",
            "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /usr/local/lib/python3.10/dist-packages (from thinc<8.3.0,>=8.2.2->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (0.7.11)\n",
            "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /usr/local/lib/python3.10/dist-packages (from thinc<8.3.0,>=8.2.2->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (0.1.5)\n",
            "Requirement already satisfied: click>=8.0.0 in /usr/local/lib/python3.10/dist-packages (from typer<1.0.0,>=0.3.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (8.1.7)\n",
            "Requirement already satisfied: shellingham>=1.3.0 in /usr/local/lib/python3.10/dist-packages (from typer<1.0.0,>=0.3.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (1.5.4)\n",
            "Requirement already satisfied: rich>=10.11.0 in /usr/local/lib/python3.10/dist-packages (from typer<1.0.0,>=0.3.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (13.9.4)\n",
            "Requirement already satisfied: cloudpathlib<1.0.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from weasel<0.5.0,>=0.1.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (0.20.0)\n",
            "Requirement already satisfied: smart-open<8.0.0,>=5.2.1 in /usr/local/lib/python3.10/dist-packages (from weasel<0.5.0,>=0.1.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (7.0.5)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (3.0.2)\n",
            "Requirement already satisfied: marisa-trie>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from language-data>=1.2->langcodes<4.0.0,>=3.2.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (1.2.1)\n",
            "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich>=10.11.0->typer<1.0.0,>=0.3.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (3.0.0)\n",
            "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich>=10.11.0->typer<1.0.0,>=0.3.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (2.18.0)\n",
            "Requirement already satisfied: wrapt in /usr/local/lib/python3.10/dist-packages (from smart-open<8.0.0,>=5.2.1->weasel<0.5.0,>=0.1.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (1.17.0)\n",
            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich>=10.11.0->typer<1.0.0,>=0.3.0->spacy<3.8.0,>=3.7.2->en-core-web-sm==3.7.1) (0.1.2)\n",
            "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
            "You can now load the package via spacy.load('en_core_web_sm')\n",
            "\u001b[38;5;3m⚠ Restart to reload dependencies\u001b[0m\n",
            "If you are in a Jupyter or Colab notebook, you may need to restart Python in\n",
            "order to load all the package's dependencies. You can do this by selecting the\n",
            "'Restart kernel' or 'Restart runtime' option.\n",
            "Collecting de-core-news-sm==3.7.0\n",
            "  Downloading https://github.com/explosion/spacy-models/releases/download/de_core_news_sm-3.7.0/de_core_news_sm-3.7.0-py3-none-any.whl (14.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m14.6/14.6 MB\u001b[0m \u001b[31m67.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: spacy<3.8.0,>=3.7.0 in /usr/local/lib/python3.10/dist-packages (from de-core-news-sm==3.7.0) (3.7.5)\n",
            "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.11 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (3.0.12)\n",
            "Requirement already satisfied: spacy-loggers<2.0.0,>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (1.0.5)\n",
            "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (1.0.11)\n",
            "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (2.0.10)\n",
            "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (3.0.9)\n",
            "Requirement already satisfied: thinc<8.3.0,>=8.2.2 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (8.2.5)\n",
            "Requirement already satisfied: wasabi<1.2.0,>=0.9.1 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (1.1.3)\n",
            "Requirement already satisfied: srsly<3.0.0,>=2.4.3 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (2.4.8)\n",
            "Requirement already satisfied: catalogue<2.1.0,>=2.0.6 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (2.0.10)\n",
            "Requirement already satisfied: weasel<0.5.0,>=0.1.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (0.4.1)\n",
            "Requirement already satisfied: typer<1.0.0,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (0.15.0)\n",
            "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (4.66.6)\n",
            "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (2.32.3)\n",
            "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (2.10.3)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (3.1.4)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (75.1.0)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (24.2)\n",
            "Requirement already satisfied: langcodes<4.0.0,>=3.2.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (3.5.0)\n",
            "Requirement already satisfied: numpy>=1.19.0 in /usr/local/lib/python3.10/dist-packages (from spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (1.26.4)\n",
            "Requirement already satisfied: language-data>=1.2 in /usr/local/lib/python3.10/dist-packages (from langcodes<4.0.0,>=3.2.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (1.3.0)\n",
            "Requirement already satisfied: annotated-types>=0.6.0 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (0.7.0)\n",
            "Requirement already satisfied: pydantic-core==2.27.1 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (2.27.1)\n",
            "Requirement already satisfied: typing-extensions>=4.12.2 in /usr/local/lib/python3.10/dist-packages (from pydantic!=1.8,!=1.8.1,<3.0.0,>=1.7.4->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (4.12.2)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (3.4.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (2.2.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3.0.0,>=2.13.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (2024.8.30)\n",
            "Requirement already satisfied: blis<0.8.0,>=0.7.8 in /usr/local/lib/python3.10/dist-packages (from thinc<8.3.0,>=8.2.2->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (0.7.11)\n",
            "Requirement already satisfied: confection<1.0.0,>=0.0.1 in /usr/local/lib/python3.10/dist-packages (from thinc<8.3.0,>=8.2.2->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (0.1.5)\n",
            "Requirement already satisfied: click>=8.0.0 in /usr/local/lib/python3.10/dist-packages (from typer<1.0.0,>=0.3.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (8.1.7)\n",
            "Requirement already satisfied: shellingham>=1.3.0 in /usr/local/lib/python3.10/dist-packages (from typer<1.0.0,>=0.3.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (1.5.4)\n",
            "Requirement already satisfied: rich>=10.11.0 in /usr/local/lib/python3.10/dist-packages (from typer<1.0.0,>=0.3.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (13.9.4)\n",
            "Requirement already satisfied: cloudpathlib<1.0.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from weasel<0.5.0,>=0.1.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (0.20.0)\n",
            "Requirement already satisfied: smart-open<8.0.0,>=5.2.1 in /usr/local/lib/python3.10/dist-packages (from weasel<0.5.0,>=0.1.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (7.0.5)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (3.0.2)\n",
            "Requirement already satisfied: marisa-trie>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from language-data>=1.2->langcodes<4.0.0,>=3.2.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (1.2.1)\n",
            "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich>=10.11.0->typer<1.0.0,>=0.3.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (3.0.0)\n",
            "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich>=10.11.0->typer<1.0.0,>=0.3.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (2.18.0)\n",
            "Requirement already satisfied: wrapt in /usr/local/lib/python3.10/dist-packages (from smart-open<8.0.0,>=5.2.1->weasel<0.5.0,>=0.1.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (1.17.0)\n",
            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich>=10.11.0->typer<1.0.0,>=0.3.0->spacy<3.8.0,>=3.7.0->de-core-news-sm==3.7.0) (0.1.2)\n",
            "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
            "You can now load the package via spacy.load('de_core_news_sm')\n",
            "\u001b[38;5;3m⚠ Restart to reload dependencies\u001b[0m\n",
            "If you are in a Jupyter or Colab notebook, you may need to restart Python in\n",
            "order to load all the package's dependencies. You can do this by selecting the\n",
            "'Restart kernel' or 'Restart runtime' option.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import math\n",
        "\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "from torch.nn.functional import softmax"
      ],
      "metadata": {
        "id": "hxQHtkoK9yqA"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class MultiHeadAttention(nn.Module):\n",
        "    def __init__(self, d_model, num_heads):\n",
        "        super().__init__()\n",
        "        assert d_model % num_heads == 0, \"d_model must be divisible by num_heads\"\n",
        "\n",
        "        self.d_model = d_model\n",
        "        self.num_heads = num_heads\n",
        "        self.d_k = d_model // num_heads  # Note: use integer division //\n",
        "\n",
        "        # Create the learnable projection matrices\n",
        "        # Why do we need d_model -> d_model size for these?\n",
        "        self.W_q = nn.Linear(d_model, d_model)\n",
        "        self.W_k = nn.Linear(d_model, d_model)\n",
        "        self.W_v = nn.Linear(d_model, d_model)\n",
        "        self.W_o = nn.Linear(d_model, d_model)\n",
        "\n",
        "    @staticmethod\n",
        "    def scaled_dot_product_attention(query, key, value, mask=None):\n",
        "      \"\"\"\n",
        "      Args:\n",
        "          query: (batch_size, num_heads, seq_len_q, d_k)\n",
        "          key: (batch_size, num_heads, seq_len_k, d_k)\n",
        "          value: (batch_size, num_heads, seq_len_v, d_v)\n",
        "          mask: Optional mask to prevent attention to certain positions\n",
        "      \"\"\"\n",
        "      # Shape checks\n",
        "      assert query.dim() == 4, f\"Query should be 4-dim but got {query.dim()}-dim\"\n",
        "      assert key.size(-1) == query.size(-1), \"Key and query depth must be equal\"\n",
        "      assert key.size(-2) == value.size(-2), \"Key and value sequence length must be equal\"\n",
        "\n",
        "      d_k = query.size(-1)\n",
        "\n",
        "      # Attention scores\n",
        "      scores = torch.matmul(query, key.transpose(-2, -1)) / math.sqrt(d_k)\n",
        "\n",
        "      if mask is not None:\n",
        "          scores = scores.masked_fill(mask == 0, float('-inf'))\n",
        "\n",
        "      attention_weights = softmax(scores, dim=-1)\n",
        "\n",
        "      return torch.matmul(attention_weights, value)\n",
        "\n",
        "    def forward(self, query, key, value, mask=None):\n",
        "      batch_size = query.size(0)\n",
        "      seq_len = query.size(1)\n",
        "\n",
        "      # 1. Linear projections\n",
        "      Q = self.W_q(query)  # (batch_size, seq_len, d_model)\n",
        "      K = self.W_k(key)\n",
        "      V = self.W_v(value)\n",
        "\n",
        "      # 2. Split into heads\n",
        "      # Your code: Q = Q.view(:,self.num_heads,:,self.d_k)\n",
        "      # Issues:\n",
        "      # 1. Missing batch_size\n",
        "      # 2. Incorrect dimension ordering\n",
        "      Q = Q.view(batch_size, seq_len, self.num_heads, self.d_k).transpose(1, 2)\n",
        "      K = K.view(batch_size, seq_len, self.num_heads, self.d_k).transpose(1, 2)\n",
        "      V = V.view(batch_size, seq_len, self.num_heads, self.d_k).transpose(1, 2)\n",
        "\n",
        "      # 3. Apply attention\n",
        "      output = self.scaled_dot_product_attention(Q, K, V, mask)\n",
        "\n",
        "      # 4. Concatenate heads\n",
        "      # Your code: scaled_dot_product.view(:,:,d_model)\n",
        "      # Issues:\n",
        "      # 1. Need to transpose back\n",
        "      # 2. Need proper dimensions\n",
        "      output = output.transpose(1, 2).contiguous().view(batch_size, seq_len, self.d_model)\n",
        "\n",
        "      # 5. Final projection\n",
        "      # Your code: scaled_dot_product @ self.W_o.T\n",
        "      # Issues: Use the layer directly\n",
        "      return self.W_o(output)"
      ],
      "metadata": {
        "id": "fJhAMliLGrxj"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class FeedForwardNetwork(nn.Module):\n",
        "    \"\"\"Position-wise Feed-Forward Network\n",
        "\n",
        "    Args:\n",
        "        d_model: input/output dimension\n",
        "        d_ff: hidden dimension\n",
        "        dropout: dropout rate (default=0.1)\n",
        "    \"\"\"\n",
        "    def __init__(self, d_model, d_ff, dropout=0.1):\n",
        "        super().__init__()\n",
        "        self.model = nn.Sequential(\n",
        "            nn.Linear(d_model, d_ff),\n",
        "            nn.ReLU(),\n",
        "            nn.Dropout(dropout),\n",
        "            nn.Linear(d_ff, d_model),\n",
        "            nn.Dropout(dropout)\n",
        "        )\n",
        "\n",
        "    def forward(self, x):\n",
        "        \"\"\"\n",
        "        Args:\n",
        "            x: Input tensor of shape (batch_size, seq_len, d_model)\n",
        "        Returns:\n",
        "            Output tensor of shape (batch_size, seq_len, d_model)\n",
        "        \"\"\"\n",
        "        return self.model(x)"
      ],
      "metadata": {
        "id": "PC5VMOddGvno"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class PositionalEncoding(nn.Module):\n",
        "    def __init__(self, d_model, max_seq_length=5000):\n",
        "        super().__init__()\n",
        "\n",
        "        # Create matrix of shape (max_seq_length, d_model)\n",
        "        pe = torch.zeros(max_seq_length, d_model)\n",
        "\n",
        "        # Create position vector\n",
        "        position = torch.arange(0, max_seq_length).unsqueeze(1) # Shape: (max_seq_length, 1)\n",
        "\n",
        "        # Create division term\n",
        "        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))\n",
        "\n",
        "        # Compute positional encodings\n",
        "        pe[:, 0::2] = torch.sin(position * div_term)\n",
        "        pe[:, 1::2] = torch.cos(position * div_term)\n",
        "\n",
        "        # Register buffer\n",
        "        self.register_buffer('pe', pe.unsqueeze(0))  # Shape: (1, max_seq_length, d_model)\n",
        "\n",
        "    def forward(self, x):\n",
        "        \"\"\"\n",
        "        Args:\n",
        "            x: Tensor shape (batch_size, seq_len, d_model)\n",
        "        \"\"\"\n",
        "        return x + self.pe[:, :x.size(1)]  # Add positional encoding up to sequence length"
      ],
      "metadata": {
        "id": "E1rX5gv-NEFL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class EncoderLayer(nn.Module):\n",
        "    def __init__(self, d_model, num_heads, d_ff, dropout=0.1):\n",
        "        super().__init__()\n",
        "\n",
        "        # 1. Multi-head attention\n",
        "        self.mha = MultiHeadAttention(d_model,num_heads)\n",
        "\n",
        "        # 2. Layer normalization\n",
        "        self.layer_norm_1 = nn.LayerNorm(d_model)\n",
        "\n",
        "        # 3. Feed forward\n",
        "        self.ff = FeedForwardNetwork(d_model,d_ff)\n",
        "\n",
        "        # 4. Another layer normalization\n",
        "        self.layer_norm_2 = nn.LayerNorm(d_model)\n",
        "        # 5. Dropout\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "\n",
        "    def forward(self, x, mask=None):\n",
        "        \"\"\"\n",
        "        Args:\n",
        "            x: Input tensor of shape (batch_size, seq_len, d_model)\n",
        "            mask: Optional mask for padding\n",
        "        Returns:\n",
        "            x: Output tensor of shape (batch_size, seq_len, d_model)\n",
        "        \"\"\"\n",
        "        # 1. Multi-head attention with residual connection and layer norm\n",
        "        # att_output = self.attention(...)\n",
        "        # x = x + att_output  # residual connection\n",
        "        # x = self.norm1(x)  # layer normalization\n",
        "        att_output = self.mha(x, x, x, mask)\n",
        "        x = self.dropout(x + att_output)  # Apply dropout after residual\n",
        "        x = self.layer_norm_1(x)\n",
        "\n",
        "        ff_output = self.ff(x)\n",
        "        x = self.dropout(x + ff_output)  # Apply dropout after residual\n",
        "        x = self.layer_norm_2(x)\n",
        "\n",
        "        # 2. Feed forward with residual connection and layer norm\n",
        "\n",
        "        return x"
      ],
      "metadata": {
        "id": "2peILDjhNIet"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class DecoderLayer(nn.Module):\n",
        "    def __init__(self, d_model, num_heads, d_ff, dropout=0.1):\n",
        "        super().__init__()\n",
        "\n",
        "        # 1. Masked Multi-head attention\n",
        "        self.mha_1 = MultiHeadAttention(d_model,num_heads)\n",
        "\n",
        "        # 2. Layer norm for first sub-layer\n",
        "        self.layer_norm_1 = nn.LayerNorm(d_model)\n",
        "\n",
        "        # 3. Multi-head attention for cross attention with encoder output\n",
        "        # This will take encoder output as key and value\n",
        "        self.mha_2 = MultiHeadAttention(d_model,num_heads)\n",
        "\n",
        "        # 4. Layer norm for second sub-layer\n",
        "        self.layer_norm_2 = nn.LayerNorm(d_model)\n",
        "\n",
        "        # 5. Feed forward network\n",
        "        self.ff = FeedForwardNetwork(d_model,d_ff)\n",
        "\n",
        "        # 6. Layer norm for third sub-layer\n",
        "        self.layer_norm_3 = nn.LayerNorm(d_model)\n",
        "\n",
        "        # 7. Dropout\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "\n",
        "    def forward(self, x, encoder_output, src_mask=None, tgt_mask=None):\n",
        "        \"\"\"\n",
        "        Args:\n",
        "            x: Target sequence embedding (batch_size, target_seq_len, d_model)\n",
        "            encoder_output: Output from encoder (batch_size, source_seq_len, d_model)\n",
        "            src_mask: Mask for source padding\n",
        "            tgt_mask: Mask for target padding and future positions\n",
        "        \"\"\"\n",
        "        # 1. Masked self-attention\n",
        "        # Remember: In decoder self-attention, query, key, value are all x\n",
        "        att_output = self.mha_1(x,x,x,tgt_mask)\n",
        "        x = self.dropout(x + att_output)\n",
        "        x = self.layer_norm_1(x)\n",
        "\n",
        "        att_output_2 = self.mha_2(x, encoder_output,encoder_output, src_mask)\n",
        "        x = self.dropout(x + att_output_2)\n",
        "        x = self.layer_norm_2(x)\n",
        "\n",
        "        ff_output = self.ff(x)\n",
        "        x = self.dropout(x + ff_output)\n",
        "        x = self.layer_norm_3(x)\n",
        "\n",
        "        return x"
      ],
      "metadata": {
        "id": "Dk0wLXd3NFBl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class Encoder(nn.Module):\n",
        "    def __init__(self,\n",
        "                 vocab_size,\n",
        "                 d_model,\n",
        "                 num_layers=6,\n",
        "                 num_heads=8,\n",
        "                 d_ff=2048,\n",
        "                 dropout=0.1,\n",
        "                 max_seq_length=5000):\n",
        "        super().__init__()\n",
        "\n",
        "        # 1. Input embedding\n",
        "        self.embeddings = nn.Embedding(vocab_size, d_model)\n",
        "        self.scale = math.sqrt(d_model)\n",
        "\n",
        "        # 2. Positional encoding\n",
        "        self.pe = PositionalEncoding(d_model, max_seq_length)\n",
        "\n",
        "        # 3. Dropout\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "\n",
        "        # 4. Stack of N encoder layers\n",
        "        self.encoder_layers = nn.ModuleList([\n",
        "            EncoderLayer(d_model, num_heads, d_ff, dropout)\n",
        "            for _ in range(num_layers)\n",
        "        ])\n",
        "\n",
        "    def forward(self, x, mask=None):\n",
        "        \"\"\"\n",
        "        Args:\n",
        "            x: Input tokens (batch_size, seq_len)\n",
        "            mask: Mask for padding positions\n",
        "        Returns:\n",
        "            encoder_output: (batch_size, seq_len, d_model)\n",
        "        \"\"\"\n",
        "        # 1. Pass through embedding layer and scale\n",
        "        x = self.embeddings(x) * self.scale\n",
        "\n",
        "        # 2. Add positional encoding and apply dropout\n",
        "        x = self.dropout(self.pe(x))\n",
        "\n",
        "        # 3. Pass through each encoder layer\n",
        "        for layer in self.encoder_layers:\n",
        "            x = layer(x, mask)\n",
        "\n",
        "        return x"
      ],
      "metadata": {
        "id": "fdaeeJTUNF5a"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class Decoder(nn.Module):\n",
        "    def __init__(self,\n",
        "                 vocab_size,\n",
        "                 d_model,\n",
        "                 num_layers=6,\n",
        "                 num_heads=8,\n",
        "                 d_ff=2048,\n",
        "                 dropout=0.1,\n",
        "                 max_seq_length=5000):\n",
        "        super().__init__()\n",
        "\n",
        "        # 1. Output embedding\n",
        "        self.embeddings = nn.Embedding(vocab_size, d_model)\n",
        "        self.scale = math.sqrt(d_model)\n",
        "\n",
        "        # 2. Positional encoding\n",
        "        self.pe = PositionalEncoding(d_model, max_seq_length)\n",
        "\n",
        "        # 3. Dropout\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "\n",
        "        # 4. Stack of N decoder layers\n",
        "        self.decoder_layers = nn.ModuleList([\n",
        "            DecoderLayer(d_model, num_heads, d_ff, dropout)\n",
        "            for _ in range(num_layers)\n",
        "        ])\n",
        "\n",
        "    def forward(self, x, encoder_output, src_mask=None, tgt_mask=None):\n",
        "        \"\"\"\n",
        "        Args:\n",
        "            x: Target tokens (batch_size, target_seq_len)\n",
        "            encoder_output: Output from encoder (batch_size, source_seq_len, d_model)\n",
        "            src_mask: Mask for source padding\n",
        "            tgt_mask: Mask for target padding and future positions\n",
        "        Returns:\n",
        "            decoder_output: (batch_size, target_seq_len, d_model)\n",
        "        \"\"\"\n",
        "        # 1. Pass through embedding layer and scale\n",
        "        x = self.embeddings(x) * self.scale\n",
        "\n",
        "        # 2. Add positional encoding and dropout\n",
        "        x = self.dropout(self.pe(x))\n",
        "\n",
        "        # 3. Pass through each decoder layer\n",
        "        for layer in self.decoder_layers:\n",
        "            x = layer(x, encoder_output, src_mask, tgt_mask)\n",
        "\n",
        "        return x"
      ],
      "metadata": {
        "id": "gVD027H1NGWT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def create_padding_mask(seq):\n",
        "    \"\"\"\n",
        "    Create mask for padding tokens (0s)\n",
        "    Args:\n",
        "        seq: Input sequence tensor (batch_size, seq_len)\n",
        "    Returns:\n",
        "        mask: Padding mask (batch_size, 1, 1, seq_len)\n",
        "    \"\"\"\n",
        "    batch_size, seq_len = seq.shape\n",
        "    output = torch.eq(seq, 0).float()\n",
        "    return output.view(batch_size, 1, 1, seq_len)\n",
        "\n",
        "def create_future_mask(size):\n",
        "    \"\"\"\n",
        "    Create mask to prevent attention to future positions\n",
        "    Args:\n",
        "        size: Size of square mask (target_seq_len)\n",
        "    Returns:\n",
        "        mask: Future mask (1, 1, size, size)\n",
        "    \"\"\"\n",
        "    # Create upper triangular matrix and invert it\n",
        "    mask = torch.triu(torch.ones((1, 1, size, size)), diagonal=1) == 0\n",
        "    return mask\n",
        "\n",
        "def create_masks(src, tgt):\n",
        "    \"\"\"\n",
        "    Create all masks needed for training\n",
        "    Args:\n",
        "        src: Source sequence (batch_size, src_len)\n",
        "        tgt: Target sequence (batch_size, tgt_len)\n",
        "    Returns:\n",
        "        src_mask: Padding mask for encoder\n",
        "        tgt_mask: Combined padding and future mask for decoder\n",
        "    \"\"\"\n",
        "    # 1. Create padding masks\n",
        "    src_padding_mask = create_padding_mask(src)\n",
        "    tgt_padding_mask = create_padding_mask(tgt)\n",
        "\n",
        "    # 2. Create future mask\n",
        "    tgt_len = tgt.size(1)\n",
        "    tgt_future_mask = create_future_mask(tgt_len)\n",
        "\n",
        "    # 3. Combine padding and future mask for target\n",
        "    # Both masks should be True for allowed positions\n",
        "    tgt_mask = tgt_padding_mask & tgt_future_mask\n",
        "\n",
        "    return src_padding_mask, tgt_mask"
      ],
      "metadata": {
        "id": "7GanNyZFNHqc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class Transformer(nn.Module):\n",
        "    def __init__(self,\n",
        "                 src_vocab_size,\n",
        "                 tgt_vocab_size,\n",
        "                 d_model,\n",
        "                 num_layers=6,\n",
        "                 num_heads=8,\n",
        "                 d_ff=2048,\n",
        "                 dropout=0.1,\n",
        "                 max_seq_length=5000):\n",
        "        super().__init__()\n",
        "\n",
        "        # Pass all necessary parameters to Encoder and Decoder\n",
        "        self.encoder = Encoder(\n",
        "            src_vocab_size,\n",
        "            d_model,\n",
        "            num_layers,\n",
        "            num_heads,\n",
        "            d_ff,\n",
        "            dropout,\n",
        "            max_seq_length\n",
        "        )\n",
        "\n",
        "        self.decoder = Decoder(\n",
        "            tgt_vocab_size,\n",
        "            d_model,\n",
        "            num_layers,\n",
        "            num_heads,\n",
        "            d_ff,\n",
        "            dropout,\n",
        "            max_seq_length\n",
        "        )\n",
        "\n",
        "        # The final linear layer should project from d_model to tgt_vocab_size\n",
        "        self.final_layer = nn.Linear(d_model, tgt_vocab_size)\n",
        "\n",
        "    def forward(self, src, tgt):\n",
        "        # Create masks for source and target\n",
        "        src_mask, tgt_mask = create_masks(src, tgt)\n",
        "\n",
        "        # Pass through encoder\n",
        "        encoder_output = self.encoder(src, src_mask)\n",
        "\n",
        "        # Pass through decoder\n",
        "        decoder_output = self.decoder(tgt, encoder_output, src_mask, tgt_mask)\n",
        "\n",
        "        # Project to vocabulary size\n",
        "        output = self.final_layer(decoder_output)\n",
        "\n",
        "        # Note: Usually don't apply softmax here if using CrossEntropyLoss\n",
        "        # as it applies log_softmax internally\n",
        "        return output"
      ],
      "metadata": {
        "id": "XREKGyXPNH0G"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "class TransformerLRScheduler:\n",
        "    def __init__(self, optimizer, d_model, warmup_steps):\n",
        "        \"\"\"\n",
        "        Args:\n",
        "            optimizer: Optimizer to adjust learning rate for\n",
        "            d_model: Model dimensionality\n",
        "            warmup_steps: Number of warmup steps\n",
        "        \"\"\"\n",
        "        # Your code here\n",
        "        # lrate = d_model^(-0.5) * min(step_num^(-0.5), step_num * warmup_steps^(-1.5))\n",
        "        self.optimizer = optimizer\n",
        "        self.d_model = d_model\n",
        "        self.warmup_steps = warmup_steps\n",
        "\n",
        "\n",
        "    def step(self, step_num):\n",
        "        \"\"\"\n",
        "        Update learning rate based on step number\n",
        "        \"\"\"\n",
        "        # Your code here - implement the formula\n",
        "        lrate = torch.pow(self.d_model,-0.5)*torch.min(torch.pow(step_num,-0.5), torch.tensor(step_num) * torch.pow(self.warmup_steps,-1.5))"
      ],
      "metadata": {
        "id": "auPKw1TwNH7W"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# https://pyimagesearch.com/2019/12/30/label-smoothing-with-keras-tensorflow-and-deep-learning/\n",
        "class LabelSmoothing(nn.Module):\n",
        "    def __init__(self, smoothing=0.1):\n",
        "        super().__init__()\n",
        "        self.smoothing = smoothing\n",
        "        self.confidence = 1.0 - smoothing\n",
        "\n",
        "    def forward(self, logits, target):\n",
        "        \"\"\"\n",
        "        Args:\n",
        "            logits: Model predictions (batch_size, vocab_size) #each row of vocab_size contains probability score of each label\n",
        "            target: True labels (batch_size) #each row of batch size contains the index to the correct label\n",
        "        \"\"\"\n",
        "        vocab_size = logits.size(-1)\n",
        "        with torch.no_grad():\n",
        "            # Create a soft target distribution\n",
        "            true_dist = torch.zeros_like(logits) #create the zeros [0,0,...]\n",
        "            true_dist.fill_(self.smoothing / (vocab_size - 1)) #fill with calculated value [0.000125..,0.000125...] (this is an arbitarary value for example purposes)\n",
        "            true_dist.scatter_(1, target.unsqueeze(1), self.confidence) #add 1 to the correct index (read more on docs of pytorch)\n",
        "        return torch.mean(torch.sum(-true_dist * torch.log_softmax(logits, dim=-1), dim=-1)) #return cross entropy loss"
      ],
      "metadata": {
        "id": "etLFtSMz_1bU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def train_transformer(model, train_dataloader, criterion, optimizer, scheduler, num_epochs, device='cuda'):\n",
        "    \"\"\"\n",
        "    Training loop for transformer\n",
        "\n",
        "    Args:\n",
        "        model: Transformer model\n",
        "        train_dataloader: DataLoader for training data\n",
        "        criterion: Loss function (with label smoothing)\n",
        "        optimizer: Optimizer\n",
        "        scheduler: Learning rate scheduler\n",
        "        num_epochs: Number of training epochs\n",
        "    \"\"\"\n",
        "    # 1. Setup\n",
        "    model = model.to(device)\n",
        "    model.train()\n",
        "\n",
        "    # For tracking training progress\n",
        "    total_loss = 0\n",
        "    all_losses = []\n",
        "\n",
        "    # 2. Training loop\n",
        "    for epoch in range(num_epochs):\n",
        "        print(f\"Epoch {epoch + 1}/{num_epochs}\")\n",
        "        epoch_loss = 0\n",
        "\n",
        "        for batch_idx, batch in enumerate(train_dataloader):\n",
        "            # Get source and target batches\n",
        "            src = batch['src'].to(device)\n",
        "            tgt = batch['tgt'].to(device)\n",
        "\n",
        "            # Create masks\n",
        "            src_mask, tgt_mask = create_masks(src, tgt)\n",
        "\n",
        "            # Prepare target for input and output\n",
        "            # Remove last token from target for input\n",
        "            tgt_input = tgt[:, :-1]\n",
        "            # Remove first token from target for output\n",
        "            tgt_output = tgt[:, 1:]\n",
        "\n",
        "            # Zero gradients\n",
        "            optimizer.zero_grad()\n",
        "\n",
        "            # Forward pass\n",
        "            outputs = model(src, tgt_input, src_mask, tgt_mask)\n",
        "\n",
        "            # Reshape outputs and target for loss calculation\n",
        "            outputs = outputs.view(-1, outputs.size(-1))\n",
        "            tgt_output = tgt_output.view(-1)\n",
        "\n",
        "            # Calculate loss\n",
        "            loss = criterion(outputs, tgt_output)\n",
        "\n",
        "            # Backward pass\n",
        "            loss.backward()\n",
        "\n",
        "            # Clip gradients\n",
        "            torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)\n",
        "\n",
        "            # Update weights\n",
        "            optimizer.step()\n",
        "            scheduler.step()\n",
        "\n",
        "            # Update loss tracking\n",
        "            epoch_loss += loss.item()\n",
        "\n",
        "            # Print progress every N batches\n",
        "            if batch_idx % 100 == 0:\n",
        "                print(f\"Batch {batch_idx}, Loss: {loss.item():.4f}\")\n",
        "\n",
        "        # Calculate average loss for epoch\n",
        "        avg_epoch_loss = epoch_loss / len(train_dataloader)\n",
        "        all_losses.append(avg_epoch_loss)\n",
        "        print(f\"Epoch {epoch + 1} Loss: {avg_epoch_loss:.4f}\")\n",
        "\n",
        "        # Save checkpoint\n",
        "        torch.save({\n",
        "            'epoch': epoch,\n",
        "            'model_state_dict': model.state_dict(),\n",
        "            'optimizer_state_dict': optimizer.state_dict(),\n",
        "            'loss': avg_epoch_loss,\n",
        "        }, f'checkpoint_epoch_{epoch+1}.pt')\n",
        "\n",
        "    return all_losses"
      ],
      "metadata": {
        "id": "Oqiu3KcZpJtg"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "import torch\n",
        "import spacy\n",
        "import urllib.request\n",
        "import zipfile\n",
        "from torch.utils.data import Dataset, DataLoader\n",
        "\n",
        "def download_multi30k():\n",
        "    \"\"\"Download Multi30k dataset if not present\"\"\"\n",
        "    # Create data directory\n",
        "    if not os.path.exists('data'):\n",
        "        os.makedirs('data')\n",
        "\n",
        "    # Download files if they don't exist\n",
        "    base_url = \"https://raw.githubusercontent.com/multi30k/dataset/master/data/task1/raw/\"\n",
        "    files = {\n",
        "        \"train.de\": \"train.de.gz\",\n",
        "        \"train.en\": \"train.en.gz\",\n",
        "        \"val.de\": \"val.de.gz\",\n",
        "        \"val.en\": \"val.en.gz\",\n",
        "        \"test.de\": \"test_2016_flickr.de.gz\",\n",
        "        \"test.en\": \"test_2016_flickr.en.gz\"\n",
        "    }\n",
        "\n",
        "    for local_name, remote_name in files.items():\n",
        "        filepath = f'data/{local_name}'\n",
        "        if not os.path.exists(filepath):\n",
        "            url = base_url + remote_name\n",
        "            urllib.request.urlretrieve(url, filepath + '.gz')\n",
        "            os.system(f'gunzip -f {filepath}.gz')\n",
        "\n",
        "def load_data(filename):\n",
        "    \"\"\"Load data from file\"\"\"\n",
        "    with open(filename, 'r', encoding='utf-8') as f:\n",
        "        return [line.strip() for line in f]\n",
        "\n",
        "def create_dataset():\n",
        "    \"\"\"Create dataset from files\"\"\"\n",
        "    # Download data if needed\n",
        "    download_multi30k()\n",
        "\n",
        "    # Load data\n",
        "    train_de = load_data('data/train.de')\n",
        "    train_en = load_data('data/train.en')\n",
        "    val_de = load_data('data/val.de')\n",
        "    val_en = load_data('data/val.en')\n",
        "\n",
        "    return (train_de, train_en), (val_de, val_en)\n",
        "\n",
        "class TranslationDataset(Dataset):\n",
        "    def __init__(self, src_texts, tgt_texts, src_vocab, tgt_vocab, src_tokenizer, tgt_tokenizer):\n",
        "        self.src_texts = src_texts\n",
        "        self.tgt_texts = tgt_texts\n",
        "        self.src_vocab = src_vocab\n",
        "        self.tgt_vocab = tgt_vocab\n",
        "        self.src_tokenizer = src_tokenizer\n",
        "        self.tgt_tokenizer = tgt_tokenizer\n",
        "\n",
        "    def __len__(self):\n",
        "        return len(self.src_texts)\n",
        "\n",
        "    def __getitem__(self, idx):\n",
        "        src_text = self.src_texts[idx]\n",
        "        tgt_text = self.tgt_texts[idx]\n",
        "\n",
        "        # Tokenize\n",
        "        src_tokens = [tok.text for tok in self.src_tokenizer(src_text)]\n",
        "        tgt_tokens = [tok.text for tok in self.tgt_tokenizer(tgt_text)]\n",
        "\n",
        "        # Convert to indices\n",
        "        src_indices = [self.src_vocab[\"<s>\"]] + [self.src_vocab[token] for token in src_tokens] + [self.src_vocab[\"</s>\"]]\n",
        "        tgt_indices = [self.tgt_vocab[\"<s>\"]] + [self.tgt_vocab[token] for token in tgt_tokens] + [self.tgt_vocab[\"</s>\"]]\n",
        "\n",
        "        return {\n",
        "            'src': torch.tensor(src_indices),\n",
        "            'tgt': torch.tensor(tgt_indices)\n",
        "        }\n",
        "\n",
        "def build_vocab_from_texts(texts, tokenizer, min_freq=2):\n",
        "    \"\"\"Build vocabulary from texts\"\"\"\n",
        "    counter = {}\n",
        "    for text in texts:\n",
        "        for token in [tok.text for tok in tokenizer(text)]:\n",
        "            counter[token] = counter.get(token, 0) + 1\n",
        "\n",
        "    # Create vocabulary\n",
        "    vocab = {\"<s>\": 0, \"</s>\": 1, \"<blank>\": 2, \"<unk>\": 3}\n",
        "    idx = 4\n",
        "    for word, freq in counter.items():\n",
        "        if freq >= min_freq:\n",
        "            vocab[word] = idx\n",
        "            idx += 1\n",
        "    return vocab\n",
        "\n",
        "def create_dataloaders(batch_size=32):\n",
        "    # Load tokenizers\n",
        "    spacy_de = spacy.load(\"de_core_news_sm\")\n",
        "    spacy_en = spacy.load(\"en_core_web_sm\")\n",
        "\n",
        "    # Get data\n",
        "    (train_de, train_en), (val_de, val_en) = create_dataset()\n",
        "\n",
        "    # Build vocabularies\n",
        "    vocab_src = build_vocab_from_texts(train_de, spacy_de)\n",
        "    vocab_tgt = build_vocab_from_texts(train_en, spacy_en)\n",
        "\n",
        "    # Create datasets\n",
        "    train_dataset = TranslationDataset(\n",
        "        train_de, train_en,\n",
        "        vocab_src, vocab_tgt,\n",
        "        spacy_de, spacy_en\n",
        "    )\n",
        "\n",
        "    val_dataset = TranslationDataset(\n",
        "        val_de, val_en,\n",
        "        vocab_src, vocab_tgt,\n",
        "        spacy_de, spacy_en\n",
        "    )\n",
        "\n",
        "    # Create dataloaders\n",
        "    train_dataloader = DataLoader(\n",
        "        train_dataset,\n",
        "        batch_size=batch_size,\n",
        "        shuffle=True,\n",
        "        collate_fn=collate_batch\n",
        "    )\n",
        "\n",
        "    val_dataloader = DataLoader(\n",
        "        val_dataset,\n",
        "        batch_size=batch_size,\n",
        "        shuffle=False,\n",
        "        collate_fn=collate_batch\n",
        "    )\n",
        "\n",
        "    return train_dataloader, val_dataloader, vocab_src, vocab_tgt\n",
        "\n",
        "def collate_batch(batch):\n",
        "    src_tensors = [item['src'] for item in batch]\n",
        "    tgt_tensors = [item['tgt'] for item in batch]\n",
        "\n",
        "    # Pad sequences\n",
        "    src_padded = torch.nn.utils.rnn.pad_sequence(src_tensors, batch_first=True, padding_value=2)\n",
        "    tgt_padded = torch.nn.utils.rnn.pad_sequence(tgt_tensors, batch_first=True, padding_value=2)\n",
        "\n",
        "    return {\n",
        "        'src': src_padded,\n",
        "        'tgt': tgt_padded\n",
        "    }\n",
        "\n",
        "# Usage:\n",
        "train_dataloader, val_dataloader, vocab_src, vocab_tgt = create_dataloaders(batch_size=32)"
      ],
      "metadata": {
        "id": "A1iM-CRW-fCt"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Initialize your transformer with the vocabulary sizes\n",
        "model = Transformer(\n",
        "    src_vocab_size=len(vocab_src),\n",
        "    tgt_vocab_size=len(vocab_tgt),\n",
        "    d_model=512,\n",
        "    num_layers=6,\n",
        "    num_heads=8,\n",
        "    d_ff=2048,\n",
        "    dropout=0.1\n",
        ")\n",
        "criterion = LabelSmoothing(smoothing=0.1).to(device)\n",
        "\n",
        "# Now you can use your training loop\n",
        "losses = train_transformer(\n",
        "    model=model,\n",
        "    train_dataloader=train_dataloader,\n",
        "    criterion=criterion,\n",
        "    optimizer=optimizer,\n",
        "    scheduler=scheduler,\n",
        "    num_epochs=10\n",
        ")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 287
        },
        "id": "jU3yR9mU9ft2",
        "outputId": "dae03db7-2449-4506-ae68-ca8fa9b4d927"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-23-a6926136c68b>\u001b[0m in \u001b[0;36m<cell line: 2>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# Get the dataloader and vocabularies\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mtrain_dataloader\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvocab_src\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvocab_tgt\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcreate_dataloaders\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch_size\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m32\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;31m# Initialize your transformer with the vocabulary sizes\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m model = Transformer(\n",
            "\u001b[0;32m<ipython-input-21-2e4992646a5a>\u001b[0m in \u001b[0;36mcreate_dataloaders\u001b[0;34m(batch_size)\u001b[0m\n\u001b[1;32m    102\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    103\u001b[0m     \u001b[0;31m# Build vocabularies\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 104\u001b[0;31m     \u001b[0mvocab_src\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbuild_vocab_from_texts\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrain_de\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mspacy_de\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    105\u001b[0m     \u001b[0mvocab_tgt\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbuild_vocab_from_texts\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrain_en\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mspacy_en\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    106\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-21-2e4992646a5a>\u001b[0m in \u001b[0;36mbuild_vocab_from_texts\u001b[0;34m(texts, tokenizer, min_freq)\u001b[0m\n\u001b[1;32m     81\u001b[0m     \u001b[0mcounter\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     82\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mtext\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mtexts\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 83\u001b[0;31m         \u001b[0;32mfor\u001b[0m \u001b[0mtoken\u001b[0m \u001b[0;32min\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mtok\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtext\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mtok\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mtokenizer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtext\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     84\u001b[0m             \u001b[0mcounter\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mtoken\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcounter\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtoken\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     85\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/spacy/language.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, text, disable, component_cfg)\u001b[0m\n\u001b[1;32m   1047\u001b[0m                 \u001b[0merror_handler\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mproc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_error_handler\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1048\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1049\u001b[0;31m                 \u001b[0mdoc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mproc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdoc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mcomponent_cfg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m  \u001b[0;31m# type: ignore[call-arg]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1050\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mKeyError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1051\u001b[0m                 \u001b[0;31m# This typically happens if a component is not initialized\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/spacy/pipeline/trainable_pipe.pyx\u001b[0m in \u001b[0;36mspacy.pipeline.trainable_pipe.TrainablePipe.__call__\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/spacy/pipeline/tok2vec.py\u001b[0m in \u001b[0;36mpredict\u001b[0;34m(self, docs)\u001b[0m\n\u001b[1;32m    124\u001b[0m             \u001b[0mwidth\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_dim\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"nO\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    125\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0malloc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwidth\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mdoc\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mdocs\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 126\u001b[0;31m         \u001b[0mtokvecs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpredict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdocs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    127\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mtokvecs\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    128\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/thinc/model.py\u001b[0m in \u001b[0;36mpredict\u001b[0;34m(self, X)\u001b[0m\n\u001b[1;32m    332\u001b[0m         \u001b[0monly\u001b[0m \u001b[0mthe\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minstead\u001b[0m \u001b[0mof\u001b[0m \u001b[0mthe\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moutput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcallback\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;31m`\u001b[0m \u001b[0mtuple\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    333\u001b[0m         \"\"\"\n\u001b[0;32m--> 334\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mis_train\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    335\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    336\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mfinish_update\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptimizer\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptimizer\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/thinc/layers/chain.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(model, X, is_train)\u001b[0m\n\u001b[1;32m     52\u001b[0m     \u001b[0mcallbacks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     53\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mlayer\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlayers\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 54\u001b[0;31m         \u001b[0mY\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minc_layer_grad\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlayer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mis_train\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mis_train\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     55\u001b[0m         \u001b[0mcallbacks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minc_layer_grad\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     56\u001b[0m         \u001b[0mX\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mY\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/thinc/model.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, X, is_train)\u001b[0m\n\u001b[1;32m    308\u001b[0m         \"\"\"Call the model's `forward` function, returning the output and a\n\u001b[1;32m    309\u001b[0m         callback to compute the gradients via backpropagation.\"\"\"\n\u001b[0;32m--> 310\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mis_train\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mis_train\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    311\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    312\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0minitialize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mInT\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mY\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mOutT\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;34m\"Model\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/thinc/layers/with_array.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(model, Xseq, is_train)\u001b[0m\n\u001b[1;32m     40\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlayers\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mXseq\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mis_train\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     41\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 42\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mcast\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mTuple\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mSeqT\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mCallable\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0m_list_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mXseq\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mis_train\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     43\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     44\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/thinc/layers/with_array.py\u001b[0m in \u001b[0;36m_list_forward\u001b[0;34m(model, Xs, is_train)\u001b[0m\n\u001b[1;32m     75\u001b[0m     \u001b[0mlengths\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mNUMPY_OPS\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masarray1i\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mseq\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mseq\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mXs\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     76\u001b[0m     \u001b[0mXf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlayer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mflatten\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mXs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpad\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mpad\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 77\u001b[0;31m     \u001b[0mYf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mget_dXf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlayer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mXf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mis_train\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     78\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     79\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mbackprop\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdYs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mListXd\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mListXd\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/thinc/model.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, X, is_train)\u001b[0m\n\u001b[1;32m    308\u001b[0m         \"\"\"Call the model's `forward` function, returning the output and a\n\u001b[1;32m    309\u001b[0m         callback to compute the gradients via backpropagation.\"\"\"\n\u001b[0;32m--> 310\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mis_train\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mis_train\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    311\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    312\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0minitialize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mInT\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mY\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mOutT\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;34m\"Model\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/thinc/layers/chain.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(model, X, is_train)\u001b[0m\n\u001b[1;32m     52\u001b[0m     \u001b[0mcallbacks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     53\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mlayer\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlayers\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 54\u001b[0;31m         \u001b[0mY\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minc_layer_grad\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlayer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mis_train\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mis_train\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     55\u001b[0m         \u001b[0mcallbacks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minc_layer_grad\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     56\u001b[0m         \u001b[0mX\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mY\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/thinc/model.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, X, is_train)\u001b[0m\n\u001b[1;32m    308\u001b[0m         \"\"\"Call the model's `forward` function, returning the output and a\n\u001b[1;32m    309\u001b[0m         callback to compute the gradients via backpropagation.\"\"\"\n\u001b[0;32m--> 310\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mis_train\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mis_train\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    311\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    312\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0minitialize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mInT\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mY\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mOutT\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;34m\"Model\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/thinc/layers/residual.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(model, X, is_train)\u001b[0m\n\u001b[1;32m     39\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0md_output\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mdX\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     40\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 41\u001b[0;31m     \u001b[0mY\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbackprop_layer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlayers\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mis_train\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     42\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     43\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mY\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mi\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mi\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbackprop\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/thinc/model.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, X, is_train)\u001b[0m\n\u001b[1;32m    308\u001b[0m         \"\"\"Call the model's `forward` function, returning the output and a\n\u001b[1;32m    309\u001b[0m         callback to compute the gradients via backpropagation.\"\"\"\n\u001b[0;32m--> 310\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mis_train\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mis_train\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    311\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    312\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0minitialize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mInT\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mY\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mOutT\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;34m\"Model\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/thinc/layers/chain.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(model, X, is_train)\u001b[0m\n\u001b[1;32m     52\u001b[0m     \u001b[0mcallbacks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     53\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mlayer\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlayers\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 54\u001b[0;31m         \u001b[0mY\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minc_layer_grad\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlayer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mis_train\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mis_train\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     55\u001b[0m         \u001b[0mcallbacks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minc_layer_grad\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     56\u001b[0m         \u001b[0mX\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mY\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/thinc/model.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, X, is_train)\u001b[0m\n\u001b[1;32m    308\u001b[0m         \"\"\"Call the model's `forward` function, returning the output and a\n\u001b[1;32m    309\u001b[0m         callback to compute the gradients via backpropagation.\"\"\"\n\u001b[0;32m--> 310\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mis_train\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mis_train\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    311\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    312\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0minitialize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mInT\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mY\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mOutT\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;34m\"Model\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/thinc/layers/chain.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(model, X, is_train)\u001b[0m\n\u001b[1;32m     52\u001b[0m     \u001b[0mcallbacks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     53\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mlayer\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlayers\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 54\u001b[0;31m         \u001b[0mY\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minc_layer_grad\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlayer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mis_train\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mis_train\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     55\u001b[0m         \u001b[0mcallbacks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minc_layer_grad\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     56\u001b[0m         \u001b[0mX\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mY\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/thinc/model.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, X, is_train)\u001b[0m\n\u001b[1;32m    308\u001b[0m         \"\"\"Call the model's `forward` function, returning the output and a\n\u001b[1;32m    309\u001b[0m         callback to compute the gradients via backpropagation.\"\"\"\n\u001b[0;32m--> 310\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mis_train\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mis_train\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    311\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    312\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0minitialize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mInT\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mY\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mOutT\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;34m\"Model\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/thinc/layers/chain.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(model, X, is_train)\u001b[0m\n\u001b[1;32m     52\u001b[0m     \u001b[0mcallbacks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     53\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mlayer\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlayers\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 54\u001b[0;31m         \u001b[0mY\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minc_layer_grad\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlayer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mis_train\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mis_train\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     55\u001b[0m         \u001b[0mcallbacks\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minc_layer_grad\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     56\u001b[0m         \u001b[0mX\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mY\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/thinc/model.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, X, is_train)\u001b[0m\n\u001b[1;32m    308\u001b[0m         \"\"\"Call the model's `forward` function, returning the output and a\n\u001b[1;32m    309\u001b[0m         callback to compute the gradients via backpropagation.\"\"\"\n\u001b[0;32m--> 310\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mis_train\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mis_train\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    311\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    312\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0minitialize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mInT\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mY\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mOutT\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;34m\"Model\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/thinc/layers/layernorm.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(model, X, is_train)\u001b[0m\n\u001b[1;32m     22\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     23\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mModel\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mInT\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mInT\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mInT\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mis_train\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mbool\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mTuple\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mInT\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mCallable\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 24\u001b[0;31m     \u001b[0mN\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmu\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvar\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_get_moments\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mops\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     25\u001b[0m     \u001b[0mXhat\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mmu\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mvar\u001b[0m \u001b[0;34m**\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1.0\u001b[0m \u001b[0;34m/\u001b[0m \u001b[0;36m2.0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     26\u001b[0m     \u001b[0mY\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbackprop_rescale\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_begin_update_scale_shift\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mXhat\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/thinc/layers/layernorm.py\u001b[0m in \u001b[0;36m_get_moments\u001b[0;34m(ops, X)\u001b[0m\n\u001b[1;32m     74\u001b[0m     \u001b[0mmu\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mFloats2d\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmean\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0maxis\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkeepdims\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     75\u001b[0m     \u001b[0mvar\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mFloats2d\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mX\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0maxis\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkeepdims\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;36m1e-08\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 76\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mcast\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mFloats2d\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mops\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masarray_f\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmu\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvar\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     77\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     78\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/thinc/backends/ops.py\u001b[0m in \u001b[0;36masarray_f\u001b[0;34m(self, data, dtype)\u001b[0m\n\u001b[1;32m    703\u001b[0m         \u001b[0mdtype\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mDTypes\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"float32\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    704\u001b[0m     ) -> FloatsXd:\n\u001b[0;32m--> 705\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mcast\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mFloatsXd\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0masarray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    706\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    707\u001b[0m     def asarray1i(\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/thinc/backends/numpy_ops.pyx\u001b[0m in \u001b[0;36mthinc.backends.numpy_ops.NumpyOps.asarray\u001b[0;34m()\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "0GqEOZBBDQf5"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}