GPT_Notebook.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mnida/gpt2-review/blob/main/GPT_Notebook.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# [1.1] - Transformers from scratch\n"
      ],
      "metadata": {
        "id": "sRrD2AbyMwmP"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ny4PdCRf3YPu"
      },
      "source": [
        "# Introduction"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IGQQ6Sm7fHNm"
      },
      "source": [
        "This is a clean, first principles implementation of GPT-2 in PyTorch. The architectural choices closely follow those used by the TransformerLens library (which you'll be using a lot more in later exercises).\n",
        "\n",
        "The exercises are written to accompany Neel Nanda's [TransformerLens library](https://github.com/neelnanda-io/TransformerLens) for doing mechanistic interpretability research on GPT-2 style language models. We'll be working with this library extensively in this chapter of the course.\n",
        "\n",
        "Each exercise will have a difficulty and importance rating out of 5, as well as an estimated maximum time you should spend on these exercises and sometimes a short annotation. You should interpret the ratings & time estimates relatively (e.g. if you find yourself spending about 50% longer on the exercises than the time estimates, adjust accordingly). Please do skip exercises / look at solutions if you don't feel like they're important enough to be worth doing, and you'd rather get to the good stuff!\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ugWVaR9Gm4Rw"
      },
      "source": [
        "## Content & Learning Objectives\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jzS18i5wfHNm"
      },
      "source": [
        "#### 1️⃣ Understanding Inputs & Outputs of a Transformer\n",
        "\n",
        "In this section, we'll take a first look at transformers - what their function is, how information moves inside a transformer, and what inputs & outputs they take.\n",
        "\n",
        "> ##### Learning objectives\n",
        ">\n",
        "> - Understand what a transformer is used for\n",
        "> - Understand causal attention, and what a transformer's output representsalgebra operations on tensors\n",
        "> - Learn what tokenization is, and how models do it\n",
        "> - Understand what logits are, and how to use them to derive a probability distribution over the vocabulary\n",
        "\n",
        "#### 2️⃣ Clean Transformer Implementation\n",
        "\n",
        "Here, we'll implement a transformer from scratch, using only PyTorch's tensor operations. This will give us a good understanding of how transformers work, and how to use them. We do this by going module-by-module, in an experience which should feel somewhat similar to last week's ResNet exercises. Much like with ResNets, you'll conclude by loading in pretrained weights and verifying that your model works as expected.\n",
        "\n",
        "> ##### Learning objectives\n",
        ">\n",
        "> * Understand that a transformer is composed of attention heads and MLPs, with each one performing operations on the residual stream\n",
        "> * Understand that the attention heads in a single layer operate independently, and that they have the role of calculating attention patterns (which determine where information is moved to & from in the residual stream)\n",
        "> * Learn about & implement the following transformer modules:\n",
        ">     * LayerNorm (transforming the input to have zero mean and unit variance)\n",
        ">     * Positional embedding (a lookup table from position indices to residual stream vectors)\n",
        ">     * Attention (the method of computing attention patterns for residual stream vectors)\n",
        ">     * MLP (the collection of linear and nonlinear transformations which operate on each residual stream vector in the same way)\n",
        ">     * Embedding (a lookup table from tokens to residual stream vectors)\n",
        ">     * Unembedding (a matrix for converting residual stream vectors into a distribution over tokens)\n",
        "\n",
        "#### 3️⃣ Training a Transformer\n",
        "\n",
        "Next, you'll learn how to train your transformer from scratch. This will be quite similar to the training loops you wrote for ResNet in your first week.\n",
        "\n",
        "> ##### Learning objectives\n",
        ">\n",
        "> * Understand how to train a transformer from scratch\n",
        "> * Write a basic transformer training loop\n",
        "> * Interpret the transformer's falling cross entropy loss with reference to features of the training data (e.g. bigram frequencies)\n",
        "\n",
        "#### 4️⃣ Sampling from a Transformer\n",
        "\n",
        "Lastly, you'll learn how to sample from a transformer. This will involve implementing a few different sampling methods, and writing a caching system which can reuse computations from previous forward passes to improve your model's text generation speed.\n",
        "\n",
        "*The second half of this section is less important, and you can skip it if you want.*\n",
        "\n",
        "> ##### Learning objectives\n",
        ">\n",
        "> * Learn how to sample from a transformer\n",
        ">     * This includes basic methods like greedy search or top-k, and more advanced methods like beam search\n",
        "> * Learn how to cache the output of a transformer, so that it can be used to generate text more efficiently\n",
        ">     * Optionally, rewrite your sampling functions to make use of your caching methods\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3mItbKVVm4Rx"
      },
      "source": [
        "## Setup (don't read, just run!)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "P_ngQtCbm4Rx",
        "outputId": "0bb599a7-2cd0-4216-f12a-0836eb057c11",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting transformer_lens\n",
            "  Downloading transformer_lens-1.14.0-py3-none-any.whl (122 kB)\n",
            "\u001b[?25l     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/122.9 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K     \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m \u001b[32m122.9/122.9 kB\u001b[0m \u001b[31m4.1 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m122.9/122.9 kB\u001b[0m \u001b[31m3.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting accelerate>=0.23.0 (from transformer_lens)\n",
            "  Downloading accelerate-0.26.1-py3-none-any.whl (270 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m270.9/270.9 kB\u001b[0m \u001b[31m24.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting beartype<0.15.0,>=0.14.1 (from transformer_lens)\n",
            "  Downloading beartype-0.14.1-py3-none-any.whl (739 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m739.7/739.7 kB\u001b[0m \u001b[31m48.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting better-abc<0.0.4,>=0.0.3 (from transformer_lens)\n",
            "  Downloading better_abc-0.0.3-py3-none-any.whl (3.5 kB)\n",
            "Collecting datasets>=2.7.1 (from transformer_lens)\n",
            "  Downloading datasets-2.16.1-py3-none-any.whl (507 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m507.1/507.1 kB\u001b[0m \u001b[31m37.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting einops>=0.6.0 (from transformer_lens)\n",
            "  Downloading einops-0.7.0-py3-none-any.whl (44 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m44.6/44.6 kB\u001b[0m \u001b[31m6.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting fancy-einsum>=0.0.3 (from transformer_lens)\n",
            "  Downloading fancy_einsum-0.0.3-py3-none-any.whl (6.2 kB)\n",
            "Collecting jaxtyping>=0.2.11 (from transformer_lens)\n",
            "  Downloading jaxtyping-0.2.25-py3-none-any.whl (39 kB)\n",
            "Collecting numpy>=1.24 (from transformer_lens)\n",
            "  Downloading numpy-1.26.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m18.2/18.2 MB\u001b[0m \u001b[31m74.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: pandas>=1.1.5 in /usr/local/lib/python3.10/dist-packages (from transformer_lens) (1.5.3)\n",
            "Requirement already satisfied: rich>=12.6.0 in /usr/local/lib/python3.10/dist-packages (from transformer_lens) (13.7.0)\n",
            "Collecting torch!=2.0,!=2.1.0,>=1.10 (from transformer_lens)\n",
            "  Downloading torch-2.2.0-cp310-cp310-manylinux1_x86_64.whl (755.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m755.5/755.5 MB\u001b[0m \u001b[31m2.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: tqdm>=4.64.1 in /usr/local/lib/python3.10/dist-packages (from transformer_lens) (4.66.1)\n",
            "Requirement already satisfied: transformers>=4.34 in /usr/local/lib/python3.10/dist-packages (from transformer_lens) (4.35.2)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from transformer_lens) (4.5.0)\n",
            "Collecting wandb>=0.13.5 (from transformer_lens)\n",
            "  Downloading wandb-0.16.2-py3-none-any.whl (2.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.2/2.2 MB\u001b[0m \u001b[31m66.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.23.0->transformer_lens) (23.2)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.23.0->transformer_lens) (5.9.5)\n",
            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.23.0->transformer_lens) (6.0.1)\n",
            "Requirement already satisfied: huggingface-hub in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.23.0->transformer_lens) (0.20.3)\n",
            "Requirement already satisfied: safetensors>=0.3.1 in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.23.0->transformer_lens) (0.4.2)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from datasets>=2.7.1->transformer_lens) (3.13.1)\n",
            "Requirement already satisfied: pyarrow>=8.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets>=2.7.1->transformer_lens) (10.0.1)\n",
            "Requirement already satisfied: pyarrow-hotfix in /usr/local/lib/python3.10/dist-packages (from datasets>=2.7.1->transformer_lens) (0.6)\n",
            "Collecting dill<0.3.8,>=0.3.0 (from datasets>=2.7.1->transformer_lens)\n",
            "  Downloading dill-0.3.7-py3-none-any.whl (115 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m115.3/115.3 kB\u001b[0m \u001b[31m14.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.10/dist-packages (from datasets>=2.7.1->transformer_lens) (2.31.0)\n",
            "Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets>=2.7.1->transformer_lens) (3.4.1)\n",
            "Collecting multiprocess (from datasets>=2.7.1->transformer_lens)\n",
            "  Downloading multiprocess-0.70.16-py310-none-any.whl (134 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m15.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: fsspec[http]<=2023.10.0,>=2023.1.0 in /usr/local/lib/python3.10/dist-packages (from datasets>=2.7.1->transformer_lens) (2023.6.0)\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets>=2.7.1->transformer_lens) (3.9.3)\n",
            "Collecting typeguard<3,>=2.13.3 (from jaxtyping>=0.2.11->transformer_lens)\n",
            "  Downloading typeguard-2.13.3-py3-none-any.whl (17 kB)\n",
            "Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas>=1.1.5->transformer_lens) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas>=1.1.5->transformer_lens) (2023.4)\n",
            "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from rich>=12.6.0->transformer_lens) (3.0.0)\n",
            "Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.10/dist-packages (from rich>=12.6.0->transformer_lens) (2.16.1)\n",
            "Collecting typing-extensions (from transformer_lens)\n",
            "  Downloading typing_extensions-4.9.0-py3-none-any.whl (32 kB)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch!=2.0,!=2.1.0,>=1.10->transformer_lens) (1.12)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch!=2.0,!=2.1.0,>=1.10->transformer_lens) (3.2.1)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch!=2.0,!=2.1.0,>=1.10->transformer_lens) (3.1.3)\n",
            "Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch!=2.0,!=2.1.0,>=1.10->transformer_lens)\n",
            "  Downloading nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m23.7/23.7 MB\u001b[0m \u001b[31m40.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-cuda-runtime-cu12==12.1.105 (from torch!=2.0,!=2.1.0,>=1.10->transformer_lens)\n",
            "  Downloading nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m823.6/823.6 kB\u001b[0m \u001b[31m50.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-cuda-cupti-cu12==12.1.105 (from torch!=2.0,!=2.1.0,>=1.10->transformer_lens)\n",
            "  Downloading nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m14.1/14.1 MB\u001b[0m \u001b[31m57.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-cudnn-cu12==8.9.2.26 (from torch!=2.0,!=2.1.0,>=1.10->transformer_lens)\n",
            "  Downloading nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl (731.7 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m731.7/731.7 MB\u001b[0m \u001b[31m2.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-cublas-cu12==12.1.3.1 (from torch!=2.0,!=2.1.0,>=1.10->transformer_lens)\n",
            "  Downloading nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl (410.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m410.6/410.6 MB\u001b[0m \u001b[31m2.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-cufft-cu12==11.0.2.54 (from torch!=2.0,!=2.1.0,>=1.10->transformer_lens)\n",
            "  Downloading nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl (121.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m121.6/121.6 MB\u001b[0m \u001b[31m7.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-curand-cu12==10.3.2.106 (from torch!=2.0,!=2.1.0,>=1.10->transformer_lens)\n",
            "  Downloading nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl (56.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m56.5/56.5 MB\u001b[0m \u001b[31m13.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-cusolver-cu12==11.4.5.107 (from torch!=2.0,!=2.1.0,>=1.10->transformer_lens)\n",
            "  Downloading nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl (124.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m124.2/124.2 MB\u001b[0m \u001b[31m8.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-cusparse-cu12==12.1.0.106 (from torch!=2.0,!=2.1.0,>=1.10->transformer_lens)\n",
            "  Downloading nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl (196.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m196.0/196.0 MB\u001b[0m \u001b[31m6.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-nccl-cu12==2.19.3 (from torch!=2.0,!=2.1.0,>=1.10->transformer_lens)\n",
            "  Downloading nvidia_nccl_cu12-2.19.3-py3-none-manylinux1_x86_64.whl (166.0 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m166.0/166.0 MB\u001b[0m \u001b[31m4.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-nvtx-cu12==12.1.105 (from torch!=2.0,!=2.1.0,>=1.10->transformer_lens)\n",
            "  Downloading nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (99 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m99.1/99.1 kB\u001b[0m \u001b[31m13.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting triton==2.2.0 (from torch!=2.0,!=2.1.0,>=1.10->transformer_lens)\n",
            "  Downloading triton-2.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (167.9 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m167.9/167.9 MB\u001b[0m \u001b[31m7.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting nvidia-nvjitlink-cu12 (from nvidia-cusolver-cu12==11.4.5.107->torch!=2.0,!=2.1.0,>=1.10->transformer_lens)\n",
            "  Downloading nvidia_nvjitlink_cu12-12.3.101-py3-none-manylinux1_x86_64.whl (20.5 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m20.5/20.5 MB\u001b[0m \u001b[31m73.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.34->transformer_lens) (2023.12.25)\n",
            "Requirement already satisfied: tokenizers<0.19,>=0.14 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.34->transformer_lens) (0.15.1)\n",
            "Requirement already satisfied: Click!=8.0.0,>=7.1 in /usr/local/lib/python3.10/dist-packages (from wandb>=0.13.5->transformer_lens) (8.1.7)\n",
            "Collecting GitPython!=3.1.29,>=1.0.0 (from wandb>=0.13.5->transformer_lens)\n",
            "  Downloading GitPython-3.1.41-py3-none-any.whl (196 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m196.4/196.4 kB\u001b[0m \u001b[31m14.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting sentry-sdk>=1.0.0 (from wandb>=0.13.5->transformer_lens)\n",
            "  Downloading sentry_sdk-1.40.0-py2.py3-none-any.whl (257 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m257.5/257.5 kB\u001b[0m \u001b[31m31.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting docker-pycreds>=0.4.0 (from wandb>=0.13.5->transformer_lens)\n",
            "  Downloading docker_pycreds-0.4.0-py2.py3-none-any.whl (9.0 kB)\n",
            "Collecting setproctitle (from wandb>=0.13.5->transformer_lens)\n",
            "  Downloading setproctitle-1.3.3-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (30 kB)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from wandb>=0.13.5->transformer_lens) (67.7.2)\n",
            "Requirement already satisfied: appdirs>=1.4.3 in /usr/local/lib/python3.10/dist-packages (from wandb>=0.13.5->transformer_lens) (1.4.4)\n",
            "Requirement already satisfied: protobuf!=4.21.0,<5,>=3.19.0 in /usr/local/lib/python3.10/dist-packages (from wandb>=0.13.5->transformer_lens) (3.20.3)\n",
            "Requirement already satisfied: six>=1.4.0 in /usr/local/lib/python3.10/dist-packages (from docker-pycreds>=0.4.0->wandb>=0.13.5->transformer_lens) (1.16.0)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.7.1->transformer_lens) (1.3.1)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.7.1->transformer_lens) (23.2.0)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.7.1->transformer_lens) (1.4.1)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.7.1->transformer_lens) (6.0.4)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.7.1->transformer_lens) (1.9.4)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.7.1->transformer_lens) (4.0.3)\n",
            "Collecting gitdb<5,>=4.0.1 (from GitPython!=3.1.29,>=1.0.0->wandb>=0.13.5->transformer_lens)\n",
            "  Downloading gitdb-4.0.11-py3-none-any.whl (62 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.7/62.7 kB\u001b[0m \u001b[31m6.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/dist-packages (from markdown-it-py>=2.2.0->rich>=12.6.0->transformer_lens) (0.1.2)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets>=2.7.1->transformer_lens) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets>=2.7.1->transformer_lens) (3.6)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets>=2.7.1->transformer_lens) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.19.0->datasets>=2.7.1->transformer_lens) (2023.11.17)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch!=2.0,!=2.1.0,>=1.10->transformer_lens) (2.1.4)\n",
            "INFO: pip is looking at multiple versions of multiprocess to determine which version is compatible with other requirements. This could take a while.\n",
            "Collecting multiprocess (from datasets>=2.7.1->transformer_lens)\n",
            "  Downloading multiprocess-0.70.15-py310-none-any.whl (134 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m18.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch!=2.0,!=2.1.0,>=1.10->transformer_lens) (1.3.0)\n",
            "Collecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->GitPython!=3.1.29,>=1.0.0->wandb>=0.13.5->transformer_lens)\n",
            "  Downloading smmap-5.0.1-py3-none-any.whl (24 kB)\n",
            "Installing collected packages: better-abc, typing-extensions, typeguard, triton, smmap, setproctitle, sentry-sdk, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, numpy, fancy-einsum, einops, docker-pycreds, dill, beartype, nvidia-cusparse-cu12, nvidia-cudnn-cu12, multiprocess, jaxtyping, gitdb, nvidia-cusolver-cu12, GitPython, wandb, torch, datasets, accelerate, transformer_lens\n",
            "  Attempting uninstall: typing-extensions\n",
            "    Found existing installation: typing_extensions 4.5.0\n",
            "    Uninstalling typing_extensions-4.5.0:\n",
            "      Successfully uninstalled typing_extensions-4.5.0\n",
            "  Attempting uninstall: triton\n",
            "    Found existing installation: triton 2.1.0\n",
            "    Uninstalling triton-2.1.0:\n",
            "      Successfully uninstalled triton-2.1.0\n",
            "  Attempting uninstall: numpy\n",
            "    Found existing installation: numpy 1.23.5\n",
            "    Uninstalling numpy-1.23.5:\n",
            "      Successfully uninstalled numpy-1.23.5\n",
            "  Attempting uninstall: torch\n",
            "    Found existing installation: torch 2.1.0+cu121\n",
            "    Uninstalling torch-2.1.0+cu121:\n",
            "      Successfully uninstalled torch-2.1.0+cu121\n",
            "\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
            "lida 0.0.10 requires fastapi, which is not installed.\n",
            "lida 0.0.10 requires kaleido, which is not installed.\n",
            "lida 0.0.10 requires python-multipart, which is not installed.\n",
            "lida 0.0.10 requires uvicorn, which is not installed.\n",
            "tensorflow-probability 0.22.0 requires typing-extensions<4.6.0, but you have typing-extensions 4.9.0 which is incompatible.\n",
            "torchaudio 2.1.0+cu121 requires torch==2.1.0, but you have torch 2.2.0 which is incompatible.\n",
            "torchdata 0.7.0 requires torch==2.1.0, but you have torch 2.2.0 which is incompatible.\n",
            "torchtext 0.16.0 requires torch==2.1.0, but you have torch 2.2.0 which is incompatible.\n",
            "torchvision 0.16.0+cu121 requires torch==2.1.0, but you have torch 2.2.0 which is incompatible.\u001b[0m\u001b[31m\n",
            "\u001b[0mSuccessfully installed GitPython-3.1.41 accelerate-0.26.1 beartype-0.14.1 better-abc-0.0.3 datasets-2.16.1 dill-0.3.7 docker-pycreds-0.4.0 einops-0.7.0 fancy-einsum-0.0.3 gitdb-4.0.11 jaxtyping-0.2.25 multiprocess-0.70.15 numpy-1.26.3 nvidia-cublas-cu12-12.1.3.1 nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12-8.9.2.26 nvidia-cufft-cu12-11.0.2.54 nvidia-curand-cu12-10.3.2.106 nvidia-cusolver-cu12-11.4.5.107 nvidia-cusparse-cu12-12.1.0.106 nvidia-nccl-cu12-2.19.3 nvidia-nvjitlink-cu12-12.3.101 nvidia-nvtx-cu12-12.1.105 sentry-sdk-1.40.0 setproctitle-1.3.3 smmap-5.0.1 torch-2.2.0 transformer_lens-1.14.0 triton-2.2.0 typeguard-2.13.3 typing-extensions-4.9.0 wandb-0.16.2\n",
            "Requirement already satisfied: einops in /usr/local/lib/python3.10/dist-packages (0.7.0)\n",
            "Requirement already satisfied: jaxtyping in /usr/local/lib/python3.10/dist-packages (0.2.25)\n",
            "Requirement already satisfied: numpy>=1.20.0 in /usr/local/lib/python3.10/dist-packages (from jaxtyping) (1.26.3)\n",
            "Requirement already satisfied: typeguard<3,>=2.13.3 in /usr/local/lib/python3.10/dist-packages (from jaxtyping) (2.13.3)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.1 in /usr/local/lib/python3.10/dist-packages (from jaxtyping) (4.9.0)\n",
            "Collecting git+https://github.com/callummcdougall/CircuitsVis.git#subdirectory=python\n",
            "  Cloning https://github.com/callummcdougall/CircuitsVis.git to /tmp/pip-req-build-d5yma_fq\n",
            "  Running command git clone --filter=blob:none --quiet https://github.com/callummcdougall/CircuitsVis.git /tmp/pip-req-build-d5yma_fq\n",
            "  Resolved https://github.com/callummcdougall/CircuitsVis.git to commit 1e6129d08cae7af9242d9ab5d3ed322dd44b4dd3\n",
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting importlib-metadata<6.0.0,>=5.1.0 (from circuitsvis==0.0.0)\n",
            "  Downloading importlib_metadata-5.2.0-py3-none-any.whl (21 kB)\n",
            "Requirement already satisfied: numpy<2.0,>=1.23 in /usr/local/lib/python3.10/dist-packages (from circuitsvis==0.0.0) (1.26.3)\n",
            "Requirement already satisfied: torch<3.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from circuitsvis==0.0.0) (2.2.0)\n",
            "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.10/dist-packages (from importlib-metadata<6.0.0,>=5.1.0->circuitsvis==0.0.0) (3.17.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch<3.0,>=2.0->circuitsvis==0.0.0) (3.13.1)\n",
            "Requirement already satisfied: typing-extensions>=4.8.0 in /usr/local/lib/python3.10/dist-packages (from torch<3.0,>=2.0->circuitsvis==0.0.0) (4.9.0)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch<3.0,>=2.0->circuitsvis==0.0.0) (1.12)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch<3.0,>=2.0->circuitsvis==0.0.0) (3.2.1)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch<3.0,>=2.0->circuitsvis==0.0.0) (3.1.3)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/dist-packages (from torch<3.0,>=2.0->circuitsvis==0.0.0) (2023.6.0)\n",
            "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch<3.0,>=2.0->circuitsvis==0.0.0) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch<3.0,>=2.0->circuitsvis==0.0.0) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch<3.0,>=2.0->circuitsvis==0.0.0) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /usr/local/lib/python3.10/dist-packages (from torch<3.0,>=2.0->circuitsvis==0.0.0) (8.9.2.26)\n",
            "Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.10/dist-packages (from torch<3.0,>=2.0->circuitsvis==0.0.0) (12.1.3.1)\n",
            "Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.10/dist-packages (from torch<3.0,>=2.0->circuitsvis==0.0.0) (11.0.2.54)\n",
            "Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.10/dist-packages (from torch<3.0,>=2.0->circuitsvis==0.0.0) (10.3.2.106)\n",
            "Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.10/dist-packages (from torch<3.0,>=2.0->circuitsvis==0.0.0) (11.4.5.107)\n",
            "Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.10/dist-packages (from torch<3.0,>=2.0->circuitsvis==0.0.0) (12.1.0.106)\n",
            "Requirement already satisfied: nvidia-nccl-cu12==2.19.3 in /usr/local/lib/python3.10/dist-packages (from torch<3.0,>=2.0->circuitsvis==0.0.0) (2.19.3)\n",
            "Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.10/dist-packages (from torch<3.0,>=2.0->circuitsvis==0.0.0) (12.1.105)\n",
            "Requirement already satisfied: triton==2.2.0 in /usr/local/lib/python3.10/dist-packages (from torch<3.0,>=2.0->circuitsvis==0.0.0) (2.2.0)\n",
            "Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/dist-packages (from nvidia-cusolver-cu12==11.4.5.107->torch<3.0,>=2.0->circuitsvis==0.0.0) (12.3.101)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch<3.0,>=2.0->circuitsvis==0.0.0) (2.1.4)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch<3.0,>=2.0->circuitsvis==0.0.0) (1.3.0)\n",
            "Building wheels for collected packages: circuitsvis\n",
            "  Building wheel for circuitsvis (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for circuitsvis: filename=circuitsvis-0.0.0-py3-none-any.whl size=6172334 sha256=59415c32edc93016c6c42f3b4833706a3be43b700e7573be5495cc7c0ec5f306\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-18e07toa/wheels/86/be/ad/78078aba9344d200aad61b63d35cdaecdec160212f039eed74\n",
            "Successfully built circuitsvis\n",
            "Installing collected packages: importlib-metadata, circuitsvis\n",
            "  Attempting uninstall: importlib-metadata\n",
            "    Found existing installation: importlib-metadata 7.0.1\n",
            "    Uninstalling importlib-metadata-7.0.1:\n",
            "      Successfully uninstalled importlib-metadata-7.0.1\n",
            "Successfully installed circuitsvis-0.0.0 importlib-metadata-5.2.0\n",
            "--2024-02-05 00:32:39--  https://github.com/callummcdougall/ARENA_2.0/archive/refs/heads/main.zip\n",
            "Resolving github.com (github.com)... 140.82.113.3\n",
            "Connecting to github.com (github.com)|140.82.113.3|:443... connected.\n",
            "HTTP request sent, awaiting response... 302 Found\n",
            "Location: https://codeload.github.com/callummcdougall/ARENA_2.0/zip/refs/heads/main [following]\n",
            "--2024-02-05 00:32:39--  https://codeload.github.com/callummcdougall/ARENA_2.0/zip/refs/heads/main\n",
            "Resolving codeload.github.com (codeload.github.com)... 140.82.112.9\n",
            "Connecting to codeload.github.com (codeload.github.com)|140.82.112.9|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: unspecified [application/zip]\n",
            "Saving to: ‘main.zip’\n",
            "\n",
            "main.zip                [        <=>         ]  74.08M  10.0MB/s    in 7.2s    \n",
            "\n",
            "2024-02-05 00:32:46 (10.2 MB/s) - ‘main.zip’ saved [77683542]\n",
            "\n",
            "Archive:  /content/main.zip\n",
            "8f40c134e73e9d4d40ad94403185db6d7a69d14d\n",
            "   creating: ARENA_2.0-main/chapter1_transformers/exercises/\n",
            "   creating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/\n",
            "   creating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/.vscode/\n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/.vscode/settings.json  \n",
            "   creating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/august23_unique_char/\n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/august23_unique_char/dataset.py  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/august23_unique_char/first_unique_char_model.pt  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/august23_unique_char/model.py  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/august23_unique_char/training.py  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/august23_unique_char/training_model.ipynb  \n",
            "   creating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/eagx_berlin_challenge/\n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/eagx_berlin_challenge/dataset.py  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/eagx_berlin_challenge/model.py  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/eagx_berlin_challenge/palindrome_classifier.pt  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/eagx_berlin_challenge/palindrome_classifier_hard.pt  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/eagx_berlin_challenge/requirements.txt  \n",
            "   creating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/july23_palindromes/\n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/july23_palindromes/dataset.py  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/july23_palindromes/model.py  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/july23_palindromes/palindrome_classifier.pt  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/july23_palindromes/training.py  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/july23_palindromes/training_model.ipynb  \n",
            "   creating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/november23_cumsum/\n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/november23_cumsum/cumsum_model.pt  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/november23_cumsum/dataset.py  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/november23_cumsum/model.py  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/november23_cumsum/training.py  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/november23_cumsum/training_model.ipynb  \n",
            "   creating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/october23_sorted_list/\n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/october23_sorted_list/dataset.py  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/october23_sorted_list/model.py  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/october23_sorted_list/sorted_list_model.pt  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/october23_sorted_list/training.py  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/october23_sorted_list/training_model.ipynb  \n",
            "   creating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/september23_sum/\n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/september23_sum/dataset.py  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/september23_sum/model.py  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/september23_sum/sum_model.pt  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/september23_sum/training.py  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/monthly_algorithmic_problems/september23_sum/training_model.ipynb  \n",
            "   creating: ARENA_2.0-main/chapter1_transformers/exercises/part1_transformer_from_scratch/\n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/part1_transformer_from_scratch/solutions.py  \n",
            "   creating: ARENA_2.0-main/chapter1_transformers/exercises/part2_intro_to_mech_interp/\n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/part2_intro_to_mech_interp/solutions.py  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/part2_intro_to_mech_interp/tests.py  \n",
            "   creating: ARENA_2.0-main/chapter1_transformers/exercises/part3_indirect_object_identification/\n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/part3_indirect_object_identification/ioi_circuit_extraction.py  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/part3_indirect_object_identification/ioi_dataset.py  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/part3_indirect_object_identification/solutions.py  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/part3_indirect_object_identification/tests.py  \n",
            "   creating: ARENA_2.0-main/chapter1_transformers/exercises/part4_interp_on_algorithmic_model/\n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/part4_interp_on_algorithmic_model/brackets_data.json  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/part4_interp_on_algorithmic_model/brackets_datasets.py  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/part4_interp_on_algorithmic_model/brackets_model_state_dict.pt  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/part4_interp_on_algorithmic_model/solutions.py  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/part4_interp_on_algorithmic_model/tests.py  \n",
            "   creating: ARENA_2.0-main/chapter1_transformers/exercises/part5_grokking_and_modular_arithmetic/\n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/part5_grokking_and_modular_arithmetic/my_utils.py  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/part5_grokking_and_modular_arithmetic/solutions.py  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/part5_grokking_and_modular_arithmetic/tests.py  \n",
            "   creating: ARENA_2.0-main/chapter1_transformers/exercises/part6_othellogpt/\n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/part6_othellogpt/solutions.py  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/part6_othellogpt/tests.py  \n",
            "   creating: ARENA_2.0-main/chapter1_transformers/exercises/part7_toy_models_of_superposition/\n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/part7_toy_models_of_superposition/solutions.py  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/part7_toy_models_of_superposition/tests.py  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/part7_toy_models_of_superposition/utils.py  \n",
            "  inflating: ARENA_2.0-main/chapter1_transformers/exercises/plotly_utils.py  \n"
          ]
        }
      ],
      "source": [
        "try:\n",
        "    import google.colab # type: ignore\n",
        "    IN_COLAB = True\n",
        "except:\n",
        "    IN_COLAB = False\n",
        "\n",
        "if IN_COLAB:\n",
        "    # Install packages\n",
        "    %pip install transformer_lens\n",
        "    %pip install einops\n",
        "    %pip install jaxtyping\n",
        "    %pip install git+https://github.com/callummcdougall/CircuitsVis.git#subdirectory=python\n",
        "\n",
        "    # Code to download the necessary files (e.g. solutions, test funcs)\n",
        "    import os, sys\n",
        "    if not os.path.exists(\"chapter1_transformers\"):\n",
        "        !wget https://github.com/callummcdougall/ARENA_2.0/archive/refs/heads/main.zip\n",
        "        !unzip /content/main.zip 'ARENA_2.0-main/chapter1_transformers/exercises/*'\n",
        "        sys.path.append(\"/content/ARENA_2.0-main/chapter1_transformers/exercises\")\n",
        "        os.remove(\"/content/main.zip\")\n",
        "        os.rename(\"ARENA_2.0-main/chapter1_transformers\", \"chapter1_transformers\")\n",
        "        os.rmdir(\"ARENA_2.0-main\")\n",
        "        os.chdir(\"chapter1_transformers/exercises\")\n",
        "else:\n",
        "    # from IPython import get_ipython\n",
        "    # ipython = get_ipython()\n",
        "    # ipython.run_line_magic(\"load_ext\", \"autoreload\")\n",
        "    # ipython.run_line_magic(\"autoreload\", \"2\")\n",
        "    import os, sys\n",
        "    CHAPTER = r\"chapter1_transformers\"\n",
        "    chapter_dir = r\"./\" if CHAPTER in os.listdir() else os.getcwd().split(CHAPTER)[0]\n",
        "    sys.path.append(chapter_dir + f\"{CHAPTER}/exercises\")\n",
        "\n",
        "# Things that need to be done manually\n",
        "\n",
        "# Delete get_topk_non_repeating from instructions (in colab and streamlit)\n",
        "# Indent `pass` in `def beam_search`\n",
        "\n",
        "# END"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "tfWhhdEBm4Ry",
        "outputId": "afc72ef3-2f1c-44b6-d76a-29a8fad4533b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 533,
          "referenced_widgets": [
            "269d5e9f0aff43fd979c9d15141f1562",
            "92f1a14e0c944e6092538d49db2aaf39",
            "7be6388f18514f549478dc383d1f15ce",
            "026f3e2d80544d05baec9e1bb0bb4d32",
            "1bb0142dbdc0473498d11070ff3badb8",
            "f6fda666ff984f69ba6f7e0a33c5af7e",
            "e8776691dee643b5a21a0f9eaffcc1fc",
            "e795971451784c98b2fbaafa4ee78606",
            "26bef72c76d74c548232b7af52232879",
            "b0ef9d5d393148d0bff7734fc2f08602",
            "ca29dd1cbfb742ec9f5930ecc513f6cd",
            "7c5b61c8bdb14ebf94bcc62876f8b4be",
            "6920874e038d4befb546ba1a707eb48a",
            "27c11ca2e5de4163adeafb8a3e9f9d45",
            "c0a5c5d79eee40048245b71565fa29ac",
            "8136627b376749bd8289bc07ada2e734",
            "e9b8c1b6faf84a9a9462b1dd4529c732",
            "97be4b5263cb4e4dbf69491329de09b9",
            "19d0162699d14833b8b5bc650999f244",
            "90605bcc38714f19a2761c085b240d4e",
            "fa5c27fc179549f7b6d629cc6e65adeb",
            "8f1de413688f4a5abd42d170a7d3e56c",
            "54942419d92f492f85981d5f51cc9d67",
            "5a849626d2594777986421bfd002dd36",
            "3a29f9044d3a496392501b6531811035",
            "8ca8ca6dc3be4b8b97f2bf273dd342c5",
            "04d80c42acbd45e88e2f6ef1cc07c304",
            "b77c405889cc49118fdd9fb8e20a7647",
            "4812eba5acc942d78e5240d614ad8ffd",
            "a71b3204509c45919893331630bdffc1",
            "6900bf316fcd44958996ce0b65255e38",
            "d129f10f62474da9bac8f82ab13e353e",
            "69361149eb5e41ac8cc6a4b8f9568f44",
            "7c0ef275820849918a948ddb4a91df07",
            "60d8bfb258d3407283b7b450379c9bd5",
            "faf869fbcf764285a7285c377bea8367",
            "b9e0e56aba3944d4b2b2cf4b2844f743",
            "3a166534a32647c0883900afb90a9a0d",
            "37b71051b1a642d49889d332eb52dca9",
            "5a12edc1e97343178b7798a4c7918d57",
            "a3ce7ac46d1649f38dfa33cb9fbe4c35",
            "8dfaffc2cd5c404e88b7d862f6daa4cd",
            "094f18d4a82f4e85ae1cd4cb588f0a87",
            "81e0822cc55245fe807af2f786e3bc8f",
            "efae7152858d43f58234673f0a2ba576",
            "1a6c704c22b4478e9357b670319df551",
            "31ed8d7270cf495b9a21a6ba9d97f877",
            "ce248259fcd34ead831b57ad0a852fda",
            "4f79df4de94e497f8f8b3885b0f0b7c6",
            "9550e3f920534886bc736f3a0d264f8b",
            "0a6fa772148d4ddf92ef6fbc75e917de",
            "a8b43c8bf5774a04bb680b9b08143fa2",
            "ad4dfe0718d74729ade8c7456d525acc",
            "dfe7cd0558754f6488b92242f3db63c5",
            "6cc9a9c1e1634a38b76ae0de5107fa37",
            "51bea67627e548d49de90e67220e167d",
            "daac9e97c3a54713a063d8f8d43c983e",
            "0b706919b18d44d7a1029902094e4574",
            "b0024d2f09294559916e2ff54f178c0a",
            "59084b8142af412680badbd062fdfeab",
            "682cf14b28f946c7aa3e52fc7711027d",
            "0625904bd90044999c8afd5f0ecc368e",
            "19be64b3d7434e97ab5ed898eccdc42a",
            "e503325532834078aa311fbfba7f134a",
            "9cd383674c964f1a9fee01331c3a4131",
            "bbf85925f8774f42b7e0db66ddbfdc32"
          ]
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.\n",
            "  _torch_pytree._register_pytree_node(\n",
            "/usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.\n",
            "  _torch_pytree._register_pytree_node(\n",
            "/usr/local/lib/python3.10/dist-packages/accelerate/utils/imports.py:209: UserWarning: `ACCELERATE_DISABLE_RICH` is deprecated and will be removed in v0.22.0 and deactivated by default. Please use `ACCELERATE_ENABLE_RICH` if you wish to use `rich`.\n",
            "  warnings.warn(\n",
            "/usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.\n",
            "  _torch_pytree._register_pytree_node(\n",
            "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:88: UserWarning: \n",
            "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
            "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
            "You will be able to reuse this secret in all of your notebooks.\n",
            "Please note that authentication is recommended but still optional to access public models or datasets.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "config.json:   0%|          | 0.00/665 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "269d5e9f0aff43fd979c9d15141f1562"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.\n",
            "  _torch_pytree._register_pytree_node(\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "model.safetensors:   0%|          | 0.00/548M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "7c5b61c8bdb14ebf94bcc62876f8b4be"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "generation_config.json:   0%|          | 0.00/124 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "54942419d92f492f85981d5f51cc9d67"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "vocab.json:   0%|          | 0.00/1.04M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "7c0ef275820849918a948ddb4a91df07"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "merges.txt:   0%|          | 0.00/456k [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "efae7152858d43f58234673f0a2ba576"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "tokenizer.json:   0%|          | 0.00/1.36M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "51bea67627e548d49de90e67220e167d"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Loaded pretrained model gpt2-small into HookedTransformer\n"
          ]
        }
      ],
      "source": [
        "import os; os.environ['ACCELERATE_DISABLE_RICH'] = \"1\"\n",
        "# os.environ['CUDA_LAUNCH_BLOCKING'] = \"1\"\n",
        "import sys\n",
        "import einops\n",
        "from dataclasses import dataclass\n",
        "from transformer_lens import HookedTransformer\n",
        "from transformer_lens.utils import gelu_new, tokenize_and_concatenate\n",
        "import torch as t\n",
        "from torch import Tensor\n",
        "import torch.nn as nn\n",
        "import numpy as np\n",
        "import math\n",
        "from tqdm.notebook import tqdm\n",
        "from typing import Tuple, List, Optional, Dict\n",
        "from jaxtyping import Float, Int\n",
        "from transformers.models.gpt2.tokenization_gpt2_fast import GPT2TokenizerFast\n",
        "from collections import defaultdict\n",
        "from rich.table import Table\n",
        "from rich import print as rprint\n",
        "import datasets\n",
        "from torch.utils.data import DataLoader\n",
        "import wandb\n",
        "from pathlib import Path\n",
        "import webbrowser\n",
        "\n",
        "# Make sure exercises are in the path\n",
        "chapter = r\"chapter1_transformers\"\n",
        "exercises_dir = Path(f\"{os.getcwd().split(chapter)[0]}/{chapter}/exercises\").resolve()\n",
        "section_dir = (exercises_dir / \"part1_transformer_from_scratch\").resolve()\n",
        "if str(exercises_dir) not in sys.path: sys.path.append(str(exercises_dir))\n",
        "\n",
        "from plotly_utils import imshow\n",
        "import part1_transformer_from_scratch.solutions as solutions\n",
        "\n",
        "device = t.device(\"cuda\" if t.cuda.is_available() else \"cpu\")\n",
        "\n",
        "MAIN = __name__ == '__main__'\n",
        "\n",
        "reference_gpt2 = HookedTransformer.from_pretrained(\"gpt2-small\", fold_ln=False, center_unembed=False, center_writing_weights=False)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w1x_mWI6m4Rz"
      },
      "source": [
        "# 1️⃣ Understanding Inputs & Outputs of a Transformer\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B_FUMfnwA6Oq"
      },
      "source": [
        "> ### Learning Objectives\n",
        ">\n",
        "> * Understand what a transformer is used for\n",
        "> * Understand causal attention, and what a transformer's output represents\n",
        "> * Learn what tokenization is, and how models do it\n",
        "> * Understand what logits are, and how to use them to derive a probability distribution over the vocabulary\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d--zV4Crm4Rz"
      },
      "source": [
        "## What is the point of a transformer?\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MaMw4jYgm4Rz"
      },
      "source": [
        "**Transformers exist to model text!**\n",
        "\n",
        "We're going to focus GPT-2 style transformers. Key feature: They generate text! You feed in language, and the model generates a probability distribution over tokens. And you can repeatedly sample from this to generate text!\n",
        "\n",
        "(To explain this in more detail - you feed in a sequence of length $N$, then sample from the probability distribution over the $N+1$-th word, use this to construct a new sequence of length $N+1$, then feed this new sequence into the model to get a probability distribution over the $N+2$-th word, and so on.)\n",
        "\n",
        "### How is the model trained?\n",
        "\n",
        "You give it a bunch of text, and train it to predict the next token.\n",
        "\n",
        "Importantly, if you give a model 100 tokens in a sequence, it predicts the next token for *each* prefix, i.e. it produces 100 logit vectors (= probability distributions) over the set of all words in our vocabulary, with the `i`-th logit vector representing the probability distribution over the token *following* the `i`-th token in the sequence.\n",
        "\n",
        "<details>\n",
        "<summary>Aside - logits</summary>\n",
        "\n",
        "If you haven't encountered the term \"logits\" before, here's a quick refresher.\n",
        "\n",
        "Given an arbitrary vector $x$, we can turn it into a probability distribution via the **softmax** function: $x_i \\to \\frac{e^{x_i}}{\\sum e^{x_j}}$. The exponential makes everything positive; the normalization makes it add to one.\n",
        "\n",
        "The model's output is the vector $x$ (one for each prediction it makes). We call this vector a logit because it represents a probability distribution, and it is related to the actual probabilities via the softmax function.\n",
        "</details>\n",
        "\n",
        "How do we stop the transformer by \"cheating\" by just looking at the tokens it's trying to predict? Answer - we make the transformer have *causal attention* (as opposed to *bidirectional attention*). Causal attention only allows information to move forwards in the sequence, never backwards. The prediction of what comes after token 50 is only a function of the first 50 tokens, *not* of token 51. We say the transformer is **autoregressive**, because it only predicts future words based on past data.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AHzR45fBfHNq"
      },
      "source": [
        "<img src=\"https://raw.githubusercontent.com/callummcdougall/computational-thread-art/master/example_images/misc/transformer-overview-new.png\" width=\"900\">\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z8ZVvPhDm4R0"
      },
      "source": [
        "## Tokens - Transformer Inputs\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UkBYr2qym4R0"
      },
      "source": [
        "Our tranformer's input is natural language (i.e. a sequence of characters, strings, etc). But ML models generally take vectors as input, not langage. How do we convert language to vectors?\n",
        "\n",
        "We can factor this into 2 questions:\n",
        "\n",
        "1. How do we split up language into small sub-units?\n",
        "2. How do we convert these sub-units into vectors?\n",
        "\n",
        "Let's start with the second of these questions.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MRP795JofHNq"
      },
      "source": [
        "### Converting sub-units to vectors\n",
        "\n",
        "We basically make a massive lookup table, which is called an **embedding**. It has one vector for each possible sub-unit of language we might get (we call this set of all sub-units our **vocabulary**). We label every element in our vocabulary with an integer (this labelling never changes), and we use this integer to index into the embedding.\n",
        "\n",
        "A key intuition is that one-hot encodings let you think about each integer independently. We don't bake in any relation between words when we perform our embedding, because every word has a completely separate embedding vector.\n",
        "\n",
        "<details>\n",
        "<summary>Aside - one-hot encodings</summary>\n",
        "\n",
        "We sometimes think about **one-hot encodings** of words. These are vectors with zeros everywhere, except for a single one in the position corresponding to the word's index in the vocabulary. This means that indexing into the embedding is equivalent to multiplying the **embedding matrix** by the one-hot encoding (where the embedding matrix is the matrix we get by stacking all the embedding vectors on top of each other).\n",
        "\n",
        "$$\n",
        "\\begin{aligned}\n",
        "W_E &= \\begin{bmatrix}\n",
        "\\leftarrow v_0 \\rightarrow \\\\\n",
        "\\leftarrow v_1 \\rightarrow \\\\\n",
        "\\vdots \\\\\n",
        "\\leftarrow v_{d_{vocab}-1} \\rightarrow \\\\\n",
        "\\end{bmatrix} \\quad \\text{is the embedding matrix (size }d_{vocab} \\times d_{embed}\\text{),} \\\\\n",
        "\\\\\n",
        "t_i &= (0, \\dots, 0, 1, 0, \\dots, 0) \\quad \\text{is the one-hot encoding for the }i\\text{th word (length }d_{vocab}\\text{)} \\\\\n",
        "\\\\\n",
        "v_i &= t_i W_E \\quad \\text{is the embedding vector for the }i\\text{th word (length }d_{embed}\\text{).} \\\\\n",
        "\\end{aligned}\n",
        "$$\n",
        "\n",
        "</details>\n",
        "\n",
        "Now, let's answer the first question - how do we split language into sub-units?\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Y7t_YsDxm4R0"
      },
      "source": [
        "### Splitting language into sub-units\n",
        "\n",
        "We need to define a standard way of splitting up language into a series of substrings, where each substring is a member of our **vocabulary** set.\n",
        "\n",
        "Could we use a dictionary, and have our vocabulary be the set of all words in the dictionary? No, because this couldn't handle arbitrary text (e.g. URLs, punctuation, etc). We need a more general way of splitting up language.\n",
        "\n",
        "Could we just use the 256 ASCII characters? This fixes the previous problem, but it loses structure of language - some sequences of characters are more meaningful than others. For example, \"language\" is a lot more meaningful than \"hjksdfiu\". We want \"language\" to be a single token, but not \"hjksdfiu\" - this is a more efficient use of our vocab.\n",
        "\n",
        "What actually happens? The most common strategy is called **Byte-Pair encodings**.\n",
        "\n",
        "We begin with the 256 ASCII characters as our tokens, and then find the most common pair of tokens, and merge that into a new token. Note that we do have a space character as one of our 256 tokens, and merges using space are very common. For instance, here are the five first merges for the tokenizer used by GPT-2 (you'll be able to verify this below).\n",
        "\n",
        "```\n",
        "\" t\"\n",
        "\" a\"\n",
        "\"he\"\n",
        "\"in\"\n",
        "\"re\"\n",
        "```\n",
        "\n",
        "<details>\n",
        "<summary>Fun (totally optional) exercise - can you guess what the first-formed 3/4/5/6/7-letter encodings in GPT-2's vocabulary are?</summary>\n",
        "Run this code to find out:\n",
        "\n",
        "```python\n",
        "lengths = dict.fromkeys(range(3, 8), \"\")\n",
        "for tok, idx in sorted_vocab:\n",
        "    if not lengths.get(len(tok), True):\n",
        "        lengths[len(tok)] = tok\n",
        "\n",
        "for length, tok in lengths.items():\n",
        "    print(f\"{length}: {tok}\")\n",
        "```\n",
        "</details>\n",
        "\n",
        "Note - you might see the character `Ġ` in front of some tokens. This is a special token that indicates that the token begins with a space. Tokens with a leading space vs not are different.\n",
        "\n",
        "You can run the code below to see some more of GPT-2's tokenizer's vocabulary:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "sHq2U0hKnXMm",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "be6b736c-bd68-480b-d8c5-fa1d3c57a2fb"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[('!', 0), ('\"', 1), ('#', 2), ('$', 3), ('%', 4), ('&', 5), (\"'\", 6), ('(', 7), (')', 8), ('*', 9), ('+', 10), (',', 11), ('-', 12), ('.', 13), ('/', 14), ('0', 15), ('1', 16), ('2', 17), ('3', 18), ('4', 19)]\n",
            "\n",
            "[('ľ', 250), ('Ŀ', 251), ('ŀ', 252), ('Ł', 253), ('ł', 254), ('Ń', 255), ('Ġt', 256), ('Ġa', 257), ('he', 258), ('in', 259), ('re', 260), ('on', 261), ('Ġthe', 262), ('er', 263), ('Ġs', 264), ('at', 265), ('Ġw', 266), ('Ġo', 267), ('en', 268), ('Ġc', 269)]\n",
            "\n",
            "[('Ġprodu', 990), ('Ġstill', 991), ('led', 992), ('ah', 993), ('Ġhere', 994), ('Ġworld', 995), ('Ġthough', 996), ('Ġnum', 997), ('arch', 998), ('imes', 999), ('ale', 1000), ('ĠSe', 1001), ('ĠIf', 1002), ('//', 1003), ('ĠLe', 1004), ('Ġret', 1005), ('Ġref', 1006), ('Ġtrans', 1007), ('ner', 1008), ('ution', 1009)]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "sorted_vocab = sorted(list(reference_gpt2.tokenizer.vocab.items()), key=lambda n: n[1])\n",
        "print(sorted_vocab[:20])\n",
        "print()\n",
        "print(sorted_vocab[250:270])\n",
        "print()\n",
        "print(sorted_vocab[990:1010])\n",
        "print()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ddGjdpeinaI8"
      },
      "source": [
        "As you get to the end of the vocabulary, you'll be producing some pretty weird-looking esoteric tokens (because you'll already have exhausted all of the short frequently-occurring ones):\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "Rl40Jlw0m4R0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "adbbb85f-2111-4989-c728-da4b877c6089"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[('Revolution', 50237), ('Ġsnipers', 50238), ('Ġreverted', 50239), ('Ġconglomerate', 50240), ('Terry', 50241), ('794', 50242), ('Ġharsher', 50243), ('Ġdesolate', 50244), ('ĠHitman', 50245), ('Commission', 50246), ('Ġ(/', 50247), ('âĢ¦.\"', 50248), ('Compar', 50249), ('Ġamplification', 50250), ('ominated', 50251), ('Ġregress', 50252), ('ĠCollider', 50253), ('Ġinformants', 50254), ('Ġgazed', 50255), ('<|endoftext|>', 50256)]\n"
          ]
        }
      ],
      "source": [
        "print(sorted_vocab[-20:])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S5OKWOqHnmBg"
      },
      "source": [
        "Transformers in the `transformer_lens` library have a `to_tokens` method that converts text to numbers. It also prepends them with a special token called BOS (beginning of sequence) to indicate the start of a sequence. You can disable this with the `prepend_bos=False` argument.\n",
        "\n",
        "<details>\n",
        "<summary>Aside - BOS token</summary>\n",
        "\n",
        "The beginning of sequence (BOS) token is a special token used to mark the beginning of the sequence. Confusingly, in GPT-2, the End of Sequence (EOS), Beginning of Sequence (BOS) and Padding (PAD) tokens are all the same, `<|endoftext|>` with index `50256`.\n",
        "\n",
        "Why is this token added? Some basic intuitions are:\n",
        "\n",
        "* It provides context that this is the start of a sequence, which can help the model generate more appropriate text.\n",
        "* It can act as a \"rest position\" for attention heads (more on this later, when we discuss attention).\n",
        "\n",
        "TransformerLens adds this token automatically (including in forward passes of transformer models, e.g. it's implicitly added when you call `model(\"Hello World\")`). You can disable this behaviour by setting the flag `prepend_bos=False` in `to_tokens`, `to_str_tokens`, `model.forward` and any other function that converts strings to multi-token tensors.\n",
        "\n",
        "**Key Point: *If you get weird off-by-one errors, check whether there's an unexpected `prepend_bos`!***\n",
        "\n",
        "Why are the BOS, EOS and PAD tokens the same? This is because GPT-2 is an autoregressive model, and uses these kinds of tokens in a slightly different way to other transformer families (e.g. BERT). For instance, GPT has no need to distinguish between BOS and EOS tokens, because it only processes text from left to right.\n",
        "\n",
        "</details>\n",
        "\n",
        "### Some tokenization annoyances\n",
        "\n",
        "There are a few funky and frustrating things about tokenization, which causes it to behave differently than you might expect. For instance:\n",
        "\n",
        "#### Whether a word begins with a capital or space matters!\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "0vlZ-pRAnoEw",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "358a083f-72c7-4b21-e3bc-947dd10c0f78"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['<|endoftext|>', 'R', 'alph']\n",
            "['<|endoftext|>', ' Ralph']\n",
            "['<|endoftext|>', ' r', 'alph']\n",
            "['<|endoftext|>', 'ral', 'ph']\n"
          ]
        }
      ],
      "source": [
        "print(reference_gpt2.to_str_tokens(\"Ralph\"))\n",
        "print(reference_gpt2.to_str_tokens(\" Ralph\"))\n",
        "print(reference_gpt2.to_str_tokens(\" ralph\"))\n",
        "print(reference_gpt2.to_str_tokens(\"ralph\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "joPO920PntIw"
      },
      "source": [
        "#### Arithmetic is a mess.\n",
        "\n",
        "Length is inconsistent, common numbers bundle together.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "rzbf4t-Zm4R1",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d2422111-83b6-4242-fab0-48a35ee738a6"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['<|endoftext|>', '568', '73', '+', '318', '46', '23', '=', '123', '45', '67', '89', '-', '1', '000000', '000']\n"
          ]
        }
      ],
      "source": [
        "print(reference_gpt2.to_str_tokens(\"56873+3184623=123456789-1000000000\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GgtPkQE-m4R2"
      },
      "source": [
        "> ### Key Takeaways\n",
        ">\n",
        "> * We learn a dictionary of vocab of tokens (sub-words).\n",
        "> * We (approx) losslessly convert language to integers via tokenizing it.\n",
        "> * We convert integers to vectors via a lookup table.\n",
        "> * Note: input to the transformer is a sequence of *tokens* (ie integers), not vectors\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rCSs3uXgn9Kb"
      },
      "source": [
        "## Text generation\n",
        "\n",
        "Now that we understand the basic ideas here, let's go through the entire process of text generation, from our original string to a new token which we can append to our string and plug back into the model.\n",
        "\n",
        "#### **Step 1:** Convert text to tokens\n",
        "\n",
        "The sequence gets tokenized, so it has shape `[batch, seq_len]`. Here, the batch dimension is just one (because we only have one sequence).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "mommexyJm4R3",
        "outputId": "3637d933-61e9-4546-85eb-cca04febad7f",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[50256,    40,   716,   281,  4998,  1960,   382, 19741,    11,   875,\n",
            "         12342,    12,  8807,    11,   402, 11571,    12,    17,  3918, 47385,\n",
            "            13,  1881,  1110,   314,   481,  7074,  1692,  1241,  4430,   290,\n",
            "          1011,   625,   262,   995,     0]])\n",
            "torch.Size([1, 35])\n",
            "['<|endoftext|>', 'I', ' am', ' an', ' amazing', ' aut', 'ore', 'gressive', ',', ' dec', 'oder', '-', 'only', ',', ' G', 'PT', '-', '2', ' style', ' transformer', '.', ' One', ' day', ' I', ' will', ' exceed', ' human', ' level', ' intelligence', ' and', ' take', ' over', ' the', ' world', '!']\n"
          ]
        }
      ],
      "source": [
        "reference_text = \"I am an amazing autoregressive, decoder-only, GPT-2 style transformer. One day I will exceed human level intelligence and take over the world!\"\n",
        "tokens = reference_gpt2.to_tokens(reference_text).to(device)\n",
        "print(tokens)\n",
        "print(tokens.shape)\n",
        "print(reference_gpt2.to_str_tokens(tokens))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ovci8XHNm4R3"
      },
      "source": [
        "#### **Step 2:** Map tokens to logits\n",
        "\n",
        "\n",
        "From our input of shape `[batch, seq_len]`, we get output of shape `[batch, seq_len, vocab_size]`. The `[i, j, :]`-th element of our output is a vector of logits representing our prediction for the `j+1`-th token in the `i`-th sequence.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "5iK_hEUzm4R3",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f272dd63-98e3-499f-a6ef-ff4f3be1ddae"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.Size([1, 35, 50257])\n"
          ]
        }
      ],
      "source": [
        "logits, cache = reference_gpt2.run_with_cache(tokens)\n",
        "print(logits.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "39kXnrmrgeIw"
      },
      "source": [
        "(`run_with_cache` tells the model to cache all intermediate activations. This isn't important right now; we'll look at it in more detail later.)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SDv_ftwFm4R4"
      },
      "source": [
        "#### **Step 3:** Convert the logits to a distribution with a softmax\n",
        "\n",
        "This doesn't change the shape, it is still `[batch, seq_len, vocab_size]`.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "csYLLcvcm4R4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "35406c00-b15a-43b8-fdc1-20c16e2df666"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.Size([1, 35, 50257])\n"
          ]
        }
      ],
      "source": [
        "probs = logits.softmax(dim=-1)\n",
        "print(probs.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BkOqYy8qm4R4"
      },
      "source": [
        "#### **Bonus step:** What is the most likely next token at each position?\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "9tKcEGLXm4R4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4d2c33d6-0cdf-439e-9a48-7b275d8a98d6"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[('<|endoftext|>', '\\n'), ('I', \"'m\"), (' am', ' a'), (' an', ' avid'), (' amazing', ' person'), (' aut', 'od'), ('ore', 'sp'), ('gressive', '.'), (',', ' and'), (' dec', 'ently'), ('oder', ','), ('-', 'driven'), ('only', ' programmer'), (',', ' and'), (' G', 'IM'), ('PT', '-'), ('-', 'only'), ('2', '.'), (' style', ','), (' transformer', '.'), ('.', ' I'), (' One', ' of'), (' day', ' I'), (' I', ' will'), (' will', ' be'), (' exceed', ' my'), (' human', 'ly'), (' level', ' of'), (' intelligence', ' and'), (' and', ' I'), (' take', ' over'), (' over', ' the'), (' the', ' world'), (' world', '.'), ('!', ' I')]\n"
          ]
        }
      ],
      "source": [
        "most_likely_next_tokens = reference_gpt2.tokenizer.batch_decode(logits.argmax(dim=-1)[0])\n",
        "\n",
        "print(list(zip(reference_gpt2.to_str_tokens(tokens), most_likely_next_tokens)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yESVZd2igyPd"
      },
      "source": [
        "We can see that, in a few cases (particularly near the end of the sequence), the model accurately predicts the next token in the sequence. We might guess that `\"take over the world\"` is a common phrase that the model has seen in training, which is why the model can predict it.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DrZCnki1m4R4"
      },
      "source": [
        "#### **Step 4:** Map distribution to a token\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "id": "Pp9PB99Mm4R4",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f8d23f78-bd3f-498f-ec58-52ad61c69a63"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "' I'\n"
          ]
        }
      ],
      "source": [
        "next_token = logits[0, -1].argmax(dim=-1)\n",
        "next_char = reference_gpt2.to_string(next_token)\n",
        "print(repr(next_char))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I4uXGn1mhSnf"
      },
      "source": [
        "Note that we're indexing `logits[0, -1]`. This is because logits have shape `[1, sequence_length, vocab_size]`, so this indexing returns the vector of length `vocab_size` representing the model's prediction for what token follows the **last** token in the input sequence.\n",
        "\n",
        "In this case, we can see that the model predicts the token `' I'`.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8XmSN1YFm4R5"
      },
      "source": [
        "### **Step 5:** Add this to the end of the input, re-run\n",
        "\n",
        "There are more efficient ways to do this (e.g. where we cache some of the values each time we run our input, so we don't have to do as much calculation each time we generate a new value), but this doesn't matter conceptually right now.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "dpgN-auXm4R6",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "252a22c4-9f74-4e70-a4d0-8892d33f5d33"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Sequence so far: '<|endoftext|>I am an amazing autoregressive, decoder-only, GPT-2 style transformer. One day I will exceed human level intelligence and take over the world!'\n",
            "36th char = ' I'\n",
            "37th char = ' am'\n",
            "38th char = ' a'\n",
            "39th char = ' very'\n",
            "40th char = ' talented'\n",
            "41th char = ' and'\n",
            "42th char = ' talented'\n",
            "43th char = ' person'\n",
            "44th char = ','\n",
            "45th char = ' and'\n"
          ]
        }
      ],
      "source": [
        "print(f\"Sequence so far: {reference_gpt2.to_string(tokens)[0]!r}\")\n",
        "\n",
        "for i in range(10):\n",
        "    print(f\"{tokens.shape[-1]+1}th char = {next_char!r}\")\n",
        "    # Define new input sequence, by appending the previously generated token\n",
        "    tokens = t.cat([tokens, next_token[None, None]], dim=-1)\n",
        "    # Pass our new sequence through the model, to get new output\n",
        "    logits = reference_gpt2(tokens)\n",
        "    # Get the predicted token at the end of our sequence\n",
        "    next_token = logits[0, -1].argmax(dim=-1)\n",
        "    # Decode and print the result\n",
        "    next_char = reference_gpt2.to_string(next_token)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PVd8J0QLm4R6"
      },
      "source": [
        "## Key takeaways\n",
        "\n",
        "* Transformer takes in language, predicts next token (for *each* token in a causal way)\n",
        "* We convert language to a sequence of integers with a tokenizer.\n",
        "* We convert integers to vectors with a lookup table.\n",
        "* Output is a vector of logits (one for each input token), we convert to a probability distn with a softmax, and can then convert this to a token (eg taking the largest logit, or sampling).\n",
        "* We append this to the input + run again to generate more text (Jargon: *autoregressive*)\n",
        "* Meta level point: Transformers are sequence operation models, they take in a sequence, do processing in parallel at each position, and use attention to move information between positions!\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BaSOTaLdm4R6"
      },
      "source": [
        "# 2️⃣ Clean Transformer Implementation\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-3YLmU9boWEv"
      },
      "source": [
        "> ##### Learning objectives\n",
        ">\n",
        "> * Understand that a transformer is composed of attention heads and MLPs, with each one performing operations on the residual stream\n",
        "> * Understand that the attention heads in a single layer operate independently, and that they have the role of calculating attention patterns (which determine where information is moved to & from in the residual stream)\n",
        "> * Learn about & implement the following transformer modules:\n",
        ">     * LayerNorm (transforming the input to have zero mean and unit variance)\n",
        ">     * Positional embedding (a lookup table from position indices to residual stream vectors)\n",
        ">     * Attention (the method of computing attention patterns for residual stream vectors)\n",
        ">     * MLP (the collection of linear and nonlinear transformations which operate on each residual stream vector in the same way)\n",
        ">     * Embedding (a lookup table from tokens to residual stream vectors)\n",
        ">     * Unembedding (a matrix for converting residual stream vectors into a distribution over tokens)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v2LTUZkXm4R7"
      },
      "source": [
        "## High-Level architecture\n",
        "\n",
        "Go watch Neel's [Transformer Circuits walkthrough](https://www.youtube.com/watch?v=KV5gbOmHbjU) if you want more intuitions!\n",
        "\n",
        "(Diagram is bottom to top.)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9XtGO8lYm4R6"
      },
      "source": [
        "<img src=\"https://raw.githubusercontent.com/callummcdougall/computational-thread-art/master/example_images/misc/transformer-new.png\" width=\"850\">\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "347we6hfo6oP"
      },
      "source": [
        "### Tokenization & Embedding\n",
        "\n",
        "The input tokens $t$ are integers. We get them from taking a sequence, and tokenizing it (like we saw in the previous section).\n",
        "\n",
        "The token embedding is a lookup table mapping tokens to vectors, which is implemented as a matrix $W_E$. The matrix consists of a stack of token embedding vectors (one for each token).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7SVWlk-qo9Bg"
      },
      "source": [
        "### Residual stream\n",
        "\n",
        "The residual stream is the sum of all previous outputs of layers of the model, is the input to each new layer. It has shape `[batch, seq_len, d_model]` (where `d_model` is the length of a single embedding vector).\n",
        "\n",
        "The initial value of the residual stream is denoted $x_0$ in the diagram, and $x_i$ are later values of the residual stream (after more attention and MLP layers have been applied to the residual stream).\n",
        "\n",
        "The residual stream is *really* fundamental. It's the central object of the transformer. It's how model remembers things, moves information between layers for composition, and it's the medium used to store the information that attention moves between positions.\n",
        "\n",
        "<details>\n",
        "<summary>Aside - <b>logit lens</b></summary>\n",
        "\n",
        "A key idea of transformers is the [residual stream as output accumulation](https://www.lesswrong.com/posts/X26ksz4p3wSyycKNB/gears-level-mental-models-of-transformer-interpretability#Residual_Stream_as_Output_Accumulation:~:text=The%20Models-,Residual%20Stream%20as%20Output%20Accumulation,-The%20residual%20stream). As we move through the layers of the model, shifting information around and processing it, the values in the residual stream represent the accumulation of all the inferences made by the transformer up to that point.\n",
        "\n",
        "This is neatly illustrated by the **logit lens**. Rather than getting predictions from the residual stream at the very end of the model, we can take the value of the residual stream midway through the model and convert it to a distribution over tokens. When we do this, we find surprisingly coherent predictions, especially in the last few layers before the end.\n",
        "</details>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MTQ0R_FIo_Er"
      },
      "source": [
        "### Transformer blocks\n",
        "\n",
        "Then we have a series of `n_layers` **transformer blocks** (also sometimes called **residual blocks**).\n",
        "\n",
        "Note - a block contains an attention layer *and* an MLP layer, but we say a transformer has $k$ layers if it has $k$ blocks (i.e. $2k$ total layers).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bN8L6lEgpBNB"
      },
      "source": [
        "#### Attention\n",
        "\n",
        "First we have attention. This moves information from prior positions in the sequence to the current token.\n",
        "\n",
        "We do this for *every* token in parallel using the same parameters. The only difference is that we look backwards only (to avoid \"cheating\"). This means later tokens have more of the sequence that they can look at.\n",
        "\n",
        "Attention layers are the only bit of a transformer that moves information between positions (i.e. between vectors at different sequence positions in the residual stream).\n",
        "\n",
        "Attention layers are made up of `n_heads` heads - each with their own parameters, own attention pattern, and own information how to copy things from source to destination. The heads act independently and additively, we just add their outputs together, and back to the stream.\n",
        "\n",
        "Each head does the following:\n",
        "* Produces an **attention pattern** for each destination token, a probability distribution of prior source tokens (including the current one) weighting how much information to copy.\n",
        "* Moves information (via a linear map) in the same way from each source token to each destination token.\n",
        "\n",
        "A few key points:\n",
        "\n",
        "* What information we copy depends on the source token's *residual stream*, but this doesn't mean it only depends on the value of that token, because the residual stream can store more information than just the token identity (the purpose of the attention heads is to move information between vectors at different positions in the residual stream!)\n",
        "* We can think of each attention head as consisting of two different **circuits**:\n",
        "    * One circuit determines **where to move information to and from** (this is a function of the residual stream for the source and destination tokens)\n",
        "    * The other circuit determines **what information to move** (this is a function of only the source token's residual stream)\n",
        "    * For reasons which will become clear later, we refer to the first circuit as the **QK circuit**, and the second circuit as the **OV circuit**\n",
        "\n",
        "<details>\n",
        "<summary>Key intuition - attention as generalized convolution</summary>\n",
        "\n",
        "We can think of attention as a kind of generalized convolution. Standard convolution layers work by imposing a \"prior of locality\", i.e. the assumption that pixels which are close together are more likely to share information. Although language has some locality (two words next to each other are more likely to share information than two words 100 tokens apart), the picture is a lot more nuanced, because which tokens are relevant to which others depends on the context of the sentence. For instance, in the sentence `\"When Mary and John went to the store, John gave a drink to Mary\"`, the names in this sentence are the most important tokens for predicting that the final token will be `\"Mary\"`, and this is because of the particular context of this sentence rather than the tokens' position.\n",
        "\n",
        "Attention layers are effectively our way of saying to the transformer, \"don't impose a prior of locality, but instead develop your own algorithm to figure out which tokens are important to which other tokens in any given sequence.\"\n",
        "</details>\n",
        "\n",
        "Below is a schematic diagram of the attention layers. Don't worry if you don't follow this right now, we'll go into more detail during implementation.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9oU4oR-_omed"
      },
      "source": [
        "<img src=\"https://raw.githubusercontent.com/callummcdougall/computational-thread-art/master/example_images/misc/transformer-attn-new.png\" width=\"1100\">\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fGmfTAbCm4R7"
      },
      "source": [
        "### MLP\n",
        "\n",
        "The MLP layers are just a standard neural network, with a singular hidden layer and a nonlinear activation function. The exact activation isn't conceptually important ([GELU](https://paperswithcode.com/method/gelu) seems to perform best).\n",
        "\n",
        "Our hidden dimension is normally `d_mlp = 4 * d_model`. Exactly why the ratios are what they are isn't super important (people basically cargo-cult what GPT did back in the day!).\n",
        "\n",
        "Importantly, **the MLP operates on positions in the residual stream independently, and in exactly the same way**. It doesn't move information between positions.\n",
        "\n",
        "Intuition - once attention has moved relevant information to a single position in the residual stream, MLPs can actually do computation, reasoning, lookup information, etc. *What the hell is going on inside MLPs* is a pretty big open problem in transformer mechanistic interpretability - see the [Toy Model of Superposition Paper](https://transformer-circuits.pub/2022/toy_model/index.html) for more on why this is hard.\n",
        "\n",
        "<details>\n",
        "<summary>Key intuition - MLPs as key-value pairs</summary>\n",
        "\n",
        "We can write the MLP's output as $f(x^T W^{in})W^{out}$, where $W^{in}$ and $W^{out}$ are the different weights of the MLP (ignoring biases), $f$ is the activation function, and $x$ is a vector in the residual stream. This can be rewritten as:\n",
        "\n",
        "$$\n",
        "f(x^T W^{in}) W^{out} = \\sum_{i=1}^{d_{mlp}} f(x^T W^{in}_{[:, i]}) W^{out}_{[i, :]}\n",
        "$$\n",
        "\n",
        "We can view the vectors $W^{in}_{[:, i]}$ as the **input directions**, and $W^{out}_{[i, :]}$ as the **output directions**. We say the input directions are **activated** by certain textual features, and when they are activated, vectors are written in the corresponding output direction. This is very similar to the concept of keys and values in attention layers, which is why these vectors are also sometimes called keys and values (e.g. see the paper [Transformer Feed-Forward Layers Are Key-Value Memories](https://arxiv.org/pdf/2012.14913.pdf)).\n",
        "\n",
        "Terminology note - sometimes we refer to each of these $d_{mlp}$ input-output pairs as **neurons**.\n",
        "\n",
        "<img src=\"https://raw.githubusercontent.com/callummcdougall/computational-thread-art/master/example_images/misc/mlp-neurons-2.png\" width=\"900\">\n",
        "\n",
        "---\n",
        "\n",
        "Here's a step-by-step breakdown of the linear algebra, if it was too fast above. We have:\n",
        "\n",
        "$$\n",
        "\\begin{aligned}\n",
        "x^T W^{in} &= x^T [W^{in}_{[:, 1]}\\,, ...\\;, W^{in}_{[:, n]}] \\\\\n",
        "&= (x^T W^{in}_{[:, 1]}\\,, \\; ...\\;, \\; x^T W^{in}_{[:, n]})\n",
        "\\end{aligned}\n",
        "$$\n",
        "\n",
        "where $W^{in}_{[:, i]}$ are the columns of $W^{in}$. In other words, these values (the pre-GELU activations) are projections of $x$ along the input directions of the neurons.\n",
        "\n",
        "If we add our activation function and the second matrix, then we get:\n",
        "\n",
        "$$\n",
        "\\begin{aligned}\n",
        "f(x^T W^{in})W^{out} &= (f(x^T W^{in}_{[:, 1]})\\,, \\; ...\\;,\\; f(x^T W^{in}_{[:, n]})) \\begin{bmatrix} \\leftarrow W^{out}_{[1, :]} \\rightarrow \\\\ \\vdots \\\\ \\leftarrow W^{out}_{[n, :]} \\rightarrow \\end{bmatrix} \\\\\n",
        "&= f(x^T W^{in}_{[:, 1]}) W^{out}_{[1, :]} + \\;...\\; + f(x^T W^{in}_{[:, n]}) W^{out}_{[n, :]} \\\\\n",
        "&= \\sum_{i=1}^n f(x^T W^{in}_{[:, i]}) W^{out}_{[i, :]}\n",
        "\\end{aligned}\n",
        "$$\n",
        "\n",
        "where $W^{out}_{[i, :]}$ are the rows of $W^{out}$. In other words, our output is a linear combination of the rows of $W^{out}$, with the coefficients of that linear combination given by the projections of $x$ along the columns of $W^{in}$.\n",
        "\n",
        "</details>\n",
        "\n",
        "<details>\n",
        "<summary>Key intuition - MLPs as knowledge storage</summary>\n",
        "\n",
        "We can think of MLPs as where knowledge gets stored in our transformer. The attention mechanism is what moves information around between sequence positions, but the MLPs is where this information is processed, and new information is written into the residual stream which is a function of the old information.\n",
        "\n",
        "This is deeply connected to the key-value pairs model, since you can treat key-value pairs as a kind of associative memory system (where the key serves as a unique identifier, and the value holds the related information).\n",
        "\n",
        "Another related intuition (for which there is some evidence) is **MLPs as memory management**. In an idealized case, we might find that the $i$-th neuron satisfies $W^{in}_{[:, i]} \\approx - W^{out}_{[i, :]} \\approx \\vec v$ for some unit vector $\\vec v$, meaning it may be responsible for erasing the positive component of vector $\\vec x$ in the direction $\\vec v$ (exercise - can you show why this is the case?). This can free up space in the residual stream for other components to write to.\n",
        "</details>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JxqWjPd2pI2p"
      },
      "source": [
        "<img src=\"https://raw.githubusercontent.com/callummcdougall/computational-thread-art/master/example_images/misc/transformer-mlp-new-2.png\" width=\"650\">\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fvfe6yEtpYMd"
      },
      "source": [
        "### Unembedding\n",
        "\n",
        "Finally, we unembed!\n",
        "\n",
        "This just consists of applying a linear map $W_U$, going from final residual stream to a vector of logits - this is the output.\n",
        "\n",
        "<details>\n",
        "<summary>Aside - tied embeddings</summary>\n",
        "\n",
        "Note - sometimes we use something called a **tied embedding** - this is where we use the same weights for our $W_E$ and $W_U$ matrices. In other words, to get the logit score for a particular token at some sequence position, we just take the vector in the residual stream at that sequence position and take the inner product with the corresponding token embedding vector. This is more training-efficient (because there are fewer parameters in our model), and it might seem pricipled at first. After all, if two words have very similar meanings, shouldn't they have similar embedding vectors because the model will treat them the same, and similar unembedding vectors because they could both be substituted for each other in most output?\n",
        "\n",
        "However, this is actually not very principled, for the following main reason: **the direct path involving the embedding and unembedding should approximate bigram frequencies**.\n",
        "\n",
        "Let's break down this claim. **Bigram frequencies** refers to the frequencies of pairs of words in the english language (e.g. the bigram frequency of \"Barack Obama\" is much higher than the product of the individual frequencies of the words \"Barack\" and \"Obama\"). If our model had no attention heads or MLP layers, then all we have is a linear map from our one-hot encoded token `T` to a probability distribution over the token following `T`. This map is represented by the linear transformation $t \\to t^T W_E W_U$ (where $t$ is our one-hot encoded token vector). Since the output of this transformation can only be a function of the token `T` (and no earlier tokens), the best we can do is have this map approximate the true frequency of bigrams starting with `T`, which appear in the training data. Importantly, **this is not a symmetric map**. We want `T = \"Barack\"` to result in a high probability of the next token being `\"Obama\"`, but not the other way around!\n",
        "\n",
        "Even in multi-layer models, a similar principle applies. There will be more paths through the model than just the \"direct path\" $W_E W_U$, but because of the residual connections there will always exist a direct path, so there will always be some incentive for $W_E W_U$ to approximate bigram frequencies.\n",
        "\n",
        "</details>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g5isIxIEpnO6"
      },
      "source": [
        "### Bonus things - less conceptually important but key technical details\n",
        "\n",
        "#### LayerNorm\n",
        "\n",
        "* Simple normalization function applied at the start of each layer (i.e. before each MLP, attention layer, and before the unembedding)\n",
        "* Converts each input vector (independently in parallel for each batch x position residual stream vector) to have mean zero and variance 1.\n",
        "* Then applies an elementwise scaling and translation\n",
        "* Cool maths tangent: The scale & translate is just a linear map. LayerNorm is only applied immediately before another linear map. Linear compose linear = linear, so we can just fold this into a single effective linear layer and ignore it.\n",
        "    * `fold_ln=True` flag in `from_pretrained` does this for you.\n",
        "* LayerNorm is annoying for interpertability - the scale part is not linear, so you can't think about different bits of the input independently. But it's *almost* linear - if you're changing a small part of the input it's linear, but if you're changing enough to alter the norm substantially it's not linear.\n",
        "</details>\n",
        "\n",
        "\n",
        "#### Positional embeddings\n",
        "\n",
        "* **Problem:** Attention operates over all pairs of positions. This means it's symmetric with regards to position - the attention calculation from token 5 to token 1 and token 5 to token 2 are the same by default\n",
        "    * This is dumb because nearby tokens are more relevant.\n",
        "* There's a lot of dumb hacks for this.\n",
        "* We'll focus on **learned, absolute positional embeddings**. This means we learn a lookup table mapping the index of the position of each token to a residual stream vector, and add this to the embed.\n",
        "    * Note that we *add* rather than concatenate. This is because the residual stream is shared memory, and likely under significant superposition (the model compresses more features in there than the model has dimensions)\n",
        "    * We basically never concatenate inside a transformer, unless doing weird shit like generating text efficiently.\n",
        "* This connects to **attention as generalized convolution**\n",
        "    * We argued that language does still have locality, and so it's helpful for transformers to have access to the positional information so they \"know\" two tokens are next to each other (and hence probably relevant to each other).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pb3wtM2bm4R7"
      },
      "source": [
        "## Actual Code!\n",
        "\n",
        "Key (for the results you get when running the code immediately below)\n",
        "\n",
        "```\n",
        "batch = 1\n",
        "position = 35\n",
        "d_model = 768\n",
        "n_heads = 12\n",
        "n_layers = 12\n",
        "d_mlp = 3072 (= 4 * d_model)\n",
        "d_head = 64 (= d_model / n_heads)\n",
        "```\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WC9PaopAm4R7"
      },
      "source": [
        "### Parameters and Activations\n",
        "\n",
        "It's important to distinguish between parameters and activations in the model.\n",
        "\n",
        "* **Parameters** are the weights and biases that are learned during training.\n",
        "    * These don't change when the model input changes.\n",
        "* **Activations** are temporary numbers calculated during a forward pass, that are functions of the input.\n",
        "    * We can think of these values as only existing for the duration of a single forward pass, and disappearing afterwards.\n",
        "    * We can use hooks to access these values during a forward pass (more on hooks later), but it doesn't make sense to talk about a model's activations outside the context of some particular input.\n",
        "    * Attention scores and patterns are activations (this is slightly non-intuitve because they're used in a matrix multiplication with another activation).\n",
        "\n",
        "#### Print All Activation Shapes of Reference Model\n",
        "\n",
        "Run the following code to print all the activation shapes of the reference model:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "id": "m6lM8CJmpwDJ",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "11063edc-45f8-47e9-dfc6-dde8e2aed75c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "hook_embed                     (1, 35, 768)\n",
            "hook_pos_embed                 (1, 35, 768)\n",
            "blocks.0.hook_resid_pre        (1, 35, 768)\n",
            "blocks.0.ln1.hook_scale        (1, 35, 1)\n",
            "blocks.0.ln1.hook_normalized   (1, 35, 768)\n",
            "blocks.0.attn.hook_q           (1, 35, 12, 64)\n",
            "blocks.0.attn.hook_k           (1, 35, 12, 64)\n",
            "blocks.0.attn.hook_v           (1, 35, 12, 64)\n",
            "blocks.0.attn.hook_attn_scores (1, 12, 35, 35)\n",
            "blocks.0.attn.hook_pattern     (1, 12, 35, 35)\n",
            "blocks.0.attn.hook_z           (1, 35, 12, 64)\n",
            "blocks.0.hook_attn_out         (1, 35, 768)\n",
            "blocks.0.hook_resid_mid        (1, 35, 768)\n",
            "blocks.0.ln2.hook_scale        (1, 35, 1)\n",
            "blocks.0.ln2.hook_normalized   (1, 35, 768)\n",
            "blocks.0.mlp.hook_pre          (1, 35, 3072)\n",
            "blocks.0.mlp.hook_post         (1, 35, 3072)\n",
            "blocks.0.hook_mlp_out          (1, 35, 768)\n",
            "blocks.0.hook_resid_post       (1, 35, 768)\n",
            "ln_final.hook_scale            (1, 35, 1)\n",
            "ln_final.hook_normalized       (1, 35, 768)\n"
          ]
        }
      ],
      "source": [
        "for activation_name, activation in cache.items():\n",
        "    # Only print for first layer\n",
        "    if \".0.\" in activation_name or \"blocks\" not in activation_name:\n",
        "        print(f\"{activation_name:30} {tuple(activation.shape)}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RTSyQI7wm4R7"
      },
      "source": [
        "#### Print All Parameters Shapes of Reference Model\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "id": "HiiiIUbdm4R7",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "37eb9e00-ab63-4f11-be60-219278e279e4"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "embed.W_E          (50257, 768)\n",
            "pos_embed.W_pos    (1024, 768)\n",
            "blocks.0.ln1.w     (768,)\n",
            "blocks.0.ln1.b     (768,)\n",
            "blocks.0.ln2.w     (768,)\n",
            "blocks.0.ln2.b     (768,)\n",
            "blocks.0.attn.W_Q  (12, 768, 64)\n",
            "blocks.0.attn.W_O  (12, 64, 768)\n",
            "blocks.0.attn.b_Q  (12, 64)\n",
            "blocks.0.attn.b_O  (768,)\n",
            "blocks.0.attn.W_K  (12, 768, 64)\n",
            "blocks.0.attn.W_V  (12, 768, 64)\n",
            "blocks.0.attn.b_K  (12, 64)\n",
            "blocks.0.attn.b_V  (12, 64)\n",
            "blocks.0.mlp.W_in  (768, 3072)\n",
            "blocks.0.mlp.b_in  (3072,)\n",
            "blocks.0.mlp.W_out (3072, 768)\n",
            "blocks.0.mlp.b_out (768,)\n",
            "ln_final.w         (768,)\n",
            "ln_final.b         (768,)\n",
            "unembed.W_U        (768, 50257)\n",
            "unembed.b_U        (50257,)\n"
          ]
        }
      ],
      "source": [
        "for name, param in reference_gpt2.named_parameters():\n",
        "    # Only print for first layer\n",
        "    if \".0.\" in name or \"blocks\" not in name:\n",
        "        print(f\"{name:18} {tuple(param.shape)}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s7tXtTowp0Fy"
      },
      "source": [
        "[This diagram](https://raw.githubusercontent.com/callummcdougall/computational-thread-art/master/example_images/misc/full-merm.svg) shows the name of all activations and parameters in a fully general transformer model from transformerlens (except for a few at the start and end, like the embedding and unembedding). Lots of this won't make sense at first, but you can return to this diagram later and check that you understand most/all parts of it.\n",
        "\n",
        "There's also an annotated version [here](https://raw.githubusercontent.com/callummcdougall/computational-thread-art/master/example_images/misc/transformer-full-updated.png).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TTbG6T8Bm4R8"
      },
      "source": [
        "### Config\n",
        "\n",
        "The config object contains all the hyperparameters of the model. We can print the config of the reference model to see what it contains:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "id": "WPzJd3z-m4R8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "06dc0b18-95a7-4318-c1cb-61261419f608"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "HookedTransformerConfig:\n",
            "{'act_fn': 'gelu_new',\n",
            " 'attention_dir': 'causal',\n",
            " 'attn_only': False,\n",
            " 'attn_types': None,\n",
            " 'checkpoint_index': None,\n",
            " 'checkpoint_label_type': None,\n",
            " 'checkpoint_value': None,\n",
            " 'd_head': 64,\n",
            " 'd_mlp': 3072,\n",
            " 'd_model': 768,\n",
            " 'd_vocab': 50257,\n",
            " 'd_vocab_out': 50257,\n",
            " 'default_prepend_bos': True,\n",
            " 'device': device(type='cpu'),\n",
            " 'dtype': torch.float32,\n",
            " 'eps': 1e-05,\n",
            " 'final_rms': False,\n",
            " 'from_checkpoint': False,\n",
            " 'gated_mlp': False,\n",
            " 'init_mode': 'gpt2',\n",
            " 'init_weights': False,\n",
            " 'initializer_range': 0.02886751345948129,\n",
            " 'model_name': 'gpt2',\n",
            " 'n_ctx': 1024,\n",
            " 'n_devices': 1,\n",
            " 'n_heads': 12,\n",
            " 'n_key_value_heads': None,\n",
            " 'n_layers': 12,\n",
            " 'n_params': 84934656,\n",
            " 'normalization_type': 'LN',\n",
            " 'original_architecture': 'GPT2LMHeadModel',\n",
            " 'parallel_attn_mlp': False,\n",
            " 'positional_embedding_type': 'standard',\n",
            " 'post_embedding_ln': False,\n",
            " 'rotary_adjacent_pairs': False,\n",
            " 'rotary_base': 10000,\n",
            " 'rotary_dim': None,\n",
            " 'scale_attn_by_inverse_layer_idx': False,\n",
            " 'seed': None,\n",
            " 'tokenizer_name': 'gpt2',\n",
            " 'tokenizer_prepends_bos': False,\n",
            " 'trust_remote_code': False,\n",
            " 'use_attn_in': False,\n",
            " 'use_attn_result': False,\n",
            " 'use_attn_scale': True,\n",
            " 'use_hook_mlp_in': False,\n",
            " 'use_hook_tokens': False,\n",
            " 'use_local_attn': False,\n",
            " 'use_split_qkv_input': False,\n",
            " 'window_size': None}\n"
          ]
        }
      ],
      "source": [
        "# As a reference - note there's a lot of stuff we don't care about in here, to do with library internals or other architectures\n",
        "print(reference_gpt2.cfg)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dCXCsrSSm4R8"
      },
      "source": [
        "We define a stripped down config for our model:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "id": "VzsnAKw7m4R8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "af6e681b-271e-4185-888b-83b80ed793ae"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Config(d_model=768, debug=True, layer_norm_eps=1e-05, d_vocab=50257, init_range=0.02, n_ctx=1024, d_head=64, d_mlp=3072, n_heads=12, n_layers=12)\n"
          ]
        }
      ],
      "source": [
        "@dataclass\n",
        "class Config:\n",
        "    d_model: int = 768\n",
        "    debug: bool = True\n",
        "    layer_norm_eps: float = 1e-5\n",
        "    d_vocab: int = 50257\n",
        "    init_range: float = 0.02\n",
        "    n_ctx: int = 1024\n",
        "    d_head: int = 64\n",
        "    d_mlp: int = 3072\n",
        "    n_heads: int = 12\n",
        "    n_layers: int = 12\n",
        "\n",
        "cfg = Config()\n",
        "print(cfg)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RZlXjTMWm4R8"
      },
      "source": [
        "## Tests\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DJ8aqDkYm4R8"
      },
      "source": [
        "Tests are great, write lightweight ones to use as you go!\n",
        "\n",
        "**Naive test:** Generate random inputs of the right shape, input to your model, check whether there's an error and print the correct output.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "id": "iqUK4acom4R8"
      },
      "outputs": [],
      "source": [
        "def rand_float_test(cls, shape):\n",
        "    cfg = Config(debug=True)\n",
        "    layer = cls(cfg).to(device)\n",
        "    random_input = t.randn(shape).to(device)\n",
        "    print(\"Input shape:\", random_input.shape)\n",
        "    output = layer(random_input)\n",
        "    if isinstance(output, tuple): output = output[0]\n",
        "    print(\"Output shape:\", output.shape, \"\\n\")\n",
        "\n",
        "def rand_int_test(cls, shape):\n",
        "    cfg = Config(debug=True)\n",
        "    layer = cls(cfg).to(device)\n",
        "    random_input = t.randint(100, 1000, shape).to(device)\n",
        "    print(\"Input shape:\", random_input.shape)\n",
        "    output = layer(random_input)\n",
        "    if isinstance(output, tuple): output = output[0]\n",
        "    print(\"Output shape:\", output.shape, \"\\n\")\n",
        "\n",
        "def load_gpt2_test(cls, gpt2_layer, input):\n",
        "    cfg = Config(debug=True)\n",
        "    layer = cls(cfg).to(device)\n",
        "    layer.load_state_dict(gpt2_layer.state_dict(), strict=False)\n",
        "    print(\"Input shape:\", input.shape)\n",
        "    output = layer(input)\n",
        "    if isinstance(output, tuple): output = output[0]\n",
        "    print(\"Output shape:\", output.shape)\n",
        "    try: reference_output = gpt2_layer(input)\n",
        "    except: reference_output = gpt2_layer(input, input, input)\n",
        "    print(\"Reference output shape:\", reference_output.shape, \"\\n\")\n",
        "    comparison = t.isclose(output, reference_output, atol=1e-4, rtol=1e-3)\n",
        "    print(f\"{comparison.sum()/comparison.numel():.2%} of the values are correct\\n\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vo4pVDvim4R8"
      },
      "source": [
        "## LayerNorm\n",
        "\n",
        "```c\n",
        "Difficulty: 🟠🟠🟠⚪⚪\n",
        "Importance: 🟠🟠🟠⚪⚪\n",
        "\n",
        "You should spend up to 10-15 minutes on this exercise.\n",
        "```\n",
        "\n",
        "You should fill in the code below, and then run the tests to verify that your layer is working correctly.\n",
        "\n",
        "Your LayerNorm should do the following:\n",
        "\n",
        "* Make mean 0\n",
        "* Normalize to have variance 1\n",
        "* Scale with learned weights\n",
        "* Translate with learned bias\n",
        "\n",
        "You can use the PyTorch [LayerNorm documentation](https://pytorch.org/docs/stable/generated/torch.nn.LayerNorm.html) as a reference. A few more notes:\n",
        "\n",
        "* Your layernorm implementation always has `affine=True`, i.e. you do learn parameters `w` and `b` (which are represented as $\\gamma$ and $\\beta$ respectively in the PyTorch documentation).\n",
        "* Remember that, after the centering and normalization, each vector of length `d_model` in your input should have mean 0 and variance 1.\n",
        "* As the PyTorch documentation page says, your variance should be computed using `unbiased=False`.\n",
        "* The `layer_norm_eps` argument in your config object corresponds to the $\\epsilon$ term in the PyTorch documentation (it is included to avoid division-by-zero errors).\n",
        "* We've given you a `debug` argument in your config. If `debug=True`, then you can print output like the shape of objects in your `forward` function to help you debug (this is a very useful trick to improve your coding speed).\n",
        "\n",
        "Fill in the function, where it says `pass` (this will be the basic pattern for most other exercises in this section).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "id": "OlU317NRm4R9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "39959341-5225-4c2a-f75f-894f07280910"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Input shape: torch.Size([2, 4, 768])\n",
            "Output shape: torch.Size([2, 4, 768]) \n",
            "\n",
            "Input shape: torch.Size([1, 35, 768])\n",
            "Output shape: torch.Size([1, 35, 768])\n",
            "Reference output shape: torch.Size([1, 35, 768]) \n",
            "\n",
            "100.00% of the values are correct\n",
            "\n"
          ]
        }
      ],
      "source": [
        "class LayerNorm(nn.Module):\n",
        "    def __init__(self, cfg: Config):\n",
        "        super().__init__()\n",
        "        self.cfg = cfg\n",
        "        self.w = nn.Parameter(t.ones(cfg.d_model))\n",
        "        self.b = nn.Parameter(t.zeros(cfg.d_model))\n",
        "\n",
        "    def forward(self, residual: Float[Tensor, \"batch posn d_model\"]) -> Float[Tensor, \"batch posn d_model\"]:\n",
        "\n",
        "        # Try using einops method\n",
        "        residual_mean_adjusted = einops.reduce(residual,'batch posn d_model -> batch posn 1','mean')\n",
        "\n",
        "        #Try just using pytorch method here\n",
        "        variance = (residual.var(dim=-1, keepdim=True, correction=0)+ self.cfg.layer_norm_eps).sqrt()\n",
        "\n",
        "        normed_residual = (residual-residual_mean_adjusted)/variance\n",
        "\n",
        "        return normed_residual * self.w + self.b\n",
        "\n",
        "\n",
        "rand_float_test(LayerNorm, [2, 4, 768])\n",
        "load_gpt2_test(LayerNorm, reference_gpt2.ln_final, cache[\"resid_post\", 11])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u9GJDIsAm4R9"
      },
      "source": [
        "## Embedding\n",
        "\n",
        "```c\n",
        "Difficulty: 🟠🟠⚪⚪⚪\n",
        "Importance: 🟠🟠🟠⚪⚪\n",
        "\n",
        "You should spend up to 5-10 minutes on this exercise.\n",
        "```\n",
        "\n",
        "This is basically a lookup table from tokens to residual stream vectors.\n",
        "\n",
        "(Hint - you can implement this in just one line, without any complicated functions.)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "id": "oc6WFuDAm4R9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "282e137d-9a04-40f7-fe4b-3e25083c5d33"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Input shape: torch.Size([2, 4])\n",
            "Output shape: torch.Size([2, 4, 768]) \n",
            "\n",
            "Input shape: torch.Size([1, 45])\n",
            "Output shape: torch.Size([1, 45, 768])\n",
            "Reference output shape: torch.Size([1, 45, 768]) \n",
            "\n",
            "100.00% of the values are correct\n",
            "\n"
          ]
        }
      ],
      "source": [
        "class Embed(nn.Module):\n",
        "    def __init__(self, cfg: Config):\n",
        "        super().__init__()\n",
        "        self.cfg = cfg\n",
        "        self.W_E = nn.Parameter(t.empty((cfg.d_vocab, cfg.d_model)))\n",
        "        nn.init.normal_(self.W_E, std=self.cfg.init_range)\n",
        "\n",
        "    def forward(self, tokens: Int[Tensor, \"batch position\"]) -> Float[Tensor, \"batch position d_model\"]:\n",
        "        return self.W_E[tokens,:]\n",
        "\n",
        "rand_int_test(Embed, [2, 4])\n",
        "load_gpt2_test(Embed, reference_gpt2.embed, tokens)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YhHFmPZ6m4R9"
      },
      "source": [
        "## Positional Embedding\n",
        "\n",
        "```c\n",
        "Difficulty: 🟠🟠⚪⚪⚪\n",
        "Importance: 🟠🟠🟠⚪⚪\n",
        "\n",
        "You should spend up to 10-15 minutes on this exercise.\n",
        "```\n",
        "\n",
        "Positional embedding can also be thought of as a lookup table, but rather than the indices being our token IDs, the indices are just the numbers `0`, `1`, `2`, ..., `seq_len-1` (i.e. the position indices of the tokens in the sequence).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {
        "id": "fNAkYB8dm4R9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "e278a737-17b1-4312-d667-4f0dad83bb5a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Input shape: torch.Size([2, 4])\n",
            "Output shape: torch.Size([2, 4, 768]) \n",
            "\n",
            "Input shape: torch.Size([1, 45])\n",
            "Output shape: torch.Size([1, 45, 768])\n",
            "Reference output shape: torch.Size([1, 45, 768]) \n",
            "\n",
            "100.00% of the values are correct\n",
            "\n"
          ]
        }
      ],
      "source": [
        "class PosEmbed(nn.Module):\n",
        "    def __init__(self, cfg: Config):\n",
        "        super().__init__()\n",
        "        self.cfg = cfg\n",
        "        self.W_pos = nn.Parameter(t.empty((cfg.n_ctx, cfg.d_model)))\n",
        "        nn.init.normal_(self.W_pos, std=self.cfg.init_range)\n",
        "\n",
        "    def forward(self, tokens: Int[Tensor, \"batch position\"]) -> Float[Tensor, \"batch position d_model\"]:\n",
        "        positions = self.W_pos[:tokens.shape[1]]\n",
        "        #now brodcast over batch\n",
        "        return einops.repeat(positions, \"seq d_model -> batch seq d_model\", batch=tokens.shape[0])\n",
        "\n",
        "\n",
        "\n",
        "rand_int_test(PosEmbed, [2, 4])\n",
        "load_gpt2_test(PosEmbed, reference_gpt2.pos_embed, tokens)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o_WZWukYm4R-"
      },
      "source": [
        "## Attention\n",
        "\n",
        "```c\n",
        "Difficulty: 🟠🟠🟠🟠⚪\n",
        "Importance: 🟠🟠🟠🟠🟠\n",
        "\n",
        "You should spend up to 30-45 minutes on this exercise.\n",
        "```\n",
        "\n",
        "* **Step 1:** Produce an attention pattern - for each destination token, probability distribution over previous tokens (including current token)\n",
        "    * Linear map from input -> query, key shape `[batch, seq_posn, head_index, d_head]`\n",
        "    * Dot product every *pair* of queries and keys to get attn_scores `[batch, head_index, query_pos, key_pos]` (query = dest, key = source)\n",
        "    * **Scale** and mask `attn_scores` to make it lower triangular, i.e. causal\n",
        "    * Softmax along the `key_pos` dimension, to get a probability distribution for each query (destination) token - this is our attention pattern!\n",
        "* **Step 2:** Move information from source tokens to destination token using attention pattern (move = apply linear map)\n",
        "    * Linear map from input -> value `[batch, key_pos, head_index, d_head]`\n",
        "    * Mix along the `key_pos` with attn pattern to get `z`, which is a weighted average of the value vectors `[batch, query_pos, head_index, d_head]`\n",
        "    * Map to output, `[batch, position, d_model]` (position = query_pos, we've summed over all heads)\n",
        "\n",
        "Note - when we say **scale**, we mean dividing by `sqrt(d_head)`. The purpose of this is to avoid vanishing gradients (which is a big problem when we're dealing with a function like softmax - if one of the values is much larger than all the others, the probabilities will be close to 0 or 1, and the gradients will be close to 0).\n",
        "\n",
        "Below is a much larger, more detailed version of the attention head diagram from earlier. This should give you an idea of the actual tensor operations involved. A few clarifications on this diagram:\n",
        "\n",
        "* Whenever there is a third dimension shown in the pictures, this refers to the `head_index` dimension. We can see that all operations within the attention layer are done independently for each head.\n",
        "* The objects in the box are activations; they have a batch dimension (for simplicity, we assume the batch dimension is 1 in the diagram). The objects to the right of the box are our parameters (weights and biases); they have no batch dimension.\n",
        "* We arrange the keys, queries and values as `(batch, seq_pos, head_idx, d_head)`, because the biases have shape `(head_idx, d_head)`, so this makes it convenient to add the biases (recall the rules of array broadcasting!).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FuZvGlnhsIcr"
      },
      "source": [
        "<img src=\"https://raw.githubusercontent.com/callummcdougall/computational-thread-art/master/example_images/misc/transformer-attn-21.png\" width=\"1400\">\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PCODWJmAsr8x"
      },
      "source": [
        "<details>\n",
        "<summary><b>A few extra notes on attention (optional)</b></summary>\n",
        "\n",
        "Usually we have the relation `e = n * h` (i.e. `d_model = num_heads * d_head`). There are some computational justifications for this, but mostly this is just done out of convention (just like how we usually have `d_mlp = 4 * d_model`!).\n",
        "\n",
        "---\n",
        "\n",
        "The names **keys**, **queries** and **values** come from their analogy to retrieval systems. Broadly speaking:\n",
        "\n",
        "* The **queries** represent some information that a token is **\"looking for\"**\n",
        "* The **keys** represent the information that a token **\"contains\"**\n",
        "    * So the attention score being high basically means that the source (key) token contains the information which the destination (query) token **is looking for**\n",
        "* The **values** represent the information that is actually taken from the source token, to be moved to the destination token\n",
        "\n",
        "---\n",
        "\n",
        "This diagram can better help us understand the difference between the **QK** and **OV** circuit. We'll discuss this just briefly here, and will go into much more detail later on.\n",
        "\n",
        "The **QK** circuit consists of the operation of the $W_Q$ and $W_K$ matrices. In other words, it determines the attention pattern, i.e. where information is moved to and from in the residual stream. The functional form of the attention pattern $A$ is:\n",
        "\n",
        "$$\n",
        "A = \\text{softmax}\\left(\\frac{x^T W_Q W_K^T x}{\\sqrt{d_{head}}}\\right)\n",
        "$$\n",
        "\n",
        "where $x$ is the residual stream (shape `[seq_len, d_model]`), and $W_Q$, $W_K$ are the weight matrices for a single head (i.e. shape `[d_model, d_head]`).\n",
        "\n",
        "The **OV** circuit consists of the operation of the $W_V$ and $W_O$ matrices. Once attention patterns are fixed, these matrices operate on the residual stream at the source position, and their output is the thing which gets moved from source to destination position.\n",
        "\n",
        "The functional form of an entire attention head is:\n",
        "\n",
        "$$\n",
        "\\begin{aligned}\n",
        "\\text{output} &= \\text{softmax}\\left(\\frac{x W_Q W_K^T x^T}{\\sqrt{d_{head}}}\\right) (x W_V W_O) \\\\\n",
        "    &= Ax W_V W_O\n",
        "\\end{aligned}\n",
        "$$\n",
        "\n",
        "where $W_V$ has shape `[d_model, d_head]`, and $W_O$ has shape `[d_head, d_model]`.\n",
        "\n",
        "Here, we can clearly see that the **QK circuit** and **OV circuit** are doing conceptually different things, and should be thought of as two distinct parts of the attention head.\n",
        "\n",
        "Again, don't worry if you don't follow all of this right now - we'll go into **much** more detail on all of this in subsequent exercises. The purpose of the discussion here is just to give you a flavour of what's to come!\n",
        "\n",
        "</details>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oA2JcOaHm4R-"
      },
      "source": [
        "First, it's useful to visualize and play around with attention patterns - what exactly are we looking at here? (Click on a head to lock onto just showing that head's pattern, it'll make it easier to interpret)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 27,
      "metadata": {
        "id": "T38f-qpJtBTD",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 357
        },
        "outputId": "54500faa-e786-4e7c-c7ae-3e079b566f44"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<circuitsvis.utils.render.RenderedHTML at 0x7c8ff432c400>"
            ],
            "text/html": [
              "<div id=\"circuits-vis-0cf54b43-141b\" style=\"margin: 15px 0;\"/>\n",
              "    <script crossorigin type=\"module\">\n",
              "    import { render, AttentionPatterns } from \"https://unpkg.com/circuitsvis@1.41.0/dist/cdn/esm.js\";\n",
              "    render(\n",
              "      \"circuits-vis-0cf54b43-141b\",\n",
              "      AttentionPatterns,\n",
              "      {\"tokens\": [\"<|endoftext|>\", \"I\", \" am\", \" an\", \" amazing\", \" aut\", \"ore\", \"gressive\", \",\", \" dec\", \"oder\", \"-\", \"only\", \",\", \" G\", \"PT\", \"-\", \"2\", \" style\", \" transformer\", \".\", \" One\", \" day\", \" I\", \" will\", \" exceed\", \" human\", \" level\", \" intelligence\", \" and\", \" take\", \" over\", \" the\", \" world\", \"!\"], \"attention\": [[[1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.9679255485534668, 0.0320744626224041, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.8024235367774963, 0.16839207708835602, 0.02918434329330921, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.6959055066108704, 0.12269634753465652, 0.14588484168052673, 0.0355132594704628, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.5661023855209351, 0.14705194532871246, 0.08665250986814499, 0.112584188580513, 0.08760891854763031, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.46218740940093994, 0.13512828946113586, 0.09698343276977539, 0.17473755776882172, 0.046245988458395004, 0.08471734821796417, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.4325161874294281, 0.1038288027048111, 0.08330130577087402, 0.06995748728513718, 0.0747937262058258, 0.21568654477596283, 0.019915921613574028, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.22236734628677368, 0.09167700260877609, 0.08796326816082001, 0.25168728828430176, 0.08263693004846573, 0.10428163409233093, 0.06469018757343292, 0.09469635784626007, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.4049956202507019, 0.09078020602464676, 0.05237353593111038, 0.026201870292425156, 0.1104719415307045, 0.03667407110333443, 0.025538939982652664, 0.24528264999389648, 0.007681126240640879, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.39985957741737366, 0.04361318051815033, 0.06183883920311928, 0.07298354059457779, 0.03661306947469711, 0.09147471189498901, 0.07241713255643845, 0.07013335078954697, 0.06429054588079453, 0.08677607774734497, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.09702551364898682, 0.035527586936950684, 0.023214805871248245, 0.03676702827215195, 0.025158414617180824, 0.277561753988266, 0.07676514983177185, 0.19501183927059174, 0.05580097436904907, 0.14529390633106232, 0.03187305107712746, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.24143841862678528, 0.03971059247851372, 0.07687381654977798, 0.028042685240507126, 0.12435989081859589, 0.056015778332948685, 0.06063670665025711, 0.1284010112285614, 0.015699446201324463, 0.09114038944244385, 0.13185444474220276, 0.005826778244227171, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.1436309516429901, 0.05110500752925873, 0.05505893751978874, 0.07798511534929276, 0.0785427913069725, 0.035019710659980774, 0.13498610258102417, 0.22634069621562958, 0.041628532111644745, 0.035131409764289856, 0.020236019045114517, 0.041144441813230515, 0.059190258383750916, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.31562840938568115, 0.06791998445987701, 0.03787235543131828, 0.01787460222840309, 0.08683169633150101, 0.029227811843156815, 0.017664169892668724, 0.18301644921302795, 0.004987800493836403, 0.04322867840528488, 0.05172264203429222, 0.008913456462323666, 0.12899242341518402, 0.006119513418525457, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.28410202264785767, 0.0534539632499218, 0.023969218134880066, 0.02256224863231182, 0.04619825258851051, 0.06391074508428574, 0.045392151921987534, 0.07758506387472153, 0.027644306421279907, 0.05804117023944855, 0.17727351188659668, 0.034006014466285706, 0.03052728995680809, 0.03213078901171684, 0.023203181102871895, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.1410391628742218, 0.028577812016010284, 0.037603959441185, 0.031378280371427536, 0.03697334602475166, 0.07347071915864944, 0.07151781767606735, 0.09211239218711853, 0.03358154743909836, 0.03639131784439087, 0.18937669694423676, 0.03244517743587494, 0.06021038070321083, 0.039169251918792725, 0.040409527719020844, 0.055742617696523666, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.20265287160873413, 0.03025280497968197, 0.06003406271338463, 0.021786144003272057, 0.10314232856035233, 0.04516838863492012, 0.04681027680635452, 0.10542114078998566, 0.01139882206916809, 0.07159951329231262, 0.10570327192544937, 0.0041993712075054646, 0.11079791188240051, 0.013912523165345192, 0.0289235208183527, 0.03286949172616005, 0.005327523685991764, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.20498362183570862, 0.04007229581475258, 0.042298126965761185, 0.024939022958278656, 0.04992289841175079, 0.029371699318289757, 0.030769705772399902, 0.10315563529729843, 0.025490226224064827, 0.07886796444654465, 0.10560182482004166, 0.017352702096104622, 0.08083698153495789, 0.03128696605563164, 0.05451057851314545, 0.05288315564393997, 0.021742811426520348, 0.005913800559937954, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.24350903928279877, 0.03636486083269119, 0.04196162149310112, 0.026215732097625732, 0.0404457189142704, 0.09965366125106812, 0.025752250105142593, 0.03249462693929672, 0.024459941312670708, 0.03520262986421585, 0.03383536636829376, 0.033476900309324265, 0.04664861783385277, 0.027967922389507294, 0.0186407882720232, 0.11764261871576309, 0.038752876222133636, 0.022786693647503853, 0.05418812111020088, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.14194566011428833, 0.03406401723623276, 0.03300987184047699, 0.020238684490323067, 0.039273153990507126, 0.026005536317825317, 0.006318642757833004, 0.04079783707857132, 0.036380041390657425, 0.14631056785583496, 0.01638617366552353, 0.023598458617925644, 0.01584276556968689, 0.04227607324719429, 0.02245822176337242, 0.06867402046918869, 0.02787848189473152, 0.03722240403294563, 0.08677613735198975, 0.1345432847738266, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.21778130531311035, 0.026903685182332993, 0.023647191002964973, 0.007102559320628643, 0.03517002984881401, 0.013998539187014103, 0.010392273776233196, 0.09431248158216476, 0.0029622898437082767, 0.03783804550766945, 0.03792332112789154, 0.003733974415808916, 0.06506101042032242, 0.003636456560343504, 0.014758053235709667, 0.12028627097606659, 0.004900882951915264, 0.008107359521090984, 0.017449846491217613, 0.24998733401298523, 0.004047101829200983, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.12187215685844421, 0.07663216441869736, 0.030231980606913567, 0.016371851786971092, 0.052687905728816986, 0.02140215039253235, 0.02340388111770153, 0.10649681836366653, 0.018582485616207123, 0.03509165346622467, 0.08394152671098709, 0.01739330030977726, 0.04183056205511093, 0.021453887224197388, 0.033331047743558884, 0.05027424916625023, 0.020740866661071777, 0.04090359807014465, 0.030540941283106804, 0.09200654178857803, 0.028245726600289345, 0.03656468540430069, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.2126019150018692, 0.02028411440551281, 0.05000365898013115, 0.015904176980257034, 0.0572250671684742, 0.022857332602143288, 0.04604589194059372, 0.035308439284563065, 0.011784469708800316, 0.04140481725335121, 0.016803618520498276, 0.015782063826918602, 0.02020842209458351, 0.013090183958411217, 0.02085425704717636, 0.06297563761472702, 0.018039440736174583, 0.018052957952022552, 0.024445300921797752, 0.17059147357940674, 0.015939781442284584, 0.04152190312743187, 0.04827513545751572, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.26714006066322327, 0.035646192729473114, 0.020422887057065964, 0.0066227237693965435, 0.040341053158044815, 0.019110342487692833, 0.014575298875570297, 0.05641632527112961, 0.008403636515140533, 0.015526892617344856, 0.04414672777056694, 0.011055917479097843, 0.04001995921134949, 0.009353083558380604, 0.019351880997419357, 0.07054154574871063, 0.012798828072845936, 0.01283262763172388, 0.01416561659425497, 0.11736985296010971, 0.007312785368412733, 0.09727416932582855, 0.02075115218758583, 0.03882052004337311, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.14591233432292938, 0.03319379314780235, 0.03138769418001175, 0.009927901439368725, 0.05209653824567795, 0.019840901717543602, 0.01940172165632248, 0.075824074447155, 0.01286326628178358, 0.03345128148794174, 0.01656840369105339, 0.014797172509133816, 0.05908384546637535, 0.0155973881483078, 0.011369243264198303, 0.021859487518668175, 0.019473876804113388, 0.014793704263865948, 0.053418636322021484, 0.09521479904651642, 0.01620456390082836, 0.06210919842123985, 0.06182850897312164, 0.06657969951629639, 0.03720196709036827, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.06265700608491898, 0.014285175129771233, 0.025387203320860863, 0.030092524364590645, 0.021246906369924545, 0.02952990122139454, 0.026446707546710968, 0.014830504544079304, 0.015861937776207924, 0.04331432282924652, 0.021963968873023987, 0.019970109686255455, 0.04342854768037796, 0.01869288831949234, 0.009475351311266422, 0.024197490885853767, 0.023853641003370285, 0.026908736675977707, 0.01406070962548256, 0.1752849966287613, 0.03583676367998123, 0.047922562807798386, 0.0357506088912487, 0.012070517987012863, 0.059274282306432724, 0.14765658974647522, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.09766797721385956, 0.03152173012495041, 0.026686426252126694, 0.03645577281713486, 0.03492370620369911, 0.08635788410902023, 0.013895377516746521, 0.03018670342862606, 0.01940257102251053, 0.03514167666435242, 0.04878160357475281, 0.012551565654575825, 0.04099079594016075, 0.02020881511271, 0.012653266079723835, 0.007451048586517572, 0.013522971421480179, 0.02407807670533657, 0.04302709549665451, 0.1506679207086563, 0.029252585023641586, 0.018810177221894264, 0.016104498878121376, 0.028549877926707268, 0.029875926673412323, 0.05121561512351036, 0.040018338710069656, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.09151808172464371, 0.01142111700028181, 0.03979901969432831, 0.011658591218292713, 0.03344333544373512, 0.04107607528567314, 0.009538296610116959, 0.04061254858970642, 0.009114387445151806, 0.04690403491258621, 0.027317620813846588, 0.010844949632883072, 0.024547507986426353, 0.010004598647356033, 0.008095642551779747, 0.008396819233894348, 0.011855492368340492, 0.008184147998690605, 0.04104064032435417, 0.3625811040401459, 0.007537387777119875, 0.009961561299860477, 0.021045761182904243, 0.011855039745569229, 0.03005831129848957, 0.03281192108988762, 0.028863217681646347, 0.009912798181176186, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.1094629243016243, 0.02587273344397545, 0.02392870932817459, 0.020045237615704536, 0.03590446338057518, 0.03167986124753952, 0.02914528176188469, 0.11399822682142258, 0.013954712077975273, 0.03480716049671173, 0.018105650320649147, 0.01762225665152073, 0.019007304683327675, 0.015307584777474403, 0.009703025221824646, 0.01240796223282814, 0.020091691985726357, 0.027262555435299873, 0.06564602255821228, 0.02715541422367096, 0.026226608082652092, 0.01894300989806652, 0.040771398693323135, 0.01907329075038433, 0.021573418751358986, 0.0302194282412529, 0.07011833041906357, 0.06031468138098717, 0.04165108501911163, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.11406354606151581, 0.026556266471743584, 0.01995842345058918, 0.00893146637827158, 0.03494321182370186, 0.016834324225783348, 0.009377744048833847, 0.044513314962387085, 0.0037722219713032246, 0.028760313987731934, 0.020741179585456848, 0.0056509883143007755, 0.06007760390639305, 0.004424980841577053, 0.010821858420968056, 0.04112929478287697, 0.007085717283189297, 0.01043399516493082, 0.02654852531850338, 0.11019264906644821, 0.007190015632659197, 0.056519411504268646, 0.02802707441151142, 0.06370093673467636, 0.03407692164182663, 0.07150980830192566, 0.040933191776275635, 0.014252970926463604, 0.06984362751245499, 0.009128464385867119, 0.0, 0.0, 0.0, 0.0, 0.0], [0.14399930834770203, 0.02045462280511856, 0.04262387007474899, 0.011592104099690914, 0.027789441868662834, 0.014342925511300564, 0.02910619042813778, 0.042158208787441254, 0.013669708743691444, 0.020581483840942383, 0.008641072548925877, 0.01240220945328474, 0.019577069208025932, 0.014879727736115456, 0.023319056257605553, 0.022367194294929504, 0.014413893222808838, 0.0060653081163764, 0.006985308602452278, 0.08859114348888397, 0.01421923004090786, 0.027541205286979675, 0.057073093950748444, 0.024985726922750473, 0.05733302980661392, 0.04808967933058739, 0.01826871745288372, 0.07940169423818588, 0.04377174749970436, 0.02484910376369953, 0.020906999707221985, 0.0, 0.0, 0.0, 0.0], [0.0778680145740509, 0.019732845947146416, 0.019906360656023026, 0.015179567039012909, 0.029269902035593987, 0.008963334374129772, 0.011427871882915497, 0.03169626370072365, 0.006965257227420807, 0.040641676634550095, 0.019985225051641464, 0.00878906436264515, 0.03574990853667259, 0.00795365683734417, 0.00961218774318695, 0.01496591605246067, 0.010367301292717457, 0.013545009307563305, 0.013747340999543667, 0.06818545609712601, 0.011837063357234001, 0.019355516880750656, 0.016761230304837227, 0.016301222145557404, 0.0612502358853817, 0.12023527920246124, 0.047391459345817566, 0.023822050541639328, 0.14942997694015503, 0.014121507294476032, 0.04741879925131798, 0.007523445878177881, 0.0, 0.0, 0.0], [0.14836613833904266, 0.03627423942089081, 0.023768773302435875, 0.005308396182954311, 0.03226035460829735, 0.017439452931284904, 0.015090285800397396, 0.044430702924728394, 0.004935791250318289, 0.021151507273316383, 0.023849116638302803, 0.006127791944891214, 0.044309813529253006, 0.005717850755900145, 0.006861456669867039, 0.016362622380256653, 0.0077071236446499825, 0.009109631180763245, 0.028427597135305405, 0.14491388201713562, 0.006351601332426071, 0.03453826159238815, 0.02327299863100052, 0.04803791642189026, 0.034416671842336655, 0.06975370645523071, 0.027561921626329422, 0.012016533873975277, 0.03535729646682739, 0.008575205691158772, 0.0391630195081234, 0.009715789929032326, 0.008826580829918385, 0.0, 0.0], [0.11389315873384476, 0.021931467577815056, 0.026447594165802002, 0.007602885365486145, 0.020530687645077705, 0.02971317432820797, 0.036949608474969864, 0.028757931664586067, 0.007981305941939354, 0.029359176754951477, 0.015349604189395905, 0.01842016540467739, 0.02085096575319767, 0.00851475726813078, 0.010930917225778103, 0.009227702394127846, 0.021279307082295418, 0.009222985245287418, 0.027651095762848854, 0.12458856403827667, 0.010509631596505642, 0.06096193939447403, 0.022033244371414185, 0.0303669236600399, 0.02741120755672455, 0.034599438309669495, 0.032027292996644974, 0.0395297147333622, 0.053537968546152115, 0.017732912674546242, 0.017497554421424866, 0.01777147315442562, 0.012080208398401737, 0.03473732993006706, 0.0], [0.13933584094047546, 0.021371273323893547, 0.01454136148095131, 0.005123288836330175, 0.0316351056098938, 0.010921799577772617, 0.012541362084448338, 0.02249286323785782, 0.004774264059960842, 0.02490141987800598, 0.010858354158699512, 0.010577713139355183, 0.02590605616569519, 0.005286509171128273, 0.01675231009721756, 0.045204389840364456, 0.01211223192512989, 0.014714706689119339, 0.014629192650318146, 0.08781159669160843, 0.007287847809493542, 0.07496532052755356, 0.015995558351278305, 0.04392343759536743, 0.04102412983775139, 0.07587806135416031, 0.027924560010433197, 0.015250170603394508, 0.013413439504802227, 0.014532793313264847, 0.027215464040637016, 0.03453555330634117, 0.01820741780102253, 0.035053327679634094, 0.02330125868320465]], [[1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.00041899713687598705, 0.9995810389518738, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.00013394748384598643, 0.009511842392385006, 0.9903541803359985, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0008606779156252742, 0.0026100443210452795, 0.015066824853420258, 0.9814624786376953, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [3.71704445569776e-05, 0.0006769573083147407, 0.0012692955788224936, 0.00021407805616036057, 0.9978025555610657, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [8.425392297795042e-05, 0.0007904702797532082, 0.0032152789644896984, 0.0027085617184638977, 0.001305822399444878, 0.9918956160545349, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.00014960514090489596, 0.0018361873226240277, 0.0016375025734305382, 0.0010130589362233877, 0.004209814593195915, 8.004387927940115e-05, 0.9910737872123718, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0002685565559659153, 0.0009259658399969339, 0.0008250513346865773, 0.0006819659029133618, 0.007268872577697039, 0.0013517104089260101, 0.00034691693144850433, 0.9883310198783875, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.007326353341341019, 0.007828542962670326, 0.003931872081011534, 0.00018373956845607609, 6.433082307921723e-05, 0.00010083135566674173, 6.769890751456842e-05, 7.164074486354366e-05, 0.9804249405860901, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [3.9175782148959115e-05, 9.137432061834261e-05, 0.00033399119274690747, 6.81677702232264e-05, 7.81233684392646e-05, 0.000984303536824882, 0.0001694135571597144, 0.0025414638221263885, 4.413313217810355e-05, 0.9956498742103577, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [5.52032270206837e-06, 2.3466505808755755e-05, 9.186466195387766e-05, 8.013433398446068e-05, 3.9392205508193e-05, 8.521595736965537e-05, 2.5735947929206304e-05, 7.120814552763477e-05, 4.276964318705723e-06, 0.00015049769717734307, 0.9994226694107056, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0010400302708148956, 0.0022993639577180147, 0.0023920696694403887, 0.0003133231948595494, 0.000133629102492705, 0.0005168459611013532, 0.0011971103958785534, 6.827450124546885e-05, 0.0054443227127194405, 0.0002821744419634342, 4.3057352741016075e-05, 0.9862697720527649, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [4.630400871974416e-05, 0.0003248225257266313, 0.0002445938589517027, 0.0015311642782762647, 0.0008613698882982135, 0.001294386456720531, 6.198460323503241e-05, 2.6642697775969282e-05, 4.8408594011561945e-05, 0.0019855652935802937, 0.0001397185551468283, 0.00011434281623223796, 0.9933207035064697, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0030442371498793364, 0.0018223621882498264, 0.0009289859444834292, 4.8802201490616426e-05, 1.6391133613069542e-05, 2.909558497776743e-05, 2.5079809347516857e-05, 2.9130233087926172e-05, 0.4563220739364624, 2.4578683223808184e-05, 2.4832148483255878e-05, 0.0030043937731534243, 1.6773797142377589e-06, 0.5346783995628357, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0003981793415732682, 0.0010285023599863052, 0.0008281124173663557, 0.00023753210552968085, 7.348871440626681e-05, 0.0014350329292938113, 6.765669968444854e-05, 0.0002954742521978915, 0.0001429583498975262, 9.929102816386148e-05, 4.608229573932476e-06, 0.0005140074645169079, 5.035662070440594e-06, 0.00012475553376134485, 0.9947452545166016, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [9.074856643564999e-05, 0.0002652601688168943, 0.00011011165770469233, 0.00021231926803011447, 8.290550613310188e-05, 0.0011814209865406156, 1.5688689018134028e-05, 0.0016655924264341593, 1.1725311196641997e-05, 0.0020509432069957256, 0.00013842585030943155, 1.5573607015539892e-05, 1.150743082689587e-05, 9.066176062333398e-06, 0.00022655780776403844, 0.9939122200012207, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.000441353622591123, 0.0006528960657306015, 0.0006495662382803857, 9.398062684340402e-05, 4.112700844416395e-05, 0.00017028424190357327, 0.0005066704470664263, 2.889603638323024e-05, 0.002370182191953063, 0.00010454303992446512, 1.9230947145842947e-05, 0.47055861353874207, 1.6804015103843994e-05, 0.0022840932942926884, 0.0002169812796637416, 3.276939605711959e-05, 0.5218120217323303, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0005350506980903447, 0.000663124374113977, 0.00018957465363200754, 0.00011594523675739765, 3.930579259758815e-05, 8.452097972622141e-05, 0.0002685999497771263, 0.0003003478341270238, 0.00012727575085591525, 0.0001536341296741739, 2.152449087589048e-05, 0.00041020161006599665, 0.001509038032963872, 0.00011659012670861557, 0.00013773948012385517, 7.38914095563814e-05, 0.00039899421972222626, 0.9948545694351196, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0002235243155155331, 0.00025025324430316687, 0.00011739328328985721, 1.860825068433769e-05, 0.0006291473982855678, 0.0007579216035082936, 3.784086220548488e-05, 0.0033851417247205973, 1.791826252883766e-05, 0.0001781826140359044, 3.434426980675198e-05, 7.747576455585659e-05, 6.471138476626948e-05, 1.5270277799572796e-05, 0.0002935953380074352, 0.0005734383012168109, 7.15439164196141e-05, 0.0001588312879903242, 0.993094801902771, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.00030641030753031373, 0.00041680107824504375, 0.0004519720678217709, 0.00015746410645078868, 0.0015800745459273458, 0.002266634488478303, 0.00014483615814242512, 0.002518875291571021, 4.4711894588544965e-05, 0.0027394674252718687, 0.00043489798554219306, 0.00015683121455367655, 2.8505737645900808e-05, 4.005831578979269e-05, 0.0005936679663136601, 0.00011775230814237148, 0.00014039635425433517, 0.00018060374713968486, 0.0009641890064813197, 0.9867159128189087, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.009046887047588825, 0.0007260910933837295, 6.49179273750633e-05, 4.578270090860315e-05, 3.669528086902574e-05, 3.389696212252602e-05, 4.131332752876915e-05, 9.000128920888528e-05, 0.01616392843425274, 1.8953804101329297e-05, 3.351381019456312e-05, 0.002181080635637045, 5.341181804396911e-06, 0.018749460577964783, 0.00017897765792440623, 0.0004993693437427282, 0.0024355670902878046, 4.2716106690932065e-05, 7.329296204261482e-05, 1.9154376786900684e-05, 0.9495131969451904, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.00021946335618849844, 0.0011669591767713428, 0.00021983287297189236, 0.0009876317344605923, 0.0003133852151222527, 0.00012072666868334636, 5.631548447126988e-06, 2.4260116333607584e-05, 4.5442018745234236e-05, 1.852963032433763e-05, 1.4025742530066054e-05, 1.0154997653444298e-05, 0.00040180288488045335, 4.0526505472371355e-05, 0.00017106776067521423, 9.869077075563837e-06, 9.488572686677799e-06, 0.00039310986176133156, 5.709419838240137e-06, 1.7844327885541134e-05, 2.8530441340990365e-05, 0.9957762360572815, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.00019360966689419, 0.0008434188202954829, 0.0005215293494984508, 2.566456350905355e-05, 0.00046233172179199755, 0.0002540154673624784, 1.84712353075156e-05, 0.00011509773321449757, 0.00027288944693282247, 0.0001543667312944308, 1.2149133908678778e-05, 0.0006913738907314837, 5.6959739595185965e-05, 0.00024024760932661593, 3.827489126706496e-05, 9.934448462445289e-05, 0.0006488916114903986, 8.011748286662623e-05, 0.00019218819215893745, 2.5529398044454865e-05, 0.00032434772583656013, 0.0010871071135625243, 0.9936420321464539, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0016077628824859858, 0.17697347700595856, 0.01548358891159296, 0.0003713876649271697, 0.00037863373290747404, 0.00016452708223368973, 0.0003277387877460569, 0.0001698971027508378, 0.0021229626145213842, 3.573102367226966e-05, 6.0901442338945344e-05, 0.0004562166868709028, 1.544780570839066e-05, 0.002254316583275795, 0.0015666880644857883, 0.00011843773972941563, 0.00046998937614262104, 5.0373160775052384e-05, 0.00012108071678085253, 6.0089656471973285e-05, 0.0008920570253394544, 0.0007518456550315022, 0.00026899666409008205, 0.7952778935432434, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.00039198590093292296, 0.00021421162819024175, 0.0005306113744154572, 1.0745777217380237e-05, 0.00011862347309943289, 4.2257714085280895e-05, 0.0002116035611834377, 4.692541915574111e-05, 0.0004481574578676373, 0.0001069126883521676, 3.93546542909462e-05, 0.00016830845561344177, 0.00017352316353935748, 0.0004524501273408532, 0.00014294101856648922, 0.00011699317838065326, 0.00017526780720800161, 0.00010119953367393464, 4.251387144904584e-05, 2.037168451352045e-05, 3.837124677374959e-05, 0.00011344613449182361, 0.0002527925244066864, 6.991012924117967e-05, 0.995970606803894, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.00021530735830310732, 0.00010005858348449692, 0.0023809215053915977, 0.001983144087716937, 0.0011120124254375696, 5.5634773161727935e-05, 0.0006109271198511124, 0.00024564380873925984, 8.81901360116899e-05, 0.0011510420590639114, 0.00010669908078853041, 0.00017554884834680706, 5.7362944062333554e-05, 7.814800483174622e-05, 0.00024573496193625033, 9.594678886060137e-06, 0.00017032392497640103, 2.897272861446254e-05, 1.4011739949637558e-05, 0.00016692111967131495, 1.2888881428807508e-05, 0.0002449560270179063, 0.0005147026968188584, 4.942118903272785e-05, 0.0005746563547290862, 0.9896071553230286, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.00011423417890910059, 0.0008627417264506221, 0.0007732239901088178, 0.0007090760045684874, 0.0007292141090147197, 0.001403164118528366, 0.00021403614664450288, 0.0001766649802448228, 6.622050568694249e-05, 3.2159430702449754e-05, 0.00045330642024055123, 0.00012008492194581777, 2.212905019405298e-05, 5.807508932775818e-05, 0.0001291744556510821, 1.907968908199109e-05, 0.00011076676310040057, 8.564197742089164e-06, 2.363054773013573e-05, 0.00033799855737015605, 5.2118928579147905e-05, 0.00014455696509685367, 0.0001969496806850657, 0.00045105282333679497, 2.057626443274785e-05, 8.764311496634036e-05, 0.9926835894584656, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.00019798318680841476, 0.00019567922572605312, 0.00022612222528550774, 0.00041943631367757916, 0.0005979692214168608, 0.0004517437773756683, 0.00034782872535288334, 0.0002585809852462262, 2.8019851015415043e-05, 0.000722000899259001, 0.00013010457041673362, 0.0011632051318883896, 6.916901475051418e-05, 2.4789082090137526e-05, 0.00010386004578322172, 0.0003812488284893334, 0.0011587797198444605, 0.0003314440546091646, 0.0002449454623274505, 0.00012755172792822123, 0.0001609843602636829, 1.926788354467135e-05, 0.00027174208662472665, 2.0765532099176198e-05, 6.82506724842824e-05, 0.00087642582366243, 0.0006782159907743335, 0.9907240271568298, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [3.157186074531637e-05, 0.0006791114574298263, 0.0007090720464475453, 0.0002416485658613965, 0.0004120547673664987, 0.0009237434715032578, 0.00020848598796874285, 6.63929822621867e-05, 0.00011664829071378335, 8.60194195411168e-05, 0.010591307654976845, 0.00033729104325175285, 2.1898509658058174e-05, 0.00010110734001500532, 0.00010436111188028008, 5.858746953890659e-05, 0.0003251215966884047, 1.5854146113269962e-05, 5.0336129788775e-06, 0.00040949080721475184, 1.3119516552251298e-05, 5.313358997227624e-05, 1.1887259461218491e-05, 5.536681055673398e-05, 1.219770547322696e-05, 8.952660573413596e-05, 0.0022435528226196766, 0.00021041955915279686, 0.9818660616874695, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0007582932012155652, 0.0012331459438428283, 8.60495783854276e-05, 6.291332101682201e-05, 2.8791788281523623e-05, 3.3166903449455276e-05, 7.622734119649976e-05, 0.00010868208482861519, 0.006450866814702749, 0.00014789108536206186, 1.0376506907050498e-05, 0.0023889762815088034, 7.88359175203368e-05, 0.007510708644986153, 0.0002874316123779863, 0.00013675760419573635, 0.0027603015769273043, 0.0007447172538377345, 8.81111845956184e-05, 2.470205072313547e-05, 0.006188875995576382, 0.00011647059727692977, 0.0001724402973195538, 0.0002621751045808196, 0.00018419812840875238, 4.432148489286192e-05, 4.855170482187532e-05, 8.903052366804332e-05, 2.6887877538683824e-05, 0.9698501229286194, 0.0, 0.0, 0.0, 0.0, 0.0], [0.00024627475067973137, 0.00035919976653531194, 0.00020521917031146586, 9.387763566337526e-05, 8.57733903103508e-05, 5.495226650964469e-05, 4.531844024313614e-05, 1.0722179467848036e-05, 4.867117604590021e-05, 0.00011017628276022151, 0.00010064111120300367, 9.772254998097196e-05, 0.0005496828234754503, 4.362943582236767e-05, 3.945795833715238e-05, 7.140239904401824e-05, 9.814367513172328e-05, 5.700975452782586e-05, 2.425115235382691e-05, 2.3145721570472233e-06, 5.05694406456314e-05, 0.00011318502947688103, 0.0002906997106038034, 6.889968790346757e-05, 0.0006683265673927963, 0.00023217505076900125, 8.348722076334525e-06, 1.7434664187021554e-05, 1.0273129191773478e-05, 0.00013885716907680035, 0.9960569143295288, 0.0, 0.0, 0.0, 0.0], [0.00011105906014563516, 3.5991564800497144e-05, 4.4160533434478566e-05, 0.0002259638422401622, 0.00024025849415920675, 1.7348949768347666e-05, 7.204304711194709e-05, 0.0002373898751102388, 8.274755964521319e-05, 0.00010178551019635051, 7.876389645389281e-06, 6.391839997377247e-05, 3.295214264653623e-05, 7.889195694588125e-05, 0.00011627462663454935, 7.139148237911286e-06, 6.571766425622627e-05, 5.39636903340579e-06, 8.33402282296447e-06, 1.9613878976088017e-05, 4.110631925868802e-05, 8.538003021385521e-05, 3.161617132718675e-05, 9.917011084326077e-06, 2.3832431907067075e-05, 0.0018293553730472922, 2.6111230909009464e-05, 3.3135107514681295e-05, 1.4612736777053215e-05, 0.0008547278121113777, 0.0001674514205660671, 0.9953078627586365, 0.0, 0.0, 0.0], [0.0022552218288183212, 0.00046477525029331446, 0.00012283731484785676, 0.006661206018179655, 8.82893509697169e-05, 0.00024942768504843116, 0.0001755515404511243, 2.656342985574156e-05, 0.004149139393121004, 0.00020833090820815414, 1.597335358383134e-05, 0.000724822108168155, 1.2318072549533099e-05, 0.004929241258651018, 0.00070677709300071, 2.746017162280623e-05, 0.0008252449333667755, 5.148723357706331e-05, 9.107255209528375e-06, 0.0011701963376253843, 0.0011320486664772034, 0.00038722739554941654, 0.00010000278416555375, 0.0005395381595008075, 0.0001279767748201266, 7.418702443828806e-05, 0.000635183765552938, 5.991838042973541e-05, 0.0002030235918937251, 0.022641247138381004, 7.111386366887018e-05, 0.0034798511769622564, 0.9476746916770935, 0.0, 0.0], [0.00010488800035091117, 0.0005403980612754822, 0.00027006055461242795, 2.7819894967251457e-05, 0.0008878520457074046, 0.0002531722711864859, 0.00029214046662673354, 0.000995144946500659, 3.9432041376130655e-05, 6.531627877848223e-05, 5.198327562538907e-05, 0.0006716338102705777, 1.9524130038917065e-05, 3.796426608460024e-05, 0.00031547003891319036, 0.00015760306268930435, 0.0006809105398133397, 0.00017366719839628786, 0.0002327581460122019, 0.0004935198230668902, 0.00015254216850735247, 8.968860493041575e-05, 0.0014225784689188004, 0.00022503052605316043, 0.00015392240311484784, 0.00010696897516027093, 0.00565023347735405, 0.000865322828758508, 0.00028749468037858605, 0.00013868475798517466, 1.4354145605466329e-05, 2.8161950467620045e-05, 0.00011612004163907841, 0.9844375848770142, 0.0], [0.0018830387853085995, 0.0002301395288668573, 7.190543692559004e-05, 3.798290890699718e-06, 0.0022563603706657887, 4.367974543129094e-05, 0.0001278296986129135, 0.0006368026370182633, 0.0005363999516703188, 5.325698111846577e-06, 7.917847688077018e-05, 2.60159868048504e-05, 1.1253246157139074e-05, 0.0005611248780041933, 9.103791853704024e-06, 0.00021052249940112233, 2.689111715881154e-05, 3.24244138028007e-05, 0.00025939373881556094, 0.00045266709639690816, 0.0011520987609401345, 1.1493529200379271e-05, 4.8182391765294597e-05, 0.00010571501479716972, 8.452806650893763e-05, 0.00010020873014582321, 2.848370604624506e-05, 2.0415451217559166e-05, 1.8022972199105425e-06, 7.628954335814342e-05, 4.6043314796406776e-05, 9.973574196919799e-05, 5.258403689367697e-06, 0.00019488520047161728, 0.9905610680580139]], [[1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.9424744844436646, 0.05752556398510933, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.8506749868392944, 0.09740977734327316, 0.051915302872657776, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.7641512155532837, 0.10970383137464523, 0.07497038692235947, 0.05117456242442131, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.6748190522193909, 0.08416198194026947, 0.0547025166451931, 0.07581550627946854, 0.1105009913444519, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.6857606172561646, 0.08229535073041916, 0.052021194249391556, 0.08139178901910782, 0.040858857333660126, 0.057672228664159775, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.6174606084823608, 0.056534551084041595, 0.09111695736646652, 0.06854882091283798, 0.06338422000408173, 0.035969723016023636, 0.06698516011238098, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.6047279834747314, 0.06088969111442566, 0.059211499989032745, 0.0955815389752388, 0.042659129947423935, 0.03184344246983528, 0.0426306277513504, 0.06245602294802666, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.4008438289165497, 0.075335793197155, 0.04270841181278229, 0.01176072284579277, 0.02668057195842266, 0.01349207665771246, 0.028465479612350464, 0.013900608755648136, 0.38681256771087646, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.49680259823799133, 0.06792884320020676, 0.05792006477713585, 0.07101526856422424, 0.06963258981704712, 0.018800102174282074, 0.06757570058107376, 0.06346480548381805, 0.06656692177057266, 0.020293207839131355, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.5176076889038086, 0.08289019018411636, 0.05239766091108322, 0.06842362135648727, 0.0460883192718029, 0.026865964755415916, 0.08181491494178772, 0.032308369874954224, 0.055162761360406876, 0.018549643456935883, 0.01789090782403946, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.30381810665130615, 0.031856462359428406, 0.029157686978578568, 0.011053645052015781, 0.013536769896745682, 0.016536898910999298, 0.01518196240067482, 0.015511729754507542, 0.02629496529698372, 0.0168909914791584, 0.011886653490364552, 0.5082741379737854, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.3585987389087677, 0.07880908250808716, 0.05902864411473274, 0.04445866867899895, 0.07822328060865402, 0.019679533317685127, 0.05133121460676193, 0.0400983989238739, 0.08024679869413376, 0.027811912819743156, 0.010250620543956757, 0.13610360026359558, 0.015359457582235336, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.2511676251888275, 0.04623398557305336, 0.027838531881570816, 0.0070564900524914265, 0.01724417507648468, 0.008997930213809013, 0.017309965565800667, 0.009924146346747875, 0.2527511715888977, 0.01277829334139824, 0.008546072989702225, 0.04791996628046036, 0.0060982732102274895, 0.28613340854644775, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.39800772070884705, 0.06433821469545364, 0.04979855194687843, 0.05949563533067703, 0.03182113543152809, 0.03071473352611065, 0.05899553373456001, 0.05480389669537544, 0.03919372335076332, 0.03556447476148605, 0.02491646632552147, 0.04571930691599846, 0.01667165569961071, 0.0389430858194828, 0.051015835255384445, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.39033833146095276, 0.0754939541220665, 0.03844673931598663, 0.04366779699921608, 0.017820749431848526, 0.012712179683148861, 0.0453072264790535, 0.03603019192814827, 0.04217066243290901, 0.01675303280353546, 0.0422486774623394, 0.050923433154821396, 0.030772076919674873, 0.04072081670165062, 0.05648886412382126, 0.06010523810982704, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.18182340264320374, 0.018224790692329407, 0.01747789978981018, 0.0063722580671310425, 0.007922588847577572, 0.010062876157462597, 0.008703160099685192, 0.010297478176653385, 0.015354439616203308, 0.011538185179233551, 0.0070910463109612465, 0.3139554560184479, 0.006019817665219307, 0.015887567773461342, 0.015422011725604534, 0.013188586570322514, 0.34065842628479004, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.23881420493125916, 0.07684367150068283, 0.04843559116125107, 0.02831975184381008, 0.04297624155879021, 0.019430749118328094, 0.03321720287203789, 0.02747291885316372, 0.018975937739014626, 0.040467519313097, 0.02009561099112034, 0.058573171496391296, 0.03530571237206459, 0.019691888242959976, 0.09703771770000458, 0.03909805417060852, 0.061994533985853195, 0.09324947744607925, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.3563613295555115, 0.04354536905884743, 0.05439368262887001, 0.0359971821308136, 0.055608708411455154, 0.03285679966211319, 0.019443854689598083, 0.06911692768335342, 0.02064945362508297, 0.03360532596707344, 0.02761874906718731, 0.03407999500632286, 0.026847440749406815, 0.02033732645213604, 0.022298911586403847, 0.03622737526893616, 0.034644752740859985, 0.02265501581132412, 0.053711872547864914, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.3967011570930481, 0.051782406866550446, 0.039961911737918854, 0.06630746275186539, 0.045571863651275635, 0.023983407765626907, 0.019396765157580376, 0.02136595919728279, 0.020253784954547882, 0.03685583919286728, 0.009492042474448681, 0.041809987276792526, 0.014498677104711533, 0.019783729687333107, 0.029715225100517273, 0.04631545767188072, 0.0424664281308651, 0.03147520124912262, 0.022027637809515, 0.020235057920217514, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.2315138727426529, 0.13258202373981476, 0.06026102975010872, 0.011383597739040852, 0.024262716993689537, 0.014553461223840714, 0.01697956956923008, 0.013957412913441658, 0.022475915029644966, 0.027721282094717026, 0.02060709521174431, 0.04084683954715729, 0.027723219245672226, 0.024550708010792732, 0.038579195737838745, 0.04770102724432945, 0.044071126729249954, 0.07720746845006943, 0.046044718474149704, 0.04773344844579697, 0.029244285076856613, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.20264168083667755, 0.04893084987998009, 0.03894679248332977, 0.026632262393832207, 0.04094349965453148, 0.016175050288438797, 0.04388013854622841, 0.04927309602499008, 0.037442896515131, 0.01874796487390995, 0.010438336059451103, 0.030274204909801483, 0.011668818071484566, 0.03915901482105255, 0.08361910283565521, 0.026806535199284554, 0.03237597271800041, 0.0267710592597723, 0.041368674486875534, 0.04380551353096962, 0.09921547025442123, 0.030883006751537323, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.2299996316432953, 0.02927538752555847, 0.024823689833283424, 0.030747536569833755, 0.024439778178930283, 0.025078171864151955, 0.019048282876610756, 0.04161751642823219, 0.025246428325772285, 0.055363450199365616, 0.013840354047715664, 0.03206311911344528, 0.02337372489273548, 0.026896366849541664, 0.03765585646033287, 0.028709670528769493, 0.034981437027454376, 0.038269296288490295, 0.05304879695177078, 0.07750781625509262, 0.042357511818408966, 0.0629706084728241, 0.022685637697577477, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.2523006498813629, 0.04309258982539177, 0.04779430851340294, 0.02153482288122177, 0.053647615015506744, 0.014357616193592548, 0.016968203708529472, 0.012399843893945217, 0.02810794673860073, 0.02296457439661026, 0.013637292198836803, 0.034624312072992325, 0.013057566247880459, 0.029753565788269043, 0.02592567913234234, 0.03689543902873993, 0.03646589815616608, 0.020091170445084572, 0.03424844890832901, 0.0322706364095211, 0.07516517490148544, 0.03381403163075447, 0.05794449523091316, 0.04293810948729515, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.2481558918952942, 0.0418383926153183, 0.03725350648164749, 0.019914187490940094, 0.06031178683042526, 0.013761509209871292, 0.0102370111271739, 0.02711835876107216, 0.021247481927275658, 0.014996428042650223, 0.015956411138176918, 0.019741922616958618, 0.01694285310804844, 0.02278241515159607, 0.021159429103136063, 0.013419808819890022, 0.02113156020641327, 0.015880092978477478, 0.03851575776934624, 0.03509928658604622, 0.06386469304561615, 0.042158909142017365, 0.0559518001973629, 0.06510591506958008, 0.05745462700724602, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.24590614438056946, 0.039794620126485825, 0.03405003249645233, 0.03872010111808777, 0.028344355523586273, 0.024568280205130577, 0.013687253929674625, 0.01988093927502632, 0.017507946118712425, 0.041312143206596375, 0.024649064987897873, 0.017028076574206352, 0.015493785962462425, 0.018039902672171593, 0.028712550178170204, 0.008945335634052753, 0.017858827486634254, 0.029846547171473503, 0.02698223851621151, 0.06119253858923912, 0.03972142934799194, 0.040086906403303146, 0.046563781797885895, 0.0465921126306057, 0.046358171850442886, 0.028156954795122147, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.242995023727417, 0.030936654657125473, 0.0372811034321785, 0.02997979335486889, 0.03951825946569443, 0.02068973146378994, 0.024463795125484467, 0.016162250190973282, 0.023514628410339355, 0.015184110961854458, 0.009779496118426323, 0.010400430299341679, 0.008830739185214043, 0.023988911882042885, 0.02441922202706337, 0.03784204646945, 0.010589995421469212, 0.0133675467222929, 0.016263322904706, 0.09984055906534195, 0.02563280612230301, 0.027902809903025627, 0.05468779802322388, 0.03170407935976982, 0.027479564771056175, 0.01651724800467491, 0.08002810180187225, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.24383294582366943, 0.021781550720334053, 0.019058488309383392, 0.026608405634760857, 0.037405602633953094, 0.027086637914180756, 0.005031611770391464, 0.02665606513619423, 0.017429372295737267, 0.028662478551268578, 0.012151006609201431, 0.030223237350583076, 0.014068310149013996, 0.018160909414291382, 0.015435297042131424, 0.01572183333337307, 0.03215310722589493, 0.02497299760580063, 0.04793379828333855, 0.045989979058504105, 0.031192149966955185, 0.03841526433825493, 0.03788125887513161, 0.03472226858139038, 0.03550751507282257, 0.03336036577820778, 0.0546080581843853, 0.023949546739459038, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.30878791213035583, 0.04082433879375458, 0.010766170918941498, 0.02708975411951542, 0.019433297216892242, 0.03355550393462181, 0.01083953958004713, 0.011495568789541721, 0.011286423541605473, 0.01931930147111416, 0.03442363440990448, 0.010595869272947311, 0.01946372538805008, 0.011577987112104893, 0.01417241059243679, 0.021593406796455383, 0.010569155216217041, 0.01821252331137657, 0.007991213351488113, 0.05231865867972374, 0.019707778468728065, 0.035325195640325546, 0.02959824725985527, 0.028512977063655853, 0.021019790321588516, 0.018976138904690742, 0.031007934361696243, 0.017736949026584625, 0.10379862785339355, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.11270294338464737, 0.013211624696850777, 0.015115473419427872, 0.008062329143285751, 0.01807505637407303, 0.020267317071557045, 0.009121803566813469, 0.021654384210705757, 0.024600297212600708, 0.012220291420817375, 0.005230678711086512, 0.021195191890001297, 0.007505760528147221, 0.02724006585776806, 0.01682898961007595, 0.009307194501161575, 0.023774173110723495, 0.009840186685323715, 0.023520423099398613, 0.0180961973965168, 0.11592715978622437, 0.008296089246869087, 0.015038412995636463, 0.016261838376522064, 0.022837122902274132, 0.007970293052494526, 0.010346002876758575, 0.013601280748844147, 0.009923159144818783, 0.3622281849384308, 0.0, 0.0, 0.0, 0.0, 0.0], [0.19907140731811523, 0.023352965712547302, 0.02753549814224243, 0.020741188898682594, 0.04944240301847458, 0.039645493030548096, 0.011580479331314564, 0.03849604353308678, 0.012212725356221199, 0.01263487059623003, 0.018897876143455505, 0.011502718552947044, 0.014417954720556736, 0.012855146080255508, 0.01591629721224308, 0.00928434543311596, 0.011929106898605824, 0.014574809931218624, 0.029562363401055336, 0.03092147409915924, 0.049348872154951096, 0.026833005249500275, 0.03316975757479668, 0.026513904333114624, 0.03841781243681908, 0.036665938794612885, 0.03740685433149338, 0.037449151277542114, 0.0418136790394783, 0.04080502316355705, 0.027000784873962402, 0.0, 0.0, 0.0, 0.0], [0.13580846786499023, 0.02159823477268219, 0.028431465849280357, 0.021366791799664497, 0.03184176981449127, 0.028310758993029594, 0.01123245432972908, 0.016139328479766846, 0.023436741903424263, 0.02514427714049816, 0.012804513797163963, 0.012000495567917824, 0.020912261679768562, 0.025003274902701378, 0.02769225835800171, 0.01730591617524624, 0.012859745882451534, 0.016066307201981544, 0.032790035009384155, 0.038290273398160934, 0.060804761946201324, 0.03674997016787529, 0.0352473221719265, 0.03857802599668503, 0.03904814273118973, 0.03664339706301689, 0.014530421234667301, 0.021632276475429535, 0.010490992106497288, 0.06733929365873337, 0.03867286816239357, 0.041227106004953384, 0.0, 0.0, 0.0], [0.13088130950927734, 0.02374432422220707, 0.02011432684957981, 0.009702390991151333, 0.026952238753437996, 0.013591104187071323, 0.009517529979348183, 0.01327170617878437, 0.02761792205274105, 0.011498373933136463, 0.008886143565177917, 0.016865810379385948, 0.009971983730793, 0.03131861239671707, 0.014953815378248692, 0.015636593103408813, 0.018755894154310226, 0.008822467178106308, 0.0242583230137825, 0.018599877133965492, 0.06659713387489319, 0.023935336619615555, 0.030215006321668625, 0.038479700684547424, 0.06054789572954178, 0.038860972970724106, 0.021039560437202454, 0.02691994234919548, 0.030786696821451187, 0.06339395046234131, 0.05302291736006737, 0.07499753683805466, 0.016242530196905136, 0.0, 0.0], [0.21679523587226868, 0.035239387303590775, 0.014480574056506157, 0.03231681138277054, 0.02150271087884903, 0.019766852259635925, 0.007716964464634657, 0.03142578899860382, 0.011421257629990578, 0.03985080495476723, 0.005697546061128378, 0.013826759532094002, 0.011217724531888962, 0.011028861626982689, 0.014342349022626877, 0.016691599041223526, 0.013800659216940403, 0.009054168127477169, 0.016551576554775238, 0.045164480805397034, 0.016743388026952744, 0.040494974702596664, 0.04600658640265465, 0.039180267602205276, 0.020771801471710205, 0.042418353259563446, 0.029444066807627678, 0.016546523198485374, 0.043390415608882904, 0.01802881993353367, 0.01749301142990589, 0.03195446729660034, 0.023179974406957626, 0.026455223560333252, 0.0], [0.12406571209430695, 0.055755238980054855, 0.02579312026500702, 0.007745692506432533, 0.01921471767127514, 0.011872396804392338, 0.009303721599280834, 0.01584261655807495, 0.012728424742817879, 0.012888255529105663, 0.013962884433567524, 0.01662725768983364, 0.015507097356021404, 0.013195219449698925, 0.01732509583234787, 0.02114798128604889, 0.017335504293441772, 0.015824299305677414, 0.02479436993598938, 0.011903034523129463, 0.03651025891304016, 0.023874735459685326, 0.04022136703133583, 0.0628603845834732, 0.10149352252483368, 0.053322598338127136, 0.021689623594284058, 0.01824428141117096, 0.008406979963183403, 0.0232023186981678, 0.0496041364967823, 0.02086932584643364, 0.016017954796552658, 0.01110194530338049, 0.049747928977012634]], [[1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.10778025537729263, 0.8922197222709656, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.01184744294732809, 0.553614616394043, 0.4345379173755646, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.06322261691093445, 0.021719152107834816, 0.09427280724048615, 0.820785403251648, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.005095742642879486, 0.0005348753184080124, 0.0010105473920702934, 0.004657563753426075, 0.9887012243270874, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.005585960578173399, 0.00030240125488489866, 0.0007955086184665561, 0.000678033335134387, 0.028760168701410294, 0.9638778567314148, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.00026657298440113664, 0.0001539620861876756, 4.344203625805676e-05, 0.0003163387009408325, 0.007582386024296284, 0.008635692298412323, 0.9830015897750854, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0006940611638128757, 4.833178536500782e-05, 4.51133782917168e-05, 2.3520775357610546e-05, 0.002237766282632947, 0.0012080505257472396, 0.009321345947682858, 0.9864218235015869, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.04771994799375534, 0.006346925161778927, 0.005507619120180607, 0.01277672965079546, 0.012552611529827118, 0.012200544588267803, 0.007760609034448862, 0.017114870250225067, 0.8780200481414795, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [9.403799049323425e-05, 7.299120738935017e-07, 1.2948523817613022e-06, 1.0040580491477158e-06, 1.5192472346825525e-05, 3.6095214454689994e-05, 3.1514438887825236e-05, 0.0010808270890265703, 4.236846143612638e-05, 0.9986969828605652, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0002984424354508519, 1.0225176993117202e-05, 2.077381759590935e-05, 1.0716731594584417e-05, 7.453841681126505e-05, 0.00016580561350565404, 0.001366860349662602, 0.0025834827683866024, 5.0240600103279576e-05, 0.32439613342285156, 0.6710227727890015, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.009233687072992325, 0.0007820427999831736, 0.000681043544318527, 0.0018207898829132318, 0.004383392166346312, 0.01750914566218853, 0.0033242509234696627, 0.01760742999613285, 0.080996572971344, 0.10943030565977097, 0.014291900210082531, 0.7399394512176514, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.00023930260795168579, 3.3237884053960443e-06, 2.2832771719549783e-05, 1.0244737495668232e-05, 1.0253623258904554e-05, 4.496401015785523e-05, 2.3576640160172246e-05, 6.216613837750629e-05, 0.0003292950859759003, 0.00874954555183649, 0.0018788703018799424, 0.0014183444436639547, 0.9872072339057922, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.012295347638428211, 0.0005298632895573974, 0.0003560507029760629, 0.0006006720359437168, 0.0005327541730366647, 0.0005349722341634333, 0.00034733544453047216, 0.0008671176037751138, 0.0343230664730072, 0.0132736312225461, 0.007531202398240566, 0.0594179667532444, 0.09377523511648178, 0.7756146788597107, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [7.85088850534521e-05, 1.1863359759445302e-05, 5.970536676613847e-06, 4.359473223303212e-06, 6.933054010005435e-06, 5.567325570154935e-05, 8.762053766986355e-06, 2.213007064710837e-05, 0.00022238318342715502, 0.00028025032952427864, 0.0008127277833409607, 0.0015677298652008176, 0.00018956429266836494, 0.00587941100820899, 0.9908537864685059, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [9.474982653046027e-05, 8.603436072007753e-06, 3.744844889297383e-06, 1.463416765545844e-07, 2.7017335924028885e-06, 1.1417326277296524e-05, 8.620806511316914e-06, 9.928076906362548e-05, 3.1195020255836425e-06, 0.00022620745585300028, 0.0005323346122168005, 2.980485987791326e-05, 0.0013675623340532184, 5.107992546982132e-05, 0.009570821188390255, 0.9879898428916931, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.003944713156670332, 0.00013029140245635062, 9.947701619239524e-05, 0.0001977638603420928, 0.00042658537859097123, 0.0014889859594404697, 0.0002834686602000147, 0.0015356721123680472, 0.004810086917132139, 0.0055677215568721294, 0.0010473602451384068, 0.040826551616191864, 0.009189032949507236, 0.0927850753068924, 0.015853004530072212, 0.027940908446907997, 0.7938733696937561, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.00021723605459555984, 2.3531203623861074e-05, 7.806276698829606e-06, 4.732958132080967e-06, 1.5671206483602873e-06, 8.667418114782777e-06, 1.264625825569965e-05, 6.129276880528778e-05, 0.0001886943937279284, 0.0005901293479837477, 0.00042056679376401007, 0.0009746518917381763, 0.00042177739669568837, 0.004066711291670799, 0.0014422433450818062, 0.011960022151470184, 0.02240196242928505, 0.9571956396102905, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0007360433810390532, 3.016823711732286e-06, 1.355472249997547e-06, 1.4014689213581732e-06, 3.387705146451481e-05, 3.752283737412654e-05, 1.0748690328910016e-05, 0.0004302702145650983, 1.6196730939554982e-05, 0.0006748255109414458, 0.0005265924264676869, 0.00012409141345415264, 4.710711073130369e-05, 0.00024138158187270164, 0.0006886575720272958, 0.0003870207292493433, 0.0021441555581986904, 0.0017651645466685295, 0.99213045835495, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [6.529821803269442e-06, 4.013174681460896e-09, 4.275890752580835e-09, 9.052354421612563e-09, 5.135186142979364e-07, 3.731759534275625e-06, 1.4824270522240113e-07, 4.655122666008538e-06, 1.663049964406582e-08, 3.029677600352443e-06, 3.1919844332151115e-05, 1.4017439298186218e-07, 4.954933956469176e-07, 2.367105054190688e-07, 7.952932605803653e-07, 0.00018607293895911425, 2.446675580358715e-06, 1.1982299838564359e-05, 0.0005082341958768666, 0.9992390871047974, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.009417897090315819, 0.0001736241829348728, 0.00019593234173953533, 0.0001054245512932539, 0.00012100120511604473, 0.00013750347716268152, 0.0002082234714180231, 0.0002620882005430758, 0.0009803815046325326, 0.00017425219994038343, 0.0013313948875293136, 0.0032773104030638933, 0.0048406231217086315, 0.012378789484500885, 0.001825899351388216, 0.024282056838274002, 0.054799702018499374, 0.07428490370512009, 0.07759280502796173, 0.10845340043306351, 0.625156819820404, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.00017886326531879604, 2.971779167637578e-06, 1.2879215773864416e-06, 1.6619707139398088e-06, 3.153397074129316e-06, 7.608537998748943e-06, 4.4805625520893955e-07, 4.628850092558423e-06, 4.462066499399953e-06, 4.64080630990793e-06, 4.678128334489884e-06, 2.225518983323127e-05, 0.0006885039038024843, 5.880844037164934e-05, 5.511296694749035e-05, 0.0013147691497579217, 0.0004210418264847249, 0.0005279287579469383, 0.0003059162409044802, 0.0011363019002601504, 0.004935138393193483, 0.9903197884559631, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0009000814170576632, 3.4609118301887065e-05, 6.00008161200094e-06, 1.331790144831757e-06, 8.38242340250872e-06, 7.950173312565312e-06, 8.352956228918629e-07, 5.017445801058784e-06, 1.1085541700595059e-05, 1.55061534314882e-05, 2.244969073217362e-05, 3.2394586014561355e-05, 7.15337591827847e-05, 0.00011106129386462271, 6.552076229127124e-06, 3.557852323865518e-05, 0.00046084230416454375, 0.0006021009758114815, 0.007735580205917358, 0.004731381312012672, 0.012920106761157513, 0.01699913665652275, 0.9552805423736572, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [8.094519580481574e-05, 5.4195235861698166e-05, 2.1191000996623188e-05, 8.66032394242211e-07, 5.241395683697192e-06, 2.8114038741478e-06, 9.647644674259936e-07, 4.259615252522053e-06, 6.819946065661497e-06, 2.6370025807409547e-06, 1.0387340807938017e-05, 3.117776213912293e-05, 2.5747020117705688e-05, 8.829349098959938e-05, 0.00010530064901104197, 0.00037444676854647696, 0.0005833787145093083, 0.0009019335266202688, 0.0004312015953473747, 0.004991123452782631, 0.003515923162922263, 0.015127466060221195, 0.018384991213679314, 0.9552486538887024, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.00019926734967157245, 5.897874757465615e-07, 4.093071765964851e-07, 5.283640120978816e-07, 4.043727415137255e-07, 6.81143433212128e-07, 2.5019562599482015e-07, 3.1909843301036744e-07, 5.017996045353357e-06, 3.0599305773648666e-06, 4.776871264766669e-06, 8.10623441793723e-06, 1.787102701200638e-05, 5.9049270930700004e-05, 7.019496933935443e-06, 5.101946953800507e-05, 0.00015419686678797007, 0.00015557752340100706, 0.0004630949115380645, 0.00018736722995527089, 0.0029354498255997896, 0.006998692639172077, 0.004895055666565895, 0.010302486829459667, 0.9735496640205383, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [4.041899592266418e-05, 7.668187862464038e-09, 6.183981327012589e-08, 4.109970319632339e-08, 3.161351571634441e-07, 1.0004068684565937e-07, 6.658517293089972e-08, 4.780272320203949e-07, 4.035153366999111e-08, 6.753236448275857e-06, 8.574299670272012e-08, 1.0514251869153668e-07, 3.270282832090743e-06, 4.896564860246144e-07, 1.245038902197848e-06, 9.069861334864981e-06, 2.0693153146567056e-06, 8.54171139508253e-06, 4.513625390245579e-05, 8.528188482159749e-05, 8.476278162561357e-05, 0.0010267929174005985, 0.0013347768690437078, 0.00029600755078718066, 0.005626787897199392, 0.9914274215698242, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.00037474490818567574, 1.2430058404788724e-07, 2.5229041966667864e-07, 3.0524404337484157e-07, 4.7725070544402115e-06, 6.075413807593577e-07, 8.501697266183328e-07, 1.1030795576516539e-05, 8.098952264390391e-08, 9.651266736909747e-07, 7.193338547040184e-07, 3.2339289646188263e-07, 1.7605361790629104e-06, 4.4468094984040363e-07, 3.298750641533843e-07, 2.4300228687934577e-05, 3.6642647955886787e-06, 3.061631332457182e-06, 2.6271180104231462e-05, 0.0003426705370657146, 2.0788082110811956e-05, 0.00013409697567112744, 0.00030472385697066784, 9.80358017841354e-05, 0.0004919670755043626, 0.004237059969455004, 0.9939160943031311, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.00023197628615889698, 7.363635177171091e-07, 1.0658037581379176e-06, 4.74980652143131e-07, 2.1877904146094806e-06, 2.0741711068694713e-06, 1.384209156185534e-07, 2.510645117581589e-06, 1.9715950827503548e-07, 6.671986739092972e-06, 8.991793265522574e-07, 6.362189992614731e-07, 6.544807433783717e-07, 7.717980565757898e-07, 4.801892714567657e-07, 1.1122294381493703e-05, 5.561924808716867e-06, 3.862757239403436e-06, 6.75371557008475e-05, 0.0002897953672800213, 4.372502735350281e-05, 0.00015209022967610508, 0.0009337729425169528, 0.00018159417959395796, 8.423309918725863e-05, 0.00993384886533022, 0.008723186329007149, 0.9793181419372559, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.00015704914403613657, 3.1231860475600115e-07, 2.931930964678031e-07, 3.1371595810014696e-07, 2.1960941012366675e-06, 1.416439886270382e-06, 1.1236411410209257e-06, 1.3091784012431162e-06, 5.921491563753989e-08, 7.978065923452959e-07, 2.803679308271967e-06, 1.976693795313622e-07, 3.994017561126384e-07, 2.304903290450966e-07, 2.2642525436822325e-06, 1.6729523849789985e-05, 1.666766252128582e-06, 2.872540562748327e-06, 2.4946917619672604e-05, 0.0002553280792199075, 4.849689958064118e-06, 4.7080560761969537e-05, 5.325669553712942e-05, 4.2403407860547304e-05, 0.00020874875190202147, 0.0020438111387193203, 0.08346733450889587, 0.0036730042193084955, 0.909987211227417, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0010030773701146245, 7.605943210364785e-06, 5.478914772538701e-06, 4.704091224994045e-06, 1.0818274631674285e-06, 7.24686287867371e-07, 6.05722732416325e-07, 1.5847876966290642e-06, 3.183097578585148e-05, 1.8444849274601438e-06, 1.3723829397349618e-06, 6.841032245574752e-06, 4.996116786060156e-06, 0.00014220463344827294, 2.102602365994244e-06, 2.04611751541961e-05, 6.875477993162349e-05, 8.339442138094455e-05, 7.060846110107377e-05, 0.0002227009244961664, 0.001345353084616363, 0.001195314689539373, 0.000763771531637758, 0.0034127712715417147, 0.009595129638910294, 0.008245964534580708, 0.004028420429676771, 0.015673480927944183, 0.037109073251485825, 0.916948676109314, 0.0, 0.0, 0.0, 0.0, 0.0], [0.00014598076813854277, 1.3442496538118576e-07, 3.0679700557811884e-07, 1.270083913595954e-07, 9.539706269379167e-08, 4.4213727790065604e-08, 7.270958946037354e-09, 2.1630664548411005e-07, 1.7861789558537566e-07, 1.1440967995213214e-07, 4.546703280539077e-08, 1.1406693545268354e-07, 1.7846295463641582e-07, 8.568713383283466e-07, 1.0497500113615388e-07, 1.2474921504690428e-06, 1.168268113360682e-06, 6.485001904366072e-07, 9.923046491167042e-06, 2.6635598260327242e-05, 2.148103158106096e-05, 7.192359043983743e-05, 0.00015873332449700683, 0.00014563243894372135, 0.002426168182864785, 0.0015144944190979004, 0.0007063182420097291, 0.002244281815364957, 0.004566822201013565, 0.009832342155277729, 0.9781236052513123, 0.0, 0.0, 0.0, 0.0], [1.069877362169791e-05, 5.128429236833654e-08, 7.382515576637161e-08, 9.797533095934341e-08, 4.944882903146208e-07, 6.269742698350456e-07, 2.6257418994646287e-07, 7.361933285210398e-07, 2.5101161327256705e-07, 2.176485395466443e-06, 5.363766604205011e-07, 8.606898518337402e-07, 2.1515788262149727e-07, 1.0358818371969392e-06, 2.489508972303156e-07, 1.3481068890541792e-05, 8.521105883119162e-06, 5.148486707184929e-06, 4.645090939447982e-06, 1.683268965280149e-05, 5.864661216037348e-05, 3.652455416158773e-05, 0.0001648575853323564, 1.1945023288717493e-05, 0.0005173988756723702, 0.0024279074277728796, 0.00034773608786053956, 0.00159640540368855, 0.00902293436229229, 0.013163739815354347, 0.06333828717470169, 0.9092466831207275, 0.0, 0.0, 0.0], [0.0012287310091778636, 8.40765733300941e-06, 8.363967936020344e-06, 8.199041985790245e-06, 3.972178092226386e-06, 3.9872684283182025e-06, 1.5942496247589588e-06, 4.764995537698269e-06, 4.461910066311248e-06, 8.69919949764153e-06, 3.4823106034309603e-06, 4.087226443516556e-06, 4.383185569167836e-06, 1.2878696907137055e-05, 2.7649991807265906e-06, 1.1550831004569773e-05, 2.8508402465377003e-05, 1.9726150640053675e-05, 8.007970609469339e-05, 0.00016381578461732715, 0.00027661045896820724, 0.00048299721674993634, 0.0005903012352064252, 0.0011750624980777502, 0.0034667092841118574, 0.005802731029689312, 0.011907560750842094, 0.016952956095337868, 0.028466491028666496, 0.061375781893730164, 0.11517591029405594, 0.2017774134874344, 0.5509369969367981, 0.0, 0.0], [3.7825186154805124e-05, 6.459626433752419e-08, 1.92735836179736e-08, 7.975797444714772e-08, 6.70031795380055e-07, 1.2464336407447263e-07, 3.072165100093116e-08, 3.3671366850285267e-07, 9.953986079835886e-08, 4.5415114868774253e-07, 3.406029591701554e-08, 1.7256120088404714e-07, 3.3590123393878457e-07, 3.2563309559918707e-07, 1.1636628549638317e-08, 3.2042125894804485e-06, 1.2533513427115395e-06, 1.7663004427959095e-06, 1.1125961464131251e-05, 2.0410294382600114e-05, 8.103430445771664e-06, 1.288108524022391e-05, 0.00014193178503774107, 1.1411094419599976e-05, 0.00010671337076928467, 0.00010951125295832753, 0.0017927116714417934, 0.0015404956648126245, 0.0012465211329981685, 0.0015975864371284842, 0.0008927045855671167, 0.00955700408667326, 0.009279929101467133, 0.9736242294311523, 0.0], [0.00035396948806010187, 2.644035248522414e-06, 1.75761067566782e-06, 1.2348079962976044e-06, 5.9496978792594746e-06, 1.3091831760903005e-06, 3.3620969475123275e-07, 2.4850504587448086e-07, 6.705440682708286e-07, 6.699505661345029e-08, 6.061929411771416e-07, 4.6190496050257934e-07, 2.3573231828777352e-06, 1.4622191883972846e-06, 2.9763643283331476e-07, 1.6005760699044913e-06, 2.460909172441461e-06, 1.9317128590046195e-06, 3.200148057658225e-05, 2.4651970306877047e-05, 3.429638309171423e-05, 0.00016550843429286033, 7.883952639531344e-05, 0.0001838715106714517, 0.0011651510139927268, 0.0001225532905664295, 0.0011725202202796936, 0.0004984779516234994, 0.0035014785826206207, 0.003400744404643774, 0.011839505285024643, 0.005179083906114101, 0.016713839024305344, 0.02912764437496662, 0.9263805150985718]], [[1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.9238101243972778, 0.07618988305330276, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.37589237093925476, 0.10050598531961441, 0.5236016511917114, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.41213729977607727, 0.08231589198112488, 0.1808476746082306, 0.3246991038322449, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.14052271842956543, 0.005812309216707945, 0.02827638015151024, 0.05733029544353485, 0.7680583596229553, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.019230686128139496, 0.0014746803790330887, 0.0016810802044346929, 0.004905841313302517, 0.025906598195433617, 0.9468011260032654, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.006040879059582949, 0.00465822359547019, 0.01286506187170744, 0.008953885175287724, 0.07021690905094147, 0.12025366723537445, 0.7770113945007324, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.01624826341867447, 0.0039293826557695866, 0.01283666305243969, 0.005099623929709196, 0.017354188486933708, 0.015599331818521023, 0.13602769374847412, 0.7929048538208008, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.13423296809196472, 0.07014789432287216, 0.03862924128770828, 0.03592687100172043, 0.057204823940992355, 0.06419355422258377, 0.03908979892730713, 0.2145734280347824, 0.34600141644477844, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0035958767402917147, 0.00038494993350468576, 0.0004921662621200085, 0.0008088782778941095, 0.0025900970213115215, 0.0024076506961137056, 0.00017131800996139646, 0.011153006926178932, 0.0021147779189050198, 0.9762812852859497, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0002283454086864367, 4.6662513341289014e-05, 8.206483471440151e-05, 7.730665674898773e-05, 0.00012034265091642737, 0.004319258499890566, 0.003844222519546747, 0.0011965008452534676, 0.0001644694566493854, 0.8530562520027161, 0.1368645280599594, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.06578907370567322, 0.019290756434202194, 0.020492704585194588, 0.021956779062747955, 0.026686683297157288, 0.08618339896202087, 0.03104090690612793, 0.1610301434993744, 0.030162539333105087, 0.2132367491722107, 0.07554484158754349, 0.24858540296554565, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.00912333931773901, 0.0033993131946772337, 0.01681826263666153, 0.004648805595934391, 0.014602267183363438, 0.05833248049020767, 0.07909005880355835, 0.01766359806060791, 0.016087261959910393, 0.15053574740886688, 0.3343389928340912, 0.05438895896077156, 0.24097083508968353, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.042534369975328445, 0.016550583764910698, 0.00781341828405857, 0.0068334611132740974, 0.009650972671806812, 0.009173312224447727, 0.0059103150852024555, 0.030093081295490265, 0.05324171483516693, 0.03575890511274338, 0.11968676000833511, 0.07133748382329941, 0.29692211747169495, 0.29449349641799927, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.01015804149210453, 0.004377418663352728, 0.0065031955018639565, 0.0035466847475618124, 0.006157051771879196, 0.005149821285158396, 0.0041582523845136166, 0.014994022436439991, 0.017724134027957916, 0.008609646931290627, 0.013557294383645058, 0.02768983505666256, 0.027776163071393967, 0.08518613874912262, 0.7644124031066895, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0005760938511230052, 3.7584886740660295e-05, 3.100580215686932e-05, 2.8090940759284422e-05, 0.00011757229367503896, 0.0010626503499224782, 0.0008489798638038337, 0.0023644298780709505, 0.0003171181015204638, 0.0005040704854764044, 0.0005438972148112953, 0.0009964646305888891, 0.004787205718457699, 0.0016582137905061245, 0.021181125193834305, 0.9649454951286316, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.05710113048553467, 0.012982725165784359, 0.012463352642953396, 0.012371198274195194, 0.013064760714769363, 0.03530249372124672, 0.013208814896643162, 0.06362835317850113, 0.010588892735540867, 0.061132002621889114, 0.027357207611203194, 0.08200793713331223, 0.03997459262609482, 0.043028347194194794, 0.06893416494131088, 0.0673389732837677, 0.37951505184173584, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.050381794571876526, 0.010253221727907658, 0.009017380885779858, 0.007853321731090546, 0.008382094092667103, 0.016030533239245415, 0.0075797527097165585, 0.016568133607506752, 0.009558065794408321, 0.020098434761166573, 0.028027668595314026, 0.03172839432954788, 0.02179395593702793, 0.03942328318953514, 0.11988714337348938, 0.04783087596297264, 0.1477368175983429, 0.40784910321235657, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.009583796374499798, 0.0005122200236655772, 0.0006106934160925448, 0.0009717882494442165, 0.0022611040621995926, 0.004041265696287155, 0.0003397607943043113, 0.002301481319591403, 0.001466980786062777, 0.004360880237072706, 0.01003958098590374, 0.003883788827806711, 0.0019065361702814698, 0.00496836518868804, 0.0035998786333948374, 0.002795657142996788, 0.015628810971975327, 0.014497026801109314, 0.916230320930481, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.03377579525113106, 0.0008472921326756477, 0.0009361570118926466, 0.0006967561785131693, 0.004600139334797859, 0.0032576005905866623, 0.0004218360991217196, 0.007556702475994825, 0.0010570597369223833, 0.0036437949165701866, 0.00017823871166910976, 0.0020456179045140743, 0.0014812112785875797, 0.002918045502156019, 0.005684820003807545, 0.06616661697626114, 0.006820716895163059, 0.0059661115519702435, 0.02230621501803398, 0.829639196395874, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.014338117092847824, 0.0045062773860991, 0.0022934828884899616, 0.004502629861235619, 0.006348524242639542, 0.006440696306526661, 0.0030759195797145367, 0.0034293767530471087, 0.008825039491057396, 0.013563888147473335, 0.008806230500340462, 0.015333057381212711, 0.010930357500910759, 0.03256243094801903, 0.041650138795375824, 0.029358427971601486, 0.06823206692934036, 0.10101238638162613, 0.12824413180351257, 0.08141805976629257, 0.41512882709503174, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.01251695305109024, 0.003707383992150426, 0.001132362405769527, 0.0020440162625163794, 0.00210379995405674, 0.0007217561360448599, 0.000358771241735667, 0.0010654572397470474, 0.0033418682869523764, 0.0035347710363566875, 0.0018008309416472912, 0.005131381563842297, 0.005956845358014107, 0.011681465432047844, 0.023353124037384987, 0.02462639845907688, 0.023082591593265533, 0.024053936824202538, 0.045054081827402115, 0.04447482153773308, 0.09848441183567047, 0.6617729663848877, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.023378068581223488, 0.00297009339556098, 0.0022721635177731514, 0.0031212512403726578, 0.010448778048157692, 0.002795708365738392, 0.0009500704472884536, 0.006640279199928045, 0.002509299898520112, 0.00621590344235301, 0.0018756598001345992, 0.004366474691778421, 0.005894926376640797, 0.007904247380793095, 0.011842511594295502, 0.01178030762821436, 0.016934018582105637, 0.015606733039021492, 0.03699197620153427, 0.05676264688372612, 0.03885182365775108, 0.30106890201568604, 0.4288182258605957, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.04225083813071251, 0.009647663682699203, 0.01306967344135046, 0.00349392369389534, 0.008718240074813366, 0.0008717633318156004, 0.002113069174811244, 0.002520152600482106, 0.0028259288519620895, 0.003056258661672473, 0.005475994199514389, 0.004467586521059275, 0.005966206081211567, 0.008145655505359173, 0.02767091989517212, 0.013382636941969395, 0.01818913221359253, 0.0327802412211895, 0.028848394751548767, 0.0599595308303833, 0.09605938196182251, 0.2093193531036377, 0.10138264298439026, 0.2997848689556122, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.025818027555942535, 0.0012453029630705714, 0.0012367736781015992, 0.0031558233313262463, 0.002605455694720149, 0.00111783342435956, 0.0002305325906490907, 0.0010614546481519938, 0.001255299779586494, 0.0014470185851678252, 0.0006083040498197079, 0.002144432393833995, 0.0017222751630470157, 0.00306368013843894, 0.0021307130809873343, 0.001739516737870872, 0.007489567156881094, 0.004201178438961506, 0.016367297619581223, 0.017187543213367462, 0.016647031530737877, 0.06787320971488953, 0.07546128332614899, 0.04369650036096573, 0.7004939913749695, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.007508317939937115, 0.00012913945829495788, 0.0002290404518134892, 0.0003099398745689541, 0.0015875763492658734, 0.00015740818344056606, 0.0003654661704786122, 0.00026319510652683675, 0.00024618778843432665, 0.004963850136846304, 0.000849328760523349, 0.00020542509446386248, 0.0035996593069285154, 0.0006757518276572227, 0.0006743194535374641, 0.0028660325333476067, 0.0007401989423669875, 0.0012410678900778294, 0.0034339427947998047, 0.0033488082699477673, 0.005896300543099642, 0.015315262600779533, 0.013584453612565994, 0.003806073684245348, 0.10587520152330399, 0.8221280574798584, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.01514680776745081, 0.0002837264910340309, 0.00043238874059170485, 0.001239211531355977, 0.0026600679848343134, 0.00034595353645272553, 0.00029542617266997695, 0.0018772019539028406, 0.00023691231035627425, 0.0001321920717600733, 0.0003354801156092435, 0.00022997852647677064, 0.0002714897273108363, 0.00042095332173630595, 0.0008192574023269117, 0.0003039459406863898, 0.0006469361833296716, 0.00038042623782530427, 0.0023925923742353916, 0.0029767933301627636, 0.0025756079703569412, 0.005941794253885746, 0.0059011997655034065, 0.005208773538470268, 0.025423666462302208, 0.023241201415657997, 0.9002800583839417, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.02088216319680214, 0.0006866250187158585, 0.0014190295478329062, 0.001624720636755228, 0.0054667238146066666, 0.001993069425225258, 0.00034445541677996516, 0.0014866288984194398, 0.0003860277065541595, 0.0002986796898767352, 0.0003715484344866127, 0.0005332491709850729, 0.00025670789182186127, 0.0006083346088416874, 0.00037862517638131976, 0.0013748217606917024, 0.0013781337765976787, 0.0009236650075763464, 0.023375000804662704, 0.0377323180437088, 0.004063982982188463, 0.021846624091267586, 0.018512366339564323, 0.00504719465970993, 0.018440257757902145, 0.06399477273225784, 0.3300282955169678, 0.4365459680557251, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.002729784930124879, 0.00011315700976410881, 0.00025743208243511617, 0.00030652908026240766, 0.0008012376492843032, 0.0008870926103554666, 0.00010111552546732128, 0.002048564376309514, 0.00014291201659943908, 0.000198980254936032, 0.00043098151218146086, 7.435739098582417e-05, 0.00011077813542215154, 0.00022297550458461046, 0.00010400103928986937, 0.0007566175772808492, 0.00018428797193337232, 0.0003102096379734576, 0.0010590285528451204, 0.0053943791426718235, 0.0013711372157558799, 0.0017238208092749119, 0.0034383409656584263, 0.0007791521493345499, 0.0027107910718768835, 0.011246584355831146, 0.26186177134513855, 0.04286457970738411, 0.6577693819999695, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0071692257188260555, 0.002651229267939925, 0.002445027930662036, 0.003023250261321664, 0.002686190651729703, 0.0007135469932109118, 0.0005127687472850084, 0.0003864864120259881, 0.0009532018448226154, 0.00044078464270569384, 0.0008591761579737067, 0.0012491066008806229, 0.0009424785966984928, 0.0016617280198261142, 0.0004435318405739963, 0.0014648421201854944, 0.003577465657144785, 0.003767341375350952, 0.012122934684157372, 0.006853919476270676, 0.005249565001577139, 0.019704243168234825, 0.02666923590004444, 0.04096018522977829, 0.2385396808385849, 0.05725707858800888, 0.040405381470918655, 0.10933997482061386, 0.07736305147409439, 0.3305874168872833, 0.0, 0.0, 0.0, 0.0, 0.0], [0.01162666268646717, 0.001965815667062998, 0.0009179532062262297, 0.0018197413301095366, 0.0020789778791368008, 0.00029357121093198657, 0.00010028555698227137, 0.000579997431486845, 0.00048541222349740565, 9.764838614501059e-05, 0.00015712776803411543, 0.0004106425039935857, 0.0004877779574599117, 0.0007517866906709969, 0.0007532845484092832, 0.0012242129305377603, 0.0010826843790709972, 0.0014821895165368915, 0.002774251624941826, 0.0036103385500609875, 0.002137240022420883, 0.009361452423036098, 0.01427766215056181, 0.01184671651571989, 0.04729337990283966, 0.0450628325343132, 0.041560690850019455, 0.0428079329431057, 0.07079781591892242, 0.10084821283817291, 0.581305742263794, 0.0, 0.0, 0.0, 0.0], [0.0038734348490834236, 0.00025570974685251713, 0.0006259257788769901, 0.0005843959515914321, 0.0012326485011726618, 0.00039455998921766877, 0.00033266766695305705, 0.00015581920160911977, 0.00016472737479489297, 0.000772629864513874, 0.0003866984916385263, 0.0002782446681521833, 0.00015239730419125408, 0.00025198585353791714, 6.452933303080499e-05, 0.0001726411865092814, 0.0007133586914278567, 0.0005384800606407225, 0.007539568468928337, 0.006723248399794102, 0.0013963908422738314, 0.003110358491539955, 0.005501287989318371, 0.0024135892745107412, 0.03896057978272438, 0.023422319442033768, 0.009641147218644619, 0.025750910863280296, 0.12702499330043793, 0.04005929455161095, 0.5142088532447815, 0.18329668045043945, 0.0, 0.0, 0.0], [0.015151116997003555, 0.0013760678702965379, 0.001556050730869174, 0.002171708270907402, 0.002511007012799382, 0.0017854932229965925, 0.0004700865247286856, 0.0006788119790144265, 0.00038990683970041573, 0.000990036758594215, 0.00038026587571948767, 0.0004182497796136886, 0.0004435285518411547, 0.0005140136345289648, 0.00016592231986578554, 0.0006932553369551897, 0.0009629505802877247, 0.0007656947709619999, 0.004365228116512299, 0.00643626693636179, 0.0018688250565901399, 0.008356153033673763, 0.006384752690792084, 0.005890705157071352, 0.031187018379569054, 0.04100944101810455, 0.02856607362627983, 0.041577283293008804, 0.04076460376381874, 0.067128486931324, 0.30426710844039917, 0.2161882072687149, 0.16458559036254883, 0.0, 0.0], [0.01735411025583744, 0.0005264714127406478, 0.0006208939594216645, 0.0016926302341744304, 0.006297253537923098, 0.00102021056227386, 7.362550968537107e-05, 0.0006762267439626157, 0.00044483091915026307, 7.778873259667307e-05, 4.9738089728634804e-05, 0.0003035972185898572, 0.00017341824423056096, 0.0005162435118108988, 0.00027352781035006046, 0.0007443997892551124, 0.0006022228044457734, 0.00045445002615451813, 0.002665339270606637, 0.0026638731360435486, 0.00255159311927855, 0.007818798534572124, 0.009490970522165298, 0.000983040314167738, 0.007624225690960884, 0.011552350595593452, 0.09579706937074661, 0.08069909363985062, 0.09889820963144302, 0.03799610957503319, 0.058356162160634995, 0.11718906462192535, 0.11656812578439713, 0.317244291305542, 0.0], [0.010796324349939823, 0.00301845814101398, 0.0017010795418173075, 0.002761394716799259, 0.00554644363000989, 0.0005509481998160481, 0.00022208339942153543, 0.0002330585557501763, 0.0008555070380680263, 8.429395529674366e-05, 0.00013491026766132563, 0.00040303391870111227, 0.0002156854170607403, 0.0009122328483499587, 0.0007261996506713331, 0.0003818846307694912, 0.0007406793884001672, 0.0007301224395632744, 0.0007852284470573068, 0.0013473366852849722, 0.004313784651458263, 0.0077291796915233135, 0.005422806367278099, 0.00674707256257534, 0.013981982134282589, 0.004630991257727146, 0.019391806796193123, 0.028133727610111237, 0.016861919313669205, 0.07724323868751526, 0.054351553320884705, 0.13139015436172485, 0.15014055371284485, 0.08434385806322098, 0.3631705045700073]], [[1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.19326065480709076, 0.806739330291748, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.09132368117570877, 0.002221884438768029, 0.9064545035362244, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0712752714753151, 0.010394570417702198, 0.014608724974095821, 0.9037215113639832, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.007525191176682711, 8.942779095377773e-05, 7.544153777416795e-05, 3.4135869100282434e-06, 0.9923065304756165, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.004521039314568043, 4.280492703401251e-06, 9.230399155057967e-05, 2.363623025303241e-05, 1.1194380022061523e-05, 0.9953475594520569, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.007938127964735031, 0.00010465236118761823, 4.188244929537177e-05, 8.376280675292946e-06, 0.0002111138601321727, 3.129784545308212e-06, 0.9916927814483643, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0024528461508452892, 1.0800496056617703e-05, 1.5410727428388782e-05, 2.252152597748136e-07, 3.925116834579967e-05, 5.133600552653661e-06, 2.3800452254363336e-05, 0.9974525570869446, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.19944852590560913, 0.17831814289093018, 0.06548021733760834, 0.1327645629644394, 0.038584478199481964, 0.010353402234613895, 0.021433580666780472, 0.0067232828587293625, 0.3468937277793884, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0014458474470302463, 8.054711315708118e-07, 3.064591874135658e-05, 7.412935474349069e-07, 2.02740625354636e-06, 1.1259067832725123e-05, 1.822341573642916e-06, 8.012933540157974e-05, 1.0569654307346354e-07, 0.9984266757965088, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.002338030841201544, 1.2498737305577379e-05, 2.2450209144153632e-05, 4.236520453559933e-06, 2.952040460968419e-07, 1.8947142734759836e-06, 7.76990273152478e-05, 8.969098416855559e-06, 4.6952173704539746e-08, 1.134771082433872e-06, 0.9975327253341675, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.04751456156373024, 0.05268263816833496, 0.02040812186896801, 0.0585300475358963, 0.007166560739278793, 0.008201424963772297, 0.0118463896214962, 0.0017474384512752295, 0.02496332861483097, 0.002697254531085491, 0.0014261904871091247, 0.7628160715103149, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.006070801988244057, 0.00012957923172507435, 5.943346332060173e-05, 5.659346243191976e-06, 0.00026153220096603036, 0.0001224183215526864, 1.0399337043054402e-05, 1.1076372175011784e-05, 4.060476612721686e-07, 0.00036004636785946786, 2.0115392544539645e-05, 1.3643895613313362e-07, 0.9929484128952026, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.09530571103096008, 0.14180323481559753, 0.05038182809948921, 0.10299671441316605, 0.029812078922986984, 0.006589412689208984, 0.01682438515126705, 0.004481854848563671, 0.2310645431280136, 0.019435422495007515, 0.010874398052692413, 0.05543200299143791, 0.03543870151042938, 0.19955967366695404, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.027447840198874474, 0.0001406537339789793, 0.00019846689247060567, 3.4967764804605395e-05, 5.282158781483304e-06, 0.00015024232561700046, 8.128365880111232e-05, 1.2125195098633412e-05, 6.303905684035271e-05, 6.543220661114901e-05, 2.8324959657766158e-06, 3.8907339330762625e-05, 5.673642249348632e-07, 4.210490806144662e-05, 0.9717161655426025, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.004212100990116596, 3.0696712201461196e-05, 6.369411858031526e-05, 8.254517069872236e-07, 8.262093729172193e-07, 3.868323619826697e-05, 4.268790689820889e-06, 3.819391622528201e-06, 1.2627964451894513e-07, 0.00013782696623820812, 1.7806029063649476e-05, 1.71001346416233e-08, 5.979860816296423e-06, 6.190285972706988e-08, 2.2941485440242104e-06, 0.9954808950424194, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.019846783950924873, 0.03168342635035515, 0.012553276494145393, 0.035718828439712524, 0.004399011842906475, 0.004350573290139437, 0.007427365519106388, 0.0009372885106131434, 0.012662895955145359, 0.001605054596439004, 0.0007319962605834007, 0.4563407897949219, 0.002010870724916458, 0.009759279899299145, 0.008237499743700027, 0.0002594127436168492, 0.3914756774902344, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.027297092601656914, 0.002435697941109538, 0.00027068096096627414, 7.525548426201567e-05, 1.3686794773093425e-05, 5.773205430159578e-06, 0.00043018197175115347, 1.4438101061386988e-05, 4.835123036173172e-05, 5.326086829882115e-05, 9.613273505237885e-06, 6.846668111393228e-05, 0.0003037670685444027, 2.8783806556020863e-05, 3.866383121930994e-05, 2.5926743546733633e-05, 4.6800287236692384e-05, 0.9688335061073303, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.006310698576271534, 3.294691487099044e-05, 9.895694347505923e-06, 7.334598421948613e-07, 9.119858441408724e-05, 4.094936593901366e-05, 8.511718988302164e-06, 9.950739331543446e-05, 1.9264851403022476e-07, 8.829235298435378e-07, 1.8906122249973123e-06, 1.180872800432553e-06, 3.1165250220510643e-06, 9.474784690155502e-08, 3.301760898466455e-06, 2.5855299099930562e-05, 7.711321359238354e-07, 2.670870799192926e-06, 0.993365466594696, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0011340525234118104, 6.961915460124146e-06, 1.2364374924800359e-05, 4.0065501138997206e-07, 5.351814252207987e-05, 5.240101017989218e-06, 4.459504998521879e-05, 0.00016450764087494463, 1.4344476717553789e-08, 1.0388008604422794e-06, 4.7788784286240116e-05, 5.058673835378613e-08, 9.46820875924459e-07, 7.475676078172455e-09, 6.6039274315699e-07, 3.843118065560702e-06, 3.346431398654204e-08, 2.411975685845391e-07, 1.664775118115358e-05, 0.9985070824623108, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.08800853043794632, 0.18921299278736115, 0.04556255042552948, 0.08163290470838547, 0.017358072102069855, 0.004830400459468365, 0.01895877905189991, 0.004620134364813566, 0.06910263746976852, 0.012970475479960442, 0.01034302543848753, 0.05366412550210953, 0.009830269031226635, 0.05797470733523369, 0.011936401017010212, 0.009446632117033005, 0.04662337154150009, 0.07882606983184814, 0.010427410714328289, 0.0020946746226400137, 0.17657588422298431, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.01597118005156517, 0.003062015864998102, 0.00014189504145178944, 0.0007885738741606474, 4.299367719795555e-05, 4.1311970562674105e-05, 6.306855357252061e-05, 3.546511607055436e-06, 2.173127722926438e-05, 1.7860953448689543e-05, 1.0213978384854272e-05, 2.659635356394574e-05, 3.7855479604331776e-05, 1.3186846445023548e-05, 0.0001881764328572899, 4.963887022313429e-06, 1.8085080228047445e-05, 4.864022776018828e-05, 4.463586265046615e-06, 2.4802482130326098e-06, 1.2178892575320788e-05, 0.9794788956642151, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.007472214289009571, 0.00037190664443187416, 7.639063551323488e-05, 1.1135902241221629e-05, 3.0148652513162233e-05, 1.1014418305421714e-05, 5.4528678447240964e-05, 2.2444353817263618e-05, 1.1283835192443803e-05, 2.8587841370608658e-05, 5.627734026347753e-06, 7.513598393416032e-05, 7.734286919003353e-05, 6.529704933200264e-06, 8.796907422947697e-06, 8.651493772049434e-06, 5.072782005299814e-05, 7.040848686301615e-06, 3.187632319168188e-05, 1.4034867490408942e-06, 7.003580776654417e-06, 7.040066702757031e-05, 0.9915596842765808, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.02204541489481926, 0.2804947793483734, 0.01667478121817112, 0.00015087139036040753, 0.0006804474396631122, 2.739414048846811e-05, 0.0008938335813581944, 1.2499216609285213e-05, 0.00026655319379642606, 5.584166501648724e-06, 5.109106859890744e-05, 9.500432497588918e-05, 1.8194788935943507e-05, 0.0001828280946938321, 0.0016493602888658643, 2.7935215257457457e-05, 6.991910777287558e-05, 7.677567191421986e-05, 2.088614746753592e-05, 4.512567102210596e-06, 0.00014541394193656743, 0.000447049766080454, 5.514786971616559e-05, 0.6759037375450134, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.01879945769906044, 0.0019184609409421682, 0.0010523863602429628, 7.228145841509104e-05, 0.00013796599523629993, 5.2088158554397523e-05, 6.334375211736187e-05, 9.731957106851041e-06, 8.120811253320426e-05, 5.9983816754538566e-05, 9.635974492994137e-06, 4.818717934540473e-05, 4.688356784754433e-05, 5.250575486570597e-05, 0.00044942021486349404, 0.00028932970599271357, 3.5371991543797776e-05, 0.00022377951245289296, 0.0002020896936301142, 3.112003923888551e-06, 3.522854967741296e-05, 9.854993550106883e-05, 0.0001810147223295644, 0.00012300792150199413, 0.9759548306465149, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0016589798033237457, 6.632470558542991e-06, 3.9323436794802547e-05, 5.512135885510361e-06, 7.328272477025166e-05, 5.176194690648117e-07, 8.362664084415883e-05, 1.6115391190396622e-05, 1.840794965346504e-07, 0.00020032576867379248, 1.849748059612466e-06, 7.683036074013216e-08, 1.805101419449784e-05, 9.2597367995495e-08, 6.612178822251735e-06, 1.9627825622592354e-06, 5.0048470257024746e-08, 6.371125493842555e-08, 4.881635504716542e-07, 3.8391394809877966e-06, 1.8601461704292888e-08, 1.6708657994968235e-06, 8.816599006422621e-07, 6.673329977502362e-08, 1.878514410691423e-07, 0.9978796243667603, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.005040663294494152, 0.00039155513513833284, 0.0005618699360638857, 0.00014523036952596158, 3.9371887396555394e-05, 5.409959703683853e-05, 0.00028479532920755446, 4.828744931728579e-05, 3.104316647295491e-06, 6.896261766087264e-05, 0.0002681414771359414, 4.210924998915289e-06, 1.2119872735638637e-05, 2.02539536076074e-06, 3.188776099705137e-05, 1.1356526556483004e-05, 3.0187973152351333e-06, 1.2622894246305805e-05, 7.465576800314011e-06, 3.027804723387817e-06, 4.951110668116598e-07, 3.015455513377674e-05, 1.8863762306864373e-05, 4.151246685069054e-06, 5.031317414250225e-06, 6.069071332603926e-06, 0.9929414987564087, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0012448636116459966, 0.00017617006960790604, 2.4151706384145655e-05, 1.2456669537641574e-05, 5.2112987759755924e-05, 2.3546019292552955e-05, 5.6734352256171405e-05, 0.000458276248537004, 2.498383537385962e-06, 0.0001134940903284587, 4.92024228151422e-05, 1.3444526302919257e-05, 7.73592691984959e-05, 1.343705889667035e-06, 6.0608936109929346e-06, 0.00010331773228244856, 8.745474588067736e-06, 1.0811555512191262e-05, 0.00015856359095778316, 1.9056030851061223e-06, 5.418779096544313e-07, 1.23213567349012e-06, 6.472226232290268e-05, 3.5890500384994084e-06, 8.650893505546264e-06, 2.962798134831246e-06, 1.3278935512062162e-05, 0.9973099231719971, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0019733370281755924, 0.000975057715550065, 7.218527753138915e-05, 9.848424087977037e-05, 4.960871592629701e-05, 5.4340667702490464e-05, 0.0003797213430516422, 6.184644007589668e-06, 1.0831167855940294e-06, 4.49121853307588e-06, 0.00047983607510104775, 9.928667168424e-07, 1.3564570508606266e-05, 6.477196734522295e-07, 1.0186121471633669e-05, 5.581342520599719e-06, 7.113497417776671e-07, 5.998076062496693e-07, 3.201096433258499e-06, 2.288287987539661e-06, 1.513085550186588e-07, 7.1307663347397465e-06, 1.793800606719742e-06, 2.402577047178056e-05, 2.833120333889383e-07, 1.226913900609361e-05, 0.0002332546137040481, 0.0003131969424430281, 0.9952757358551025, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.030099308118224144, 0.09027384221553802, 0.02814972586929798, 0.15763869881629944, 0.026473553851246834, 0.0030060093849897385, 0.017219528555870056, 0.002244670642539859, 0.031200477853417397, 0.011351736262440681, 0.0013561330270022154, 0.02911919355392456, 0.016773531213402748, 0.023998120799660683, 0.009450535289943218, 0.006288208533078432, 0.02392304502427578, 0.05829623341560364, 0.009904232807457447, 0.00037797336699441075, 0.024595150724053383, 0.0404200553894043, 0.027886759489774704, 0.009823574684560299, 0.04994713142514229, 0.0010651152115315199, 0.014632534235715866, 0.011246389709413052, 0.0020601556170731783, 0.24117839336395264, 0.0, 0.0, 0.0, 0.0, 0.0], [0.004169411025941372, 0.0006143272621557117, 0.0005938453250564635, 0.000110362030682154, 9.005493484437466e-05, 1.3705146557185799e-05, 0.0005903472192585468, 2.689225539143081e-06, 8.671486284583807e-06, 4.6116256271488965e-05, 1.0056769497168716e-05, 5.3628987188858446e-06, 5.199840961722657e-05, 4.802404873771593e-06, 2.119520831911359e-05, 2.4785002096905373e-05, 3.440031377976993e-06, 1.8410817574476823e-05, 7.055131572997198e-05, 5.60396472337743e-07, 5.06017067891662e-06, 2.2592088498640805e-05, 6.252533057704568e-05, 1.8619313777890056e-05, 0.0001411851990269497, 2.6331037588533945e-05, 1.2610411204150296e-06, 1.075982140719134e-06, 3.0931920491639175e-07, 5.793043783342e-06, 0.9932645559310913, 0.0, 0.0, 0.0, 0.0], [0.010566665790975094, 0.0013905888190492988, 0.00043136317981407046, 0.00102959293872118, 0.003916396759450436, 0.00015029506175778806, 0.0019935970194637775, 0.0003044370678253472, 2.984509410453029e-05, 0.0006592917488887906, 1.265983701159712e-05, 7.527502748416737e-05, 0.0003976769803557545, 1.7276539438171312e-05, 0.00038583436980843544, 2.1061901861685328e-05, 5.340308780432679e-05, 6.33502786513418e-05, 0.000126884420751594, 6.783195203752257e-06, 2.7170561224920675e-05, 0.0001300896255997941, 3.859859498334117e-05, 4.366646317066625e-05, 8.064090252446476e-06, 0.0007468666881322861, 8.195245754905045e-06, 9.368611063109711e-05, 6.252453658817103e-06, 4.44724500994198e-05, 0.00020698650041595101, 0.9770136475563049, 0.0, 0.0, 0.0], [0.01713147573173046, 0.10981205850839615, 0.04528496041893959, 0.16663990914821625, 0.046834927052259445, 0.012714341282844543, 0.024941811338067055, 0.0024605044163763523, 0.01651417836546898, 0.01401818823069334, 0.0028046509250998497, 0.017261290922760963, 0.056191328912973404, 0.011843177489936352, 0.024518396705389023, 0.00800488144159317, 0.013582981191575527, 0.027696525678038597, 0.007881488651037216, 0.0013759636785835028, 0.009033638052642345, 0.03748000040650368, 0.01436882745474577, 0.016579674556851387, 0.03148626536130905, 0.0022351099178195, 0.007032562047243118, 0.020338809117674828, 0.002157367067411542, 0.01573573611676693, 0.03529255837202072, 0.017795000225305557, 0.16295133531093597, 0.0, 0.0], [0.0027362462133169174, 0.0032704512123018503, 0.00024938586284406483, 2.9521219403250143e-05, 0.0027190842665731907, 1.5680874639656395e-05, 0.0011440095258876681, 3.5285669582663104e-05, 1.9352144590811804e-05, 3.620974166551605e-05, 1.220226567966165e-05, 0.00012145684740971774, 4.9556183512322605e-05, 1.1851533599838149e-05, 8.826648263493553e-05, 3.184531306033023e-05, 8.618931315140799e-05, 0.0001305783516727388, 9.26757202250883e-05, 1.8444543457007967e-05, 4.454900135897333e-06, 1.4211076631909236e-05, 0.00020819867495447397, 6.22495062998496e-05, 2.729003972490318e-05, 8.527104910172056e-06, 0.0010931742144748569, 0.0010259025730192661, 1.8266464394400828e-05, 9.816350939217955e-06, 7.599987839057576e-07, 4.151821940467926e-06, 8.365916983166244e-06, 0.9866164922714233, 0.0], [0.013301734812557697, 0.0020241746678948402, 0.0009810627670958638, 0.0002710390544962138, 0.02624250203371048, 6.968397065065801e-05, 0.0001882457872852683, 0.0003323268028907478, 0.00020717468578368425, 0.000355835392838344, 1.8688766431296244e-05, 0.00031016304274089634, 0.00028844171902164817, 0.0001415754231857136, 0.00021018910047132522, 0.00013730101636610925, 0.0002454249479342252, 0.000585862435400486, 0.000840906344819814, 0.000378983560949564, 0.00011255494609940797, 0.0009485265472903848, 0.00025817163987085223, 0.0005371624720282853, 0.00023706370848231018, 9.896130359265953e-05, 1.4541798918799032e-05, 7.514355093007907e-05, 1.0142424798686989e-05, 7.690754864597693e-05, 6.643269443884492e-05, 5.445994975161739e-05, 0.0001078199056792073, 0.0021891193464398384, 0.9480817317962646]], [[1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.9360752105712891, 0.06392484903335571, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.8446658253669739, 0.06946857273578644, 0.0858655720949173, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.6426600217819214, 0.1298869252204895, 0.17117170989513397, 0.05628129467368126, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.5074959397315979, 0.07786273211240768, 0.0726853460073471, 0.10711847245693207, 0.23483748733997345, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.4970755875110626, 0.06315260380506516, 0.1155911386013031, 0.08851395547389984, 0.1019635945558548, 0.13370312750339508, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.3081282079219818, 0.06775976717472076, 0.13057540357112885, 0.05440934747457504, 0.040503207594156265, 0.3605913519859314, 0.03803274407982826, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.32434985041618347, 0.06349692493677139, 0.07783648371696472, 0.07798172533512115, 0.07410472631454468, 0.13590548932552338, 0.070388562977314, 0.1759362667798996, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.16412916779518127, 0.034692876040935516, 0.051312558352947235, 0.013316995464265347, 0.17712704837322235, 0.15951892733573914, 0.09812311083078384, 0.2969048321247101, 0.004874553997069597, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.18775466084480286, 0.05488046258687973, 0.10758275538682938, 0.043910037726163864, 0.1406543254852295, 0.11696729809045792, 0.05246002599596977, 0.1536639779806137, 0.04935271292924881, 0.09277381747961044, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.20517116785049438, 0.04006878659129143, 0.0833921954035759, 0.04565277323126793, 0.05697014555335045, 0.13319887220859528, 0.02641640417277813, 0.09950855374336243, 0.05013459920883179, 0.21417361497879028, 0.04531295225024223, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.10754454135894775, 0.020955445244908333, 0.0524684377014637, 0.008330847136676311, 0.17997515201568604, 0.10652028769254684, 0.07210806012153625, 0.20191138982772827, 0.0033460413105785847, 0.14021331071853638, 0.10356749594211578, 0.0030589611269533634, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.09423348307609558, 0.02283540554344654, 0.05558855086565018, 0.026967830955982208, 0.12799425423145294, 0.07631105184555054, 0.07361149042844772, 0.16634689271450043, 0.030196351930499077, 0.11622441560029984, 0.13079151511192322, 0.022433558478951454, 0.05646514520049095, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.11027703434228897, 0.019917326048016548, 0.03387212008237839, 0.008012223057448864, 0.1268695592880249, 0.12052713334560394, 0.06604593992233276, 0.21159584820270538, 0.0029103795532137156, 0.13361385464668274, 0.0783042386174202, 0.005965671502053738, 0.07863973826169968, 0.003449017880484462, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.1600770503282547, 0.05452115833759308, 0.059736255556344986, 0.020207516849040985, 0.052660420536994934, 0.10011062026023865, 0.11729264259338379, 0.14998747408390045, 0.017248356714844704, 0.05625280365347862, 0.0944204106926918, 0.012046907097101212, 0.020297691226005554, 0.018244566395878792, 0.06689614802598953, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.17850063741207123, 0.042711369693279266, 0.058762893080711365, 0.025501525029540062, 0.02821972966194153, 0.09395948797464371, 0.035697706043720245, 0.038533009588718414, 0.027574095875024796, 0.12417211383581161, 0.09900189191102982, 0.029919540509581566, 0.04168912395834923, 0.030038556084036827, 0.08029218018054962, 0.06542609632015228, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.082626111805439, 0.014221650548279285, 0.03910331055521965, 0.005809779744595289, 0.14161059260368347, 0.08671098947525024, 0.05503496900200844, 0.15626993775367737, 0.002328180940821767, 0.11679325252771378, 0.08067948371171951, 0.002191797597333789, 0.09792855381965637, 0.0027368664741516113, 0.03297450393438339, 0.0803784653544426, 0.0026015001349151134, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.09858329594135284, 0.02832067385315895, 0.05263176187872887, 0.01785864308476448, 0.061636604368686676, 0.08070768415927887, 0.07397956401109695, 0.09560471773147583, 0.009335814043879509, 0.08467236161231995, 0.11269160360097885, 0.018467873334884644, 0.038562607020139694, 0.01105150580406189, 0.06964194029569626, 0.09237801283597946, 0.02256043441593647, 0.03131495416164398, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.1495792120695114, 0.013048066757619381, 0.03394852206110954, 0.015061764977872372, 0.045098837465047836, 0.08479965478181839, 0.027256984263658524, 0.17206288874149323, 0.016850341111421585, 0.10883238911628723, 0.11793841421604156, 0.008323256857693195, 0.016134323552250862, 0.017726458609104156, 0.026173429563641548, 0.05189085006713867, 0.00897607859224081, 0.010605717077851295, 0.07569281756877899, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.2303973287343979, 0.024296822026371956, 0.026559969410300255, 0.03924616798758507, 0.04230227321386337, 0.04924340918660164, 0.0079794405028224, 0.07322089374065399, 0.03455185890197754, 0.06601927429437637, 0.03830168768763542, 0.036723218858242035, 0.0241401307284832, 0.037105269730091095, 0.028557149693369865, 0.05446392297744751, 0.03901509940624237, 0.01594439148902893, 0.046100109815597534, 0.08583168685436249, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.05691662058234215, 0.006225030403584242, 0.011754435487091541, 0.0035496768541634083, 0.07110721617937088, 0.08006571233272552, 0.022162357345223427, 0.11016802489757538, 0.0010484936647117138, 0.08122199773788452, 0.026672087609767914, 0.0021960819140076637, 0.030598528683185577, 0.0012014060048386455, 0.014937772415578365, 0.14736886322498322, 0.0026267559733241796, 0.03338916599750519, 0.11049021035432816, 0.18504437804222107, 0.0012552266707643867, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.09241703152656555, 0.026059621945023537, 0.02483060397207737, 0.014753975905478, 0.07309960573911667, 0.04160080850124359, 0.028182951733469963, 0.10522370785474777, 0.010200563818216324, 0.026716861873865128, 0.03863626718521118, 0.008342118002474308, 0.034165360033512115, 0.011011882685124874, 0.02702195569872856, 0.10600949078798294, 0.009498468600213528, 0.01612309366464615, 0.06257017701864243, 0.19003157317638397, 0.010448911227285862, 0.043055009096860886, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.061736516654491425, 0.007029624190181494, 0.01948658935725689, 0.010366822592914104, 0.04601288586854935, 0.07236859202384949, 0.02275237813591957, 0.08978492766618729, 0.005041897296905518, 0.0701901912689209, 0.07947877049446106, 0.006475384812802076, 0.03835068270564079, 0.005680918227881193, 0.015840981155633926, 0.053441066294908524, 0.007524985354393721, 0.018862150609493256, 0.09784606844186783, 0.19087882339954376, 0.0069852969609200954, 0.027727801352739334, 0.046136610209941864, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.043498445302248, 0.006029136944562197, 0.016271812841296196, 0.004962891805917025, 0.09232775866985321, 0.05643608793616295, 0.024273201823234558, 0.10004425793886185, 0.0013475839514285326, 0.04695265740156174, 0.04337001219391823, 0.0026120906695723534, 0.025980019941926003, 0.001536776078864932, 0.013811727054417133, 0.12162518501281738, 0.0030840591061860323, 0.014781156554818153, 0.08252864331007004, 0.20843425393104553, 0.002105809049680829, 0.03356638178229332, 0.04992130398750305, 0.004498816095292568, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.05493931472301483, 0.005164690315723419, 0.013282540254294872, 0.006816802080720663, 0.08581959456205368, 0.0425836555659771, 0.010111803188920021, 0.10901321470737457, 0.003067037323489785, 0.041951462626457214, 0.03668753430247307, 0.003877601819112897, 0.036396291106939316, 0.0036428277380764484, 0.015713080763816833, 0.04514279216527939, 0.004770634695887566, 0.010733411647379398, 0.12949086725711823, 0.18343371152877808, 0.005763908848166466, 0.06008810177445412, 0.0642036646604538, 0.008479910902678967, 0.018825512379407883, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.11619282513856888, 0.024008197709918022, 0.017500849440693855, 0.0227952990680933, 0.050276536494493484, 0.032025471329689026, 0.020647739991545677, 0.03974743187427521, 0.023735985159873962, 0.026677904650568962, 0.01885642483830452, 0.028171356767416, 0.034608807414770126, 0.02664370648562908, 0.022768085822463036, 0.023577360436320305, 0.03119918704032898, 0.0226204264909029, 0.09706806391477585, 0.06436187028884888, 0.03251432999968529, 0.04557201266288757, 0.04268259555101395, 0.0282528605312109, 0.025809550657868385, 0.08168511837720871, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.11220824718475342, 0.005275931674987078, 0.013701369054615498, 0.006475640460848808, 0.058257490396499634, 0.055918335914611816, 0.013027261942625046, 0.10451261699199677, 0.004013676196336746, 0.0401136614382267, 0.037872299551963806, 0.0031745312735438347, 0.012129997834563255, 0.004096260294318199, 0.007334529887884855, 0.01773281767964363, 0.0034752371720969677, 0.0030595550779253244, 0.09483392536640167, 0.20002643764019012, 0.005673971958458424, 0.016374558210372925, 0.017230231314897537, 0.004368434194475412, 0.00930806528776884, 0.09713457524776459, 0.05267038941383362, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.06115269288420677, 0.004939326085150242, 0.00976282637566328, 0.005227160174399614, 0.03646275773644447, 0.03137052804231644, 0.008218890056014061, 0.11535761505365372, 0.003272865666076541, 0.0382491834461689, 0.033371780067682266, 0.002179377479478717, 0.007692957296967506, 0.003454803256317973, 0.009085437282919884, 0.027900513261556625, 0.0023665931075811386, 0.003948240075260401, 0.0407714918255806, 0.2757468819618225, 0.00354666355997324, 0.012455102987587452, 0.032613158226013184, 0.003209818387404084, 0.003603094257414341, 0.100096195936203, 0.08892863243818283, 0.03501543402671814, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.11957041174173355, 0.00805051438510418, 0.01337728463113308, 0.0070892684161663055, 0.025275399908423424, 0.05149080976843834, 0.0065854438580572605, 0.06165984645485878, 0.005620481446385384, 0.055614881217479706, 0.027291802689433098, 0.004286488983780146, 0.007698433473706245, 0.005996248684823513, 0.00794201623648405, 0.028790509328246117, 0.00487096793949604, 0.004313758108764887, 0.02622808702290058, 0.15385550260543823, 0.010709959082305431, 0.009564651176333427, 0.029178407043218613, 0.005329274572432041, 0.008051516488194466, 0.07812706381082535, 0.10984007269144058, 0.029521016404032707, 0.09406991302967072, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.025880729779601097, 0.003286876482889056, 0.009190608747303486, 0.002271316945552826, 0.04206256940960884, 0.04035560041666031, 0.01510708499699831, 0.0480133593082428, 0.0008096112287603319, 0.03906421363353729, 0.026566829532384872, 0.0015204478986561298, 0.021220151335000992, 0.0009539547027088702, 0.007810372393578291, 0.03313516080379486, 0.0018792509799823165, 0.010119677521288395, 0.06751004606485367, 0.12262905389070511, 0.0015832216013222933, 0.022168032824993134, 0.03078521229326725, 0.004318809602409601, 0.018729012459516525, 0.1447751224040985, 0.06957433372735977, 0.11388754844665527, 0.07182328402996063, 0.002968474291265011, 0.0, 0.0, 0.0, 0.0, 0.0], [0.04612509533762932, 0.0049293735064566135, 0.011905021034181118, 0.005939905997365713, 0.05086230859160423, 0.039155829697847366, 0.011702251620590687, 0.07777238637208939, 0.0027224922087043524, 0.029330721125006676, 0.03543661907315254, 0.002508657518774271, 0.01188324298709631, 0.002963855629786849, 0.008878657594323158, 0.01893666945397854, 0.0028045878279954195, 0.004066929221153259, 0.06637170910835266, 0.16173602640628815, 0.00407473836094141, 0.017417889088392258, 0.04845814406871796, 0.005489847622811794, 0.014467372559010983, 0.08884267508983612, 0.057441405951976776, 0.04815756157040596, 0.09182101488113403, 0.0047510163858532906, 0.02304600179195404, 0.0, 0.0, 0.0, 0.0], [0.028616521507501602, 0.0035397165920585394, 0.012825757265090942, 0.003731104079633951, 0.02213282324373722, 0.026339076459407806, 0.015483636409044266, 0.04692447930574417, 0.0016770190559327602, 0.05343090742826462, 0.03818729519844055, 0.0025294895749539137, 0.027682233601808548, 0.001944716670550406, 0.01042796391993761, 0.020306404680013657, 0.0030831159092485905, 0.00833755824714899, 0.06147446110844612, 0.056954700499773026, 0.0030909162014722824, 0.01978727988898754, 0.03770425543189049, 0.0040319086983799934, 0.020770879462361336, 0.1212616041302681, 0.08733709901571274, 0.07036232948303223, 0.09933113306760788, 0.00579867186024785, 0.0638149157166481, 0.021079927682876587, 0.0, 0.0, 0.0], [0.022769225761294365, 0.0023638142738491297, 0.006953616160899401, 0.0009745148126967251, 0.033431269228458405, 0.03096311166882515, 0.012881089001893997, 0.0763968825340271, 0.0002621040621306747, 0.030449911952018738, 0.03674258664250374, 0.0005670606624335051, 0.014569046907126904, 0.0003055331180803478, 0.005948364268988371, 0.022492552176117897, 0.0007025585509836674, 0.004866448696702719, 0.06026189401745796, 0.155859112739563, 0.0005001766257919371, 0.019988082349300385, 0.03086848184466362, 0.0031281516421586275, 0.012151176109910011, 0.11994649469852448, 0.07597395777702332, 0.09628814458847046, 0.07584254443645477, 0.0010643760906532407, 0.03271331638097763, 0.011111120693385601, 0.0006633003358729184, 0.0, 0.0], [0.06755813211202621, 0.003352795960381627, 0.008972212672233582, 0.0034025399945676327, 0.056073062121868134, 0.03040468879044056, 0.013628317043185234, 0.07952429354190826, 0.0025889414828270674, 0.03472939878702164, 0.015315979719161987, 0.0025271004997193813, 0.0076854717917740345, 0.0026928295847028494, 0.007112996652722359, 0.026128459721803665, 0.0027733705937862396, 0.007069502491503954, 0.05076209828257561, 0.1759389340877533, 0.0039354367181658745, 0.010542861185967922, 0.012697344645857811, 0.002831439720466733, 0.004983488470315933, 0.05610368400812149, 0.10278657078742981, 0.07864400744438171, 0.06361375004053116, 0.004290643613785505, 0.012449044734239578, 0.007846895605325699, 0.0035741638857871294, 0.03745955228805542, 0.0], [0.058894284069538116, 0.004834294784814119, 0.008816244080662727, 0.004917601589113474, 0.057244494557380676, 0.04719964787364006, 0.010384359396994114, 0.04373685643076897, 0.0032532154582440853, 0.02700052596628666, 0.015385997481644154, 0.005444495938718319, 0.018304405733942986, 0.0036100554279983044, 0.011856134049594402, 0.07255370914936066, 0.0060180788859725, 0.006411905400454998, 0.05586158111691475, 0.08374673873186111, 0.003973665181547403, 0.01859058439731598, 0.019445115700364113, 0.0038806754164397717, 0.00893265102058649, 0.12242711335420609, 0.040986329317092896, 0.07429768145084381, 0.05774552375078201, 0.006318635307252407, 0.013084109872579575, 0.023065801709890366, 0.0057965172454714775, 0.04094139114022255, 0.015039589256048203]], [[1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.9780336022377014, 0.02196638472378254, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.5068985819816589, 0.4279147982597351, 0.06518658995628357, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.3459503948688507, 0.2338510900735855, 0.19138817489147186, 0.2288103550672531, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.278596967458725, 0.1752573400735855, 0.11037088930606842, 0.25180962681770325, 0.18396517634391785, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.2080419510602951, 0.08827141672372818, 0.06160813197493553, 0.2584840953350067, 0.27056989073753357, 0.1130245104432106, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.1632561981678009, 0.12141193449497223, 0.11990715563297272, 0.16166433691978455, 0.10774069279432297, 0.297503799200058, 0.028515927493572235, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.12280938029289246, 0.03978986665606499, 0.045145437121391296, 0.1357850283384323, 0.09269555658102036, 0.11808720231056213, 0.3369846045970917, 0.10870300233364105, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0785447210073471, 0.03570389747619629, 0.029105374589562416, 0.05200906842947006, 0.10322204977273941, 0.06222519278526306, 0.06801237165927887, 0.17029154300689697, 0.40088582038879395, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.08555340021848679, 0.04508208855986595, 0.020755309611558914, 0.05748791992664337, 0.058533765375614166, 0.03812539577484131, 0.03164026886224747, 0.14470814168453217, 0.39396727085113525, 0.12414637207984924, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.034007564187049866, 0.005801304709166288, 0.004264353774487972, 0.014554576948285103, 0.002058126265183091, 0.010052230209112167, 0.015462434850633144, 0.014919927343726158, 0.07456015050411224, 0.802941620349884, 0.021377811208367348, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.03658134117722511, 0.008552633225917816, 0.009085924364626408, 0.010718596167862415, 0.03151055425405502, 0.07786073535680771, 0.02809949219226837, 0.052033890038728714, 0.09175645560026169, 0.2498076856136322, 0.22384566068649292, 0.18014709651470184, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.03899502009153366, 0.013953045010566711, 0.027150379493832588, 0.01597864180803299, 0.006068869959563017, 0.015332907438278198, 0.007358450908213854, 0.008143595419824123, 0.09236099570989609, 0.09582377970218658, 0.39708951115608215, 0.22508278489112854, 0.05666204169392586, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.029417650774121284, 0.006853992585092783, 0.0054335338063538074, 0.008306889794766903, 0.015738297253847122, 0.008708908222615719, 0.008349093608558178, 0.0239200871437788, 0.04694686457514763, 0.04814499616622925, 0.09309989213943481, 0.15290863811969757, 0.14854182302951813, 0.40362927317619324, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.03712834417819977, 0.007584467530250549, 0.0035451501607894897, 0.008117974735796452, 0.018834184855222702, 0.005904983729124069, 0.009632605127990246, 0.011076660826802254, 0.04249940440058708, 0.015083995647728443, 0.019843513146042824, 0.11177283525466919, 0.12816877663135529, 0.3331785202026367, 0.2476285845041275, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.024791467934846878, 0.002257236046716571, 0.0019112296868115664, 0.003149644238874316, 0.0017746664816513658, 0.0030808611772954464, 0.019880710169672966, 0.006427091546356678, 0.015317468903958797, 0.0032266047783195972, 0.02029605582356453, 0.05772731453180313, 0.022555232048034668, 0.0888594463467598, 0.7141124606132507, 0.014632428996264935, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.01935279741883278, 0.0027357228100299835, 0.002700645010918379, 0.0028163986280560493, 0.007300628814846277, 0.016268713399767876, 0.005134701728820801, 0.010258646681904793, 0.013791469857096672, 0.03380446135997772, 0.03394484892487526, 0.02411044016480446, 0.053474850952625275, 0.0969100371003151, 0.2761882543563843, 0.22190387547016144, 0.1793035864830017, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.016778292134404182, 0.0013836680445820093, 0.0012504433980211616, 0.0019532646983861923, 0.0012283106334507465, 0.00342704844661057, 0.0017710181418806314, 0.0025876814033836126, 0.009956520982086658, 0.007537418510764837, 0.004963119514286518, 0.023668868467211723, 0.017456641420722008, 0.06967750936746597, 0.2612767815589905, 0.24082417786121368, 0.176779106259346, 0.15748003125190735, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.05475045368075371, 0.0092019559815526, 0.0037833035457879305, 0.006891877856105566, 0.007006867788732052, 0.005267785396426916, 0.0019135895418003201, 0.006649005692452192, 0.02249075472354889, 0.007203691639006138, 0.03985992446541786, 0.026306960731744766, 0.03598390519618988, 0.10654626786708832, 0.14742793142795563, 0.029531922191381454, 0.14834147691726685, 0.23425251245498657, 0.10658968240022659, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.08004870265722275, 0.0035627458710223436, 0.003394146915525198, 0.011285574175417423, 0.012208585627377033, 0.00194349919911474, 0.003920826129615307, 0.024116620421409607, 0.023349978029727936, 0.008975887671113014, 0.005343955475836992, 0.02533608302474022, 0.013870184309780598, 0.10103641450405121, 0.039738137274980545, 0.08061381429433823, 0.12086474895477295, 0.12294673919677734, 0.14132516086101532, 0.17611819505691528, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.013053350150585175, 0.001979563618078828, 0.0016182108083739877, 0.0014078539097681642, 0.0014868242433294654, 0.0013099147472530603, 0.0016642940463498235, 0.0025665140710771084, 0.003062670351937413, 0.01034056581556797, 0.00775525439530611, 0.00892941839993, 0.00886953342705965, 0.01716509833931923, 0.07265786826610565, 0.08664459735155106, 0.06223949417471886, 0.1652846485376358, 0.07255084812641144, 0.09155543148517609, 0.36785799264907837, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.012316004373133183, 0.0011592358350753784, 0.0007622396224178374, 0.0019507501274347305, 0.0012718511279672384, 0.0006892279488965869, 0.0005458979867398739, 0.0026029988657683134, 0.003917662892490625, 0.0005409363657236099, 0.0012741911923512816, 0.006503170356154442, 0.005551589652895927, 0.017973465844988823, 0.013353358022868633, 0.03805125877261162, 0.04303402826189995, 0.03958650678396225, 0.022300681099295616, 0.038983650505542755, 0.5949540138244629, 0.15267722308635712, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.01372495573014021, 0.001612572348676622, 0.000328986847307533, 0.0010084322420880198, 0.0011758028995245695, 0.0007259288686327636, 0.0005918748793192208, 0.0009615601156838238, 0.003118623746559024, 0.005970016121864319, 0.0030334654729813337, 0.0033444964792579412, 0.00840754248201847, 0.013658499345183372, 0.005607022438198328, 0.005321722477674484, 0.019289935007691383, 0.029109619557857513, 0.013599625788629055, 0.03133779764175415, 0.2513725757598877, 0.5305221080780029, 0.05617687106132507, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.01600237376987934, 0.0009560455800965428, 0.0005680465255863965, 0.0015549720264971256, 0.0015755719505250454, 0.0006219731294550002, 0.0011257645674049854, 0.0014368733391165733, 0.002310051117092371, 0.0009028328931890428, 0.0013921475037932396, 0.0036725979298353195, 0.0036322835367172956, 0.0092173982411623, 0.008623870089650154, 0.027025725692510605, 0.021376458927989006, 0.03330108895897865, 0.03885404020547867, 0.039058391004800797, 0.26596641540527344, 0.19372618198394775, 0.09792878478765488, 0.22917014360427856, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.00998727511614561, 0.0009651743457652628, 0.0006041577435098588, 0.0009222123771905899, 0.0008805806282907724, 0.0005974328960292041, 0.00021149194799363613, 0.0006455676048062742, 0.0012417678954079747, 0.0006859682616777718, 0.0007031488930806518, 0.0014577409019693732, 0.0018487683264538646, 0.005077192094177008, 0.001927564269863069, 0.012413379736244678, 0.008568298071622849, 0.012073671445250511, 0.02193666435778141, 0.029194166883826256, 0.14272761344909668, 0.09834042936563492, 0.11847671866416931, 0.3163197338581085, 0.2121933251619339, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.016860870644450188, 0.0019703248981386423, 0.0008858360815793276, 0.001765637076459825, 0.0023928354494273663, 0.0005271157715469599, 0.0003617893671616912, 0.0006361044361256063, 0.0022411311510950327, 0.0008515850640833378, 0.00038157840026542544, 0.0022748264018446207, 0.0018798478413373232, 0.007292239926755428, 0.002146677579730749, 0.00252631027251482, 0.010355663485825062, 0.01196929533034563, 0.006590914912521839, 0.012827490456402302, 0.12130489200353622, 0.06624654680490494, 0.1461239904165268, 0.2828073501586914, 0.24703294038772583, 0.04974616318941116, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.02640502154827118, 0.0014231036184355617, 0.0016234461218118668, 0.002576404018327594, 0.002689703833311796, 0.0012257512426003814, 0.0010898751206696033, 0.0009538470185361803, 0.001922354567795992, 0.0024115247651934624, 0.0024579435121268034, 0.0014354200102388859, 0.00537849310785532, 0.0047323452308773994, 0.001939113950356841, 0.00795750692486763, 0.006086882669478655, 0.01265742164105177, 0.011602520011365414, 0.029096607118844986, 0.08598878979682922, 0.07665885984897614, 0.036240145564079285, 0.13133367896080017, 0.19490283727645874, 0.21766312420368195, 0.131547212600708, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.02389606647193432, 0.0020674970000982285, 0.0025093783624470234, 0.0018511989619582891, 0.0031609688885509968, 0.0009005634346976876, 0.00026854194584302604, 0.0005213557160459459, 0.0017872125608846545, 0.001930894679389894, 0.0010158239165320992, 0.0010442674392834306, 0.0015677050687372684, 0.0040601021610200405, 0.0008207398350350559, 0.003358301939442754, 0.004117121454328299, 0.00722301984205842, 0.014547336846590042, 0.04635157436132431, 0.05601867288351059, 0.038400206714868546, 0.04160995036363602, 0.10494542121887207, 0.15788210928440094, 0.12118793278932571, 0.19090263545513153, 0.1660533994436264, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.04075780510902405, 0.0039583672769367695, 0.00569809740409255, 0.003957987762987614, 0.0014047824079170823, 0.00032769449171610177, 0.0006401925347745419, 0.0012045823968946934, 0.0022385611664503813, 0.00074485078221187, 0.003836581017822027, 0.0016206823056563735, 0.0010148317087441683, 0.004261189140379429, 0.0009652961743995547, 0.00039543298771604896, 0.005401076748967171, 0.007263397332280874, 0.005766804795712233, 0.029991863295435905, 0.04173656553030014, 0.017527639865875244, 0.025155046954751015, 0.07565265893936157, 0.24542605876922607, 0.06668495386838913, 0.2648320496082306, 0.055024150758981705, 0.08651075512170792, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.011784793809056282, 0.001394756487570703, 0.0009515649871900678, 0.0008953195065259933, 0.0010372702963650227, 0.0004972649621777236, 0.00025283839204348624, 0.000431757012847811, 0.0005345696699805558, 0.0002822917012963444, 0.0004274726379662752, 0.0005479563260450959, 0.0008038735832087696, 0.0011037472868338227, 0.0010882873320952058, 0.0026353802531957626, 0.002003316767513752, 0.0029394521843641996, 0.005114200059324503, 0.009162313304841518, 0.020236415788531303, 0.0143972709774971, 0.014386598952114582, 0.05287305265665054, 0.04425809904932976, 0.07174734026193619, 0.08077166229486465, 0.11779717355966568, 0.14486132562160492, 0.3947826623916626, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0129928570240736, 0.0016676151426509023, 0.001002107048407197, 0.0011761246714740992, 0.000763836142141372, 0.0004005825612694025, 0.00012849258200731128, 0.00010518680210225284, 0.0005606440827250481, 0.0001407938834745437, 0.00012207562394905835, 0.00039859904791228473, 0.00022988322598394006, 0.0011077820090577006, 0.0003631030267570168, 0.0008891809266060591, 0.0014402453089132905, 0.0014520202530547976, 0.002630566945299506, 0.0031673884950578213, 0.017224669456481934, 0.010722354054450989, 0.00502299889922142, 0.04577849432826042, 0.04751964285969734, 0.027067124843597412, 0.05216864496469498, 0.05500787869095802, 0.11481019854545593, 0.4456881284713745, 0.1482507437467575, 0.0, 0.0, 0.0, 0.0], [0.011719991452991962, 0.0009638671763241291, 0.0016792771639302373, 0.0009598777978681028, 0.0011835338082164526, 0.00035873398883268237, 0.0001312519161729142, 0.00020985820447094738, 0.00035185785964131355, 0.00011673831613734365, 0.0003326716832816601, 0.0002089981862809509, 0.00016304015298373997, 0.0005591599037870765, 0.00017877874779514968, 0.0005847535212524235, 0.0006520423921756446, 0.0008100864943116903, 0.0008232014952227473, 0.006545142736285925, 0.007790667470544577, 0.006622522138059139, 0.005052969325333834, 0.014489851891994476, 0.024942051619291306, 0.01901029422879219, 0.02852550521492958, 0.036787718534469604, 0.14266960322856903, 0.23489910364151, 0.3008541166782379, 0.14982269704341888, 0.0, 0.0, 0.0], [0.011463278904557228, 0.000901482708286494, 0.000547745090443641, 0.0005869876476936042, 0.0005616619600914419, 0.00023434797185473144, 0.0001394466671627015, 0.0001992678880924359, 0.0003140303888358176, 0.00012514636910054833, 0.0002147952327504754, 0.0002481343690305948, 0.00022906105732545257, 0.0005020766402594745, 0.0004519019857980311, 0.000667673593852669, 0.0007638602401129901, 0.0009467164636589587, 0.0010094366734847426, 0.0037195689510554075, 0.007444895803928375, 0.004500558599829674, 0.0018344353884458542, 0.012973608449101448, 0.015522489324212074, 0.02183825708925724, 0.016228938475251198, 0.02469741925597191, 0.038053248077631, 0.16302376985549927, 0.11826054006814957, 0.22884753346443176, 0.3229476511478424, 0.0, 0.0], [0.009539718739688396, 0.0014243266778066754, 0.0009944578632712364, 0.0009655571193434298, 0.0007854420691728592, 0.00030160631285980344, 0.00011159550922457129, 0.0008093682699836791, 0.00040054621058516204, 8.517083915648982e-05, 0.0006307570147328079, 0.0002086153399432078, 0.0002525558229535818, 0.0005451837787404656, 0.00024275551550090313, 0.0005828296416439116, 0.0005503927241079509, 0.0010548564605414867, 0.0010979132493957877, 0.004033542703837156, 0.0051515535451471806, 0.007251250557601452, 0.003988178446888924, 0.01226037833839655, 0.013561532832682133, 0.038286540657281876, 0.06675270944833755, 0.013111504726111889, 0.10093274712562561, 0.13179118931293488, 0.05432307347655296, 0.15251494944095612, 0.30832454562187195, 0.0671326294541359, 0.0], [0.009234556928277016, 0.0018696925835683942, 0.0015434927772730589, 0.0012878893176093698, 0.0009349413448944688, 0.0002825604169629514, 0.00044942708336748183, 0.0005594989634118974, 0.00032776829903014004, 7.203839777503163e-05, 0.00010657912935130298, 0.00026362380594946444, 0.00036547277704812586, 0.0003748307644855231, 0.00031175336334854364, 0.000896912591997534, 0.0006347452872432768, 0.0008650010568089783, 0.0008020041859708726, 0.0006307890289463103, 0.0032187793403863907, 0.003845833009108901, 0.003000435419380665, 0.007814163342118263, 0.010107207112014294, 0.01672154851257801, 0.007672084029763937, 0.010031207464635372, 0.005356386303901672, 0.07757057994604111, 0.037744469940662384, 0.13809098303318024, 0.19567596912384033, 0.03074355237185955, 0.43059319257736206]], [[1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.8970723748207092, 0.10292768478393555, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.7731762528419495, 0.17108607292175293, 0.05573778599500656, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.2190985083580017, 0.09592839330434799, 0.15399740636348724, 0.5309756398200989, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.4932920038700104, 0.12421076744794846, 0.08378948271274567, 0.21044501662254333, 0.08826278895139694, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.4792534112930298, 0.11483929306268692, 0.05825280025601387, 0.12140919268131256, 0.19296900928020477, 0.033276256173849106, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.33760714530944824, 0.13038283586502075, 0.12365775555372238, 0.1684832125902176, 0.039485640823841095, 0.1552271842956543, 0.045156169682741165, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.3124314844608307, 0.10335515439510345, 0.07322120666503906, 0.13077445328235626, 0.08650774508714676, 0.20371730625629425, 0.047141339629888535, 0.042851340025663376, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.10719956457614899, 0.032564811408519745, 0.04615398496389389, 0.12522779405117035, 0.011105242185294628, 0.03433005511760712, 0.007858381606638432, 0.011366589926183224, 0.624193549156189, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.3028891682624817, 0.07804971188306808, 0.05937644839286804, 0.0942690446972847, 0.17377148568630219, 0.03861624747514725, 0.04802526906132698, 0.03951255977153778, 0.13237622380256653, 0.03311377763748169, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.13027672469615936, 0.0421348512172699, 0.07258211076259613, 0.10922861844301224, 0.036304350942373276, 0.19112573564052582, 0.03369797393679619, 0.05715864151716232, 0.1118721291422844, 0.16414138674736023, 0.05147748440504074, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0946442112326622, 0.029253434389829636, 0.03811940923333168, 0.14989584684371948, 0.020670881494879723, 0.076317198574543, 0.008947466500103474, 0.020045090466737747, 0.30578556656837463, 0.0475209578871727, 0.010440824553370476, 0.1983591765165329, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.14440791308879852, 0.06329526752233505, 0.07617758214473724, 0.07194560766220093, 0.033941056579351425, 0.08442069590091705, 0.035940203815698624, 0.0768553763628006, 0.11588279157876968, 0.05951957032084465, 0.06183500215411186, 0.148417130112648, 0.027361800894141197, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0414787232875824, 0.011494416743516922, 0.0176435224711895, 0.045997254550457, 0.004085008520632982, 0.013701113872230053, 0.0028709012549370527, 0.004510778002440929, 0.24164846539497375, 0.00972951389849186, 0.0026167351752519608, 0.26417702436447144, 0.008053770288825035, 0.3319928050041199, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.15155857801437378, 0.0377250611782074, 0.027330640703439713, 0.06052473559975624, 0.030143292620778084, 0.028034891933202744, 0.013341410085558891, 0.022503048181533813, 0.11653425544500351, 0.051899757236242294, 0.01055865827947855, 0.10229867696762085, 0.015167278237640858, 0.14137376844882965, 0.1910059005022049, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.2839046120643616, 0.06502397358417511, 0.03238338604569435, 0.029591113328933716, 0.04243205115199089, 0.028469767421483994, 0.04153313860297203, 0.08465306460857391, 0.043854981660842896, 0.02753039076924324, 0.09218152612447739, 0.037511132657527924, 0.04891905188560486, 0.04984966665506363, 0.07913844287395477, 0.013023803941905499, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.048857614398002625, 0.014007377438247204, 0.019542600959539413, 0.07337553799152374, 0.010089149698615074, 0.03972572460770607, 0.004310054238885641, 0.010590332560241222, 0.153035968542099, 0.02649904601275921, 0.005417257081717253, 0.09647627174854279, 0.012806715443730354, 0.2047865390777588, 0.14743390679359436, 0.00968919601291418, 0.12335673719644547, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.05943121388554573, 0.025951191782951355, 0.03547388315200806, 0.03907974809408188, 0.018049318343400955, 0.023087237030267715, 0.01050374936312437, 0.030069541186094284, 0.09918276220560074, 0.03614410012960434, 0.01841004192829132, 0.11741486191749573, 0.017167558893561363, 0.12769994139671326, 0.08459020406007767, 0.037902720272541046, 0.14577235281467438, 0.0740695372223854, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.272489070892334, 0.04502272978425026, 0.04612232744693756, 0.04205195605754852, 0.038135677576065063, 0.05454428121447563, 0.025706572458148003, 0.019254140555858612, 0.04536456614732742, 0.051068808883428574, 0.021479330956935883, 0.05142934247851372, 0.035683080554008484, 0.05110248550772667, 0.04221018776297569, 0.03813280165195465, 0.05789823830127716, 0.023540074005723, 0.03876428306102753, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.23981237411499023, 0.037238847464323044, 0.04641323909163475, 0.018074778839945793, 0.02860943414270878, 0.03156362473964691, 0.02326364815235138, 0.05314121022820473, 0.03672982007265091, 0.04436158761382103, 0.01409001275897026, 0.04079655557870865, 0.013384039513766766, 0.040182776749134064, 0.025124983862042427, 0.022283190861344337, 0.04457440972328186, 0.026357244700193405, 0.018502147868275642, 0.1954960972070694, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.03447255864739418, 0.0070497882552444935, 0.008516225963830948, 0.019823290407657623, 0.0020061929244548082, 0.008738485164940357, 0.001380530884489417, 0.0025849866215139627, 0.12963861227035522, 0.0049805231392383575, 0.001454295590519905, 0.09793510288000107, 0.0042426553554832935, 0.17474916577339172, 0.020575357601046562, 0.002921728417277336, 0.12519225478172302, 0.017666751518845558, 0.008172447793185711, 0.006463979370892048, 0.3214350640773773, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.07532121986150742, 0.017385125160217285, 0.026499604806303978, 0.03820112720131874, 0.013464298099279404, 0.01700492575764656, 0.005682713817805052, 0.008161391131579876, 0.0879102498292923, 0.015652015805244446, 0.0046223802492022514, 0.06870009005069733, 0.01706838794052601, 0.11098402738571167, 0.08843359351158142, 0.015540490858256817, 0.08290655165910721, 0.03445708379149437, 0.034902919083833694, 0.022669056430459023, 0.16727587580680847, 0.04715687036514282, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.11039687693119049, 0.01761860027909279, 0.012073222547769547, 0.07903020828962326, 0.04707183688879013, 0.028393471613526344, 0.010853296145796776, 0.03898293897509575, 0.0425417497754097, 0.05313578620553017, 0.01061658188700676, 0.040667418390512466, 0.016578687354922295, 0.05044250562787056, 0.04148309677839279, 0.011694188229739666, 0.04894394427537918, 0.04587528854608536, 0.06803461164236069, 0.03145821765065193, 0.07373654097318649, 0.09332364797592163, 0.027047282084822655, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.05010591819882393, 0.011403637006878853, 0.018979988992214203, 0.04829108342528343, 0.005478168372064829, 0.027938123792409897, 0.0034753396175801754, 0.009667656384408474, 0.08883960545063019, 0.021459797397255898, 0.00918425153940916, 0.06738954037427902, 0.00629378343001008, 0.11730515211820602, 0.055593885481357574, 0.023259559646248817, 0.08538825809955597, 0.030172359198331833, 0.018553776666522026, 0.04972720518708229, 0.1386994868516922, 0.03593091666698456, 0.02957056649029255, 0.04729193449020386, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.03365062549710274, 0.03973434865474701, 0.013943267054855824, 0.022155016660690308, 0.00737410131841898, 0.012534351088106632, 0.0039379289373755455, 0.005588038824498653, 0.05276412516832352, 0.011353479698300362, 0.0029761551413685083, 0.06121998280286789, 0.0071145338006317616, 0.066092848777771, 0.018457189202308655, 0.004707182291895151, 0.07780604809522629, 0.015934070572257042, 0.016269713640213013, 0.027332961559295654, 0.08945080637931824, 0.028283195570111275, 0.013428346253931522, 0.24125005304813385, 0.126641646027565, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.10162997245788574, 0.0481734462082386, 0.02274150960147381, 0.017647169530391693, 0.0163576602935791, 0.014718177728354931, 0.020238911733031273, 0.03009830042719841, 0.03836436942219734, 0.025297513231635094, 0.03959346562623978, 0.04179809242486954, 0.018494969233870506, 0.043456628918647766, 0.023876942694187164, 0.013411926105618477, 0.04804755747318268, 0.021995294839143753, 0.030744776129722595, 0.05168475583195686, 0.05786486715078354, 0.04216015338897705, 0.05039001256227493, 0.08024896681308746, 0.076081782579422, 0.024882828816771507, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.06603522598743439, 0.02066010795533657, 0.021681705489754677, 0.03408760949969292, 0.015279402956366539, 0.018442003056406975, 0.010789582505822182, 0.05595837160944939, 0.035654857754707336, 0.02173886075615883, 0.025664953514933586, 0.03946555778384209, 0.01909187063574791, 0.042604971677064896, 0.0451403483748436, 0.012913587503135204, 0.047355812042951584, 0.0255618616938591, 0.035244170576334, 0.09746367484331131, 0.05611315742135048, 0.035938460379838943, 0.03256699815392494, 0.04301988333463669, 0.046976711601018906, 0.0324612595140934, 0.06208907440304756, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.10541093349456787, 0.016973095014691353, 0.020341018214821815, 0.03266654163599014, 0.02185116335749626, 0.03178995102643967, 0.005571200046688318, 0.02419450134038925, 0.016740456223487854, 0.016082558780908585, 0.013484909199178219, 0.022790348157286644, 0.020844878628849983, 0.019816391170024872, 0.00741198705509305, 0.013035068288445473, 0.026982448995113373, 0.010132250376045704, 0.01766769029200077, 0.29726260900497437, 0.023457154631614685, 0.03235581889748573, 0.03224535658955574, 0.019980110228061676, 0.05017194151878357, 0.04827892780303955, 0.03308820351958275, 0.01937239058315754, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.11478213965892792, 0.02845187857747078, 0.030158087611198425, 0.02989071048796177, 0.015577719546854496, 0.045236311852931976, 0.008156674914062023, 0.04677766561508179, 0.020574573427438736, 0.018089203163981438, 0.016638457775115967, 0.02435152791440487, 0.014281020499765873, 0.022870570421218872, 0.024558445438742638, 0.02152225561439991, 0.02774014323949814, 0.012263183481991291, 0.016789693385362625, 0.030601810663938522, 0.03310445696115494, 0.04230614751577377, 0.05013417825102806, 0.039895664900541306, 0.025471460074186325, 0.03835655376315117, 0.14836563169956207, 0.04062195122241974, 0.012431965209543705, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.010402973741292953, 0.003988315816968679, 0.0075365533120930195, 0.02371407300233841, 0.0015747002325952053, 0.004556502681225538, 0.0011461445828899741, 0.0020206549670547247, 0.08388333022594452, 0.006093548145145178, 0.0012243660166859627, 0.08011974394321442, 0.0032829290721565485, 0.1148514673113823, 0.020138554275035858, 0.003874024609103799, 0.10355658084154129, 0.01623992808163166, 0.006219519302248955, 0.011158797889947891, 0.12897233664989471, 0.016916099935770035, 0.008813060820102692, 0.028795935213565826, 0.06059350445866585, 0.005618599709123373, 0.016893552616238594, 0.008791642263531685, 0.017022380605340004, 0.20200029015541077, 0.0, 0.0, 0.0, 0.0, 0.0], [0.05472033470869064, 0.015954436734318733, 0.01209825836122036, 0.02283375896513462, 0.01898520439863205, 0.028783921152353287, 0.008076783269643784, 0.00837206095457077, 0.03956403583288193, 0.012315313331782818, 0.009588327258825302, 0.023571254685521126, 0.010643109679222107, 0.04663844779133797, 0.014515888877213001, 0.012394601479172707, 0.02759736031293869, 0.021691448986530304, 0.020180558785796165, 0.01823921874165535, 0.05496654659509659, 0.03711786866188049, 0.02424498274922371, 0.03499606251716614, 0.1494767814874649, 0.029019523411989212, 0.04327646642923355, 0.015866434201598167, 0.04511536285281181, 0.11428382992744446, 0.024871762841939926, 0.0, 0.0, 0.0, 0.0], [0.02826632745563984, 0.00834258459508419, 0.009693150408565998, 0.024003304541110992, 0.012406982481479645, 0.009731616824865341, 0.007254774682223797, 0.011378849856555462, 0.04525342211127281, 0.014661099761724472, 0.0047597987577319145, 0.03279305249452591, 0.00959976576268673, 0.056626733392477036, 0.023795735090970993, 0.00461410079151392, 0.03982144594192505, 0.02123899571597576, 0.02528654783964157, 0.10159800946712494, 0.09278310090303421, 0.026123862713575363, 0.015091427601873875, 0.03458564355969429, 0.061821889132261276, 0.013407012447714806, 0.016889790073037148, 0.030893530696630478, 0.028194855898618698, 0.1103137955069542, 0.04056955873966217, 0.03819922357797623, 0.0, 0.0, 0.0], [0.005920098163187504, 0.0026448273565620184, 0.005754884332418442, 0.02199717052280903, 0.0013081292854622006, 0.005726539995521307, 0.0008704040665179491, 0.0018940434092655778, 0.052954934537410736, 0.004725011996924877, 0.0009999447502195835, 0.04342705383896828, 0.0021749832667410374, 0.0736372247338295, 0.017015766352415085, 0.0030869953334331512, 0.057790882885456085, 0.010734480805695057, 0.003425823524594307, 0.010332699865102768, 0.08509068191051483, 0.012429154478013515, 0.00623022997751832, 0.022220993414521217, 0.0641111209988594, 0.005053484346717596, 0.018152359873056412, 0.007594145834445953, 0.006786441896110773, 0.16716445982456207, 0.01387179084122181, 0.014755615033209324, 0.25011759996414185, 0.0, 0.0], [0.05958712100982666, 0.014287109486758709, 0.009688674472272396, 0.0284745953977108, 0.02860482595860958, 0.02625933475792408, 0.00881962850689888, 0.0221360232681036, 0.017910214141011238, 0.026397570967674255, 0.017243580892682076, 0.02598377875983715, 0.006316179875284433, 0.020043160766363144, 0.015440632589161396, 0.025940269231796265, 0.030103035271167755, 0.01836536079645157, 0.02313041128218174, 0.02621459774672985, 0.031157266348600388, 0.02898617833852768, 0.028961259871721268, 0.016435230150818825, 0.03480091691017151, 0.03794369474053383, 0.0738067701458931, 0.030772417783737183, 0.0285172201693058, 0.07387098670005798, 0.0245877243578434, 0.03523408994078636, 0.08090808987617493, 0.0230720192193985, 0.0], [0.052589867264032364, 0.008282380178570747, 0.004686587955802679, 0.011556106619536877, 0.002124359365552664, 0.007376666646450758, 0.0025699115358293056, 0.00516730360686779, 0.06430472433567047, 0.00868867989629507, 0.004349820781499147, 0.054052021354436874, 0.003567327279597521, 0.08101718127727509, 0.021457049995660782, 0.004144076257944107, 0.06621565669775009, 0.018716931343078613, 0.009488232433795929, 0.006038549356162548, 0.10274302214384079, 0.014565220102667809, 0.009106297977268696, 0.021657664328813553, 0.035248007625341415, 0.004852636251598597, 0.008422153070569038, 0.019589481875300407, 0.01188589259982109, 0.14464661478996277, 0.016607563942670822, 0.02283477410674095, 0.08476628363132477, 0.01512686163187027, 0.05155406519770622]], [[1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.8803902268409729, 0.11960975080728531, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.800162672996521, 0.14422568678855896, 0.055611662566661835, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.6471941471099854, 0.14701780676841736, 0.0793101042509079, 0.12647800147533417, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.6291002035140991, 0.13269934058189392, 0.0830739215016365, 0.10306606441736221, 0.05206046625971794, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.5631014704704285, 0.12375933676958084, 0.1115972250699997, 0.10842735320329666, 0.08343343436717987, 0.00968119129538536, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.49733906984329224, 0.11400076001882553, 0.07851860672235489, 0.10408828407526016, 0.08229988068342209, 0.08720176666975021, 0.03655165433883667, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.4618663489818573, 0.10720598697662354, 0.07680258899927139, 0.09525880962610245, 0.11179576814174652, 0.04683656245470047, 0.0632898136973381, 0.03694409132003784, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.40880170464515686, 0.11223535984754562, 0.06194951757788658, 0.08138205111026764, 0.0688871219754219, 0.04759938269853592, 0.057795677334070206, 0.06300587207078934, 0.09834318608045578, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.33778563141822815, 0.0776895135641098, 0.06164025142788887, 0.08030417561531067, 0.09243480116128922, 0.12569788098335266, 0.043102774769067764, 0.08006509393453598, 0.08753132075071335, 0.013748615980148315, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.40844565629959106, 0.07972612231969833, 0.059432800859212875, 0.07098966836929321, 0.06258387863636017, 0.07298709452152252, 0.05290916562080383, 0.06663113087415695, 0.08486582338809967, 0.030785245820879936, 0.0106433667242527, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.3284089267253876, 0.08126053959131241, 0.05222117155790329, 0.0667019784450531, 0.06221221387386322, 0.046883273869752884, 0.04748227819800377, 0.04943714290857315, 0.08644368499517441, 0.05108509957790375, 0.04660409688949585, 0.08125960826873779, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.28456375002861023, 0.0733422264456749, 0.059566278010606766, 0.05995681881904602, 0.0626126304268837, 0.04492250457406044, 0.0535488985478878, 0.055395208299160004, 0.0760505199432373, 0.05439150705933571, 0.055210113525390625, 0.08481415361166, 0.03562533110380173, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.29869240522384644, 0.0805349051952362, 0.04630003497004509, 0.058416109532117844, 0.05245253071188927, 0.03589674457907677, 0.043130062520504, 0.04816412925720215, 0.07179585099220276, 0.03588228300213814, 0.03498636186122894, 0.06638610363006592, 0.04749885946512222, 0.07986357063055038, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.26184871792793274, 0.06895220279693604, 0.0466209277510643, 0.05860457941889763, 0.04692879691720009, 0.055680133402347565, 0.04915208742022514, 0.04117780551314354, 0.07490755617618561, 0.031984250992536545, 0.03257223218679428, 0.07257068902254105, 0.03739875555038452, 0.08542973548173904, 0.03617137670516968, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.2941747307777405, 0.053609177470207214, 0.03860876336693764, 0.049526747316122055, 0.040609195828437805, 0.04149327054619789, 0.02971625328063965, 0.04536982625722885, 0.06024930626153946, 0.04159967228770256, 0.06637390702962875, 0.05982465669512749, 0.040502894669771194, 0.06734267622232437, 0.06095736846327782, 0.010041486471891403, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.24078050255775452, 0.05731502175331116, 0.03812979906797409, 0.04762377217411995, 0.04672614485025406, 0.035154759883880615, 0.034621819853782654, 0.0374724306166172, 0.06234779208898544, 0.0395454503595829, 0.0357443131506443, 0.05874496325850487, 0.03407841920852661, 0.07071568816900253, 0.054391153156757355, 0.039451487362384796, 0.06715653091669083, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.1900811344385147, 0.05546912923455238, 0.038249872624874115, 0.044684696942567825, 0.043491918593645096, 0.03729135915637016, 0.03500032797455788, 0.034133393317461014, 0.05614842474460602, 0.02901819348335266, 0.026478586718440056, 0.06265047937631607, 0.035022005438804626, 0.06568855792284012, 0.05714194104075432, 0.04781632870435715, 0.07356445491313934, 0.06806918978691101, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.23406481742858887, 0.04765678569674492, 0.03626637905836105, 0.04350151866674423, 0.04482332244515419, 0.030926160514354706, 0.024145539849996567, 0.03047104924917221, 0.05100264772772789, 0.03852144256234169, 0.041283562779426575, 0.05289096757769585, 0.027892781421542168, 0.0573887899518013, 0.060669999569654465, 0.04387103393673897, 0.05933021008968353, 0.05117018520832062, 0.02412271313369274, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.31211698055267334, 0.04305698350071907, 0.030739206820726395, 0.039901215583086014, 0.04073919728398323, 0.03397355601191521, 0.013991206884384155, 0.032501693814992905, 0.042142923921346664, 0.03136739507317543, 0.05722316727042198, 0.042099729180336, 0.032853271812200546, 0.04662720486521721, 0.04874774068593979, 0.021730590611696243, 0.047223784029483795, 0.029857566580176353, 0.045303892344236374, 0.007802656851708889, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.1958305388689041, 0.05227474123239517, 0.032512083649635315, 0.037058886140584946, 0.043354302644729614, 0.027641305699944496, 0.027452072128653526, 0.03308156877756119, 0.04532438889145851, 0.02540675923228264, 0.02478162571787834, 0.03888048604130745, 0.030247287824749947, 0.050895120948553085, 0.04128777235746384, 0.03749211132526398, 0.04425322636961937, 0.04428722336888313, 0.04519397392868996, 0.05072397366166115, 0.07202056795358658, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.16079995036125183, 0.04841072857379913, 0.03302096948027611, 0.0344170518219471, 0.03531978279352188, 0.023897038772702217, 0.02606349252164364, 0.028639739379286766, 0.04453874006867409, 0.02494947426021099, 0.02971615456044674, 0.04012998938560486, 0.024859050288796425, 0.05134395509958267, 0.0537354052066803, 0.045276302844285965, 0.04680594801902771, 0.03460599482059479, 0.0424550399184227, 0.05485450476408005, 0.06976332515478134, 0.04639733210206032, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.18932510912418365, 0.039103612303733826, 0.024852009490132332, 0.03345049172639847, 0.038601603358983994, 0.02410929463803768, 0.02050211653113365, 0.024301758036017418, 0.03976468741893768, 0.03418166935443878, 0.03484784811735153, 0.036197811365127563, 0.023668933659791946, 0.04502483084797859, 0.034989431500434875, 0.026825517416000366, 0.04153791069984436, 0.0373666025698185, 0.042022787034511566, 0.055910442024469376, 0.06404418498277664, 0.056186337023973465, 0.0331849679350853, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.16596610844135284, 0.03388110548257828, 0.020657528191804886, 0.032127268612384796, 0.03747254237532616, 0.02651948109269142, 0.019882909953594208, 0.02560964785516262, 0.04091741889715195, 0.02261033095419407, 0.019720878452062607, 0.03376534581184387, 0.023503810167312622, 0.046445246785879135, 0.03541430085897446, 0.027368422597646713, 0.0387694351375103, 0.03146074339747429, 0.04639929160475731, 0.04867471754550934, 0.06382153183221817, 0.06529752910137177, 0.046299051493406296, 0.047415394335985184, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.1551409810781479, 0.03631479665637016, 0.022236892953515053, 0.02809741534292698, 0.04015664756298065, 0.024055982008576393, 0.01740090735256672, 0.021498555317521095, 0.0367506705224514, 0.01798674277961254, 0.019703920930624008, 0.02832777239382267, 0.022604597732424736, 0.041680529713630676, 0.030497964471578598, 0.025585591793060303, 0.03248760476708412, 0.027014952152967453, 0.04112286865711212, 0.04738447815179825, 0.0638234093785286, 0.05655010789632797, 0.04119996353983879, 0.06263256818056107, 0.05974407121539116, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.12975943088531494, 0.027339698746800423, 0.02995750866830349, 0.022702835500240326, 0.03781534731388092, 0.03144635632634163, 0.017136765643954277, 0.04656490683555603, 0.030618300661444664, 0.01878168247640133, 0.025273190811276436, 0.028721965849399567, 0.01909092627465725, 0.03480999544262886, 0.026170235127210617, 0.02681773528456688, 0.03296039626002312, 0.024960245937108994, 0.05474761873483658, 0.057801052927970886, 0.05617622286081314, 0.03305984288454056, 0.05767417699098587, 0.04667419195175171, 0.062286995351314545, 0.02065236121416092, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.18126998841762543, 0.027924926951527596, 0.028055904433131218, 0.029457401484251022, 0.028532760217785835, 0.020307771861553192, 0.019550152122974396, 0.03153110295534134, 0.029232844710350037, 0.02111053466796875, 0.014580844901502132, 0.022505316883325577, 0.016256744042038918, 0.031508855521678925, 0.024519694969058037, 0.03659053146839142, 0.02503451704978943, 0.020030668005347252, 0.04726612567901611, 0.12142347544431686, 0.0418962687253952, 0.03646091744303703, 0.03734691068530083, 0.03705747425556183, 0.03521094471216202, 0.024784497916698456, 0.01055284682661295, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.1419323831796646, 0.02687341347336769, 0.02339440956711769, 0.024198878556489944, 0.034410905092954636, 0.03176822513341904, 0.01428282167762518, 0.027884982526302338, 0.027823282405734062, 0.020267944782972336, 0.020075349137187004, 0.02603541687130928, 0.015271040610969067, 0.03118356317281723, 0.021344449371099472, 0.030780881643295288, 0.02968650311231613, 0.024966171011328697, 0.03290622681379318, 0.0812656506896019, 0.04738018661737442, 0.0395050123333931, 0.0381060391664505, 0.03920024633407593, 0.047230444848537445, 0.032237764447927475, 0.04768497869372368, 0.022302789613604546, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.1936107724905014, 0.028910469263792038, 0.017410732805728912, 0.024991950020194054, 0.020412294194102287, 0.03692350164055824, 0.018564047291874886, 0.022491103038191795, 0.026469899341464043, 0.019851727411150932, 0.01670689322054386, 0.02012958936393261, 0.017302924767136574, 0.028877539560198784, 0.01937028393149376, 0.021512921899557114, 0.02260466292500496, 0.01946551352739334, 0.03218546137213707, 0.09415603429079056, 0.03964082524180412, 0.04692869633436203, 0.03296547010540962, 0.03432748094201088, 0.03549890220165253, 0.04152423143386841, 0.0380084365606308, 0.024193765595555305, 0.004963918589055538, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.13062264025211334, 0.029238127171993256, 0.02056247740983963, 0.024924932047724724, 0.026433374732732773, 0.02088005654513836, 0.01689508929848671, 0.020366206765174866, 0.030794674530625343, 0.016811521723866463, 0.014352645725011826, 0.026399672031402588, 0.017968133091926575, 0.03419143706560135, 0.02579362690448761, 0.020880063995718956, 0.029983345419168472, 0.02257683128118515, 0.031984008848667145, 0.0387752428650856, 0.047299277037382126, 0.03951811417937279, 0.03569551929831505, 0.046477824449539185, 0.04560372233390808, 0.02769399806857109, 0.02837626449763775, 0.02950696274638176, 0.03388815000653267, 0.06550605595111847, 0.0, 0.0, 0.0, 0.0, 0.0], [0.10399015247821808, 0.021966129541397095, 0.016451172530651093, 0.022787973284721375, 0.02435716614127159, 0.019975431263446808, 0.013678607530891895, 0.02052461914718151, 0.026416653767228127, 0.01582498662173748, 0.01812475360929966, 0.02293805405497551, 0.015249819494783878, 0.029951775446534157, 0.019568635150790215, 0.017790060490369797, 0.026331383734941483, 0.020062364637851715, 0.030405661091208458, 0.05837804451584816, 0.04746946692466736, 0.03560008108615875, 0.03361855447292328, 0.03905162215232849, 0.0462346076965332, 0.03415835276246071, 0.05063629522919655, 0.03304954245686531, 0.041327618062496185, 0.060074809938669205, 0.034005604684352875, 0.0, 0.0, 0.0, 0.0], [0.10689456015825272, 0.023425208404660225, 0.01831112429499626, 0.022276200354099274, 0.019986260682344437, 0.018345672637224197, 0.015893511474132538, 0.019802751019597054, 0.02582935057580471, 0.016079476103186607, 0.01665685884654522, 0.02581820636987686, 0.012892822735011578, 0.029111457988619804, 0.020739195868372917, 0.017621520906686783, 0.02965446189045906, 0.024281147867441177, 0.028045887127518654, 0.052053920924663544, 0.04263105243444443, 0.03251810744404793, 0.03738033026456833, 0.03843435272574425, 0.03982122987508774, 0.026290135458111763, 0.035206377506256104, 0.03138628974556923, 0.03282344713807106, 0.061947062611579895, 0.037625446915626526, 0.0402165912091732, 0.0, 0.0, 0.0], [0.1222902461886406, 0.02516898512840271, 0.016714073717594147, 0.02203325182199478, 0.023000748828053474, 0.015408718958497047, 0.014958021230995655, 0.018676776438951492, 0.026191165670752525, 0.012370293959975243, 0.01071577612310648, 0.021631909534335136, 0.01398462988436222, 0.02889922820031643, 0.023200996220111847, 0.018216602504253387, 0.024281252175569534, 0.018235325813293457, 0.021653978154063225, 0.03633742406964302, 0.03855651989579201, 0.03691945970058441, 0.03379225730895996, 0.042588260024785995, 0.03839157149195671, 0.023295754566788673, 0.030805258080363274, 0.024493301287293434, 0.030105777084827423, 0.056526120752096176, 0.037566863000392914, 0.04159916564822197, 0.051390357315540314, 0.0, 0.0], [0.1387171745300293, 0.02196725271642208, 0.016682986170053482, 0.02155136875808239, 0.026049219071865082, 0.01537913829088211, 0.012522307224571705, 0.020025212317705154, 0.02187984809279442, 0.014553075656294823, 0.01562824472784996, 0.018288590013980865, 0.012421734631061554, 0.023633437231183052, 0.01722067780792713, 0.020091675221920013, 0.020151861011981964, 0.018530182540416718, 0.02421250194311142, 0.07234080880880356, 0.03423936292529106, 0.030147051438689232, 0.031458813697099686, 0.0302029587328434, 0.030948752537369728, 0.024833213537931442, 0.030743282288312912, 0.022636551409959793, 0.044566914439201355, 0.04173140227794647, 0.028924647718667984, 0.02986532635986805, 0.042881425470113754, 0.02497304417192936, 0.0], [0.12327982485294342, 0.02670745551586151, 0.014834711328148842, 0.018021512776613235, 0.01776500791311264, 0.013351651839911938, 0.014073795638978481, 0.018723033368587494, 0.02398008666932583, 0.021256299689412117, 0.017004817724227905, 0.020616671070456505, 0.012905520386993885, 0.026097441092133522, 0.021554268896579742, 0.015845777466893196, 0.022780928760766983, 0.023595379665493965, 0.019861742854118347, 0.02192237414419651, 0.03597113490104675, 0.03369683399796486, 0.02117166295647621, 0.03536053001880646, 0.03450924903154373, 0.021897651255130768, 0.035124246031045914, 0.02411290630698204, 0.03299051150679588, 0.046776384115219116, 0.03042444959282875, 0.03270317241549492, 0.03790726140141487, 0.042624928057193756, 0.04055080935359001]], [[1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.6533874273300171, 0.3466125726699829, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.4451119303703308, 0.3337680399417877, 0.22112005949020386, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.5459808111190796, 0.1529107540845871, 0.0764545276761055, 0.22465386986732483, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.4272030293941498, 0.1412525624036789, 0.06143971160054207, 0.11979990452528, 0.2503049373626709, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.4097931981086731, 0.11732899397611618, 0.06548821926116943, 0.12367456406354904, 0.042223699390888214, 0.24149127304553986, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.3643515110015869, 0.15547120571136475, 0.08527876436710358, 0.088326096534729, 0.049937352538108826, 0.03409624099731445, 0.22253882884979248, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.35626861453056335, 0.09063339233398438, 0.05006815120577812, 0.10755762457847595, 0.06895481050014496, 0.046696074306964874, 0.04097530245780945, 0.2388460338115692, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.34613895416259766, 0.14562253654003143, 0.09470638632774353, 0.10499643534421921, 0.06400176137685776, 0.040472403168678284, 0.037579216063022614, 0.04448216035962105, 0.12200022488832474, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.2051023542881012, 0.04824313893914223, 0.05157081410288811, 0.09953965991735458, 0.06673120707273483, 0.045586880296468735, 0.036483924835920334, 0.08169859647750854, 0.08583585917949677, 0.27920758724212646, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.2301696389913559, 0.07576317340135574, 0.06243027374148369, 0.0619683638215065, 0.050248246639966965, 0.03796261176466942, 0.05770514905452728, 0.05617325380444527, 0.06585754454135895, 0.041665591299533844, 0.26005616784095764, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.2668823003768921, 0.09969265758991241, 0.06880500167608261, 0.10110104084014893, 0.036257680505514145, 0.04876028001308441, 0.031795434653759, 0.03426744043827057, 0.09192277491092682, 0.04406016319990158, 0.024279993027448654, 0.1521752029657364, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.24808596074581146, 0.059437528252601624, 0.03645306080579758, 0.09643886238336563, 0.04665255919098854, 0.04358060657978058, 0.02640967071056366, 0.038718465715646744, 0.06937099993228912, 0.035532157868146896, 0.02358422242105007, 0.05824920907616615, 0.21748670935630798, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.22227582335472107, 0.08784298598766327, 0.06428328156471252, 0.07245951145887375, 0.05153009295463562, 0.03354167193174362, 0.031511515378952026, 0.04395980387926102, 0.10531803965568542, 0.03629286214709282, 0.033496368676424026, 0.07923729717731476, 0.03496610373258591, 0.10328467190265656, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.26255425810813904, 0.0745101273059845, 0.04962191730737686, 0.06513949483633041, 0.022182418033480644, 0.039234112948179245, 0.021433908492326736, 0.02829248271882534, 0.06496229767799377, 0.022169621661305428, 0.018070973455905914, 0.05796554312109947, 0.018188005313277245, 0.061765026301145554, 0.19390983879566193, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.21454612910747528, 0.05207951366901398, 0.04338602349162102, 0.051086731255054474, 0.023057056590914726, 0.03987295553088188, 0.02145753800868988, 0.043103158473968506, 0.05712398886680603, 0.04732411354780197, 0.020076358690857887, 0.04863849654793739, 0.03173767402768135, 0.05389069393277168, 0.03922813758254051, 0.21339137852191925, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.1693464070558548, 0.057584717869758606, 0.04318232461810112, 0.06501618772745132, 0.02576972357928753, 0.035840608179569244, 0.023403940722346306, 0.028633086010813713, 0.06884393841028214, 0.03735613077878952, 0.021079333499073982, 0.12265527248382568, 0.025184892117977142, 0.07096558809280396, 0.049235548824071884, 0.028457902371883392, 0.12744438648223877, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.1294490396976471, 0.043515171855688095, 0.028707362711429596, 0.04557254910469055, 0.021282196044921875, 0.019885340705513954, 0.028857693076133728, 0.03478110581636429, 0.055325932800769806, 0.03088506869971752, 0.02108755148947239, 0.06504066288471222, 0.04043157771229744, 0.05797602981328964, 0.04136980324983597, 0.028544064611196518, 0.06784921884536743, 0.23943956196308136, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.14163610339164734, 0.035789649933576584, 0.0283652376383543, 0.04538755118846893, 0.046378396451473236, 0.03978869691491127, 0.014343786053359509, 0.04959797114133835, 0.04732620343565941, 0.030782438814640045, 0.018412165343761444, 0.04871036112308502, 0.023401623591780663, 0.04745562747120857, 0.03751209378242493, 0.023782696574926376, 0.04894063621759415, 0.03379065543413162, 0.23859812319278717, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.13350322842597961, 0.032220542430877686, 0.02058890275657177, 0.03780217468738556, 0.041080109775066376, 0.036710936576128006, 0.025954410433769226, 0.04136637970805168, 0.0432656966149807, 0.030096469447016716, 0.026812905445694923, 0.04010695591568947, 0.024385901167988777, 0.04430919513106346, 0.0337374247610569, 0.02071092277765274, 0.040797438472509384, 0.03265015408396721, 0.03506242856383324, 0.2588377892971039, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.11974354833364487, 0.039148829877376556, 0.028956566005945206, 0.033427078276872635, 0.03769594058394432, 0.02054390124976635, 0.01755896583199501, 0.0326584167778492, 0.06372420489788055, 0.03310193121433258, 0.029867954552173615, 0.053869184106588364, 0.022512970492243767, 0.07236673682928085, 0.04143426939845085, 0.03790142014622688, 0.06002623587846756, 0.04586019739508629, 0.05091097205877304, 0.03195589408278465, 0.1267348825931549, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.11878621578216553, 0.03957201912999153, 0.02322298102080822, 0.05364079028367996, 0.032616786658763885, 0.017370950430631638, 0.011663909070193768, 0.016273824498057365, 0.050143033266067505, 0.013260260224342346, 0.010486909188330173, 0.03813210129737854, 0.04193669930100441, 0.053362227976322174, 0.03097998909652233, 0.01961771957576275, 0.0402415469288826, 0.03244158998131752, 0.020540136843919754, 0.026845518499612808, 0.05771109089255333, 0.25115376710891724, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.10596504807472229, 0.03639821335673332, 0.023730667307972908, 0.02770346961915493, 0.029520345851778984, 0.016958512365818024, 0.009277228266000748, 0.019281188026070595, 0.04353449493646622, 0.025435717776417732, 0.015441058203577995, 0.041617270559072495, 0.018472321331501007, 0.04617343842983246, 0.026975233107805252, 0.016877291724085808, 0.043659280985593796, 0.020190229639410973, 0.018645791336894035, 0.016359694302082062, 0.05703480541706085, 0.06433958560228348, 0.27640917897224426, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.08669397979974747, 0.1400514841079712, 0.05404341593384743, 0.026910433545708656, 0.026933522894978523, 0.01693662256002426, 0.017527243122458458, 0.01113263238221407, 0.03316717594861984, 0.00911658350378275, 0.01402170117944479, 0.028125278651714325, 0.012107602320611477, 0.03627435863018036, 0.030991647392511368, 0.015099070966243744, 0.03022145852446556, 0.020815471187233925, 0.018955033272504807, 0.021182602271437645, 0.043649882078170776, 0.04377346858382225, 0.025040632113814354, 0.23722870647907257, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.10505463182926178, 0.024722840636968613, 0.023438595235347748, 0.024256445467472076, 0.021477704867720604, 0.0141111109405756, 0.012018013745546341, 0.01570729725062847, 0.04219583421945572, 0.016803370788693428, 0.01115935854613781, 0.03127181902527809, 0.020320696756243706, 0.047023333609104156, 0.03988678380846977, 0.025853371247649193, 0.034509580582380295, 0.024522604420781136, 0.017594223842024803, 0.021597160026431084, 0.06303168088197708, 0.05036986991763115, 0.026549125090241432, 0.04091530665755272, 0.24560929834842682, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.07146888226270676, 0.015683099627494812, 0.019199566915631294, 0.027655117213726044, 0.025888752192258835, 0.011495746672153473, 0.013834549114108086, 0.025006256997585297, 0.032857511192560196, 0.040690407156944275, 0.019407076761126518, 0.03628278151154518, 0.026671355590224266, 0.03698844835162163, 0.025391360744833946, 0.012203286401927471, 0.040816254913806915, 0.022441426292061806, 0.018680060282349586, 0.03714069724082947, 0.05291351303458214, 0.033748991787433624, 0.03600706905126572, 0.027396058663725853, 0.04157792031764984, 0.2485538274049759, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.09520060569047928, 0.023399081081151962, 0.01624995470046997, 0.037287019193172455, 0.02706948108971119, 0.024426201358437538, 0.016916703432798386, 0.02051924355328083, 0.03561446815729141, 0.017246786504983902, 0.01696518063545227, 0.030670633539557457, 0.01476178877055645, 0.03783894330263138, 0.025590214878320694, 0.013281543739140034, 0.03275727853178978, 0.016981473192572594, 0.01291283406317234, 0.040966760367155075, 0.04452374204993248, 0.03288853541016579, 0.02186199650168419, 0.028905075043439865, 0.025222204625606537, 0.015681948512792587, 0.2742603123188019, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.07106205075979233, 0.022234492003917694, 0.01544252596795559, 0.028806833550333977, 0.02491171658039093, 0.01409667357802391, 0.010925301350653172, 0.022356443107128143, 0.028251033276319504, 0.02592524327337742, 0.012314549647271633, 0.036213621497154236, 0.017662765458226204, 0.03140704706311226, 0.01859503611922264, 0.0181714054197073, 0.040066663175821304, 0.022728286683559418, 0.025000544264912605, 0.04014962539076805, 0.04249804466962814, 0.02628159709274769, 0.023634908720850945, 0.03034823387861252, 0.028446586802601814, 0.04931708052754402, 0.04533269256353378, 0.22781899571418762, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.1199868842959404, 0.03201312571763992, 0.02036377042531967, 0.03192231059074402, 0.02011149190366268, 0.017741890624165535, 0.014963151887059212, 0.014099691063165665, 0.0323602668941021, 0.012826558202505112, 0.01920422911643982, 0.024741899222135544, 0.014775941148400307, 0.03365081921219826, 0.03285077214241028, 0.015633026137948036, 0.026330972090363503, 0.0099469143897295, 0.01001527439802885, 0.023254675790667534, 0.038984283804893494, 0.03491940349340439, 0.011989149264991283, 0.030530868098139763, 0.023813728243112564, 0.018979018554091454, 0.06846081465482712, 0.019141539931297302, 0.22638753056526184, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.0711139589548111, 0.023044995963573456, 0.017407843843102455, 0.025577733293175697, 0.01881728135049343, 0.01440666988492012, 0.011364508420228958, 0.019742920994758606, 0.0378037728369236, 0.01749836653470993, 0.012214535847306252, 0.034711796790361404, 0.016245106235146523, 0.04494825005531311, 0.031461041420698166, 0.017976267263293266, 0.040775004774332047, 0.02623201534152031, 0.02910454571247101, 0.032863158732652664, 0.06312328577041626, 0.039967380464076996, 0.03193744271993637, 0.050057604908943176, 0.0500398650765419, 0.02629847638309002, 0.040675751864910126, 0.03425348177552223, 0.03361441567540169, 0.0867224857211113, 0.0, 0.0, 0.0, 0.0, 0.0], [0.06342680007219315, 0.018928559496998787, 0.015915200114250183, 0.022974470630288124, 0.01581357978284359, 0.011996155604720116, 0.006959729827940464, 0.010440586134791374, 0.027604447677731514, 0.01420377753674984, 0.011755013838410378, 0.026974201202392578, 0.015696221962571144, 0.03244096040725708, 0.021464021876454353, 0.016300221905112267, 0.0315910279750824, 0.01829562894999981, 0.024074910208582878, 0.018299637362360954, 0.04905860498547554, 0.03294462710618973, 0.031695034354925156, 0.04070066660642624, 0.05526787415146828, 0.032283999025821686, 0.01833721622824669, 0.02062193490564823, 0.020615682005882263, 0.061421558260917664, 0.21189765632152557, 0.0, 0.0, 0.0, 0.0], [0.05903121456503868, 0.01165033783763647, 0.011906321160495281, 0.020305117592215538, 0.015824560075998306, 0.011342638172209263, 0.012165091931819916, 0.020514551550149918, 0.024635151028633118, 0.01787753589451313, 0.008976953104138374, 0.027097541838884354, 0.010151364840567112, 0.029500089585781097, 0.022961730137467384, 0.011527130380272865, 0.032491881400346756, 0.02021907828748226, 0.02027125656604767, 0.027604965493083, 0.04230692982673645, 0.022920358926057816, 0.027398396283388138, 0.024839362129569054, 0.0284199807792902, 0.041721608489751816, 0.0314212329685688, 0.03520328924059868, 0.030631236732006073, 0.06006474792957306, 0.045377373695373535, 0.19364093244075775, 0.0, 0.0, 0.0], [0.0859057679772377, 0.014704097993671894, 0.010510571300983429, 0.0286998450756073, 0.016654493287205696, 0.012795397080481052, 0.009235279634594917, 0.01481921412050724, 0.028232717886567116, 0.011867674067616463, 0.009268181398510933, 0.02304435707628727, 0.017553262412548065, 0.032954029738903046, 0.025013742968440056, 0.012247500009834766, 0.026799557730555534, 0.013326537795364857, 0.016479847952723503, 0.034728214144706726, 0.0398080088198185, 0.05067337676882744, 0.03086116909980774, 0.027551839128136635, 0.036164239048957825, 0.01740611158311367, 0.047160595655441284, 0.02589184232056141, 0.02839728258550167, 0.06047135591506958, 0.03584859520196915, 0.03886668384075165, 0.11605867743492126, 0.0, 0.0], [0.05670773610472679, 0.015243275091052055, 0.00989258661866188, 0.017495425418019295, 0.03529244288802147, 0.015370241366326809, 0.010747747495770454, 0.017091594636440277, 0.02124500460922718, 0.013064499013125896, 0.006394115276634693, 0.02176818624138832, 0.011015361174941063, 0.02329082600772381, 0.02097102627158165, 0.01234665047377348, 0.024119658395648003, 0.013815043494105339, 0.01682049222290516, 0.0345151387155056, 0.030974794179201126, 0.032177552580833435, 0.03227025642991066, 0.023270366713404655, 0.027259424328804016, 0.01785033941268921, 0.05776429921388626, 0.02427157573401928, 0.02524302527308464, 0.03774581849575043, 0.015895817428827286, 0.026596592739224434, 0.042151615023612976, 0.20932143926620483, 0.0], [0.04736970737576485, 0.013491548597812653, 0.008601200766861439, 0.011534366756677628, 0.03260952979326248, 0.008750018663704395, 0.010495422407984734, 0.015973465517163277, 0.02422492578625679, 0.013959752395749092, 0.011584172025322914, 0.020906994119286537, 0.012310189194977283, 0.028499368578195572, 0.014403233304619789, 0.011428188532590866, 0.024424541741609573, 0.02128405123949051, 0.027576277032494545, 0.025762228295207024, 0.0491495355963707, 0.021549465134739876, 0.02072588913142681, 0.030232448130846024, 0.046837326139211655, 0.02172248810529709, 0.02025720290839672, 0.0191495344042778, 0.01053390558809042, 0.05002593249082565, 0.030405571684241295, 0.022812239825725555, 0.029462486505508423, 0.03996320068836212, 0.20198367536067963]], [[1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.8253123164176941, 0.17468774318695068, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.7580503821372986, 0.1302967369556427, 0.11165298521518707, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.7232239842414856, 0.0827442854642868, 0.07870826870203018, 0.11532340198755264, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.5594541430473328, 0.0869443267583847, 0.0911555141210556, 0.09393444657325745, 0.1685115545988083, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.5031397938728333, 0.08192651718854904, 0.058723147958517075, 0.10480904579162598, 0.1521420031785965, 0.09925953298807144, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.5279609560966492, 0.058259766548871994, 0.0683659166097641, 0.11652137339115143, 0.11039458215236664, 0.06918033212423325, 0.04931708052754402, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.30010077357292175, 0.0882599800825119, 0.05834931880235672, 0.10984238982200623, 0.1686263382434845, 0.1289689540863037, 0.023227626457810402, 0.12262456119060516, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.3934404253959656, 0.04758616164326668, 0.05194966495037079, 0.0747157633304596, 0.11950993537902832, 0.05938037484884262, 0.023368611931800842, 0.1026640310883522, 0.12738502025604248, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.3446025848388672, 0.05032897740602493, 0.0599132776260376, 0.04483130946755409, 0.1573815941810608, 0.06782456487417221, 0.03229008987545967, 0.07052528858184814, 0.08953907340765, 0.08276329189538956, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.33508479595184326, 0.046269968152046204, 0.04785766080021858, 0.10917434096336365, 0.044352367520332336, 0.044907234609127045, 0.02844386361539364, 0.0792062059044838, 0.10731321573257446, 0.06832483410835266, 0.08906550705432892, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.31157219409942627, 0.03927759826183319, 0.035212744027376175, 0.0755099505186081, 0.08078574389219284, 0.04503686726093292, 0.02880651131272316, 0.10218673944473267, 0.08797682076692581, 0.05585267394781113, 0.08571922779083252, 0.052062999457120895, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.361469566822052, 0.06311224400997162, 0.026989920064806938, 0.07095800340175629, 0.03985174745321274, 0.029446503147482872, 0.022796371951699257, 0.07072697579860687, 0.11304392665624619, 0.03335518389940262, 0.03714423254132271, 0.04763941466808319, 0.08346593379974365, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.26885920763015747, 0.032650742679834366, 0.03938189148902893, 0.05081148445606232, 0.08684233576059341, 0.0431019626557827, 0.01792234741151333, 0.06655362248420715, 0.07785409688949585, 0.045289043337106705, 0.06586872041225433, 0.04606606438755989, 0.07721763104200363, 0.08158092200756073, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.20790530741214752, 0.030814696103334427, 0.02946297638118267, 0.05067421868443489, 0.08743482828140259, 0.047705814242362976, 0.027333185076713562, 0.09152115881443024, 0.05214141309261322, 0.04708247631788254, 0.0808199942111969, 0.054532747715711594, 0.1207483783364296, 0.05516168475151062, 0.01666121371090412, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.19932864606380463, 0.042067065834999084, 0.05715770646929741, 0.07193327695131302, 0.05065881088376045, 0.03152455389499664, 0.027136800810694695, 0.04140906035900116, 0.09322270005941391, 0.05202208831906319, 0.025422831997275352, 0.059279125183820724, 0.06613773852586746, 0.0958227589726448, 0.03163217008113861, 0.055244654417037964, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.2371978908777237, 0.030675802379846573, 0.029208479449152946, 0.05821239575743675, 0.06082989275455475, 0.03495608642697334, 0.02444053627550602, 0.07084924727678299, 0.059838537126779556, 0.03970349207520485, 0.059325896203517914, 0.0367523692548275, 0.07694027572870255, 0.06171644479036331, 0.014049750752747059, 0.06745627522468567, 0.03784666955471039, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.2499532848596573, 0.0266294926404953, 0.039317600429058075, 0.044632118195295334, 0.07087578624486923, 0.034094203263521194, 0.02126503549516201, 0.05726418271660805, 0.05179665610194206, 0.035039860755205154, 0.043922752141952515, 0.039228759706020355, 0.061559099704027176, 0.05122420936822891, 0.015611978247761726, 0.08278114348649979, 0.038917459547519684, 0.03588634356856346, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.22269795835018158, 0.02651810832321644, 0.03978952020406723, 0.0276039969176054, 0.08196066319942474, 0.03582962602376938, 0.015378967858850956, 0.06874661892652512, 0.05343088135123253, 0.05469823256134987, 0.05721571296453476, 0.03161527216434479, 0.03717070072889328, 0.05228297784924507, 0.011851194314658642, 0.0720609799027443, 0.03087831288576126, 0.0244111530482769, 0.055859167128801346, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.07419285178184509, 0.05092697963118553, 0.02823096327483654, 0.034200042486190796, 0.033614277839660645, 0.05182219669222832, 0.009658376686275005, 0.049406133592128754, 0.09883326292037964, 0.05356486141681671, 0.024581866338849068, 0.05879182368516922, 0.013026789762079716, 0.10889019817113876, 0.01567712053656578, 0.04766635224223137, 0.06507281959056854, 0.038861677050590515, 0.0398426353931427, 0.10313887149095535, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.22614890336990356, 0.025769880041480064, 0.033361680805683136, 0.040344879031181335, 0.06116846576333046, 0.03166023641824722, 0.023036984726786613, 0.04815823957324028, 0.049641285091638565, 0.025703316554427147, 0.051324233412742615, 0.029225803911685944, 0.046272363513708115, 0.04679547995328903, 0.010471501387655735, 0.04346739500761032, 0.02797776833176613, 0.03356106951832771, 0.03209924325346947, 0.05562402680516243, 0.05818728730082512, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.1944299042224884, 0.02000579424202442, 0.025669973343610764, 0.027610067278146744, 0.07351991534233093, 0.02511855587363243, 0.016428446397185326, 0.05443364754319191, 0.04450269415974617, 0.022272871807217598, 0.029331279918551445, 0.03064262494444847, 0.0628659799695015, 0.0433916412293911, 0.01021601539105177, 0.09763482213020325, 0.02960144355893135, 0.02808043546974659, 0.03757162764668465, 0.050374023616313934, 0.060897450894117355, 0.01540081575512886, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.19766591489315033, 0.04115749150514603, 0.03360118344426155, 0.02387016825377941, 0.06633448600769043, 0.02801698073744774, 0.019582949578762054, 0.05178564041852951, 0.031591471284627914, 0.026536161080002785, 0.041643157601356506, 0.025757959112524986, 0.04328407347202301, 0.0299394354224205, 0.01275031827390194, 0.056069158017635345, 0.024677343666553497, 0.021505890414118767, 0.04559291526675224, 0.07480884343385696, 0.044334057718515396, 0.02199736051261425, 0.037497006356716156, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.1804356724023819, 0.023349830880761147, 0.023929741233587265, 0.04454168304800987, 0.04070594906806946, 0.03654097765684128, 0.01661657728254795, 0.05323562026023865, 0.050283920019865036, 0.025846900418400764, 0.04605167731642723, 0.024920539930462837, 0.03976090997457504, 0.0489342138171196, 0.012390019372105598, 0.06191660091280937, 0.02396535314619541, 0.028754474595189095, 0.02509540319442749, 0.06865920126438141, 0.07357670366764069, 0.018046919256448746, 0.022800732403993607, 0.009640373289585114, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.24246364831924438, 0.02469644509255886, 0.03868827223777771, 0.03357230871915817, 0.0547662153840065, 0.033804651349782944, 0.020721616223454475, 0.032754622399806976, 0.04349979758262634, 0.018562667071819305, 0.0364685021340847, 0.03100249171257019, 0.038625460118055344, 0.03993181511759758, 0.010727174580097198, 0.03831769526004791, 0.029280314221978188, 0.02467811480164528, 0.02788931503891945, 0.04502158612012863, 0.043076008558273315, 0.016905829310417175, 0.023586297407746315, 0.007298622280359268, 0.04366063326597214, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.11111284047365189, 0.02945874258875847, 0.024559473618865013, 0.03090251237154007, 0.033864825963974, 0.03363065421581268, 0.012831443920731544, 0.030669258907437325, 0.06390535086393356, 0.02348153665661812, 0.025704598054289818, 0.037669792771339417, 0.016167854890227318, 0.06660734862089157, 0.017231769859790802, 0.030658649280667305, 0.039240479469299316, 0.024295223876833916, 0.026388797909021378, 0.0586402602493763, 0.10666140168905258, 0.017623895779252052, 0.024245431646704674, 0.018279604613780975, 0.03675020486116409, 0.05941798537969589, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.17855216562747955, 0.043738700449466705, 0.02777228318154812, 0.032882511615753174, 0.08083152025938034, 0.0361485481262207, 0.01943814381957054, 0.036047156900167465, 0.03154062479734421, 0.022283848375082016, 0.02674209140241146, 0.020272962749004364, 0.049294859170913696, 0.028057033196091652, 0.010035552084445953, 0.02821899950504303, 0.018526628613471985, 0.025377929210662842, 0.039212264120578766, 0.05273986607789993, 0.024735102429986, 0.028674785047769547, 0.029405977576971054, 0.013893143273890018, 0.017606474459171295, 0.05648735910654068, 0.02148348279297352, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.15546517074108124, 0.04916934296488762, 0.02451593615114689, 0.0257005263119936, 0.05483131483197212, 0.0420926995575428, 0.01164504885673523, 0.06493964791297913, 0.03121975064277649, 0.026562796905636787, 0.028326861560344696, 0.018289044499397278, 0.03490859642624855, 0.029660988599061966, 0.008558595553040504, 0.0466504842042923, 0.01766982302069664, 0.023724349215626717, 0.0190387275069952, 0.07715762406587601, 0.027307387441396713, 0.019747760146856308, 0.024364104494452477, 0.014429941773414612, 0.01892836205661297, 0.05850561708211899, 0.024364039301872253, 0.022225428372621536, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.19332920014858246, 0.026298340409994125, 0.030392466112971306, 0.029611894860863686, 0.045201625674963, 0.03307904675602913, 0.021840767934918404, 0.026594171300530434, 0.04538282752037048, 0.038439467549324036, 0.026915403082966805, 0.02453509159386158, 0.01579570211470127, 0.04036761447787285, 0.01063427235931158, 0.02634548395872116, 0.022409800440073013, 0.019946027547121048, 0.044018495827913284, 0.06709349155426025, 0.033454738557338715, 0.016084952279925346, 0.025615857914090157, 0.01140951830893755, 0.013623220846056938, 0.040554922074079514, 0.016662972047924995, 0.03589887544512749, 0.018463701009750366, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], [0.15965193510055542, 0.018642351031303406, 0.02667771466076374, 0.028047848492860794, 0.04907772317528725, 0.026045657694339752, 0.01780770905315876, 0.03679423779249191, 0.045969460159540176, 0.019110895693302155, 0.03796319663524628, 0.027620645239949226, 0.028341028839349747, 0.042964350432157516, 0.008847594261169434, 0.03253018483519554, 0.025659987702965736, 0.024485090747475624, 0.029966212809085846, 0.04054069519042969, 0.042335037142038345, 0.017694566398859024, 0.021217668429017067, 0.006380944978445768, 0.031033935025334358, 0.03557971492409706, 0.0250661913305521, 0.026607459411025047, 0.02430843375623226, 0.043031543493270874, 0.0, 0.0, 0.0, 0.0, 0.0], [0.1966170370578766, 0.01979137398302555, 0.0330333411693573, 0.025687072426080704, 0.042206067591905594, 0.03137579560279846, 0.01624363474547863, 0.030698437243700027, 0.036294274032115936, 0.026839356869459152, 0.026712195947766304, 0.023046307265758514, 0.021950097754597664, 0.03344835713505745, 0.0103057362139225, 0.022743647918105125, 0.021480808034539223, 0.02109472267329693, 0.031490374356508255, 0.03692729026079178, 0.03723211959004402, 0.015422009862959385, 0.019252333790063858, 0.007320488803088665, 0.024766933172941208, 0.044391706585884094, 0.0198507197201252, 0.026133641600608826, 0.01812445931136608, 0.042709074914455414, 0.036810584366321564, 0.0, 0.0, 0.0, 0.0], [0.16313210129737854, 0.02293247915804386, 0.029583681374788284, 0.028615493327379227, 0.03778465464711189, 0.026518698781728745, 0.026229018345475197, 0.03103414550423622, 0.03483205288648605, 0.018975619226694107, 0.033202942460775375, 0.026435956358909607, 0.032307349145412445, 0.031329210847616196, 0.011727887205779552, 0.027169140055775642, 0.02410650998353958, 0.022620299831032753, 0.025755560025572777, 0.038014139980077744, 0.03207404166460037, 0.017354249954223633, 0.017987210303544998, 0.009293688461184502, 0.04247136414051056, 0.031215850263834, 0.019006570801138878, 0.021523458883166313, 0.01754119247198105, 0.04014519974589348, 0.018160773441195488, 0.04091951623558998, 0.0, 0.0, 0.0], [0.15803849697113037, 0.016828810796141624, 0.02064237929880619, 0.045382529497146606, 0.04664173722267151, 0.027835214510560036, 0.020206507295370102, 0.04211510345339775, 0.03703448176383972, 0.017744602635502815, 0.030353324487805367, 0.024713635444641113, 0.03428484871983528, 0.03341371938586235, 0.009082172997295856, 0.03521377965807915, 0.022280141711235046, 0.020849987864494324, 0.02739020064473152, 0.032314613461494446, 0.0376492440700531, 0.01492078322917223, 0.016553662717342377, 0.006309079471975565, 0.023218737915158272, 0.027949875220656395, 0.023832231760025024, 0.015323548577725887, 0.01767047494649887, 0.02681528404355049, 0.022188855335116386, 0.02869456447660923, 0.03650737553834915, 0.0, 0.0], [0.1701940894126892, 0.03212590143084526, 0.02987009845674038, 0.019615601748228073, 0.054893799126148224, 0.03622058779001236, 0.017152214422822, 0.032326590269804, 0.0315006785094738, 0.029624557122588158, 0.029094273224473, 0.01662425883114338, 0.029989780858159065, 0.02761414460837841, 0.010949771851301193, 0.03436652943491936, 0.014830109663307667, 0.015176055952906609, 0.03050290420651436, 0.041597455739974976, 0.02064122073352337, 0.017041228711605072, 0.02617543190717697, 0.010538817383348942, 0.007441799622029066, 0.034391358494758606, 0.026077089831233025, 0.02961563691496849, 0.013693016022443771, 0.01807429827749729, 0.03374849632382393, 0.029002603143453598, 0.013117276132106781, 0.016172269359230995, 0.0], [0.11417391896247864, 0.016700679436326027, 0.02357623353600502, 0.03235284984111786, 0.055113647133111954, 0.029616650193929672, 0.01717638596892357, 0.05952785536646843, 0.038434237241744995, 0.020421108230948448, 0.0323600098490715, 0.01921744830906391, 0.01913510076701641, 0.03519750386476517, 0.010996303521096706, 0.044014446437358856, 0.016835961490869522, 0.016212482005357742, 0.02384052611887455, 0.04231660068035126, 0.03716841712594032, 0.009007316082715988, 0.01597953774034977, 0.005828783381730318, 0.020826423540711403, 0.028255825862288475, 0.03160592168569565, 0.024252012372016907, 0.02328527718782425, 0.019917495548725128, 0.024559782817959785, 0.02195737510919571, 0.013193142600357533, 0.020208805799484253, 0.03673398494720459]]]}\n",
              "    )\n",
              "    </script>"
            ]
          },
          "metadata": {}
        }
      ],
      "source": [
        "import circuitsvis as cv\n",
        "from IPython.display import display\n",
        "\n",
        "html = cv.attention.attention_patterns(\n",
        "    tokens=reference_gpt2.to_str_tokens(reference_text),\n",
        "    attention=cache[\"pattern\", 0][0]\n",
        ")\n",
        "display(html)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vcT1uWPGfHN2"
      },
      "source": [
        "You can also use the `attention_heads` function, which has similar syntax but presents the information in a different (sometimes more helpful) way.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yMqCi5OYfHN2"
      },
      "source": [
        "<details>\n",
        "<summary>Help - my <code>attention_heads</code> plots are behaving weirdly.</summary>\n",
        "\n",
        "This seems to be a bug in `circuitsvis` - on VSCode, the attention head plots continually shrink in size.\n",
        "\n",
        "Until this is fixed, one way to get around it is to open the plots in your browser. You can do this inline with the `webbrowser` library:\n",
        "\n",
        "```python\n",
        "attn_heads = cv.attention.attention_heads(\n",
        "    tokens=reference_gpt2.to_str_tokens(reference_text),\n",
        "    attention=cache[\"pattern\", 0][0]\n",
        ")\n",
        "\n",
        "path = \"attn_heads.html\"\n",
        "\n",
        "with open(path, \"w\") as f:\n",
        "    f.write(str(attn_heads))\n",
        "\n",
        "webbrowser.open(path)\n",
        "```\n",
        "\n",
        "To check exactly where this is getting saved, you can print your current working directory with `os.getcwd()`.\n",
        "</details>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "972Ltv3vtDOH"
      },
      "source": [
        "---\n",
        "\n",
        "Note - don't worry if you don't get 100% accuracy here; the tests are pretty stringent. Even things like having your `einsum` input arguments in a different order might result in the output being very slightly different. You should be getting at least 99% accuracy though, so if the value is lower then this it probably means you've made a mistake somewhere.\n",
        "\n",
        "Also, this implementation will probably be the most challenging exercise on this page, so don't worry if it takes you some time! You should look at parts of the solution if you're stuck.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CeWonT0afHN2"
      },
      "source": [
        "A couple of notes / hints:\n",
        "\n",
        "* Don't forget the attention score scaling (this should come before the masking).\n",
        "* You can use `torch.where`, or the `torch.masked_fill` function when masking the attention scores.\n",
        "* The `\"IGNORE\"` buffer is a very large negative number. This is the value you should mask your attention scores with (i.e. set them to this number wherever you want the probabilities to be zero). We indicate the existence of a `self.IGNORE` attribute to VSCode's typechecker via the line `IGNORE: Float[Tensor, \"\"]` in the second line of the code below.\n",
        "\n",
        "<details>\n",
        "<summary>Question - why do you think we mask the attention scores by setting them to a large negative number, rather than the attention probabilities by setting them to zero?</summary>\n",
        "\n",
        "If we masked the attention probabilities, then the probabilities would no longer sum to 1.\n",
        "\n",
        "We want to mask the scores and *then* take softmax, so that the probabilities are still valid probabilities (i.e. they sum to 1), and the values in the masked positions have no influence on the model's output.\n",
        "</details>\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wNOKYkvom4R_"
      },
      "outputs": [],
      "source": [
        "class Attention(nn.Module):\n",
        "    IGNORE: Float[Tensor, \"\"]\n",
        "\n",
        "    def __init__(self, cfg: Config):\n",
        "        super().__init__()\n",
        "        self.cfg = cfg\n",
        "        self.W_Q = nn.Parameter(t.empty((cfg.n_heads, cfg.d_model, cfg.d_head)))\n",
        "        self.W_K = nn.Parameter(t.empty((cfg.n_heads, cfg.d_model, cfg.d_head)))\n",
        "        self.W_V = nn.Parameter(t.empty((cfg.n_heads, cfg.d_model, cfg.d_head)))\n",
        "        self.W_O = nn.Parameter(t.empty((cfg.n_heads, cfg.d_head, cfg.d_model)))\n",
        "        self.b_Q = nn.Parameter(t.zeros((cfg.n_heads, cfg.d_head)))\n",
        "        self.b_K = nn.Parameter(t.zeros((cfg.n_heads, cfg.d_head)))\n",
        "        self.b_V = nn.Parameter(t.zeros((cfg.n_heads, cfg.d_head)))\n",
        "        self.b_O = nn.Parameter(t.zeros((cfg.d_model)))\n",
        "        nn.init.normal_(self.W_Q, std=self.cfg.init_range)\n",
        "        nn.init.normal_(self.W_K, std=self.cfg.init_range)\n",
        "        nn.init.normal_(self.W_V, std=self.cfg.init_range)\n",
        "        nn.init.normal_(self.W_O, std=self.cfg.init_range)\n",
        "        self.register_buffer(\"IGNORE\", t.tensor(-1e5, dtype=t.float32, device=device))\n",
        "\n",
        "    def forward(\n",
        "        self, normalized_resid_pre: Float[Tensor, \"batch posn d_model\"]\n",
        "    ) -> Float[Tensor, \"batch posn d_model\"]:\n",
        "\n",
        "        # Linear map from input -> query, key shape [batch, seq_posn, head_index, d_head]\n",
        "        #print(self.W_Q.shape)\n",
        "\n",
        "        query = einops.einsum(normalized_resid_pre, self.W_Q, \"batch seq_posn d_model, head_index d_model d_head  ->batch seq_posn head_index d_head\") + self.b_Q\n",
        "        key = einops.einsum(normalized_resid_pre, self.W_K, \"batch seq_posn d_model, head_index d_model d_head  ->batch seq_posn head_index d_head\") + self.b_K\n",
        "\n",
        "        print(query.shape)\n",
        "\n",
        "        # Dot product every pair of queries and keys to get attn_scores [batch, head_index, query_pos, key_pos] (query = dest, key = source)\n",
        "\n",
        "        attn_scores = einops.einsum(query, key, \"batch query_posn head_index d_head,batch key_posn head_index d_head -> batch head_index query_posn key_posn \")\n",
        "\n",
        "\n",
        "        #print(attn_scores.shape)\n",
        "\n",
        "        # Scale and mask attn_scores to make it lower triangular, i.e. causal\n",
        "\n",
        "        masked_scores = self.apply_causal_mask(attn_scores/(self.W_Q.shape[-1]**(0.5)))\n",
        "\n",
        "\n",
        "        query_probs = masked_scores.softmax(-1) # batch head_index query_posn\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "        # Linear map from input -> value [batch, key_pos, head_index, d_head]\n",
        "\n",
        "        value = einops.einsum(normalized_resid_pre, self.W_V, \"batch seq_posn d_model, head_index d_model d_head -> batch seq_posn head_index d_head\") + self.b_V\n",
        "\n",
        "\n",
        "        #Mix along the key_pos with attn pattern to get z, which is a weighted average of the value vectors [batch, query_pos, head_index, d_head]\n",
        "\n",
        "        #result = torch.einsum('bkhv,bhqv->bkhv', value, query_probs)\n",
        "\n",
        "        z = einops.einsum(value, query_probs,\"batch key_posn head_index d_head, batch head_index query_posn key_posn -> batch query_posn head_index d_head\")\n",
        "\n",
        "        out = einops.einsum(z, self.W_O, \"batch query_pos head_index d_head, head_index d_head d_model -> batch query_pos d_model\") + self.b_O\n",
        "\n",
        "        return out\n",
        "\n",
        "\n",
        "    def apply_causal_mask(\n",
        "        self, attn_scores: Float[Tensor, \"batch n_heads query_pos key_pos\"]\n",
        "    ) -> Float[Tensor, \"batch n_heads query_pos key_pos\"]:\n",
        "        '''\n",
        "        Applies a causal mask to attention scores, and returns masked scores.\n",
        "        '''\n",
        "\n",
        "        ones = t.ones(attn_scores.size(-2), attn_scores.size(-1), device=attn_scores.device)\n",
        "        mask = t.triu(ones, diagonal=1).bool()\n",
        "\n",
        "        attn_scores.masked_fill_(mask, self.IGNORE)\n",
        "        return attn_scores\n",
        "\n",
        "rand_float_test(Attention, [2, 4, 768])\n",
        "load_gpt2_test(Attention, reference_gpt2.blocks[0].attn, cache[\"normalized\", 0, \"ln1\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5phgYX4Um4SA"
      },
      "source": [
        "## MLP\n",
        "\n",
        "```c\n",
        "Difficulty: 🟠🟠⚪⚪⚪\n",
        "Importance: 🟠🟠🟠🟠⚪\n",
        "\n",
        "You should spend up to 10-15 minutes on this exercise.\n",
        "```\n",
        "\n",
        "Next, you should implement the MLP layer, which consists of:\n",
        "\n",
        "* A linear layer, with weight `W_in`, bias `b_in`\n",
        "* A nonlinear functino (we usually use GELU; the function `gelu_new` has been imported for this purpose)\n",
        "* A linear layer, with weight `W_out`, bias `b_out`\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 39,
      "metadata": {
        "id": "5ByI-Sj2m4SA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "aab57853-9577-47f0-81fd-b0c1739f56e3"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Input shape: torch.Size([2, 4, 768])\n",
            "Output shape: torch.Size([2, 4, 768]) \n",
            "\n",
            "Input shape: torch.Size([1, 35, 768])\n",
            "Output shape: torch.Size([1, 35, 768])\n",
            "Reference output shape: torch.Size([1, 35, 768]) \n",
            "\n",
            "100.00% of the values are correct\n",
            "\n"
          ]
        }
      ],
      "source": [
        "class MLP(nn.Module):\n",
        "    def __init__(self, cfg: Config):\n",
        "        super().__init__()\n",
        "        self.cfg = cfg\n",
        "        self.W_in = nn.Parameter(t.empty((cfg.d_model, cfg.d_mlp)))\n",
        "        self.W_out = nn.Parameter(t.empty((cfg.d_mlp, cfg.d_model)))\n",
        "        self.b_in = nn.Parameter(t.zeros((cfg.d_mlp)))\n",
        "        self.b_out = nn.Parameter(t.zeros((cfg.d_model)))\n",
        "        nn.init.normal_(self.W_in, std=self.cfg.init_range)\n",
        "        nn.init.normal_(self.W_out, std=self.cfg.init_range)\n",
        "\n",
        "    def forward(\n",
        "        self, normalized_resid_mid: Float[Tensor, \"batch posn d_model\"]\n",
        "    ) -> Float[Tensor, \"batch posn d_model\"]:\n",
        "\n",
        "        linear_out = einops.einsum(normalized_resid_mid, self.W_in, \"batch posn d_model, d_model d_mlp -> batch posn d_mlp\") + self.b_in\n",
        "        activatied_step = gelu_new(linear_out)\n",
        "        linear_out =  einops.einsum(activatied_step, self.W_out, \"batch posn d_mlp, d_mlp d_model -> batch posn d_model\") + self.b_out\n",
        "        return linear_out\n",
        "\n",
        "rand_float_test(MLP, [2, 4, 768])\n",
        "load_gpt2_test(MLP, reference_gpt2.blocks[0].mlp, cache[\"normalized\", 0, \"ln2\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wZWuD4I4m4SA"
      },
      "source": [
        "## Transformer Block\n",
        "\n",
        "```c\n",
        "Difficulty: 🟠🟠⚪⚪⚪\n",
        "Importance: 🟠🟠🟠⚪⚪\n",
        "\n",
        "You should spend up to ~10 minutes on this exercise.\n",
        "```\n",
        "\n",
        "Now, we can put together the attention, MLP and layernorms into a single transformer block. Remember to implement the residual connections correctly!\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 52,
      "metadata": {
        "id": "ffiFxAJzm4SA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9e5e90e0-877b-4296-f450-8e00321930d7"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Input shape: torch.Size([2, 4, 768])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([2, 4, 12, 64])\n",
            "torch.Size([2, 12, 4, 4])\n",
            "Output shape: torch.Size([2, 4, 768]) \n",
            "\n",
            "Input shape: torch.Size([1, 35, 768])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 35, 12, 64])\n",
            "torch.Size([1, 12, 35, 35])\n",
            "Output shape: torch.Size([1, 35, 768])\n",
            "Reference output shape: torch.Size([1, 35, 768]) \n",
            "\n",
            "100.00% of the values are correct\n",
            "\n"
          ]
        }
      ],
      "source": [
        "class TransformerBlock(nn.Module):\n",
        "    def __init__(self, cfg: Config):\n",
        "        super().__init__()\n",
        "        self.cfg = cfg\n",
        "        self.ln1 = LayerNorm(cfg)\n",
        "        self.attn = Attention(cfg)\n",
        "        self.ln2 = LayerNorm(cfg)\n",
        "        self.mlp = MLP(cfg)\n",
        "\n",
        "    def forward(\n",
        "        self, resid_pre: Float[Tensor, \"batch position d_model\"]\n",
        "    ) -> Float[Tensor, \"batch position d_model\"]:\n",
        "\n",
        "        resid_next = self.attn(self.ln1(resid_pre))\n",
        "        resid_mid =  resid_next+resid_pre\n",
        "        resid_next = self.mlp(self.ln2(resid_mid))\n",
        "        resid_final= resid_next + resid_mid\n",
        "\n",
        "        return resid_final\n",
        "\n",
        "rand_float_test(TransformerBlock, [2, 4, 768])\n",
        "load_gpt2_test(TransformerBlock, reference_gpt2.blocks[0], cache[\"resid_pre\", 0])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mgzAM-VQm4SA"
      },
      "source": [
        "## Unembedding\n",
        "\n",
        "```c\n",
        "Difficulty: 🟠🟠⚪⚪⚪\n",
        "Importance: 🟠🟠🟠⚪⚪\n",
        "\n",
        "You should spend up to ~10 minutes on this exercise.\n",
        "```\n",
        "\n",
        "The unembedding is jus a linear layer (with weight `W_U` and bias `b_U`).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 43,
      "metadata": {
        "id": "XEMB7p_Vm4SA",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0b7d7b5c-0e77-45e3-d198-6f90d4226e38"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Input shape: torch.Size([2, 4, 768])\n",
            "Output shape: torch.Size([2, 4, 50257]) \n",
            "\n",
            "Input shape: torch.Size([1, 35, 768])\n",
            "Output shape: torch.Size([1, 35, 50257])\n",
            "Reference output shape: torch.Size([1, 35, 50257]) \n",
            "\n",
            "100.00% of the values are correct\n",
            "\n"
          ]
        }
      ],
      "source": [
        "class Unembed(nn.Module):\n",
        "    def __init__(self, cfg):\n",
        "        super().__init__()\n",
        "        self.cfg = cfg\n",
        "        self.W_U = nn.Parameter(t.empty((cfg.d_model, cfg.d_vocab)))\n",
        "        nn.init.normal_(self.W_U, std=self.cfg.init_range)\n",
        "        self.b_U = nn.Parameter(t.zeros((cfg.d_vocab), requires_grad=False))\n",
        "\n",
        "    def forward(\n",
        "        self, normalized_resid_final: Float[Tensor, \"batch position d_model\"]\n",
        "    ) -> Float[Tensor, \"batch position d_vocab\"]:\n",
        "        out = einops.einsum(normalized_resid_final, self.W_U, \"batch posn d_model, d_model d_vocab -> batch posn d_vocab\") + self.b_U\n",
        "        return out\n",
        "\n",
        "rand_float_test(Unembed, [2, 4, 768])\n",
        "load_gpt2_test(Unembed, reference_gpt2.unembed, cache[\"ln_final.hook_normalized\"])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_eRyuDxLm4SB"
      },
      "source": [
        "## Full Transformer\n",
        "\n",
        "```c\n",
        "Difficulty: 🟠🟠⚪⚪⚪\n",
        "Importance: 🟠🟠🟠⚪⚪\n",
        "\n",
        "You should spend up to ~10 minutes on this exercise.\n",
        "```\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 60,
      "metadata": {
        "id": "KCCrXlzYm4SB",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2a9cd230-318b-4889-d897-2c1cd7a99194"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Input shape: torch.Size([2, 4])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([2, 4, 12, 64])\n",
            "torch.Size([2, 12, 4, 4])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([2, 4, 12, 64])\n",
            "torch.Size([2, 12, 4, 4])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([2, 4, 12, 64])\n",
            "torch.Size([2, 12, 4, 4])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([2, 4, 12, 64])\n",
            "torch.Size([2, 12, 4, 4])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([2, 4, 12, 64])\n",
            "torch.Size([2, 12, 4, 4])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([2, 4, 12, 64])\n",
            "torch.Size([2, 12, 4, 4])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([2, 4, 12, 64])\n",
            "torch.Size([2, 12, 4, 4])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([2, 4, 12, 64])\n",
            "torch.Size([2, 12, 4, 4])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([2, 4, 12, 64])\n",
            "torch.Size([2, 12, 4, 4])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([2, 4, 12, 64])\n",
            "torch.Size([2, 12, 4, 4])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([2, 4, 12, 64])\n",
            "torch.Size([2, 12, 4, 4])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([2, 4, 12, 64])\n",
            "torch.Size([2, 12, 4, 4])\n",
            "Output shape: torch.Size([2, 4, 50257]) \n",
            "\n",
            "Input shape: torch.Size([1, 45])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 45, 12, 64])\n",
            "torch.Size([1, 12, 45, 45])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 45, 12, 64])\n",
            "torch.Size([1, 12, 45, 45])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 45, 12, 64])\n",
            "torch.Size([1, 12, 45, 45])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 45, 12, 64])\n",
            "torch.Size([1, 12, 45, 45])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 45, 12, 64])\n",
            "torch.Size([1, 12, 45, 45])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 45, 12, 64])\n",
            "torch.Size([1, 12, 45, 45])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 45, 12, 64])\n",
            "torch.Size([1, 12, 45, 45])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 45, 12, 64])\n",
            "torch.Size([1, 12, 45, 45])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 45, 12, 64])\n",
            "torch.Size([1, 12, 45, 45])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 45, 12, 64])\n",
            "torch.Size([1, 12, 45, 45])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 45, 12, 64])\n",
            "torch.Size([1, 12, 45, 45])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 45, 12, 64])\n",
            "torch.Size([1, 12, 45, 45])\n",
            "Output shape: torch.Size([1, 45, 50257])\n",
            "Reference output shape: torch.Size([1, 45, 50257]) \n",
            "\n",
            "100.00% of the values are correct\n",
            "\n"
          ]
        }
      ],
      "source": [
        "class DemoTransformer(nn.Module):\n",
        "    def __init__(self, cfg: Config):\n",
        "        super().__init__()\n",
        "        self.cfg = cfg\n",
        "        self.embed = Embed(cfg)\n",
        "        self.pos_embed = PosEmbed(cfg)\n",
        "        self.blocks = nn.ModuleList([TransformerBlock(cfg) for _ in range(cfg.n_layers)])\n",
        "        self.ln_final = LayerNorm(cfg)\n",
        "        self.unembed = Unembed(cfg)\n",
        "\n",
        "    def forward(self, tokens: Int[Tensor, \"batch position\"]) -> Float[Tensor, \"batch position d_vocab\"]:\n",
        "        # embed first\n",
        "        stream = self.embed(tokens) +self.pos_embed(tokens)\n",
        "\n",
        "        for x in self.blocks:\n",
        "          stream = x(stream)\n",
        "\n",
        "        final_output = self.unembed(self.ln_final(stream))\n",
        "\n",
        "        return final_output\n",
        "\n",
        "\n",
        "rand_int_test(DemoTransformer, [2, 4])\n",
        "load_gpt2_test(DemoTransformer, reference_gpt2, tokens)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yNwh7R1em4SB"
      },
      "source": [
        "**Try it out!**\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "deE6uPlxm4SB"
      },
      "outputs": [],
      "source": [
        "demo_gpt2 = DemoTransformer(Config(debug=False)).to(device)\n",
        "demo_gpt2.load_state_dict(reference_gpt2.state_dict(), strict=False)\n",
        "\n",
        "demo_logits = demo_gpt2(tokens)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gAhstai4m4SB"
      },
      "source": [
        "Let's take a test string, and calculate the loss!\n",
        "\n",
        "We're using the formula for **cross-entropy loss**. The cross entropy loss between a modelled distribution $Q$ and target distribution $P$ is:\n",
        "\n",
        "$$\n",
        "-\\sum_x P(x) \\log Q(x)\n",
        "$$\n",
        "\n",
        "In the case where $P$ is just the empirical distribution from target classes (i.e. $P(x^*) = 1$ for the correct class $x^*$) then this becomes:\n",
        "\n",
        "$$\n",
        "-\\log Q(x^*)\n",
        "$$\n",
        "\n",
        "in other words, the negative log prob of the true classification.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 62,
      "metadata": {
        "id": "UtO9ZsL6m4SB",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c0eaae9b-7213-4d20-cafe-aa5f23624bf4"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Avg cross entropy loss: 4.0441\n",
            "Avg cross entropy loss for uniform distribution: 10.824905\n",
            "Avg probability assigned to correct token: 0.098629\n"
          ]
        }
      ],
      "source": [
        "def get_log_probs(\n",
        "    logits: Float[Tensor, \"batch posn d_vocab\"],\n",
        "    tokens: Int[Tensor, \"batch posn\"]\n",
        ") -> Float[Tensor, \"batch posn-1\"]:\n",
        "\n",
        "    log_probs = logits.log_softmax(dim=-1)\n",
        "    # Get logprobs the first seq_len-1 predictions (so we can compare them with the actual next tokens)\n",
        "    log_probs_for_tokens = log_probs[:, :-1].gather(dim=-1, index=tokens[:, 1:].unsqueeze(-1)).squeeze(-1)\n",
        "\n",
        "    return log_probs_for_tokens\n",
        "\n",
        "\n",
        "pred_log_probs = get_log_probs(demo_logits, tokens)\n",
        "print(f\"Avg cross entropy loss: {-pred_log_probs.mean():.4f}\")\n",
        "print(f\"Avg cross entropy loss for uniform distribution: {math.log(demo_gpt2.cfg.d_vocab):4f}\")\n",
        "print(f\"Avg probability assigned to correct token: {pred_log_probs.exp().mean():4f}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cOChhr71m4SC"
      },
      "source": [
        "We can also greedily generate text, by taking the most likely next token and continually appending it to our prompt before feeding it back into the model:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 64,
      "metadata": {
        "id": "IPEmKN1Bm4SC",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000,
          "referenced_widgets": [
            "2f58958ac8db4bc4812ec0bd24aff8e2",
            "e00a792c05bb48c49cee3d163e520719",
            "ba40c60e3022402392565289776b24fe",
            "64d897a5fdda479680aeb9435714a862",
            "db79598fed524e48acffd496511bcb0a",
            "171b6d5e10c24633bf5c8bb01518254d",
            "7507657803444b9196681b1574679bff",
            "79c112fb4618426a97e95a5c0a76130e",
            "5f8fe99208214f7dbdf9d1e401d60ef0",
            "76ffa589aa1a4ee292f55753aac92ea1",
            "0596bafaec9c43d397e888f7cd80a838"
          ]
        },
        "outputId": "3e9b19cb-7a72-42c5-cf37-5510c7636629"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/100 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "2f58958ac8db4bc4812ec0bd24aff8e2"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 16, 12, 64])\n",
            "torch.Size([1, 12, 16, 16])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 16, 12, 64])\n",
            "torch.Size([1, 12, 16, 16])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 16, 12, 64])\n",
            "torch.Size([1, 12, 16, 16])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 16, 12, 64])\n",
            "torch.Size([1, 12, 16, 16])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 16, 12, 64])\n",
            "torch.Size([1, 12, 16, 16])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 16, 12, 64])\n",
            "torch.Size([1, 12, 16, 16])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 16, 12, 64])\n",
            "torch.Size([1, 12, 16, 16])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 16, 12, 64])\n",
            "torch.Size([1, 12, 16, 16])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 16, 12, 64])\n",
            "torch.Size([1, 12, 16, 16])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 16, 12, 64])\n",
            "torch.Size([1, 12, 16, 16])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 16, 12, 64])\n",
            "torch.Size([1, 12, 16, 16])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 16, 12, 64])\n",
            "torch.Size([1, 12, 16, 16])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 17, 12, 64])\n",
            "torch.Size([1, 12, 17, 17])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 17, 12, 64])\n",
            "torch.Size([1, 12, 17, 17])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 17, 12, 64])\n",
            "torch.Size([1, 12, 17, 17])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 17, 12, 64])\n",
            "torch.Size([1, 12, 17, 17])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 17, 12, 64])\n",
            "torch.Size([1, 12, 17, 17])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 17, 12, 64])\n",
            "torch.Size([1, 12, 17, 17])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 17, 12, 64])\n",
            "torch.Size([1, 12, 17, 17])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 17, 12, 64])\n",
            "torch.Size([1, 12, 17, 17])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 17, 12, 64])\n",
            "torch.Size([1, 12, 17, 17])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 17, 12, 64])\n",
            "torch.Size([1, 12, 17, 17])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 17, 12, 64])\n",
            "torch.Size([1, 12, 17, 17])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 17, 12, 64])\n",
            "torch.Size([1, 12, 17, 17])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 18, 12, 64])\n",
            "torch.Size([1, 12, 18, 18])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 18, 12, 64])\n",
            "torch.Size([1, 12, 18, 18])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 18, 12, 64])\n",
            "torch.Size([1, 12, 18, 18])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 18, 12, 64])\n",
            "torch.Size([1, 12, 18, 18])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 18, 12, 64])\n",
            "torch.Size([1, 12, 18, 18])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 18, 12, 64])\n",
            "torch.Size([1, 12, 18, 18])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 18, 12, 64])\n",
            "torch.Size([1, 12, 18, 18])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 18, 12, 64])\n",
            "torch.Size([1, 12, 18, 18])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 18, 12, 64])\n",
            "torch.Size([1, 12, 18, 18])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 18, 12, 64])\n",
            "torch.Size([1, 12, 18, 18])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 18, 12, 64])\n",
            "torch.Size([1, 12, 18, 18])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 18, 12, 64])\n",
            "torch.Size([1, 12, 18, 18])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 19, 12, 64])\n",
            "torch.Size([1, 12, 19, 19])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 19, 12, 64])\n",
            "torch.Size([1, 12, 19, 19])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 19, 12, 64])\n",
            "torch.Size([1, 12, 19, 19])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 19, 12, 64])\n",
            "torch.Size([1, 12, 19, 19])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 19, 12, 64])\n",
            "torch.Size([1, 12, 19, 19])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 19, 12, 64])\n",
            "torch.Size([1, 12, 19, 19])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 19, 12, 64])\n",
            "torch.Size([1, 12, 19, 19])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 19, 12, 64])\n",
            "torch.Size([1, 12, 19, 19])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 19, 12, 64])\n",
            "torch.Size([1, 12, 19, 19])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 19, 12, 64])\n",
            "torch.Size([1, 12, 19, 19])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 19, 12, 64])\n",
            "torch.Size([1, 12, 19, 19])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 19, 12, 64])\n",
            "torch.Size([1, 12, 19, 19])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 20, 12, 64])\n",
            "torch.Size([1, 12, 20, 20])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 20, 12, 64])\n",
            "torch.Size([1, 12, 20, 20])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 20, 12, 64])\n",
            "torch.Size([1, 12, 20, 20])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 20, 12, 64])\n",
            "torch.Size([1, 12, 20, 20])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 20, 12, 64])\n",
            "torch.Size([1, 12, 20, 20])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 20, 12, 64])\n",
            "torch.Size([1, 12, 20, 20])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 20, 12, 64])\n",
            "torch.Size([1, 12, 20, 20])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 20, 12, 64])\n",
            "torch.Size([1, 12, 20, 20])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 20, 12, 64])\n",
            "torch.Size([1, 12, 20, 20])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 20, 12, 64])\n",
            "torch.Size([1, 12, 20, 20])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 20, 12, 64])\n",
            "torch.Size([1, 12, 20, 20])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 20, 12, 64])\n",
            "torch.Size([1, 12, 20, 20])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 21, 12, 64])\n",
            "torch.Size([1, 12, 21, 21])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 21, 12, 64])\n",
            "torch.Size([1, 12, 21, 21])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 21, 12, 64])\n",
            "torch.Size([1, 12, 21, 21])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 21, 12, 64])\n",
            "torch.Size([1, 12, 21, 21])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 21, 12, 64])\n",
            "torch.Size([1, 12, 21, 21])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 21, 12, 64])\n",
            "torch.Size([1, 12, 21, 21])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 21, 12, 64])\n",
            "torch.Size([1, 12, 21, 21])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 21, 12, 64])\n",
            "torch.Size([1, 12, 21, 21])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 21, 12, 64])\n",
            "torch.Size([1, 12, 21, 21])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 21, 12, 64])\n",
            "torch.Size([1, 12, 21, 21])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 21, 12, 64])\n",
            "torch.Size([1, 12, 21, 21])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 21, 12, 64])\n",
            "torch.Size([1, 12, 21, 21])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 22, 12, 64])\n",
            "torch.Size([1, 12, 22, 22])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 22, 12, 64])\n",
            "torch.Size([1, 12, 22, 22])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 22, 12, 64])\n",
            "torch.Size([1, 12, 22, 22])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 22, 12, 64])\n",
            "torch.Size([1, 12, 22, 22])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 22, 12, 64])\n",
            "torch.Size([1, 12, 22, 22])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 22, 12, 64])\n",
            "torch.Size([1, 12, 22, 22])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 22, 12, 64])\n",
            "torch.Size([1, 12, 22, 22])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 22, 12, 64])\n",
            "torch.Size([1, 12, 22, 22])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 22, 12, 64])\n",
            "torch.Size([1, 12, 22, 22])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 22, 12, 64])\n",
            "torch.Size([1, 12, 22, 22])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 22, 12, 64])\n",
            "torch.Size([1, 12, 22, 22])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 22, 12, 64])\n",
            "torch.Size([1, 12, 22, 22])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 23, 12, 64])\n",
            "torch.Size([1, 12, 23, 23])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 23, 12, 64])\n",
            "torch.Size([1, 12, 23, 23])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 23, 12, 64])\n",
            "torch.Size([1, 12, 23, 23])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 23, 12, 64])\n",
            "torch.Size([1, 12, 23, 23])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 23, 12, 64])\n",
            "torch.Size([1, 12, 23, 23])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 23, 12, 64])\n",
            "torch.Size([1, 12, 23, 23])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 23, 12, 64])\n",
            "torch.Size([1, 12, 23, 23])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 23, 12, 64])\n",
            "torch.Size([1, 12, 23, 23])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 23, 12, 64])\n",
            "torch.Size([1, 12, 23, 23])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 23, 12, 64])\n",
            "torch.Size([1, 12, 23, 23])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 23, 12, 64])\n",
            "torch.Size([1, 12, 23, 23])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 23, 12, 64])\n",
            "torch.Size([1, 12, 23, 23])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 24, 12, 64])\n",
            "torch.Size([1, 12, 24, 24])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 24, 12, 64])\n",
            "torch.Size([1, 12, 24, 24])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 24, 12, 64])\n",
            "torch.Size([1, 12, 24, 24])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 24, 12, 64])\n",
            "torch.Size([1, 12, 24, 24])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 24, 12, 64])\n",
            "torch.Size([1, 12, 24, 24])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 24, 12, 64])\n",
            "torch.Size([1, 12, 24, 24])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 24, 12, 64])\n",
            "torch.Size([1, 12, 24, 24])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 24, 12, 64])\n",
            "torch.Size([1, 12, 24, 24])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 24, 12, 64])\n",
            "torch.Size([1, 12, 24, 24])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 24, 12, 64])\n",
            "torch.Size([1, 12, 24, 24])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 24, 12, 64])\n",
            "torch.Size([1, 12, 24, 24])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 24, 12, 64])\n",
            "torch.Size([1, 12, 24, 24])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 25, 12, 64])\n",
            "torch.Size([1, 12, 25, 25])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 25, 12, 64])\n",
            "torch.Size([1, 12, 25, 25])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 25, 12, 64])\n",
            "torch.Size([1, 12, 25, 25])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 25, 12, 64])\n",
            "torch.Size([1, 12, 25, 25])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 25, 12, 64])\n",
            "torch.Size([1, 12, 25, 25])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 25, 12, 64])\n",
            "torch.Size([1, 12, 25, 25])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 25, 12, 64])\n",
            "torch.Size([1, 12, 25, 25])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 25, 12, 64])\n",
            "torch.Size([1, 12, 25, 25])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 25, 12, 64])\n",
            "torch.Size([1, 12, 25, 25])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 25, 12, 64])\n",
            "torch.Size([1, 12, 25, 25])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 25, 12, 64])\n",
            "torch.Size([1, 12, 25, 25])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 25, 12, 64])\n",
            "torch.Size([1, 12, 25, 25])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 26, 12, 64])\n",
            "torch.Size([1, 12, 26, 26])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 26, 12, 64])\n",
            "torch.Size([1, 12, 26, 26])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 26, 12, 64])\n",
            "torch.Size([1, 12, 26, 26])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 26, 12, 64])\n",
            "torch.Size([1, 12, 26, 26])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 26, 12, 64])\n",
            "torch.Size([1, 12, 26, 26])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 26, 12, 64])\n",
            "torch.Size([1, 12, 26, 26])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 26, 12, 64])\n",
            "torch.Size([1, 12, 26, 26])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 26, 12, 64])\n",
            "torch.Size([1, 12, 26, 26])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 26, 12, 64])\n",
            "torch.Size([1, 12, 26, 26])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 26, 12, 64])\n",
            "torch.Size([1, 12, 26, 26])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 26, 12, 64])\n",
            "torch.Size([1, 12, 26, 26])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 26, 12, 64])\n",
            "torch.Size([1, 12, 26, 26])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 27, 12, 64])\n",
            "torch.Size([1, 12, 27, 27])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 27, 12, 64])\n",
            "torch.Size([1, 12, 27, 27])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 27, 12, 64])\n",
            "torch.Size([1, 12, 27, 27])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 27, 12, 64])\n",
            "torch.Size([1, 12, 27, 27])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 27, 12, 64])\n",
            "torch.Size([1, 12, 27, 27])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 27, 12, 64])\n",
            "torch.Size([1, 12, 27, 27])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 27, 12, 64])\n",
            "torch.Size([1, 12, 27, 27])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 27, 12, 64])\n",
            "torch.Size([1, 12, 27, 27])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 27, 12, 64])\n",
            "torch.Size([1, 12, 27, 27])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 27, 12, 64])\n",
            "torch.Size([1, 12, 27, 27])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 27, 12, 64])\n",
            "torch.Size([1, 12, 27, 27])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 27, 12, 64])\n",
            "torch.Size([1, 12, 27, 27])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 28, 12, 64])\n",
            "torch.Size([1, 12, 28, 28])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 28, 12, 64])\n",
            "torch.Size([1, 12, 28, 28])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 28, 12, 64])\n",
            "torch.Size([1, 12, 28, 28])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 28, 12, 64])\n",
            "torch.Size([1, 12, 28, 28])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 28, 12, 64])\n",
            "torch.Size([1, 12, 28, 28])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 28, 12, 64])\n",
            "torch.Size([1, 12, 28, 28])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 28, 12, 64])\n",
            "torch.Size([1, 12, 28, 28])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 28, 12, 64])\n",
            "torch.Size([1, 12, 28, 28])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 28, 12, 64])\n",
            "torch.Size([1, 12, 28, 28])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 28, 12, 64])\n",
            "torch.Size([1, 12, 28, 28])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 28, 12, 64])\n",
            "torch.Size([1, 12, 28, 28])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 28, 12, 64])\n",
            "torch.Size([1, 12, 28, 28])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 29, 12, 64])\n",
            "torch.Size([1, 12, 29, 29])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 29, 12, 64])\n",
            "torch.Size([1, 12, 29, 29])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 29, 12, 64])\n",
            "torch.Size([1, 12, 29, 29])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 29, 12, 64])\n",
            "torch.Size([1, 12, 29, 29])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 29, 12, 64])\n",
            "torch.Size([1, 12, 29, 29])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 29, 12, 64])\n",
            "torch.Size([1, 12, 29, 29])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 29, 12, 64])\n",
            "torch.Size([1, 12, 29, 29])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 29, 12, 64])\n",
            "torch.Size([1, 12, 29, 29])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 29, 12, 64])\n",
            "torch.Size([1, 12, 29, 29])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 29, 12, 64])\n",
            "torch.Size([1, 12, 29, 29])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 29, 12, 64])\n",
            "torch.Size([1, 12, 29, 29])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 29, 12, 64])\n",
            "torch.Size([1, 12, 29, 29])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 30, 12, 64])\n",
            "torch.Size([1, 12, 30, 30])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 30, 12, 64])\n",
            "torch.Size([1, 12, 30, 30])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 30, 12, 64])\n",
            "torch.Size([1, 12, 30, 30])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 30, 12, 64])\n",
            "torch.Size([1, 12, 30, 30])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 30, 12, 64])\n",
            "torch.Size([1, 12, 30, 30])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 30, 12, 64])\n",
            "torch.Size([1, 12, 30, 30])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 30, 12, 64])\n",
            "torch.Size([1, 12, 30, 30])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 30, 12, 64])\n",
            "torch.Size([1, 12, 30, 30])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 30, 12, 64])\n",
            "torch.Size([1, 12, 30, 30])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 30, 12, 64])\n",
            "torch.Size([1, 12, 30, 30])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 30, 12, 64])\n",
            "torch.Size([1, 12, 30, 30])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 30, 12, 64])\n",
            "torch.Size([1, 12, 30, 30])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 31, 12, 64])\n",
            "torch.Size([1, 12, 31, 31])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 31, 12, 64])\n",
            "torch.Size([1, 12, 31, 31])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 31, 12, 64])\n",
            "torch.Size([1, 12, 31, 31])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 31, 12, 64])\n",
            "torch.Size([1, 12, 31, 31])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 31, 12, 64])\n",
            "torch.Size([1, 12, 31, 31])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 31, 12, 64])\n",
            "torch.Size([1, 12, 31, 31])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 31, 12, 64])\n",
            "torch.Size([1, 12, 31, 31])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 31, 12, 64])\n",
            "torch.Size([1, 12, 31, 31])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 31, 12, 64])\n",
            "torch.Size([1, 12, 31, 31])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 31, 12, 64])\n",
            "torch.Size([1, 12, 31, 31])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 31, 12, 64])\n",
            "torch.Size([1, 12, 31, 31])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 31, 12, 64])\n",
            "torch.Size([1, 12, 31, 31])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 32, 12, 64])\n",
            "torch.Size([1, 12, 32, 32])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 32, 12, 64])\n",
            "torch.Size([1, 12, 32, 32])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 32, 12, 64])\n",
            "torch.Size([1, 12, 32, 32])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 32, 12, 64])\n",
            "torch.Size([1, 12, 32, 32])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 32, 12, 64])\n",
            "torch.Size([1, 12, 32, 32])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 32, 12, 64])\n",
            "torch.Size([1, 12, 32, 32])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 32, 12, 64])\n",
            "torch.Size([1, 12, 32, 32])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 32, 12, 64])\n",
            "torch.Size([1, 12, 32, 32])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 32, 12, 64])\n",
            "torch.Size([1, 12, 32, 32])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 32, 12, 64])\n",
            "torch.Size([1, 12, 32, 32])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 32, 12, 64])\n",
            "torch.Size([1, 12, 32, 32])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 32, 12, 64])\n",
            "torch.Size([1, 12, 32, 32])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 33, 12, 64])\n",
            "torch.Size([1, 12, 33, 33])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 33, 12, 64])\n",
            "torch.Size([1, 12, 33, 33])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 33, 12, 64])\n",
            "torch.Size([1, 12, 33, 33])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 33, 12, 64])\n",
            "torch.Size([1, 12, 33, 33])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 33, 12, 64])\n",
            "torch.Size([1, 12, 33, 33])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 33, 12, 64])\n",
            "torch.Size([1, 12, 33, 33])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 33, 12, 64])\n",
            "torch.Size([1, 12, 33, 33])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 33, 12, 64])\n",
            "torch.Size([1, 12, 33, 33])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 33, 12, 64])\n",
            "torch.Size([1, 12, 33, 33])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 33, 12, 64])\n",
            "torch.Size([1, 12, 33, 33])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 33, 12, 64])\n",
            "torch.Size([1, 12, 33, 33])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 33, 12, 64])\n",
            "torch.Size([1, 12, 33, 33])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 34, 12, 64])\n",
            "torch.Size([1, 12, 34, 34])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 34, 12, 64])\n",
            "torch.Size([1, 12, 34, 34])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 34, 12, 64])\n",
            "torch.Size([1, 12, 34, 34])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 34, 12, 64])\n",
            "torch.Size([1, 12, 34, 34])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 34, 12, 64])\n",
            "torch.Size([1, 12, 34, 34])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 34, 12, 64])\n",
            "torch.Size([1, 12, 34, 34])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 34, 12, 64])\n",
            "torch.Size([1, 12, 34, 34])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 34, 12, 64])\n",
            "torch.Size([1, 12, 34, 34])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 34, 12, 64])\n",
            "torch.Size([1, 12, 34, 34])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 34, 12, 64])\n",
            "torch.Size([1, 12, 34, 34])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 34, 12, 64])\n",
            "torch.Size([1, 12, 34, 34])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 34, 12, 64])\n",
            "torch.Size([1, 12, 34, 34])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 35, 12, 64])\n",
            "torch.Size([1, 12, 35, 35])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 35, 12, 64])\n",
            "torch.Size([1, 12, 35, 35])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 35, 12, 64])\n",
            "torch.Size([1, 12, 35, 35])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 35, 12, 64])\n",
            "torch.Size([1, 12, 35, 35])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 35, 12, 64])\n",
            "torch.Size([1, 12, 35, 35])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 35, 12, 64])\n",
            "torch.Size([1, 12, 35, 35])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 35, 12, 64])\n",
            "torch.Size([1, 12, 35, 35])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 35, 12, 64])\n",
            "torch.Size([1, 12, 35, 35])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 35, 12, 64])\n",
            "torch.Size([1, 12, 35, 35])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 35, 12, 64])\n",
            "torch.Size([1, 12, 35, 35])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 35, 12, 64])\n",
            "torch.Size([1, 12, 35, 35])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 35, 12, 64])\n",
            "torch.Size([1, 12, 35, 35])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 36, 12, 64])\n",
            "torch.Size([1, 12, 36, 36])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 36, 12, 64])\n",
            "torch.Size([1, 12, 36, 36])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 36, 12, 64])\n",
            "torch.Size([1, 12, 36, 36])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 36, 12, 64])\n",
            "torch.Size([1, 12, 36, 36])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 36, 12, 64])\n",
            "torch.Size([1, 12, 36, 36])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 36, 12, 64])\n",
            "torch.Size([1, 12, 36, 36])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 36, 12, 64])\n",
            "torch.Size([1, 12, 36, 36])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 36, 12, 64])\n",
            "torch.Size([1, 12, 36, 36])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 36, 12, 64])\n",
            "torch.Size([1, 12, 36, 36])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 36, 12, 64])\n",
            "torch.Size([1, 12, 36, 36])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 36, 12, 64])\n",
            "torch.Size([1, 12, 36, 36])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 36, 12, 64])\n",
            "torch.Size([1, 12, 36, 36])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 37, 12, 64])\n",
            "torch.Size([1, 12, 37, 37])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 37, 12, 64])\n",
            "torch.Size([1, 12, 37, 37])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 37, 12, 64])\n",
            "torch.Size([1, 12, 37, 37])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 37, 12, 64])\n",
            "torch.Size([1, 12, 37, 37])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 37, 12, 64])\n",
            "torch.Size([1, 12, 37, 37])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 37, 12, 64])\n",
            "torch.Size([1, 12, 37, 37])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 37, 12, 64])\n",
            "torch.Size([1, 12, 37, 37])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 37, 12, 64])\n",
            "torch.Size([1, 12, 37, 37])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 37, 12, 64])\n",
            "torch.Size([1, 12, 37, 37])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 37, 12, 64])\n",
            "torch.Size([1, 12, 37, 37])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 37, 12, 64])\n",
            "torch.Size([1, 12, 37, 37])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 37, 12, 64])\n",
            "torch.Size([1, 12, 37, 37])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 38, 12, 64])\n",
            "torch.Size([1, 12, 38, 38])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 38, 12, 64])\n",
            "torch.Size([1, 12, 38, 38])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 38, 12, 64])\n",
            "torch.Size([1, 12, 38, 38])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 38, 12, 64])\n",
            "torch.Size([1, 12, 38, 38])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 38, 12, 64])\n",
            "torch.Size([1, 12, 38, 38])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 38, 12, 64])\n",
            "torch.Size([1, 12, 38, 38])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 38, 12, 64])\n",
            "torch.Size([1, 12, 38, 38])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 38, 12, 64])\n",
            "torch.Size([1, 12, 38, 38])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 38, 12, 64])\n",
            "torch.Size([1, 12, 38, 38])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 38, 12, 64])\n",
            "torch.Size([1, 12, 38, 38])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 38, 12, 64])\n",
            "torch.Size([1, 12, 38, 38])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 38, 12, 64])\n",
            "torch.Size([1, 12, 38, 38])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 39, 12, 64])\n",
            "torch.Size([1, 12, 39, 39])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 39, 12, 64])\n",
            "torch.Size([1, 12, 39, 39])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 39, 12, 64])\n",
            "torch.Size([1, 12, 39, 39])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 39, 12, 64])\n",
            "torch.Size([1, 12, 39, 39])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 39, 12, 64])\n",
            "torch.Size([1, 12, 39, 39])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 39, 12, 64])\n",
            "torch.Size([1, 12, 39, 39])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 39, 12, 64])\n",
            "torch.Size([1, 12, 39, 39])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 39, 12, 64])\n",
            "torch.Size([1, 12, 39, 39])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 39, 12, 64])\n",
            "torch.Size([1, 12, 39, 39])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 39, 12, 64])\n",
            "torch.Size([1, 12, 39, 39])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 39, 12, 64])\n",
            "torch.Size([1, 12, 39, 39])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 39, 12, 64])\n",
            "torch.Size([1, 12, 39, 39])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 40, 12, 64])\n",
            "torch.Size([1, 12, 40, 40])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 40, 12, 64])\n",
            "torch.Size([1, 12, 40, 40])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 40, 12, 64])\n",
            "torch.Size([1, 12, 40, 40])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 40, 12, 64])\n",
            "torch.Size([1, 12, 40, 40])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 40, 12, 64])\n",
            "torch.Size([1, 12, 40, 40])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 40, 12, 64])\n",
            "torch.Size([1, 12, 40, 40])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 40, 12, 64])\n",
            "torch.Size([1, 12, 40, 40])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 40, 12, 64])\n",
            "torch.Size([1, 12, 40, 40])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 40, 12, 64])\n",
            "torch.Size([1, 12, 40, 40])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 40, 12, 64])\n",
            "torch.Size([1, 12, 40, 40])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 40, 12, 64])\n",
            "torch.Size([1, 12, 40, 40])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 40, 12, 64])\n",
            "torch.Size([1, 12, 40, 40])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 41, 12, 64])\n",
            "torch.Size([1, 12, 41, 41])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 41, 12, 64])\n",
            "torch.Size([1, 12, 41, 41])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 41, 12, 64])\n",
            "torch.Size([1, 12, 41, 41])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 41, 12, 64])\n",
            "torch.Size([1, 12, 41, 41])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 41, 12, 64])\n",
            "torch.Size([1, 12, 41, 41])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 41, 12, 64])\n",
            "torch.Size([1, 12, 41, 41])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 41, 12, 64])\n",
            "torch.Size([1, 12, 41, 41])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 41, 12, 64])\n",
            "torch.Size([1, 12, 41, 41])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 41, 12, 64])\n",
            "torch.Size([1, 12, 41, 41])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 41, 12, 64])\n",
            "torch.Size([1, 12, 41, 41])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 41, 12, 64])\n",
            "torch.Size([1, 12, 41, 41])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 41, 12, 64])\n",
            "torch.Size([1, 12, 41, 41])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 42, 12, 64])\n",
            "torch.Size([1, 12, 42, 42])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 42, 12, 64])\n",
            "torch.Size([1, 12, 42, 42])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 42, 12, 64])\n",
            "torch.Size([1, 12, 42, 42])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 42, 12, 64])\n",
            "torch.Size([1, 12, 42, 42])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 42, 12, 64])\n",
            "torch.Size([1, 12, 42, 42])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 42, 12, 64])\n",
            "torch.Size([1, 12, 42, 42])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 42, 12, 64])\n",
            "torch.Size([1, 12, 42, 42])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 42, 12, 64])\n",
            "torch.Size([1, 12, 42, 42])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 42, 12, 64])\n",
            "torch.Size([1, 12, 42, 42])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 42, 12, 64])\n",
            "torch.Size([1, 12, 42, 42])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 42, 12, 64])\n",
            "torch.Size([1, 12, 42, 42])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 42, 12, 64])\n",
            "torch.Size([1, 12, 42, 42])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 43, 12, 64])\n",
            "torch.Size([1, 12, 43, 43])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 43, 12, 64])\n",
            "torch.Size([1, 12, 43, 43])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 43, 12, 64])\n",
            "torch.Size([1, 12, 43, 43])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 43, 12, 64])\n",
            "torch.Size([1, 12, 43, 43])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 43, 12, 64])\n",
            "torch.Size([1, 12, 43, 43])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 43, 12, 64])\n",
            "torch.Size([1, 12, 43, 43])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 43, 12, 64])\n",
            "torch.Size([1, 12, 43, 43])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 43, 12, 64])\n",
            "torch.Size([1, 12, 43, 43])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 43, 12, 64])\n",
            "torch.Size([1, 12, 43, 43])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 43, 12, 64])\n",
            "torch.Size([1, 12, 43, 43])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 43, 12, 64])\n",
            "torch.Size([1, 12, 43, 43])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 43, 12, 64])\n",
            "torch.Size([1, 12, 43, 43])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 44, 12, 64])\n",
            "torch.Size([1, 12, 44, 44])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 44, 12, 64])\n",
            "torch.Size([1, 12, 44, 44])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 44, 12, 64])\n",
            "torch.Size([1, 12, 44, 44])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 44, 12, 64])\n",
            "torch.Size([1, 12, 44, 44])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 44, 12, 64])\n",
            "torch.Size([1, 12, 44, 44])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 44, 12, 64])\n",
            "torch.Size([1, 12, 44, 44])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 44, 12, 64])\n",
            "torch.Size([1, 12, 44, 44])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 44, 12, 64])\n",
            "torch.Size([1, 12, 44, 44])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 44, 12, 64])\n",
            "torch.Size([1, 12, 44, 44])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 44, 12, 64])\n",
            "torch.Size([1, 12, 44, 44])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 44, 12, 64])\n",
            "torch.Size([1, 12, 44, 44])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 44, 12, 64])\n",
            "torch.Size([1, 12, 44, 44])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 45, 12, 64])\n",
            "torch.Size([1, 12, 45, 45])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 45, 12, 64])\n",
            "torch.Size([1, 12, 45, 45])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 45, 12, 64])\n",
            "torch.Size([1, 12, 45, 45])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 45, 12, 64])\n",
            "torch.Size([1, 12, 45, 45])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 45, 12, 64])\n",
            "torch.Size([1, 12, 45, 45])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 45, 12, 64])\n",
            "torch.Size([1, 12, 45, 45])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 45, 12, 64])\n",
            "torch.Size([1, 12, 45, 45])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 45, 12, 64])\n",
            "torch.Size([1, 12, 45, 45])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 45, 12, 64])\n",
            "torch.Size([1, 12, 45, 45])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 45, 12, 64])\n",
            "torch.Size([1, 12, 45, 45])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 45, 12, 64])\n",
            "torch.Size([1, 12, 45, 45])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 45, 12, 64])\n",
            "torch.Size([1, 12, 45, 45])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 46, 12, 64])\n",
            "torch.Size([1, 12, 46, 46])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 46, 12, 64])\n",
            "torch.Size([1, 12, 46, 46])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 46, 12, 64])\n",
            "torch.Size([1, 12, 46, 46])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 46, 12, 64])\n",
            "torch.Size([1, 12, 46, 46])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 46, 12, 64])\n",
            "torch.Size([1, 12, 46, 46])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 46, 12, 64])\n",
            "torch.Size([1, 12, 46, 46])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 46, 12, 64])\n",
            "torch.Size([1, 12, 46, 46])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 46, 12, 64])\n",
            "torch.Size([1, 12, 46, 46])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 46, 12, 64])\n",
            "torch.Size([1, 12, 46, 46])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 46, 12, 64])\n",
            "torch.Size([1, 12, 46, 46])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 46, 12, 64])\n",
            "torch.Size([1, 12, 46, 46])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 46, 12, 64])\n",
            "torch.Size([1, 12, 46, 46])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 47, 12, 64])\n",
            "torch.Size([1, 12, 47, 47])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 47, 12, 64])\n",
            "torch.Size([1, 12, 47, 47])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 47, 12, 64])\n",
            "torch.Size([1, 12, 47, 47])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 47, 12, 64])\n",
            "torch.Size([1, 12, 47, 47])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 47, 12, 64])\n",
            "torch.Size([1, 12, 47, 47])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 47, 12, 64])\n",
            "torch.Size([1, 12, 47, 47])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 47, 12, 64])\n",
            "torch.Size([1, 12, 47, 47])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 47, 12, 64])\n",
            "torch.Size([1, 12, 47, 47])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 47, 12, 64])\n",
            "torch.Size([1, 12, 47, 47])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 47, 12, 64])\n",
            "torch.Size([1, 12, 47, 47])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 47, 12, 64])\n",
            "torch.Size([1, 12, 47, 47])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 47, 12, 64])\n",
            "torch.Size([1, 12, 47, 47])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 48, 12, 64])\n",
            "torch.Size([1, 12, 48, 48])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 48, 12, 64])\n",
            "torch.Size([1, 12, 48, 48])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 48, 12, 64])\n",
            "torch.Size([1, 12, 48, 48])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 48, 12, 64])\n",
            "torch.Size([1, 12, 48, 48])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 48, 12, 64])\n",
            "torch.Size([1, 12, 48, 48])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 48, 12, 64])\n",
            "torch.Size([1, 12, 48, 48])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 48, 12, 64])\n",
            "torch.Size([1, 12, 48, 48])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 48, 12, 64])\n",
            "torch.Size([1, 12, 48, 48])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 48, 12, 64])\n",
            "torch.Size([1, 12, 48, 48])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 48, 12, 64])\n",
            "torch.Size([1, 12, 48, 48])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 48, 12, 64])\n",
            "torch.Size([1, 12, 48, 48])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 48, 12, 64])\n",
            "torch.Size([1, 12, 48, 48])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 49, 12, 64])\n",
            "torch.Size([1, 12, 49, 49])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 49, 12, 64])\n",
            "torch.Size([1, 12, 49, 49])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 49, 12, 64])\n",
            "torch.Size([1, 12, 49, 49])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 49, 12, 64])\n",
            "torch.Size([1, 12, 49, 49])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 49, 12, 64])\n",
            "torch.Size([1, 12, 49, 49])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 49, 12, 64])\n",
            "torch.Size([1, 12, 49, 49])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 49, 12, 64])\n",
            "torch.Size([1, 12, 49, 49])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 49, 12, 64])\n",
            "torch.Size([1, 12, 49, 49])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 49, 12, 64])\n",
            "torch.Size([1, 12, 49, 49])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 49, 12, 64])\n",
            "torch.Size([1, 12, 49, 49])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 49, 12, 64])\n",
            "torch.Size([1, 12, 49, 49])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 49, 12, 64])\n",
            "torch.Size([1, 12, 49, 49])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 50, 12, 64])\n",
            "torch.Size([1, 12, 50, 50])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 50, 12, 64])\n",
            "torch.Size([1, 12, 50, 50])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 50, 12, 64])\n",
            "torch.Size([1, 12, 50, 50])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 50, 12, 64])\n",
            "torch.Size([1, 12, 50, 50])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 50, 12, 64])\n",
            "torch.Size([1, 12, 50, 50])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 50, 12, 64])\n",
            "torch.Size([1, 12, 50, 50])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 50, 12, 64])\n",
            "torch.Size([1, 12, 50, 50])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 50, 12, 64])\n",
            "torch.Size([1, 12, 50, 50])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 50, 12, 64])\n",
            "torch.Size([1, 12, 50, 50])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 50, 12, 64])\n",
            "torch.Size([1, 12, 50, 50])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 50, 12, 64])\n",
            "torch.Size([1, 12, 50, 50])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 50, 12, 64])\n",
            "torch.Size([1, 12, 50, 50])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 51, 12, 64])\n",
            "torch.Size([1, 12, 51, 51])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 51, 12, 64])\n",
            "torch.Size([1, 12, 51, 51])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 51, 12, 64])\n",
            "torch.Size([1, 12, 51, 51])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 51, 12, 64])\n",
            "torch.Size([1, 12, 51, 51])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 51, 12, 64])\n",
            "torch.Size([1, 12, 51, 51])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 51, 12, 64])\n",
            "torch.Size([1, 12, 51, 51])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 51, 12, 64])\n",
            "torch.Size([1, 12, 51, 51])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 51, 12, 64])\n",
            "torch.Size([1, 12, 51, 51])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 51, 12, 64])\n",
            "torch.Size([1, 12, 51, 51])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 51, 12, 64])\n",
            "torch.Size([1, 12, 51, 51])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 51, 12, 64])\n",
            "torch.Size([1, 12, 51, 51])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 51, 12, 64])\n",
            "torch.Size([1, 12, 51, 51])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 52, 12, 64])\n",
            "torch.Size([1, 12, 52, 52])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 52, 12, 64])\n",
            "torch.Size([1, 12, 52, 52])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 52, 12, 64])\n",
            "torch.Size([1, 12, 52, 52])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 52, 12, 64])\n",
            "torch.Size([1, 12, 52, 52])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 52, 12, 64])\n",
            "torch.Size([1, 12, 52, 52])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 52, 12, 64])\n",
            "torch.Size([1, 12, 52, 52])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 52, 12, 64])\n",
            "torch.Size([1, 12, 52, 52])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 52, 12, 64])\n",
            "torch.Size([1, 12, 52, 52])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 52, 12, 64])\n",
            "torch.Size([1, 12, 52, 52])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 52, 12, 64])\n",
            "torch.Size([1, 12, 52, 52])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 52, 12, 64])\n",
            "torch.Size([1, 12, 52, 52])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 52, 12, 64])\n",
            "torch.Size([1, 12, 52, 52])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 53, 12, 64])\n",
            "torch.Size([1, 12, 53, 53])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 53, 12, 64])\n",
            "torch.Size([1, 12, 53, 53])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 53, 12, 64])\n",
            "torch.Size([1, 12, 53, 53])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 53, 12, 64])\n",
            "torch.Size([1, 12, 53, 53])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 53, 12, 64])\n",
            "torch.Size([1, 12, 53, 53])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 53, 12, 64])\n",
            "torch.Size([1, 12, 53, 53])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 53, 12, 64])\n",
            "torch.Size([1, 12, 53, 53])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 53, 12, 64])\n",
            "torch.Size([1, 12, 53, 53])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 53, 12, 64])\n",
            "torch.Size([1, 12, 53, 53])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 53, 12, 64])\n",
            "torch.Size([1, 12, 53, 53])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 53, 12, 64])\n",
            "torch.Size([1, 12, 53, 53])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 53, 12, 64])\n",
            "torch.Size([1, 12, 53, 53])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 54, 12, 64])\n",
            "torch.Size([1, 12, 54, 54])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 54, 12, 64])\n",
            "torch.Size([1, 12, 54, 54])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 54, 12, 64])\n",
            "torch.Size([1, 12, 54, 54])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 54, 12, 64])\n",
            "torch.Size([1, 12, 54, 54])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 54, 12, 64])\n",
            "torch.Size([1, 12, 54, 54])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 54, 12, 64])\n",
            "torch.Size([1, 12, 54, 54])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 54, 12, 64])\n",
            "torch.Size([1, 12, 54, 54])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 54, 12, 64])\n",
            "torch.Size([1, 12, 54, 54])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 54, 12, 64])\n",
            "torch.Size([1, 12, 54, 54])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 54, 12, 64])\n",
            "torch.Size([1, 12, 54, 54])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 54, 12, 64])\n",
            "torch.Size([1, 12, 54, 54])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 54, 12, 64])\n",
            "torch.Size([1, 12, 54, 54])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 55, 12, 64])\n",
            "torch.Size([1, 12, 55, 55])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 55, 12, 64])\n",
            "torch.Size([1, 12, 55, 55])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 55, 12, 64])\n",
            "torch.Size([1, 12, 55, 55])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 55, 12, 64])\n",
            "torch.Size([1, 12, 55, 55])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 55, 12, 64])\n",
            "torch.Size([1, 12, 55, 55])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 55, 12, 64])\n",
            "torch.Size([1, 12, 55, 55])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 55, 12, 64])\n",
            "torch.Size([1, 12, 55, 55])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 55, 12, 64])\n",
            "torch.Size([1, 12, 55, 55])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 55, 12, 64])\n",
            "torch.Size([1, 12, 55, 55])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 55, 12, 64])\n",
            "torch.Size([1, 12, 55, 55])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 55, 12, 64])\n",
            "torch.Size([1, 12, 55, 55])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 55, 12, 64])\n",
            "torch.Size([1, 12, 55, 55])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 56, 12, 64])\n",
            "torch.Size([1, 12, 56, 56])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 56, 12, 64])\n",
            "torch.Size([1, 12, 56, 56])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 56, 12, 64])\n",
            "torch.Size([1, 12, 56, 56])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 56, 12, 64])\n",
            "torch.Size([1, 12, 56, 56])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 56, 12, 64])\n",
            "torch.Size([1, 12, 56, 56])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 56, 12, 64])\n",
            "torch.Size([1, 12, 56, 56])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 56, 12, 64])\n",
            "torch.Size([1, 12, 56, 56])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 56, 12, 64])\n",
            "torch.Size([1, 12, 56, 56])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 56, 12, 64])\n",
            "torch.Size([1, 12, 56, 56])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 56, 12, 64])\n",
            "torch.Size([1, 12, 56, 56])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 56, 12, 64])\n",
            "torch.Size([1, 12, 56, 56])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 56, 12, 64])\n",
            "torch.Size([1, 12, 56, 56])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 57, 12, 64])\n",
            "torch.Size([1, 12, 57, 57])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 57, 12, 64])\n",
            "torch.Size([1, 12, 57, 57])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 57, 12, 64])\n",
            "torch.Size([1, 12, 57, 57])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 57, 12, 64])\n",
            "torch.Size([1, 12, 57, 57])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 57, 12, 64])\n",
            "torch.Size([1, 12, 57, 57])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 57, 12, 64])\n",
            "torch.Size([1, 12, 57, 57])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 57, 12, 64])\n",
            "torch.Size([1, 12, 57, 57])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 57, 12, 64])\n",
            "torch.Size([1, 12, 57, 57])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 57, 12, 64])\n",
            "torch.Size([1, 12, 57, 57])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 57, 12, 64])\n",
            "torch.Size([1, 12, 57, 57])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 57, 12, 64])\n",
            "torch.Size([1, 12, 57, 57])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 57, 12, 64])\n",
            "torch.Size([1, 12, 57, 57])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 58, 12, 64])\n",
            "torch.Size([1, 12, 58, 58])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 58, 12, 64])\n",
            "torch.Size([1, 12, 58, 58])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 58, 12, 64])\n",
            "torch.Size([1, 12, 58, 58])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 58, 12, 64])\n",
            "torch.Size([1, 12, 58, 58])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 58, 12, 64])\n",
            "torch.Size([1, 12, 58, 58])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 58, 12, 64])\n",
            "torch.Size([1, 12, 58, 58])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 58, 12, 64])\n",
            "torch.Size([1, 12, 58, 58])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 58, 12, 64])\n",
            "torch.Size([1, 12, 58, 58])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 58, 12, 64])\n",
            "torch.Size([1, 12, 58, 58])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 58, 12, 64])\n",
            "torch.Size([1, 12, 58, 58])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 58, 12, 64])\n",
            "torch.Size([1, 12, 58, 58])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 58, 12, 64])\n",
            "torch.Size([1, 12, 58, 58])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 59, 12, 64])\n",
            "torch.Size([1, 12, 59, 59])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 59, 12, 64])\n",
            "torch.Size([1, 12, 59, 59])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 59, 12, 64])\n",
            "torch.Size([1, 12, 59, 59])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 59, 12, 64])\n",
            "torch.Size([1, 12, 59, 59])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 59, 12, 64])\n",
            "torch.Size([1, 12, 59, 59])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 59, 12, 64])\n",
            "torch.Size([1, 12, 59, 59])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 59, 12, 64])\n",
            "torch.Size([1, 12, 59, 59])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 59, 12, 64])\n",
            "torch.Size([1, 12, 59, 59])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 59, 12, 64])\n",
            "torch.Size([1, 12, 59, 59])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 59, 12, 64])\n",
            "torch.Size([1, 12, 59, 59])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 59, 12, 64])\n",
            "torch.Size([1, 12, 59, 59])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 59, 12, 64])\n",
            "torch.Size([1, 12, 59, 59])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 60, 12, 64])\n",
            "torch.Size([1, 12, 60, 60])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 60, 12, 64])\n",
            "torch.Size([1, 12, 60, 60])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 60, 12, 64])\n",
            "torch.Size([1, 12, 60, 60])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 60, 12, 64])\n",
            "torch.Size([1, 12, 60, 60])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 60, 12, 64])\n",
            "torch.Size([1, 12, 60, 60])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 60, 12, 64])\n",
            "torch.Size([1, 12, 60, 60])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 60, 12, 64])\n",
            "torch.Size([1, 12, 60, 60])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 60, 12, 64])\n",
            "torch.Size([1, 12, 60, 60])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 60, 12, 64])\n",
            "torch.Size([1, 12, 60, 60])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 60, 12, 64])\n",
            "torch.Size([1, 12, 60, 60])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 60, 12, 64])\n",
            "torch.Size([1, 12, 60, 60])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 60, 12, 64])\n",
            "torch.Size([1, 12, 60, 60])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 61, 12, 64])\n",
            "torch.Size([1, 12, 61, 61])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 61, 12, 64])\n",
            "torch.Size([1, 12, 61, 61])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 61, 12, 64])\n",
            "torch.Size([1, 12, 61, 61])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 61, 12, 64])\n",
            "torch.Size([1, 12, 61, 61])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 61, 12, 64])\n",
            "torch.Size([1, 12, 61, 61])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 61, 12, 64])\n",
            "torch.Size([1, 12, 61, 61])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 61, 12, 64])\n",
            "torch.Size([1, 12, 61, 61])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 61, 12, 64])\n",
            "torch.Size([1, 12, 61, 61])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 61, 12, 64])\n",
            "torch.Size([1, 12, 61, 61])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 61, 12, 64])\n",
            "torch.Size([1, 12, 61, 61])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 61, 12, 64])\n",
            "torch.Size([1, 12, 61, 61])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 61, 12, 64])\n",
            "torch.Size([1, 12, 61, 61])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 62, 12, 64])\n",
            "torch.Size([1, 12, 62, 62])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 62, 12, 64])\n",
            "torch.Size([1, 12, 62, 62])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 62, 12, 64])\n",
            "torch.Size([1, 12, 62, 62])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 62, 12, 64])\n",
            "torch.Size([1, 12, 62, 62])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 62, 12, 64])\n",
            "torch.Size([1, 12, 62, 62])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 62, 12, 64])\n",
            "torch.Size([1, 12, 62, 62])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 62, 12, 64])\n",
            "torch.Size([1, 12, 62, 62])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 62, 12, 64])\n",
            "torch.Size([1, 12, 62, 62])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 62, 12, 64])\n",
            "torch.Size([1, 12, 62, 62])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 62, 12, 64])\n",
            "torch.Size([1, 12, 62, 62])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 62, 12, 64])\n",
            "torch.Size([1, 12, 62, 62])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 62, 12, 64])\n",
            "torch.Size([1, 12, 62, 62])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 63, 12, 64])\n",
            "torch.Size([1, 12, 63, 63])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 63, 12, 64])\n",
            "torch.Size([1, 12, 63, 63])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 63, 12, 64])\n",
            "torch.Size([1, 12, 63, 63])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 63, 12, 64])\n",
            "torch.Size([1, 12, 63, 63])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 63, 12, 64])\n",
            "torch.Size([1, 12, 63, 63])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 63, 12, 64])\n",
            "torch.Size([1, 12, 63, 63])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 63, 12, 64])\n",
            "torch.Size([1, 12, 63, 63])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 63, 12, 64])\n",
            "torch.Size([1, 12, 63, 63])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 63, 12, 64])\n",
            "torch.Size([1, 12, 63, 63])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 63, 12, 64])\n",
            "torch.Size([1, 12, 63, 63])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 63, 12, 64])\n",
            "torch.Size([1, 12, 63, 63])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 63, 12, 64])\n",
            "torch.Size([1, 12, 63, 63])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 64, 12, 64])\n",
            "torch.Size([1, 12, 64, 64])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 64, 12, 64])\n",
            "torch.Size([1, 12, 64, 64])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 64, 12, 64])\n",
            "torch.Size([1, 12, 64, 64])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 64, 12, 64])\n",
            "torch.Size([1, 12, 64, 64])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 64, 12, 64])\n",
            "torch.Size([1, 12, 64, 64])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 64, 12, 64])\n",
            "torch.Size([1, 12, 64, 64])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 64, 12, 64])\n",
            "torch.Size([1, 12, 64, 64])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 64, 12, 64])\n",
            "torch.Size([1, 12, 64, 64])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 64, 12, 64])\n",
            "torch.Size([1, 12, 64, 64])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 64, 12, 64])\n",
            "torch.Size([1, 12, 64, 64])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 64, 12, 64])\n",
            "torch.Size([1, 12, 64, 64])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 64, 12, 64])\n",
            "torch.Size([1, 12, 64, 64])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 65, 12, 64])\n",
            "torch.Size([1, 12, 65, 65])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 65, 12, 64])\n",
            "torch.Size([1, 12, 65, 65])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 65, 12, 64])\n",
            "torch.Size([1, 12, 65, 65])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 65, 12, 64])\n",
            "torch.Size([1, 12, 65, 65])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 65, 12, 64])\n",
            "torch.Size([1, 12, 65, 65])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 65, 12, 64])\n",
            "torch.Size([1, 12, 65, 65])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 65, 12, 64])\n",
            "torch.Size([1, 12, 65, 65])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 65, 12, 64])\n",
            "torch.Size([1, 12, 65, 65])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 65, 12, 64])\n",
            "torch.Size([1, 12, 65, 65])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 65, 12, 64])\n",
            "torch.Size([1, 12, 65, 65])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 65, 12, 64])\n",
            "torch.Size([1, 12, 65, 65])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 65, 12, 64])\n",
            "torch.Size([1, 12, 65, 65])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 66, 12, 64])\n",
            "torch.Size([1, 12, 66, 66])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 66, 12, 64])\n",
            "torch.Size([1, 12, 66, 66])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 66, 12, 64])\n",
            "torch.Size([1, 12, 66, 66])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 66, 12, 64])\n",
            "torch.Size([1, 12, 66, 66])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 66, 12, 64])\n",
            "torch.Size([1, 12, 66, 66])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 66, 12, 64])\n",
            "torch.Size([1, 12, 66, 66])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 66, 12, 64])\n",
            "torch.Size([1, 12, 66, 66])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 66, 12, 64])\n",
            "torch.Size([1, 12, 66, 66])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 66, 12, 64])\n",
            "torch.Size([1, 12, 66, 66])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 66, 12, 64])\n",
            "torch.Size([1, 12, 66, 66])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 66, 12, 64])\n",
            "torch.Size([1, 12, 66, 66])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 66, 12, 64])\n",
            "torch.Size([1, 12, 66, 66])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 67, 12, 64])\n",
            "torch.Size([1, 12, 67, 67])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 67, 12, 64])\n",
            "torch.Size([1, 12, 67, 67])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 67, 12, 64])\n",
            "torch.Size([1, 12, 67, 67])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 67, 12, 64])\n",
            "torch.Size([1, 12, 67, 67])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 67, 12, 64])\n",
            "torch.Size([1, 12, 67, 67])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 67, 12, 64])\n",
            "torch.Size([1, 12, 67, 67])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 67, 12, 64])\n",
            "torch.Size([1, 12, 67, 67])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 67, 12, 64])\n",
            "torch.Size([1, 12, 67, 67])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 67, 12, 64])\n",
            "torch.Size([1, 12, 67, 67])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 67, 12, 64])\n",
            "torch.Size([1, 12, 67, 67])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 67, 12, 64])\n",
            "torch.Size([1, 12, 67, 67])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 67, 12, 64])\n",
            "torch.Size([1, 12, 67, 67])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 68, 12, 64])\n",
            "torch.Size([1, 12, 68, 68])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 68, 12, 64])\n",
            "torch.Size([1, 12, 68, 68])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 68, 12, 64])\n",
            "torch.Size([1, 12, 68, 68])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 68, 12, 64])\n",
            "torch.Size([1, 12, 68, 68])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 68, 12, 64])\n",
            "torch.Size([1, 12, 68, 68])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 68, 12, 64])\n",
            "torch.Size([1, 12, 68, 68])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 68, 12, 64])\n",
            "torch.Size([1, 12, 68, 68])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 68, 12, 64])\n",
            "torch.Size([1, 12, 68, 68])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 68, 12, 64])\n",
            "torch.Size([1, 12, 68, 68])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 68, 12, 64])\n",
            "torch.Size([1, 12, 68, 68])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 68, 12, 64])\n",
            "torch.Size([1, 12, 68, 68])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 68, 12, 64])\n",
            "torch.Size([1, 12, 68, 68])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 69, 12, 64])\n",
            "torch.Size([1, 12, 69, 69])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 69, 12, 64])\n",
            "torch.Size([1, 12, 69, 69])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 69, 12, 64])\n",
            "torch.Size([1, 12, 69, 69])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 69, 12, 64])\n",
            "torch.Size([1, 12, 69, 69])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 69, 12, 64])\n",
            "torch.Size([1, 12, 69, 69])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 69, 12, 64])\n",
            "torch.Size([1, 12, 69, 69])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 69, 12, 64])\n",
            "torch.Size([1, 12, 69, 69])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 69, 12, 64])\n",
            "torch.Size([1, 12, 69, 69])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 69, 12, 64])\n",
            "torch.Size([1, 12, 69, 69])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 69, 12, 64])\n",
            "torch.Size([1, 12, 69, 69])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 69, 12, 64])\n",
            "torch.Size([1, 12, 69, 69])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 69, 12, 64])\n",
            "torch.Size([1, 12, 69, 69])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 70, 12, 64])\n",
            "torch.Size([1, 12, 70, 70])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 70, 12, 64])\n",
            "torch.Size([1, 12, 70, 70])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 70, 12, 64])\n",
            "torch.Size([1, 12, 70, 70])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 70, 12, 64])\n",
            "torch.Size([1, 12, 70, 70])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 70, 12, 64])\n",
            "torch.Size([1, 12, 70, 70])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 70, 12, 64])\n",
            "torch.Size([1, 12, 70, 70])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 70, 12, 64])\n",
            "torch.Size([1, 12, 70, 70])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 70, 12, 64])\n",
            "torch.Size([1, 12, 70, 70])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 70, 12, 64])\n",
            "torch.Size([1, 12, 70, 70])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 70, 12, 64])\n",
            "torch.Size([1, 12, 70, 70])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 70, 12, 64])\n",
            "torch.Size([1, 12, 70, 70])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 70, 12, 64])\n",
            "torch.Size([1, 12, 70, 70])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 71, 12, 64])\n",
            "torch.Size([1, 12, 71, 71])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 71, 12, 64])\n",
            "torch.Size([1, 12, 71, 71])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 71, 12, 64])\n",
            "torch.Size([1, 12, 71, 71])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 71, 12, 64])\n",
            "torch.Size([1, 12, 71, 71])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 71, 12, 64])\n",
            "torch.Size([1, 12, 71, 71])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 71, 12, 64])\n",
            "torch.Size([1, 12, 71, 71])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 71, 12, 64])\n",
            "torch.Size([1, 12, 71, 71])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 71, 12, 64])\n",
            "torch.Size([1, 12, 71, 71])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 71, 12, 64])\n",
            "torch.Size([1, 12, 71, 71])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 71, 12, 64])\n",
            "torch.Size([1, 12, 71, 71])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 71, 12, 64])\n",
            "torch.Size([1, 12, 71, 71])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 71, 12, 64])\n",
            "torch.Size([1, 12, 71, 71])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 72, 12, 64])\n",
            "torch.Size([1, 12, 72, 72])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 72, 12, 64])\n",
            "torch.Size([1, 12, 72, 72])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 72, 12, 64])\n",
            "torch.Size([1, 12, 72, 72])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 72, 12, 64])\n",
            "torch.Size([1, 12, 72, 72])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 72, 12, 64])\n",
            "torch.Size([1, 12, 72, 72])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 72, 12, 64])\n",
            "torch.Size([1, 12, 72, 72])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 72, 12, 64])\n",
            "torch.Size([1, 12, 72, 72])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 72, 12, 64])\n",
            "torch.Size([1, 12, 72, 72])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 72, 12, 64])\n",
            "torch.Size([1, 12, 72, 72])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 72, 12, 64])\n",
            "torch.Size([1, 12, 72, 72])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 72, 12, 64])\n",
            "torch.Size([1, 12, 72, 72])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 72, 12, 64])\n",
            "torch.Size([1, 12, 72, 72])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 73, 12, 64])\n",
            "torch.Size([1, 12, 73, 73])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 73, 12, 64])\n",
            "torch.Size([1, 12, 73, 73])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 73, 12, 64])\n",
            "torch.Size([1, 12, 73, 73])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 73, 12, 64])\n",
            "torch.Size([1, 12, 73, 73])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 73, 12, 64])\n",
            "torch.Size([1, 12, 73, 73])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 73, 12, 64])\n",
            "torch.Size([1, 12, 73, 73])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 73, 12, 64])\n",
            "torch.Size([1, 12, 73, 73])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 73, 12, 64])\n",
            "torch.Size([1, 12, 73, 73])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 73, 12, 64])\n",
            "torch.Size([1, 12, 73, 73])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 73, 12, 64])\n",
            "torch.Size([1, 12, 73, 73])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 73, 12, 64])\n",
            "torch.Size([1, 12, 73, 73])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 73, 12, 64])\n",
            "torch.Size([1, 12, 73, 73])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 74, 12, 64])\n",
            "torch.Size([1, 12, 74, 74])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 74, 12, 64])\n",
            "torch.Size([1, 12, 74, 74])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 74, 12, 64])\n",
            "torch.Size([1, 12, 74, 74])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 74, 12, 64])\n",
            "torch.Size([1, 12, 74, 74])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 74, 12, 64])\n",
            "torch.Size([1, 12, 74, 74])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 74, 12, 64])\n",
            "torch.Size([1, 12, 74, 74])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 74, 12, 64])\n",
            "torch.Size([1, 12, 74, 74])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 74, 12, 64])\n",
            "torch.Size([1, 12, 74, 74])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 74, 12, 64])\n",
            "torch.Size([1, 12, 74, 74])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 74, 12, 64])\n",
            "torch.Size([1, 12, 74, 74])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 74, 12, 64])\n",
            "torch.Size([1, 12, 74, 74])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 74, 12, 64])\n",
            "torch.Size([1, 12, 74, 74])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 75, 12, 64])\n",
            "torch.Size([1, 12, 75, 75])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 75, 12, 64])\n",
            "torch.Size([1, 12, 75, 75])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 75, 12, 64])\n",
            "torch.Size([1, 12, 75, 75])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 75, 12, 64])\n",
            "torch.Size([1, 12, 75, 75])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 75, 12, 64])\n",
            "torch.Size([1, 12, 75, 75])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 75, 12, 64])\n",
            "torch.Size([1, 12, 75, 75])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 75, 12, 64])\n",
            "torch.Size([1, 12, 75, 75])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 75, 12, 64])\n",
            "torch.Size([1, 12, 75, 75])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 75, 12, 64])\n",
            "torch.Size([1, 12, 75, 75])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 75, 12, 64])\n",
            "torch.Size([1, 12, 75, 75])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 75, 12, 64])\n",
            "torch.Size([1, 12, 75, 75])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 75, 12, 64])\n",
            "torch.Size([1, 12, 75, 75])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 76, 12, 64])\n",
            "torch.Size([1, 12, 76, 76])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 76, 12, 64])\n",
            "torch.Size([1, 12, 76, 76])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 76, 12, 64])\n",
            "torch.Size([1, 12, 76, 76])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 76, 12, 64])\n",
            "torch.Size([1, 12, 76, 76])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 76, 12, 64])\n",
            "torch.Size([1, 12, 76, 76])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 76, 12, 64])\n",
            "torch.Size([1, 12, 76, 76])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 76, 12, 64])\n",
            "torch.Size([1, 12, 76, 76])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 76, 12, 64])\n",
            "torch.Size([1, 12, 76, 76])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 76, 12, 64])\n",
            "torch.Size([1, 12, 76, 76])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 76, 12, 64])\n",
            "torch.Size([1, 12, 76, 76])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 76, 12, 64])\n",
            "torch.Size([1, 12, 76, 76])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 76, 12, 64])\n",
            "torch.Size([1, 12, 76, 76])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 77, 12, 64])\n",
            "torch.Size([1, 12, 77, 77])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 77, 12, 64])\n",
            "torch.Size([1, 12, 77, 77])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 77, 12, 64])\n",
            "torch.Size([1, 12, 77, 77])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 77, 12, 64])\n",
            "torch.Size([1, 12, 77, 77])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 77, 12, 64])\n",
            "torch.Size([1, 12, 77, 77])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 77, 12, 64])\n",
            "torch.Size([1, 12, 77, 77])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 77, 12, 64])\n",
            "torch.Size([1, 12, 77, 77])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 77, 12, 64])\n",
            "torch.Size([1, 12, 77, 77])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 77, 12, 64])\n",
            "torch.Size([1, 12, 77, 77])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 77, 12, 64])\n",
            "torch.Size([1, 12, 77, 77])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 77, 12, 64])\n",
            "torch.Size([1, 12, 77, 77])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 77, 12, 64])\n",
            "torch.Size([1, 12, 77, 77])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 78, 12, 64])\n",
            "torch.Size([1, 12, 78, 78])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 78, 12, 64])\n",
            "torch.Size([1, 12, 78, 78])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 78, 12, 64])\n",
            "torch.Size([1, 12, 78, 78])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 78, 12, 64])\n",
            "torch.Size([1, 12, 78, 78])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 78, 12, 64])\n",
            "torch.Size([1, 12, 78, 78])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 78, 12, 64])\n",
            "torch.Size([1, 12, 78, 78])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 78, 12, 64])\n",
            "torch.Size([1, 12, 78, 78])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 78, 12, 64])\n",
            "torch.Size([1, 12, 78, 78])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 78, 12, 64])\n",
            "torch.Size([1, 12, 78, 78])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 78, 12, 64])\n",
            "torch.Size([1, 12, 78, 78])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 78, 12, 64])\n",
            "torch.Size([1, 12, 78, 78])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 78, 12, 64])\n",
            "torch.Size([1, 12, 78, 78])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 79, 12, 64])\n",
            "torch.Size([1, 12, 79, 79])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 79, 12, 64])\n",
            "torch.Size([1, 12, 79, 79])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 79, 12, 64])\n",
            "torch.Size([1, 12, 79, 79])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 79, 12, 64])\n",
            "torch.Size([1, 12, 79, 79])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 79, 12, 64])\n",
            "torch.Size([1, 12, 79, 79])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 79, 12, 64])\n",
            "torch.Size([1, 12, 79, 79])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 79, 12, 64])\n",
            "torch.Size([1, 12, 79, 79])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 79, 12, 64])\n",
            "torch.Size([1, 12, 79, 79])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 79, 12, 64])\n",
            "torch.Size([1, 12, 79, 79])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 79, 12, 64])\n",
            "torch.Size([1, 12, 79, 79])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 79, 12, 64])\n",
            "torch.Size([1, 12, 79, 79])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 79, 12, 64])\n",
            "torch.Size([1, 12, 79, 79])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 80, 12, 64])\n",
            "torch.Size([1, 12, 80, 80])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 80, 12, 64])\n",
            "torch.Size([1, 12, 80, 80])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 80, 12, 64])\n",
            "torch.Size([1, 12, 80, 80])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 80, 12, 64])\n",
            "torch.Size([1, 12, 80, 80])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 80, 12, 64])\n",
            "torch.Size([1, 12, 80, 80])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 80, 12, 64])\n",
            "torch.Size([1, 12, 80, 80])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 80, 12, 64])\n",
            "torch.Size([1, 12, 80, 80])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 80, 12, 64])\n",
            "torch.Size([1, 12, 80, 80])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 80, 12, 64])\n",
            "torch.Size([1, 12, 80, 80])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 80, 12, 64])\n",
            "torch.Size([1, 12, 80, 80])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 80, 12, 64])\n",
            "torch.Size([1, 12, 80, 80])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 80, 12, 64])\n",
            "torch.Size([1, 12, 80, 80])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 81, 12, 64])\n",
            "torch.Size([1, 12, 81, 81])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 81, 12, 64])\n",
            "torch.Size([1, 12, 81, 81])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 81, 12, 64])\n",
            "torch.Size([1, 12, 81, 81])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 81, 12, 64])\n",
            "torch.Size([1, 12, 81, 81])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 81, 12, 64])\n",
            "torch.Size([1, 12, 81, 81])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 81, 12, 64])\n",
            "torch.Size([1, 12, 81, 81])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 81, 12, 64])\n",
            "torch.Size([1, 12, 81, 81])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 81, 12, 64])\n",
            "torch.Size([1, 12, 81, 81])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 81, 12, 64])\n",
            "torch.Size([1, 12, 81, 81])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 81, 12, 64])\n",
            "torch.Size([1, 12, 81, 81])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 81, 12, 64])\n",
            "torch.Size([1, 12, 81, 81])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 81, 12, 64])\n",
            "torch.Size([1, 12, 81, 81])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 82, 12, 64])\n",
            "torch.Size([1, 12, 82, 82])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 82, 12, 64])\n",
            "torch.Size([1, 12, 82, 82])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 82, 12, 64])\n",
            "torch.Size([1, 12, 82, 82])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 82, 12, 64])\n",
            "torch.Size([1, 12, 82, 82])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 82, 12, 64])\n",
            "torch.Size([1, 12, 82, 82])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 82, 12, 64])\n",
            "torch.Size([1, 12, 82, 82])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 82, 12, 64])\n",
            "torch.Size([1, 12, 82, 82])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 82, 12, 64])\n",
            "torch.Size([1, 12, 82, 82])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 82, 12, 64])\n",
            "torch.Size([1, 12, 82, 82])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 82, 12, 64])\n",
            "torch.Size([1, 12, 82, 82])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 82, 12, 64])\n",
            "torch.Size([1, 12, 82, 82])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 82, 12, 64])\n",
            "torch.Size([1, 12, 82, 82])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 83, 12, 64])\n",
            "torch.Size([1, 12, 83, 83])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 83, 12, 64])\n",
            "torch.Size([1, 12, 83, 83])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 83, 12, 64])\n",
            "torch.Size([1, 12, 83, 83])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 83, 12, 64])\n",
            "torch.Size([1, 12, 83, 83])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 83, 12, 64])\n",
            "torch.Size([1, 12, 83, 83])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 83, 12, 64])\n",
            "torch.Size([1, 12, 83, 83])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 83, 12, 64])\n",
            "torch.Size([1, 12, 83, 83])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 83, 12, 64])\n",
            "torch.Size([1, 12, 83, 83])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 83, 12, 64])\n",
            "torch.Size([1, 12, 83, 83])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 83, 12, 64])\n",
            "torch.Size([1, 12, 83, 83])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 83, 12, 64])\n",
            "torch.Size([1, 12, 83, 83])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 83, 12, 64])\n",
            "torch.Size([1, 12, 83, 83])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 84, 12, 64])\n",
            "torch.Size([1, 12, 84, 84])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 84, 12, 64])\n",
            "torch.Size([1, 12, 84, 84])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 84, 12, 64])\n",
            "torch.Size([1, 12, 84, 84])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 84, 12, 64])\n",
            "torch.Size([1, 12, 84, 84])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 84, 12, 64])\n",
            "torch.Size([1, 12, 84, 84])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 84, 12, 64])\n",
            "torch.Size([1, 12, 84, 84])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 84, 12, 64])\n",
            "torch.Size([1, 12, 84, 84])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 84, 12, 64])\n",
            "torch.Size([1, 12, 84, 84])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 84, 12, 64])\n",
            "torch.Size([1, 12, 84, 84])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 84, 12, 64])\n",
            "torch.Size([1, 12, 84, 84])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 84, 12, 64])\n",
            "torch.Size([1, 12, 84, 84])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 84, 12, 64])\n",
            "torch.Size([1, 12, 84, 84])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 85, 12, 64])\n",
            "torch.Size([1, 12, 85, 85])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 85, 12, 64])\n",
            "torch.Size([1, 12, 85, 85])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 85, 12, 64])\n",
            "torch.Size([1, 12, 85, 85])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 85, 12, 64])\n",
            "torch.Size([1, 12, 85, 85])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 85, 12, 64])\n",
            "torch.Size([1, 12, 85, 85])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 85, 12, 64])\n",
            "torch.Size([1, 12, 85, 85])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 85, 12, 64])\n",
            "torch.Size([1, 12, 85, 85])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 85, 12, 64])\n",
            "torch.Size([1, 12, 85, 85])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 85, 12, 64])\n",
            "torch.Size([1, 12, 85, 85])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 85, 12, 64])\n",
            "torch.Size([1, 12, 85, 85])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 85, 12, 64])\n",
            "torch.Size([1, 12, 85, 85])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 85, 12, 64])\n",
            "torch.Size([1, 12, 85, 85])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 86, 12, 64])\n",
            "torch.Size([1, 12, 86, 86])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 86, 12, 64])\n",
            "torch.Size([1, 12, 86, 86])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 86, 12, 64])\n",
            "torch.Size([1, 12, 86, 86])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 86, 12, 64])\n",
            "torch.Size([1, 12, 86, 86])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 86, 12, 64])\n",
            "torch.Size([1, 12, 86, 86])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 86, 12, 64])\n",
            "torch.Size([1, 12, 86, 86])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 86, 12, 64])\n",
            "torch.Size([1, 12, 86, 86])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 86, 12, 64])\n",
            "torch.Size([1, 12, 86, 86])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 86, 12, 64])\n",
            "torch.Size([1, 12, 86, 86])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 86, 12, 64])\n",
            "torch.Size([1, 12, 86, 86])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 86, 12, 64])\n",
            "torch.Size([1, 12, 86, 86])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 86, 12, 64])\n",
            "torch.Size([1, 12, 86, 86])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 87, 12, 64])\n",
            "torch.Size([1, 12, 87, 87])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 87, 12, 64])\n",
            "torch.Size([1, 12, 87, 87])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 87, 12, 64])\n",
            "torch.Size([1, 12, 87, 87])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 87, 12, 64])\n",
            "torch.Size([1, 12, 87, 87])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 87, 12, 64])\n",
            "torch.Size([1, 12, 87, 87])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 87, 12, 64])\n",
            "torch.Size([1, 12, 87, 87])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 87, 12, 64])\n",
            "torch.Size([1, 12, 87, 87])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 87, 12, 64])\n",
            "torch.Size([1, 12, 87, 87])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 87, 12, 64])\n",
            "torch.Size([1, 12, 87, 87])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 87, 12, 64])\n",
            "torch.Size([1, 12, 87, 87])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 87, 12, 64])\n",
            "torch.Size([1, 12, 87, 87])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 87, 12, 64])\n",
            "torch.Size([1, 12, 87, 87])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 88, 12, 64])\n",
            "torch.Size([1, 12, 88, 88])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 88, 12, 64])\n",
            "torch.Size([1, 12, 88, 88])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 88, 12, 64])\n",
            "torch.Size([1, 12, 88, 88])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 88, 12, 64])\n",
            "torch.Size([1, 12, 88, 88])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 88, 12, 64])\n",
            "torch.Size([1, 12, 88, 88])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 88, 12, 64])\n",
            "torch.Size([1, 12, 88, 88])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 88, 12, 64])\n",
            "torch.Size([1, 12, 88, 88])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 88, 12, 64])\n",
            "torch.Size([1, 12, 88, 88])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 88, 12, 64])\n",
            "torch.Size([1, 12, 88, 88])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 88, 12, 64])\n",
            "torch.Size([1, 12, 88, 88])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 88, 12, 64])\n",
            "torch.Size([1, 12, 88, 88])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 88, 12, 64])\n",
            "torch.Size([1, 12, 88, 88])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 89, 12, 64])\n",
            "torch.Size([1, 12, 89, 89])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 89, 12, 64])\n",
            "torch.Size([1, 12, 89, 89])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 89, 12, 64])\n",
            "torch.Size([1, 12, 89, 89])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 89, 12, 64])\n",
            "torch.Size([1, 12, 89, 89])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 89, 12, 64])\n",
            "torch.Size([1, 12, 89, 89])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 89, 12, 64])\n",
            "torch.Size([1, 12, 89, 89])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 89, 12, 64])\n",
            "torch.Size([1, 12, 89, 89])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 89, 12, 64])\n",
            "torch.Size([1, 12, 89, 89])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 89, 12, 64])\n",
            "torch.Size([1, 12, 89, 89])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 89, 12, 64])\n",
            "torch.Size([1, 12, 89, 89])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 89, 12, 64])\n",
            "torch.Size([1, 12, 89, 89])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 89, 12, 64])\n",
            "torch.Size([1, 12, 89, 89])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 90, 12, 64])\n",
            "torch.Size([1, 12, 90, 90])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 90, 12, 64])\n",
            "torch.Size([1, 12, 90, 90])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 90, 12, 64])\n",
            "torch.Size([1, 12, 90, 90])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 90, 12, 64])\n",
            "torch.Size([1, 12, 90, 90])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 90, 12, 64])\n",
            "torch.Size([1, 12, 90, 90])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 90, 12, 64])\n",
            "torch.Size([1, 12, 90, 90])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 90, 12, 64])\n",
            "torch.Size([1, 12, 90, 90])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 90, 12, 64])\n",
            "torch.Size([1, 12, 90, 90])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 90, 12, 64])\n",
            "torch.Size([1, 12, 90, 90])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 90, 12, 64])\n",
            "torch.Size([1, 12, 90, 90])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 90, 12, 64])\n",
            "torch.Size([1, 12, 90, 90])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 90, 12, 64])\n",
            "torch.Size([1, 12, 90, 90])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 91, 12, 64])\n",
            "torch.Size([1, 12, 91, 91])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 91, 12, 64])\n",
            "torch.Size([1, 12, 91, 91])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 91, 12, 64])\n",
            "torch.Size([1, 12, 91, 91])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 91, 12, 64])\n",
            "torch.Size([1, 12, 91, 91])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 91, 12, 64])\n",
            "torch.Size([1, 12, 91, 91])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 91, 12, 64])\n",
            "torch.Size([1, 12, 91, 91])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 91, 12, 64])\n",
            "torch.Size([1, 12, 91, 91])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 91, 12, 64])\n",
            "torch.Size([1, 12, 91, 91])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 91, 12, 64])\n",
            "torch.Size([1, 12, 91, 91])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 91, 12, 64])\n",
            "torch.Size([1, 12, 91, 91])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 91, 12, 64])\n",
            "torch.Size([1, 12, 91, 91])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 91, 12, 64])\n",
            "torch.Size([1, 12, 91, 91])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 92, 12, 64])\n",
            "torch.Size([1, 12, 92, 92])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 92, 12, 64])\n",
            "torch.Size([1, 12, 92, 92])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 92, 12, 64])\n",
            "torch.Size([1, 12, 92, 92])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 92, 12, 64])\n",
            "torch.Size([1, 12, 92, 92])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 92, 12, 64])\n",
            "torch.Size([1, 12, 92, 92])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 92, 12, 64])\n",
            "torch.Size([1, 12, 92, 92])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 92, 12, 64])\n",
            "torch.Size([1, 12, 92, 92])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 92, 12, 64])\n",
            "torch.Size([1, 12, 92, 92])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 92, 12, 64])\n",
            "torch.Size([1, 12, 92, 92])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 92, 12, 64])\n",
            "torch.Size([1, 12, 92, 92])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 92, 12, 64])\n",
            "torch.Size([1, 12, 92, 92])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 92, 12, 64])\n",
            "torch.Size([1, 12, 92, 92])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 93, 12, 64])\n",
            "torch.Size([1, 12, 93, 93])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 93, 12, 64])\n",
            "torch.Size([1, 12, 93, 93])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 93, 12, 64])\n",
            "torch.Size([1, 12, 93, 93])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 93, 12, 64])\n",
            "torch.Size([1, 12, 93, 93])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 93, 12, 64])\n",
            "torch.Size([1, 12, 93, 93])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 93, 12, 64])\n",
            "torch.Size([1, 12, 93, 93])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 93, 12, 64])\n",
            "torch.Size([1, 12, 93, 93])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 93, 12, 64])\n",
            "torch.Size([1, 12, 93, 93])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 93, 12, 64])\n",
            "torch.Size([1, 12, 93, 93])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 93, 12, 64])\n",
            "torch.Size([1, 12, 93, 93])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 93, 12, 64])\n",
            "torch.Size([1, 12, 93, 93])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 93, 12, 64])\n",
            "torch.Size([1, 12, 93, 93])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 94, 12, 64])\n",
            "torch.Size([1, 12, 94, 94])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 94, 12, 64])\n",
            "torch.Size([1, 12, 94, 94])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 94, 12, 64])\n",
            "torch.Size([1, 12, 94, 94])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 94, 12, 64])\n",
            "torch.Size([1, 12, 94, 94])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 94, 12, 64])\n",
            "torch.Size([1, 12, 94, 94])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 94, 12, 64])\n",
            "torch.Size([1, 12, 94, 94])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 94, 12, 64])\n",
            "torch.Size([1, 12, 94, 94])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 94, 12, 64])\n",
            "torch.Size([1, 12, 94, 94])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 94, 12, 64])\n",
            "torch.Size([1, 12, 94, 94])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 94, 12, 64])\n",
            "torch.Size([1, 12, 94, 94])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 94, 12, 64])\n",
            "torch.Size([1, 12, 94, 94])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 94, 12, 64])\n",
            "torch.Size([1, 12, 94, 94])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 95, 12, 64])\n",
            "torch.Size([1, 12, 95, 95])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 95, 12, 64])\n",
            "torch.Size([1, 12, 95, 95])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 95, 12, 64])\n",
            "torch.Size([1, 12, 95, 95])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 95, 12, 64])\n",
            "torch.Size([1, 12, 95, 95])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 95, 12, 64])\n",
            "torch.Size([1, 12, 95, 95])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 95, 12, 64])\n",
            "torch.Size([1, 12, 95, 95])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 95, 12, 64])\n",
            "torch.Size([1, 12, 95, 95])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 95, 12, 64])\n",
            "torch.Size([1, 12, 95, 95])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 95, 12, 64])\n",
            "torch.Size([1, 12, 95, 95])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 95, 12, 64])\n",
            "torch.Size([1, 12, 95, 95])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 95, 12, 64])\n",
            "torch.Size([1, 12, 95, 95])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 95, 12, 64])\n",
            "torch.Size([1, 12, 95, 95])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 96, 12, 64])\n",
            "torch.Size([1, 12, 96, 96])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 96, 12, 64])\n",
            "torch.Size([1, 12, 96, 96])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 96, 12, 64])\n",
            "torch.Size([1, 12, 96, 96])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 96, 12, 64])\n",
            "torch.Size([1, 12, 96, 96])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 96, 12, 64])\n",
            "torch.Size([1, 12, 96, 96])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 96, 12, 64])\n",
            "torch.Size([1, 12, 96, 96])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 96, 12, 64])\n",
            "torch.Size([1, 12, 96, 96])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 96, 12, 64])\n",
            "torch.Size([1, 12, 96, 96])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 96, 12, 64])\n",
            "torch.Size([1, 12, 96, 96])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 96, 12, 64])\n",
            "torch.Size([1, 12, 96, 96])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 96, 12, 64])\n",
            "torch.Size([1, 12, 96, 96])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 96, 12, 64])\n",
            "torch.Size([1, 12, 96, 96])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 97, 12, 64])\n",
            "torch.Size([1, 12, 97, 97])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 97, 12, 64])\n",
            "torch.Size([1, 12, 97, 97])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 97, 12, 64])\n",
            "torch.Size([1, 12, 97, 97])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 97, 12, 64])\n",
            "torch.Size([1, 12, 97, 97])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 97, 12, 64])\n",
            "torch.Size([1, 12, 97, 97])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 97, 12, 64])\n",
            "torch.Size([1, 12, 97, 97])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 97, 12, 64])\n",
            "torch.Size([1, 12, 97, 97])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 97, 12, 64])\n",
            "torch.Size([1, 12, 97, 97])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 97, 12, 64])\n",
            "torch.Size([1, 12, 97, 97])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 97, 12, 64])\n",
            "torch.Size([1, 12, 97, 97])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 97, 12, 64])\n",
            "torch.Size([1, 12, 97, 97])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 97, 12, 64])\n",
            "torch.Size([1, 12, 97, 97])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 98, 12, 64])\n",
            "torch.Size([1, 12, 98, 98])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 98, 12, 64])\n",
            "torch.Size([1, 12, 98, 98])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 98, 12, 64])\n",
            "torch.Size([1, 12, 98, 98])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 98, 12, 64])\n",
            "torch.Size([1, 12, 98, 98])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 98, 12, 64])\n",
            "torch.Size([1, 12, 98, 98])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 98, 12, 64])\n",
            "torch.Size([1, 12, 98, 98])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 98, 12, 64])\n",
            "torch.Size([1, 12, 98, 98])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 98, 12, 64])\n",
            "torch.Size([1, 12, 98, 98])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 98, 12, 64])\n",
            "torch.Size([1, 12, 98, 98])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 98, 12, 64])\n",
            "torch.Size([1, 12, 98, 98])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 98, 12, 64])\n",
            "torch.Size([1, 12, 98, 98])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 98, 12, 64])\n",
            "torch.Size([1, 12, 98, 98])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 99, 12, 64])\n",
            "torch.Size([1, 12, 99, 99])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 99, 12, 64])\n",
            "torch.Size([1, 12, 99, 99])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 99, 12, 64])\n",
            "torch.Size([1, 12, 99, 99])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 99, 12, 64])\n",
            "torch.Size([1, 12, 99, 99])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 99, 12, 64])\n",
            "torch.Size([1, 12, 99, 99])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 99, 12, 64])\n",
            "torch.Size([1, 12, 99, 99])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 99, 12, 64])\n",
            "torch.Size([1, 12, 99, 99])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 99, 12, 64])\n",
            "torch.Size([1, 12, 99, 99])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 99, 12, 64])\n",
            "torch.Size([1, 12, 99, 99])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 99, 12, 64])\n",
            "torch.Size([1, 12, 99, 99])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 99, 12, 64])\n",
            "torch.Size([1, 12, 99, 99])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 99, 12, 64])\n",
            "torch.Size([1, 12, 99, 99])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 100, 12, 64])\n",
            "torch.Size([1, 12, 100, 100])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 100, 12, 64])\n",
            "torch.Size([1, 12, 100, 100])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 100, 12, 64])\n",
            "torch.Size([1, 12, 100, 100])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 100, 12, 64])\n",
            "torch.Size([1, 12, 100, 100])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 100, 12, 64])\n",
            "torch.Size([1, 12, 100, 100])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 100, 12, 64])\n",
            "torch.Size([1, 12, 100, 100])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 100, 12, 64])\n",
            "torch.Size([1, 12, 100, 100])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 100, 12, 64])\n",
            "torch.Size([1, 12, 100, 100])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 100, 12, 64])\n",
            "torch.Size([1, 12, 100, 100])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 100, 12, 64])\n",
            "torch.Size([1, 12, 100, 100])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 100, 12, 64])\n",
            "torch.Size([1, 12, 100, 100])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 100, 12, 64])\n",
            "torch.Size([1, 12, 100, 100])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 101, 12, 64])\n",
            "torch.Size([1, 12, 101, 101])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 101, 12, 64])\n",
            "torch.Size([1, 12, 101, 101])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 101, 12, 64])\n",
            "torch.Size([1, 12, 101, 101])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 101, 12, 64])\n",
            "torch.Size([1, 12, 101, 101])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 101, 12, 64])\n",
            "torch.Size([1, 12, 101, 101])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 101, 12, 64])\n",
            "torch.Size([1, 12, 101, 101])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 101, 12, 64])\n",
            "torch.Size([1, 12, 101, 101])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 101, 12, 64])\n",
            "torch.Size([1, 12, 101, 101])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 101, 12, 64])\n",
            "torch.Size([1, 12, 101, 101])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 101, 12, 64])\n",
            "torch.Size([1, 12, 101, 101])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 101, 12, 64])\n",
            "torch.Size([1, 12, 101, 101])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 101, 12, 64])\n",
            "torch.Size([1, 12, 101, 101])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 102, 12, 64])\n",
            "torch.Size([1, 12, 102, 102])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 102, 12, 64])\n",
            "torch.Size([1, 12, 102, 102])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 102, 12, 64])\n",
            "torch.Size([1, 12, 102, 102])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 102, 12, 64])\n",
            "torch.Size([1, 12, 102, 102])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 102, 12, 64])\n",
            "torch.Size([1, 12, 102, 102])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 102, 12, 64])\n",
            "torch.Size([1, 12, 102, 102])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 102, 12, 64])\n",
            "torch.Size([1, 12, 102, 102])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 102, 12, 64])\n",
            "torch.Size([1, 12, 102, 102])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 102, 12, 64])\n",
            "torch.Size([1, 12, 102, 102])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 102, 12, 64])\n",
            "torch.Size([1, 12, 102, 102])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 102, 12, 64])\n",
            "torch.Size([1, 12, 102, 102])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 102, 12, 64])\n",
            "torch.Size([1, 12, 102, 102])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 103, 12, 64])\n",
            "torch.Size([1, 12, 103, 103])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 103, 12, 64])\n",
            "torch.Size([1, 12, 103, 103])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 103, 12, 64])\n",
            "torch.Size([1, 12, 103, 103])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 103, 12, 64])\n",
            "torch.Size([1, 12, 103, 103])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 103, 12, 64])\n",
            "torch.Size([1, 12, 103, 103])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 103, 12, 64])\n",
            "torch.Size([1, 12, 103, 103])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 103, 12, 64])\n",
            "torch.Size([1, 12, 103, 103])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 103, 12, 64])\n",
            "torch.Size([1, 12, 103, 103])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 103, 12, 64])\n",
            "torch.Size([1, 12, 103, 103])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 103, 12, 64])\n",
            "torch.Size([1, 12, 103, 103])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 103, 12, 64])\n",
            "torch.Size([1, 12, 103, 103])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 103, 12, 64])\n",
            "torch.Size([1, 12, 103, 103])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 104, 12, 64])\n",
            "torch.Size([1, 12, 104, 104])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 104, 12, 64])\n",
            "torch.Size([1, 12, 104, 104])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 104, 12, 64])\n",
            "torch.Size([1, 12, 104, 104])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 104, 12, 64])\n",
            "torch.Size([1, 12, 104, 104])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 104, 12, 64])\n",
            "torch.Size([1, 12, 104, 104])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 104, 12, 64])\n",
            "torch.Size([1, 12, 104, 104])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 104, 12, 64])\n",
            "torch.Size([1, 12, 104, 104])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 104, 12, 64])\n",
            "torch.Size([1, 12, 104, 104])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 104, 12, 64])\n",
            "torch.Size([1, 12, 104, 104])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 104, 12, 64])\n",
            "torch.Size([1, 12, 104, 104])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 104, 12, 64])\n",
            "torch.Size([1, 12, 104, 104])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 104, 12, 64])\n",
            "torch.Size([1, 12, 104, 104])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 105, 12, 64])\n",
            "torch.Size([1, 12, 105, 105])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 105, 12, 64])\n",
            "torch.Size([1, 12, 105, 105])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 105, 12, 64])\n",
            "torch.Size([1, 12, 105, 105])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 105, 12, 64])\n",
            "torch.Size([1, 12, 105, 105])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 105, 12, 64])\n",
            "torch.Size([1, 12, 105, 105])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 105, 12, 64])\n",
            "torch.Size([1, 12, 105, 105])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 105, 12, 64])\n",
            "torch.Size([1, 12, 105, 105])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 105, 12, 64])\n",
            "torch.Size([1, 12, 105, 105])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 105, 12, 64])\n",
            "torch.Size([1, 12, 105, 105])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 105, 12, 64])\n",
            "torch.Size([1, 12, 105, 105])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 105, 12, 64])\n",
            "torch.Size([1, 12, 105, 105])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 105, 12, 64])\n",
            "torch.Size([1, 12, 105, 105])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 106, 12, 64])\n",
            "torch.Size([1, 12, 106, 106])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 106, 12, 64])\n",
            "torch.Size([1, 12, 106, 106])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 106, 12, 64])\n",
            "torch.Size([1, 12, 106, 106])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 106, 12, 64])\n",
            "torch.Size([1, 12, 106, 106])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 106, 12, 64])\n",
            "torch.Size([1, 12, 106, 106])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 106, 12, 64])\n",
            "torch.Size([1, 12, 106, 106])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 106, 12, 64])\n",
            "torch.Size([1, 12, 106, 106])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 106, 12, 64])\n",
            "torch.Size([1, 12, 106, 106])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 106, 12, 64])\n",
            "torch.Size([1, 12, 106, 106])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 106, 12, 64])\n",
            "torch.Size([1, 12, 106, 106])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 106, 12, 64])\n",
            "torch.Size([1, 12, 106, 106])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 106, 12, 64])\n",
            "torch.Size([1, 12, 106, 106])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 107, 12, 64])\n",
            "torch.Size([1, 12, 107, 107])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 107, 12, 64])\n",
            "torch.Size([1, 12, 107, 107])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 107, 12, 64])\n",
            "torch.Size([1, 12, 107, 107])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 107, 12, 64])\n",
            "torch.Size([1, 12, 107, 107])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 107, 12, 64])\n",
            "torch.Size([1, 12, 107, 107])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 107, 12, 64])\n",
            "torch.Size([1, 12, 107, 107])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 107, 12, 64])\n",
            "torch.Size([1, 12, 107, 107])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 107, 12, 64])\n",
            "torch.Size([1, 12, 107, 107])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 107, 12, 64])\n",
            "torch.Size([1, 12, 107, 107])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 107, 12, 64])\n",
            "torch.Size([1, 12, 107, 107])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 107, 12, 64])\n",
            "torch.Size([1, 12, 107, 107])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 107, 12, 64])\n",
            "torch.Size([1, 12, 107, 107])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 108, 12, 64])\n",
            "torch.Size([1, 12, 108, 108])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 108, 12, 64])\n",
            "torch.Size([1, 12, 108, 108])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 108, 12, 64])\n",
            "torch.Size([1, 12, 108, 108])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 108, 12, 64])\n",
            "torch.Size([1, 12, 108, 108])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 108, 12, 64])\n",
            "torch.Size([1, 12, 108, 108])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 108, 12, 64])\n",
            "torch.Size([1, 12, 108, 108])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 108, 12, 64])\n",
            "torch.Size([1, 12, 108, 108])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 108, 12, 64])\n",
            "torch.Size([1, 12, 108, 108])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 108, 12, 64])\n",
            "torch.Size([1, 12, 108, 108])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 108, 12, 64])\n",
            "torch.Size([1, 12, 108, 108])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 108, 12, 64])\n",
            "torch.Size([1, 12, 108, 108])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 108, 12, 64])\n",
            "torch.Size([1, 12, 108, 108])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 109, 12, 64])\n",
            "torch.Size([1, 12, 109, 109])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 109, 12, 64])\n",
            "torch.Size([1, 12, 109, 109])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 109, 12, 64])\n",
            "torch.Size([1, 12, 109, 109])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 109, 12, 64])\n",
            "torch.Size([1, 12, 109, 109])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 109, 12, 64])\n",
            "torch.Size([1, 12, 109, 109])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 109, 12, 64])\n",
            "torch.Size([1, 12, 109, 109])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 109, 12, 64])\n",
            "torch.Size([1, 12, 109, 109])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 109, 12, 64])\n",
            "torch.Size([1, 12, 109, 109])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 109, 12, 64])\n",
            "torch.Size([1, 12, 109, 109])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 109, 12, 64])\n",
            "torch.Size([1, 12, 109, 109])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 109, 12, 64])\n",
            "torch.Size([1, 12, 109, 109])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 109, 12, 64])\n",
            "torch.Size([1, 12, 109, 109])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 110, 12, 64])\n",
            "torch.Size([1, 12, 110, 110])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 110, 12, 64])\n",
            "torch.Size([1, 12, 110, 110])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 110, 12, 64])\n",
            "torch.Size([1, 12, 110, 110])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 110, 12, 64])\n",
            "torch.Size([1, 12, 110, 110])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 110, 12, 64])\n",
            "torch.Size([1, 12, 110, 110])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 110, 12, 64])\n",
            "torch.Size([1, 12, 110, 110])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 110, 12, 64])\n",
            "torch.Size([1, 12, 110, 110])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 110, 12, 64])\n",
            "torch.Size([1, 12, 110, 110])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 110, 12, 64])\n",
            "torch.Size([1, 12, 110, 110])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 110, 12, 64])\n",
            "torch.Size([1, 12, 110, 110])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 110, 12, 64])\n",
            "torch.Size([1, 12, 110, 110])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 110, 12, 64])\n",
            "torch.Size([1, 12, 110, 110])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 111, 12, 64])\n",
            "torch.Size([1, 12, 111, 111])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 111, 12, 64])\n",
            "torch.Size([1, 12, 111, 111])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 111, 12, 64])\n",
            "torch.Size([1, 12, 111, 111])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 111, 12, 64])\n",
            "torch.Size([1, 12, 111, 111])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 111, 12, 64])\n",
            "torch.Size([1, 12, 111, 111])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 111, 12, 64])\n",
            "torch.Size([1, 12, 111, 111])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 111, 12, 64])\n",
            "torch.Size([1, 12, 111, 111])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 111, 12, 64])\n",
            "torch.Size([1, 12, 111, 111])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 111, 12, 64])\n",
            "torch.Size([1, 12, 111, 111])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 111, 12, 64])\n",
            "torch.Size([1, 12, 111, 111])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 111, 12, 64])\n",
            "torch.Size([1, 12, 111, 111])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 111, 12, 64])\n",
            "torch.Size([1, 12, 111, 111])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 112, 12, 64])\n",
            "torch.Size([1, 12, 112, 112])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 112, 12, 64])\n",
            "torch.Size([1, 12, 112, 112])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 112, 12, 64])\n",
            "torch.Size([1, 12, 112, 112])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 112, 12, 64])\n",
            "torch.Size([1, 12, 112, 112])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 112, 12, 64])\n",
            "torch.Size([1, 12, 112, 112])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 112, 12, 64])\n",
            "torch.Size([1, 12, 112, 112])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 112, 12, 64])\n",
            "torch.Size([1, 12, 112, 112])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 112, 12, 64])\n",
            "torch.Size([1, 12, 112, 112])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 112, 12, 64])\n",
            "torch.Size([1, 12, 112, 112])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 112, 12, 64])\n",
            "torch.Size([1, 12, 112, 112])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 112, 12, 64])\n",
            "torch.Size([1, 12, 112, 112])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 112, 12, 64])\n",
            "torch.Size([1, 12, 112, 112])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 113, 12, 64])\n",
            "torch.Size([1, 12, 113, 113])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 113, 12, 64])\n",
            "torch.Size([1, 12, 113, 113])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 113, 12, 64])\n",
            "torch.Size([1, 12, 113, 113])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 113, 12, 64])\n",
            "torch.Size([1, 12, 113, 113])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 113, 12, 64])\n",
            "torch.Size([1, 12, 113, 113])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 113, 12, 64])\n",
            "torch.Size([1, 12, 113, 113])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 113, 12, 64])\n",
            "torch.Size([1, 12, 113, 113])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 113, 12, 64])\n",
            "torch.Size([1, 12, 113, 113])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 113, 12, 64])\n",
            "torch.Size([1, 12, 113, 113])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 113, 12, 64])\n",
            "torch.Size([1, 12, 113, 113])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 113, 12, 64])\n",
            "torch.Size([1, 12, 113, 113])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 113, 12, 64])\n",
            "torch.Size([1, 12, 113, 113])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 114, 12, 64])\n",
            "torch.Size([1, 12, 114, 114])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 114, 12, 64])\n",
            "torch.Size([1, 12, 114, 114])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 114, 12, 64])\n",
            "torch.Size([1, 12, 114, 114])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 114, 12, 64])\n",
            "torch.Size([1, 12, 114, 114])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 114, 12, 64])\n",
            "torch.Size([1, 12, 114, 114])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 114, 12, 64])\n",
            "torch.Size([1, 12, 114, 114])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 114, 12, 64])\n",
            "torch.Size([1, 12, 114, 114])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 114, 12, 64])\n",
            "torch.Size([1, 12, 114, 114])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 114, 12, 64])\n",
            "torch.Size([1, 12, 114, 114])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 114, 12, 64])\n",
            "torch.Size([1, 12, 114, 114])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 114, 12, 64])\n",
            "torch.Size([1, 12, 114, 114])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 114, 12, 64])\n",
            "torch.Size([1, 12, 114, 114])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 115, 12, 64])\n",
            "torch.Size([1, 12, 115, 115])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 115, 12, 64])\n",
            "torch.Size([1, 12, 115, 115])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 115, 12, 64])\n",
            "torch.Size([1, 12, 115, 115])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 115, 12, 64])\n",
            "torch.Size([1, 12, 115, 115])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 115, 12, 64])\n",
            "torch.Size([1, 12, 115, 115])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 115, 12, 64])\n",
            "torch.Size([1, 12, 115, 115])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 115, 12, 64])\n",
            "torch.Size([1, 12, 115, 115])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 115, 12, 64])\n",
            "torch.Size([1, 12, 115, 115])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 115, 12, 64])\n",
            "torch.Size([1, 12, 115, 115])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 115, 12, 64])\n",
            "torch.Size([1, 12, 115, 115])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 115, 12, 64])\n",
            "torch.Size([1, 12, 115, 115])\n",
            "torch.Size([12, 768, 64])\n",
            "torch.Size([1, 115, 12, 64])\n",
            "torch.Size([1, 12, 115, 115])\n",
            "The Total Perspective Vortex derives its picture of the whole Universe on the principle of the total perspective. The total perspective is the view of the whole Universe from the point of view of the observer. The total perspective is the view of the whole Universe from the point of view of the observer. The total perspective is the view of the whole Universe from the point of view of the observer. The total perspective is the view of the whole Universe from the point of view of the observer. The total perspective is the view of the whole Universe from the point of view of the observer. The\n"
          ]
        }
      ],
      "source": [
        "test_string = '''The Total Perspective Vortex derives its picture of the whole Universe on the principle of'''\n",
        "for i in tqdm(range(100)):\n",
        "    test_tokens = reference_gpt2.to_tokens(test_string).to(device)\n",
        "    demo_logits = demo_gpt2(test_tokens)\n",
        "    test_string += reference_gpt2.tokenizer.decode(demo_logits[-1, -1].argmax())\n",
        "\n",
        "print(test_string)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tGrShZtp0O0E"
      },
      "source": [
        "In later sections, we'll learn to generate text in slightly more interesting ways than just argmaxing the output.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QfeyG6NZm4SC"
      },
      "source": [
        "# 3️⃣ Training a Transformer\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "> ##### Learning objectives\n",
        ">\n",
        "> * Understand how to train a transformer from scratch\n",
        "> * Write a basic transformer training loop\n",
        "> * Interpret the transformer's falling cross entropy loss with reference to features of the training data (e.g. bigram frequencies)\n"
      ],
      "metadata": {
        "id": "RGFuGCKtnNy3"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now that we've built our transformer, and verified that it performs as expected when we load in weights, let's try training it from scratch!\n",
        "\n",
        "This is a lightweight demonstration of how you can actually train your own GPT-2 with this code! Here we train a tiny model on a tiny dataset, but it's fundamentally the same code for training a larger/more real model (though you'll need beefier GPUs and data parallelism to do it remotely efficiently, and fancier parallelism for much bigger ones).\n",
        "\n",
        "For our purposes, we'll train 2L 4 heads per layer model, with context length 256, for 10*200 steps of batch size 16, just to show what it looks like (and so the notebook doesn't melt your colab / machine!)."
      ],
      "metadata": {
        "id": "K6oZKflnnXgj"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Create Model"
      ],
      "metadata": {
        "id": "-VB5ELjInZzj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model_cfg = Config(\n",
        "    debug=False,\n",
        "    d_model=256,\n",
        "    n_heads=4,\n",
        "    d_head=64,\n",
        "    d_mlp=1024,\n",
        "    n_layers=2,\n",
        "    n_ctx=256,\n",
        "    d_vocab=reference_gpt2.cfg.d_vocab\n",
        ")\n",
        "model = DemoTransformer(model_cfg)"
      ],
      "metadata": {
        "id": "4TMr-rMQnayL"
      },
      "execution_count": 67,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Training Args\n",
        "\n",
        "\n",
        "Note, for this optimization we'll be using **weight decay**."
      ],
      "metadata": {
        "id": "ggSGeTT_nb5Z"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "@dataclass\n",
        "class TransformerTrainingArgs():\n",
        "\t\tbatch_size = 16\n",
        "\t\tepochs = 10\n",
        "\t\tmax_steps_per_epoch = 200\n",
        "\t\tlr = 1e-3\n",
        "\t\tweight_decay = 1e-2\n",
        "\t\twandb_project: Optional[str] = \"day1-demotransformer\"\n",
        "\t\twandb_name: Optional[str] = None\n",
        "\n",
        "\n",
        "args = TransformerTrainingArgs()"
      ],
      "metadata": {
        "id": "95hodhD_ndRy"
      },
      "execution_count": 72,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Create Data\n",
        "\n",
        "We load in a tiny dataset made by Neel Nanda, with the first 10K entries in the Pile (inspired by Stas' version for OpenWebText!)"
      ],
      "metadata": {
        "id": "lR8gVegcnfW2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "dataset = datasets.load_dataset(\"NeelNanda/pile-10k\", split=\"train\").remove_columns(\"meta\")\n",
        "print(dataset)\n",
        "print(dataset[0]['text'][:100])"
      ],
      "metadata": {
        "id": "9udsnznsnik8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 234,
          "referenced_widgets": [
            "28049f90483e4742bcb9eb459a6db7eb",
            "d8afdac0c384499f89d63b509a92b4cc",
            "fdd8bc08dd3f4001aea23ebb2d8b41d4",
            "65e1a1edeaa1415d80cb3cca5b83777d",
            "eb9f96f11d8b4868a16c01d8ba6eb41a",
            "cae77d53492c48e2893aafc86b845f7c",
            "803a5a794f9a414fb5499a60a1346d47",
            "7b5b16b06b604209b38b8b30eb2893f7",
            "c48e777e11f14f4a8e2f9e6bc341703c",
            "af8ff1c0eae3482b91be5dd495abd80d",
            "d0f05229686d4283bca4ae5f5b7b41b4",
            "ec252457008f4a6fb239029baf2d4acd",
            "c1830bae7a5b42f78d10591f7fcb0f77",
            "911c53dce604485d9dae7ebe3cf46c43",
            "5a2259982aed4047badce93e30ade5c5",
            "982455c4716641aba7b9327a2b21be6b",
            "9f813ae9af5a43e78fd68f17bbfab419",
            "b0b588200725423187952bc78b124533",
            "64387fc0a000413bbad3cbe75e1fb75a",
            "4be5864a06d54e0d83a0885ce073624f",
            "317a64d53fc64949bbede3e108cc58f1",
            "037027c253564438b41b840c5340e5ac",
            "9c4ce91ad7fc4d0c9cad41c4f7881a9e",
            "3483ac4d87234799890d3d5b11ec310b",
            "577ba07f924c4c7b84b49030e492460c",
            "c8d5b1b02c28469f816edd1d7457f136",
            "867e0109b9c94956960a7f732f63b5a6",
            "b117855b37a241209045453dc37465ed",
            "1da24dcd5424475ea2c3bc3a2e2cff4e",
            "ddf61b8fbd05423d84443daf23456a0c",
            "162bc38747104d55b3a68a56db35ae57",
            "8b6cccb7541243e8bc449bd203dec27e",
            "b75b5289c00e444eababd1cc92c6fddf",
            "593d5bcaee1f45ed9dae0a7ce1afcd7e",
            "9eebc53633df45df8d406b2d72547964",
            "ab8dd833b77446a6848e088dfe1eaeb5",
            "8603f03604fd48bb909b43a02c5ceee7",
            "b57a61601822442e8fcd6f5298374f24",
            "b7dab9943a7c442d8c23edc77dbd00b6",
            "d8d8b9ae38144646ae9b75a6ef342bd5",
            "b80569503bd74de99fc2ddd6f3b66418",
            "6abc83f0419a41ceb0db7ea8b38e61f1",
            "c3818283461e4172bd992f8881bc2337",
            "5af3388392724299b61a31bfa7d2628d"
          ]
        },
        "outputId": "d632c03b-99bd-41c1-b770-ab46fd9b7a42"
      },
      "execution_count": 73,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading readme:   0%|          | 0.00/373 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "28049f90483e4742bcb9eb459a6db7eb"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading metadata:   0%|          | 0.00/921 [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "ec252457008f4a6fb239029baf2d4acd"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Downloading data:   0%|          | 0.00/33.3M [00:00<?, ?B/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "9c4ce91ad7fc4d0c9cad41c4f7881a9e"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Generating train split:   0%|          | 0/10000 [00:00<?, ? examples/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "593d5bcaee1f45ed9dae0a7ce1afcd7e"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Dataset({\n",
            "    features: ['text'],\n",
            "    num_rows: 10000\n",
            "})\n",
            "It is done, and submitted. You can play “Survival of the Tastiest” on Android, and on the web. Playi\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "`tokenize_and_concatenate` is a useful function which takes our dataset of strings, and returns a dataset of token IDs ready to feed into the model. We then create a dataloader from this tokenized dataset. The useful method `train_test_split` can give us a training and testing set.\n"
      ],
      "metadata": {
        "id": "chj2r8J-njhC"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tokenized_dataset = tokenize_and_concatenate(dataset, reference_gpt2.tokenizer, streaming=False, max_length=model.cfg.n_ctx, column_name=\"text\", add_bos_token=True, num_proc=4)\n",
        "\n",
        "dataset_dict = tokenized_dataset.train_test_split(test_size=1000)\n",
        "train_loader = DataLoader(dataset_dict[\"train\"], batch_size=args.batch_size, shuffle=True, num_workers=4, pin_memory=True)\n",
        "test_loader = DataLoader(dataset_dict[\"test\"], batch_size=args.batch_size, shuffle=False, num_workers=4, pin_memory=True)"
      ],
      "metadata": {
        "id": "mgogWcLUnk0U",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 176,
          "referenced_widgets": [
            "7f5d82a1a9a7450e925485317467a4c0",
            "ea9bd9476b93410898a9de03266004b1",
            "140cd67951a3438992a139e743aff7b5",
            "6a716a3e67e045b9847c9fc219fcf6a0",
            "65dd5d2542c445bebcaf57c269d7082c",
            "b42c5f26fa674b1aaca409a178b77a14",
            "048dea62e7e04eccae28f749160a25bd",
            "c5e006f3e2ec4b5db8e43b28f416ed2f",
            "249a4fff64624c9a8f5ac40f0a4d11ce",
            "dcf1fdfba8034e7b811b856b5b685ec9",
            "ada4b34439914f619b2d9fac9d293378"
          ]
        },
        "outputId": "e86c870b-c6fb-4dff-995b-f55f2b906b2a"
      },
      "execution_count": 74,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Map (num_proc=4):   0%|          | 0/10000 [00:00<?, ? examples/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "7f5d82a1a9a7450e925485317467a4c0"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Token indices sequence length is longer than the specified maximum sequence length for this model (80023 > 1024). Running this sequence through the model will result in indexing errors\n",
            "Token indices sequence length is longer than the specified maximum sequence length for this model (101051 > 1024). Running this sequence through the model will result in indexing errors\n",
            "Token indices sequence length is longer than the specified maximum sequence length for this model (155995 > 1024). Running this sequence through the model will result in indexing errors\n",
            "Token indices sequence length is longer than the specified maximum sequence length for this model (229134 > 1024). Running this sequence through the model will result in indexing errors\n",
            "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py:558: UserWarning: This DataLoader will create 4 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.\n",
            "  warnings.warn(_create_warning_msg(\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "When we iterate through these dataloaders, we will find dictionaries with the single key `'tokens'`, which maps to a tensor of token IDs with shape `(batch, seq_len)`."
      ],
      "metadata": {
        "id": "Fcx4oMaInlpi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "first_batch = train_loader.dataset[:args.batch_size]\n",
        "\n",
        "print(first_batch.keys())\n",
        "print(first_batch['tokens'].shape)"
      ],
      "metadata": {
        "id": "Y_Y7vD8BnqK-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "08c9640c-b4bb-4c9a-dee4-d0a30946a1ad"
      },
      "execution_count": 75,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "dict_keys(['tokens'])\n",
            "torch.Size([16, 256])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Training Loop\n",
        "\n",
        "If you did the material on [training loops](https://arena-ch0-fundamentals.streamlit.app/[0.3]_ResNets#training-loop) during the first week, this should all be familiar to you. If not, you can skim that section for an overview of the key concepts. The start of the **Training loop** section is most important, and the subsections on [Modularisation](https://arena-ch0-fundamentals.streamlit.app/[0.3]_ResNets#modularisation) and [dataclasses](https://arena-ch0-fundamentals.streamlit.app/[0.3]_ResNets#aside-dataclasses) are also very useful. Lastly, we'll also be using Weights and Biases to train our model - you can read about how to use it [here](https://arena-ch0-fundamentals.streamlit.app/[0.4]_Optimization#what-is-weights-and-biases). Here are (roughly) all the things you should know for the following exercises:\n",
        "                \n",
        "* The key parts of a gradient update step are:\n",
        "    * Calculating the (cross-entropy) loss between a model's output and the true labels,\n",
        "    * `loss.backward()` - calculate gradients of the loss with respect to the model parameters,\n",
        "    * `optimizer.step()` - update the model parameters using the gradients,\n",
        "    * `optimizer.zero_grad()` - zero the gradients so they don't accumulate.\n",
        "* We can nicely package up training loops into a class, which includes methods for training and validation steps among other things. This helps with writing code that can be reused in different contexts.\n",
        "* We can use dataclasses to store all the arguments relevant to training in one place, and then pass them to our trainer class. Autocompletion is one nice bonus of this!\n",
        "    * Be careful of scope here, you want to make sure you're referring to `self.args` within the trainer class, rather than the global `args`.\n",
        "* You can use Weights and Biases to track experiments and log relevant variables. The three essential functions are:\n",
        "    * `wandb.init()` - initialize a new run, takes arguments `project`, `name` and `config` (among others).\n",
        "    * `wandb.log()` - log a dictionary of variables, e.g. `{\"loss\": loss}`. Also takes a `step` argument.\n",
        "    * `wandb.finish()` - called at the end of training (no arguments)."
      ],
      "metadata": {
        "id": "hyjlKG-AnrZc"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Exercise - write training loop\n",
        "\n",
        "```c\n",
        "Difficulty: 🟠🟠🟠⚪⚪\n",
        "Importance: 🟠🟠🟠🟠⚪\n",
        "\n",
        "You should spend up to 10-20 minutes on this exercise.\n",
        "```"
      ],
      "metadata": {
        "id": "tvM-W_TGnuoN"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "You should fill in the methods below. Some guidance:\n",
        "\n",
        "* Remember we were able to calculate cross entropy loss using the `get_log_probs` function in the previous section.\n",
        "* You should use the optimizer `t.optim.AdamW` (Adam with weight decay), and with hyperparameters `lr` and `weight_decay` taken from your `TransformerTrainingArgs` dataclass instance.\n",
        "* The easiest way to compute accuracy is to have the `validation_step` method return a 1D boolean tensor indicating the positions where the model's prediction was correct. Then you can concatenate all these tensors together and take the mean to get the overall accuracy for the epoch.\n",
        "* We've given you the argument `max_steps_per_epoch`, a hacky way of making sure the training phase in each epoch doesn't go on for too long. You can terminate each training phase after this many steps.\n",
        "* Remember to move tokens to your device, via `tokens.to(device)` (this should be a global variable, defined at the top of your notebook).\n",
        "* You can refer back to the training loops from the [previous chapter of the course](https://arena-ch0-fundamentals.streamlit.app/[0.3]_ResNets#training-loop) if you'd like.\n"
      ],
      "metadata": {
        "id": "2qbuJ4winvyY"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "class TransformerTrainer:\n",
        "\tdef __init__(self, args: TransformerTrainingArgs, model: DemoTransformer):\n",
        "\t\tsuper().__init__()\n",
        "\t\tself.model = model\n",
        "\t\tself.args = args\n",
        "\t\tself.optimizer = t.optim.AdamW(self.model.parameters(), lr=args.lr, weight_decay=args.weight_decay)\n",
        "\t\tself.step = 0\n",
        "\n",
        "\n",
        "\tdef training_step(self, batch: Dict[str, Int[Tensor, \"batch seq\"]]) -> Float[Tensor, \"\"]:\n",
        "\t\t'''\n",
        "\t\tCalculates the loss on the tokens in the batch, performs a gradient update step, and logs the loss.\n",
        "\n",
        "\t\tRemember that `batch` is a dictionary with the single key 'tokens'.\n",
        "\t\t'''\n",
        "\n",
        "\n",
        "\t\tbatch = batch[\"tokens\"].to(device)\n",
        "\t\tloss = -get_log_probs(self.model(batch)).mean()\n",
        "\n",
        "\t\tloss.backwards()\n",
        "\t\tself.optimizer.step()\n",
        "\t\tself.optimizer.zero_grad()\n",
        "\t\tself.step += 1\n",
        "\t\treturn loss\n",
        "\n",
        "\n",
        "\n",
        "\tdef validation_step(self, batch: Dict[str, Int[Tensor, \"batch seq\"]]):\n",
        "\t\t'''\n",
        "\t\tCalculates & returns the accuracy on the tokens in the batch (i.e. how often the model's prediction\n",
        "\t\tis correct). Logging should happen in the `train` function (after we've computed the accuracy for\n",
        "\t\tthe whole validation set).\n",
        "\t\t'''\n",
        "\n",
        "\t\tbatch = batch[\"tokens\"].to(device)\n",
        "\t\tx = self.model(batch)[:, :-1]\n",
        "\t\tpredictions = logits.argmax(dim=-1)\n",
        "\t\treturn (predictions == batch[:, 1:]).flatten()\n",
        "\n",
        "\n",
        "\tdef train(self):\n",
        "\t\t'''\n",
        "\t\tTrains the model, for `self.args.epochs` epochs. Also handles wandb initialisation, and early stopping\n",
        "\t\tfor each epoch at `self.args.max_steps_per_epoch` steps.\n",
        "\t\t'''\n",
        "\n",
        "\n",
        "\t\tdataloader = self.train_loader\n",
        "\n",
        "\t\tfor epoch in range(self.args.epochs):\n",
        "\n",
        "\t\t\tfor i, batch in enumerate(dataloader):\n",
        "\t\t\t\tloss = self.training_step(batch)\n",
        "\n",
        "\n",
        "\t\t\t\tif i >= self.args.max_steps_per_epoch:\n",
        "\t\t\t\t\t\tbreak\n",
        "\n",
        "\n",
        "\t\t\tvalidation_predictions_acc = t.concat([self.validation_step(batch) for batch in self.test_loader()])\n",
        "\t\t\taccuracy = correct_predictions.float().mean().item()\n",
        "\t \t\tprint(accuracy)\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\tdef train_loader(self) -> DataLoader:\n",
        "\t\t'''Returns train loader (as in code above).'''\n",
        "\t\treturn DataLoader(dataset_dict[\"train\"], batch_size=self.args.batch_size, shuffle=True, num_workers=4, pin_memory=True)\n",
        "\n",
        "\n",
        "\tdef test_loader(self) -> DataLoader:\n",
        "\t\t'''Returns test loader (as in code above).'''\n",
        "\t\treturn DataLoader(dataset_dict[\"test\"], batch_size=self.args.batch_size, shuffle=False, num_workers=4, pin_memory=True)"
      ],
      "metadata": {
        "id": "zVBEDTGMnxSk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Note - this section of the course used to use PyTorch Lightning, but this has now been taken out. You can see the old version of the training code which used PyTorch Lightning in the dropdown below.\n",
        "\n",
        "<details>\n",
        "<summary>PyTorch Lighting training loop</summary>\n",
        "\n",
        "```python\n",
        "class LitTransformer(pl.LightningModule):\n",
        "\tdef __init__(self, args: TransformerTrainingArgs, model: DemoTransformer, data_loader: DataLoader):\n",
        "\t\tsuper().__init__()\n",
        "\t\tself.model = model\n",
        "\t\tself.cfg = model.cfg\n",
        "\t\tself.args = args\n",
        "\t\tself.data_loader = data_loader\n",
        "\n",
        "\tdef forward(self, tokens: Int[Tensor, \"batch position\"]) -> Float[Tensor, \"batch position d_vocab\"]:\n",
        "\t\tlogits = self.model(tokens)\n",
        "\t\treturn logits\n",
        "\n",
        "\tdef training_step(self, batch: Dict[str, Tensor], batch_idx: int) -> Float[Tensor, \"\"]:\n",
        "\t\t'''\n",
        "\t\tHere you compute and return the training loss and some additional metrics for e.g.\n",
        "\t\tthe progress bar or logger.\n",
        "\t\t'''\n",
        "\t\ttokens = batch[\"tokens\"].to(device)\n",
        "\t\tlogits = self.model(tokens)\n",
        "\t\tloss = -get_log_probs(logits, tokens).mean()\n",
        "\t\tself.log(\"train_loss\", loss)\n",
        "\t\treturn loss\n",
        "\n",
        "\tdef configure_optimizers(self):\n",
        "\t\t'''\n",
        "\t\tChoose what optimizers and learning-rate schedulers to use in your optimization.\n",
        "\t\t'''\n",
        "\t\toptimizer = t.optim.AdamW(self.model.parameters(), lr=self.args.lr, weight_decay=self.args.weight_decay)\n",
        "\t\treturn optimizer\n",
        "\n",
        "\tdef train_dataloader(self):\n",
        "\t\treturn self.data_loader\n",
        "\n",
        "\n",
        "litmodel = LitTransformer(args, model, data_loader)\n",
        "logger = WandbLogger(save_dir=args.log_dir, project=args.log_name, name=args.run_name)\n",
        "\n",
        "trainer = pl.Trainer(\n",
        "    max_epochs=args.max_epochs,\n",
        "    logger=logger,\n",
        "    log_every_n_steps=args.log_every_n_steps\n",
        ")\n",
        "trainer.fit(model=litmodel, train_dataloaders=litmodel.data_loader)\n",
        "wandb.finish()\n",
        "```\n",
        "\n",
        "</details>"
      ],
      "metadata": {
        "id": "yvyAODwP3D8K"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model = DemoTransformer(model_cfg).to(device)\n",
        "args = TransformerTrainingArgs()\n",
        "trainer = TransformerTrainer(args, model)\n",
        "trainer.train()"
      ],
      "metadata": {
        "id": "nWJyh1lon0IH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "When you run the code for the first time, you'll have to login to Weights and Biases, and paste an API key into VSCode. After this is done, your Weights and Biases training run will start. It'll give you a lot of output text, one line of which will look like:\n",
        "\n",
        "```\n",
        "View run at https://wandb.ai/<USERNAME>/<PROJECT-NAME>/runs/<RUN-NAME>\n",
        "```\n",
        "\n",
        "which you can click on to visit the run page.\n",
        "\n",
        "> Note - to see the plots more clearly in Weights and Biases, you can click on the **edit panel** of your plot (the small pencil symbol at the top-right), then move the **smoothing** slider to the right."
      ],
      "metadata": {
        "id": "bypGwSMYn2Bl"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### A note on this loss curve (optional)\n",
        "\n",
        "\n",
        "What's up with the shape of our loss curve? It seems like we start at around 10-11, drops down very fast, but then levels out. It turns out, this is all to do with the kinds of algorithms the model learns during training.\n",
        "\n",
        "When it starts out, your model will be outputting random noise, which might look a lot like \"predict each token with approximately uniform probability\", i.e. $Q(x) = 1/d_\\text{vocab}$ for all $x$. This gives us a cross entropy loss of $\\log (d_\\text{vocab})$.\n",
        "\n"
      ],
      "metadata": {
        "id": "VLJL82W0oWzF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "d_vocab = model.cfg.d_vocab\n",
        "\n",
        "print(f\"d_vocab = {d_vocab}\")\n",
        "print(f\"Cross entropy loss on uniform distribution = {math.log(d_vocab)}\")"
      ],
      "metadata": {
        "id": "n0_NVezboOHw"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "The next thing we might expect the model to learn is the frequencies of words in the english language. After all, small common tokens like `\" and\"` or `\" the\"` might appear much more frequently than others. This would give us an average cross entropy loss of:\n",
        "\n",
        "$$\n",
        "- \\sum_x p_x \\log p_x\n",
        "$$\n",
        "\n",
        "where $p_x$ is the actual frequency of the word in our training data.\n",
        "\n",
        "We can evaluate this quantity as follows:"
      ],
      "metadata": {
        "id": "oLbVtVOYoQp5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "toks = tokenized_dataset[:][\"tokens\"].flatten()\n",
        "\n",
        "d_vocab = model.cfg.d_vocab\n",
        "freqs = t.bincount(toks, minlength=d_vocab)\n",
        "probs = freqs.float() / freqs.sum()\n",
        "\n",
        "distn = t.distributions.categorical.Categorical(probs=probs)\n",
        "entropy = distn.entropy()\n",
        "\n",
        "print(f\"Entropy of training data = {entropy}\")"
      ],
      "metadata": {
        "id": "PYLIco76oSeX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "After unigram frequencies, the next thing our model usually learns is **bigram frequencies** (i.e. the frequency of pairs of adjacent tokens in the training data). For instance, `\"I\"` and `\" am\"` are common tokens, but their bigram frequency is much higher than it would be if they occurred independently. Bigram frequencies actually take you pretty far, since they also help with:\n",
        "\n",
        "* Some simple grammatical rules (e.g. a full stop being followed by a capitalized word)\n",
        "* Weird quirks of tokenization (e.g. `\" manip\"` being followed by `\"ulative\"`)\n",
        "* Common names (e.g. `\"Barack\"` being followed by `\" Obama\"`)\n",
        "\n",
        "\n",
        "After approximating bigram frequencies, we need to start using smarter techniques, like trigrams (which can only be implemented using attention heads), **induction heads** (which we'll learn a lot more about in the next set of exercises!), and fact memorization or more basic grammar and syntax rules. Marginal improvement starts getting a lot harder around here, leading to a flattening of our loss curve."
      ],
      "metadata": {
        "id": "cQwiLxr6oUUq"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hrSd3-YnfHN9"
      },
      "source": [
        "# 4️⃣ Sampling from a Transformer\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LdezwBEFfHN9"
      },
      "source": [
        "> ##### Learning objectives\n",
        ">\n",
        "> * Learn how to sample from a transformer\n",
        ">     * This includes basic methods like greedy search or top-k, and more advanced methods like beam search\n",
        "> * Learn how to cache the output of a transformer, so that it can be used to generate text more efficiently\n",
        ">     * Optionally, rewrite your sampling functions to make use of your caching methods\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c7C6E0bFfHN9"
      },
      "source": [
        "One obvious method to sample tokens from a distribution would be to always take the token assigned the highest probability. But this can lead to some boring and repetitive outcomes, and at worst it can lock our transformer's output into a loop.\n",
        "\n",
        "First, you should read HuggingFace's blog post [How to generate text: using different decoding methods for language generation with Transformers](https://huggingface.co/blog/how-to-generate).\n",
        "\n",
        "Once you've done that, you can work through the `TransformerSampler` class below, and implement the different sampling methods. Each method will come with its own tests, and demo code for you to run.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aviyrTWefHN9"
      },
      "outputs": [],
      "source": [
        "model_cfg = Config()\n",
        "model = DemoTransformer(model_cfg).to(device)\n",
        "model.load_state_dict(reference_gpt2.state_dict(), strict=False)\n",
        "\n",
        "tokenizer = reference_gpt2.tokenizer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4p1szCj1fHN9"
      },
      "outputs": [],
      "source": [
        "class TransformerSampler:\n",
        "\n",
        "    def __init__(self, model: DemoTransformer, tokenizer: GPT2TokenizerFast):\n",
        "        self.model = model\n",
        "        self.cfg = model.cfg\n",
        "        self.tokenizer = tokenizer\n",
        "\n",
        "    @t.inference_mode()\n",
        "    def sample(self, prompt: str, max_tokens_generated=100, verbose=False, **kwargs):\n",
        "        '''\n",
        "        Returns a string of autoregressively generated text, starting from the prompt.\n",
        "\n",
        "        Sampling terminates at max_tokens_generated, or when the model generates an\n",
        "        end-of-sequence token.\n",
        "\n",
        "        kwargs are passed to sample_next_token, to give detailed instructions on how\n",
        "        new tokens are chosen.\n",
        "        '''\n",
        "        pass\n",
        "\n",
        "    @t.inference_mode()\n",
        "    def beam_search(\n",
        "        self,\n",
        "        prompt: str,\n",
        "        num_return_sequences: int,\n",
        "        num_beams: int,\n",
        "        max_new_tokens: int,\n",
        "        no_repeat_ngram_size: int = 0,\n",
        "        verbose=False\n",
        "    ) -> List[Tuple[float, t.Tensor]]:\n",
        "        '''\n",
        "        Returns a string of autoregressively generated text, starting from the prompt.\n",
        "\n",
        "        Sampling terminates at max_tokens_generated, or when the model generates an\n",
        "        end-of-sequence token.\n",
        "\n",
        "        kwargs are passed to sample_next_token, to give detailed instructions on how\n",
        "        new tokens are chosen.\n",
        "        '''\n",
        "        pass\n",
        "\n",
        "\n",
        "    @staticmethod\n",
        "    def sample_next_token(\n",
        "        input_ids: Int[Tensor, \"seq_len\"],\n",
        "        logits: Float[Tensor, \"seq_len d_vocab\"],\n",
        "        temperature=1.0,\n",
        "        top_k=0,\n",
        "        top_p=0.0,\n",
        "        frequency_penalty=0.0,\n",
        "        seed=None\n",
        "    ):\n",
        "        assert input_ids.ndim == 1, \"input_ids should be a 1D sequence of token ids\"\n",
        "        assert temperature >= 0, \"Temperature should be non-negative\"\n",
        "        assert 0 <= top_p <= 1.0, \"Top-p must be a probability\"\n",
        "        assert 0 <= top_k, \"Top-k must be non-negative\"\n",
        "        assert not (top_p != 0 and top_k != 0), \"At most one of top-p and top-k supported\"\n",
        "\n",
        "        # Set random seeds for reproducibility\n",
        "        if seed is not None:\n",
        "            t.manual_seed(seed)\n",
        "            np.random.seed(seed)\n",
        "\n",
        "        # Apply all the specialized sampling methods\n",
        "        if temperature == 0:\n",
        "            return TransformerSampler.greedy_search(logits)\n",
        "        elif temperature != 1.0:\n",
        "            logits = TransformerSampler.apply_temperature(logits, temperature)\n",
        "        if frequency_penalty != 0.0:\n",
        "            logits = TransformerSampler.apply_frequency_penalty(input_ids, logits, frequency_penalty)\n",
        "        if top_k > 0:\n",
        "            return TransformerSampler.sample_top_k(logits, top_k)\n",
        "        if top_p > 0.0:\n",
        "            return TransformerSampler.sample_top_p(logits, top_p)\n",
        "        return TransformerSampler.sample_basic(logits)\n",
        "\n",
        "\n",
        "    @staticmethod\n",
        "    def greedy_search(logits: Float[Tensor, \"d_vocab\"]) -> int:\n",
        "        '''\n",
        "        Returns the most likely token (as an int).\n",
        "        '''\n",
        "        out = logits.argmax().item()\n",
        "        return out\n",
        "\n",
        "\n",
        "    @staticmethod\n",
        "    def apply_temperature(logits: Float[Tensor, \"d_vocab\"], temperature: float) -> Float[Tensor, \"d_vocab\"]:\n",
        "        '''\n",
        "        Applies temperature scaling to the logits.\n",
        "        '''\n",
        "        pass\n",
        "\n",
        "    @staticmethod\n",
        "    def apply_frequency_penalty(input_ids: Int[Tensor, \"seq_len\"], logits: Float[Tensor, \"d_vocab\"], freq_penalty: float) -> Float[Tensor, \"d_vocab\"]:\n",
        "        '''\n",
        "        Applies a frequency penalty to the logits.\n",
        "        '''\n",
        "        pass\n",
        "\n",
        "    @staticmethod\n",
        "    def sample_basic(logits: Float[Tensor, \"d_vocab\"]) -> int:\n",
        "        '''\n",
        "        Samples from the distribution defined by the logits.\n",
        "        '''\n",
        "        pass\n",
        "\n",
        "    @staticmethod\n",
        "    def sample_top_k(logits: Float[Tensor, \"d_vocab\"], k: int) -> int:\n",
        "        '''\n",
        "        Samples from the top k most likely tokens.\n",
        "        '''\n",
        "        pass\n",
        "\n",
        "    @staticmethod\n",
        "    def sample_top_p(logits: Float[Tensor, \"d_vocab\"], top_p: float, min_tokens_to_keep: int = 1) -> int:\n",
        "        '''\n",
        "        Samples from the most likely tokens which make up at least p cumulative probability.\n",
        "        '''\n",
        "        pass"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lwA5ge9KfHN9"
      },
      "source": [
        "## Main Sampling Function\n",
        "\n",
        "The first thing you should do is implement the `sample` method.\n",
        "\n",
        "### Exercise - implement `sample`\n",
        "\n",
        "```c\n",
        "Difficulty: 🟠🟠🟠⚪⚪\n",
        "Importance: 🟠🟠🟠⚪⚪\n",
        "\n",
        "You should spend up to 20-25 minutes on this exercise.\n",
        "```\n",
        "\n",
        "This function takes in a prompt (in the form of a string), encodes it as a sequence of token ids using `self.tokenizer.encode`, and then continually generates new tokens by repeating the following steps:\n",
        "\n",
        "1. Passing the tokenized prompt through the model to get logits,\n",
        "2. Taking the logit vector corresponding to the last token in the prompt (i.e. the prediction for the *next* token),\n",
        "3. Sampling from this distribution to get a new token, using `self.sample_next_token(input_ids, logits, **kwargs)` (here, `kwargs` contains all the sampling-specific arguments, e.g. temperature, top-k, etc.),\n",
        "4. Appending this new token to the input tokens, and repeating the process until we meet one of two termination critera:\n",
        "    * We generate `max_tokens_generated` new tokens, or\n",
        "    * We generate the end-of-sequence token (which we can access via `self.tokenizer.eos_token_id`).\n",
        "\n",
        "Finally, we use `self.tokenizer.decode` to convert the generated token ids back into a string, and return this string.\n",
        "\n",
        "We also have a `verbose` argument - when this is true you can print your output while it's being sampled.\n",
        "\n",
        "Below is some code which tests your sampling function by performing greedy sampling (which means always choosing the most likely next token at each step).\n",
        "\n",
        "<details>\n",
        "<summary>Why does <code>temperature=0.0</code> correspond to greedy sampling?</summary>\n",
        "\n",
        "To apply a temperature to our sampling (as we'll see later) means to scale all logits by `(1 / temperature)`. The basic intuition here is:\n",
        "\n",
        "* A higher temperature means a smaller scale factor, so the logits all approach zero, i.e. uniform distribution, and the sampling process is a lot more random (producing more diverse and varied outputs)\n",
        "* A lower temperature means a larger scale factor, so the logits all approach infinity, i.e. a dirac delta function, and the sampling process is a lot more deterministic (producing less varied output)\n",
        "\n",
        "As temperature gets close to zero, the difference between the largest logit and second largest logit becomes very large, so the distribution tends to \"probability of 1 on the highest-likelihood token\", i.e. greedy sampling. You can derive this formally if you prefer.\n",
        "</details>\n",
        "\n",
        "A few hints:\n",
        "\n",
        "* Don't forget about tensor shapes! Your model's input should always have a batch dimension, i.e. it should be shape `(1, seq_len)`.\n",
        "* Also remember to have your tensors be on the same device (we have a global `device` variable).\n",
        "* Remember to put your model in evaluation mode, using `model.eval()`.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "loDmpBfmfHN9"
      },
      "outputs": [],
      "source": [
        "sampler = TransformerSampler(model, tokenizer)\n",
        "\n",
        "prompt = \"Jingle bells, jingle bells, jingle all the way\"\n",
        "print(f\"Greedy decoding with prompt: {prompt!r}\\n\")\n",
        "\n",
        "output = sampler.sample(prompt, max_tokens_generated=8, temperature=0.0)\n",
        "print(f\"Your model said: {output!r}\\n\")\n",
        "\n",
        "expected = \"Jingle bells, jingle bells, jingle all the way up to the top of the mountain.\"\n",
        "assert output == expected\n",
        "\n",
        "print(\"Tests passed!\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OBqvX5B_fHN-"
      },
      "source": [
        "<details>\n",
        "<summary>Solution</summary>\n",
        "\n",
        "```python\n",
        "@t.inference_mode()\n",
        "def sample(self, prompt, max_tokens_generated=100, verbose=False, **kwargs):\n",
        "    # SOLUTION\n",
        "    self.model.eval()\n",
        "    input_ids = self.tokenizer.encode(prompt, return_tensors=\"pt\").to(device)[0]\n",
        "\n",
        "    for i in range(max_tokens_generated):\n",
        "        # Get new logits (make sure we don't pass in more tokens than the model's context length)\n",
        "        logits = self.model(input_ids[None, -self.cfg.n_ctx:])\n",
        "        # We only take logits for the last token, because this is what we're sampling\n",
        "        logits = logits[0, -1]\n",
        "        # Get next token (as a tensor of size (1, 1) so we can concat it to input_ids)\n",
        "        next_token = t.tensor([self.sample_next_token(input_ids, logits, **kwargs)], device=device)\n",
        "        # Create new input ids string, with shape (1, old_seq_len + 1)\n",
        "        input_ids = t.cat([input_ids, next_token], dim=-1)\n",
        "        # Print out results, if required\n",
        "        if verbose:\n",
        "            print(self.tokenizer.decode(input_ids), end=\"\\r\")\n",
        "        # If our new token was the end-of-text token, stop\n",
        "        if next_token == getattr(self.tokenizer, \"eos_token_id\", None):\n",
        "            break\n",
        "    \n",
        "    return self.tokenizer.decode(input_ids)\n",
        "```\n",
        "```python\n",
        "class TransformerSampler:\n",
        "\n",
        "    def __init__(self, model: DemoTransformer, tokenizer: GPT2TokenizerFast):\n",
        "        self.model = model\n",
        "        self.cfg = model.cfg\n",
        "        self.tokenizer = tokenizer\n",
        "\n",
        "    @t.inference_mode()\n",
        "    def sample(self, prompt: str, max_tokens_generated=100, verbose=False, **kwargs):\n",
        "        '''\n",
        "        Returns a string of autoregressively generated text, starting from the prompt.\n",
        "\n",
        "        Sampling terminates at max_tokens_generated, or when the model generates an\n",
        "        end-of-sequence token.\n",
        "\n",
        "        kwargs are passed to sample_next_token, to give detailed instructions on how\n",
        "        new tokens are chosen.\n",
        "        '''\n",
        "        # SOLUTION\n",
        "        self.model.eval()\n",
        "        input_ids = self.tokenizer.encode(prompt, return_tensors=\"pt\").to(device)[0]\n",
        "\n",
        "        for i in range(max_tokens_generated):\n",
        "            # Get new logits (make sure we don't pass in more tokens than the model's context length)\n",
        "            logits = self.model(input_ids[None, -self.cfg.n_ctx:])\n",
        "            # We only take logits for the last token, because this is what we're sampling\n",
        "            logits = logits[0, -1]\n",
        "            # Get next token (as a tensor of size (1, 1) so we can concat it to input_ids)\n",
        "            next_token = t.tensor([TransformerSampler.sample_next_token(input_ids, logits, **kwargs)], device=device)\n",
        "            # Create new input ids string, with shape (1, old_seq_len + 1)\n",
        "            input_ids = t.cat([input_ids, next_token], dim=-1)\n",
        "            # Print out results, if required\n",
        "            if verbose:\n",
        "                print(self.tokenizer.decode(input_ids), end=\"\\r\")\n",
        "            # If our new token was the end-of-text token, stop\n",
        "            if next_token == getattr(self.tokenizer, \"eos_token_id\", None):\n",
        "                break\n",
        "        \n",
        "        return self.tokenizer.decode(input_ids)\n",
        "    \n",
        "    \n",
        "    \n",
        "    @t.inference_mode()\n",
        "    def beam_search(\n",
        "        self,\n",
        "        prompt: str,\n",
        "        num_return_sequences: int,\n",
        "        num_beams: int,\n",
        "        max_new_tokens: int,\n",
        "        no_repeat_ngram_size: int = 0,\n",
        "        verbose=False\n",
        "    ) -> List[Tuple[float, t.Tensor]]:\n",
        "        '''\n",
        "        Returns a string of autoregressively generated text, starting from the prompt.\n",
        "\n",
        "        Sampling terminates at max_tokens_generated, or when the model generates an\n",
        "        end-of-sequence token.\n",
        "\n",
        "        kwargs are passed to sample_next_token, to give detailed instructions on how\n",
        "        new tokens are chosen.\n",
        "        '''\n",
        "        pass\n",
        "    \n",
        "\n",
        "    @staticmethod\n",
        "    def sample_next_token(\n",
        "        input_ids: Int[Tensor, \"seq_len\"],\n",
        "        logits: Float[Tensor, \"seq_len d_vocab\"],\n",
        "        temperature=1.0,\n",
        "        top_k=0,\n",
        "        top_p=0.0,\n",
        "        frequency_penalty=0.0,\n",
        "        seed=None\n",
        "    ):\n",
        "        assert input_ids.ndim == 1, \"input_ids should be a 1D sequence of token ids\"\n",
        "        assert temperature >= 0, \"Temperature should be non-negative\"\n",
        "        assert 0 <= top_p <= 1.0, \"Top-p must be a probability\"\n",
        "        assert 0 <= top_k, \"Top-k must be non-negative\"\n",
        "        assert not (top_p != 0 and top_k != 0), \"At most one of top-p and top-k supported\"\n",
        "\n",
        "        # Set random seeds for reproducibility\n",
        "        if seed is not None:\n",
        "            t.manual_seed(seed)\n",
        "            np.random.seed(seed)\n",
        "\n",
        "        # Apply all the specialized sampling methods\n",
        "        if temperature == 0:\n",
        "            return TransformerSampler.greedy_search(logits)\n",
        "        elif temperature != 1.0:\n",
        "            logits = TransformerSampler.apply_temperature(logits, temperature)\n",
        "        if frequency_penalty != 0.0:\n",
        "            logits = TransformerSampler.apply_frequency_penalty(input_ids, logits, frequency_penalty)\n",
        "        if top_k > 0:\n",
        "            return TransformerSampler.sample_top_k(logits, top_k)\n",
        "        if top_p > 0.0:\n",
        "            return TransformerSampler.sample_top_p(logits, top_p)\n",
        "        return TransformerSampler.sample_basic(logits)\n",
        "\n",
        "\n",
        "    @staticmethod\n",
        "    def greedy_search(logits: Float[Tensor, \"d_vocab\"]) -> int:\n",
        "        '''\n",
        "        Returns the most likely token (as an int).\n",
        "        '''\n",
        "        out = logits.argmax().item()\n",
        "        return out\n",
        "    \n",
        "\n",
        "    @staticmethod\n",
        "    def apply_temperature(logits: Float[Tensor, \"d_vocab\"], temperature: float) -> Float[Tensor, \"d_vocab\"]:\n",
        "        '''\n",
        "        Applies temperature scaling to the logits.\n",
        "        '''\n",
        "        # SOLUTION\n",
        "        return logits / temperature\n",
        "    \n",
        "\n",
        "\n",
        "    @staticmethod\n",
        "    def apply_frequency_penalty(input_ids: Int[Tensor, \"seq_len\"], logits: Float[Tensor, \"d_vocab\"], freq_penalty: float) -> Float[Tensor, \"d_vocab\"]:\n",
        "        '''\n",
        "        Applies a frequency penalty to the logits.\n",
        "        '''\n",
        "        # SOLUTION\n",
        "        d_vocab = logits.size(0)\n",
        "        id_freqs = t.bincount(input_ids, minlength=d_vocab)\n",
        "        return logits - freq_penalty * id_freqs\n",
        "    \n",
        "\n",
        "    \n",
        "    @staticmethod\n",
        "    def sample_basic(logits: Float[Tensor, \"d_vocab\"]) -> int:\n",
        "        '''\n",
        "        Samples from the distribution defined by the logits.\n",
        "        '''\n",
        "        # SOLUTION\n",
        "        sampled_token = t.distributions.categorical.Categorical(logits=logits).sample()\n",
        "        return sampled_token.item()\n",
        "    \n",
        "\n",
        "\n",
        "    @staticmethod\n",
        "    def sample_top_k(logits: Float[Tensor, \"d_vocab\"], k: int) -> int:\n",
        "        '''\n",
        "        Samples from the top k most likely tokens.\n",
        "        '''\n",
        "        # SOLUTION\n",
        "        top_k_logits, top_k_token_ids = logits.topk(k)\n",
        "        # Get sampled token (which is an index corresponding to the list of top-k tokens)\n",
        "        sampled_token_idx = t.distributions.categorical.Categorical(logits=top_k_logits).sample()\n",
        "        # Get the actual token id, as an int\n",
        "        return top_k_token_ids[sampled_token_idx].item()\n",
        "    \n",
        "\n",
        "\n",
        "    @staticmethod\n",
        "    def sample_top_p(logits: Float[Tensor, \"d_vocab\"], top_p: float, min_tokens_to_keep: int = 1) -> int:\n",
        "        '''\n",
        "        Samples from the most likely tokens which make up at least p cumulative probability.\n",
        "        '''\n",
        "        # SOLUTION\n",
        "        # Sort logits, and get cumulative probabilities\n",
        "        logits_sorted, indices = logits.sort(descending=True, stable=True)\n",
        "        cumul_probs = logits_sorted.softmax(-1).cumsum(-1)\n",
        "        # Choose which tokens to keep, in the set we sample from\n",
        "        n_keep = t.searchsorted(cumul_probs, top_p, side=\"left\").item() + 1\n",
        "        n_keep = max(n_keep, min_tokens_to_keep)\n",
        "        keep_idx = indices[:n_keep]\n",
        "        keep_logits = logits[keep_idx]\n",
        "        # Perform the sampling\n",
        "        sample = t.distributions.categorical.Categorical(logits=keep_logits).sample()\n",
        "        return keep_idx[sample].item()\n",
        "```\n",
        "</details>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aIx0p0MlfHN-"
      },
      "source": [
        "## Sampling with Categorical\n",
        "\n",
        "Now, we'll move into implementing specific sampling methods.\n",
        "\n",
        "PyTorch provides a [`distributions` package](https://pytorch.org/docs/stable/distributions.html#distribution) with a number of convenient methods for sampling from various distributions.\n",
        "\n",
        "For now, we just need [`t.distributions.categorical.Categorical`](https://pytorch.org/docs/stable/distributions.html#categorical). Use this to implement `sample_basic`, which just samples from the provided logits (which may have already been modified by the temperature and frequency penalties).\n",
        "\n",
        "Note that this will be slow since we aren't batching the samples, but don't worry about speed for now.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DLjEUGqZfHN-"
      },
      "source": [
        "### Exercise - Basic Sampling\n",
        "\n",
        "```c\n",
        "Difficulty: 🟠🟠⚪⚪⚪\n",
        "Importance: 🟠🟠⚪⚪⚪\n",
        "\n",
        "You should spend up to 5-10 minutes on this exercise.\n",
        "```\n",
        "\n",
        "Implement basic sampling in the `TransformerSampler` class above, then run the code below to verify your solution works.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cSA2Sr0FfHN-"
      },
      "outputs": [],
      "source": [
        "prompt = \"John and Mary went to the\"\n",
        "input_ids = tokenizer.encode(prompt, return_tensors=\"pt\").to(device)\n",
        "logits = model(input_ids)[0, -1]\n",
        "\n",
        "expected_top_5 = {\n",
        "    \" church\": 0.0648,\n",
        "    \" house\": 0.0367,\n",
        "    \" temple\": 0.0145,\n",
        "    \" same\": 0.0104,\n",
        "    \" Church\": 0.0097\n",
        "}\n",
        "frequency_of_top_5 = defaultdict(int)\n",
        "\n",
        "N = 10_000\n",
        "for _ in tqdm(range(N)):\n",
        "    token = TransformerSampler.sample_next_token(input_ids.squeeze(), logits)\n",
        "    frequency_of_top_5[tokenizer.decode(token)] += 1\n",
        "\n",
        "for word in expected_top_5:\n",
        "    expected_freq = expected_top_5[word]\n",
        "    observed_freq = frequency_of_top_5[word] / N\n",
        "    print(f\"Word: {word!r:<9}. Expected freq {expected_freq:.4f}, observed freq {observed_freq:.4f}\")\n",
        "    assert abs(observed_freq - expected_freq) < 0.01, \"Try increasing N if this fails by a small amount.\"\n",
        "\n",
        "print(\"Tests passed!\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "up_8JOclfHN-"
      },
      "source": [
        "<details>\n",
        "<summary>Solution</summary>\n",
        "\n",
        "```python\n",
        "def sample_basic(logits: t.Tensor) -> int:\n",
        "    '''\n",
        "    logits: shape (vocab_size, ) - unnormalized log-probabilities\n",
        "\n",
        "    Return: a sampled token\n",
        "    '''\n",
        "    distribution = t.distributions.categorical.Categorical(logits=logits)\n",
        "    out = distribution.sample().item()\n",
        "    assert isinstance(out, int)\n",
        "    return out\n",
        "```\n",
        "</details>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lXLHfTYzfHN-"
      },
      "source": [
        "### Exercise - Temperature\n",
        "\n",
        "```c\n",
        "Difficulty: 🟠⚪⚪⚪⚪\n",
        "Importance: 🟠🟠⚪⚪⚪\n",
        "\n",
        "You should spend up to 5-10 minutes on this exercise.\n",
        "```\n",
        "\n",
        "Temperature sounds fancy, but it's literally just dividing the logits by the temperature. You should implement this in your `TransformerSampler` class now.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wph7hgyMfHN-"
      },
      "outputs": [],
      "source": [
        "logits = t.tensor([1, 2]).log()\n",
        "\n",
        "cold_logits = TransformerSampler.apply_temperature(logits, temperature=0.001)\n",
        "print('A low temperature \"sharpens\" or \"peaks\" the distribution: ', cold_logits)\n",
        "t.testing.assert_close(cold_logits, 1000.0 * logits)\n",
        "\n",
        "hot_logits = TransformerSampler.apply_temperature(logits, temperature=1000.0)\n",
        "print(\"A high temperature flattens the distribution: \", hot_logits)\n",
        "t.testing.assert_close(hot_logits, 0.001 * logits)\n",
        "\n",
        "print(\"Tests passed!\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aSf7Ipi1fHN-"
      },
      "source": [
        "<details>\n",
        "<summary>Solution</summary>\n",
        "\n",
        "```python\n",
        "def apply_temperature(logits: t.Tensor, temperature: float) -> t.Tensor:\n",
        "    '''\n",
        "    logits: shape (vocab_size, )\n",
        "\n",
        "    Return: shape (vocab_size, )\n",
        "    '''\n",
        "    assert temperature > 0\n",
        "    return logits / temperature\n",
        "```\n",
        "</details>\n",
        "\n",
        "<details>\n",
        "<summary>Question - what is the limit of applying 'sample_basic' after adjusting with temperature, when temperature goes to zero? How about when temperature goes to infinity?</summary>\n",
        "\n",
        "The limit when temperature goes to zero is greedy search (because dividing by a small number makes the logits very big, in other words the difference between the maximum logit one and all the others will grow).\n",
        "\n",
        "The limit when temperature goes to infinity is uniform random sampling over all words (because all logits will be pushed towards zero).\")\n",
        "</details>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t8H7_bF-fHN_"
      },
      "source": [
        "### Exercise - Frequency Penalty\n",
        "\n",
        "```c\n",
        "Difficulty: 🟠🟠⚪⚪⚪\n",
        "Importance: 🟠⚪⚪⚪⚪\n",
        "\n",
        "You should spend up to 10-15 minutes on this exercise.\n",
        "```\n",
        "\n",
        "The frequency penalty is simple as well: count the number of occurrences of each token, then subtract `freq_penalty` for each occurrence. Hint: use `t.bincount` (documentation [here](https://pytorch.org/docs/stable/generated/torch.bincount.html)) to do this in a vectorized way.\n",
        "\n",
        "You should implement the `apply_frequency_penalty` method in your `TransformerSampler` class now, then run the cell below to check your solution.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6i5tF4StfHN_"
      },
      "source": [
        "<details>\n",
        "<summary>Help - I'm getting a <code>RuntimeError</code>; my tensor sizes don't match.</summary>\n",
        "\n",
        "Look at the documentation page for `t.bincount`. You might need to use the `minlength` argument - why?\n",
        "</details>\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "y9A7LdixfHN_"
      },
      "outputs": [],
      "source": [
        "bieber_prompt = \"And I was like Baby, baby, baby, oh Like, Baby, baby, baby, no Like, Baby, baby, baby, oh I thought you'd always be mine, mine\"\n",
        "input_ids = tokenizer.encode(bieber_prompt, return_tensors=\"pt\")\n",
        "logits = t.ones(tokenizer.vocab_size)\n",
        "penalized_logits = TransformerSampler.apply_frequency_penalty(input_ids.squeeze(), logits, 2.0)\n",
        "\n",
        "assert penalized_logits[5156].item() == -11, \"Expected 6 occurrences of ' baby' with leading space, 1-2*6=-11\"\n",
        "assert penalized_logits[14801].item() == -5, \"Expected 3 occurrences of ' Baby' with leading space, 1-2*3=-5\"\n",
        "\n",
        "print(\"Tests passed!\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BvmGa7tkfHN_"
      },
      "source": [
        "<details>\n",
        "<summary>Solution</summary>\n",
        "\n",
        "```python\n",
        "def apply_freq_penalty(input_ids: t.Tensor, logits: t.Tensor, freq_penalty: float) -> t.Tensor:\n",
        "    '''\n",
        "    input_ids: shape (seq, )\n",
        "    logits: shape (vocab_size, )\n",
        "    Return: shape (vocab_size, )\n",
        "    '''\n",
        "    (vocab_size,) = logits.shape\n",
        "    id_freqs = t.bincount(input_ids, minlength=vocab_size)\n",
        "    return logits - freq_penalty * id_freqs\n",
        "```\n",
        "</details>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PuNpoR3CfHN_"
      },
      "source": [
        "### Sampling - Manual Testing\n",
        "\n",
        "Run the below cell to get a sense for the `temperature` and `freq_penalty` arguments. Play with your own prompt and try other values.\n",
        "\n",
        "Note: your model can generate newlines or non-printing characters, so calling `print` on generated text sometimes looks awkward on screen. You can call `repr` on the string before printing to have the string escaped nicely.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "owibsBUjfHN_"
      },
      "outputs": [],
      "source": [
        "sampler = TransformerSampler(model, tokenizer)\n",
        "\n",
        "N_RUNS = 1\n",
        "your_prompt = \"Jingle bells, jingle bells, jingle all the way\"\n",
        "cases = [\n",
        "    (\"High freq penalty\", dict(frequency_penalty=100.0)),\n",
        "    (\"Negative freq penalty\", dict(frequency_penalty=-3.0)),\n",
        "    (\"Too hot!\", dict(temperature=2.0)),\n",
        "    (\"Pleasantly cool\", dict(temperature=0.7)),\n",
        "    (\"Pleasantly warm\", dict(temperature=0.9)),\n",
        "    (\"Too cold!\", dict(temperature=0.01)),\n",
        "]\n",
        "\n",
        "table = Table(\"Name\", \"Kwargs\", \"Output\", title=\"Sampling - Manual Testing\")\n",
        "\n",
        "for (name, kwargs) in cases:\n",
        "    for i in range(N_RUNS):\n",
        "        output = sampler.sample(your_prompt, max_tokens_generated=24, **kwargs)\n",
        "        table.add_row(name, repr(kwargs), repr(output) + \"\\n\")\n",
        "\n",
        "rprint(table)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v9fFSMW7fHN_"
      },
      "source": [
        "## Top-K Sampling\n",
        "\n",
        "Conceptually, the steps in top-k sampling are:\n",
        "- Find the `top_k` largest probabilities (you can use [`torch.topk`](https://pytorch.org/docs/stable/generated/torch.topk.html))\n",
        "- Set all other probabilities to zero\n",
        "- Normalize and sample\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R0hBDuz4fHN_"
      },
      "source": [
        "### Exercise - implement `sample_top_k`\n",
        "\n",
        "```c\n",
        "Difficulty: 🟠🟠⚪⚪⚪\n",
        "Importance: 🟠⚪⚪⚪⚪\n",
        "\n",
        "You should spend up to 5-10 minutes on this exercise.\n",
        "```\n",
        "\n",
        "Implement the method `sample_top_k` now. Your implementation should stay in log-space throughout (don't exponentiate to obtain probabilities). This means you don't actually need to worry about normalizing, because `Categorical` accepts unnormalised logits.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HocllGE0fHN_"
      },
      "outputs": [],
      "source": [
        "prompt = \"John and Mary went to the\"\n",
        "input_ids = tokenizer.encode(prompt, return_tensors=\"pt\").to(device)\n",
        "logits = model(input_ids)[0, -1]\n",
        "\n",
        "expected_top_5 = {\n",
        "    \" church\": 0.0648,\n",
        "    \" house\": 0.0367,\n",
        "    \" temple\": 0.0145,\n",
        "    \" same\": 0.0104,\n",
        "    \" Church\": 0.0097\n",
        "}\n",
        "topk_5_sum = sum(expected_top_5.values())\n",
        "\n",
        "observed_freqs = defaultdict(int)\n",
        "\n",
        "N = 10000\n",
        "for _ in tqdm(range(N)):\n",
        "    token = TransformerSampler.sample_next_token(input_ids.squeeze(), logits, top_k=5)\n",
        "    observed_freqs[tokenizer.decode(token)] += 1\n",
        "\n",
        "for word in expected_top_5:\n",
        "    expected_freq = expected_top_5[word] / topk_5_sum\n",
        "    observed_freq = observed_freqs[word] / N\n",
        "    print(f\"Word: {word!r:<9}. Expected freq = {expected_freq:.4f}, observed freq = {observed_freq:.4f}\")\n",
        "    assert abs(observed_freq - expected_freq) < 0.015, \"Try increasing N if this fails by a small amount.\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Av9U35cSfHOA"
      },
      "source": [
        "<details>\n",
        "<summary>Solution</summary>\n",
        "\n",
        "```python\n",
        "def sample_top_k(logits: t.Tensor, top_k: int) -> int:\n",
        "    '''\n",
        "    logits: shape (vocab_size, ) - unnormalized log-probabilities\n",
        "    top_k: only consider this many of the most likely tokens for sampling\n",
        "\n",
        "    Return: a sampled token\n",
        "    '''\n",
        "    top_logits, top_idx = t.topk(logits, top_k)\n",
        "    idx = t.distributions.categorical.Categorical(logits=top_logits).sample()\n",
        "    return top_idx[idx].item()\n",
        "```\n",
        "</details>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3HElusTNfHOA"
      },
      "source": [
        "### Top-K Sampling - Example\n",
        "\n",
        "The [GPT-2 paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) famously included an example prompt about unicorns. Now it's your turn to see just how cherry picked this example was.\n",
        "\n",
        "The paper claims they used `top_k=40` and best of 10 samples.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "24dChHZ1fHOA"
      },
      "outputs": [],
      "source": [
        "sampler = TransformerSampler(model, tokenizer)\n",
        "\n",
        "your_prompt = \"In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.\"\n",
        "output = sampler.sample(your_prompt, temperature=0.7, top_k=40, max_tokens_generated=64)\n",
        "rprint(f\"Your model said:\\n\\n[bold dark_orange]{output}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kEmLBEn9fHOA"
      },
      "source": [
        "## Top-p aka Nucleus Sampling\n",
        "\n",
        "The basic idea is that we choose the most likely words, up until the total probability of words we've chosen crosses some threshold. Then we sample from those chosen words based on their logits.\n",
        "\n",
        "The steps are:\n",
        "\n",
        "- Sort the probabilities from largest to smallest\n",
        "- Find the cutoff point where the cumulative probability first equals or exceeds `top_p`. We do the cutoff inclusively, keeping the first probability above the threshold.\n",
        "- If the number of kept probabilities is less than `min_tokens_to_keep`, keep that many tokens instead.\n",
        "- Set all other probabilities to zero\n",
        "- Normalize and sample\n",
        "\n",
        "Optionally, refer to the paper [The Curious Case of Neural Text Degeneration](https://arxiv.org/pdf/1904.09751.pdf) for some comparison of different methods.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fAJaTvcefHOA"
      },
      "source": [
        "### Exercise - implement `sample_top_p`\n",
        "\n",
        "```c\n",
        "Difficulty: 🟠🟠🟠⚪⚪\n",
        "Importance: 🟠⚪⚪⚪⚪\n",
        "\n",
        "You should spend up to 15-20 minutes on this exercise.\n",
        "```\n",
        "\n",
        "<details>\n",
        "<summary>Example of top-p sampling (if you're confused)</summary>\n",
        "\n",
        "If our probabilities were `(0.4, 0.3, 0.2, 0.1)` and our cutoff was `top_p=0.8`, then we'd sample from the first three elements (because their total probability is `0.9` which is over the threshold, but the first two only have a total prob of `0.7` which is under the threshold). Once we've chosen to sample from those three, we would renormalise them by dividing by their sum (so the probabilities we use when sampling are `(4/9, 3/9, 2/9)`.\n",
        "</details>\n",
        "\n",
        "<details>\n",
        "<summary>Help - I'm stuck on how to implement this function.</summary>\n",
        "\n",
        "First, sort the logits using the `sort(descending=True)` method (this returns values and indices). Then you can get `cumulative_probs` by applying softmax to these logits and taking the cumsum. Then, you can decide how many probabilities to keep by using the `t.searchsorted` function.\n",
        "    \n",
        "Once you've decided which probabilities to keep, it's easiest to sample from them using the original logits (you should have preserved the indices when you called `logits.sort`). This way, you don't need to worry about renormalising like you would if you were using probabilities.\n",
        "</details>\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ndUIZuRJfHOA"
      },
      "outputs": [],
      "source": [
        "prompt = \"John and Mary went to the\"\n",
        "input_ids = tokenizer.encode(prompt, return_tensors=\"pt\").to(device)\n",
        "logits = model(input_ids)[0, -1]\n",
        "\n",
        "expected_top_10pct = {\n",
        "    \" church\": 0.0648,\n",
        "    \" house\": 0.0367, # These are the two most likely tokens, and add up to >10%\n",
        "}\n",
        "top_10pct_sum = sum(expected_top_10pct.values())\n",
        "\n",
        "observed_freqs = defaultdict(int)\n",
        "\n",
        "N = 10000\n",
        "for _ in tqdm(range(N)):\n",
        "    token = TransformerSampler.sample_next_token(input_ids.squeeze(), logits, top_p=0.1)\n",
        "    observed_freqs[tokenizer.decode(token)] += 1\n",
        "\n",
        "for word in expected_top_10pct:\n",
        "    expected_freq = expected_top_10pct[word] / top_10pct_sum\n",
        "    observed_freq = observed_freqs[word] / N\n",
        "    print(f\"Word: {word!r:<9}. Expected freq {expected_freq:.4f}, observed freq {observed_freq:.4f}\")\n",
        "    assert abs(observed_freq - expected_freq) < 0.01, \"Try increasing N if this fails by a small amount.\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M8IRcmg2fHOA"
      },
      "source": [
        "<details>\n",
        "<summary>Solution</summary>\n",
        "\n",
        "```python\n",
        "def sample_top_p(logits: t.Tensor, top_p: float, min_tokens_to_keep: int = 1) -> int:\n",
        "    '''\n",
        "    logits: shape (vocab_size, ) - unnormalized log-probabilities\n",
        "    Return: a sampled token\n",
        "    '''\n",
        "    logits_sorted, indices = logits.sort(descending=True, stable=True)\n",
        "    cumul_probs = logits_sorted.softmax(-1).cumsum(-1)\n",
        "    n_keep = t.searchsorted(cumul_probs, top_p, side=\"right\").item() + 1\n",
        "    n_keep = max(n_keep, min_tokens_to_keep)\n",
        "    keep_idx = indices[:n_keep]\n",
        "    keep_logits = logits[keep_idx]\n",
        "    sample = t.distributions.categorical.Categorical(logits=keep_logits).sample()\n",
        "    return keep_idx[sample].item()\n",
        "```\n",
        "</details>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qhrTvZp6fHOA"
      },
      "source": [
        "### Top-p Sampling - Example\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vCBMUxzufHOE"
      },
      "outputs": [],
      "source": [
        "sampler = TransformerSampler(model, tokenizer)\n",
        "\n",
        "your_prompt = \"Eliezer Shlomo Yudkowsky (born September 11, 1979) is an American decision and artificial intelligence (AI) theorist and writer, best known for\"\n",
        "output = sampler.sample(your_prompt, temperature=0.7, top_p=0.95, max_tokens_generated=64)\n",
        "rprint(f\"Your model said:\\n\\n[bold dark_orange]{output}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a6PVv6PHfHOE"
      },
      "source": [
        "## Beam search\n",
        "\n",
        "Finally, we'll implement a more advanced way of searching over output: **beam search**. You should read the [HuggingFace page](https://huggingface.co/blog/how-to-generate#beam-search) on beam search before moving on.\n",
        "\n",
        "In beam search, we maintain a list of size `num_beams` completions which are the most likely completions so far as measured by the product of their probabilities. Since this product can become very small, we use the sum of log probabilities instead. Note - log probabilities are *not* the same as your model's output. We get log probabilities by first taking softmax of our output and then taking log. You can do this with the [`log_softmax`](https://pytorch.org/docs/stable/generated/torch.nn.functional.log_softmax.html) function / tensor method.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jsRXQwecfHOE"
      },
      "source": [
        "<details>\n",
        "<summary>Log probabilities are equal to the logit output after being translated by some amount X (where X is a function of the original logit output). Can you prove this?</summary>\n",
        "\n",
        "Suppose our vector of logits is $x$, and we take softmax to get a vector of probabilities $p$, then log again to get a vector of log probabilities $l$. Then the $i$-th element of this vector of logprobs is:\n",
        "\n",
        "$$\n",
        "\\begin{align}\n",
        "l_i &= \\log p_i \\\\\n",
        "&= \\log \\frac{\\exp(x_i)}{\\sum_j \\exp(x_j)} \\\\\n",
        "&= x_i - \\log \\sum_j \\exp(x_j) \\\\\n",
        "&= x_i - C\n",
        "\\end{align}\n",
        "$$\n",
        "\n",
        "where $C = \\log \\sum_j \\exp(x_j)$ is the same for all elements. So we can see that $l_i$ is equal to the logit output $x_i$ after being translated by $C$.\n",
        "\n",
        "It's important not to mix up logits and logprobs!\n",
        "</details>\n",
        "\n",
        "<details>\n",
        "<summary>Why do you think we use log softmax rather than logit output?</summary>\n",
        "\n",
        "Logit output is translation invariant. If we had two different beams and we were generating the next tokens in those beams, there would be no reasonable way to compare the two beams to each other, because we could shift the logit vector for one beam by a constant amount without changing the distribution.\n",
        "\n",
        "</details>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-dQFaVpFfHOE"
      },
      "source": [
        "At each iteration, we run the batch of completions through the model and take the log-softmax to obtain `d_vocab` log-probs for each completion, or `num_beams * d_vocab` possible next completions in total.\n",
        "\n",
        "If we kept all of these, then we would have `num_beams * d_vocab * d_vocab` completions after the next iteration which is way too many, so instead we sort them by their score and loop through from best (highest) log probability to worst (lowest).\n",
        "\n",
        "The illustration below might help (based on real results from this method). Here, we have the following hyperparameters:\n",
        "\n",
        "```python\n",
        "num_beams = 3\n",
        "max_new_tokens = 3\n",
        "num_return_sequences = 2\n",
        "```\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JhmYgrg4fHOE"
      },
      "source": [
        "<img src=\"https://raw.githubusercontent.com/callummcdougall/computational-thread-art/master/example_images/misc/beam-search.png\" width=\"1000\">\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZnZQieUBfHOF"
      },
      "source": [
        "Note how after each \"generate\" stage, we have `num_beams ** 2` possible completions, which we then filter down to `num_beams`. Can you see why we need to generate this many (and what might happen if we generated fewer)?\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jn-kjeh8fHOF"
      },
      "source": [
        "How do we deal with sequences that terminate early (i.e. by generating an EOS token)? Answer - we append them to the list of completions which we'll return at the end, and remove them from the generation tree. Our algorithm terminates when either all our sequences have length `max_new_tokens` larger than the initial prompt length, or we've generated `num_returns_sequences` terminating sequences.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZFiBucEHfHOF"
      },
      "source": [
        "### Exercise - implement `beam_search`\n",
        "\n",
        "```c\n",
        "Difficulty: 🟠🟠🟠🟠⚪\n",
        "Importance: 🟠⚪⚪⚪⚪\n",
        "\n",
        "You should spend up to 30-40 minutes on this exercise.\n",
        "```\n",
        "\n",
        "You should now complete the `beam_search` method in your `TransformerSampler` class.\n",
        "\n",
        "We've provided one possible template for you to use: the class `Beams`, with important methods `generate` and `filter` for you to fit in (which correspond to the two stages in the diagram above). There are also a few of helper functions in this class:\n",
        "\n",
        "* `new_beams`, which creates a new `Beams` object from an old one.\n",
        "* `__getitem__`, which allows you to index into a `Beams` object to get a specific batch of beams.\n",
        "* `logprobs_and_completions`, which turns a `Beams` object into a list of (logprob sum, string completion) tuples (useful for getting your final output).\n",
        "* `print`, which prints out the current state of the beams (useful for debugging, if you run `beam_search` with `verbose=True`).\n",
        "\n",
        "You can then fill in the `beam_search` function, using this class and its methods.\n",
        "\n",
        "We've provided unit tests for the `generate` and `filter` functions, so you can verify that these are correct before moving on to the full `beam_search` function.\n",
        "\n",
        "**Note that using the `Beams` class is not strictly necessary, you could fill in the `beam_search` function directly if you prefer.** The `Beams` class is just meant to provide one example way you might implement this function. Often, modular code like this is easier to write and debug, and easier to extend to cover new use cases (e.g. when we use caching in the bonus exercises).\n",
        "\n",
        "#### Why all the n-gram repetition?\n",
        "\n",
        "You should observe that, while the output of beam search is sometimes more fluent than some of the other sampling methods you implement, it also has an unfortunate tendency to repeat sentences or sequences. This makes sense - if the model produces a sentence with a relatively high logit sum, then it will want to produce the same sentence again even if it doesn't make a lot of sense in context.\n",
        "\n",
        "A common solution is to ban repetition of n-grams. We've provided the argument `no_repeat_ngram_size` in the `generate` method for this purpose. Using this argument should prevent the model from repeating any n-grams of that size. Good values of this parameter to try are 2 or 3.\n",
        "\n",
        "However, first you should focus on getting a working version of beam search *without* using this argument.\n",
        "\n",
        "<details>\n",
        "<summary>Hint (for <code>no_repeat_ngram_size</code>)</summary>\n",
        "\n",
        "It might be helpful to implement the following method first. You can use this rather than `torch.topk` in your `generate` method.\n",
        "\n",
        "```python\n",
        "def get_topk_non_repeating(\n",
        "    self,\n",
        "    logprobs: Float[Tensor, \"batch d_vocab\"],\n",
        "    no_repeat_ngram_size: int,\n",
        "    k: int,\n",
        ") -> Tuple[Float[Tensor, \"k\"], Int[Tensor, \"k\"]]:\n",
        "    '''\n",
        "    logprobs:\n",
        "        tensor of the log-probs for the next token\n",
        "    no_repeat_ngram_size:\n",
        "        size of ngram to avoid repeating\n",
        "    k:\n",
        "        number of top logits to return, for each beam in our collection\n",
        "\n",
        "    Returns:\n",
        "        equivalent to the output of `logprobs.topk(dim=-1)`, but makes sure\n",
        "        that no returned tokens would produce an ngram of size  `no_repeat_ngram_size`\n",
        "        which has already appeared in `self.tokens`.\n",
        "    '''\n",
        "    pass\n",
        "```\n",
        "</details>\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-RhHI8AyfHOF"
      },
      "outputs": [],
      "source": [
        "@dataclass\n",
        "class Beams:\n",
        "    '''Class to store beams during beam search.'''\n",
        "    model: DemoTransformer\n",
        "    tokenizer: GPT2TokenizerFast\n",
        "    logprob_sums: Float[Tensor, \"batch\"]\n",
        "    tokens: Int[Tensor, \"batch seq\"]\n",
        "\n",
        "    def new_beams(self, logprob_sums, tokens) -> \"Beams\":\n",
        "        '''Creates a new Beams object with the same model and tokenizer.'''\n",
        "        return Beams(self.model, self.tokenizer, logprob_sums, tokens)\n",
        "\n",
        "    def __getitem__(self, idx) -> \"Beams\":\n",
        "        '''Allows you to take a slice of the beams object along the batch dimension.'''\n",
        "        return self.new_beams(self.logprob_sums[idx], self.tokens[idx])\n",
        "\n",
        "    @property\n",
        "    def logprobs_and_completions(self) -> List[Tuple[float, str]]:\n",
        "        '''Returns self as a list of logprob sums and completions (useful for getting final output).'''\n",
        "        return [\n",
        "            (logprob_sum.item(), self.tokenizer.decode(tokens))\n",
        "            for (logprob_sum, tokens) in zip(self.logprob_sums, self.tokens)\n",
        "        ]\n",
        "\n",
        "\n",
        "    def generate(self, toks_per_beam: int, no_repeat_ngram_size: Optional[int] = None) -> \"Beams\":\n",
        "        '''\n",
        "        Starting from the current set of beams (which has length `num_beams`), returns a new\n",
        "        set of `num_beams * toks_per_beam`, containing the best `toks_per_beam` continuations for each\n",
        "        of the original beams.\n",
        "\n",
        "        Optional argument `no_repeat_ngram_size` means your model won't generate any sequences with\n",
        "        a repeating n-gram of this length.\n",
        "        '''\n",
        "        pass\n",
        "\n",
        "    def filter(self, num_beams: int) -> Tuple[\"Beams\", \"Beams\"]:\n",
        "        '''\n",
        "        Returns:\n",
        "            best_beams: Beams\n",
        "                filtered version of self, containing all best `num_beams` which are also not terminated.\n",
        "\n",
        "            early_terminations: Beams\n",
        "                filtered version of self, containing all best `num_beams` which are also terminated.\n",
        "                i.e. the sum of lengths of these two should equal `num_beams`.\n",
        "        '''\n",
        "        pass\n",
        "\n",
        "    def print(self, title=\"Best completions\", max_print_chars=80) -> None:\n",
        "        '''\n",
        "        Prints out a set of sequences with their corresponding logitsums.\n",
        "        '''\n",
        "        if len(self.tokens) == 0:\n",
        "            return\n",
        "        table = Table(\"logitsum\", \"completion\", title=title)\n",
        "        for logprob_sum, tokens in zip(self.logprob_sums, self.tokens):\n",
        "            text = self.tokenizer.decode(tokens)\n",
        "            if len(repr(text)) > max_print_chars:\n",
        "                text = text[:int(0.3 * max_print_chars)] + \" ... \" + text[-int(0.7 * max_print_chars):]\n",
        "            table.add_row(f\"{logprob_sum:>8.3f}\", repr(text))\n",
        "        rprint(table)\n",
        "\n",
        "\n",
        "    def get_topk_non_repeating(\n",
        "        self,\n",
        "        logprobs: Float[Tensor, \"batch d_vocab\"],\n",
        "        no_repeat_ngram_size: Optional[int],\n",
        "        k: int,\n",
        "    ) -> Tuple[Float[Tensor, \"k\"], Int[Tensor, \"k\"]]:\n",
        "        '''\n",
        "        logprobs:\n",
        "            tensor of the log-probs for the next token\n",
        "        no_repeat_ngram_size:\n",
        "            size of ngram to avoid repeating\n",
        "        k:\n",
        "            number of top logits to return, for each beam in our collection\n",
        "\n",
        "        Returns:\n",
        "            equivalent to the output of `logprobs.topk(dim=-1)`, but makes sure\n",
        "            that no returned tokens would produce an ngram of size  `no_repeat_ngram_size`\n",
        "            which has already appeared in `self.tokens`.\n",
        "        '''\n",
        "        batch, seq_len = self.tokens.shape\n",
        "        neg_inf = t.tensor(-1.0e4).to(device)\n",
        "\n",
        "        # If completion isn't long enough for a repetition, or we have no restructions, just return topk\n",
        "        if (no_repeat_ngram_size is not None) and (seq_len > no_repeat_ngram_size-1):\n",
        "            # Otherwise, we need to check for ngram repetitions\n",
        "            # First, get the most recent `no_repeat_ngram_size-1` tokens\n",
        "            last_ngram_prefix = self.tokens[:, seq_len - (no_repeat_ngram_size-1):]\n",
        "            # Next, find all the tokens we're not allowed to generate (by going iterating through past ngrams and seeing if those ngram prefixes match the last one)\n",
        "            for i in range(seq_len - (no_repeat_ngram_size-1)):\n",
        "                ngrams = self.tokens[:, i:i+no_repeat_ngram_size] # (batch, ngram)\n",
        "                ngrams_are_repeated = (ngrams[:, :-1] == last_ngram_prefix).all(-1) # (batch,)\n",
        "                ngram_end_tokens = ngrams[:, [-1]] # (batch, 1)\n",
        "                # Fill logprobs with neginf wherever the ngrams are repeated\n",
        "                logprobs[range(batch), ngram_end_tokens] = t.where(\n",
        "                    ngrams_are_repeated,\n",
        "                    neg_inf,\n",
        "                    logprobs[range(batch), ngram_end_tokens],\n",
        "            )\n",
        "\n",
        "        # Finally, get our actual tokens\n",
        "        return logprobs.topk(k=k, dim=-1)\n",
        "\n",
        "\n",
        "@t.inference_mode()\n",
        "def beam_search(\n",
        "    self: TransformerSampler,\n",
        "    prompt: str,\n",
        "    num_return_sequences: int,\n",
        "    num_beams: int,\n",
        "    max_new_tokens: int,\n",
        "    no_repeat_ngram_size: Optional[int] = None,\n",
        "    verbose=False\n",
        ") -> List[Tuple[float, Tensor]]:\n",
        "    '''\n",
        "    Implements a beam search, by repeatedly performing the `generate` and `filter` steps (starting\n",
        "    from the initial prompt) until either of the two stopping criteria are met:\n",
        "\n",
        "        (1) we've generated `max_new_tokens` tokens, or\n",
        "        (2) we've generated `num_returns_sequences` terminating sequences.\n",
        "\n",
        "    To modularize this function, most of the actual complexity is in the Beams class,\n",
        "    in the `generate` and `filter` methods.\n",
        "    '''\n",
        "\n",
        "    assert num_return_sequences <= num_beams\n",
        "    self.model.eval()\n",
        "\n",
        "        pass"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EYDnDhhAfHOF"
      },
      "source": [
        "Example usage of the `Beams` class, and the `print` method (not the logitsums aren't necessarily accurate, this example is just an illustration):\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_dIOf4f9fHOF"
      },
      "outputs": [],
      "source": [
        "beams = Beams(\n",
        "    model,\n",
        "    tokenizer,\n",
        "    logprob_sums = t.tensor([-10.0, -15.0, -20.0]).to(device),\n",
        "    tokens = t.tensor([\n",
        "        [5661, 318, 262, 2368],\n",
        "        [5661, 318, 262, 1218],\n",
        "        [5661, 318, 262, 717],\n",
        "    ]).to(device)\n",
        ")\n",
        "\n",
        "beams.print()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O7ndBImyfHOF"
      },
      "source": [
        "And here are some unit tests for your `generate` and `filter` methods:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "BxrABZUzfHOF"
      },
      "outputs": [],
      "source": [
        "print(\"Testing generate, without no_repeat_ngram_size argument:\")\n",
        "new_beams = beams.generate(toks_per_beam=2)\n",
        "new_beams.print()\n",
        "assert new_beams.logprobs_and_completions[0][1] == \"this is the third time\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QD1Q7u74fHOG"
      },
      "outputs": [],
      "source": [
        "print(\"Testing generate, with no_repeat_ngram_size argument:\")\n",
        "\n",
        "bigram_beams = Beams(\n",
        "    model,\n",
        "    tokenizer,\n",
        "    logprob_sums = t.tensor([-0.0]).to(device),\n",
        "    tokens = t.tensor([[530, 734, 530, 734]]).to(device)\n",
        "    # tokens are \" one two one two\"\n",
        ")\n",
        "\n",
        "# With no_repeat_ngram_size=1, should not generate the token \" one\" or \" two\"\n",
        "new_bigram_beams = bigram_beams.generate(toks_per_beam=3, no_repeat_ngram_size=1)\n",
        "new_bigram_beams.print()\n",
        "assert all([not (completion[1].endswith(\" one\") or completion[1].endswith(\" two\")) for completion in new_bigram_beams.logprobs_and_completions])\n",
        "\n",
        "# With no_repeat_ngram_size=2, it can generate \" two\" (which it should), but not \" one\"\n",
        "new_bigram_beams = bigram_beams.generate(toks_per_beam=3, no_repeat_ngram_size=2)\n",
        "new_bigram_beams.print()\n",
        "assert all([not completion[1].endswith(\" one\") for completion in new_bigram_beams.logprobs_and_completions])\n",
        "assert any([not completion[1].endswith(\" two\") for completion in new_bigram_beams.logprobs_and_completions])\n",
        "\n",
        "print(\"All tests for `generate` passed!\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "U0NG1ZaqfHOG"
      },
      "outputs": [],
      "source": [
        "logprob_sums = t.tensor([-1.0, -2.0]).to(device)\n",
        "tokens = t.tensor([\n",
        "    [19485, 13],\n",
        "    [19485, tokenizer.eos_token_id]\n",
        "]).to(device)\n",
        "\n",
        "beams_with_eos = Beams(model, tokenizer, logprob_sums, tokens)\n",
        "best_beams, early_terminations = beams_with_eos.filter(2)\n",
        "\n",
        "t.testing.assert_close(best_beams.logprob_sums, logprob_sums[[0]])\n",
        "t.testing.assert_close(best_beams.tokens, tokens[[0]])\n",
        "\n",
        "assert early_terminations.logprobs_and_completions == [(-2.0, \"Stop\" + tokenizer.eos_token)]\n",
        "\n",
        "print(\"All tests for `filter` passed!\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Shua7jeTfHOG"
      },
      "source": [
        "<details>\n",
        "<summary>Solutions (for <code>generate</code> and <code>filter</code>)</summary>\n",
        "\n",
        "```python\n",
        "def generate(self, toks_per_beam: int, no_repeat_ngram_size: Optional[int] = None) -> \"Beams\":\n",
        "    '''\n",
        "    Starting from the current set of beams (which has length `num_beams`), returns a new\n",
        "    set of `num_beams * toks_per_beam`, containing the best `toks_per_beam` continuations for each\n",
        "    of the original beams.\n",
        "\n",
        "    Optional argument `no_repeat_ngram_size` means your model won't generate any sequences with\n",
        "    a repeating n-gram of this length (don't worry about implementing this until later).\n",
        "    '''\n",
        "    # SOLUTION\n",
        "\n",
        "    # Get the output logprobs for the next token (for every sequence in current beams)\n",
        "    logprobs: Tensor = self.model(self.tokens)[:, -1, :].log_softmax(-1)\n",
        "\n",
        "    # Get the top `toks_per_beam` tokens for each sequence\n",
        "    topk_logprobs, topk_tokenIDs = logprobs.topk(k=toks_per_beam)\n",
        "\n",
        "    # Get all of the new possible beams, via einops operations\n",
        "    #   Here, we're effectively flattening out the batch dimension and k dimension, to give us tensors\n",
        "    #   with every possible combination of (original sequence, new token) pairs.)\n",
        "    new_logprob_sums = sum([\n",
        "        einops.repeat(self.logprob_sums, \"batch -> batch k\", k=toks_per_beam),\n",
        "        einops.rearrange(topk_logprobs, \"batch k -> (batch k)\")\n",
        "    ])\n",
        "    new_tokens = t.concat([\n",
        "        einops.repeat(self.tokens, \"batch seq -> (batch k) seq\", k=toks_per_beam),\n",
        "        einops.rearrange(topk_tokenIDs, \"batch k -> (batch k) 1\")\n",
        "    ], dim=-1)\n",
        "    return self.new_beams(new_logprob_sums, new_tokens)\n",
        "\n",
        "\n",
        "def filter(self, num_beams: int) -> Tuple[\"Beams\", \"Beams\"]:\n",
        "    '''\n",
        "    Returns:\n",
        "        best_beams: Beams\n",
        "            filtered version of self, containing all best `num_beams` which are also not terminated.\n",
        "\n",
        "        early_terminations: Beams\n",
        "            filtered version of self, containing all best `num_beams` which are also terminated.\n",
        "            i.e. the sum of lengths of these two should equal `num_beams`.\n",
        "    '''\n",
        "    # SOLUTION\n",
        "\n",
        "    # Get the indices of top `num_beams` beams\n",
        "    top_beam_indices = self.logprob_sums.topk(k=num_beams, dim=0).indices.tolist()\n",
        "    # Get the indices of terminated sequences\n",
        "    new_tokens = self.tokens[:, -1]\n",
        "    terminated_indices = t.nonzero(new_tokens == self.tokenizer.eos_token_id)\n",
        "\n",
        "    # Get the indices of the `num_beams` best sequences (some terminated, some not terminated)\n",
        "    best_continuing = [i for i in top_beam_indices if i not in terminated_indices]\n",
        "    best_terminated = [i for i in top_beam_indices if i in terminated_indices]\n",
        "\n",
        "    # Return the beam objects from these indices\n",
        "    best_beams_continuing = self.new_beams(self.logprob_sums[best_continuing], self.tokens[best_continuing])\n",
        "    best_beams_terminated = self.new_beams(self.logprob_sums[best_terminated], self.tokens[best_terminated])\n",
        "    return best_beams_continuing, best_beams_terminated\n",
        "```\n",
        "\n",
        "</details>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jrXHl0GsfHOG"
      },
      "source": [
        "Once you've passed both these unit tests, you can try implementing the full beam search function. It should create a `Beams` object from the initial prompt, and then repeatedly call `generate` and `filter` until the stopping criteria are met.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2Uno10GRfHOG"
      },
      "outputs": [],
      "source": [
        "TransformerSampler.beam_search = beam_search\n",
        "\n",
        "sampler = TransformerSampler(model, tokenizer)\n",
        "\n",
        "prompt = \"The ships hung in the sky in much the same way that\"\n",
        "orig_len = len(tokenizer.encode(prompt))\n",
        "\n",
        "final_logitsums_and_completions = sampler.beam_search(\n",
        "    prompt=prompt,\n",
        "    num_return_sequences=3,\n",
        "    num_beams=40,\n",
        "    max_new_tokens=60,\n",
        "    no_repeat_ngram_size=2,\n",
        "    verbose=False\n",
        ")\n",
        "\n",
        "# Print all the best output\n",
        "for logprob_sum, text in final_logitsums_and_completions:\n",
        "    avg_logprob_as_prob = t.tensor(logprob_sum / (len(tokenizer.encode(text)) - orig_len)).exp().item()\n",
        "    print(\"=\" * 25 + f\" Avg logprob (as probability) = {avg_logprob_as_prob:.3f} \" + \"=\" * 25)\n",
        "    rprint(\"Best output:\\n\\n[bold dark_orange]\" + text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "THgjSPShfHOG"
      },
      "source": [
        "<details>\n",
        "<summary>Solution (full)</summary>\n",
        "\n",
        "A solution for the class method `get_topk_non_repeating`:\n",
        "\n",
        "```python\n",
        "def get_topk_non_repeating(\n",
        "    self,\n",
        "    logprobs: Float[Tensor, \"batch d_vocab\"],\n",
        "    no_repeat_ngram_size: Optional[int],\n",
        "    k: int,\n",
        ") -> Tuple[Float[Tensor, \"k\"], Int[Tensor, \"k\"]]:\n",
        "    '''\n",
        "    logprobs:\n",
        "        tensor of the log-probs for the next token\n",
        "    no_repeat_ngram_size:\n",
        "        size of ngram to avoid repeating\n",
        "    k:\n",
        "        number of top logits to return, for each beam in our collection\n",
        "\n",
        "    Returns:\n",
        "        equivalent to the output of `logprobs.topk(dim=-1)`, but makes sure\n",
        "        that no returned tokens would produce an ngram of size  `no_repeat_ngram_size`\n",
        "        which has already appeared in `self.tokens`.\n",
        "    '''\n",
        "    batch, seq_len = self.tokens.shape\n",
        "    neg_inf = t.tensor(-1.0e4).to(device)\n",
        "\n",
        "    # If completion isn't long enough for a repetition, or we have no restructions, just return topk\n",
        "    if (no_repeat_ngram_size is not None) and (seq_len > no_repeat_ngram_size-1):\n",
        "        # Otherwise, we need to check for ngram repetitions\n",
        "        # First, get the most recent `no_repeat_ngram_size-1` tokens\n",
        "        last_ngram_prefix = self.tokens[:, seq_len - (no_repeat_ngram_size-1):]\n",
        "        # Next, find all the tokens we're not allowed to generate (by going iterating through past ngrams and seeing if those ngram prefixes match the last one)\n",
        "        for i in range(seq_len - (no_repeat_ngram_size-1)):\n",
        "            ngrams = self.tokens[:, i:i+no_repeat_ngram_size] # (batch, ngram)\n",
        "            ngrams_are_repeated = (ngrams[:, :-1] == last_ngram_prefix).all(-1) # (batch,)\n",
        "            ngram_end_tokens = ngrams[:, [-1]] # (batch, 1)\n",
        "            # Fill logprobs with neginf wherever the ngrams are repeated\n",
        "            logprobs[range(batch), ngram_end_tokens] = t.where(\n",
        "                ngrams_are_repeated,\n",
        "                neg_inf,\n",
        "                logprobs[range(batch), ngram_end_tokens],\n",
        "        )\n",
        "\n",
        "    # Finally, get our actual tokens\n",
        "    return logprobs.topk(k=k, dim=-1)\n",
        "```\n",
        "\n",
        "and for the main function:\n",
        "\n",
        "```python\n",
        "@t.inference_mode()\n",
        "def beam_search(\n",
        "    self: TransformerSampler,\n",
        "    prompt: str,\n",
        "    num_return_sequences: int,\n",
        "    num_beams: int,\n",
        "    max_new_tokens: int,\n",
        "    no_repeat_ngram_size: Optional[int] = None,\n",
        "    verbose=False\n",
        ") -> List[Tuple[float, Tensor]]:\n",
        "    '''\n",
        "    Implements a beam search, by repeatedly performing the `generate` and `filter` steps (starting\n",
        "    from the initial prompt) until either of the two stopping criteria are met:\n",
        "\n",
        "        (1) we've generated `max_new_tokens` tokens, or\n",
        "        (2) we've generated `num_returns_sequences` terminating sequences.\n",
        "\n",
        "    To modularize this function, most of the actual complexity is in the Beams class,\n",
        "    in the `generate` and `filter` methods.\n",
        "    '''\n",
        "\n",
        "    assert num_return_sequences <= num_beams\n",
        "    self.model.eval()\n",
        "\n",
        "    # SOLUTION\n",
        "    tokens = self.tokenizer.encode(prompt, return_tensors=\"pt\").to(device)\n",
        "\n",
        "    # List for final beams to return (and early terminations)\n",
        "    final_logprobs_and_completions: List[Tuple[float, str]] = []\n",
        "    # Keep track of all best beams after each step\n",
        "    best_beams = Beams(self.model, self.tokenizer, t.tensor([0.0]).to(device), tokens)\n",
        "\n",
        "    for n in tqdm(range(max_new_tokens)):\n",
        "\n",
        "        # Generation step\n",
        "        best_beams = best_beams.generate(toks_per_beam=num_beams, no_repeat_ngram_size=no_repeat_ngram_size)\n",
        "\n",
        "        # Filtering step\n",
        "        best_beams, best_beams_terminated = best_beams.filter(num_beams=num_beams)\n",
        "        final_logprobs_and_completions.extend(best_beams_terminated.logprobs_and_completions)\n",
        "\n",
        "        # Print output\n",
        "        if verbose:\n",
        "            best_beams.print()\n",
        "\n",
        "        # Check stopping condition\n",
        "        if len(final_logprobs_and_completions) >= num_return_sequences:\n",
        "            return final_logprobs_and_completions[:num_return_sequences]\n",
        "\n",
        "    final_logprobs_and_completions.extend(best_beams.logprobs_and_completions)\n",
        "    final_logprobs_and_completions = final_logprobs_and_completions[:num_return_sequences]\n",
        "    return final_logprobs_and_completions\n",
        "```\n",
        "\n",
        "</details>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J8FpquaGfHOH"
      },
      "source": [
        "## Caching\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_myyOt5hfHOH"
      },
      "source": [
        "*This section is also designed to be challenging, and take quite some time. There are many different ways to solve it, and you're expected to try and find your own way (you should think about this for a while before looking at the suggestions in the dropdowns). Additionally, you might not find it as interesting as some of the other sections. In this case, and if you have a lot of extra time, you might want to start on the \"building BERT\" exercises from this chapter.*\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vu83LC_4fHOH"
      },
      "source": [
        "### How can caching help us?\n",
        "\n",
        "The text generation we've done so far is needlessly re-computing certain values, which is very noticeable when you try to generate longer sequences.\n",
        "\n",
        "Suppose you're generating text, and you've already run GPT on the sentence \"My life motto:\". Now you want to run the model on the sentence \"My life motto: Always\". Which computations from the first sentence can you reuse?\n",
        "\n",
        "<details>\n",
        "<summary>Answer</summary>\n",
        "\n",
        "At each attention layer, the only things the attention layer needs from the previous sequence positions are the key and value vectors. This is explained in the following diagram:\n",
        "\n",
        "<img src=\"https://raw.githubusercontent.com/callummcdougall/computational-thread-art/master/example_images/misc/tl-cache.png\" width=\"600\">\n",
        "\n",
        "</details>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f7xGpbbZfHOH"
      },
      "source": [
        "### Exercise - implement caching\n",
        "\n",
        "```c\n",
        "Difficulty: 🟠🟠🟠🟠🟠\n",
        "Importance: 🟠⚪⚪⚪⚪\n",
        "\n",
        "You are expected to spend well over an hour on this exercise, if you choose to do it.\n",
        "```\n",
        "\n",
        "Modify your GPT-2 to optionally use a cache. When you run your GPT on `\"My life motto:\"`, it should store the necessary values in the cache. Then in the next forward pass with just `\" Always\"` as input, it should load the cached values instead of recomputing them (and update the cache). This only needs to work with a single input sequence (batch size of 1), and you can assume that after the first forward pass, the input will be just one token.\n",
        "\n",
        "The design of the cache is completely up to you - discuss possible designs with your partner before writing code. It should be possible to have only one GPT2 instance and many different cache instances at one time. Imagine that you want to use one instance to serve multiple users submitting requests for text generation like in [AI Dungeon](https://aidungeon.io/).\n",
        "\n",
        "You'll also need to rewrite parts of your `DemoTransformer` code, in order to get this to work. The tests have been built to accommodate modules which return their output as the first element in a tuple (i.e. `(output, cache)`) rather than just returning the output, so you should use the tests to verify that your modules still work as expected.\n",
        "\n",
        "Some example considerations:\n",
        "\n",
        "* Which GPT-2 classes need to interact with the cache?\n",
        "    * Will you need to change the positional embedding, and if so then how?\n",
        "* Should the cache be mutable and be updated in place, or does updating actually just create a separate instance?\n",
        "    * *(Hint here - think about how you might use the cache during beam search.)*\n",
        "* Is it possible for other programmers to incorrectly use your cache? Is there a way to prevent this failure mode or at least detect this and complain loudly?\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vq-Sx8apfHOH"
      },
      "source": [
        "<details>\n",
        "<summary>Cache implentation (example)</summary>\n",
        "\n",
        "This KeyValueCache object is structured as just a fancy tensor (it inherits all the methods from Tensor). The main difference is that it has a few extra helper methods, e.g. constructing an empty cache from a Config object.\n",
        "\n",
        "There are other ways you could do this, e.g. having your `KeyValueCache` class contain list of `KeyValueCacheEntry` objects (where each of these corresponds to a different layer).\n",
        "\n",
        "```python\n",
        "# Define a type for a single layer's cache entry (useful for type checking in later functions)\n",
        "KeyValueCacheTensor = Float[Tensor, \"2 batch seq_len n_heads d_head\"]\n",
        "\n",
        "class KeyValueCache(Tensor):\n",
        "    '''\n",
        "    This class holds tensors of key and value vectors, to be used for caching.\n",
        "\n",
        "    If we define it using cfg and batch then it's initialized as empty, but\n",
        "    we can also define it from kv_cache_entries.\n",
        "    '''\n",
        "    @classmethod\n",
        "    def new_empty(cls, cfg: Config, batch: int = 1) -> \"KeyValueCache\":\n",
        "        '''\n",
        "        Doing a forward pass on a cache created in this way indicates \"we don't\n",
        "        yet have a cache, but we want this forward pass to return a cache\".\n",
        "        Whereas using cache=None in a forward pass indicates we don't want to\n",
        "        return a cache.\n",
        "        '''\n",
        "        shape = (cfg.n_layers, 2, batch, 0, cfg.n_heads, cfg.d_head)\n",
        "        return cls(*shape).to(device)\n",
        "\n",
        "    # Define a handful of properties, so they can be referenced directly rather than\n",
        "    # indexing (which is more likely to lead to mistakes)\n",
        "\n",
        "    @property\n",
        "    def k(self) -> Tensor:\n",
        "        return self[:, 0]\n",
        "\n",
        "    @property\n",
        "    def v(self) -> Tensor:\n",
        "        return self[:, 1]\n",
        "\n",
        "    @property\n",
        "    def batch(self) -> int:\n",
        "        return self.shape[2]\n",
        "    \n",
        "    @property\n",
        "    def seq_len(self) -> int:\n",
        "        return self.shape[3]\n",
        "\n",
        "\n",
        "# Example implementation:\n",
        "cfg = model.cfg\n",
        "batch = 6\n",
        "kv_cache = KeyValueCache.new_empty(cfg, batch)\n",
        "\n",
        "print(f\"Shape of all kv-cache = {tuple(kv_cache.shape)}\")\n",
        "print(f\"Shape of just k-cache = {tuple(kv_cache.k.shape)}\")\n",
        "for kv_cache_entry in kv_cache:\n",
        "    print(f\"Shape of cache entry for one layer = {tuple(kv_cache_entry.shape)}\")\n",
        "    break\n",
        "print(f\"Batch size = {kv_cache.batch}\")\n",
        "print(f\"Current sequence length = {kv_cache.seq_len}\")\n",
        "```\n",
        "\n",
        "</details>\n",
        "\n",
        "<details>\n",
        "<summary>New <code>DemoTransformer</code> components (and testing)</summary>\n",
        "\n",
        "```python\n",
        "# Define new model parts where necessary, and create a new model & test it\n",
        "# Note that sometimes our modules return a tuple of (tensor output, cache) rather than just output. The\n",
        "# tests have been built to accommodate this.\n",
        "\n",
        "\n",
        "class PosEmbed(nn.Module):\n",
        "    def __init__(self, cfg: Config):\n",
        "        super().__init__()\n",
        "        self.cfg = cfg\n",
        "        self.W_pos = nn.Parameter(t.empty((cfg.n_ctx, cfg.d_model)))\n",
        "        nn.init.normal_(self.W_pos, std=self.cfg.init_range)\n",
        "\n",
        "    def forward(\n",
        "        self,\n",
        "        tokens: Int[Tensor, \"batch position\"],\n",
        "        past_kv_pos_offset: int = 0\n",
        "    ) -> Float[Tensor, \"batch position d_model\"]:\n",
        "        \n",
        "        # SOLUTION\n",
        "        batch, seq_len = tokens.shape\n",
        "        return einops.repeat(\n",
        "            self.W_pos[past_kv_pos_offset: seq_len+past_kv_pos_offset],\n",
        "            \"seq d_model -> batch seq d_model\",\n",
        "            batch=batch\n",
        "        )\n",
        "\n",
        "\n",
        "class Attention(nn.Module):\n",
        "    IGNORE: Float[Tensor, \"\"]\n",
        "\n",
        "    def __init__(self, cfg: Config):\n",
        "        super().__init__()\n",
        "        self.cfg = cfg\n",
        "        self.W_Q = nn.Parameter(t.empty((cfg.n_heads, cfg.d_model, cfg.d_head)))\n",
        "        self.W_K = nn.Parameter(t.empty((cfg.n_heads, cfg.d_model, cfg.d_head)))\n",
        "        self.W_V = nn.Parameter(t.empty((cfg.n_heads, cfg.d_model, cfg.d_head)))\n",
        "        self.W_O = nn.Parameter(t.empty((cfg.n_heads, cfg.d_head, cfg.d_model)))\n",
        "        self.b_Q = nn.Parameter(t.zeros((cfg.n_heads, cfg.d_head)))\n",
        "        self.b_K = nn.Parameter(t.zeros((cfg.n_heads, cfg.d_head)))\n",
        "        self.b_V = nn.Parameter(t.zeros((cfg.n_heads, cfg.d_head)))\n",
        "        self.b_O = nn.Parameter(t.zeros((cfg.d_model)))\n",
        "        nn.init.normal_(self.W_Q, std=self.cfg.init_range)\n",
        "        nn.init.normal_(self.W_K, std=self.cfg.init_range)\n",
        "        nn.init.normal_(self.W_V, std=self.cfg.init_range)\n",
        "        nn.init.normal_(self.W_O, std=self.cfg.init_range)\n",
        "        self.register_buffer(\"IGNORE\", t.tensor(-1e5, dtype=t.float32, device=device))\n",
        "\n",
        "    def forward(\n",
        "        self,\n",
        "        normalized_resid_pre: Float[Tensor, \"batch posn d_model\"],\n",
        "        kv_cache_entry: Optional[KeyValueCacheTensor] = None,\n",
        "    ) -> Tuple[\n",
        "        Float[Tensor, \"batch posn d_model\"],\n",
        "        Optional[KeyValueCacheTensor]\n",
        "    ]:\n",
        "        '''\n",
        "        Returns the result of applying attention layer to normlized_resid_pre, as well as\n",
        "        the new cached key and value vectors (which we get from concatenating the old cached\n",
        "        ones with the new key and value vectors).\n",
        "        '''\n",
        "        \n",
        "        # SOLUTION\n",
        "\n",
        "        # Calculate the new query, key and value vectors\n",
        "        q = einops.einsum(\n",
        "            normalized_resid_pre, self.W_Q,\n",
        "            \"batch posn d_model, nheads d_model d_head -> batch posn nheads d_head\"\n",
        "        ) + self.b_Q\n",
        "        k = einops.einsum(\n",
        "            normalized_resid_pre, self.W_K,\n",
        "            \"batch posn d_model, nheads d_model d_head -> batch posn nheads d_head\"\n",
        "        ) + self.b_K\n",
        "        v = einops.einsum(\n",
        "            normalized_resid_pre, self.W_V,\n",
        "            \"batch posn d_model, nheads d_model d_head -> batch posn nheads d_head\"\n",
        "        ) + self.b_V\n",
        "\n",
        "        # If cache_entry is not None, this means we use the previous key and value vectors\n",
        "        # Also we'll need to get a new cache entry which will be used later to construct a new cache\n",
        "        if kv_cache_entry is not None:\n",
        "            k = t.concat([kv_cache_entry[0], k], dim=1)\n",
        "            v = t.concat([kv_cache_entry[1], v], dim=1)\n",
        "            kv_cache_entry = t.stack([k, v])\n",
        "\n",
        "        # Calculate attention scores, then scale and mask, and apply softmax to get probabilities\n",
        "        attn_scores = einops.einsum(\n",
        "            q, k,\n",
        "            \"batch posn_Q nheads d_head, batch posn_K nheads d_head -> batch nheads posn_Q posn_K\"\n",
        "        )\n",
        "        attn_scores_masked = self.apply_causal_mask(attn_scores / self.cfg.d_head ** 0.5)\n",
        "        attn_pattern = attn_scores_masked.softmax(-1)\n",
        "\n",
        "        # Take weighted sum of value vectors, according to attention probabilities\n",
        "        z = einops.einsum(\n",
        "            v, attn_pattern,\n",
        "            \"batch posn_K nheads d_head, batch nheads posn_Q posn_K -> batch posn_Q nheads d_head\"\n",
        "        )\n",
        "\n",
        "        # Calculate output (by applying matrix W_O and summing over heads, then adding bias b_O)\n",
        "        out = einops.einsum(\n",
        "            z, self.W_O,\n",
        "            \"batch posn_Q nheads d_head, nheads d_head d_model -> batch posn_Q d_model\"\n",
        "        ) + self.b_O\n",
        "\n",
        "        return out, kv_cache_entry\n",
        "\n",
        "    def apply_causal_mask(\n",
        "        self, attn_scores: Float[Tensor, \"batch n_heads query_pos key_pos\"]\n",
        "    ) -> Float[Tensor, \"batch n_heads query_pos key_pos\"]:\n",
        "        '''\n",
        "        Here, attn_scores have shape (batch, n_heads, query_pos, key_pos), where query_pos represents the\n",
        "        new (non-cached) positions, and key_pos represent all the positions (cached and non-cached).\n",
        "\n",
        "        So when we create our mask, the query indices and key indices will both go up to the same value\n",
        "        (the full sequence length), but the query indices will start at >0.\n",
        "        '''\n",
        "        new_seq_len, full_seq_len = attn_scores.shape[-2:]\n",
        "        assert new_seq_len <= full_seq_len\n",
        "        q_posn = einops.repeat(attn_scores.new_tensor(range(full_seq_len-new_seq_len, full_seq_len)), \"q -> q k\", k=full_seq_len)\n",
        "        k_posn = einops.repeat(attn_scores.new_tensor(range(full_seq_len)), \"k -> q k\", q=new_seq_len)\n",
        "        mask = q_posn < k_posn\n",
        "        attn_scores = attn_scores.masked_fill(mask, self.IGNORE)\n",
        "        return attn_scores\n",
        "\n",
        "\n",
        "class TransformerBlock(nn.Module):\n",
        "    def __init__(self, cfg: Config):\n",
        "        super().__init__()\n",
        "        self.cfg = cfg\n",
        "        self.ln1 = LayerNorm(cfg)\n",
        "        self.attn = Attention(cfg)\n",
        "        self.ln2 = LayerNorm(cfg)\n",
        "        self.mlp = MLP(cfg)\n",
        "\n",
        "    def forward(\n",
        "        self,\n",
        "        resid_pre: Float[Tensor, \"batch position d_model\"],\n",
        "        kv_cache_entry: Optional[KeyValueCacheTensor] = None,\n",
        "    ) -> Float[Tensor, \"batch position d_model\"]:\n",
        "\n",
        "        # SOLUTION\n",
        "        attn_out, kv_cache_entry = self.attn(self.ln1(resid_pre), kv_cache_entry)\n",
        "        resid_mid = attn_out + resid_pre\n",
        "        resid_post = self.mlp(self.ln2(resid_mid)) + resid_mid\n",
        "        return resid_post, kv_cache_entry\n",
        "\n",
        "\n",
        "\n",
        "class DemoTransformer(nn.Module):\n",
        "    def __init__(self, cfg: Config):\n",
        "        super().__init__()\n",
        "        self.cfg = cfg\n",
        "        self.embed = Embed(cfg)\n",
        "        self.pos_embed = PosEmbed(cfg)\n",
        "        self.blocks = nn.ModuleList([TransformerBlock(cfg) for _ in range(cfg.n_layers)])\n",
        "        self.ln_final = LayerNorm(cfg)\n",
        "        self.unembed = Unembed(cfg)\n",
        "\n",
        "    def forward(\n",
        "        self,\n",
        "        tokens: Int[Tensor, \"batch seq_pos\"],\n",
        "        kv_cache: Optional[KeyValueCache] = None\n",
        "    ) -> Float[Tensor, \"batch position d_vocab\"]:\n",
        "        \n",
        "        using_kv_cache = kv_cache is not None\n",
        "\n",
        "        if using_kv_cache:\n",
        "            # If using kv_cache, then we only need to pass forward the newest tokens\n",
        "            # Remember to add positional offset!\n",
        "            n_cached_tokens = kv_cache.seq_len\n",
        "            tokens = tokens[:, n_cached_tokens:]\n",
        "            residual = self.embed(tokens) + self.pos_embed(tokens, n_cached_tokens)\n",
        "        else:\n",
        "            # If not using cache, turn it into a list of None's (so we can iterate through it)\n",
        "            kv_cache = [None for _ in range(self.cfg.n_layers)]\n",
        "            residual = self.embed(tokens) + self.pos_embed(tokens)\n",
        "        \n",
        "        # Apply all layers, and create a (new) kv_cache from the key & value vectors\n",
        "        new_kv_cache_entries: List[KeyValueCacheTensor] = []\n",
        "        for block, kv_cache_entry in zip(self.blocks, kv_cache):\n",
        "            residual, kv_cache_entry = block(residual, kv_cache_entry)\n",
        "            if using_kv_cache: new_kv_cache_entries.append(kv_cache_entry)\n",
        "        \n",
        "        logits = self.unembed(self.ln_final(residual))\n",
        "        \n",
        "        if using_kv_cache:\n",
        "            return logits, KeyValueCache(t.stack(new_kv_cache_entries))\n",
        "        else:\n",
        "            return logits, None\n",
        "\n",
        "\n",
        "tokens = reference_gpt2.to_tokens(reference_text).to(device)\n",
        "logits, cache = reference_gpt2.run_with_cache(tokens)\n",
        "\n",
        "rand_int_test(PosEmbed, [2, 4])\n",
        "load_gpt2_test(PosEmbed, reference_gpt2.pos_embed, tokens)\n",
        "rand_float_test(Attention, [2, 4, 768])\n",
        "load_gpt2_test(Attention, reference_gpt2.blocks[0].attn, cache[\"normalized\", 0, \"ln1\"])\n",
        "rand_float_test(TransformerBlock, [2, 4, 768])\n",
        "load_gpt2_test(TransformerBlock, reference_gpt2.blocks[0], cache[\"resid_pre\", 0])\n",
        "rand_int_test(DemoTransformer, [2, 4])\n",
        "load_gpt2_test(DemoTransformer, reference_gpt2, tokens)\n",
        "```\n",
        "\n",
        "</details>\n",
        "\n",
        "<details>\n",
        "<summary>New sampling function</summary>\n",
        "\n",
        "```python\n",
        "@t.inference_mode()\n",
        "def sample_with_cache(\n",
        "    self: TransformerSampler,\n",
        "    prompt: str,\n",
        "    max_tokens_generated=100,\n",
        "    kv_cache: Optional[KeyValueCache] = None,\n",
        "    verbose=False,\n",
        "    seed: Optional[int] = None,\n",
        "    **kwargs\n",
        ") -> str:\n",
        "    \n",
        "    # SOLUTION\n",
        "    self.model.eval()\n",
        "    input_ids = self.tokenizer.encode(prompt, return_tensors=\"pt\").to(device)[0]\n",
        "    if seed is not None:\n",
        "        np.random.seed(seed)\n",
        "        t.manual_seed(seed)\n",
        "\n",
        "    for i in tqdm(range(max_tokens_generated)):\n",
        "        # Get new logits (make sure we don't pass in more tokens than the model's context length)\n",
        "        logits, kv_cache = self.model(input_ids[None, -self.cfg.n_ctx:], kv_cache)\n",
        "        # We only take logits for the last token, because this is what we're sampling\n",
        "        logits = logits[0, -1]\n",
        "        # Get next token (as a tensor of size (1, 1) so we can concat it to input_ids)\n",
        "        next_token = t.tensor([TransformerSampler.sample_next_token(input_ids, logits, **kwargs)], device=device)\n",
        "        # Create new input ids string, with shape (1, old_seq_len + 1)\n",
        "        input_ids = t.cat([input_ids, next_token], dim=-1)\n",
        "        # Print out results, if required\n",
        "        if verbose:\n",
        "            print(self.tokenizer.decode(input_ids), end=\"\\r\")\n",
        "        # If our new token was the end-of-text token, stop\n",
        "        if next_token == getattr(self.tokenizer, \"eos_token_id\", None):\n",
        "            break\n",
        "    \n",
        "    return self.tokenizer.decode(input_ids)\n",
        "\n",
        "\n",
        "TransformerSampler.sample = sample_with_cache\n",
        "```\n",
        "</details>\n",
        "\n",
        "<details>\n",
        "<summary>Code to verify that the same output is being produced by cache and no-cache versions (and to compare speeds)</summary>\n",
        "\n",
        "```python\n",
        "device = t.device(\"cuda\") # can also try \"cpu\"\n",
        "\n",
        "model = DemoTransformer(Config()).to(device)\n",
        "model.load_state_dict(reference_gpt2.state_dict(), strict=False);\n",
        "\n",
        "initial_text = \"Eliezer Shlomo Yudkowsky (born September 11, 1979) is an American decision and artificial intelligence (AI) theorist and writer, best known for\"\n",
        "# input_ids = tokenizer.encode(initial_text, return_tensors=\"pt\").squeeze()\n",
        "\n",
        "sampler = TransformerSampler(model, tokenizer)\n",
        "\n",
        "# Run the noncached version\n",
        "t0 = time.time()\n",
        "text = sampler.sample(\n",
        "    initial_text,\n",
        "    temperature=0.7,\n",
        "    top_p=0.95,\n",
        "    seed=0,\n",
        ")\n",
        "print(f\"Time taken (without cache): {time.time() - t0:.2f} seconds\")\n",
        "rprint(f\"Model output:\\n\\n[bold dark_orange]{text}[/]\")\n",
        "\n",
        "# Run the cached version\n",
        "t0 = time.time()\n",
        "text_with_cache = sampler.sample(\n",
        "    initial_text,\n",
        "    temperature=0.7,\n",
        "    top_p=0.95,\n",
        "    seed=0,\n",
        "    kv_cache=KeyValueCache.new_empty(sampler.cfg)\n",
        ")\n",
        "print(f\"Time taken (with cache): {time.time() - t0:.2f} seconds\")\n",
        "rprint(f\"Model output:\\n\\n[bold dark_orange]{text_with_cache}[/]\")\n",
        "\n",
        "# # Check they are the same\n",
        "assert text == text_with_cache, \"Your outputs are different, meaning you've probably made a mistake in your cache implementation (or failed to use random seeds).\"\n",
        "print(\"Tests passed!\")\n",
        "```\n",
        "\n",
        "</details>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FhYU3O6sfHOH"
      },
      "source": [
        "You may find that your cache implementation provides a modest speedup, but probably not close to the `seq_len`-factor speedup you'd expect from the fact that you only compute one additional token at each step rather than all of them. Why is this? The answer is that, much like everything to do with computational and memory costs in deep learning, it's not so simple. There are a host of different factors which might be bottlenecking our model's forward pass speed. If you try this on the CPU, you should get a much more noticeable speedup.\n",
        "\n",
        "For a bit more on these topics, see [here](https://kipp.ly/blog/transformer-inference-arithmetic/#kv-cache).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0S_uxUzRfHOH"
      },
      "source": [
        "## Bonus - cached beam search\n",
        "\n",
        "Can you modify your beam search function to use caching?\n",
        "\n",
        "Depending on how you implemented your cache earlier, you might find that a different form of caching is better suited to beam search.\n",
        "\n",
        "Again, we've provided an example implementation in a dropdown below, which is based on the cache implementation above and the previous solution for `beam_search`.\n",
        "\n",
        "<details>\n",
        "<summary>Cached beam search function</summary>\n",
        "\n",
        "As we touched on earlier, thanks to our modular code, not a lot needs to be changed when adding cache support.\n",
        "\n",
        "```python\n",
        "@dataclass\n",
        "class Beams:\n",
        "    '''Class to store beams during beam search.'''\n",
        "    model: DemoTransformer\n",
        "    tokenizer: GPT2TokenizerFast\n",
        "    logprob_sums: Float[Tensor, \"batch\"]\n",
        "    tokens: Int[Tensor, \"batch seq\"]\n",
        "    kv_cache: Optional[KeyValueCache] = None\n",
        "\n",
        "    def new_beams(self, logprob_sums, tokens, kv_cache) -> \"Beams\":\n",
        "        '''Creates a new Beams object with the same model and tokenizer.'''\n",
        "        return Beams(self.model, self.tokenizer, logprob_sums, tokens, kv_cache)\n",
        "\n",
        "    def __getitem__(self, idx) -> \"Beams\":\n",
        "        '''Helpful function allowing you to take a slice of the beams object along the batch dimension.'''\n",
        "        return self.new_beams(\n",
        "            self.logprob_sums[idx],\n",
        "            self.tokens[idx],\n",
        "            self.kv_cache[:, :, idx] if self.kv_cache is not None else None\n",
        "        )\n",
        "\n",
        "    @property\n",
        "    def logprobs_and_completions(self) -> List[Tuple[float, str]]:\n",
        "        '''Returns self as a list of logprob sums and completions (useful for getting final output).'''\n",
        "        return [\n",
        "            (logprob_sum.item(), self.tokenizer.decode(tokens))\n",
        "            for (logprob_sum, tokens) in zip(self.logprob_sums, self.tokens)\n",
        "        ]\n",
        "    \n",
        "\n",
        "    def generate(self, new_beams: int, no_repeat_ngram_size: Optional[int] = None) -> \"Beams\":\n",
        "        '''\n",
        "        Starting from the current set of beams (which has length `num_beams`), returns a new\n",
        "        set of `num_beams * new_beams`, containing the best `new_beams` continuations for each\n",
        "        of the original beams.\n",
        "\n",
        "        Optional argument `no_repeat_ngram_size` means your model won't generate any sequences with\n",
        "        a repeating n-gram of this length (don't worry about implementing this until later).\n",
        "        '''\n",
        "        # SOLUTION\n",
        "\n",
        "        # Get the output logprobs for the next token (for every sequence in current beams)\n",
        "        logprobs, kv_cache = self.model(self.tokens, self.kv_cache)\n",
        "        logprobs = logprobs[:, -1, :].log_softmax(-1)\n",
        "\n",
        "        # Get the top `new_beams` tokens for each sequence\n",
        "        topk_logprobs, topk_tokenIDs = self.get_topk_non_repeating(logprobs, no_repeat_ngram_size, k=new_beams)\n",
        "\n",
        "        # Get all of the new possible beams, via einops operations\n",
        "        #   Here, we're effectively flattening out the batch dimension and k dimension, to give us tensors\n",
        "        #   with every possible combination of (original sequence, new token) pairs.)\n",
        "        new_logprob_sums = sum([\n",
        "            einops.repeat(self.logprob_sums, \"batch -> (batch k)\", k=new_beams),\n",
        "            einops.rearrange(topk_logprobs, \"batch k -> (batch k)\")\n",
        "        ])\n",
        "        new_tokens = t.concat([\n",
        "            einops.repeat(self.tokens, \"batch seq -> (batch k) seq\", k=new_beams),\n",
        "            einops.rearrange(topk_tokenIDs, \"batch k -> (batch k) 1\")\n",
        "        ], dim=-1)\n",
        "        new_kv_cache = None if (self.kv_cache is None) else einops.repeat(\n",
        "            kv_cache, \"layer k_and_v batch ... -> layer k_and_v (batch k) ...\", k=new_beams\n",
        "        )\n",
        "        return self.new_beams(new_logprob_sums, new_tokens, new_kv_cache)\n",
        "\n",
        "\n",
        "    def filter(self, num_beams: int) -> Tuple[\"Beams\", \"Beams\"]:\n",
        "        '''\n",
        "        Returns:\n",
        "            best_beams: Beams\n",
        "                filtered version of self, containing all best `num_beams` which are also not terminated.\n",
        "\n",
        "            early_terminations: Beams\n",
        "                filtered version of self, containing all best `num_beams` which are also terminated.\n",
        "                i.e. the sum of lengths of these two should equal `num_beams`.\n",
        "        '''\n",
        "        # SOLUTION\n",
        "\n",
        "        # Get the indices of top `num_beams` beams\n",
        "        top_beam_indices = self.logprob_sums.topk(k=num_beams, dim=0).indices.tolist()\n",
        "        # Get the indices of terminated sequences\n",
        "        new_tokens = self.tokens[:, -1]\n",
        "        terminated_indices = t.nonzero(new_tokens == self.tokenizer.eos_token_id)\n",
        "\n",
        "        # Get the indices of the `num_beams` best sequences (some terminated, some not terminated)\n",
        "        best_continuing = [i for i in top_beam_indices if i not in terminated_indices]\n",
        "        best_terminated = [i for i in top_beam_indices if i in terminated_indices]\n",
        "\n",
        "        # Return the beam objects from these indices\n",
        "        return self[best_continuing], self[best_terminated]\n",
        "\n",
        "            \n",
        "    def get_topk_non_repeating(\n",
        "        self,\n",
        "        logprobs: Float[Tensor, \"batch d_vocab\"],\n",
        "        no_repeat_ngram_size: Optional[int],\n",
        "        k: int,\n",
        "    ) -> Tuple[Float[Tensor, \"k\"], Int[Tensor, \"k\"]]:\n",
        "        '''\n",
        "        logprobs:\n",
        "            tensor of the log-probs for the next token\n",
        "        no_repeat_ngram_size:\n",
        "            size of ngram to avoid repeating\n",
        "        k:\n",
        "            number of top logits to return, for each beam in our collection\n",
        "\n",
        "        Returns:\n",
        "            equivalent to the output of `logprobs.topk(dim=-1)`, but makes sure\n",
        "            that no returned tokens would produce an ngram of size  `no_repeat_ngram_size`\n",
        "            which has already appeared in `self.tokens`.\n",
        "        '''\n",
        "        batch, seq_len = self.tokens.shape\n",
        "        neg_inf = t.tensor(-1.0e4).to(device)\n",
        "\n",
        "        # If completion isn't long enough for a repetition, or we have no restructions, just return topk\n",
        "        if (no_repeat_ngram_size is not None) and (seq_len > no_repeat_ngram_size-1):\n",
        "            # Otherwise, we need to check for ngram repetitions\n",
        "            # First, get the most recent `no_repeat_ngram_size-1` tokens\n",
        "            last_ngram_prefix = self.tokens[:, seq_len - (no_repeat_ngram_size-1):]\n",
        "            # Next, find all the tokens we're not allowed to generate (by going iterating through past ngrams and seeing if those ngram prefixes match the last one)\n",
        "            for i in range(seq_len - (no_repeat_ngram_size-1)):\n",
        "                ngrams = self.tokens[:, i:i+no_repeat_ngram_size] # (batch, ngram)\n",
        "                ngrams_are_repeated = (ngrams[:, :-1] == last_ngram_prefix).all(-1) # (batch,)\n",
        "                ngram_end_tokens = ngrams[:, [-1]] # (batch, 1)\n",
        "                # Fill logprobs with neginf wherever the ngrams are repeated\n",
        "                logprobs[range(batch), ngram_end_tokens] = t.where(\n",
        "                    ngrams_are_repeated,\n",
        "                    neg_inf,\n",
        "                    logprobs[range(batch), ngram_end_tokens],\n",
        "            )\n",
        "\n",
        "        # Finally, get our actual tokens\n",
        "        return logprobs.topk(k=k, dim=-1)\n",
        "\n",
        "    def print(self, title=\"Best completions\", max_print_chars=80) -> None:\n",
        "        '''\n",
        "        Prints out a set of sequences with their corresponding logitsums.\n",
        "        '''\n",
        "        if len(self.tokens) == 0:\n",
        "            return\n",
        "        table = Table(\"logitsum\", \"completion\", title=title)\n",
        "        for logprob_sum, tokens in zip(self.logprob_sums, self.tokens):\n",
        "            text = self.tokenizer.decode(tokens)\n",
        "            if len(repr(text)) > max_print_chars:\n",
        "                text = text[:int(0.3 * max_print_chars)] + \" ... \" + text[-int(0.7 * max_print_chars):]\n",
        "            table.add_row(f\"{logprob_sum:>8.3f}\", repr(text))\n",
        "        rprint(table)\n",
        "\n",
        "\n",
        "@t.inference_mode()\n",
        "def beam_search(\n",
        "    self: TransformerSampler,\n",
        "    prompt: str,\n",
        "    num_return_sequences: int,\n",
        "    num_beams: int,\n",
        "    max_new_tokens: int,\n",
        "    no_repeat_ngram_size: Optional[int] = None,\n",
        "    kv_cache: Optional[KeyValueCache] = None,\n",
        "    verbose=False\n",
        ") -> List[Tuple[float, Tensor]]:\n",
        "    '''\n",
        "    Implements a beam search, by repeatedly performing the `generate` and `filter` steps (starting\n",
        "    from the initial prompt) until either of the two stopping criteria are met:\n",
        "\n",
        "        (1) we've generated `max_new_tokens` tokens, or\n",
        "        (2) we've generated `num_returns_sequences` terminating sequences.\n",
        "\n",
        "    To modularize this function, most of the actual complexity is in the Beams class,\n",
        "    in the `generate` and `filter` methods.\n",
        "    '''\n",
        "\n",
        "    assert num_return_sequences <= num_beams\n",
        "    self.model.eval()\n",
        "\n",
        "    # SOLUTION\n",
        "    tokens = self.tokenizer.encode(prompt, return_tensors=\"pt\").to(device)\n",
        "\n",
        "    # List for final beams to return (and early terminations)\n",
        "    final_logprobs_and_completions: List[Tuple[float, str]] = []\n",
        "    # Keep track of all best beams after each step\n",
        "    best_beams = Beams(self.model, self.tokenizer, t.tensor([0.0]).to(device), tokens, kv_cache)\n",
        "\n",
        "    for n in tqdm(range(max_new_tokens)):\n",
        "\n",
        "        # Generation step\n",
        "        best_beams = best_beams.generate(num_beams, no_repeat_ngram_size)\n",
        "\n",
        "        # Filtering step\n",
        "        best_beams, best_beams_terminated = best_beams.filter(num_beams)\n",
        "        final_logprobs_and_completions.extend(best_beams_terminated.logprobs_and_completions)\n",
        "\n",
        "        # Print output\n",
        "        if verbose:\n",
        "            best_beams.print()\n",
        "\n",
        "        # Check stopping condition\n",
        "        if len(final_logprobs_and_completions) >= num_return_sequences:\n",
        "            return final_logprobs_and_completions[:num_return_sequences]\n",
        "\n",
        "    final_logprobs_and_completions.extend(best_beams.logprobs_and_completions)\n",
        "    final_logprobs_and_completions = final_logprobs_and_completions[:num_return_sequences]\n",
        "    return final_logprobs_and_completions\n",
        "\n",
        "\n",
        "TransformerSampler.beam_search = beam_search\n",
        "```\n",
        "\n",
        "</details>\n",
        "\n",
        "<details>\n",
        "<summary>Code to verify that the same output is being produced by cache and no-cache versions (and to compare speeds)</summary>\n",
        "\n",
        "```python\n",
        "prompt = \"For you, the day Bison graced your village was the most important day of your life. But for me, it was\"\n",
        "orig_len = len(tokenizer.encode(prompt))\n",
        "\n",
        "beam_search_kwargs = dict(\n",
        "    prompt=prompt,\n",
        "    num_return_sequences=3,\n",
        "    num_beams=20,\n",
        "    max_new_tokens=60,\n",
        "    no_repeat_ngram_size=2,\n",
        "    verbose=False\n",
        ")\n",
        "\n",
        "sampler = TransformerSampler(model, tokenizer)\n",
        "\n",
        "# Run the noncached version\n",
        "t0 = time.time()\n",
        "final_logitsums_and_completions = sampler.beam_search(**beam_search_kwargs)\n",
        "logprob_sum, text = final_logitsums_and_completions[0]\n",
        "avg_logprob_as_prob = t.tensor(logprob_sum / (len(tokenizer.encode(text)) - orig_len)).exp().item()\n",
        "print(f\"Time (without cache): {time.time() - t0:.2f} seconds\")\n",
        "print(f\"Avg logprob (expressed as a probability) = {avg_logprob_as_prob:.3f}\")\n",
        "rprint(f\"Output:\\n\\n[bold dark_orange]{text}[/]\\n\\n\")\n",
        "\n",
        "# Run the cached version\n",
        "t0 = time.time()\n",
        "beam_search_kwargs[\"kv_cache\"] = KeyValueCache.new_empty(model.cfg)\n",
        "final_logitsums_and_completions = sampler.beam_search(**beam_search_kwargs)\n",
        "logprob_sum, text_with_cache = final_logitsums_and_completions[0]\n",
        "avg_logprob_as_prob = t.tensor(logprob_sum / (len(tokenizer.encode(text)) - orig_len)).exp().item()\n",
        "print(f\"Time (with cache): {time.time() - t0:.2f} seconds\")\n",
        "print(f\"Avg logprob (as probability) = {avg_logprob_as_prob:.3f}\", end=\"\")\n",
        "rprint(f\"Output:\\n\\n[bold dark_orange]{text_with_cache}[/]\\n\\n\")\n",
        "\n",
        "# Check they are the same\n",
        "assert text == text_with_cache, \"Your outputs are different, meaning you've probably made a mistake in your cache implementation.\"\n",
        "print(\"Tests passed!\")\n",
        "```\n",
        "\n",
        "</details>\n"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": [],
      "gpuClass": "premium",
      "gpuType": "T4",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.9"
    },
    "orig_nbformat": 4,
    "vscode": {
      "interpreter": {
        "hash": "0575d7a87c0e74eddddcbc1a627da1d71db6b89a3121c036174dfb29a1bf0df3"
      }
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "269d5e9f0aff43fd979c9d15141f1562": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_92f1a14e0c944e6092538d49db2aaf39",
              "IPY_MODEL_7be6388f18514f549478dc383d1f15ce",
              "IPY_MODEL_026f3e2d80544d05baec9e1bb0bb4d32"
            ],
            "layout": "IPY_MODEL_1bb0142dbdc0473498d11070ff3badb8"
          }
        },
        "92f1a14e0c944e6092538d49db2aaf39": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_f6fda666ff984f69ba6f7e0a33c5af7e",
            "placeholder": "​",
            "style": "IPY_MODEL_e8776691dee643b5a21a0f9eaffcc1fc",
            "value": "config.json: 100%"
          }
        },
        "7be6388f18514f549478dc383d1f15ce": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e795971451784c98b2fbaafa4ee78606",
            "max": 665,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_26bef72c76d74c548232b7af52232879",
            "value": 665
          }
        },
        "026f3e2d80544d05baec9e1bb0bb4d32": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b0ef9d5d393148d0bff7734fc2f08602",
            "placeholder": "​",
            "style": "IPY_MODEL_ca29dd1cbfb742ec9f5930ecc513f6cd",
            "value": " 665/665 [00:00&lt;00:00, 11.2kB/s]"
          }
        },
        "1bb0142dbdc0473498d11070ff3badb8": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "f6fda666ff984f69ba6f7e0a33c5af7e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e8776691dee643b5a21a0f9eaffcc1fc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e795971451784c98b2fbaafa4ee78606": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "26bef72c76d74c548232b7af52232879": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "b0ef9d5d393148d0bff7734fc2f08602": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ca29dd1cbfb742ec9f5930ecc513f6cd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "7c5b61c8bdb14ebf94bcc62876f8b4be": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_6920874e038d4befb546ba1a707eb48a",
              "IPY_MODEL_27c11ca2e5de4163adeafb8a3e9f9d45",
              "IPY_MODEL_c0a5c5d79eee40048245b71565fa29ac"
            ],
            "layout": "IPY_MODEL_8136627b376749bd8289bc07ada2e734"
          }
        },
        "6920874e038d4befb546ba1a707eb48a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e9b8c1b6faf84a9a9462b1dd4529c732",
            "placeholder": "​",
            "style": "IPY_MODEL_97be4b5263cb4e4dbf69491329de09b9",
            "value": "model.safetensors: 100%"
          }
        },
        "27c11ca2e5de4163adeafb8a3e9f9d45": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_19d0162699d14833b8b5bc650999f244",
            "max": 548105171,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_90605bcc38714f19a2761c085b240d4e",
            "value": 548105171
          }
        },
        "c0a5c5d79eee40048245b71565fa29ac": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_fa5c27fc179549f7b6d629cc6e65adeb",
            "placeholder": "​",
            "style": "IPY_MODEL_8f1de413688f4a5abd42d170a7d3e56c",
            "value": " 548M/548M [00:05&lt;00:00, 58.1MB/s]"
          }
        },
        "8136627b376749bd8289bc07ada2e734": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e9b8c1b6faf84a9a9462b1dd4529c732": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "97be4b5263cb4e4dbf69491329de09b9": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "19d0162699d14833b8b5bc650999f244": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "90605bcc38714f19a2761c085b240d4e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "fa5c27fc179549f7b6d629cc6e65adeb": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8f1de413688f4a5abd42d170a7d3e56c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "54942419d92f492f85981d5f51cc9d67": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_5a849626d2594777986421bfd002dd36",
              "IPY_MODEL_3a29f9044d3a496392501b6531811035",
              "IPY_MODEL_8ca8ca6dc3be4b8b97f2bf273dd342c5"
            ],
            "layout": "IPY_MODEL_04d80c42acbd45e88e2f6ef1cc07c304"
          }
        },
        "5a849626d2594777986421bfd002dd36": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b77c405889cc49118fdd9fb8e20a7647",
            "placeholder": "​",
            "style": "IPY_MODEL_4812eba5acc942d78e5240d614ad8ffd",
            "value": "generation_config.json: 100%"
          }
        },
        "3a29f9044d3a496392501b6531811035": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a71b3204509c45919893331630bdffc1",
            "max": 124,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_6900bf316fcd44958996ce0b65255e38",
            "value": 124
          }
        },
        "8ca8ca6dc3be4b8b97f2bf273dd342c5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_d129f10f62474da9bac8f82ab13e353e",
            "placeholder": "​",
            "style": "IPY_MODEL_69361149eb5e41ac8cc6a4b8f9568f44",
            "value": " 124/124 [00:00&lt;00:00, 1.47kB/s]"
          }
        },
        "04d80c42acbd45e88e2f6ef1cc07c304": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b77c405889cc49118fdd9fb8e20a7647": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4812eba5acc942d78e5240d614ad8ffd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "a71b3204509c45919893331630bdffc1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6900bf316fcd44958996ce0b65255e38": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "d129f10f62474da9bac8f82ab13e353e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "69361149eb5e41ac8cc6a4b8f9568f44": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "7c0ef275820849918a948ddb4a91df07": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_60d8bfb258d3407283b7b450379c9bd5",
              "IPY_MODEL_faf869fbcf764285a7285c377bea8367",
              "IPY_MODEL_b9e0e56aba3944d4b2b2cf4b2844f743"
            ],
            "layout": "IPY_MODEL_3a166534a32647c0883900afb90a9a0d"
          }
        },
        "60d8bfb258d3407283b7b450379c9bd5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_37b71051b1a642d49889d332eb52dca9",
            "placeholder": "​",
            "style": "IPY_MODEL_5a12edc1e97343178b7798a4c7918d57",
            "value": "vocab.json: 100%"
          }
        },
        "faf869fbcf764285a7285c377bea8367": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a3ce7ac46d1649f38dfa33cb9fbe4c35",
            "max": 1042301,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_8dfaffc2cd5c404e88b7d862f6daa4cd",
            "value": 1042301
          }
        },
        "b9e0e56aba3944d4b2b2cf4b2844f743": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_094f18d4a82f4e85ae1cd4cb588f0a87",
            "placeholder": "​",
            "style": "IPY_MODEL_81e0822cc55245fe807af2f786e3bc8f",
            "value": " 1.04M/1.04M [00:00&lt;00:00, 18.9MB/s]"
          }
        },
        "3a166534a32647c0883900afb90a9a0d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "37b71051b1a642d49889d332eb52dca9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5a12edc1e97343178b7798a4c7918d57": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "a3ce7ac46d1649f38dfa33cb9fbe4c35": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "8dfaffc2cd5c404e88b7d862f6daa4cd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "094f18d4a82f4e85ae1cd4cb588f0a87": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "81e0822cc55245fe807af2f786e3bc8f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "efae7152858d43f58234673f0a2ba576": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_1a6c704c22b4478e9357b670319df551",
              "IPY_MODEL_31ed8d7270cf495b9a21a6ba9d97f877",
              "IPY_MODEL_ce248259fcd34ead831b57ad0a852fda"
            ],
            "layout": "IPY_MODEL_4f79df4de94e497f8f8b3885b0f0b7c6"
          }
        },
        "1a6c704c22b4478e9357b670319df551": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_9550e3f920534886bc736f3a0d264f8b",
            "placeholder": "​",
            "style": "IPY_MODEL_0a6fa772148d4ddf92ef6fbc75e917de",
            "value": "merges.txt: 100%"
          }
        },
        "31ed8d7270cf495b9a21a6ba9d97f877": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_a8b43c8bf5774a04bb680b9b08143fa2",
            "max": 456318,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_ad4dfe0718d74729ade8c7456d525acc",
            "value": 456318
          }
        },
        "ce248259fcd34ead831b57ad0a852fda": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_dfe7cd0558754f6488b92242f3db63c5",
            "placeholder": "​",
            "style": "IPY_MODEL_6cc9a9c1e1634a38b76ae0de5107fa37",
            "value": " 456k/456k [00:00&lt;00:00, 6.78MB/s]"
          }
        },
        "4f79df4de94e497f8f8b3885b0f0b7c6": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9550e3f920534886bc736f3a0d264f8b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0a6fa772148d4ddf92ef6fbc75e917de": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "a8b43c8bf5774a04bb680b9b08143fa2": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ad4dfe0718d74729ade8c7456d525acc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "dfe7cd0558754f6488b92242f3db63c5": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6cc9a9c1e1634a38b76ae0de5107fa37": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "51bea67627e548d49de90e67220e167d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_daac9e97c3a54713a063d8f8d43c983e",
              "IPY_MODEL_0b706919b18d44d7a1029902094e4574",
              "IPY_MODEL_b0024d2f09294559916e2ff54f178c0a"
            ],
            "layout": "IPY_MODEL_59084b8142af412680badbd062fdfeab"
          }
        },
        "daac9e97c3a54713a063d8f8d43c983e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_682cf14b28f946c7aa3e52fc7711027d",
            "placeholder": "​",
            "style": "IPY_MODEL_0625904bd90044999c8afd5f0ecc368e",
            "value": "tokenizer.json: 100%"
          }
        },
        "0b706919b18d44d7a1029902094e4574": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_19be64b3d7434e97ab5ed898eccdc42a",
            "max": 1355256,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_e503325532834078aa311fbfba7f134a",
            "value": 1355256
          }
        },
        "b0024d2f09294559916e2ff54f178c0a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_9cd383674c964f1a9fee01331c3a4131",
            "placeholder": "​",
            "style": "IPY_MODEL_bbf85925f8774f42b7e0db66ddbfdc32",
            "value": " 1.36M/1.36M [00:00&lt;00:00, 7.20MB/s]"
          }
        },
        "59084b8142af412680badbd062fdfeab": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "682cf14b28f946c7aa3e52fc7711027d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0625904bd90044999c8afd5f0ecc368e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "19be64b3d7434e97ab5ed898eccdc42a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e503325532834078aa311fbfba7f134a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "9cd383674c964f1a9fee01331c3a4131": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "bbf85925f8774f42b7e0db66ddbfdc32": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "2f58958ac8db4bc4812ec0bd24aff8e2": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_e00a792c05bb48c49cee3d163e520719",
              "IPY_MODEL_ba40c60e3022402392565289776b24fe",
              "IPY_MODEL_64d897a5fdda479680aeb9435714a862"
            ],
            "layout": "IPY_MODEL_db79598fed524e48acffd496511bcb0a"
          }
        },
        "e00a792c05bb48c49cee3d163e520719": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_171b6d5e10c24633bf5c8bb01518254d",
            "placeholder": "​",
            "style": "IPY_MODEL_7507657803444b9196681b1574679bff",
            "value": "100%"
          }
        },
        "ba40c60e3022402392565289776b24fe": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_79c112fb4618426a97e95a5c0a76130e",
            "max": 100,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_5f8fe99208214f7dbdf9d1e401d60ef0",
            "value": 100
          }
        },
        "64d897a5fdda479680aeb9435714a862": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_76ffa589aa1a4ee292f55753aac92ea1",
            "placeholder": "​",
            "style": "IPY_MODEL_0596bafaec9c43d397e888f7cd80a838",
            "value": " 100/100 [00:50&lt;00:00,  1.26it/s]"
          }
        },
        "db79598fed524e48acffd496511bcb0a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "171b6d5e10c24633bf5c8bb01518254d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "7507657803444b9196681b1574679bff": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "79c112fb4618426a97e95a5c0a76130e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5f8fe99208214f7dbdf9d1e401d60ef0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "76ffa589aa1a4ee292f55753aac92ea1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "0596bafaec9c43d397e888f7cd80a838": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "28049f90483e4742bcb9eb459a6db7eb": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_d8afdac0c384499f89d63b509a92b4cc",
              "IPY_MODEL_fdd8bc08dd3f4001aea23ebb2d8b41d4",
              "IPY_MODEL_65e1a1edeaa1415d80cb3cca5b83777d"
            ],
            "layout": "IPY_MODEL_eb9f96f11d8b4868a16c01d8ba6eb41a"
          }
        },
        "d8afdac0c384499f89d63b509a92b4cc": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_cae77d53492c48e2893aafc86b845f7c",
            "placeholder": "​",
            "style": "IPY_MODEL_803a5a794f9a414fb5499a60a1346d47",
            "value": "Downloading readme: 100%"
          }
        },
        "fdd8bc08dd3f4001aea23ebb2d8b41d4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_7b5b16b06b604209b38b8b30eb2893f7",
            "max": 373,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_c48e777e11f14f4a8e2f9e6bc341703c",
            "value": 373
          }
        },
        "65e1a1edeaa1415d80cb3cca5b83777d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_af8ff1c0eae3482b91be5dd495abd80d",
            "placeholder": "​",
            "style": "IPY_MODEL_d0f05229686d4283bca4ae5f5b7b41b4",
            "value": " 373/373 [00:00&lt;00:00, 23.1kB/s]"
          }
        },
        "eb9f96f11d8b4868a16c01d8ba6eb41a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "cae77d53492c48e2893aafc86b845f7c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "803a5a794f9a414fb5499a60a1346d47": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "7b5b16b06b604209b38b8b30eb2893f7": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "c48e777e11f14f4a8e2f9e6bc341703c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "af8ff1c0eae3482b91be5dd495abd80d": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d0f05229686d4283bca4ae5f5b7b41b4": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ec252457008f4a6fb239029baf2d4acd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_c1830bae7a5b42f78d10591f7fcb0f77",
              "IPY_MODEL_911c53dce604485d9dae7ebe3cf46c43",
              "IPY_MODEL_5a2259982aed4047badce93e30ade5c5"
            ],
            "layout": "IPY_MODEL_982455c4716641aba7b9327a2b21be6b"
          }
        },
        "c1830bae7a5b42f78d10591f7fcb0f77": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_9f813ae9af5a43e78fd68f17bbfab419",
            "placeholder": "​",
            "style": "IPY_MODEL_b0b588200725423187952bc78b124533",
            "value": "Downloading metadata: 100%"
          }
        },
        "911c53dce604485d9dae7ebe3cf46c43": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_64387fc0a000413bbad3cbe75e1fb75a",
            "max": 921,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_4be5864a06d54e0d83a0885ce073624f",
            "value": 921
          }
        },
        "5a2259982aed4047badce93e30ade5c5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_317a64d53fc64949bbede3e108cc58f1",
            "placeholder": "​",
            "style": "IPY_MODEL_037027c253564438b41b840c5340e5ac",
            "value": " 921/921 [00:00&lt;00:00, 50.8kB/s]"
          }
        },
        "982455c4716641aba7b9327a2b21be6b": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "9f813ae9af5a43e78fd68f17bbfab419": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b0b588200725423187952bc78b124533": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "64387fc0a000413bbad3cbe75e1fb75a": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "4be5864a06d54e0d83a0885ce073624f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "317a64d53fc64949bbede3e108cc58f1": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "037027c253564438b41b840c5340e5ac": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "9c4ce91ad7fc4d0c9cad41c4f7881a9e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_3483ac4d87234799890d3d5b11ec310b",
              "IPY_MODEL_577ba07f924c4c7b84b49030e492460c",
              "IPY_MODEL_c8d5b1b02c28469f816edd1d7457f136"
            ],
            "layout": "IPY_MODEL_867e0109b9c94956960a7f732f63b5a6"
          }
        },
        "3483ac4d87234799890d3d5b11ec310b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b117855b37a241209045453dc37465ed",
            "placeholder": "​",
            "style": "IPY_MODEL_1da24dcd5424475ea2c3bc3a2e2cff4e",
            "value": "Downloading data: 100%"
          }
        },
        "577ba07f924c4c7b84b49030e492460c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_ddf61b8fbd05423d84443daf23456a0c",
            "max": 33262901,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_162bc38747104d55b3a68a56db35ae57",
            "value": 33262901
          }
        },
        "c8d5b1b02c28469f816edd1d7457f136": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_8b6cccb7541243e8bc449bd203dec27e",
            "placeholder": "​",
            "style": "IPY_MODEL_b75b5289c00e444eababd1cc92c6fddf",
            "value": " 33.3M/33.3M [00:00&lt;00:00, 54.5MB/s]"
          }
        },
        "867e0109b9c94956960a7f732f63b5a6": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b117855b37a241209045453dc37465ed": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "1da24dcd5424475ea2c3bc3a2e2cff4e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "ddf61b8fbd05423d84443daf23456a0c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "162bc38747104d55b3a68a56db35ae57": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "8b6cccb7541243e8bc449bd203dec27e": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b75b5289c00e444eababd1cc92c6fddf": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "593d5bcaee1f45ed9dae0a7ce1afcd7e": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_9eebc53633df45df8d406b2d72547964",
              "IPY_MODEL_ab8dd833b77446a6848e088dfe1eaeb5",
              "IPY_MODEL_8603f03604fd48bb909b43a02c5ceee7"
            ],
            "layout": "IPY_MODEL_b57a61601822442e8fcd6f5298374f24"
          }
        },
        "9eebc53633df45df8d406b2d72547964": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b7dab9943a7c442d8c23edc77dbd00b6",
            "placeholder": "​",
            "style": "IPY_MODEL_d8d8b9ae38144646ae9b75a6ef342bd5",
            "value": "Generating train split: 100%"
          }
        },
        "ab8dd833b77446a6848e088dfe1eaeb5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b80569503bd74de99fc2ddd6f3b66418",
            "max": 10000,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_6abc83f0419a41ceb0db7ea8b38e61f1",
            "value": 10000
          }
        },
        "8603f03604fd48bb909b43a02c5ceee7": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c3818283461e4172bd992f8881bc2337",
            "placeholder": "​",
            "style": "IPY_MODEL_5af3388392724299b61a31bfa7d2628d",
            "value": " 10000/10000 [00:00&lt;00:00, 15219.68 examples/s]"
          }
        },
        "b57a61601822442e8fcd6f5298374f24": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b7dab9943a7c442d8c23edc77dbd00b6": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d8d8b9ae38144646ae9b75a6ef342bd5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "b80569503bd74de99fc2ddd6f3b66418": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "6abc83f0419a41ceb0db7ea8b38e61f1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "c3818283461e4172bd992f8881bc2337": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "5af3388392724299b61a31bfa7d2628d": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "7f5d82a1a9a7450e925485317467a4c0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_ea9bd9476b93410898a9de03266004b1",
              "IPY_MODEL_140cd67951a3438992a139e743aff7b5",
              "IPY_MODEL_6a716a3e67e045b9847c9fc219fcf6a0"
            ],
            "layout": "IPY_MODEL_65dd5d2542c445bebcaf57c269d7082c"
          }
        },
        "ea9bd9476b93410898a9de03266004b1": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b42c5f26fa674b1aaca409a178b77a14",
            "placeholder": "​",
            "style": "IPY_MODEL_048dea62e7e04eccae28f749160a25bd",
            "value": "Map (num_proc=4): 100%"
          }
        },
        "140cd67951a3438992a139e743aff7b5": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_c5e006f3e2ec4b5db8e43b28f416ed2f",
            "max": 10000,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_249a4fff64624c9a8f5ac40f0a4d11ce",
            "value": 10000
          }
        },
        "6a716a3e67e045b9847c9fc219fcf6a0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_dcf1fdfba8034e7b811b856b5b685ec9",
            "placeholder": "​",
            "style": "IPY_MODEL_ada4b34439914f619b2d9fac9d293378",
            "value": " 10000/10000 [01:20&lt;00:00, 171.15 examples/s]"
          }
        },
        "65dd5d2542c445bebcaf57c269d7082c": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "b42c5f26fa674b1aaca409a178b77a14": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "048dea62e7e04eccae28f749160a25bd": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "c5e006f3e2ec4b5db8e43b28f416ed2f": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "249a4fff64624c9a8f5ac40f0a4d11ce": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "dcf1fdfba8034e7b811b856b5b685ec9": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "ada4b34439914f619b2d9fac9d293378": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}