numbers_pos.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9A2Vv1FysshA"
   },
   "source": [
    "# A Transformer based Language Model from scratch\n",
    "> Building transformer with simple building blocks\n",
    "\n",
    "- toc: true\n",
    "- branch: master\n",
    "- badges: true\n",
    "- comments: true\n",
    "- author: Arto\n",
    "- categories: [fastai, pytorch]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device('cuda:1' if torch.cuda.is_available() else 'cpu')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "executionInfo": {
     "elapsed": 14938,
     "status": "ok",
     "timestamp": 1683920469541,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "J1vMAp9RdEME"
   },
   "outputs": [],
   "source": [
    "#hide\n",
    "import sys\n",
    "if 'google.colab' in sys.modules:\n",
    "    !pip install -Uqq fastai"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "x60WAzoFsshC"
   },
   "source": [
    "In this notebook i'm going to construct transformer based language model from scratch starting with the simplest building blocks. This is inspired by Chapter 12 of [Deep Learning for Coders book](https://www.amazon.com/Deep-Learning-Coders-fastai-PyTorch/dp/1492045527) in which it's demonstrated how to create a Recurrent Neural Network. It provides a strong intuition of how RNNs relate to regular feed-forward neural nets and why certain design choices were made. Here we aim to aquire similar kind of intuition about Transfomer based architectures.\n",
    "\n",
    "But as always we should start with the data to be modeled, 'cause without data any model makes no particular sense."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "n-11Hp1FsshD"
   },
   "source": [
    "## Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4sY7YjqxsshE"
   },
   "source": [
    "Similar to authors of the book I'll use simple Human numbers dataset which is specifically designed to prototyping model fast and straightforward. For more details on the data one can refer to the aforemantioned book chapter which is also available for free as [a notebook](https://github.com/fastai/fastbook/blob/master/12_nlp_dive.ipynb) (isn't that awesome?!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 54
    },
    "executionInfo": {
     "elapsed": 10043,
     "status": "ok",
     "timestamp": 1683920479580,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "ZVbdeXJLfDWA",
    "outputId": "249b4241-d491-422e-8329-aac3cf7cf00f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#2) [Path('train.txt'),Path('valid.txt')]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from fastai.text.all import *\n",
    "path = untar_data(URLs.HUMAN_NUMBERS)\n",
    "Path.BASE_PATH = path\n",
    "path.ls()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Yvaqd-EFsshI"
   },
   "source": [
    "The data consists of consecutive numbers from 1 to 9999 inclusive spelled as words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 13,
     "status": "ok",
     "timestamp": 1683920479581,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "12L4Y2IZfP3U",
    "outputId": "2d99a82b-c249-4b66-8863-d088f471c23c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#9998) ['one \\n','two \\n','three \\n','four \\n','five \\n','six \\n','seven \\n','eight \\n','nine \\n','ten \\n'...]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lines = L()\n",
    "with open(path/'train.txt') as f: lines += L(*f.readlines())\n",
    "with open(path/'valid.txt') as f: lines += L(*f.readlines())\n",
    "lines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 11,
     "status": "ok",
     "timestamp": 1683920479581,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "3DmYFw56fVkG",
    "outputId": "4a1cf3aa-0de4-48d0-f944-3cb21c7a3874"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['one', '.', 'two', '.', 'three', '.', 'four', '.', 'five', '.']"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = ' . '.join([l.strip() for l in lines])\n",
    "tokens = text.split(' ')\n",
    "tokens[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 8,
     "status": "ok",
     "timestamp": 1683920479581,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "ShoK-h1tfd7v",
    "outputId": "065ff0bf-bd70-4e8e-90e3-f014a73bb2a4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#30) ['one','.','two','three','four','five','six','seven','eight','nine'...]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab = L(*tokens).unique()\n",
    "vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 7,
     "status": "ok",
     "timestamp": 1683920479582,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "tEJDCek3fi1Q",
    "outputId": "0005f9a8-81ed-4d44-e78e-3832689dca7a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#63095) [0,1,2,1,3,1,4,1,5,1...]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word2idx = {w:i for i,w in enumerate(vocab)}\n",
    "nums = L(word2idx[i] for i in tokens)\n",
    "nums"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "N0h8tG_xsshO"
   },
   "source": [
    "The task will be to predict subsequent token given preceding three. This kind of tasks when the goal is to predict next token from previous ones is called autoregresive language modeling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 346,
     "status": "ok",
     "timestamp": 1683920479922,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "ZvtW2XIVfqa_",
    "outputId": "9b18e0d9-3323-44ce-b74b-4e7a04fc4eaa"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#21031) [(['one', '.', 'two'], '.'),(['.', 'three', '.'], 'four'),(['four', '.', 'five'], '.'),(['.', 'six', '.'], 'seven'),(['seven', '.', 'eight'], '.'),(['.', 'nine', '.'], 'ten'),(['ten', '.', 'eleven'], '.'),(['.', 'twelve', '.'], 'thirteen'),(['thirteen', '.', 'fourteen'], '.'),(['.', 'fifteen', '.'], 'sixteen')...]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L((tokens[i:i+3], tokens[i+3]) for i in range(0,len(tokens)-4,3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 654,
     "status": "ok",
     "timestamp": 1683920480575,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "GLkeIFFafwyb",
    "outputId": "650e6e18-31b2-400b-9299-6e72b1674bca"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(#21031) [(tensor([0, 1, 2]), 1),(tensor([1, 3, 1]), 4),(tensor([4, 1, 5]), 1),(tensor([1, 6, 1]), 7),(tensor([7, 1, 8]), 1),(tensor([1, 9, 1]), 10),(tensor([10,  1, 11]), 1),(tensor([ 1, 12,  1]), 13),(tensor([13,  1, 14]), 1),(tensor([ 1, 15,  1]), 16)...]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "seqs = L((tensor(nums[i:i+3]), nums[i+3]) for i in range(0,len(nums)-4,3))\n",
    "seqs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "executionInfo": {
     "elapsed": 6,
     "status": "ok",
     "timestamp": 1683920480576,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "OiZ42KiSf4xZ"
   },
   "outputs": [],
   "source": [
    "bs = 64\n",
    "cut = int(len(seqs) * 0.8)\n",
    "dls = DataLoaders.from_dsets(seqs[:cut], seqs[cut:], bs=64, shuffle=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 5,
     "status": "ok",
     "timestamp": 1683920480576,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "KgdGMBpdsshQ",
    "outputId": "4a1d8ce2-abda-447f-ff94-5bc5e2dbfd0e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([64, 3]), torch.Size([64]))"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x, y = dls.one_batch()\n",
    "x.shape, y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ey8OWvHTtdPQ"
   },
   "source": [
    "## Dot product attention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LcFJaSnwgfuz"
   },
   "source": [
    "![Multi head attention](https://github.com/fastai/course-v3/blob/master/nbs/dl2/images/attention.png?raw=1 \"credit: Attention is all you need\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tB8UKIdlsshQ"
   },
   "source": [
    "The core idea behind Transformers is Attention. Since the release of famous paper [Attention is All You Need](https://arxiv.org/abs/1706.03762) transformers has become most popular architecture for language modelling. \n",
    "\n",
    "There are a lot of great resourses explaining transformers architecture. I'll list some of those I found useful and comprehensive:\n",
    "1. [The Annotated Transformer](http://nlp.seas.harvard.edu/2018/04/03/attention.html) completes the original paper with code\n",
    "2. [Encoder-Decoder Model](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/Encoder_Decoder_Model.ipynb) notebook by huggingface gives mathemetically grounded explanation of how transformer encoder-decoder models work\n",
    "3. [The Illustrated GPT-2](https://jalammar.github.io/illustrated-gpt2/) one of the great blogposts by Jay Alammar visualizing generative language modelling on exaple of GPT-2\n",
    "4. [minGPT](https://github.com/karpathy/minGPT) cool repo by A. Karpathy providing clear minimal implementation of GPT model\n",
    "\n",
    "There exist multiple attention mechanisms. The particular one used in the original transformer paper is Scaled Dot Product attention.\n",
    "Given query vector for particular token we will compare it with a key vector for each token in a sequence and decide how much value vectors of those will effect resulting representetion of the token of interest. One way to view this from a linguistic prospective is: a key is a question each word respondes to, value is information that word represent and a query is related to what every word was looking to combine with.\n",
    "\n",
    "Mathemetically we can compute attention for all _q_, _k_, _v_ in a matrix form:\n",
    "\n",
    "$$\\textbf {Attention}(Q,K,V) = \\textbf {softmax}({QK^T\\over\\sqrt d_k})V $$\n",
    "\n",
    "Note that dot product $QK^T$ results in matrix of shape (seq_len x seq_len). Then it is devided by $ \\sqrt d_k$ to compensate the fact, that longer sequences will have larger dot product. $ \\textbf{softmax}$ is applied to rescale the attention matrix to be betwin 0 and 1. When multiplied by $V$ it produces a matrix of the same shape as $V$ (seq_len x dv).\n",
    "\n",
    "So where those _q_, _k_, _v_ come from. Well that's fairly straitforward queries are culculated from the embeddings of tokens we want to find representation for by simple linear projection. Keys and values are calculated from the embeddings of context tokens. In case of self attention all of them come from the original sequence."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Hoz346ussshU"
   },
   "source": [
    "To evaluete the model performance we need to compare it to some baseline. Let's see what would be the accuracy if of the model which would always predict most common token."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 375,
     "status": "ok",
     "timestamp": 1683920510695,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "Hf16xi3CsshU",
    "outputId": "32190184-4612-4745-cd12-ec3782c9aec5"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor(29), 'thousand', 0.15165200855716662)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n,counts = 0,torch.zeros(len(vocab))\n",
    "for x,y in dls.valid:\n",
    "    n += y.shape[0]\n",
    "    for i in range_of(vocab): counts[i] += (y==i).long().sum()\n",
    "idx = torch.argmax(counts)\n",
    "idx, vocab[idx.item()], counts[idx].item()/n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qOxKN6XcsshU"
   },
   "source": [
    "As you can see, always predicting \"thousand\" which turn out to be the most common token in the dataset would result in ~15% accuracy. Our simple transformer does much better then that. It feels promising, so let's try to improve the architecture and check if we can get better results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "id": "jlSYyLCrsshV"
   },
   "source": [
    "### Multihead attention"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true,
    "id": "j7qA-aotsshW"
   },
   "source": [
    "A structured sequence may comprise multiple distinctive kinds of relationships. Our model is forced to learn only one way in which queries, keys and values are constructed from the original token embedding. To remove this limitation we can modify attention layer include multiple heads which would correspond to extracting different kinds of relationships between tokens. The MultiHeadAttention layer consits of several heads each of those is similar to SelfAttention layer we made before. To keep computational cost of the multi-head layer we set $d_k = d_v = d_{model}/n_h$, where $n_h$ is number of heads."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7nKA2px-sshZ"
   },
   "source": [
    "### Arranging data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZxyhwqVHssha"
   },
   "source": [
    "Unlike RNN the tranformer is not a stateful model. This means it treats each sequence indepently and can only attend within fixed length context. This limitation was addressed by authors of [Transformer-XL paper](https://arxiv.org/abs/1901.02860) where adding a segment-level recurrence mechanism and a novel positional encoding scheme were proposed to enable capturing long-term dependencies. I will not go into details of TransformerXL architecture here. As we shell see stateless transformer can also learn a lot about the structure of our data.\n",
    "\n",
    "One thing to note in this case is that we don't need to maintain the structure of the data outside of the sequences, so we can shuffle the sequences randomly in the dataloader."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 212,
     "status": "ok",
     "timestamp": 1683920510904,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "O3_tjztpssha",
    "outputId": "8de20c39-1b0c-4c1c-fbf6-87c12149d128"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([64, 16]), torch.Size([64, 16]))"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sl = 16\n",
    "seqs = L((tensor(nums[i:i+sl]), tensor(nums[i+1:i+sl+1]))\n",
    "         for i in range(0,len(nums)-sl-1,sl))\n",
    "cut = int(len(seqs) * 0.8)\n",
    "dls = DataLoaders.from_dsets(seqs[:cut], seqs[cut:],\n",
    "                             bs=bs, drop_last=True, shuffle=True)\n",
    "xb, yb = dls.one_batch()\n",
    "xb.shape, yb.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 6,
     "status": "ok",
     "timestamp": 1683920510904,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "6o8UZeyHssha",
    "outputId": "d4526dd6-08d3-4239-c6e2-d34c80bcab36"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(#16) ['one','.','two','.','three','.','four','.','five','.'...],\n",
       " (#16) ['.','two','.','three','.','four','.','five','.','six'...]]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[L(vocab[o] for o in s) for s in seqs[0]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6S2x1m1Wssha"
   },
   "source": [
    "### Positional encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3W1jUH7Ossha"
   },
   "source": [
    "Before we did average pooling over seq_len dimension. Our model didn't care about the order of the tokens at all. But actually order of the tokens in a sentence matter a lot. In our case `one hundred two` and `two hundred one` are pretty different and `hundred one two` doesn't make sense.\n",
    "\n",
    "To encorporate positional information into the model authors of the transformer architecture proposed to use positional encodings in addition to regular token embeddings. Positional encodings may be learned, but it's also possible to use hardcoded encodings. For instance encodings may be composed of sin and cos.\n",
    "In this way each position in a sequence will get unique vector associated with it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "executionInfo": {
     "elapsed": 3,
     "status": "ok",
     "timestamp": 1683920510905,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "9ozy8DEcsshb"
   },
   "outputs": [],
   "source": [
    "class PositionalEncoding(Module):\n",
    "    def __init__(self, d):\n",
    "        self.register_buffer('freq', 1/(10000 ** (torch.arange(0., d, 2.)/d)))\n",
    "        self.scale = d**0.5\n",
    "    def forward(self, x):\n",
    "        device = x.device\n",
    "        pos_enc = torch.cat([torch.sin(torch.outer(torch.arange(x.size(1), device=device), self.freq)),\n",
    "                             torch.cos(torch.outer(torch.arange(x.size(1), device=device), self.freq))],\n",
    "                            axis=-1)\n",
    "        return x*self.scale + pos_enc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 395
    },
    "executionInfo": {
     "elapsed": 609,
     "status": "ok",
     "timestamp": 1683920511511,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "klMWBmezsshc",
    "outputId": "255ae147-eeb2-44a7-84f2-55e4e84b5025"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABRoAAAF1CAYAAACHyFa3AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAA9hAAAPYQGoP6dpAAA/s0lEQVR4nO3deZhcZZk34F91J+lsnYYA2SCBsO/7IovDlpFNBHEElYGIgIogYBQhjgiCEPUbHEQRBITAKKKOgsgMYCayiewhbDKQsCUsISAknQSykK7vD6W1TYJp6u10N7nv66orqXNO/eo5XfV2nXr6LJVqtVoNAAAAAEAN6jq7AAAAAACg+9NoBAAAAABqptEIAAAAANRMoxEAAAAAqJlGIwAAAABQM41GAAAAAKBmGo0AAAAAQM00GgEAAACAmmk0AgAAAAA102gEAAAAAGqm0ZjkwgsvzDrrrJPevXtnp512yr333tvZJUGHuf3223PggQdm2LBhqVQque6669rMr1ar+drXvpahQ4emT58+GTVqVKZMmdI5xUIHGDduXHbYYYc0NjZm0KBBOfjgg/PEE0+0WWb+/Pk5/vjjs9pqq6V///75yEc+kpdffrmTKobyLrroomy55ZYZMGBABgwYkJ133jk33nhj63xjgJXRN7/5zVQqlZx88smt04wF3uvOPPPMVCqVNreNN964db4xwMrghRdeyL/+679mtdVWS58+fbLFFlvk/vvvb53vO3L7rPSNxp/97GcZM2ZMzjjjjEyaNClbbbVV9tlnn8ycObOzS4MOMW/evGy11Va58MILlzr/29/+di644IJcfPHFueeee9KvX7/ss88+mT9//gquFDrGbbfdluOPPz533313JkyYkEWLFuUDH/hA5s2b17rMF77whfzmN7/JL37xi9x222158cUXc8ghh3Ri1VDWWmutlW9+85t54IEHcv/992evvfbKQQcdlMceeyyJMcDK57777ssPf/jDbLnllm2mGwusDDbbbLO89NJLrbff//73rfOMAd7rXn/99ey6667p2bNnbrzxxvzxj3/Meeedl1VXXbV1Gd+R26m6kttxxx2rxx9/fOv9xYsXV4cNG1YdN25cJ1YFK0aS6rXXXtt6v6WlpTpkyJDq//t//6912qxZs6oNDQ3Vn/70p51QIXS8mTNnVpNUb7vttmq1+uf3fM+ePau/+MUvWpd5/PHHq0mqd911V2eVCR1u1VVXrV522WXGACudOXPmVDfYYIPqhAkTqrvvvnv1pJNOqlarPg9YOZxxxhnVrbbaaqnzjAFWBqeeemp1t912W+Z835Hbb6Xeo3HhwoV54IEHMmrUqNZpdXV1GTVqVO66665OrAw6xzPPPJMZM2a0GRNNTU3ZaaedjAnes2bPnp0kGThwYJLkgQceyKJFi9qMg4033jgjRowwDnhPWrx4ca655prMmzcvO++8szHASuf444/PAQcc0OY9n/g8YOUxZcqUDBs2LOuuu24OP/zwTJs2LYkxwMrh+uuvz/bbb5+PfvSjGTRoULbZZptceumlrfN9R26/lbrR+Oqrr2bx4sUZPHhwm+mDBw/OjBkzOqkq6Dxvv++NCVYWLS0tOfnkk7Prrrtm8803T/LncdCrV6+sssoqbZY1DniveeSRR9K/f/80NDTks5/9bK699tpsuummxgArlWuuuSaTJk3KuHHjlphnLLAy2GmnnTJ+/PjcdNNNueiii/LMM8/k/e9/f+bMmWMMsFJ4+umnc9FFF2WDDTbIzTffnOOOOy4nnnhirrzyyiS+I78bPTq7AADoLMcff3weffTRNucigpXFRhttlMmTJ2f27Nn5r//6r4wePTq33XZbZ5cFK8z06dNz0kknZcKECendu3dnlwOdYr/99mv9/5Zbbpmddtopa6+9dn7+85+nT58+nVgZrBgtLS3Zfvvtc+655yZJttlmmzz66KO5+OKLM3r06E6urntaqfdoXH311VNfX7/EVbNefvnlDBkypJOqgs7z9vvemGBlcMIJJ+SGG27ILbfckrXWWqt1+pAhQ7Jw4cLMmjWrzfLGAe81vXr1yvrrr5/tttsu48aNy1ZbbZXvfve7xgArjQceeCAzZ87Mtttumx49eqRHjx657bbbcsEFF6RHjx4ZPHiwscBKZ5VVVsmGG26YqVOn+jxgpTB06NBsuummbaZtsskmracQ8B25/VbqRmOvXr2y3XbbZeLEia3TWlpaMnHixOy8886dWBl0jpEjR2bIkCFtxkRzc3PuueceY4L3jGq1mhNOOCHXXnttfve732XkyJFt5m+33Xbp2bNnm3HwxBNPZNq0acYB72ktLS1ZsGCBMcBKY++9984jjzySyZMnt9623377HH744a3/NxZY2cydOzdPPfVUhg4d6vOAlcKuu+6aJ554os20J598MmuvvXYS35HfjZX+0OkxY8Zk9OjR2X777bPjjjvm/PPPz7x583LUUUd1dmnQIebOnZupU6e23n/mmWcyefLkDBw4MCNGjMjJJ5+cb3zjG9lggw0ycuTInH766Rk2bFgOPvjgzisaCjr++ONz9dVX59e//nUaGxtbz63S1NSUPn36pKmpKUcffXTGjBmTgQMHZsCAAfn85z+fnXfeOe973/s6uXooY+zYsdlvv/0yYsSIzJkzJ1dffXVuvfXW3HzzzcYAK43GxsbW8/O+rV+/fllttdVapxsLvNd96UtfyoEHHpi11147L774Ys4444zU19fn4x//uM8DVgpf+MIXsssuu+Tcc8/NoYcemnvvvTeXXHJJLrnkkiRJpVLxHbm9Ovuy113B9773veqIESOqvXr1qu64447Vu+++u7NLgg5zyy23VJMscRs9enS1Wq1WW1paqqeffnp18ODB1YaGhuree+9dfeKJJzq3aChoae//JNUrrriidZk333yz+rnPfa666qqrVvv27Vv98Ic/XH3ppZc6r2go7FOf+lR17bXXrvbq1au6xhprVPfee+/qb3/729b5xgArq91337160kkntd43FnivO+yww6pDhw6t9urVq7rmmmtWDzvssOrUqVNb5xsDrAx+85vfVDfffPNqQ0NDdeONN65ecsklbeb7jtw+lWq1Wu2kHicAAAAA8B6xUp+jEQAAAAAoQ6MRAAAAAKiZRiMAAAAAUDONRgAAAACgZhqNAAAAAEDNNBoBAAAAgJppNP7FggULcuaZZ2bBggWdXQp0CmMAjANIjANIjAMwBsA4eLcq1Wq12tlFdAXNzc1pamrK7NmzM2DAgM4uB1Y4YwCMA0iMA0iMAzAGwDh4t+zRCAAAAADUTKMRAAAAAKhZj84uoKO1tLTkxRdfTGNjYyqVyjKXa25ubvMvrGyMATAOIDEOIDEOwBgA4+DvVavVzJkzJ8OGDUtd3bL3W3zPn6Px+eefz/Dhwzu7DAAAAADo1qZPn5611lprmfPf83s0NjY2JknWOuOrqevdu5OrAQBgaR76yOWdXQIAAMvQPLcla2/7bGufbVne843Gtw+XruvdW6MRAKCLGtDo1OEAAF3dO52WMHExGAAAAACgAI1GAAAAAKBmGo0AAAAAQM26RaPxwgsvzDrrrJPevXtnp512yr333tvZJQEAAAAAf6PLNxp/9rOfZcyYMTnjjDMyadKkbLXVVtlnn30yc+bMzi4NAAAAAPiLLt9o/M53vpNjjz02Rx11VDbddNNcfPHF6du3by6//PLOLg0AAAAA+Isu3WhcuHBhHnjggYwaNap1Wl1dXUaNGpW77rprqY9ZsGBBmpub29wAAAAAgI7VpRuNr776ahYvXpzBgwe3mT548ODMmDFjqY8ZN25cmpqaWm/Dhw9fEaUCAAAAwEqtSzca342xY8dm9uzZrbfp06d3dkkAAAAA8J7Xo7MLeCerr7566uvr8/LLL7eZ/vLLL2fIkCFLfUxDQ0MaGhpWRHkAAAAAwF906T0ae/Xqle222y4TJ05sndbS0pKJEydm55137sTKAAAAAIC/1aX3aEySMWPGZPTo0dl+++2z44475vzzz8+8efNy1FFHdXZpAAAAAMBfdPlG42GHHZZXXnklX/va1zJjxoxsvfXWuemmm5a4QAwAAAAA0Hm6fKMxSU444YSccMIJnV0GAAAAALAMXfocjQAAAABA96DRCAAAAADUrFscOl3C+A9enP6NtfdV73tzZIFq/urhecOLZT09Z7ViWa/M61csK0nemN9QLGvhgrJv25ZFBfvtJbNaKuWykqSlXFSlZG3VclHF86plX4NK6XUtpavWBaxUTnhhp2JZuw14slhWkmzca0axrNXrFxXLaqyrL5aVJL0r5bax6grvz1CXcp/J9RX7WgBAR/EpCwAAAADUTKMRAAAAAKiZRiMAAAAAUDONRgAAAACgZhqNAAAAAEDNNBoBAAAAgJppNAIAAAAANdNoBAAAAABqptEIAAAAANRMoxEAAAAAqJlGIwAAAABQM41GAAAAAKBmGo0AAAAAQM00GgEAAACAmmk0AgAAAAA102gEAAAAAGqm0QgAAAAA1EyjEQAAAAComUYjAAAAAFCzSrVarXZ2ER2pubk5TU1NefSPg9LYWHtfda0e/QtU9VfT3ppbLOu++cOKZT0wb51iWUny5NxBxbJemjegWFaSzH6jT7GsBfN7FstavKjs3wGqJfPeKphV+DdQpaVSLqylXFSSsutaLbeelff0p8DfWFnWE7qpNe4vl/Xy+8v+At9ik2nFsv559ceLZW3V57liWUmyZn257dKmuoKfx0n6VsptY/Ws1BfLqkvZ9ayv2A8EgK6peU5LVt3w6cyePTsDBiy7L+OTDAAAAAComUYjAAAAAFAzjUYAAAAAoGYajQAAAABAzTQaAQAAAICadelG47hx47LDDjuksbExgwYNysEHH5wnnniis8sCAAAAAP5Ol2403nbbbTn++ONz9913Z8KECVm0aFE+8IEPZN68eZ1dGgAAAADwN3p0dgHv5Kabbmpzf/z48Rk0aFAeeOCB/NM//VMnVQUAAAAA/L0u3Wj8e7Nnz06SDBw4cJnLLFiwIAsWLGi939zc3OF1AQAAAMDKrksfOv23WlpacvLJJ2fXXXfN5ptvvszlxo0bl6amptbb8OHDV2CVAAAAALBy6jaNxuOPPz6PPvporrnmmndcbuzYsZk9e3brbfr06SuoQgAAAABYeXWLQ6dPOOGE3HDDDbn99tuz1lprveOyDQ0NaWhoWEGVAQAAAABJF280VqvVfP7zn8+1116bW2+9NSNHjuzskgAAAACApejSjcbjjz8+V199dX7961+nsbExM2bMSJI0NTWlT58+nVwdAAAAAPC2Ln2OxosuuiizZ8/OHnvskaFDh7befvazn3V2aQAAAADA3+jSezRWq9XOLgEAAAAAWA5deo9GAAAAAKB70GgEAAAAAGrWpQ+dLmm/249LXZ/eNef8YLcfF6jmr/btWy5rYN+ZxbKG9JhVLCtJVu+5frGsx3oOK5aVJM/2GFgs67X6ci/oG/N7FctKkrfq6otltZT8E8VblYJhSbVgXCVla0vJs0G0lAurll7Pgiolf2alV9PZPaCopqvvKZbVOG3rYllJ8uSodYtlPb9NU7Gsfx6+arGsJNm5/9RiWRv0fKVYVpIMrFtYLKuxrtxXoJ6VcttXSbrsZ0t9xf4pACwfnxgAAAAAQM00GgEAAACAmmk0AgAAAAA102gEAAAAAGqm0QgAAAAA1EyjEQAAAAComUYjAAAAAFAzjUYAAAAAoGYajQAAAABAzTQaAQAAAICaaTQCAAAAADXTaAQAAAAAaqbRCAAAAADUTKMRAAAAAKiZRiMAAAAAUDONRgAAAACgZhqNAAAAAEDNNBoBAAAAgJr16OwCVpQNz5+bHvWLas75XP61QDV/9YPdflwsa9++xaKyVa/55cKSJFML59Feb6RXsay3iiUlLakvmJaixVXLRSVJKosr5cJK/pmopfCaVsutZ8GoVIq/oAWzStcG3VB15y2LZdX9fnKxrCQZnq2LZU3PasWyJmSjYllJkuEFs/oXzEqyQc9XCqaV21hoLL3bRsHPlh4Ft7EWV1uKZSVJfcX+LgDvVX7DAwAAAAA102gEAAAAAGqm0QgAAAAA1EyjEQAAAAComUYjAAAAAFCzbtVo/OY3v5lKpZKTTz65s0sBAAAAAP5Gt2k03nffffnhD3+YLbfcsrNLAQAAAAD+TrdoNM6dOzeHH354Lr300qy66qqdXQ4AAAAA8He6RaPx+OOPzwEHHJBRo0Z1dikAAAAAwFL06OwC/pFrrrkmkyZNyn333bdcyy9YsCALFixovd/c3NxRpQEAAAAAf9Gl92icPn16TjrppPzkJz9J7969l+sx48aNS1NTU+tt+PDhHVwlAAAAANClG40PPPBAZs6cmW233TY9evRIjx49ctttt+WCCy5Ijx49snjx4iUeM3bs2MyePbv1Nn369E6oHAAAAABWLl360Om99947jzzySJtpRx11VDbeeOOceuqpqa+vX+IxDQ0NaWhoWFElAgAAAADp4o3GxsbGbL755m2m9evXL6utttoS0wEAAACAztOlD50GAAAAALqHLr1H49LceuutnV0CAAAAAPB37NEIAAAAANRMoxEAAAAAqJlGIwAAAABQs253jsZ3q+WZ6Wmp9Kw5Z52rtyhQzV99bY0PFcsattl/FsvaslfvYllJskHPecWy5vR+oVhWkrzR0qtY1puLa3+PvW3R4vpiWUmyuFopllVtKZhVsK4/5xX8+0m1XFSSlCytUvA1SNmXIEV/cCXft4XXs1Ly/VH6NSj83oUV4eUvLyyWtdZJaxXLSpKW+x4vljVo4JbFsl7sv2qxrCS5t8/axbKa6t8slpUkfSsLimX17DmrXFZ1cbGs0uoq5TY86spvLADwHmWPRgAAAACgZu3eo3Hx4sUZP358Jk6cmJkzZ6alpaXN/N/97nfFigMAAAAAuod2NxpPOumkjB8/PgcccEA233zzVCp2owcAAACAlV27G43XXHNNfv7zn2f//ffviHoAAAAAgG6o3edo7NWrV9Zff/2OqAUAAAAA6Kba3Wj84he/mO9+97upVl3SEgAAAAD4s+U6dPqQQw5pc/93v/tdbrzxxmy22Wbp2bNnm3m/+tWvylUHAAAAAHQLy9VobGpqanP/wx/+cIcUAwAAAAB0T8vVaLziiis6ug4AAAAAoBtr9zka99prr8yaNWuJ6c3Nzdlrr71K1AQAAAAAdDPtbjTeeuutWbhw4RLT58+fnzvuuKNIUQAAAABA97Jch04nycMPP9z6/z/+8Y+ZMWNG6/3Fixfnpptuypprrlm2OgAAAACgW1juRuPWW2+dSqWSSqWy1EOk+/Tpk+9973tFiwMAAAAAuoflbjQ+88wzqVarWXfddXPvvfdmjTXWaJ3Xq1evDBo0KPX19R1SJAAAAADQtS13o3HttddOkrS0tHRYMQAAAABA97Tcjca3XX/99UudXqlU0rt376y//voZOXJkzYUBAAAAAN1HuxuNBx98cCqVSqrVapvpb0+rVCrZbbfdct1112XVVVctVigAAAAA0HXVtfcBEyZMyA477JAJEyZk9uzZmT17diZMmJCddtopN9xwQ26//fb86U9/ype+9KWOqBcAAAAA6ILavUfjSSedlEsuuSS77LJL67S99947vXv3zqc//ek89thjOf/88/OpT32qaKG1evmTW6e+oXfNOYMvvKdANX81f4udimX959Cdi2V9ddAfimUlyWp1fYplrdvztWJZSTKzV2OxrNd79y2WNfethmJZSbJwcbmLNS1e3O6/USxTS7VSLCtJWlrK5VULZv05sGBUpVxYpVJ4PYsq+kMrl9XVlVzVgi8BvJM7t7+iWNbOh44plpUka/7HS8WyGh+aUSyraciaxbKS5LnVVi+W9WjfYcWykmRwz9nFsgbUzS+W1bvyZrGsJKlPufPh12VxsayelcIX/ayWW8/6SrntUgBq1+7fyk899VQGDBiwxPQBAwbk6aefTpJssMEGefXVV2uvDgAAAADoFtrdaNxuu+1yyimn5JVXXmmd9sorr+TLX/5ydthhhyTJlClTMnz48HJVAgAAAABdWrsPnf7Rj36Ugw46KGuttVZrM3H69OlZd9118+tf/zpJMnfu3Hz1q18tWykAAAAA0GW1u9G40UYb5Y9//GN++9vf5sknn2yd9s///M+pq/vzDpIHH3xw0SIBAAAAgK6t3Y3GJKmrq8u+++6bfffdt3Q9AAAAAEA39K4ajRMnTszEiRMzc+bMtLS0vWLY5ZdfXqSwt73wwgs59dRTc+ONN+aNN97I+uuvnyuuuCLbb7990ecBAAAAAN69djcav/71r+ess87K9ttvn6FDh6ZSqXREXUmS119/Pbvuumv23HPP3HjjjVljjTUyZcqUrLrqqh32nAAAAABA+7W70XjxxRdn/PjxOeKIIzqinja+9a1vZfjw4bniiitap40cObLDnxcAAAAAaJ+69j5g4cKF2WWXXTqiliVcf/312X777fPRj340gwYNyjbbbJNLL730HR+zYMGCNDc3t7kBAAAAAB2r3Y3GY445JldffXVH1LKEp59+OhdddFE22GCD3HzzzTnuuONy4okn5sorr1zmY8aNG5empqbW2/Dhw1dIrQAAAACwMmv3odPz58/PJZdckv/93//NlltumZ49e7aZ/53vfKdYcS0tLdl+++1z7rnnJkm22WabPProo7n44oszevTopT5m7NixGTNmTOv95uZmzUYAAAAA6GDtbjQ+/PDD2XrrrZMkjz76aJt5pS8MM3To0Gy66aZtpm2yySb55S9/uczHNDQ0pKGhoWgdAAAAAMA7a3ej8ZZbbumIOpZq1113zRNPPNFm2pNPPpm11157hdUAAAAAAPxj7T5H49umTp2am2++OW+++WaSpFqtFivqbV/4whdy991359xzz83UqVNz9dVX55JLLsnxxx9f/LkAAAAAgHev3Y3GP/3pT9l7772z4YYbZv/9989LL72UJDn66KPzxS9+sWhxO+ywQ6699tr89Kc/zeabb56zzz47559/fg4//PCizwMAAAAA1KbdjcYvfOEL6dmzZ6ZNm5a+ffu2Tj/ssMNy0003FS0uST74wQ/mkUceyfz58/P444/n2GOPLf4cAAAAAEBt2n2Oxt/+9re5+eabs9Zaa7WZvsEGG+S5554rVhgAAAAA0H20e4/GefPmtdmT8W2vvfaaqz0DAAAAwEqq3Y3G97///bnqqqta71cqlbS0tOTb3/529txzz6LFAQAAAADdQ7sPnf72t7+dvffeO/fff38WLlyYL3/5y3nsscfy2muv5c477+yIGos4/Ojfpnf/dq/uEib818YFqvmrNSfOKpZ17XZbFcs6oOmhYllJskeflmJZg+vf9cXSl2pEz9eKZb3c0FQs67VF/YplJcm8Rb2KZS1YVPtYettbb9UXy0qSan2lXNbicllJknLDIKkUrK3wahZV7ewClq1a8iXowusJK8r/vrl6sawdP/JwsawkeeF3GxXLeuvB/yuWtcqUNYplJcnc4eW2FaasXra2dfu9WixrjR7NxbIa6xYWy0qS3vXlNhZ6FtzwaCm8sVCXstt/AHQd7e7YbL755nnyySez22675aCDDsq8efNyyCGH5MEHH8x6663XETUCAAAAAF3cu9otqampKf/2b/9WuhYAAAAAoJtarkbjww8v/+EnW2655bsuBgAAAADonpar0bj11lunUqmkWn3nk0hVKpUsXry4SGEAAAAAQPexXI3GZ555pqPrAAAAAAC6seVqNK699todXQcAAAAA0I21+6rTAAAAAAB/T6MRAAAAAKiZRiMAAAAAUDONRgAAAACgZu+q0Thr1qxcdtllGTt2bF577bUkyaRJk/LCCy8ULQ4AAAAA6B6W66rTf+vhhx/OqFGj0tTUlGeffTbHHntsBg4cmF/96leZNm1arrrqqo6oEwAAAADowtq9R+OYMWPyyU9+MlOmTEnv3r1bp++///65/fbbixYHAAAAAHQP7W403nffffnMZz6zxPQ111wzM2bMKFIUAAAAANC9tLvR2NDQkObm5iWmP/nkk1ljjTWKFAUAAAAAdC/tbjR+6EMfyllnnZVFixYlSSqVSqZNm5ZTTz01H/nIR4oXCAAAAAB0fe1uNJ533nmZO3duBg0alDfffDO777571l9//TQ2Nuacc87piBoBAAAAgC6u3VedbmpqyoQJE3LnnXfmoYceyty5c7Pttttm1KhRHVEfAAAAANANtLvR+LZdd901u+66a8laOtRnmp7NgMZ278C5hPH/sn+Bav5q8IX3FMvq+8BOxbL+e+OtimUlyTYNfyiW1b/SUCwrSYb1mFMsa3CP2cWyZvRsKpaVJH/q2a9Y1ryevYplLXyrvlhWkixeXPs4f1ulvlosK0mqLZWCYQWjKmXXs1ItuJ4Fo4r+0JKk5Hp2ZSVXs/BLwHvLV646sljWvZ/5TrGsJNl5rzHFstZ8qNyg6v30K8WykqT/BmsWy/rTWo3FspLk6VVXL5Y1sqHcz21QfbntyCRprHuzWFbPtBTLqmv/gXDvnFdy26Nabj2TpL5Sdl0BVjbt/i164okn5oILLlhi+ve///2cfPLJJWoCAAAAALqZdjcaf/nLXy51T8Zddtkl//Vf/1WkKAAAAACge2l3o/FPf/pTmpqWPKRzwIABefXVV4sUBQAAAAB0L+1uNK6//vq56aablph+4403Zt111y1SFAAAAADQvbT7YjBjxozJCSeckFdeeSV77bVXkmTixIk577zzcv7555euDwAAAADoBtq9R+OnPvWpnHfeefnRj36UPffcM3vuuWd+/OMf56KLLsqxxx5btLjFixfn9NNPz8iRI9OnT5+st956Ofvss1OtumwlAAAAAHQl7d6jMUmOO+64HHfccXnllVfSp0+f9O/fv3RdSZJvfetbueiii3LllVdms802y/3335+jjjoqTU1NOfHEEzvkOQEAAACA9ntXjca3rbHGGqXqWKo//OEPOeigg3LAAQckSdZZZ5389Kc/zb333tuhzwsAAAAAtE+7D51++eWXc8QRR2TYsGHp0aNH6uvr29xK2mWXXTJx4sQ8+eSTSZKHHnoov//977Pffvst8zELFixIc3NzmxsAAAAA0LHavUfjJz/5yUybNi2nn356hg4dmkql0hF1JUlOO+20NDc3Z+ONN059fX0WL16cc845J4cffvgyHzNu3Lh8/etf77CaAAAAAIAltbvR+Pvf/z533HFHtt566w4op62f//zn+clPfpKrr746m222WSZPnpyTTz45w4YNy+jRo5f6mLFjx2bMmDGt95ubmzN8+PAOrxUAAAAAVmbtbjQOHz58hV31+ZRTTslpp52Wj33sY0mSLbbYIs8991zGjRu3zEZjQ0NDGhoaVkh9AAAAAMCftfscjeeff35OO+20PPvssx1QTltvvPFG6urallhfX5+WlpYOf24AAAAAYPm1e4/Gww47LG+88UbWW2+99O3bNz179mwz/7XXXitW3IEHHphzzjknI0aMyGabbZYHH3ww3/nOd/KpT32q2HMAAAAAALVrd6Px/PPP74Aylu573/teTj/99Hzuc5/LzJkzM2zYsHzmM5/J1772tRVWAwAAAADwj7W70biscyN2hMbGxpx//vkrtLkJAAAAALRfu8/RmCRPPfVUvvrVr+bjH/94Zs6cmSS58cYb89hjjxUtDgAAAADoHtrdaLztttuyxRZb5J577smvfvWrzJ07N0ny0EMP5YwzziheIAAAAADQ9bX70OnTTjst3/jGNzJmzJg0Nja2Tt9rr73y/e9/v2hxJX10yr7p0a+h5px1PzqlQDV/9eaPaq/pbYMmzS+W9b//tFGxrCQ5fNW7i2Vt3fCudsRdpoEF44b0mF0sa1rPN4plJcmAnuXeH809ehfLeqO+5z9eqB3q6spdlb6lrux7LXXVclnlVjOpVAqGJSkZV/BH1pVVS78EK8nPjfeWdb5X7siYV455q1hWkvTf6+ViWXXXDi+WtXj6i8WykqTx+cHFsma/0qtYVpI8P3SVYlkvNZbLWrPn68WykqSpZUGxrN715T4Mehbd8EiS+sJ5AHQV7f4W/cgjj+TDH/7wEtMHDRqUV199tUhRAAAAAED30u5G4yqrrJKXXnppiekPPvhg1lxzzSJFAQAAAADdS7sbjR/72Mdy6qmnZsaMGalUKmlpacmdd96ZL33pSznyyCM7okYAAAAAoItrd6Px3HPPzcYbb5zhw4dn7ty52XTTTfNP//RP2WWXXfLVr361I2oEAAAAALq4dl8MplevXrn00ktz+umn59FHH83cuXOzzTbbZIMNNuiI+gAAAACAbqDdjca3jRgxIiNGjChZCwAAAADQTbW70fipT33qHedffvnl77oYAAAAAKB7anej8fXXX29zf9GiRXn00Ucza9as7LXXXsUKAwAAAAC6j3Y3Gq+99tolprW0tOS4447LeuutV6QoAAAAAKB7afdVp5caUleXMWPG5D/+4z9KxAEAAAAA3UyRRmOSPPXUU3nrrbdKxQEAAAAA3Ui7D50eM2ZMm/vVajUvvfRS/vu//zujR48uVhgAAAAA0H20u9H44IMPtrlfV1eXNdZYI+edd94/vCI1AAAAAPDe1O5G4y233NIRdQAAAAAA3VixczQCAAAAACuvdu/RuM0226RSqSzXspMmTWp3QQAAAABA99PuRuO+++6bH/zgB9l0002z8847J0nuvvvuPPbYYznuuOPSp0+f4kUCAAAAAF1buxuNr7zySk488cScffbZbaafccYZmT59ei6//PJixQEAAAAA3UO7z9H4i1/8IkceeeQS0//1X/81v/zlL4sUBQAAAAB0L+3eo7FPnz658847s8EGG7SZfuedd6Z3797FCitt/gVD06Nn7fX99IffLVDNX310xxOKZfV6dHqxrDlT1iuWlSQPbLh2sazNer1QLCtJGut6FctarX5esazVe8wplpUkA3quUiyrd49FxbJ61pf7+SfJovp2/1pbpsV11WJZSVItmbec58pdvqxyUUlSrZRbz+U9J3DnKPh6VrvyesKKUenfv1jWvz6+5B/Fa/GNDa8rlnXG1scUy+o39ZliWUnS54Vy2zF9XlmlWFaSvNJc7v0xY2BTsaxZDX2LZSXJkPrmYlkLq+W21xoKfrYnSUtaCqYVvr5ptVxt9RXXXgVWPu3+Rn7yySfnuOOOy6RJk7LjjjsmSe65555cfvnlOf3004sXCAAAAAB0fe1uNJ522mlZd911893vfjc//vGPkySbbLJJrrjiihx66KHFCwQAAAAAur53dYzhoYceqqkIAAAAALR6VyeNmDVrVi677LJ85StfyWuvvZYkmTRpUl54oey58wAAAACA7qHdezQ+/PDDGTVqVJqamvLss8/mmGOOycCBA/OrX/0q06ZNy1VXXdURdQIAAAAAXVi792gcM2ZMPvnJT2bKlCltrjK9//775/bbb29X1u23354DDzwww4YNS6VSyXXXXddmfrVazde+9rUMHTo0ffr0yahRozJlypT2lgwAAAAAdLB2Nxrvu+++fOYzn1li+pprrpkZM2a0K2vevHnZaqutcuGFFy51/re//e1ccMEFufjii3PPPfekX79+2WeffTJ//vz2lg0AAAAAdKB2Hzrd0NCQ5ubmJaY/+eSTWWONNdqVtd9++2W//fZb6rxqtZrzzz8/X/3qV3PQQQclSa666qoMHjw41113XT72sY+1t3QAAAAAoIO0e4/GD33oQznrrLOyaNGiJEmlUsm0adNy6qmn5iMf+Uixwp555pnMmDEjo0aNap3W1NSUnXbaKXfdddcyH7dgwYI0Nze3uQEAAAAAHavdjcbzzjsvc+fOzaBBg/Lmm29m9913z/rrr5/Gxsacc845xQp7+zDswYMHt5k+ePDgdzxEe9y4cWlqamq9DR8+vFhNAAAAAMDStfvQ6aampkyYMCF33nlnHnroocydOzfbbrttmz0PO9PYsWMzZsyY1vvNzc2ajQAAAADQwdrdaHzbrrvuml133bVkLW0MGTIkSfLyyy9n6NChrdNffvnlbL311st8XENDQxoaGjqsLgAAAABgSct96PRdd92VG264oc20q666KiNHjsygQYPy6U9/OgsWLChW2MiRIzNkyJBMnDixdVpzc3Puueee7LzzzsWeBwAAAACo3XI3Gs8666w89thjrfcfeeSRHH300Rk1alROO+20/OY3v8m4cePa9eRz587N5MmTM3ny5CR/vgDM5MmTM23atFQqlZx88sn5xje+keuvvz6PPPJIjjzyyAwbNiwHH3xwu54HAAAAAOhYy33o9OTJk3P22We33r/mmmuy00475dJLL02SDB8+PGeccUbOPPPM5X7y+++/P3vuuWfr/bfPrTh69OiMHz8+X/7ylzNv3rx8+tOfzqxZs7LbbrvlpptuSu/evZf7OQAAAACAjrfcjcbXX3+9zRWgb7vttuy3336t93fYYYdMnz69XU++xx57pFqtLnN+pVLJWWedlbPOOqtduQAAAADAirXch04PHjw4zzzzTJJk4cKFmTRpUt73vve1zp8zZ0569uxZvkIAAAAAoMtb7kbj/vvvn9NOOy133HFHxo4dm759++b9739/6/yHH3446623XocUCQAAAAB0bct96PTZZ5+dQw45JLvvvnv69++fK6+8Mr169Wqdf/nll+cDH/hAhxRZQsNvJ6VHpfY9Lhdl2Yd6vxvT924olrXOra8Uy2p6cv1iWUnyh93K5R3Y/6liWUkyqL5fsayBdXPLZfUol5UkjT3mF8vq22NhsaxePcqec3VBfUuxrEpd2fFeqZTLqpasraVgYfBOSr/Vyg5ROtn/fXF4sazVflr2zbb3uMXFso7bdrn/zv8Prfc/fYtlJUndjD8Vy+ozs6lYVpK8Mqvc9sKM+Y3Fsv7Ur3+xrCSZVy13hFhTdVGxrJZ3ON3Vu7G44C/wOpsxAF3KcjcaV1999dx+++2ZPXt2+vfvn/r6+jbzf/GLX6R//7IftAAAAABA97Dcjca3NTUt/a+TAwcOrLkYAAAAAKB7KnfsBgAAAACw0tJoBAAAAABqptEIAAAAANRMoxEAAAAAqJlGIwAAAABQM41GAAAAAKBmGo0AAAAAQM00GgEAAACAmmk0AgAAAAA102gEAAAAAGqm0QgAAAAA1EyjEQAAAAComUYjAAAAAFAzjUYAAAAAoGYajQAAAABAzTQaAQAAAICa9ejsAlaUBftsm8U9e9ecc+SUNQtU81eb7T61WNabffsWy1p1yoJiWUky6eW1imW9OLjs23ZQfbmsxrpKsaxV6t4olpUkjfXzi2X17bGoWFZD/eJiWUkyr66lWFZdwawkaakr+LedSrn3WirVcllJ4drKRaXwanZl1S789oBl+eXB3y2W9ZWvf6BYVpI8c9bcYlmrbTOzWFZl2OBiWUmyePqLxbL6vvJWsawkqZ9Vbvtv5huNxbJeb+pXLCtJ5rT0KZY1v67ctt+iwh+iPVNyG6vgxjwANbNHIwAAAABQM41GAAAAAKBmGo0AAAAAQM00GgEAAACAmmk0AgAAAAA169RG4+23354DDzwww4YNS6VSyXXXXdc6b9GiRTn11FOzxRZbpF+/fhk2bFiOPPLIvPhiuavhAQAAAABldGqjcd68edlqq61y4YUXLjHvjTfeyKRJk3L66adn0qRJ+dWvfpUnnngiH/rQhzqhUgAAAADgnfTozCffb7/9st9++y11XlNTUyZMmNBm2ve///3suOOOmTZtWkaMGLEiSgQAAAAAlkO3Okfj7NmzU6lUssoqq3R2KQAAAADA3+jUPRrbY/78+Tn11FPz8Y9/PAMGDFjmcgsWLMiCBQta7zc3N6+I8gAAAABgpdYt9mhctGhRDj300FSr1Vx00UXvuOy4cePS1NTUehs+fPgKqhIAAAAAVl5dvtH4dpPxueeey4QJE95xb8YkGTt2bGbPnt16mz59+gqqFAAAAABWXl360Om3m4xTpkzJLbfcktVWW+0fPqahoSENDQ0roDoAAAAA4G2d2micO3dupk6d2nr/mWeeyeTJkzNw4MAMHTo0//Iv/5JJkyblhhtuyOLFizNjxowkycCBA9OrV6/OKhsAAAAA+Dud2mi8//77s+eee7beHzNmTJJk9OjROfPMM3P99dcnSbbeeus2j7vllluyxx57rKgyAQAAAIB/oFMbjXvssUeq1eoy57/TPAAAAACg6+jyF4MBAAAAALo+jUYAAAAAoGYajQAAAABAzTr1HI0rUt8TX0yPfg0158z42doFqvmr8af9R7GsUzf8VLGshqdfKZaVJLNeGFos68lNBhXLSpIter1eLKtvpWexrAF184tlJUlT/ZvFsvrULyqW1VD/VrGsJOlR11Isq66u7HliKwXzqpViUUnJrMJ51Uq5n1mlUnhFnUYYilq94GdLdf6CYllJcu6MfYplHbPOncWyfrzeB4tlJUmvqc8Uy2p4pex2TK9ZjcWyXn+jT7GsVxf1L5aVJPNaav++8rZFBfcpWVwtu722uODne0vKbfslSV3qi2Utrparrb5iHyGge/DbCgAAAAComUYjAAAAAFAzjUYAAAAAoGYajQAAAABAzTQaAQAAAICaaTQCAAAAADXTaAQAAAAAaqbRCAAAAADUTKMRAAAAAKiZRiMAAAAAUDONRgAAAACgZhqNAAAAAEDNNBoBAAAAgJppNAIAAAAANdNoBAAAAABqptEIAAAAANRMoxEAAAAAqJlGIwAAAABQsx6dXcCK8rP1b86Axtr7qgf+cp8C1fzVll/rXSzr1W2aimWtdvVTxbKSpO/04cWyHntzrWJZSbJ/35eLZfWp9CqW1VQ3v1hWkjTWv1ksq1+PBcWyetYvLpaVJD3qW4pl1dVVi2UlSaVSMqxgbUULW4kU/bGVfa+lupK8piVXs/BLQPu9f+JJxbLWPKDsJu4td5R7s51z2G+LZX1/g3LbHUkyaEJ9saz61+YWy0qShlmNxbKa5zYUy3ptYd9iWUkyp6Xcd4P51XKv56K8VSyrq2sp+IFQV3ZjAaBbsEcjAAAAAFAzjUYAAAAAoGYajQAAAABAzTQaAQAAAICaaTQCAAAAADXr1Ebj7bffngMPPDDDhg1LpVLJddddt8xlP/vZz6ZSqeT8889fYfUBAAAAAMunUxuN8+bNy1ZbbZULL7zwHZe79tprc/fdd2fYsGErqDIAAAAAoD16dOaT77ffftlvv/3ecZkXXnghn//853PzzTfngAMOWEGVAQAAAADt0amNxn+kpaUlRxxxRE455ZRsttlmy/WYBQsWZMGCBa33m5ubO6o8AAAAAOAvuvTFYL71rW+lR48eOfHEE5f7MePGjUtTU1Prbfjw4R1YIQAAAACQdOFG4wMPPJDvfve7GT9+fCqVynI/buzYsZk9e3brbfr06R1YJQAAAACQdOFG4x133JGZM2dmxIgR6dGjR3r06JHnnnsuX/ziF7POOuss83ENDQ0ZMGBAmxsAAAAA0LG67DkajzjiiIwaNarNtH322SdHHHFEjjrqqE6qCgAAAABYmk5tNM6dOzdTp05tvf/MM89k8uTJGThwYEaMGJHVVlutzfI9e/bMkCFDstFGG63oUgEAAACAd9Cpjcb7778/e+65Z+v9MWPGJElGjx6d8ePHd1JVAAAAAEB7dWqjcY899ki1Wl3u5Z999tmOKwYAAAAAeNe67MVgAAAAAIDuQ6MRAAAAAKiZRiMAAAAAULNOPUfjinTRrHXT+63aV7dl1uwC1fzVvQsWFcv603YtxbIGXrGgWFaS9H9++c/F+Y881jy0WFaSzFr1rWJZ/Xv0LpbVr67c65kkjXXzi2X1rVtYLKt3fbkxkCQ96xcXy6qrK/e+TZJKpVxepWBt1YJ1/VmlS0YVzUqS0j82WMlt8u3mYlnPfaNXsawkWeuyvsWyBn2iX7Gs2RuW+8xLkqH9y9WWWeVezyRpmDW4WFbLvJ7FsmYtLPfeSJI5i/sUy5pfLbeei6plvxssqpbbzu1Z+AO5rvT2AsBKxh6NAAAAAEDNNBoBAAAAgJppNAIAAAAANdNoBAAAAABqptEIAAAAANRMoxEAAAAAqJlGIwAAAABQM41GAAAAAKBmGo0AAAAAQM00GgEAAACAmmk0AgAAAAA102gEAAAAAGqm0QgAAAAA1EyjEQAAAAComUYjAAAAAFAzjUYAAAAAoGYajQAAAABAzXp0dgEdrVqtJknmz32rSN5b1UVFct42b05LsayWN+cXyyq9nosXlqtt0byFxbKSZE7B16C5R7msOW+Vy0qSN+YvLpa14I1y74/Sr+db8xYUy1r8Ztm/xbQUfA1a3ipXW3VR4b85Fawt1XJRlZZKubAkKTlEC67nn/PKrWvhn1rXVfo1oN3eWlzw9/cbZV/Qtwr+nmzuott+SfJWteBnckvZ16DktmTLm+Veg9LbMfN7l/nOkiTzCm5Lzq0vu11aV1fu0+WtStnaelbK1VZX8FO0fqX5QAa6qua5f/59+3afbVkq1X+0RDf3/PPPZ/jw4Z1dBgAAAAB0a9OnT89aa621zPnv+UZjS0tLXnzxxTQ2NqbyDn+dam5uzvDhwzN9+vQMGDBgBVYIXYMxAMYBJMYBJMYBGANgHPy9arWaOXPmZNiwYamrW/bRHu/5Q6fr6uresdP69wYMGOANxErNGADjABLjABLjAIwBMA7+VlNT0z9cxsVgAAAAAICaaTQCAAAAADXTaPyLhoaGnHHGGWloaOjsUqBTGANgHEBiHEBiHIAxAMbBu/WevxgMAAD/2JlnnpnrrrsukydPLpr77LPPZuTIkXnwwQez9dZbL3WZW2+9NXvuuWdef/31rLLKKhk/fnxOPvnkzJo1q2gt7dEVagAA6G7s0QgA0I188pOfTKVSWeK27777dnZpxRx22GF58sknV/oaAAC6m/f8VacBAN5r9t1331xxxRVtpr2XDuvp06dP+vTps9LXAADQ3dijEQCgm2loaMiQIUPa3FZdddXW+ZVKJT/84Q/zwQ9+MH379s0mm2ySu+66K1OnTs0ee+yRfv36ZZdddslTTz21RPYPf/jDDB8+PH379s2hhx6a2bNnt5l/2WWXZZNNNknv3r2z8cYb5wc/+EGb+ffee2+22Wab9O7dO9tvv30efPDBJZ7jf/7nf7LhhhumT58+2XPPPfPss8+2mT9+/PisssoqrffPPPPMbL311vnP//zPrLPOOmlqasrHPvaxzJkzp3WZOXPm5PDDD0+/fv0ydOjQ/Md//Ef22GOPnHzyycv8OT700EPZc88909jYmAEDBmS77bbL/fffv9Qa1llnnaXuSfq26dOn59BDD80qq6ySgQMH5qCDDlpivQAA3us0GgEA3oPOPvvsHHnkkZk8eXI23njjfOITn8hnPvOZjB07Nvfff3+q1WpOOOGENo+ZOnVqfv7zn+c3v/lNbrrppjz44IP53Oc+1zr/Jz/5Sb72ta/lnHPOyeOPP55zzz03p59+eq688sokydy5c/PBD34wm266aR544IGceeaZ+dKXvtTmOaZPn55DDjkkBx54YCZPnpxjjjkmp5122j9cn6eeeirXXXddbrjhhtxwww257bbb8s1vfrN1/pgxY3LnnXfm+uuvz4QJE3LHHXdk0qRJ75h5+OGHZ6211sp9992XBx54IKeddlp69uy51GXvu+++vPTSS3nppZfy/PPP533ve1/e//73J0kWLVqUffbZJ42Njbnjjjty5513pn///tl3332zcOHCf7huAADvFQ6dBgDoZm644Yb079+/zbSvfOUr+cpXvtJ6/6ijjsqhhx6aJDn11FOz88475/TTT88+++yTJDnppJNy1FFHtcmYP39+rrrqqqy55ppJku9973s54IADct5552XIkCE544wzct555+WQQw5JkowcOTJ//OMf88Mf/jCjR4/O1VdfnZaWlvzoRz9K7969s9lmm+X555/Pcccd1/ocF110UdZbb72cd955SZKNNtoojzzySL71rW+94zq3tLRk/PjxaWxsTJIcccQRmThxYs4555zMmTMnV155Za6++ursvffeSZIrrrgiw4YNe8fMadOm5ZRTTsnGG2+cJNlggw2Wuewaa6zR+v+TTjopL730Uu67774kyc9+9rO0tLTksssua93L8Yorrsgqq6ySW2+9NR/4wAfesQ4AgPcKjUYAgG5mzz33zEUXXdRm2sCBA9vc33LLLVv/P3jw4CTJFlts0Wba/Pnz09zcnAEDBiRJRowY0dpkTJKdd945LS0teeKJJ9LY2JinnnoqRx99dI499tjWZd566600NTUlSR5//PFsueWW6d27d5uMv/X4449np512ajPt75dZmnXWWae1yZgkQ4cOzcyZM5MkTz/9dBYtWpQdd9yxdX5TU1M22mijd8wcM2ZMjjnmmPznf/5nRo0alY9+9KNZb7313vExl1xySX70ox/lD3/4Q2vz8aGHHsrUqVPb1Jf8uXG7tMPTAQDeqzQaAQC6mX79+mX99dd/x2X+9hDgt/eyW9q0lpaW5XrOuXPnJkkuvfTSJRqF9fX1y5VRi78/pLlSqSx37cty5pln5hOf+ET++7//OzfeeGPOOOOMXHPNNfnwhz+81OVvueWWfP7zn89Pf/rTNo3cuXPnZrvttstPfvKTJR7zt3tCAgC81zlHIwAASf58KPGLL77Yev/uu+9OXV1dNtpoowwePDjDhg3L008/nfXXX7/NbeTIkUmSTTbZJA8//HDmz5/fJuNvbbLJJrn33nvbTPv7Zdpr3XXXTc+ePVsPZU6S2bNn58knn/yHj91www3zhS98Ib/97W9zyCGHLHE177dNnTo1//Iv/5KvfOUrrYeOv23bbbfNlClTMmjQoCV+Nm/v7QkAsDLQaAQA6GYWLFiQGTNmtLm9+uqrNef27t07o0ePzkMPPZQ77rgjJ554Yg499NAMGTIkSfL1r38948aNywUXXJAnn3wyjzzySK644op85zvfSZJ84hOfSKVSybHHHps//vGP+Z//+Z/8+7//e5vn+OxnP5spU6bklFNOyRNPPJGrr74648ePr6nuxsbGjB49OqecckpuueWWPPbYYzn66KNTV1fX5srQf+vNN9/MCSeckFtvvTXPPfdc7rzzztx3333ZZJNNlrrsgQcemG222Saf/vSn2/zckz9fVGb11VfPQQcdlDvuuCPPPPNMbr311px44ol5/vnna1o3AIDuRKMRAKCbuemmmzJ06NA2t912263m3PXXXz+HHHJI9t9//3zgAx/IlltumR/84Aet84855phcdtllueKKK7LFFltk9913z/jx41v3aOzfv39+85vf5JFHHsk222yTf/u3f1viIi8jRozIL3/5y1x33XXZaqutcvHFF+fcc8+tufbvfOc72XnnnfPBD34wo0aNyq677ppNNtmkzfki/1Z9fX3+9Kc/5cgjj8yGG26YQw89NPvtt1++/vWvL7Hsyy+/nP/7v//LxIkTM2zYsDY/9yTp27dvbr/99owYMSKHHHJINtlkkxx99NGZP39+6/kvAQBWBpVqtVrt7CIAAKCkefPmZc0118x5552Xo48+urPLAQBYKbgYDAAA3d6DDz6Y//u//8uOO+6Y2bNn56yzzkqSHHTQQZ1cGQDAykOjEQCA94R///d/zxNPPJFevXplu+22yx133JHVV1+9s8sCAFhpOHQaAAAAAKiZi8EAAAAAADXTaAQAAAAAaqbRCAAAAADUTKMRAAAAAKiZRiMAAAAAUDONRgAAAACgZhqNAAAAAEDNNBoBAAAAgJppNAIAAAAANfv/nO0MFKK69TEAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 1600x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#collapse-hide\n",
    "x = torch.zeros(1, 16, 64)\n",
    "encs = PositionalEncoding(64)(x)\n",
    "plt.matshow(encs.squeeze())\n",
    "plt.xlabel('Embedding size')\n",
    "plt.ylabel('Sequence length')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "executionInfo": {
     "elapsed": 7,
     "status": "ok",
     "timestamp": 1683920511513,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "MNFf06HTsshd"
   },
   "outputs": [],
   "source": [
    "def loss_func(inp, targ):\n",
    "    return F.cross_entropy(inp.view(-1, len(vocab)), targ.view(-1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "executionInfo": {
     "elapsed": 7,
     "status": "ok",
     "timestamp": 1683920511513,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "5yL28lWTsshe"
   },
   "outputs": [],
   "source": [
    "# learn = Learner(dls, Model3(len(vocab)), loss_func=loss_func, metrics=accuracy)\n",
    "# learn.fit_one_cycle(5, 1e-2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8YLUpa3Usshe"
   },
   "source": [
    "Wow! That's a great accuracy! So the problem is solved and we only needed one attention layer and 2 layer deep feed-forward block? Don't you feel somewhat skeptical about this result?\n",
    "\n",
    "Well, you should be! Think about what we did here: the goal was to predict a target sequence, say `['.','two','.','three','.','four']` from an input `['one','.','two','.','three','.']`. These two sequences intersect on all positions except the first and the last one. So models needs to learn simply to copy input tokens starting from the second one to the outputs. In our case this will result in 15 correct predictions of total 16 positions, that's almost 94% accuracy. This makes the task very simple but not very useful to learn. To train proper autoregressive language model, as we did with RNNs, a concept of masking is to be introduced."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tsc046eGsshf"
   },
   "source": [
    "### Causal Masking"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fr3MfMhZsshf"
   },
   "source": [
    "So we want to allow the model for each token to attend only to itself and those prior to it. To acomplish this we can set all the values of attention matrix above the main diagonal to $-\\infty$. After softmax this values will effectively turn to 0 thus disabling attention to the \"future\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "executionInfo": {
     "elapsed": 384,
     "status": "ok",
     "timestamp": 1683924611221,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "N64vjltmsshf"
   },
   "outputs": [],
   "source": [
    "def get_subsequent_mask(x):\n",
    "    sz = x.size(1)\n",
    "    mask = (torch.triu(torch.ones(sz, sz, device=x.device)) == 1).transpose(0, 1)\n",
    "    mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))\n",
    "    return mask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 437
    },
    "executionInfo": {
     "elapsed": 530,
     "status": "ok",
     "timestamp": 1683924612148,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "tTY5tGIEsshf",
    "outputId": "0038b2be-c015-4e6d-9cb0-f19731419c43"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZoAAAGkCAYAAAAIduO+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAA9hAAAPYQGoP6dpAAAUOElEQVR4nO3df2id9b3A8U+S2jSTJGhdq8VUOxnU/vBnqmgvzmFVvK0ojG5ChVJhjC3a1oKs3VAnrsaOTQrWVSub62XWVhlFJ1dH6dCu09JfVpRtdkNwmdJWQXJqhaPLOfePsWy5tS6n7afPOenrBQ+Sx/PkfPgqefPNOXlOU7VarQYAJGkuegAARjahASCV0ACQSmgASCU0AKQSGgBSCQ0AqYQGgFRCA0AqoQEgVcOG5pFHHolzzz03xowZE5dffnls37696JHqSm9vb8yYMSPa29tj3LhxcfPNN8dbb71V9Fh178EHH4ympqZYvHhx0aPUpXfffTduvfXWGDt2bLS1tcX06dNj586dRY9VVwYGBuLuu++OSZMmRVtbW5x33nlx//33x8l8t6+GDM2GDRtiyZIlce+998bu3bvjwgsvjOuvvz4OHDhQ9Gh14+WXX46enp7Ytm1bbNq0KT799NO47rrr4tChQ0WPVrd27NgRjz32WFxwwQVFj1KXPvzww5g5c2accsop8cILL8Qf/vCH+MlPfhKnnXZa0aPVlRUrVsTq1atj1apV8cc//jFWrFgRP/rRj+Lhhx8uerTCNDXiTTUvv/zymDFjRqxatSoiIiqVSnR1dcUdd9wRS5cuLXi6+vT+++/HuHHj4uWXX46rrrqq6HHqzkcffRSXXHJJ/PSnP40f/vCHcdFFF8XKlSuLHquuLF26NH7/+9/H7373u6JHqWtz5syJ8ePHx89+9rPBc1/72teira0tfvnLXxY4WXEabkfzySefxK5du2LWrFmD55qbm2PWrFnx6quvFjhZfevv74+IiNNPP73gSepTT09PzJ49e8j/Vwz13HPPRXd3d8ydOzfGjRsXF198cTz++ONFj1V3rrzyyti8eXPs3bs3IiJef/312Lp1a9xwww0FT1acUUUPUKsPPvggBgYGYvz48UPOjx8/Pv70pz8VNFV9q1QqsXjx4pg5c2ZMmzat6HHqzvr162P37t2xY8eOokepa2+//XasXr06lixZEt/73vdix44dsXDhwhg9enTMnz+/6PHqxtKlS6NUKsXkyZOjpaUlBgYGYvny5TFv3ryiRytMw4WG2vX09MSbb74ZW7duLXqUutPX1xeLFi2KTZs2xZgxY4oep65VKpXo7u6OBx54ICIiLr744njzzTfj0UcfFZp/8/TTT8eTTz4Z69ati6lTp8aePXti8eLFMWHChJN2nRouNGeccUa0tLTE/v37h5zfv39/nHnmmQVNVb9uv/32eP7552PLli1x9tlnFz1O3dm1a1ccOHAgLrnkksFzAwMDsWXLlli1alWUy+VoaWkpcML6cdZZZ8WUKVOGnDv//PPjV7/6VUET1ae77rorli5dGrfccktEREyfPj3eeeed6O3tPWlD03Cv0YwePTouvfTS2Lx58+C5SqUSmzdvjiuuuKLAyepLtVqN22+/PTZu3Bi//e1vY9KkSUWPVJeuueaaeOONN2LPnj2DR3d3d8ybNy/27NkjMv9m5syZh71Ffu/evXHOOecUNFF9+vjjj6O5eeiP1paWlqhUKgVNVLyG29FERCxZsiTmz58f3d3dcdlll8XKlSvj0KFDsWDBgqJHqxs9PT2xbt26ePbZZ6O9vT327dsXERGdnZ3R1tZW8HT1o729/bDXrU499dQYO3as17P+nzvvvDOuvPLKeOCBB+LrX/96bN++PdasWRNr1qwperS6cuONN8by5ctj4sSJMXXq1HjttdfioYceittuu63o0YpTbVAPP/xwdeLEidXRo0dXL7vssuq2bduKHqmuRMRnHk888UTRo9W9r3zlK9VFixYVPUZd+vWvf12dNm1atbW1tTp58uTqmjVrih6p7pRKpeqiRYuqEydOrI4ZM6b6pS99qfr973+/Wi6Xix6tMA35dzQANI6Ge40GgMYiNACkEhoAUgkNAKmEBoBUQgNAqoYNTblcjh/84AdRLpeLHqXuWavhsU7DY52Gz1r9Q8P+HU2pVIrOzs7o7++Pjo6Oosepa9ZqeKzT8Fin4bNW/9CwOxoAGoPQAJDqhN9Us1KpxHvvvRft7e3R1NR01N+nVCoN+SdHZq2GxzoNj3UavpG+VtVqNQ4ePBgTJkw47I7V/+6Ev0bzt7/9Lbq6uk7kUwKQqK+v73M/7+qE72ja29sjIuK/4r9jVJxyop/+iJ7t/5+iRwBoKKVSKbq6ugZ/rh/JCQ/NP39dNipOiVFN9ROak/kdIQDH4j+9DOLNAACkEhoAUgkNAKmEBoBUQgNAKqEBIJXQAJBKaABIJTQApBIaAFIJDQCphAaAVEcVmkceeSTOPffcGDNmTFx++eWxffv24z0XACNEzaHZsGFDLFmyJO69997YvXt3XHjhhXH99dfHgQMHMuYDoMHVHJqHHnoovvnNb8aCBQtiypQp8eijj8YXvvCF+PnPf54xHwANrqbQfPLJJ7Fr166YNWvWv75Bc3PMmjUrXn311c+8plwuR6lUGnIAcPKoKTQffPBBDAwMxPjx44ecHz9+fOzbt+8zr+nt7Y3Ozs7Bw8c4A5xc0t91tmzZsujv7x88+vr6sp8SgDpS00c5n3HGGdHS0hL79+8fcn7//v1x5plnfuY1ra2t0draevQTAtDQatrRjB49Oi699NLYvHnz4LlKpRKbN2+OK6644rgPB0Djq2lHExGxZMmSmD9/fnR3d8dll10WK1eujEOHDsWCBQsy5gOgwdUcmm984xvx/vvvxz333BP79u2Liy66KF588cXD3iAAABERTdVqtXoin7BUKkVnZ2dcHTfFqKZTTuRTf65NlWeKHgGgofzz53l/f390dHQc8XHudQZAKqEBIJXQAJBKaABIJTQApBIaAFIJDQCphAaAVEIDQCqhASCV0ACQquabao5U1zbPLXqEw7j/GjAS2NEAkEpoAEglNACkEhoAUgkNAKmEBoBUQgNAKqEBIJXQAJBKaABIJTQApBIaAFIJDQCphAaAVEIDQCqhASCV0ACQSmgASCU0AKQSGgBSCQ0AqYQGgFRCA0AqoQEgldAAkEpoAEglNACkEhoAUgkNAKmEBoBUQgNAKqEBIJXQAJBKaABIJTQApBIaAFKNKnoAjuza5rlFj/CZNlWeKXoEoIHY0QCQSmgASCU0AKQSGgBSCQ0AqYQGgFRCA0AqoQEgldAAkEpoAEglNACkEhoAUgkNAKmEBoBUNYWmt7c3ZsyYEe3t7TFu3Li4+eab46233sqaDYARoKbQvPzyy9HT0xPbtm2LTZs2xaeffhrXXXddHDp0KGs+ABpcTR989uKLLw75+he/+EWMGzcudu3aFVddddVxHQyAkeGYPmGzv78/IiJOP/30Iz6mXC5HuVwe/LpUKh3LUwLQYI76zQCVSiUWL14cM2fOjGnTph3xcb29vdHZ2Tl4dHV1He1TAtCAjjo0PT098eabb8b69es/93HLli2L/v7+waOvr+9onxKABnRUvzq7/fbb4/nnn48tW7bE2Wef/bmPbW1tjdbW1qMaDoDGV1NoqtVq3HHHHbFx48Z46aWXYtKkSVlzATBC1BSanp6eWLduXTz77LPR3t4e+/bti4iIzs7OaGtrSxkQgMZW02s0q1evjv7+/rj66qvjrLPOGjw2bNiQNR8ADa7mX50BQC3c6wyAVEIDQCqhASCV0ACQSmgASCU0AKQSGgBSCQ0AqYQGgFRCA0AqoQEgldAAkOqoPviMk9u1zXOLHuEwmyrPFD0CcAR2NACkEhoAUgkNAKmEBoBUQgNAKqEBIJXQAJBKaABIJTQApBIaAFIJDQCphAaAVEIDQCqhASCV0ACQSmgASCU0AKQSGgBSCQ0AqYQGgFRCA0AqoQEgldAAkEpoAEglNACkEhoAUgkNAKmEBoBUQgNAKqEBIJXQAJBKaABIJTQApBIaAFIJDQCphAaAVKOKHgCOh2ub5xY9wmE2VZ4pegSoC3Y0AKQSGgBSCQ0AqYQGgFRCA0AqoQEgldAAkEpoAEglNACkEhoAUgkNAKmEBoBUQgNAKqEBINUxhebBBx+MpqamWLx48XEaB4CR5qhDs2PHjnjsscfiggsuOJ7zADDCHFVoPvroo5g3b148/vjjcdpppx3vmQAYQY4qND09PTF79uyYNWvWf3xsuVyOUqk05ADg5FHzRzmvX78+du/eHTt27BjW43t7e+O+++6reTAARoaadjR9fX2xaNGiePLJJ2PMmDHDumbZsmXR398/ePT19R3VoAA0ppp2NLt27YoDBw7EJZdcMnhuYGAgtmzZEqtWrYpyuRwtLS1DrmltbY3W1tbjMy0ADaem0FxzzTXxxhtvDDm3YMGCmDx5cnz3u989LDIAUFNo2tvbY9q0aUPOnXrqqTF27NjDzgNAhDsDAJCs5ned/X8vvfTScRgDgJHKjgaAVEIDQCqhASCV0ACQSmgASCU0AKQSGgBSCQ0AqYQGgFRCA0AqoQEg1THf6wz4bNc2zy16hMNsqjxT9AichOxoAEglNACkEhoAUgkNAKmEBoBUQgNAKqEBIJXQAJBKaABIJTQApBIaAFIJDQCphAaAVEIDQCqhASCV0ACQSmgASCU0AKQSGgBSCQ0AqYQGgFRCA0AqoQEgldAAkEpoAEglNACkEhoAUgkNAKmEBoBUQgNAKqEBIJXQAJBKaABIJTQApBIaAFIJDQCpRhU9AHDiXNs8t+gRDrOp8kzRI5DMjgaAVEIDQCqhASCV0ACQSmgASCU0AKQSGgBSCQ0AqYQGgFRCA0AqoQEgldAAkEpoAEglNACkqjk07777btx6660xduzYaGtri+nTp8fOnTszZgNgBKjp82g+/PDDmDlzZnz1q1+NF154Ib74xS/Gn//85zjttNOy5gOgwdUUmhUrVkRXV1c88cQTg+cmTZp03IcCYOSo6Vdnzz33XHR3d8fcuXNj3LhxcfHFF8fjjz/+udeUy+UolUpDDgBOHjWF5u23347Vq1fHl7/85fjNb34T3/72t2PhwoWxdu3aI17T29sbnZ2dg0dXV9cxDw1A42iqVqvV4T549OjR0d3dHa+88srguYULF8aOHTvi1Vdf/cxryuVylMvlwa9LpVJ0dXXF1XFTjGo65RhGB0aCTZVnih6Bo1QqlaKzszP6+/ujo6PjiI+raUdz1llnxZQpU4acO//88+Ovf/3rEa9pbW2Njo6OIQcAJ4+aQjNz5sx46623hpzbu3dvnHPOOcd1KABGjppCc+edd8a2bdvigQceiL/85S+xbt26WLNmTfT09GTNB0CDqyk0M2bMiI0bN8ZTTz0V06ZNi/vvvz9WrlwZ8+bNy5oPgAZX09/RRETMmTMn5syZkzELACOQe50BkEpoAEglNACkEhoAUgkNAKmEBoBUQgNAKqEBIJXQAJBKaABIJTQApKr5XmcAx9O1zXOLHuEz+UC248eOBoBUQgNAKqEBIJXQAJBKaABIJTQApBIaAFIJDQCphAaAVEIDQCqhASCV0ACQSmgASCU0AKQSGgBSCQ0AqYQGgFRCA0AqoQEgldAAkEpoAEglNACkEhoAUgkNAKmEBoBUQgNAKqEBIJXQAJBKaABIJTQApBIaAFIJDQCphAaAVEIDQCqhASCV0ACQSmgASDWq6AEA6tG1zXOLHuEwmyrPFD3CUbGjASCV0ACQSmgASCU0AKQSGgBSCQ0AqYQGgFRCA0AqoQEgldAAkEpoAEglNACkEhoAUtUUmoGBgbj77rtj0qRJ0dbWFuedd17cf//9Ua1Ws+YDoMHV9DEBK1asiNWrV8fatWtj6tSpsXPnzliwYEF0dnbGwoULs2YEoIHVFJpXXnklbrrpppg9e3ZERJx77rnx1FNPxfbt21OGA6Dx1fSrsyuvvDI2b94ce/fujYiI119/PbZu3Ro33HDDEa8pl8tRKpWGHACcPGra0SxdujRKpVJMnjw5WlpaYmBgIJYvXx7z5s074jW9vb1x3333HfOgADSmmnY0Tz/9dDz55JOxbt262L17d6xduzZ+/OMfx9q1a494zbJly6K/v3/w6OvrO+ahAWgcNe1o7rrrrli6dGnccsstERExffr0eOedd6K3tzfmz5//mde0trZGa2vrsU8KQEOqaUfz8ccfR3Pz0EtaWlqiUqkc16EAGDlq2tHceOONsXz58pg4cWJMnTo1XnvttXjooYfitttuy5oPgAZXU2gefvjhuPvuu+M73/lOHDhwICZMmBDf+ta34p577smaD4AG11Q9wX/WXyqVorOzM66Om2JU0ykn8qkBGtqmyjNFjzDEP3+e9/f3R0dHxxEf515nAKQSGgBSCQ0AqYQGgFRCA0AqoQEgldAAkEpoAEglNACkEhoAUgkNAKmEBoBUNd29GYDiXNs8t+gRhvh79dNhPc6OBoBUQgNAKqEBIJXQAJBKaABIJTQApBIaAFIJDQCphAaAVEIDQCqhASCV0ACQSmgASCU0AKQSGgBSCQ0AqYQGgFRCA0AqoQEgldAAkEpoAEglNACkEhoAUgkNAKmEBoBUQgNAKqEBIJXQAJBKaABIJTQApBIaAFIJDQCphAaAVEIDQCqhASDVqBP9hNVqNSIi/h6fRlRP9LMDcLz8PT6NiH/9XD+SEx6agwcPRkTE1vjfE/3UACQ4ePBgdHZ2HvHfN1X/U4qOs0qlEu+99160t7dHU1PTUX+fUqkUXV1d0dfXFx0dHcdxwpHHWg2PdRoe6zR8I32tqtVqHDx4MCZMmBDNzUd+JeaE72iam5vj7LPPPm7fr6OjY0T+B8xgrYbHOg2PdRq+kbxWn7eT+SdvBgAgldAAkKphQ9Pa2hr33ntvtLa2Fj1K3bNWw2Odhsc6DZ+1+ocT/mYAAE4uDbujAaAxCA0AqYQGgFRCA0AqoQEgldAAkEpoAEglNACk+j9hGsdCJM4b4QAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 480x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "inp = torch.randn(8, 10, 64)\n",
    "mask = get_subsequent_mask(inp)\n",
    "plt.matshow(mask);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 186,
     "status": "ok",
     "timestamp": 1683924622535,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "RM4R9ltgWRrr",
    "outputId": "a2bd9428-431f-4780-9f6f-86c1201e8097"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0., -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf],\n",
       "        [0., 0., -inf, -inf, -inf, -inf, -inf, -inf, -inf, -inf],\n",
       "        [0., 0., 0., -inf, -inf, -inf, -inf, -inf, -inf, -inf],\n",
       "        [0., 0., 0., 0., -inf, -inf, -inf, -inf, -inf, -inf],\n",
       "        [0., 0., 0., 0., 0., -inf, -inf, -inf, -inf, -inf],\n",
       "        [0., 0., 0., 0., 0., 0., -inf, -inf, -inf, -inf],\n",
       "        [0., 0., 0., 0., 0., 0., 0., -inf, -inf, -inf],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0., -inf, -inf],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0., 0., -inf],\n",
       "        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mask"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 437
    },
    "id": "RV_Br0CZsshg",
    "outputId": "53f70b1c-c383-4e0a-c801-69eabd770e6a"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZoAAAGkCAYAAAAIduO+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAA9hAAAPYQGoP6dpAAAVCklEQVR4nO3dcYzUhZ338e/uIsPW7m4QC0JZhJpeUEAFQaM8se0j0Rg19dLYmmBCMLFNuypIYgrto9ZYWGlaQyIWxVhLUhHMNURrThtDo5RWAoIYTVtpz4tdNYDmzI7i3YK78/zh3V63C3YH+PKbWV6vZGIYZ/h98sPs29/sMtNQqVQqAQBJGoseAMDwJjQApBIaAFIJDQCphAaAVEIDQCqhASCV0ACQSmgASCU0AKSq29A88MADMXny5Bg1alRcdNFFsX379qIn1ZTOzs6YM2dOtLS0xNixY+Paa6+N119/vehZNe/ee++NhoaGWLx4cdFTatLbb78dN9xwQ4wZMyaam5tjxowZ8dJLLxU9q6b09vbGHXfcEVOmTInm5uY466yz4p577omT+d2+6jI0GzdujCVLlsRdd90Vu3btivPOOy+uuOKK2L9/f9HTasYLL7wQHR0dsW3btnjuuefi0KFDcfnll8eBAweKnlazduzYEQ899FCce+65RU+pSe+//37MnTs3TjnllHjmmWfiD3/4Q/zkJz+J0aNHFz2tpqxcuTLWrFkTq1evjj/+8Y+xcuXK+NGPfhT3339/0dMK01CPb6p50UUXxZw5c2L16tUREdHX1xft7e1xyy23xNKlSwteV5vefffdGDt2bLzwwgtx6aWXFj2n5nz44Ycxa9as+OlPfxo//OEP4/zzz49Vq1YVPaumLF26NH73u9/Fb3/726Kn1LSrr746xo0bF4888kj/fV/72teiubk5fvGLXxS4rDh1d0Vz8ODB2LlzZ8ybN6//vsbGxpg3b168+OKLBS6rbd3d3RERcdpppxW8pDZ1dHTEVVddNeC/KwZ66qmnYvbs2XHdddfF2LFjY+bMmfHwww8XPavmXHLJJbF58+bYs2dPRES88sorsXXr1rjyyisLXlacEUUPqNZ7770Xvb29MW7cuAH3jxs3Lv70pz8VtKq29fX1xeLFi2Pu3Lkxffr0oufUnA0bNsSuXbtix44dRU+paW+88UasWbMmlixZEt/73vdix44dceutt8bIkSNjwYIFRc+rGUuXLo1yuRxTp06Npqam6O3tjeXLl8f8+fOLnlaYugsN1evo6IjXXnsttm7dWvSUmtPV1RWLFi2K5557LkaNGlX0nJrW19cXs2fPjhUrVkRExMyZM+O1116LBx98UGj+xhNPPBGPPfZYrF+/PqZNmxa7d++OxYsXx4QJE07a81R3oTn99NOjqakp9u3bN+D+ffv2xRlnnFHQqtp18803x9NPPx1btmyJiRMnFj2n5uzcuTP2798fs2bN6r+vt7c3tmzZEqtXr46enp5oamoqcGHtGD9+fJxzzjkD7jv77LPjl7/8ZUGLatPtt98eS5cujeuvvz4iImbMmBFvvvlmdHZ2nrShqbvv0YwcOTIuuOCC2Lx5c/99fX19sXnz5rj44osLXFZbKpVK3HzzzbFp06b4zW9+E1OmTCl6Uk267LLL4tVXX43du3f332bPnh3z58+P3bt3i8zfmDt37qAfkd+zZ0+ceeaZBS2qTR999FE0Ng780trU1BR9fX0FLSpe3V3RREQsWbIkFixYELNnz44LL7wwVq1aFQcOHIiFCxcWPa1mdHR0xPr16+PJJ5+MlpaW2Lt3b0REtLW1RXNzc8HrakdLS8ug71udeuqpMWbMGN/P+ju33XZbXHLJJbFixYr4+te/Htu3b4+1a9fG2rVri55WU6655ppYvnx5TJo0KaZNmxYvv/xy3HfffXHjjTcWPa04lTp1//33VyZNmlQZOXJk5cILL6xs27at6Ek1JSIOe3v00UeLnlbzvvSlL1UWLVpU9Iya9Ktf/aoyffr0SqlUqkydOrWydu3aoifVnHK5XFm0aFFl0qRJlVGjRlW+8IUvVL7//e9Xenp6ip5WmLr8ezQA1I+6+x4NAPVFaABIJTQApBIaAFIJDQCphAaAVHUbmp6envjBD34QPT09RU+pec7V0DhPQ+M8DZ1z9Ym6/Xs05XI52traoru7O1pbW4ueU9Ocq6FxnobGeRo65+oTdXtFA0B9EBoAUp3wN9Xs6+uLd955J1paWqKhoeGof59yuTzgnxyZczU0ztPQOE9DN9zPVaVSiQ8++CAmTJgw6B2r/9YJ/x7NW2+9Fe3t7SfykAAk6urq+tTPuzrhVzQtLS0REfHmrsnR+tnaeeXun/9pRtETAOrKx3Eotsa/9n9dP5ITHpr/ebms9bON0dpSO6EZ0XBK0RMA6st/vx72j74NUjtf6QEYloQGgFRCA0AqoQEgldAAkEpoAEglNACkEhoAUgkNAKmEBoBUQgNAKqEBINVRheaBBx6IyZMnx6hRo+Kiiy6K7du3H+9dAAwTVYdm48aNsWTJkrjrrrti165dcd5558UVV1wR+/fvz9gHQJ2rOjT33Xdf3HTTTbFw4cI455xz4sEHH4zPfOYz8bOf/SxjHwB1rqrQHDx4MHbu3Bnz5s3739+gsTHmzZsXL7744mGf09PTE+VyecANgJNHVaF57733ore3N8aNGzfg/nHjxsXevXsP+5zOzs5oa2vrv/kYZ4CTS/pPnS1btiy6u7v7b11dXdmHBKCGVPVRzqeffno0NTXFvn37Bty/b9++OOOMMw77nFKpFKVS6egXAlDXqrqiGTlyZFxwwQWxefPm/vv6+vpi8+bNcfHFFx/3cQDUv6quaCIilixZEgsWLIjZs2fHhRdeGKtWrYoDBw7EwoULM/YBUOeqDs03vvGNePfdd+POO++MvXv3xvnnnx/PPvvsoB8QAICIiIZKpVI5kQcsl8vR1tYW7+/5QrS21M474Fwx4fyiJwDUlY8rh+L5eDK6u7ujtbX1iI+rna/0AAxLQgNAKqEBIJXQAJBKaABIJTQApBIaAFIJDQCphAaAVEIDQCqhASBV1W+qebz88z/NiBENpxR1+EF+/c7uoicM4v3XgOHAFQ0AqYQGgFRCA0AqoQEgldAAkEpoAEglNACkEhoAUgkNAKmEBoBUQgNAKqEBIJXQAJBKaABIJTQApBIaAFIJDQCphAaAVEIDQCqhASCV0ACQSmgASCU0AKQSGgBSCQ0AqYQGgFRCA0AqoQEgldAAkEpoAEglNACkEhoAUgkNAKmEBoBUQgNAKqEBINWIogfUivd6DxQ9YZDGU08tesJh9R2ovXMF1C5XNACkEhoAUgkNAKmEBoBUQgNAKqEBIJXQAJBKaABIJTQApBIaAFIJDQCphAaAVEIDQCqhASBVVaHp7OyMOXPmREtLS4wdOzauvfbaeP3117O2ATAMVBWaF154ITo6OmLbtm3x3HPPxaFDh+Lyyy+PAz6fBIAjqOqDz5599tkBv/75z38eY8eOjZ07d8all156XIcBMDwc0ydsdnd3R0TEaaeddsTH9PT0RE9PT/+vy+XysRwSgDpz1D8M0NfXF4sXL465c+fG9OnTj/i4zs7OaGtr67+1t7cf7SEBqENHHZqOjo547bXXYsOGDZ/6uGXLlkV3d3f/raur62gPCUAdOqqXzm6++eZ4+umnY8uWLTFx4sRPfWypVIpSqXRU4wCof1WFplKpxC233BKbNm2K559/PqZMmZK1C4BhoqrQdHR0xPr16+PJJ5+MlpaW2Lt3b0REtLW1RXNzc8pAAOpbVd+jWbNmTXR3d8eXv/zlGD9+fP9t48aNWfsAqHNVv3QGANXwXmcApBIaAFIJDQCphAaAVEIDQCqhASCV0ACQSmgASCU0AKQSGgBSCQ0AqYQGgFRH9cFnw9HpTacWPWGQvgMHip4AcMxc0QCQSmgASCU0AKQSGgBSCQ0AqYQGgFRCA0AqoQEgldAAkEpoAEglNACkEhoAUgkNAKmEBoBUQgNAKqEBIJXQAJBKaABIJTQApBIaAFIJDQCphAaAVEIDQCqhASCV0ACQSmgASCU0AKQSGgBSCQ0AqYQGgFRCA0AqoQEgldAAkEpoAEglNACkEhoAUo0o7MgNDZ/casT1//5/i54wyIjPjyp6wmF9/PY7RU8YpO//nF/0hEEat+4uegLUBFc0AKQSGgBSCQ0AqYQGgFRCA0AqoQEgldAAkEpoAEglNACkEhoAUgkNAKmEBoBUQgNAKqEBINUxhebee++NhoaGWLx48XGaA8Bwc9Sh2bFjRzz00ENx7rnnHs89AAwzRxWaDz/8MObPnx8PP/xwjB49+nhvAmAYOarQdHR0xFVXXRXz5s37h4/t6emJcrk84AbAyaPqj3LesGFD7Nq1K3bs2DGkx3d2dsbdd99d9TAAhoeqrmi6urpi0aJF8dhjj8WoUUP7PPtly5ZFd3d3/62rq+uohgJQn6q6otm5c2fs378/Zs2a1X9fb29vbNmyJVavXh09PT3R1NQ04DmlUilKpdLxWQtA3akqNJdddlm8+uqrA+5buHBhTJ06Nb773e8OigwAVBWalpaWmD59+oD7Tj311BgzZsyg+wEgwjsDAJCs6p86+3vPP//8cZgBwHDligaAVEIDQCqhASCV0ACQSmgASCU0AKQSGgBSCQ0AqYQGgFRCA0AqoQEg1TG/19lRq1QiolLY4f/e7ROeLXrCIP+vfFnRE+pG49bdRU8YpKm1tegJg/T6KHUK4IoGgFRCA0AqoQEgldAAkEpoAEglNACkEhoAUgkNAKmEBoBUQgNAKqEBIJXQAJBKaABIJTQApBIaAFIJDQCphAaAVEIDQCqhASCV0ACQSmgASCU0AKQSGgBSCQ0AqYQGgFRCA0AqoQEgldAAkEpoAEglNACkEhoAUgkNAKmEBoBUQgNAKqEBIJXQAJBqRGFHbmyKaGgq7PB/78ZXFhQ9YZDx//VvRU/gGOy/blrREwYZ88iLRU/gJOSKBoBUQgNAKqEBIJXQAJBKaABIJTQApBIaAFIJDQCphAaAVEIDQCqhASCV0ACQSmgASCU0AKSqOjRvv/123HDDDTFmzJhobm6OGTNmxEsvvZSxDYBhoKrPo3n//fdj7ty58ZWvfCWeeeaZ+NznPhd//vOfY/To0Vn7AKhzVYVm5cqV0d7eHo8++mj/fVOmTDnuowAYPqp66eypp56K2bNnx3XXXRdjx46NmTNnxsMPP/ypz+np6YlyuTzgBsDJo6rQvPHGG7FmzZr44he/GL/+9a/j29/+dtx6662xbt26Iz6ns7Mz2tra+m/t7e3HPBqA+lFVaPr6+mLWrFmxYsWKmDlzZnzzm9+Mm266KR588MEjPmfZsmXR3d3df+vq6jrm0QDUj6pCM378+DjnnHMG3Hf22WfHX//61yM+p1QqRWtr64AbACePqkIzd+7ceP311wfct2fPnjjzzDOP6ygAho+qQnPbbbfFtm3bYsWKFfGXv/wl1q9fH2vXro2Ojo6sfQDUuapCM2fOnNi0aVM8/vjjMX369Ljnnnti1apVMX/+/Kx9ANS5qv4eTUTE1VdfHVdffXXGFgCGIe91BkAqoQEgldAAkEpoAEglNACkEhoAUgkNAKmEBoBUQgNAKqEBIJXQAJCq6vc6O14aGhuioaGhqMMPMnrtZ4ueMFhj7Zwfqtf31f8oesJgjxQ9YLAR7ROLnnBYH3e9VfSEYcMVDQCphAaAVEIDQCqhASCV0ACQSmgASCU0AKQSGgBSCQ0AqYQGgFRCA0AqoQEgldAAkEpoAEglNACkEhoAUgkNAKmEBoBUQgNAKqEBIJXQAJBKaABIJTQApBIaAFIJDQCphAaAVEIDQCqhASCV0ACQSmgASCU0AKQSGgBSCQ0AqYQGgFRCA0AqoQEgldAAkGpEUQcuXzszRpwyqqjDD3KwpaHoCYOUDh4sekLdGPH5CUVPGKS0YXTRE+pC7979RU8gmSsaAFIJDQCphAaAVEIDQCqhASCV0ACQSmgASCU0AKQSGgBSCQ0AqYQGgFRCA0AqoQEgVVWh6e3tjTvuuCOmTJkSzc3NcdZZZ8U999wTlUolax8Ada6qjwlYuXJlrFmzJtatWxfTpk2Ll156KRYuXBhtbW1x6623Zm0EoI5VFZrf//738dWvfjWuuuqqiIiYPHlyPP7447F9+/aUcQDUv6peOrvkkkti8+bNsWfPnoiIeOWVV2Lr1q1x5ZVXHvE5PT09US6XB9wAOHlUdUWzdOnSKJfLMXXq1Ghqaore3t5Yvnx5zJ8//4jP6ezsjLvvvvuYhwJQn6q6onniiSfisccei/Xr18euXbti3bp18eMf/zjWrVt3xOcsW7Ysuru7+29dXV3HPBqA+lHVFc3tt98eS5cujeuvvz4iImbMmBFvvvlmdHZ2xoIFCw77nFKpFKVS6diXAlCXqrqi+eijj6KxceBTmpqaoq+v77iOAmD4qOqK5pprronly5fHpEmTYtq0afHyyy/HfffdFzfeeGPWPgDqXFWhuf/+++OOO+6I73znO7F///6YMGFCfOtb34o777wzax8Ada6q0LS0tMSqVati1apVSXMAGG681xkAqYQGgFRCA0AqoQEgldAAkEpoAEglNACkEhoAUgkNAKmEBoBUQgNAKqEBIFVVb6p5PLX8y44Y0XBKUYcfpOE3ny96wiCVR2r0/wMqvUUvGOTjt98pesIgo//1QNETBqm9P7mI/7zi/KInHNaop7cXPWHYqNGvZAAMF0IDQCqhASCV0ACQSmgASCU0AKQSGgBSCQ0AqYQGgFRCA0AqoQEgldAAkEpoAEglNACkEhoAUgkNAKmEBoBUQgNAKqEBIJXQAJBKaABIJTQApBIaAFIJDQCphAaAVEIDQCqhASCV0ACQSmgASCU0AKQSGgBSCQ0AqYQGgFRCA0AqoQEg1YgTfcBKpRIRER/HoYjKiT76kTUc6Cl6wiCVyqGiJxxepbfoBXWhUjlY9IRBemvwv6mPD/1X0RMO6+MaPFe15uP45Bz9z9f1I2mo/KNHHGdvvfVWtLe3n8hDApCoq6srJk6ceMR/f8JD09fXF++88060tLREQ0PDUf8+5XI52tvbo6urK1pbW4/jwuHHuRoa52lonKehG+7nqlKpxAcffBATJkyIxsYjfyfmhL901tjY+Knlq1Zra+uw/APM4FwNjfM0NM7T0A3nc9XW1vYPH+OHAQBIJTQApKrb0JRKpbjrrruiVCoVPaXmOVdD4zwNjfM0dM7VJ074DwMAcHKp2ysaAOqD0ACQSmgASCU0AKQSGgBSCQ0AqYQGgFRCA0Cq/w8Zk/Bf6MQ01QAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 480x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "q, k = torch.rand(1,10,32), torch.randn(1,10,32)\n",
    "att_ = F.softmax((q@k.permute(0,2,1)+mask), -1)\n",
    "plt.matshow(att_[0].detach());"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "V6LSGqBtsshg"
   },
   "source": [
    "We should also modify the attention layer to accept mask:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wPCJdBuSsshg"
   },
   "source": [
    "Now we get somewhat lower accuracy, which is expected given that the task has become more difficult. Also training loss is significantly lower than validation loss, which means the model is overfitting. Let's see if the same approaches as was applied to RNNs can help."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CAQ3OO6Usshg"
   },
   "source": [
    "### Multilayer transformer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "PILh_gZisshh"
   },
   "source": [
    "To solve a more difficult task we ussualy need a deeper model. For convenience let's make a TransformerLayer which will combine self-attention and feed-forward blocks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NpxQR4jjsshi"
   },
   "source": [
    "That's not good! 4 layer deep Transformer strugles to learn anything. But there are good news, this problem has been already resolved in the original transformer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Wv7HPuEosshi"
   },
   "source": [
    "### Residual connections and Regularization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "10eLL4WIsshi"
   },
   "source": [
    "If you are familiar with ResNets the proposed solution will not surprise you much. The idea is simple yet very effective. Instead of returning modified output $f(x)$ each transformer sublayer will return $x + f(x)$. This allows the original input to propagate freely through the model. So the model learns not an entirely new representation of $x$ but how to modify $x$ to add some useful information to the original representation.\n",
    "\n",
    "As we modify layers to include the residual connections let's also add some regularization by inserting Dropout layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "executionInfo": {
     "elapsed": 2,
     "status": "ok",
     "timestamp": 1683920568706,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "-79nUq88sshi"
   },
   "outputs": [],
   "source": [
    "class TransformerEmbedding(Module):\n",
    "    def __init__(self, emb_sz, d_model, p=0.1):\n",
    "        self.emb = Embedding(emb_sz, d_model)\n",
    "        nn.init.trunc_normal_(self.emb.weight, std=d_model**-0.5)\n",
    "        self.pos_enc = PositionalEncoding(d_model)\n",
    "        self.drop = nn.Dropout(p)\n",
    "    def forward(self, x):\n",
    "        return self.drop(self.pos_enc(self.emb(x)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SerAuW2ysshj"
   },
   "source": [
    "Another modification is to add layer normalization which is intended to improve learning dynamics of the network by reparametrising data statistics and is generally used in transformer based architectures."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "executionInfo": {
     "elapsed": 1,
     "status": "ok",
     "timestamp": 1683920570821,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "mgeMNw5Asshj"
   },
   "outputs": [],
   "source": [
    "class FeedForward(Module):\n",
    "    def __init__(self, d_model, d_ff, p=0.2):\n",
    "        self.lin1 = nn.Linear(d_model, d_ff)\n",
    "        self.lin2 = nn.Linear(d_ff, d_model)\n",
    "        self.act = nn.ReLU()\n",
    "        self.norm = nn.LayerNorm(d_model)\n",
    "        self.drop = nn.Dropout(p)\n",
    "    def forward(self, x):\n",
    "        x = self.norm(x)\n",
    "        out = self.act(self.lin1(x))\n",
    "        out = self.lin2(out)\n",
    "        return x + self.drop(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "from positional_encodings.torch_encodings import PositionalEncoding2D"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "pos_dim = 10\n",
    "\n",
    "p_enc_2d = PositionalEncoding2D(pos_dim)\n",
    "pos_embeddings = {}\n",
    "# for i in range(1,200):\n",
    "#     for j in range(2,100):    \n",
    "z = torch.zeros(1,257,257,pos_dim)\n",
    "# pos_embeddings[(i,j)] = p_enc_2d(z[:,:i,:j,:].to(DEVICE))\n",
    "pos_embeddings = p_enc_2d(z.to(device))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZSCo8b1U_YMN"
   },
   "source": [
    "# --------------------- My Experiments ---------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "executionInfo": {
     "elapsed": 1616,
     "status": "ok",
     "timestamp": 1683925600084,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "0N4XDqjD_S3K"
   },
   "outputs": [],
   "source": [
    "class MultiHeadAttention(Module):\n",
    "    def __init__(self, d_model, n_heads, p=0.1, attn_type='qkv'):\n",
    "        assert d_model%n_heads == 0\n",
    "        self.n_heads = n_heads\n",
    "\n",
    "        d_qk, d_v = d_model//n_heads, d_model//n_heads\n",
    "        \n",
    "        if attn_type == 'qkv':\n",
    "            self.iq = nn.Linear(d_model, d_model, bias=False)\n",
    "        else: # kv\n",
    "            self.map_pos = nn.Linear(pos_dim, 1, bias=True)  \n",
    "            \n",
    "        self.ik = nn.Linear(d_model, d_model, bias=False)\n",
    "        self.iv = nn.Linear(d_model, d_model, bias=False)\n",
    "        \n",
    "        self.scale = d_qk**0.5\n",
    "        self.out = nn.Linear(d_model, d_model, bias=False)\n",
    "        self.norm = nn.LayerNorm(d_model)\n",
    "        self.drop = nn.Dropout(p)\n",
    "        \n",
    "        \n",
    "        self.attn_type = attn_type   \n",
    "        \n",
    "\n",
    "    def forward(self, x, mask=None):\n",
    "        bs, seq_len, d = x.size()\n",
    "        mask = ifnone(mask, 0)\n",
    "        x = self.norm(x)\n",
    "        k = self.ik(x).view(bs, seq_len, self.n_heads, d//self.n_heads).transpose(1, 2)\n",
    "        v = self.iv(x).view(bs, seq_len, self.n_heads, d//self.n_heads).transpose(1, 2)\n",
    "        \n",
    "        if attn_type == 'qkv':\n",
    "            q = self.iq(x).view(bs, seq_len, self.n_heads, d//self.n_heads).transpose(1, 2)\n",
    "            att = q@k.transpose(-2,-1)/self.scale\n",
    "        elif attn_type == 'kv':\n",
    "            att = k@k.transpose(-2,-1)/self.scale\n",
    "            pos = pos_embeddings[:,:seq_len,:seq_len,:] # 1, m, m, ? \n",
    "            att = att.unsqueeze(-1) + pos.unsqueeze(0)\n",
    "            att = self.map_pos(att).squeeze(-1)\n",
    "        else: # symmetric\n",
    "            att = k@k.transpose(-2,-1)/self.scale\n",
    "        \n",
    "        \n",
    "        att = F.softmax(att + mask, dim=-1) # (B, T, T)\n",
    "        \n",
    "        out = att @ v # (bs, nh, sl, sl) x (bs, nh, sl, dh) -> (bs, nh, sl, dh)\n",
    "        out = out.transpose(1, 2).contiguous().view(bs, seq_len, d) # back to original shape\n",
    "\n",
    "        return x + self.drop(self.out(out))\n",
    "\n",
    "\n",
    "\n",
    "class TransformerLayer(Module):\n",
    "    def __init__(self, d_model, n_heads=8, d_ff=None, causal=True,\n",
    "                 p_att=0.1, p_ff=0.1, attn_type=None):\n",
    "        d_ff = ifnone(d_ff, 4*d_model)\n",
    "        self.attn = MultiHeadAttention(d_model, n_heads, attn_type=attn_type)\n",
    "        self.ff = FeedForward(d_model, d_ff, p=p_ff)\n",
    "        self.causal = causal\n",
    "        self._init()\n",
    "    def forward(self, x, mask=None):\n",
    "        if self.causal:\n",
    "            mask = get_subsequent_mask(x)\n",
    "        return self.ff(self.attn(x, mask))\n",
    "    def _init(self):\n",
    "        for p in self.parameters():\n",
    "            if p.dim()>1: nn.init.xavier_uniform_(p)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "executionInfo": {
     "elapsed": 241,
     "status": "ok",
     "timestamp": 1683925602773,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "qNWDSmP6_S4Z"
   },
   "outputs": [],
   "source": [
    "class Model7(Module):\n",
    "    def __init__(self, vocab_sz, d_model=64, n_layer=4, n_heads=8, \n",
    "                 p_emb=0.1, p_att=0.1, p_ff=0.2, tie_weights=True, attn_type=None):\n",
    "        self.emb = TransformerEmbedding(vocab_sz, d_model, p=p_emb)\n",
    "        self.encoder = nn.Sequential(*[TransformerLayer(d_model, n_heads,\n",
    "                                                     p_att=p_att, p_ff=p_ff, attn_type=attn_type)\n",
    "                                    for _ in range(n_layer)],\n",
    "                                    nn.LayerNorm(d_model))\n",
    "        self.out = nn.Linear(d_model, vocab_sz)\n",
    "        if tie_weights: self.out.weight = self.emb.emb.weight\n",
    "    def forward(self, x):\n",
    "        x = self.emb(x)\n",
    "        x = self.encoder(x)\n",
    "        return self.out(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 4,
     "status": "ok",
     "timestamp": 1683924812704,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "ZXITx4ogJ3qc"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 1191,
     "status": "ok",
     "timestamp": 1683924813892,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "DyZ5YQi0J4YF",
    "outputId": "4baa267f-fef4-40fe-c6df-629ce4bedf9b"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "executionInfo": {
     "elapsed": 608902,
     "status": "ok",
     "timestamp": 1683934820745,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "5Av9LoHz_S9h",
    "outputId": "bcf503af-d15f-41c8-d44d-d518baa53afa"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv, run number: 0 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.256110</td>\n",
       "      <td>2.905067</td>\n",
       "      <td>0.136068</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.891167</td>\n",
       "      <td>2.720767</td>\n",
       "      <td>0.160482</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.279390</td>\n",
       "      <td>1.808424</td>\n",
       "      <td>0.409017</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1.790504</td>\n",
       "      <td>1.574720</td>\n",
       "      <td>0.506836</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1.417395</td>\n",
       "      <td>1.319127</td>\n",
       "      <td>0.552979</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.142001</td>\n",
       "      <td>1.073821</td>\n",
       "      <td>0.646159</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>0.987452</td>\n",
       "      <td>1.063504</td>\n",
       "      <td>0.645508</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.888688</td>\n",
       "      <td>0.951178</td>\n",
       "      <td>0.700521</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.802699</td>\n",
       "      <td>0.850084</td>\n",
       "      <td>0.733073</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.740156</td>\n",
       "      <td>0.814712</td>\n",
       "      <td>0.735514</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.700165</td>\n",
       "      <td>0.883285</td>\n",
       "      <td>0.719320</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.671095</td>\n",
       "      <td>0.810858</td>\n",
       "      <td>0.739746</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.655420</td>\n",
       "      <td>0.802854</td>\n",
       "      <td>0.737712</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.644619</td>\n",
       "      <td>0.794536</td>\n",
       "      <td>0.738525</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.638983</td>\n",
       "      <td>0.797211</td>\n",
       "      <td>0.739095</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv, run number: 1 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.266454</td>\n",
       "      <td>2.923888</td>\n",
       "      <td>0.143473</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.903503</td>\n",
       "      <td>2.681810</td>\n",
       "      <td>0.168620</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.321858</td>\n",
       "      <td>1.945074</td>\n",
       "      <td>0.363770</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1.810326</td>\n",
       "      <td>1.662280</td>\n",
       "      <td>0.509766</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1.489406</td>\n",
       "      <td>1.531891</td>\n",
       "      <td>0.542725</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.277714</td>\n",
       "      <td>1.368430</td>\n",
       "      <td>0.568766</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.110325</td>\n",
       "      <td>1.147792</td>\n",
       "      <td>0.620361</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.990049</td>\n",
       "      <td>1.079419</td>\n",
       "      <td>0.652018</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.911535</td>\n",
       "      <td>0.989139</td>\n",
       "      <td>0.687093</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.856172</td>\n",
       "      <td>0.908478</td>\n",
       "      <td>0.707845</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.807359</td>\n",
       "      <td>0.913087</td>\n",
       "      <td>0.712321</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.773079</td>\n",
       "      <td>0.890409</td>\n",
       "      <td>0.719157</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.751292</td>\n",
       "      <td>0.884397</td>\n",
       "      <td>0.719645</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.737705</td>\n",
       "      <td>0.869899</td>\n",
       "      <td>0.720133</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.732973</td>\n",
       "      <td>0.874786</td>\n",
       "      <td>0.719727</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv, run number: 2 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.179368</td>\n",
       "      <td>2.875248</td>\n",
       "      <td>0.143962</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.869890</td>\n",
       "      <td>2.683697</td>\n",
       "      <td>0.173991</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.298279</td>\n",
       "      <td>2.032179</td>\n",
       "      <td>0.345947</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1.786012</td>\n",
       "      <td>1.722394</td>\n",
       "      <td>0.464600</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1.417061</td>\n",
       "      <td>1.286951</td>\n",
       "      <td>0.598796</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.151532</td>\n",
       "      <td>1.065494</td>\n",
       "      <td>0.651367</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.002271</td>\n",
       "      <td>1.077212</td>\n",
       "      <td>0.646729</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.917353</td>\n",
       "      <td>1.006039</td>\n",
       "      <td>0.670247</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.866430</td>\n",
       "      <td>0.978164</td>\n",
       "      <td>0.676107</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.832919</td>\n",
       "      <td>0.961366</td>\n",
       "      <td>0.677327</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.808266</td>\n",
       "      <td>0.974109</td>\n",
       "      <td>0.682048</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.786870</td>\n",
       "      <td>0.951877</td>\n",
       "      <td>0.691162</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.774935</td>\n",
       "      <td>0.971439</td>\n",
       "      <td>0.686686</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.767964</td>\n",
       "      <td>0.964784</td>\n",
       "      <td>0.685547</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.761766</td>\n",
       "      <td>0.965881</td>\n",
       "      <td>0.685303</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv, run number: 3 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.157713</td>\n",
       "      <td>2.877072</td>\n",
       "      <td>0.147380</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.843228</td>\n",
       "      <td>2.508722</td>\n",
       "      <td>0.192871</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.229631</td>\n",
       "      <td>1.915144</td>\n",
       "      <td>0.395345</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1.768398</td>\n",
       "      <td>1.570661</td>\n",
       "      <td>0.507324</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1.456425</td>\n",
       "      <td>1.418163</td>\n",
       "      <td>0.534993</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.240513</td>\n",
       "      <td>1.231260</td>\n",
       "      <td>0.598633</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.071986</td>\n",
       "      <td>1.060262</td>\n",
       "      <td>0.660563</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.950162</td>\n",
       "      <td>0.949831</td>\n",
       "      <td>0.702067</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.853404</td>\n",
       "      <td>0.882342</td>\n",
       "      <td>0.717773</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.778079</td>\n",
       "      <td>0.823782</td>\n",
       "      <td>0.731527</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.727330</td>\n",
       "      <td>0.830475</td>\n",
       "      <td>0.724447</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.696557</td>\n",
       "      <td>0.804903</td>\n",
       "      <td>0.733887</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.676709</td>\n",
       "      <td>0.814380</td>\n",
       "      <td>0.722331</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.662792</td>\n",
       "      <td>0.816314</td>\n",
       "      <td>0.724772</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.657383</td>\n",
       "      <td>0.818551</td>\n",
       "      <td>0.723389</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv, run number: 4 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.105174</td>\n",
       "      <td>2.908107</td>\n",
       "      <td>0.151855</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.742391</td>\n",
       "      <td>2.502717</td>\n",
       "      <td>0.176025</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.160787</td>\n",
       "      <td>2.055616</td>\n",
       "      <td>0.342448</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1.754310</td>\n",
       "      <td>1.741489</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1.473144</td>\n",
       "      <td>1.640367</td>\n",
       "      <td>0.511800</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.261733</td>\n",
       "      <td>1.618402</td>\n",
       "      <td>0.502441</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.082921</td>\n",
       "      <td>1.041910</td>\n",
       "      <td>0.660726</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.951288</td>\n",
       "      <td>1.005008</td>\n",
       "      <td>0.678548</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.851222</td>\n",
       "      <td>0.856578</td>\n",
       "      <td>0.740316</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.781847</td>\n",
       "      <td>0.825425</td>\n",
       "      <td>0.741943</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.738515</td>\n",
       "      <td>0.803925</td>\n",
       "      <td>0.751383</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.707328</td>\n",
       "      <td>0.799827</td>\n",
       "      <td>0.730225</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.689486</td>\n",
       "      <td>0.812231</td>\n",
       "      <td>0.734294</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.679906</td>\n",
       "      <td>0.798167</td>\n",
       "      <td>0.741130</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.673696</td>\n",
       "      <td>0.799959</td>\n",
       "      <td>0.740641</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: qkv, run number: 0 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.126371</td>\n",
       "      <td>2.854813</td>\n",
       "      <td>0.153890</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.803911</td>\n",
       "      <td>2.469023</td>\n",
       "      <td>0.234294</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.124386</td>\n",
       "      <td>1.708857</td>\n",
       "      <td>0.481934</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1.647635</td>\n",
       "      <td>1.631472</td>\n",
       "      <td>0.506836</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1.323307</td>\n",
       "      <td>1.305052</td>\n",
       "      <td>0.587972</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.069267</td>\n",
       "      <td>1.003357</td>\n",
       "      <td>0.674316</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>0.885686</td>\n",
       "      <td>0.899739</td>\n",
       "      <td>0.725505</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.769194</td>\n",
       "      <td>0.796518</td>\n",
       "      <td>0.749023</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.701276</td>\n",
       "      <td>0.789625</td>\n",
       "      <td>0.736572</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.659822</td>\n",
       "      <td>0.831674</td>\n",
       "      <td>0.743164</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.635121</td>\n",
       "      <td>0.800752</td>\n",
       "      <td>0.740234</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.617488</td>\n",
       "      <td>0.798432</td>\n",
       "      <td>0.743896</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.605799</td>\n",
       "      <td>0.800841</td>\n",
       "      <td>0.739176</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.599844</td>\n",
       "      <td>0.798512</td>\n",
       "      <td>0.744792</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.596778</td>\n",
       "      <td>0.799637</td>\n",
       "      <td>0.744303</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: qkv, run number: 1 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.438013</td>\n",
       "      <td>2.933906</td>\n",
       "      <td>0.178304</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.944735</td>\n",
       "      <td>2.708915</td>\n",
       "      <td>0.192546</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.232578</td>\n",
       "      <td>1.822949</td>\n",
       "      <td>0.445150</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1.725628</td>\n",
       "      <td>1.688974</td>\n",
       "      <td>0.522054</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1.393994</td>\n",
       "      <td>1.559400</td>\n",
       "      <td>0.544678</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.162757</td>\n",
       "      <td>1.321178</td>\n",
       "      <td>0.582764</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>0.993218</td>\n",
       "      <td>1.283090</td>\n",
       "      <td>0.598958</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.873510</td>\n",
       "      <td>1.088197</td>\n",
       "      <td>0.652507</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.781377</td>\n",
       "      <td>1.056127</td>\n",
       "      <td>0.671143</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.717942</td>\n",
       "      <td>1.070560</td>\n",
       "      <td>0.663005</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.677158</td>\n",
       "      <td>0.981954</td>\n",
       "      <td>0.694417</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.650476</td>\n",
       "      <td>1.029456</td>\n",
       "      <td>0.681478</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.633166</td>\n",
       "      <td>1.026260</td>\n",
       "      <td>0.678223</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.627566</td>\n",
       "      <td>0.999555</td>\n",
       "      <td>0.687744</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.620932</td>\n",
       "      <td>1.005780</td>\n",
       "      <td>0.681885</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: qkv, run number: 2 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.199775</td>\n",
       "      <td>2.840255</td>\n",
       "      <td>0.155924</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.885970</td>\n",
       "      <td>2.767675</td>\n",
       "      <td>0.158040</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.316361</td>\n",
       "      <td>1.987790</td>\n",
       "      <td>0.358236</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1.738982</td>\n",
       "      <td>1.669892</td>\n",
       "      <td>0.473307</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1.332402</td>\n",
       "      <td>1.229930</td>\n",
       "      <td>0.597982</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.063866</td>\n",
       "      <td>1.200493</td>\n",
       "      <td>0.626139</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>0.886864</td>\n",
       "      <td>1.028088</td>\n",
       "      <td>0.681641</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.766677</td>\n",
       "      <td>0.962205</td>\n",
       "      <td>0.704020</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.696091</td>\n",
       "      <td>0.961523</td>\n",
       "      <td>0.712565</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.653112</td>\n",
       "      <td>0.923405</td>\n",
       "      <td>0.722331</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.628153</td>\n",
       "      <td>0.899545</td>\n",
       "      <td>0.729248</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.610473</td>\n",
       "      <td>0.930577</td>\n",
       "      <td>0.722412</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.600217</td>\n",
       "      <td>0.952582</td>\n",
       "      <td>0.714437</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.594396</td>\n",
       "      <td>0.927872</td>\n",
       "      <td>0.725423</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.591937</td>\n",
       "      <td>0.925557</td>\n",
       "      <td>0.722168</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: qkv, run number: 3 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.207227</td>\n",
       "      <td>2.888162</td>\n",
       "      <td>0.138428</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.833928</td>\n",
       "      <td>2.431302</td>\n",
       "      <td>0.215413</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.196405</td>\n",
       "      <td>1.906998</td>\n",
       "      <td>0.382406</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1.699117</td>\n",
       "      <td>1.575192</td>\n",
       "      <td>0.520752</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1.333332</td>\n",
       "      <td>1.355064</td>\n",
       "      <td>0.574870</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.078866</td>\n",
       "      <td>1.135900</td>\n",
       "      <td>0.648356</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>0.911745</td>\n",
       "      <td>0.974671</td>\n",
       "      <td>0.703369</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.789000</td>\n",
       "      <td>1.061764</td>\n",
       "      <td>0.684814</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.711152</td>\n",
       "      <td>0.898400</td>\n",
       "      <td>0.727132</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.666203</td>\n",
       "      <td>0.868965</td>\n",
       "      <td>0.725993</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.640007</td>\n",
       "      <td>0.831103</td>\n",
       "      <td>0.738851</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.618669</td>\n",
       "      <td>0.837905</td>\n",
       "      <td>0.734782</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.608703</td>\n",
       "      <td>0.849219</td>\n",
       "      <td>0.733968</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.602987</td>\n",
       "      <td>0.832384</td>\n",
       "      <td>0.734049</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.598535</td>\n",
       "      <td>0.836909</td>\n",
       "      <td>0.733154</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: qkv, run number: 4 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.192735</td>\n",
       "      <td>2.827039</td>\n",
       "      <td>0.149495</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.872044</td>\n",
       "      <td>2.732201</td>\n",
       "      <td>0.129964</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.252523</td>\n",
       "      <td>1.986052</td>\n",
       "      <td>0.299479</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1.736533</td>\n",
       "      <td>1.597513</td>\n",
       "      <td>0.486247</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1.346651</td>\n",
       "      <td>1.296470</td>\n",
       "      <td>0.569499</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.085373</td>\n",
       "      <td>1.119548</td>\n",
       "      <td>0.635417</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>0.929421</td>\n",
       "      <td>0.955579</td>\n",
       "      <td>0.699300</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.813634</td>\n",
       "      <td>0.938610</td>\n",
       "      <td>0.727132</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.733221</td>\n",
       "      <td>0.875827</td>\n",
       "      <td>0.728597</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.680856</td>\n",
       "      <td>0.850010</td>\n",
       "      <td>0.734294</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.647435</td>\n",
       "      <td>0.818790</td>\n",
       "      <td>0.739665</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.627903</td>\n",
       "      <td>0.821096</td>\n",
       "      <td>0.741455</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.615833</td>\n",
       "      <td>0.814034</td>\n",
       "      <td>0.743245</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.606631</td>\n",
       "      <td>0.807765</td>\n",
       "      <td>0.744385</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.603263</td>\n",
       "      <td>0.808049</td>\n",
       "      <td>0.744303</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv, run number: 0 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.393529</td>\n",
       "      <td>3.001172</td>\n",
       "      <td>0.127035</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.041160</td>\n",
       "      <td>2.778927</td>\n",
       "      <td>0.121582</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.789026</td>\n",
       "      <td>2.348497</td>\n",
       "      <td>0.166178</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.398091</td>\n",
       "      <td>2.255796</td>\n",
       "      <td>0.203695</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.089827</td>\n",
       "      <td>2.180502</td>\n",
       "      <td>0.294189</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.814277</td>\n",
       "      <td>1.945096</td>\n",
       "      <td>0.373047</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.556556</td>\n",
       "      <td>1.525931</td>\n",
       "      <td>0.511800</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.331787</td>\n",
       "      <td>1.216697</td>\n",
       "      <td>0.602214</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.160109</td>\n",
       "      <td>1.119630</td>\n",
       "      <td>0.634277</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.035079</td>\n",
       "      <td>1.010087</td>\n",
       "      <td>0.671956</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.944026</td>\n",
       "      <td>1.070367</td>\n",
       "      <td>0.664225</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.880972</td>\n",
       "      <td>1.033161</td>\n",
       "      <td>0.672119</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.837097</td>\n",
       "      <td>1.031386</td>\n",
       "      <td>0.674479</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.807368</td>\n",
       "      <td>0.989681</td>\n",
       "      <td>0.686686</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.788521</td>\n",
       "      <td>1.000026</td>\n",
       "      <td>0.684326</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv, run number: 1 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.661893</td>\n",
       "      <td>3.171602</td>\n",
       "      <td>0.147135</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.192876</td>\n",
       "      <td>2.779485</td>\n",
       "      <td>0.139567</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.862332</td>\n",
       "      <td>2.486130</td>\n",
       "      <td>0.241536</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.475781</td>\n",
       "      <td>1.990853</td>\n",
       "      <td>0.253906</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.141692</td>\n",
       "      <td>2.050632</td>\n",
       "      <td>0.288981</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.873590</td>\n",
       "      <td>2.019208</td>\n",
       "      <td>0.304443</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.646640</td>\n",
       "      <td>2.040819</td>\n",
       "      <td>0.342529</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.458590</td>\n",
       "      <td>1.907932</td>\n",
       "      <td>0.374023</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.298475</td>\n",
       "      <td>1.650696</td>\n",
       "      <td>0.451986</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.169873</td>\n",
       "      <td>1.563750</td>\n",
       "      <td>0.507975</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.071038</td>\n",
       "      <td>1.499576</td>\n",
       "      <td>0.539632</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.995929</td>\n",
       "      <td>1.351174</td>\n",
       "      <td>0.557454</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.943771</td>\n",
       "      <td>1.300040</td>\n",
       "      <td>0.573079</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.908635</td>\n",
       "      <td>1.309141</td>\n",
       "      <td>0.574788</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.885919</td>\n",
       "      <td>1.305756</td>\n",
       "      <td>0.575684</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv, run number: 2 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.338608</td>\n",
       "      <td>2.958845</td>\n",
       "      <td>0.153727</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.041699</td>\n",
       "      <td>2.813343</td>\n",
       "      <td>0.149821</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.864466</td>\n",
       "      <td>2.419638</td>\n",
       "      <td>0.163249</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.495135</td>\n",
       "      <td>2.029039</td>\n",
       "      <td>0.244548</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.159680</td>\n",
       "      <td>2.123187</td>\n",
       "      <td>0.213053</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.892215</td>\n",
       "      <td>2.034806</td>\n",
       "      <td>0.271973</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.661490</td>\n",
       "      <td>1.956011</td>\n",
       "      <td>0.339600</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.456574</td>\n",
       "      <td>1.626559</td>\n",
       "      <td>0.426188</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.278591</td>\n",
       "      <td>1.486161</td>\n",
       "      <td>0.488688</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.140106</td>\n",
       "      <td>1.295008</td>\n",
       "      <td>0.546468</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.036828</td>\n",
       "      <td>1.245304</td>\n",
       "      <td>0.571126</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.961942</td>\n",
       "      <td>1.199774</td>\n",
       "      <td>0.590658</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.909927</td>\n",
       "      <td>1.182478</td>\n",
       "      <td>0.591797</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.874270</td>\n",
       "      <td>1.157099</td>\n",
       "      <td>0.599609</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.853803</td>\n",
       "      <td>1.152192</td>\n",
       "      <td>0.600993</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv, run number: 3 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.554806</td>\n",
       "      <td>3.157939</td>\n",
       "      <td>0.138509</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.135639</td>\n",
       "      <td>2.831849</td>\n",
       "      <td>0.149251</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.869622</td>\n",
       "      <td>2.503117</td>\n",
       "      <td>0.161784</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.465970</td>\n",
       "      <td>2.256011</td>\n",
       "      <td>0.207520</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.178779</td>\n",
       "      <td>2.242734</td>\n",
       "      <td>0.193359</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.938494</td>\n",
       "      <td>2.212176</td>\n",
       "      <td>0.237305</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.720776</td>\n",
       "      <td>2.093378</td>\n",
       "      <td>0.318604</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.535247</td>\n",
       "      <td>2.130716</td>\n",
       "      <td>0.337240</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.373782</td>\n",
       "      <td>1.691186</td>\n",
       "      <td>0.415283</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.229297</td>\n",
       "      <td>1.627900</td>\n",
       "      <td>0.490641</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.108412</td>\n",
       "      <td>1.475220</td>\n",
       "      <td>0.558594</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.017215</td>\n",
       "      <td>1.407626</td>\n",
       "      <td>0.573568</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.949689</td>\n",
       "      <td>1.387002</td>\n",
       "      <td>0.585938</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.902415</td>\n",
       "      <td>1.351619</td>\n",
       "      <td>0.587077</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.871385</td>\n",
       "      <td>1.356790</td>\n",
       "      <td>0.589518</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv, run number: 4 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.627999</td>\n",
       "      <td>3.114597</td>\n",
       "      <td>0.141357</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.141501</td>\n",
       "      <td>2.684431</td>\n",
       "      <td>0.111898</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.704339</td>\n",
       "      <td>2.100744</td>\n",
       "      <td>0.182454</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.318949</td>\n",
       "      <td>2.021231</td>\n",
       "      <td>0.248047</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.023443</td>\n",
       "      <td>1.982339</td>\n",
       "      <td>0.322266</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.774339</td>\n",
       "      <td>2.047803</td>\n",
       "      <td>0.320312</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.561549</td>\n",
       "      <td>1.801184</td>\n",
       "      <td>0.405518</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.371079</td>\n",
       "      <td>1.352699</td>\n",
       "      <td>0.548584</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.213627</td>\n",
       "      <td>1.313427</td>\n",
       "      <td>0.604248</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.089453</td>\n",
       "      <td>1.168114</td>\n",
       "      <td>0.634928</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.997568</td>\n",
       "      <td>1.092779</td>\n",
       "      <td>0.637695</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.934060</td>\n",
       "      <td>1.108246</td>\n",
       "      <td>0.646891</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.886791</td>\n",
       "      <td>1.109776</td>\n",
       "      <td>0.648763</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.856142</td>\n",
       "      <td>1.074375</td>\n",
       "      <td>0.653320</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.834467</td>\n",
       "      <td>1.086297</td>\n",
       "      <td>0.652669</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: qkv, run number: 0 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.322006</td>\n",
       "      <td>2.905356</td>\n",
       "      <td>0.154541</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.018939</td>\n",
       "      <td>2.806235</td>\n",
       "      <td>0.159098</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.844239</td>\n",
       "      <td>2.645578</td>\n",
       "      <td>0.155273</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.463679</td>\n",
       "      <td>2.166068</td>\n",
       "      <td>0.248210</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.103963</td>\n",
       "      <td>2.231103</td>\n",
       "      <td>0.335042</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.787865</td>\n",
       "      <td>2.116756</td>\n",
       "      <td>0.339030</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.520120</td>\n",
       "      <td>1.905909</td>\n",
       "      <td>0.407715</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.302812</td>\n",
       "      <td>1.619077</td>\n",
       "      <td>0.481689</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.125919</td>\n",
       "      <td>1.525010</td>\n",
       "      <td>0.502848</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.987772</td>\n",
       "      <td>1.344381</td>\n",
       "      <td>0.563558</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.886671</td>\n",
       "      <td>1.275475</td>\n",
       "      <td>0.593913</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.815300</td>\n",
       "      <td>1.232477</td>\n",
       "      <td>0.611898</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.764783</td>\n",
       "      <td>1.194154</td>\n",
       "      <td>0.621745</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.728874</td>\n",
       "      <td>1.198390</td>\n",
       "      <td>0.619710</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.706431</td>\n",
       "      <td>1.203610</td>\n",
       "      <td>0.618815</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: qkv, run number: 1 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.675939</td>\n",
       "      <td>3.286580</td>\n",
       "      <td>0.153727</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.234025</td>\n",
       "      <td>2.846916</td>\n",
       "      <td>0.152588</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.963362</td>\n",
       "      <td>2.471504</td>\n",
       "      <td>0.217855</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.519038</td>\n",
       "      <td>2.198577</td>\n",
       "      <td>0.306234</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.128155</td>\n",
       "      <td>2.028503</td>\n",
       "      <td>0.315674</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.776640</td>\n",
       "      <td>1.718396</td>\n",
       "      <td>0.434489</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.459444</td>\n",
       "      <td>1.343228</td>\n",
       "      <td>0.563558</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.214571</td>\n",
       "      <td>1.231956</td>\n",
       "      <td>0.620768</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.038331</td>\n",
       "      <td>1.078258</td>\n",
       "      <td>0.656494</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.915064</td>\n",
       "      <td>1.038143</td>\n",
       "      <td>0.661621</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.826779</td>\n",
       "      <td>1.014846</td>\n",
       "      <td>0.668701</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.763628</td>\n",
       "      <td>1.049435</td>\n",
       "      <td>0.677409</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.721294</td>\n",
       "      <td>1.065108</td>\n",
       "      <td>0.683675</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.692329</td>\n",
       "      <td>1.048420</td>\n",
       "      <td>0.684652</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.672358</td>\n",
       "      <td>1.045334</td>\n",
       "      <td>0.685547</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: qkv, run number: 2 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.283466</td>\n",
       "      <td>2.953053</td>\n",
       "      <td>0.147298</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.008454</td>\n",
       "      <td>2.792472</td>\n",
       "      <td>0.134196</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.770570</td>\n",
       "      <td>2.173727</td>\n",
       "      <td>0.199870</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.361310</td>\n",
       "      <td>2.053098</td>\n",
       "      <td>0.231283</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.022636</td>\n",
       "      <td>1.954606</td>\n",
       "      <td>0.292318</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.728628</td>\n",
       "      <td>1.872171</td>\n",
       "      <td>0.340658</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.481518</td>\n",
       "      <td>1.807692</td>\n",
       "      <td>0.391195</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.276068</td>\n",
       "      <td>1.574686</td>\n",
       "      <td>0.462728</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.105634</td>\n",
       "      <td>1.442695</td>\n",
       "      <td>0.500732</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.973168</td>\n",
       "      <td>1.301105</td>\n",
       "      <td>0.531494</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.875911</td>\n",
       "      <td>1.239028</td>\n",
       "      <td>0.554443</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.804779</td>\n",
       "      <td>1.162338</td>\n",
       "      <td>0.574219</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.754469</td>\n",
       "      <td>1.110021</td>\n",
       "      <td>0.591960</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.719821</td>\n",
       "      <td>1.101122</td>\n",
       "      <td>0.596110</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.696160</td>\n",
       "      <td>1.097466</td>\n",
       "      <td>0.596436</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: qkv, run number: 3 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.626357</td>\n",
       "      <td>3.202688</td>\n",
       "      <td>0.136963</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.206197</td>\n",
       "      <td>2.827932</td>\n",
       "      <td>0.134521</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.970742</td>\n",
       "      <td>2.744233</td>\n",
       "      <td>0.107992</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.599782</td>\n",
       "      <td>2.079245</td>\n",
       "      <td>0.225586</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.230227</td>\n",
       "      <td>1.984308</td>\n",
       "      <td>0.271077</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.922301</td>\n",
       "      <td>1.884548</td>\n",
       "      <td>0.310547</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.661190</td>\n",
       "      <td>1.882179</td>\n",
       "      <td>0.368164</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.443450</td>\n",
       "      <td>1.848705</td>\n",
       "      <td>0.406576</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.254639</td>\n",
       "      <td>1.375580</td>\n",
       "      <td>0.482178</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.094875</td>\n",
       "      <td>1.200894</td>\n",
       "      <td>0.569092</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.975859</td>\n",
       "      <td>1.059166</td>\n",
       "      <td>0.627767</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.886947</td>\n",
       "      <td>1.033764</td>\n",
       "      <td>0.642985</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.824665</td>\n",
       "      <td>1.034272</td>\n",
       "      <td>0.646322</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.781818</td>\n",
       "      <td>1.010408</td>\n",
       "      <td>0.650879</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.753029</td>\n",
       "      <td>1.007373</td>\n",
       "      <td>0.652588</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: qkv, run number: 4 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.616479</td>\n",
       "      <td>2.998990</td>\n",
       "      <td>0.158854</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.186210</td>\n",
       "      <td>2.821243</td>\n",
       "      <td>0.152018</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.958699</td>\n",
       "      <td>2.687315</td>\n",
       "      <td>0.104492</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.592973</td>\n",
       "      <td>2.173463</td>\n",
       "      <td>0.226644</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.226380</td>\n",
       "      <td>2.152412</td>\n",
       "      <td>0.257080</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.904694</td>\n",
       "      <td>2.042761</td>\n",
       "      <td>0.385173</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.608244</td>\n",
       "      <td>1.566036</td>\n",
       "      <td>0.471354</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.353468</td>\n",
       "      <td>1.278362</td>\n",
       "      <td>0.561117</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.158399</td>\n",
       "      <td>1.126146</td>\n",
       "      <td>0.632568</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.017708</td>\n",
       "      <td>1.043477</td>\n",
       "      <td>0.659098</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.918967</td>\n",
       "      <td>1.020396</td>\n",
       "      <td>0.672607</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.850062</td>\n",
       "      <td>0.978769</td>\n",
       "      <td>0.686686</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.802394</td>\n",
       "      <td>0.957578</td>\n",
       "      <td>0.696777</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.768972</td>\n",
       "      <td>0.949135</td>\n",
       "      <td>0.701091</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.747777</td>\n",
       "      <td>0.942077</td>\n",
       "      <td>0.702881</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv, run number: 0 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.786648</td>\n",
       "      <td>3.443423</td>\n",
       "      <td>0.034424</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.390014</td>\n",
       "      <td>2.907272</td>\n",
       "      <td>0.122314</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.141245</td>\n",
       "      <td>2.792983</td>\n",
       "      <td>0.151042</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.950041</td>\n",
       "      <td>2.490605</td>\n",
       "      <td>0.184408</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.694398</td>\n",
       "      <td>2.152139</td>\n",
       "      <td>0.246257</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.456744</td>\n",
       "      <td>2.060925</td>\n",
       "      <td>0.218750</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.260626</td>\n",
       "      <td>2.056519</td>\n",
       "      <td>0.253743</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.096862</td>\n",
       "      <td>2.127450</td>\n",
       "      <td>0.240072</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.955330</td>\n",
       "      <td>2.093332</td>\n",
       "      <td>0.215495</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.831894</td>\n",
       "      <td>2.152003</td>\n",
       "      <td>0.248291</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.725680</td>\n",
       "      <td>2.125997</td>\n",
       "      <td>0.258626</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.635231</td>\n",
       "      <td>2.134191</td>\n",
       "      <td>0.254395</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.561857</td>\n",
       "      <td>2.164215</td>\n",
       "      <td>0.256917</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.502430</td>\n",
       "      <td>2.178493</td>\n",
       "      <td>0.255778</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.455339</td>\n",
       "      <td>2.186927</td>\n",
       "      <td>0.253825</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv, run number: 1 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.560938</td>\n",
       "      <td>3.218148</td>\n",
       "      <td>0.127930</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.263213</td>\n",
       "      <td>2.881521</td>\n",
       "      <td>0.159180</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.079265</td>\n",
       "      <td>2.839216</td>\n",
       "      <td>0.083577</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.958641</td>\n",
       "      <td>2.786019</td>\n",
       "      <td>0.140299</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.797669</td>\n",
       "      <td>2.486819</td>\n",
       "      <td>0.222087</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.569824</td>\n",
       "      <td>2.161576</td>\n",
       "      <td>0.211995</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.365146</td>\n",
       "      <td>2.186734</td>\n",
       "      <td>0.195312</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.194447</td>\n",
       "      <td>2.157119</td>\n",
       "      <td>0.211833</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.049576</td>\n",
       "      <td>2.143222</td>\n",
       "      <td>0.236165</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.921771</td>\n",
       "      <td>2.050793</td>\n",
       "      <td>0.253581</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.810161</td>\n",
       "      <td>1.996025</td>\n",
       "      <td>0.270264</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.712961</td>\n",
       "      <td>1.972274</td>\n",
       "      <td>0.279948</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.631969</td>\n",
       "      <td>1.918643</td>\n",
       "      <td>0.284587</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.567064</td>\n",
       "      <td>1.920778</td>\n",
       "      <td>0.288493</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.514364</td>\n",
       "      <td>1.928189</td>\n",
       "      <td>0.288493</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv, run number: 2 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.764611</td>\n",
       "      <td>3.466141</td>\n",
       "      <td>0.097249</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.394081</td>\n",
       "      <td>3.061090</td>\n",
       "      <td>0.116943</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.162601</td>\n",
       "      <td>2.791243</td>\n",
       "      <td>0.141764</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.021663</td>\n",
       "      <td>2.758688</td>\n",
       "      <td>0.151042</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.886313</td>\n",
       "      <td>2.344868</td>\n",
       "      <td>0.183594</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.663899</td>\n",
       "      <td>2.028331</td>\n",
       "      <td>0.234863</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.450929</td>\n",
       "      <td>2.027122</td>\n",
       "      <td>0.278971</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.273306</td>\n",
       "      <td>2.048230</td>\n",
       "      <td>0.285970</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.123489</td>\n",
       "      <td>2.022606</td>\n",
       "      <td>0.279785</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.995157</td>\n",
       "      <td>2.009215</td>\n",
       "      <td>0.287842</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.884186</td>\n",
       "      <td>1.943247</td>\n",
       "      <td>0.275716</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.789876</td>\n",
       "      <td>1.955305</td>\n",
       "      <td>0.278239</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.710743</td>\n",
       "      <td>1.940603</td>\n",
       "      <td>0.287435</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.647813</td>\n",
       "      <td>1.927733</td>\n",
       "      <td>0.283773</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.597368</td>\n",
       "      <td>1.927303</td>\n",
       "      <td>0.281820</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv, run number: 3 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.937311</td>\n",
       "      <td>3.520140</td>\n",
       "      <td>0.022217</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.469275</td>\n",
       "      <td>2.886132</td>\n",
       "      <td>0.143066</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.194350</td>\n",
       "      <td>2.774813</td>\n",
       "      <td>0.151611</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.023705</td>\n",
       "      <td>2.573309</td>\n",
       "      <td>0.146322</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.792653</td>\n",
       "      <td>2.103400</td>\n",
       "      <td>0.157552</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.556187</td>\n",
       "      <td>2.013309</td>\n",
       "      <td>0.182048</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.364610</td>\n",
       "      <td>2.130389</td>\n",
       "      <td>0.188558</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.208972</td>\n",
       "      <td>2.147259</td>\n",
       "      <td>0.186605</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.076491</td>\n",
       "      <td>2.205057</td>\n",
       "      <td>0.210205</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.963416</td>\n",
       "      <td>2.200417</td>\n",
       "      <td>0.211995</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.864799</td>\n",
       "      <td>2.191076</td>\n",
       "      <td>0.226156</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.783617</td>\n",
       "      <td>2.160321</td>\n",
       "      <td>0.240234</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.715535</td>\n",
       "      <td>2.177067</td>\n",
       "      <td>0.234131</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.660964</td>\n",
       "      <td>2.167851</td>\n",
       "      <td>0.239176</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.618209</td>\n",
       "      <td>2.172966</td>\n",
       "      <td>0.238851</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv, run number: 4 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.742989</td>\n",
       "      <td>3.540063</td>\n",
       "      <td>0.012939</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.389758</td>\n",
       "      <td>2.908855</td>\n",
       "      <td>0.132650</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.152061</td>\n",
       "      <td>2.789101</td>\n",
       "      <td>0.149740</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.995584</td>\n",
       "      <td>2.663342</td>\n",
       "      <td>0.156494</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.762866</td>\n",
       "      <td>2.160289</td>\n",
       "      <td>0.205404</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.513678</td>\n",
       "      <td>2.102845</td>\n",
       "      <td>0.241781</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.306990</td>\n",
       "      <td>2.091374</td>\n",
       "      <td>0.266520</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.133886</td>\n",
       "      <td>2.085315</td>\n",
       "      <td>0.272868</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.987639</td>\n",
       "      <td>2.122685</td>\n",
       "      <td>0.240316</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.863372</td>\n",
       "      <td>2.136404</td>\n",
       "      <td>0.274007</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.757827</td>\n",
       "      <td>2.129403</td>\n",
       "      <td>0.265137</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.669007</td>\n",
       "      <td>2.207108</td>\n",
       "      <td>0.271729</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.595883</td>\n",
       "      <td>2.202402</td>\n",
       "      <td>0.270996</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.536824</td>\n",
       "      <td>2.207287</td>\n",
       "      <td>0.271973</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.489783</td>\n",
       "      <td>2.199377</td>\n",
       "      <td>0.273844</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: qkv, run number: 0 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.123415</td>\n",
       "      <td>3.682958</td>\n",
       "      <td>0.123698</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.663893</td>\n",
       "      <td>3.028462</td>\n",
       "      <td>0.149984</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.328714</td>\n",
       "      <td>2.812965</td>\n",
       "      <td>0.118815</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.120908</td>\n",
       "      <td>2.754214</td>\n",
       "      <td>0.109212</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.911127</td>\n",
       "      <td>2.339564</td>\n",
       "      <td>0.168783</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.648442</td>\n",
       "      <td>2.065583</td>\n",
       "      <td>0.222331</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.419489</td>\n",
       "      <td>2.058681</td>\n",
       "      <td>0.185384</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.226226</td>\n",
       "      <td>2.096517</td>\n",
       "      <td>0.208577</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.057988</td>\n",
       "      <td>2.138140</td>\n",
       "      <td>0.222900</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.913268</td>\n",
       "      <td>2.135828</td>\n",
       "      <td>0.228597</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.790560</td>\n",
       "      <td>2.144373</td>\n",
       "      <td>0.234294</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.687495</td>\n",
       "      <td>2.126568</td>\n",
       "      <td>0.247396</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.602326</td>\n",
       "      <td>2.131693</td>\n",
       "      <td>0.252930</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.533526</td>\n",
       "      <td>2.134682</td>\n",
       "      <td>0.255859</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.479849</td>\n",
       "      <td>2.134258</td>\n",
       "      <td>0.257243</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: qkv, run number: 1 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.558767</td>\n",
       "      <td>3.074125</td>\n",
       "      <td>0.068441</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.259746</td>\n",
       "      <td>2.914386</td>\n",
       "      <td>0.138021</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.074975</td>\n",
       "      <td>2.809415</td>\n",
       "      <td>0.118978</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.962560</td>\n",
       "      <td>2.792138</td>\n",
       "      <td>0.159587</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.850901</td>\n",
       "      <td>2.411124</td>\n",
       "      <td>0.181885</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.651581</td>\n",
       "      <td>2.073038</td>\n",
       "      <td>0.267660</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.438972</td>\n",
       "      <td>1.968406</td>\n",
       "      <td>0.284993</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.251179</td>\n",
       "      <td>1.978583</td>\n",
       "      <td>0.247233</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.086227</td>\n",
       "      <td>1.958969</td>\n",
       "      <td>0.264648</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.941540</td>\n",
       "      <td>1.873665</td>\n",
       "      <td>0.273275</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.813909</td>\n",
       "      <td>1.904835</td>\n",
       "      <td>0.287760</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.705416</td>\n",
       "      <td>1.906943</td>\n",
       "      <td>0.281331</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.615114</td>\n",
       "      <td>1.897701</td>\n",
       "      <td>0.290527</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.542627</td>\n",
       "      <td>1.907062</td>\n",
       "      <td>0.286947</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.486252</td>\n",
       "      <td>1.909147</td>\n",
       "      <td>0.287028</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: qkv, run number: 2 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.738606</td>\n",
       "      <td>3.277540</td>\n",
       "      <td>0.107747</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.365715</td>\n",
       "      <td>2.929230</td>\n",
       "      <td>0.138753</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.141736</td>\n",
       "      <td>2.818784</td>\n",
       "      <td>0.151042</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.010460</td>\n",
       "      <td>2.804560</td>\n",
       "      <td>0.144124</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.902296</td>\n",
       "      <td>2.607884</td>\n",
       "      <td>0.153890</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.706191</td>\n",
       "      <td>2.185438</td>\n",
       "      <td>0.237793</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.485104</td>\n",
       "      <td>2.123076</td>\n",
       "      <td>0.209798</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.288338</td>\n",
       "      <td>2.138227</td>\n",
       "      <td>0.214600</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.117483</td>\n",
       "      <td>2.222909</td>\n",
       "      <td>0.229574</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.967729</td>\n",
       "      <td>2.249841</td>\n",
       "      <td>0.241211</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.836730</td>\n",
       "      <td>2.258175</td>\n",
       "      <td>0.251058</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.727536</td>\n",
       "      <td>2.300864</td>\n",
       "      <td>0.260986</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.637313</td>\n",
       "      <td>2.293713</td>\n",
       "      <td>0.262858</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.564447</td>\n",
       "      <td>2.283865</td>\n",
       "      <td>0.266764</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.506466</td>\n",
       "      <td>2.290050</td>\n",
       "      <td>0.264974</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: qkv, run number: 3 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.799785</td>\n",
       "      <td>3.468743</td>\n",
       "      <td>0.104899</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.410474</td>\n",
       "      <td>2.906385</td>\n",
       "      <td>0.126383</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.165561</td>\n",
       "      <td>2.813470</td>\n",
       "      <td>0.150146</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.024117</td>\n",
       "      <td>2.754552</td>\n",
       "      <td>0.150228</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.896191</td>\n",
       "      <td>2.475426</td>\n",
       "      <td>0.190348</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.683941</td>\n",
       "      <td>2.095858</td>\n",
       "      <td>0.215983</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.459024</td>\n",
       "      <td>2.067529</td>\n",
       "      <td>0.198405</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.262044</td>\n",
       "      <td>2.134962</td>\n",
       "      <td>0.222331</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.088563</td>\n",
       "      <td>2.133756</td>\n",
       "      <td>0.230632</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.938598</td>\n",
       "      <td>2.139222</td>\n",
       "      <td>0.255127</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.809746</td>\n",
       "      <td>2.194618</td>\n",
       "      <td>0.264648</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.701664</td>\n",
       "      <td>2.179216</td>\n",
       "      <td>0.269694</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.612158</td>\n",
       "      <td>2.203246</td>\n",
       "      <td>0.265869</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.540274</td>\n",
       "      <td>2.185686</td>\n",
       "      <td>0.275391</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.484836</td>\n",
       "      <td>2.194176</td>\n",
       "      <td>0.272542</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: qkv, run number: 4 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.992415</td>\n",
       "      <td>3.433018</td>\n",
       "      <td>0.029622</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.539213</td>\n",
       "      <td>2.877968</td>\n",
       "      <td>0.130452</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.239172</td>\n",
       "      <td>2.809046</td>\n",
       "      <td>0.146322</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.046509</td>\n",
       "      <td>2.580565</td>\n",
       "      <td>0.138184</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.790208</td>\n",
       "      <td>2.034219</td>\n",
       "      <td>0.239665</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.530278</td>\n",
       "      <td>1.987964</td>\n",
       "      <td>0.269694</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.312816</td>\n",
       "      <td>2.026698</td>\n",
       "      <td>0.290690</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.127237</td>\n",
       "      <td>2.037697</td>\n",
       "      <td>0.293701</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.964619</td>\n",
       "      <td>2.014831</td>\n",
       "      <td>0.275960</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.825566</td>\n",
       "      <td>2.046067</td>\n",
       "      <td>0.274089</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.706381</td>\n",
       "      <td>2.038357</td>\n",
       "      <td>0.278158</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.606682</td>\n",
       "      <td>2.061013</td>\n",
       "      <td>0.277181</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.524806</td>\n",
       "      <td>2.067924</td>\n",
       "      <td>0.283040</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.458603</td>\n",
       "      <td>2.055263</td>\n",
       "      <td>0.283366</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.406925</td>\n",
       "      <td>2.056545</td>\n",
       "      <td>0.283285</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv, run number: 0 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.779950</td>\n",
       "      <td>3.620638</td>\n",
       "      <td>0.115479</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.534728</td>\n",
       "      <td>3.176333</td>\n",
       "      <td>0.147217</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.317591</td>\n",
       "      <td>2.958775</td>\n",
       "      <td>0.133423</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.169337</td>\n",
       "      <td>2.819496</td>\n",
       "      <td>0.135864</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.065526</td>\n",
       "      <td>2.741374</td>\n",
       "      <td>0.128906</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.980413</td>\n",
       "      <td>2.650137</td>\n",
       "      <td>0.132568</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.880701</td>\n",
       "      <td>2.391030</td>\n",
       "      <td>0.171387</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.762758</td>\n",
       "      <td>2.233511</td>\n",
       "      <td>0.170654</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.642931</td>\n",
       "      <td>2.124956</td>\n",
       "      <td>0.215454</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.532384</td>\n",
       "      <td>2.058960</td>\n",
       "      <td>0.221558</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.434199</td>\n",
       "      <td>2.041093</td>\n",
       "      <td>0.243530</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.349746</td>\n",
       "      <td>2.002544</td>\n",
       "      <td>0.252930</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.276571</td>\n",
       "      <td>1.984753</td>\n",
       "      <td>0.253174</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.214891</td>\n",
       "      <td>2.003264</td>\n",
       "      <td>0.250244</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.161992</td>\n",
       "      <td>1.995071</td>\n",
       "      <td>0.253296</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv, run number: 1 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.763236</td>\n",
       "      <td>3.428140</td>\n",
       "      <td>0.047852</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.510207</td>\n",
       "      <td>3.059750</td>\n",
       "      <td>0.145996</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.292147</td>\n",
       "      <td>2.895185</td>\n",
       "      <td>0.147949</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.146852</td>\n",
       "      <td>2.845733</td>\n",
       "      <td>0.146729</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.043314</td>\n",
       "      <td>2.749339</td>\n",
       "      <td>0.150879</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.940920</td>\n",
       "      <td>2.610408</td>\n",
       "      <td>0.169556</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.809343</td>\n",
       "      <td>2.417502</td>\n",
       "      <td>0.205322</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.680020</td>\n",
       "      <td>2.318129</td>\n",
       "      <td>0.238403</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.556323</td>\n",
       "      <td>2.255854</td>\n",
       "      <td>0.244385</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.442540</td>\n",
       "      <td>2.155138</td>\n",
       "      <td>0.230957</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.341932</td>\n",
       "      <td>2.144346</td>\n",
       "      <td>0.236328</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.254886</td>\n",
       "      <td>2.120174</td>\n",
       "      <td>0.227905</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.180538</td>\n",
       "      <td>2.084403</td>\n",
       "      <td>0.239014</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.117025</td>\n",
       "      <td>2.103563</td>\n",
       "      <td>0.238647</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.063135</td>\n",
       "      <td>2.095209</td>\n",
       "      <td>0.227783</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv, run number: 2 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.538267</td>\n",
       "      <td>3.188992</td>\n",
       "      <td>0.066284</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.335859</td>\n",
       "      <td>2.926776</td>\n",
       "      <td>0.142944</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.160795</td>\n",
       "      <td>2.809565</td>\n",
       "      <td>0.134888</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.039974</td>\n",
       "      <td>2.740454</td>\n",
       "      <td>0.111084</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.935435</td>\n",
       "      <td>2.522521</td>\n",
       "      <td>0.113770</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.794439</td>\n",
       "      <td>2.224814</td>\n",
       "      <td>0.187256</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.643646</td>\n",
       "      <td>2.095000</td>\n",
       "      <td>0.198486</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.508358</td>\n",
       "      <td>2.047927</td>\n",
       "      <td>0.201294</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.390952</td>\n",
       "      <td>2.045635</td>\n",
       "      <td>0.241455</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.289689</td>\n",
       "      <td>2.078646</td>\n",
       "      <td>0.231445</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.202756</td>\n",
       "      <td>2.036314</td>\n",
       "      <td>0.259521</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.128256</td>\n",
       "      <td>2.037366</td>\n",
       "      <td>0.261719</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.064673</td>\n",
       "      <td>2.041625</td>\n",
       "      <td>0.253052</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.010630</td>\n",
       "      <td>2.019937</td>\n",
       "      <td>0.258667</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.964692</td>\n",
       "      <td>2.017975</td>\n",
       "      <td>0.262573</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv, run number: 3 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.653085</td>\n",
       "      <td>3.444927</td>\n",
       "      <td>0.049927</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.410664</td>\n",
       "      <td>3.045062</td>\n",
       "      <td>0.133911</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.195814</td>\n",
       "      <td>2.848503</td>\n",
       "      <td>0.130737</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.034428</td>\n",
       "      <td>2.632292</td>\n",
       "      <td>0.164185</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.871154</td>\n",
       "      <td>2.521423</td>\n",
       "      <td>0.158203</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.724978</td>\n",
       "      <td>2.425130</td>\n",
       "      <td>0.195190</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.596575</td>\n",
       "      <td>2.426666</td>\n",
       "      <td>0.235474</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.479253</td>\n",
       "      <td>2.262484</td>\n",
       "      <td>0.200562</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.371669</td>\n",
       "      <td>2.246390</td>\n",
       "      <td>0.230835</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.277334</td>\n",
       "      <td>2.266978</td>\n",
       "      <td>0.235840</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.195402</td>\n",
       "      <td>2.269067</td>\n",
       "      <td>0.217529</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.123983</td>\n",
       "      <td>2.308841</td>\n",
       "      <td>0.217529</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.062958</td>\n",
       "      <td>2.269947</td>\n",
       "      <td>0.239868</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.010510</td>\n",
       "      <td>2.250391</td>\n",
       "      <td>0.237427</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.966735</td>\n",
       "      <td>2.257782</td>\n",
       "      <td>0.242554</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv, run number: 4 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.832075</td>\n",
       "      <td>3.593876</td>\n",
       "      <td>0.083618</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.561356</td>\n",
       "      <td>3.088274</td>\n",
       "      <td>0.153564</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.323626</td>\n",
       "      <td>2.820665</td>\n",
       "      <td>0.133545</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.164889</td>\n",
       "      <td>2.827059</td>\n",
       "      <td>0.155762</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.055079</td>\n",
       "      <td>2.721776</td>\n",
       "      <td>0.113281</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.946341</td>\n",
       "      <td>2.503030</td>\n",
       "      <td>0.129150</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.811435</td>\n",
       "      <td>2.222337</td>\n",
       "      <td>0.171753</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.669329</td>\n",
       "      <td>2.094936</td>\n",
       "      <td>0.187378</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.542354</td>\n",
       "      <td>2.044702</td>\n",
       "      <td>0.177490</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.432746</td>\n",
       "      <td>2.081752</td>\n",
       "      <td>0.181030</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.337950</td>\n",
       "      <td>2.077515</td>\n",
       "      <td>0.150757</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.257401</td>\n",
       "      <td>2.080302</td>\n",
       "      <td>0.167847</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.188289</td>\n",
       "      <td>2.091949</td>\n",
       "      <td>0.187866</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.129915</td>\n",
       "      <td>2.096571</td>\n",
       "      <td>0.192505</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.080594</td>\n",
       "      <td>2.068130</td>\n",
       "      <td>0.207764</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: qkv, run number: 0 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.818080</td>\n",
       "      <td>3.508749</td>\n",
       "      <td>0.036377</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.533694</td>\n",
       "      <td>2.972770</td>\n",
       "      <td>0.148193</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.301426</td>\n",
       "      <td>2.883048</td>\n",
       "      <td>0.127075</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.152441</td>\n",
       "      <td>2.807759</td>\n",
       "      <td>0.150269</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.046470</td>\n",
       "      <td>2.734226</td>\n",
       "      <td>0.071777</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.950099</td>\n",
       "      <td>2.571378</td>\n",
       "      <td>0.103516</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.815828</td>\n",
       "      <td>2.355838</td>\n",
       "      <td>0.164673</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.667740</td>\n",
       "      <td>2.120956</td>\n",
       "      <td>0.192505</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.533961</td>\n",
       "      <td>2.129471</td>\n",
       "      <td>0.242920</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.418701</td>\n",
       "      <td>2.122684</td>\n",
       "      <td>0.206421</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.320170</td>\n",
       "      <td>2.145543</td>\n",
       "      <td>0.218140</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.236482</td>\n",
       "      <td>2.168191</td>\n",
       "      <td>0.203003</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.164595</td>\n",
       "      <td>2.205129</td>\n",
       "      <td>0.192993</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.103129</td>\n",
       "      <td>2.205527</td>\n",
       "      <td>0.197144</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.051060</td>\n",
       "      <td>2.221873</td>\n",
       "      <td>0.186768</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: qkv, run number: 1 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.752044</td>\n",
       "      <td>3.426531</td>\n",
       "      <td>0.088135</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.528726</td>\n",
       "      <td>2.955788</td>\n",
       "      <td>0.179688</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.314596</td>\n",
       "      <td>2.850368</td>\n",
       "      <td>0.142822</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.166739</td>\n",
       "      <td>2.806790</td>\n",
       "      <td>0.149536</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.063057</td>\n",
       "      <td>2.762475</td>\n",
       "      <td>0.129028</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.975819</td>\n",
       "      <td>2.689634</td>\n",
       "      <td>0.163696</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.865307</td>\n",
       "      <td>2.410533</td>\n",
       "      <td>0.197388</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.726306</td>\n",
       "      <td>2.194293</td>\n",
       "      <td>0.210693</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.594058</td>\n",
       "      <td>2.088334</td>\n",
       "      <td>0.206787</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.475757</td>\n",
       "      <td>2.061948</td>\n",
       "      <td>0.208252</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.372826</td>\n",
       "      <td>2.049785</td>\n",
       "      <td>0.236084</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.284717</td>\n",
       "      <td>1.983080</td>\n",
       "      <td>0.222290</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.208947</td>\n",
       "      <td>2.004854</td>\n",
       "      <td>0.197388</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.144893</td>\n",
       "      <td>2.012524</td>\n",
       "      <td>0.197021</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.091278</td>\n",
       "      <td>2.020186</td>\n",
       "      <td>0.208862</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: qkv, run number: 2 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.752974</td>\n",
       "      <td>3.521869</td>\n",
       "      <td>0.064209</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.503753</td>\n",
       "      <td>3.046232</td>\n",
       "      <td>0.147339</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.282761</td>\n",
       "      <td>2.900546</td>\n",
       "      <td>0.152344</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.137003</td>\n",
       "      <td>2.797543</td>\n",
       "      <td>0.117676</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.036966</td>\n",
       "      <td>2.784209</td>\n",
       "      <td>0.123413</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.955060</td>\n",
       "      <td>2.751126</td>\n",
       "      <td>0.114014</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.863881</td>\n",
       "      <td>2.688503</td>\n",
       "      <td>0.175781</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.750164</td>\n",
       "      <td>2.494161</td>\n",
       "      <td>0.175659</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.628124</td>\n",
       "      <td>2.223534</td>\n",
       "      <td>0.223511</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.513535</td>\n",
       "      <td>2.182864</td>\n",
       "      <td>0.183838</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.411650</td>\n",
       "      <td>2.190731</td>\n",
       "      <td>0.216919</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.322968</td>\n",
       "      <td>2.177484</td>\n",
       "      <td>0.204468</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.246704</td>\n",
       "      <td>2.191951</td>\n",
       "      <td>0.196411</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.182219</td>\n",
       "      <td>2.188703</td>\n",
       "      <td>0.210083</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.127733</td>\n",
       "      <td>2.178325</td>\n",
       "      <td>0.212402</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: qkv, run number: 3 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.660381</td>\n",
       "      <td>3.298744</td>\n",
       "      <td>0.129028</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.418197</td>\n",
       "      <td>2.997445</td>\n",
       "      <td>0.124268</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.223088</td>\n",
       "      <td>2.853104</td>\n",
       "      <td>0.146606</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.095164</td>\n",
       "      <td>2.770655</td>\n",
       "      <td>0.075317</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.002727</td>\n",
       "      <td>2.720538</td>\n",
       "      <td>0.110718</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.904896</td>\n",
       "      <td>2.441361</td>\n",
       "      <td>0.145020</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.763900</td>\n",
       "      <td>2.180756</td>\n",
       "      <td>0.167358</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.622993</td>\n",
       "      <td>2.037804</td>\n",
       "      <td>0.220337</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.493161</td>\n",
       "      <td>1.990182</td>\n",
       "      <td>0.239502</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.379699</td>\n",
       "      <td>1.988093</td>\n",
       "      <td>0.208496</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.281974</td>\n",
       "      <td>1.961679</td>\n",
       "      <td>0.239136</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.197995</td>\n",
       "      <td>1.979465</td>\n",
       "      <td>0.236694</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.126114</td>\n",
       "      <td>2.001982</td>\n",
       "      <td>0.218872</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.064939</td>\n",
       "      <td>1.990827</td>\n",
       "      <td>0.211914</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.012540</td>\n",
       "      <td>1.949848</td>\n",
       "      <td>0.229248</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: qkv, run number: 4 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.744245</td>\n",
       "      <td>3.364651</td>\n",
       "      <td>0.046387</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.463769</td>\n",
       "      <td>2.942701</td>\n",
       "      <td>0.114380</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.249632</td>\n",
       "      <td>2.857760</td>\n",
       "      <td>0.123291</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.109915</td>\n",
       "      <td>2.756546</td>\n",
       "      <td>0.147705</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.008084</td>\n",
       "      <td>2.700984</td>\n",
       "      <td>0.135986</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.894123</td>\n",
       "      <td>2.481040</td>\n",
       "      <td>0.187622</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.746901</td>\n",
       "      <td>2.224102</td>\n",
       "      <td>0.209717</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.601740</td>\n",
       "      <td>2.085410</td>\n",
       "      <td>0.207764</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.473986</td>\n",
       "      <td>2.058539</td>\n",
       "      <td>0.223511</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.364160</td>\n",
       "      <td>2.033843</td>\n",
       "      <td>0.217651</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.269680</td>\n",
       "      <td>2.083060</td>\n",
       "      <td>0.216919</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.188889</td>\n",
       "      <td>2.090467</td>\n",
       "      <td>0.208496</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.119326</td>\n",
       "      <td>2.110051</td>\n",
       "      <td>0.207886</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.060334</td>\n",
       "      <td>2.077042</td>\n",
       "      <td>0.220947</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.010269</td>\n",
       "      <td>2.080143</td>\n",
       "      <td>0.211548</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv, run number: 0 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.739426</td>\n",
       "      <td>3.585954</td>\n",
       "      <td>0.045492</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.395606</td>\n",
       "      <td>2.977958</td>\n",
       "      <td>0.141846</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.066989</td>\n",
       "      <td>2.825778</td>\n",
       "      <td>0.150065</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.888824</td>\n",
       "      <td>2.780202</td>\n",
       "      <td>0.162923</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.757553</td>\n",
       "      <td>2.645855</td>\n",
       "      <td>0.168376</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.519654</td>\n",
       "      <td>2.336948</td>\n",
       "      <td>0.221354</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.251229</td>\n",
       "      <td>2.118922</td>\n",
       "      <td>0.287028</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.054818</td>\n",
       "      <td>2.040635</td>\n",
       "      <td>0.312826</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.923690</td>\n",
       "      <td>1.970276</td>\n",
       "      <td>0.350993</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.839479</td>\n",
       "      <td>1.928472</td>\n",
       "      <td>0.379639</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.783501</td>\n",
       "      <td>1.898394</td>\n",
       "      <td>0.391195</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.745867</td>\n",
       "      <td>1.880767</td>\n",
       "      <td>0.397624</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.724709</td>\n",
       "      <td>1.867640</td>\n",
       "      <td>0.399577</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.712869</td>\n",
       "      <td>1.863234</td>\n",
       "      <td>0.402751</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.704969</td>\n",
       "      <td>1.866359</td>\n",
       "      <td>0.401855</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv, run number: 1 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.811261</td>\n",
       "      <td>3.753230</td>\n",
       "      <td>0.099202</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.464804</td>\n",
       "      <td>3.080024</td>\n",
       "      <td>0.163737</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.096173</td>\n",
       "      <td>2.814667</td>\n",
       "      <td>0.186605</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.835804</td>\n",
       "      <td>2.562561</td>\n",
       "      <td>0.210856</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.509800</td>\n",
       "      <td>2.239202</td>\n",
       "      <td>0.264242</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.230158</td>\n",
       "      <td>2.069459</td>\n",
       "      <td>0.286865</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.053960</td>\n",
       "      <td>2.021962</td>\n",
       "      <td>0.317464</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.944532</td>\n",
       "      <td>1.995772</td>\n",
       "      <td>0.311035</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.868072</td>\n",
       "      <td>1.981654</td>\n",
       "      <td>0.340902</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.817046</td>\n",
       "      <td>1.955131</td>\n",
       "      <td>0.343750</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.781820</td>\n",
       "      <td>1.948407</td>\n",
       "      <td>0.359538</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.756418</td>\n",
       "      <td>1.939986</td>\n",
       "      <td>0.360189</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.734826</td>\n",
       "      <td>1.937398</td>\n",
       "      <td>0.363525</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.725647</td>\n",
       "      <td>1.937173</td>\n",
       "      <td>0.364502</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.720556</td>\n",
       "      <td>1.936023</td>\n",
       "      <td>0.364827</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv, run number: 2 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.800936</td>\n",
       "      <td>3.592150</td>\n",
       "      <td>0.045898</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.405033</td>\n",
       "      <td>3.037939</td>\n",
       "      <td>0.195801</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.075040</td>\n",
       "      <td>2.861137</td>\n",
       "      <td>0.181234</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.872208</td>\n",
       "      <td>2.777240</td>\n",
       "      <td>0.171305</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.615695</td>\n",
       "      <td>2.481252</td>\n",
       "      <td>0.197266</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.319309</td>\n",
       "      <td>2.292773</td>\n",
       "      <td>0.209391</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.121572</td>\n",
       "      <td>2.183578</td>\n",
       "      <td>0.228027</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.000262</td>\n",
       "      <td>2.135737</td>\n",
       "      <td>0.237061</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.923005</td>\n",
       "      <td>2.115013</td>\n",
       "      <td>0.262533</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.871248</td>\n",
       "      <td>2.069990</td>\n",
       "      <td>0.278402</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.833429</td>\n",
       "      <td>2.041928</td>\n",
       "      <td>0.305420</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.803798</td>\n",
       "      <td>2.024007</td>\n",
       "      <td>0.313639</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.786998</td>\n",
       "      <td>2.012340</td>\n",
       "      <td>0.322917</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.775007</td>\n",
       "      <td>2.009348</td>\n",
       "      <td>0.322917</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.771546</td>\n",
       "      <td>2.005721</td>\n",
       "      <td>0.323486</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv, run number: 3 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.805371</td>\n",
       "      <td>3.791160</td>\n",
       "      <td>0.075928</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.422752</td>\n",
       "      <td>3.183433</td>\n",
       "      <td>0.130697</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.072796</td>\n",
       "      <td>2.897424</td>\n",
       "      <td>0.160156</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.845762</td>\n",
       "      <td>2.633075</td>\n",
       "      <td>0.179362</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.515448</td>\n",
       "      <td>2.140192</td>\n",
       "      <td>0.230062</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.221329</td>\n",
       "      <td>1.997985</td>\n",
       "      <td>0.290771</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.032376</td>\n",
       "      <td>1.912972</td>\n",
       "      <td>0.330322</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.916931</td>\n",
       "      <td>1.894463</td>\n",
       "      <td>0.362223</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.836707</td>\n",
       "      <td>1.859875</td>\n",
       "      <td>0.379232</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.783240</td>\n",
       "      <td>1.831758</td>\n",
       "      <td>0.388184</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.746762</td>\n",
       "      <td>1.822118</td>\n",
       "      <td>0.396159</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.723034</td>\n",
       "      <td>1.810948</td>\n",
       "      <td>0.405436</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.706451</td>\n",
       "      <td>1.810014</td>\n",
       "      <td>0.406413</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.697441</td>\n",
       "      <td>1.807655</td>\n",
       "      <td>0.408366</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.689683</td>\n",
       "      <td>1.808923</td>\n",
       "      <td>0.407471</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv, run number: 4 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.730314</td>\n",
       "      <td>3.516700</td>\n",
       "      <td>0.092285</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.349332</td>\n",
       "      <td>2.993265</td>\n",
       "      <td>0.166585</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.015526</td>\n",
       "      <td>2.792944</td>\n",
       "      <td>0.172038</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.761664</td>\n",
       "      <td>2.522188</td>\n",
       "      <td>0.196615</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.434139</td>\n",
       "      <td>2.207585</td>\n",
       "      <td>0.234701</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.182510</td>\n",
       "      <td>2.062737</td>\n",
       "      <td>0.238200</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.036076</td>\n",
       "      <td>2.004856</td>\n",
       "      <td>0.253011</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.943094</td>\n",
       "      <td>2.003203</td>\n",
       "      <td>0.271729</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.884936</td>\n",
       "      <td>1.986014</td>\n",
       "      <td>0.300374</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.840868</td>\n",
       "      <td>1.972298</td>\n",
       "      <td>0.307861</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.806354</td>\n",
       "      <td>1.950923</td>\n",
       "      <td>0.320638</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.778509</td>\n",
       "      <td>1.946187</td>\n",
       "      <td>0.332764</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.765622</td>\n",
       "      <td>1.936598</td>\n",
       "      <td>0.334798</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.756892</td>\n",
       "      <td>1.934471</td>\n",
       "      <td>0.336670</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.752372</td>\n",
       "      <td>1.932438</td>\n",
       "      <td>0.337402</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: qkv, run number: 0 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.983528</td>\n",
       "      <td>3.730043</td>\n",
       "      <td>0.006266</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.472521</td>\n",
       "      <td>3.054789</td>\n",
       "      <td>0.138346</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.084556</td>\n",
       "      <td>2.877667</td>\n",
       "      <td>0.151367</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.891173</td>\n",
       "      <td>2.826400</td>\n",
       "      <td>0.163167</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.767120</td>\n",
       "      <td>2.727380</td>\n",
       "      <td>0.174723</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.552186</td>\n",
       "      <td>2.367991</td>\n",
       "      <td>0.248535</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.278265</td>\n",
       "      <td>2.173682</td>\n",
       "      <td>0.267741</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.081423</td>\n",
       "      <td>2.076395</td>\n",
       "      <td>0.290202</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.952221</td>\n",
       "      <td>2.029893</td>\n",
       "      <td>0.311035</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.866005</td>\n",
       "      <td>2.002422</td>\n",
       "      <td>0.336507</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.811468</td>\n",
       "      <td>1.974432</td>\n",
       "      <td>0.355632</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.777894</td>\n",
       "      <td>1.960111</td>\n",
       "      <td>0.366618</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.751652</td>\n",
       "      <td>1.955594</td>\n",
       "      <td>0.374512</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.740806</td>\n",
       "      <td>1.949867</td>\n",
       "      <td>0.378906</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.735057</td>\n",
       "      <td>1.947703</td>\n",
       "      <td>0.379639</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: qkv, run number: 1 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.800070</td>\n",
       "      <td>3.578831</td>\n",
       "      <td>0.042643</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.458818</td>\n",
       "      <td>3.027496</td>\n",
       "      <td>0.146159</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.114544</td>\n",
       "      <td>2.847957</td>\n",
       "      <td>0.160970</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.890756</td>\n",
       "      <td>2.737275</td>\n",
       "      <td>0.169840</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.633768</td>\n",
       "      <td>2.391743</td>\n",
       "      <td>0.225667</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.301673</td>\n",
       "      <td>2.101590</td>\n",
       "      <td>0.268717</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.070214</td>\n",
       "      <td>1.999889</td>\n",
       "      <td>0.318929</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.924124</td>\n",
       "      <td>1.959800</td>\n",
       "      <td>0.367513</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.826172</td>\n",
       "      <td>1.881063</td>\n",
       "      <td>0.413167</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.753621</td>\n",
       "      <td>1.866724</td>\n",
       "      <td>0.426351</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.706315</td>\n",
       "      <td>1.812531</td>\n",
       "      <td>0.446615</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.672002</td>\n",
       "      <td>1.792034</td>\n",
       "      <td>0.455485</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.652769</td>\n",
       "      <td>1.780574</td>\n",
       "      <td>0.460124</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.641860</td>\n",
       "      <td>1.774317</td>\n",
       "      <td>0.462646</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.638443</td>\n",
       "      <td>1.776227</td>\n",
       "      <td>0.462972</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: qkv, run number: 2 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.914317</td>\n",
       "      <td>3.712070</td>\n",
       "      <td>0.019368</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.474833</td>\n",
       "      <td>3.060810</td>\n",
       "      <td>0.162028</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.120632</td>\n",
       "      <td>2.852863</td>\n",
       "      <td>0.173177</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.904432</td>\n",
       "      <td>2.744458</td>\n",
       "      <td>0.191569</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.656404</td>\n",
       "      <td>2.350033</td>\n",
       "      <td>0.263346</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.340041</td>\n",
       "      <td>2.132334</td>\n",
       "      <td>0.303304</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.106686</td>\n",
       "      <td>2.007703</td>\n",
       "      <td>0.330485</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.961277</td>\n",
       "      <td>1.959396</td>\n",
       "      <td>0.348226</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.864275</td>\n",
       "      <td>1.907053</td>\n",
       "      <td>0.366536</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.796753</td>\n",
       "      <td>1.888129</td>\n",
       "      <td>0.373617</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.747746</td>\n",
       "      <td>1.889408</td>\n",
       "      <td>0.383708</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.714013</td>\n",
       "      <td>1.872278</td>\n",
       "      <td>0.385986</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.696109</td>\n",
       "      <td>1.869971</td>\n",
       "      <td>0.390625</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.685127</td>\n",
       "      <td>1.865558</td>\n",
       "      <td>0.392415</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.678375</td>\n",
       "      <td>1.866159</td>\n",
       "      <td>0.393392</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: qkv, run number: 3 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.824178</td>\n",
       "      <td>3.614142</td>\n",
       "      <td>0.081462</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.428209</td>\n",
       "      <td>2.973805</td>\n",
       "      <td>0.128092</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.069601</td>\n",
       "      <td>2.856808</td>\n",
       "      <td>0.160156</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.886776</td>\n",
       "      <td>2.799834</td>\n",
       "      <td>0.174886</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.738454</td>\n",
       "      <td>2.591961</td>\n",
       "      <td>0.201009</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.441010</td>\n",
       "      <td>2.238321</td>\n",
       "      <td>0.210449</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.185280</td>\n",
       "      <td>2.106575</td>\n",
       "      <td>0.257080</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.015402</td>\n",
       "      <td>2.055804</td>\n",
       "      <td>0.274658</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.904782</td>\n",
       "      <td>2.041633</td>\n",
       "      <td>0.307373</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.832205</td>\n",
       "      <td>2.015671</td>\n",
       "      <td>0.321696</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.780776</td>\n",
       "      <td>2.002613</td>\n",
       "      <td>0.328532</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.745169</td>\n",
       "      <td>2.001560</td>\n",
       "      <td>0.329753</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.722490</td>\n",
       "      <td>1.995259</td>\n",
       "      <td>0.335124</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.709921</td>\n",
       "      <td>1.992601</td>\n",
       "      <td>0.335856</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.706388</td>\n",
       "      <td>1.991550</td>\n",
       "      <td>0.334798</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: qkv, run number: 4 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.895893</td>\n",
       "      <td>3.653290</td>\n",
       "      <td>0.019368</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.426116</td>\n",
       "      <td>3.048022</td>\n",
       "      <td>0.149089</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.082704</td>\n",
       "      <td>2.878556</td>\n",
       "      <td>0.177897</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.903945</td>\n",
       "      <td>2.806743</td>\n",
       "      <td>0.195882</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.785312</td>\n",
       "      <td>2.701115</td>\n",
       "      <td>0.208008</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.584977</td>\n",
       "      <td>2.413975</td>\n",
       "      <td>0.237386</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.292434</td>\n",
       "      <td>2.183657</td>\n",
       "      <td>0.246745</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.080281</td>\n",
       "      <td>2.065049</td>\n",
       "      <td>0.287760</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.944641</td>\n",
       "      <td>2.021415</td>\n",
       "      <td>0.300537</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.855661</td>\n",
       "      <td>1.995325</td>\n",
       "      <td>0.321777</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.798240</td>\n",
       "      <td>1.974666</td>\n",
       "      <td>0.332520</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.762142</td>\n",
       "      <td>1.960406</td>\n",
       "      <td>0.346191</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.740542</td>\n",
       "      <td>1.956783</td>\n",
       "      <td>0.352295</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.726170</td>\n",
       "      <td>1.951063</td>\n",
       "      <td>0.354004</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.724927</td>\n",
       "      <td>1.951092</td>\n",
       "      <td>0.352783</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv, run number: 0 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.810269</td>\n",
       "      <td>3.734236</td>\n",
       "      <td>0.010661</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.584748</td>\n",
       "      <td>3.216152</td>\n",
       "      <td>0.093587</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.303396</td>\n",
       "      <td>2.973103</td>\n",
       "      <td>0.127930</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.093809</td>\n",
       "      <td>2.866926</td>\n",
       "      <td>0.129883</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.940014</td>\n",
       "      <td>2.752566</td>\n",
       "      <td>0.124674</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.776876</td>\n",
       "      <td>2.563743</td>\n",
       "      <td>0.145020</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.588048</td>\n",
       "      <td>2.360862</td>\n",
       "      <td>0.182780</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.420425</td>\n",
       "      <td>2.219026</td>\n",
       "      <td>0.199056</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.287214</td>\n",
       "      <td>2.152068</td>\n",
       "      <td>0.203288</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.187955</td>\n",
       "      <td>2.105972</td>\n",
       "      <td>0.206624</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.114762</td>\n",
       "      <td>2.089404</td>\n",
       "      <td>0.212565</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.065384</td>\n",
       "      <td>2.088450</td>\n",
       "      <td>0.221517</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.030941</td>\n",
       "      <td>2.078882</td>\n",
       "      <td>0.216227</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.008901</td>\n",
       "      <td>2.077996</td>\n",
       "      <td>0.218180</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.995393</td>\n",
       "      <td>2.078663</td>\n",
       "      <td>0.217285</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv, run number: 1 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.984528</td>\n",
       "      <td>3.728209</td>\n",
       "      <td>0.025228</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.762385</td>\n",
       "      <td>3.271311</td>\n",
       "      <td>0.094482</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.446060</td>\n",
       "      <td>2.989980</td>\n",
       "      <td>0.141846</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.205099</td>\n",
       "      <td>2.912400</td>\n",
       "      <td>0.138672</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.046065</td>\n",
       "      <td>2.879444</td>\n",
       "      <td>0.134684</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.937492</td>\n",
       "      <td>2.855197</td>\n",
       "      <td>0.134115</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.854042</td>\n",
       "      <td>2.823140</td>\n",
       "      <td>0.133138</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.771837</td>\n",
       "      <td>2.778483</td>\n",
       "      <td>0.122803</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.677628</td>\n",
       "      <td>2.728649</td>\n",
       "      <td>0.140625</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.579418</td>\n",
       "      <td>2.665212</td>\n",
       "      <td>0.156982</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.488751</td>\n",
       "      <td>2.611908</td>\n",
       "      <td>0.176758</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.412603</td>\n",
       "      <td>2.578848</td>\n",
       "      <td>0.187581</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.355131</td>\n",
       "      <td>2.556907</td>\n",
       "      <td>0.190674</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.316350</td>\n",
       "      <td>2.549632</td>\n",
       "      <td>0.197266</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.290185</td>\n",
       "      <td>2.552867</td>\n",
       "      <td>0.198324</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv, run number: 2 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.745343</td>\n",
       "      <td>3.607341</td>\n",
       "      <td>0.028646</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.563595</td>\n",
       "      <td>3.121385</td>\n",
       "      <td>0.123291</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.307624</td>\n",
       "      <td>2.876912</td>\n",
       "      <td>0.155680</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.111077</td>\n",
       "      <td>2.838729</td>\n",
       "      <td>0.146403</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.982094</td>\n",
       "      <td>2.817988</td>\n",
       "      <td>0.131755</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.893403</td>\n",
       "      <td>2.787728</td>\n",
       "      <td>0.137695</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.822708</td>\n",
       "      <td>2.754901</td>\n",
       "      <td>0.147135</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.756006</td>\n",
       "      <td>2.701516</td>\n",
       "      <td>0.153320</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.684285</td>\n",
       "      <td>2.648257</td>\n",
       "      <td>0.179362</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.609851</td>\n",
       "      <td>2.597428</td>\n",
       "      <td>0.190511</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.533827</td>\n",
       "      <td>2.560360</td>\n",
       "      <td>0.200602</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.467034</td>\n",
       "      <td>2.528384</td>\n",
       "      <td>0.212728</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.415837</td>\n",
       "      <td>2.511587</td>\n",
       "      <td>0.220378</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.378226</td>\n",
       "      <td>2.504466</td>\n",
       "      <td>0.225830</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.353044</td>\n",
       "      <td>2.499109</td>\n",
       "      <td>0.226644</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv, run number: 3 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.965988</td>\n",
       "      <td>3.984542</td>\n",
       "      <td>0.016602</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.769512</td>\n",
       "      <td>3.486216</td>\n",
       "      <td>0.052734</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.455294</td>\n",
       "      <td>3.076285</td>\n",
       "      <td>0.123047</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.195735</td>\n",
       "      <td>2.929309</td>\n",
       "      <td>0.120443</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.017612</td>\n",
       "      <td>2.877846</td>\n",
       "      <td>0.127035</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.876677</td>\n",
       "      <td>2.811169</td>\n",
       "      <td>0.115641</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.727585</td>\n",
       "      <td>2.638794</td>\n",
       "      <td>0.143311</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.560447</td>\n",
       "      <td>2.457121</td>\n",
       "      <td>0.159342</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.408159</td>\n",
       "      <td>2.340617</td>\n",
       "      <td>0.158691</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.286724</td>\n",
       "      <td>2.295203</td>\n",
       "      <td>0.169271</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.197711</td>\n",
       "      <td>2.271691</td>\n",
       "      <td>0.177490</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.132998</td>\n",
       "      <td>2.262565</td>\n",
       "      <td>0.177165</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.089068</td>\n",
       "      <td>2.251539</td>\n",
       "      <td>0.181803</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.059522</td>\n",
       "      <td>2.256901</td>\n",
       "      <td>0.181559</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.041539</td>\n",
       "      <td>2.254459</td>\n",
       "      <td>0.183512</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv, run number: 4 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.880755</td>\n",
       "      <td>3.840956</td>\n",
       "      <td>0.012207</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.684680</td>\n",
       "      <td>3.319615</td>\n",
       "      <td>0.109375</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.404265</td>\n",
       "      <td>2.961782</td>\n",
       "      <td>0.142578</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.183015</td>\n",
       "      <td>2.870802</td>\n",
       "      <td>0.128174</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.035398</td>\n",
       "      <td>2.841386</td>\n",
       "      <td>0.120361</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.939298</td>\n",
       "      <td>2.832915</td>\n",
       "      <td>0.120524</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.872284</td>\n",
       "      <td>2.810717</td>\n",
       "      <td>0.125814</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.823886</td>\n",
       "      <td>2.793444</td>\n",
       "      <td>0.106934</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.783776</td>\n",
       "      <td>2.774246</td>\n",
       "      <td>0.114176</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.744970</td>\n",
       "      <td>2.748593</td>\n",
       "      <td>0.101725</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.708301</td>\n",
       "      <td>2.717497</td>\n",
       "      <td>0.100749</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.673726</td>\n",
       "      <td>2.695105</td>\n",
       "      <td>0.104004</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.645063</td>\n",
       "      <td>2.683329</td>\n",
       "      <td>0.105143</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.622101</td>\n",
       "      <td>2.676942</td>\n",
       "      <td>0.105957</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.606497</td>\n",
       "      <td>2.676456</td>\n",
       "      <td>0.107503</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: qkv, run number: 0 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.217417</td>\n",
       "      <td>3.919697</td>\n",
       "      <td>0.033285</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.970368</td>\n",
       "      <td>3.376639</td>\n",
       "      <td>0.054932</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.577105</td>\n",
       "      <td>3.003459</td>\n",
       "      <td>0.127035</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.282501</td>\n",
       "      <td>2.916244</td>\n",
       "      <td>0.126383</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.097875</td>\n",
       "      <td>2.867589</td>\n",
       "      <td>0.125326</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.979867</td>\n",
       "      <td>2.848060</td>\n",
       "      <td>0.114339</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.897547</td>\n",
       "      <td>2.831880</td>\n",
       "      <td>0.127197</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.831774</td>\n",
       "      <td>2.813371</td>\n",
       "      <td>0.123047</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.769002</td>\n",
       "      <td>2.788873</td>\n",
       "      <td>0.116292</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.697614</td>\n",
       "      <td>2.735869</td>\n",
       "      <td>0.134847</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.617998</td>\n",
       "      <td>2.699219</td>\n",
       "      <td>0.142904</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.540433</td>\n",
       "      <td>2.668200</td>\n",
       "      <td>0.155518</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.477609</td>\n",
       "      <td>2.651919</td>\n",
       "      <td>0.162191</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.431299</td>\n",
       "      <td>2.644697</td>\n",
       "      <td>0.164062</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.401236</td>\n",
       "      <td>2.642390</td>\n",
       "      <td>0.165283</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: qkv, run number: 1 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.760406</td>\n",
       "      <td>3.663076</td>\n",
       "      <td>0.012044</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.552478</td>\n",
       "      <td>3.222616</td>\n",
       "      <td>0.088704</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.296066</td>\n",
       "      <td>2.972003</td>\n",
       "      <td>0.139567</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.101163</td>\n",
       "      <td>2.879679</td>\n",
       "      <td>0.133545</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.970060</td>\n",
       "      <td>2.835473</td>\n",
       "      <td>0.130290</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.874525</td>\n",
       "      <td>2.792941</td>\n",
       "      <td>0.114909</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.787953</td>\n",
       "      <td>2.738497</td>\n",
       "      <td>0.106689</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.690541</td>\n",
       "      <td>2.642449</td>\n",
       "      <td>0.142171</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.577505</td>\n",
       "      <td>2.539279</td>\n",
       "      <td>0.165039</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.466160</td>\n",
       "      <td>2.454911</td>\n",
       "      <td>0.169352</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.371641</td>\n",
       "      <td>2.401851</td>\n",
       "      <td>0.175456</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.299568</td>\n",
       "      <td>2.385653</td>\n",
       "      <td>0.181478</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.246471</td>\n",
       "      <td>2.372357</td>\n",
       "      <td>0.183757</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.209772</td>\n",
       "      <td>2.365506</td>\n",
       "      <td>0.184733</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.184687</td>\n",
       "      <td>2.366337</td>\n",
       "      <td>0.184896</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: qkv, run number: 2 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.962212</td>\n",
       "      <td>3.685862</td>\n",
       "      <td>0.016683</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.740550</td>\n",
       "      <td>3.265891</td>\n",
       "      <td>0.091309</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.436923</td>\n",
       "      <td>2.929611</td>\n",
       "      <td>0.135824</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.186072</td>\n",
       "      <td>2.834026</td>\n",
       "      <td>0.133301</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.018351</td>\n",
       "      <td>2.797809</td>\n",
       "      <td>0.130371</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.897051</td>\n",
       "      <td>2.741009</td>\n",
       "      <td>0.121989</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.785489</td>\n",
       "      <td>2.622373</td>\n",
       "      <td>0.120768</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.658205</td>\n",
       "      <td>2.459718</td>\n",
       "      <td>0.142334</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.517354</td>\n",
       "      <td>2.322810</td>\n",
       "      <td>0.161458</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.387485</td>\n",
       "      <td>2.231669</td>\n",
       "      <td>0.175456</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.282606</td>\n",
       "      <td>2.186506</td>\n",
       "      <td>0.180583</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.205691</td>\n",
       "      <td>2.159859</td>\n",
       "      <td>0.184082</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.150734</td>\n",
       "      <td>2.147903</td>\n",
       "      <td>0.187744</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.112981</td>\n",
       "      <td>2.137654</td>\n",
       "      <td>0.187419</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.087601</td>\n",
       "      <td>2.143871</td>\n",
       "      <td>0.186035</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: qkv, run number: 3 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.700942</td>\n",
       "      <td>3.550620</td>\n",
       "      <td>0.041341</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.529192</td>\n",
       "      <td>3.171848</td>\n",
       "      <td>0.129883</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.314997</td>\n",
       "      <td>2.946774</td>\n",
       "      <td>0.127767</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.137192</td>\n",
       "      <td>2.853841</td>\n",
       "      <td>0.132324</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.010113</td>\n",
       "      <td>2.806837</td>\n",
       "      <td>0.131104</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.913192</td>\n",
       "      <td>2.759380</td>\n",
       "      <td>0.118896</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.821257</td>\n",
       "      <td>2.653938</td>\n",
       "      <td>0.127523</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.694592</td>\n",
       "      <td>2.493128</td>\n",
       "      <td>0.177490</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.536622</td>\n",
       "      <td>2.368774</td>\n",
       "      <td>0.207113</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.397646</td>\n",
       "      <td>2.280470</td>\n",
       "      <td>0.213786</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.286027</td>\n",
       "      <td>2.229349</td>\n",
       "      <td>0.233561</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.202332</td>\n",
       "      <td>2.206851</td>\n",
       "      <td>0.231771</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.145350</td>\n",
       "      <td>2.198550</td>\n",
       "      <td>0.231771</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.106767</td>\n",
       "      <td>2.192727</td>\n",
       "      <td>0.229574</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.081530</td>\n",
       "      <td>2.193223</td>\n",
       "      <td>0.229411</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: qkv, run number: 4 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.739009</td>\n",
       "      <td>3.872558</td>\n",
       "      <td>0.023356</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.560241</td>\n",
       "      <td>3.376516</td>\n",
       "      <td>0.061849</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.310072</td>\n",
       "      <td>3.016315</td>\n",
       "      <td>0.124186</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.109187</td>\n",
       "      <td>2.893941</td>\n",
       "      <td>0.143066</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.973913</td>\n",
       "      <td>2.850288</td>\n",
       "      <td>0.145426</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.876796</td>\n",
       "      <td>2.827340</td>\n",
       "      <td>0.129395</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.793561</td>\n",
       "      <td>2.819528</td>\n",
       "      <td>0.132406</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.708748</td>\n",
       "      <td>2.793622</td>\n",
       "      <td>0.127441</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.610941</td>\n",
       "      <td>2.755081</td>\n",
       "      <td>0.135091</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.511055</td>\n",
       "      <td>2.703089</td>\n",
       "      <td>0.142334</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.420800</td>\n",
       "      <td>2.640796</td>\n",
       "      <td>0.152181</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.346896</td>\n",
       "      <td>2.597601</td>\n",
       "      <td>0.161214</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.291750</td>\n",
       "      <td>2.576217</td>\n",
       "      <td>0.160645</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.252062</td>\n",
       "      <td>2.565836</td>\n",
       "      <td>0.163411</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.224819</td>\n",
       "      <td>2.569664</td>\n",
       "      <td>0.163249</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv, run number: 0 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.852681</td>\n",
       "      <td>3.803610</td>\n",
       "      <td>0.017497</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.763639</td>\n",
       "      <td>3.455470</td>\n",
       "      <td>0.063802</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.589327</td>\n",
       "      <td>3.054032</td>\n",
       "      <td>0.096436</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.402731</td>\n",
       "      <td>2.902567</td>\n",
       "      <td>0.118490</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.253665</td>\n",
       "      <td>2.850314</td>\n",
       "      <td>0.112305</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.138574</td>\n",
       "      <td>2.826208</td>\n",
       "      <td>0.125814</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.050501</td>\n",
       "      <td>2.812085</td>\n",
       "      <td>0.118001</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.981303</td>\n",
       "      <td>2.798802</td>\n",
       "      <td>0.126872</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.925574</td>\n",
       "      <td>2.784396</td>\n",
       "      <td>0.127279</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.878669</td>\n",
       "      <td>2.772762</td>\n",
       "      <td>0.118896</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.839980</td>\n",
       "      <td>2.760521</td>\n",
       "      <td>0.116943</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.807115</td>\n",
       "      <td>2.749983</td>\n",
       "      <td>0.113037</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.779764</td>\n",
       "      <td>2.748399</td>\n",
       "      <td>0.112874</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.757448</td>\n",
       "      <td>2.744497</td>\n",
       "      <td>0.114909</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.740333</td>\n",
       "      <td>2.744803</td>\n",
       "      <td>0.113444</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv, run number: 1 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.782003</td>\n",
       "      <td>3.752938</td>\n",
       "      <td>0.006917</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.673055</td>\n",
       "      <td>3.413193</td>\n",
       "      <td>0.042806</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.503610</td>\n",
       "      <td>3.094693</td>\n",
       "      <td>0.127523</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.338161</td>\n",
       "      <td>2.937732</td>\n",
       "      <td>0.127279</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.204606</td>\n",
       "      <td>2.866712</td>\n",
       "      <td>0.130778</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.101532</td>\n",
       "      <td>2.827581</td>\n",
       "      <td>0.134847</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.021000</td>\n",
       "      <td>2.802485</td>\n",
       "      <td>0.127930</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.952673</td>\n",
       "      <td>2.777577</td>\n",
       "      <td>0.135986</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.891199</td>\n",
       "      <td>2.755721</td>\n",
       "      <td>0.132568</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.831864</td>\n",
       "      <td>2.720687</td>\n",
       "      <td>0.132324</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.773054</td>\n",
       "      <td>2.691732</td>\n",
       "      <td>0.146240</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.718019</td>\n",
       "      <td>2.671092</td>\n",
       "      <td>0.156494</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.668390</td>\n",
       "      <td>2.659008</td>\n",
       "      <td>0.158040</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.626396</td>\n",
       "      <td>2.656447</td>\n",
       "      <td>0.161377</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.592477</td>\n",
       "      <td>2.654578</td>\n",
       "      <td>0.161458</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv, run number: 2 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.119989</td>\n",
       "      <td>4.159524</td>\n",
       "      <td>0.018392</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.991966</td>\n",
       "      <td>3.706879</td>\n",
       "      <td>0.022868</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.757097</td>\n",
       "      <td>3.186787</td>\n",
       "      <td>0.111165</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.525809</td>\n",
       "      <td>2.988249</td>\n",
       "      <td>0.130046</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.346243</td>\n",
       "      <td>2.907305</td>\n",
       "      <td>0.132487</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.212708</td>\n",
       "      <td>2.860591</td>\n",
       "      <td>0.130371</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.112031</td>\n",
       "      <td>2.827976</td>\n",
       "      <td>0.132324</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.033800</td>\n",
       "      <td>2.804772</td>\n",
       "      <td>0.129476</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.972951</td>\n",
       "      <td>2.788322</td>\n",
       "      <td>0.124430</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.923354</td>\n",
       "      <td>2.774572</td>\n",
       "      <td>0.125895</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.881844</td>\n",
       "      <td>2.759404</td>\n",
       "      <td>0.124512</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.846579</td>\n",
       "      <td>2.751206</td>\n",
       "      <td>0.124186</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.817475</td>\n",
       "      <td>2.744676</td>\n",
       "      <td>0.122803</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.794602</td>\n",
       "      <td>2.743056</td>\n",
       "      <td>0.121989</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.775946</td>\n",
       "      <td>2.741575</td>\n",
       "      <td>0.121663</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv, run number: 3 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.195218</td>\n",
       "      <td>4.271805</td>\n",
       "      <td>0.011149</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4.100415</td>\n",
       "      <td>3.919659</td>\n",
       "      <td>0.016683</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.899645</td>\n",
       "      <td>3.390743</td>\n",
       "      <td>0.066813</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.667909</td>\n",
       "      <td>3.110270</td>\n",
       "      <td>0.132894</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.476866</td>\n",
       "      <td>3.009045</td>\n",
       "      <td>0.142497</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.328533</td>\n",
       "      <td>2.947477</td>\n",
       "      <td>0.141113</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.211152</td>\n",
       "      <td>2.912014</td>\n",
       "      <td>0.138021</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.119597</td>\n",
       "      <td>2.880876</td>\n",
       "      <td>0.137370</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.045233</td>\n",
       "      <td>2.860630</td>\n",
       "      <td>0.137126</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.983793</td>\n",
       "      <td>2.836925</td>\n",
       "      <td>0.131429</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.932370</td>\n",
       "      <td>2.820383</td>\n",
       "      <td>0.130290</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.889774</td>\n",
       "      <td>2.808973</td>\n",
       "      <td>0.126546</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.854914</td>\n",
       "      <td>2.801194</td>\n",
       "      <td>0.126790</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.825733</td>\n",
       "      <td>2.797756</td>\n",
       "      <td>0.125895</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.803543</td>\n",
       "      <td>2.796906</td>\n",
       "      <td>0.126628</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv, run number: 4 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.020916</td>\n",
       "      <td>3.716302</td>\n",
       "      <td>0.097249</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.909830</td>\n",
       "      <td>3.401957</td>\n",
       "      <td>0.074788</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.695959</td>\n",
       "      <td>3.042683</td>\n",
       "      <td>0.102783</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.476337</td>\n",
       "      <td>2.911928</td>\n",
       "      <td>0.142660</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.307863</td>\n",
       "      <td>2.839160</td>\n",
       "      <td>0.127441</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.180383</td>\n",
       "      <td>2.806583</td>\n",
       "      <td>0.139242</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.081534</td>\n",
       "      <td>2.780782</td>\n",
       "      <td>0.138672</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.001318</td>\n",
       "      <td>2.758491</td>\n",
       "      <td>0.143555</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.933442</td>\n",
       "      <td>2.729105</td>\n",
       "      <td>0.140462</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.873280</td>\n",
       "      <td>2.705006</td>\n",
       "      <td>0.141032</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.818486</td>\n",
       "      <td>2.681062</td>\n",
       "      <td>0.144775</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.769631</td>\n",
       "      <td>2.659479</td>\n",
       "      <td>0.142578</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.727437</td>\n",
       "      <td>2.649730</td>\n",
       "      <td>0.138753</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.692334</td>\n",
       "      <td>2.646668</td>\n",
       "      <td>0.141520</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.664065</td>\n",
       "      <td>2.644886</td>\n",
       "      <td>0.141113</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: qkv, run number: 0 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.029977</td>\n",
       "      <td>3.940488</td>\n",
       "      <td>0.011800</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.916085</td>\n",
       "      <td>3.584047</td>\n",
       "      <td>0.015788</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.704937</td>\n",
       "      <td>3.178875</td>\n",
       "      <td>0.065837</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.490649</td>\n",
       "      <td>2.992399</td>\n",
       "      <td>0.126302</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.322660</td>\n",
       "      <td>2.922877</td>\n",
       "      <td>0.127604</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.197926</td>\n",
       "      <td>2.888736</td>\n",
       "      <td>0.131755</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.103554</td>\n",
       "      <td>2.868154</td>\n",
       "      <td>0.130859</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.031625</td>\n",
       "      <td>2.859206</td>\n",
       "      <td>0.132975</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.975441</td>\n",
       "      <td>2.849168</td>\n",
       "      <td>0.130859</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.931418</td>\n",
       "      <td>2.844845</td>\n",
       "      <td>0.132487</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.895803</td>\n",
       "      <td>2.840667</td>\n",
       "      <td>0.133952</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.867830</td>\n",
       "      <td>2.837476</td>\n",
       "      <td>0.133626</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.845601</td>\n",
       "      <td>2.838989</td>\n",
       "      <td>0.131348</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.827577</td>\n",
       "      <td>2.835502</td>\n",
       "      <td>0.132080</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.814441</td>\n",
       "      <td>2.835456</td>\n",
       "      <td>0.131592</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: qkv, run number: 1 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.147433</td>\n",
       "      <td>3.940940</td>\n",
       "      <td>0.041911</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4.019802</td>\n",
       "      <td>3.551355</td>\n",
       "      <td>0.096761</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.782294</td>\n",
       "      <td>3.127396</td>\n",
       "      <td>0.143311</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.541497</td>\n",
       "      <td>2.922445</td>\n",
       "      <td>0.150960</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.357352</td>\n",
       "      <td>2.867228</td>\n",
       "      <td>0.146647</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.220800</td>\n",
       "      <td>2.842339</td>\n",
       "      <td>0.143066</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.118934</td>\n",
       "      <td>2.824513</td>\n",
       "      <td>0.138184</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.038033</td>\n",
       "      <td>2.808567</td>\n",
       "      <td>0.137858</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.974950</td>\n",
       "      <td>2.799019</td>\n",
       "      <td>0.135417</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.923671</td>\n",
       "      <td>2.785138</td>\n",
       "      <td>0.137939</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.881884</td>\n",
       "      <td>2.776775</td>\n",
       "      <td>0.136963</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.847040</td>\n",
       "      <td>2.769675</td>\n",
       "      <td>0.137451</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.818645</td>\n",
       "      <td>2.766265</td>\n",
       "      <td>0.137370</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.796660</td>\n",
       "      <td>2.763146</td>\n",
       "      <td>0.137939</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.779410</td>\n",
       "      <td>2.763057</td>\n",
       "      <td>0.137533</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: qkv, run number: 2 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.921959</td>\n",
       "      <td>3.999055</td>\n",
       "      <td>0.007406</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.802258</td>\n",
       "      <td>3.626639</td>\n",
       "      <td>0.023600</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.602234</td>\n",
       "      <td>3.250455</td>\n",
       "      <td>0.122559</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.407818</td>\n",
       "      <td>3.061406</td>\n",
       "      <td>0.166260</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.259551</td>\n",
       "      <td>2.965779</td>\n",
       "      <td>0.159749</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.147995</td>\n",
       "      <td>2.912041</td>\n",
       "      <td>0.155762</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.061290</td>\n",
       "      <td>2.873391</td>\n",
       "      <td>0.144450</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.992460</td>\n",
       "      <td>2.849613</td>\n",
       "      <td>0.142822</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.937087</td>\n",
       "      <td>2.826633</td>\n",
       "      <td>0.136882</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.889939</td>\n",
       "      <td>2.810385</td>\n",
       "      <td>0.131185</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.849852</td>\n",
       "      <td>2.792695</td>\n",
       "      <td>0.131429</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.816581</td>\n",
       "      <td>2.784283</td>\n",
       "      <td>0.128418</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.789059</td>\n",
       "      <td>2.777920</td>\n",
       "      <td>0.128011</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.766397</td>\n",
       "      <td>2.776349</td>\n",
       "      <td>0.124674</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.748534</td>\n",
       "      <td>2.775003</td>\n",
       "      <td>0.124837</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: qkv, run number: 3 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.991369</td>\n",
       "      <td>3.838861</td>\n",
       "      <td>0.019287</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.885571</td>\n",
       "      <td>3.531678</td>\n",
       "      <td>0.051432</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.698266</td>\n",
       "      <td>3.164856</td>\n",
       "      <td>0.145996</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.488981</td>\n",
       "      <td>2.981755</td>\n",
       "      <td>0.156006</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.321503</td>\n",
       "      <td>2.913943</td>\n",
       "      <td>0.164388</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.194088</td>\n",
       "      <td>2.872408</td>\n",
       "      <td>0.156494</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.096774</td>\n",
       "      <td>2.844241</td>\n",
       "      <td>0.152995</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.020845</td>\n",
       "      <td>2.820210</td>\n",
       "      <td>0.149089</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.959316</td>\n",
       "      <td>2.798866</td>\n",
       "      <td>0.148763</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.908960</td>\n",
       "      <td>2.784643</td>\n",
       "      <td>0.148763</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.866576</td>\n",
       "      <td>2.774467</td>\n",
       "      <td>0.150391</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.832157</td>\n",
       "      <td>2.765511</td>\n",
       "      <td>0.146403</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.803713</td>\n",
       "      <td>2.759660</td>\n",
       "      <td>0.145426</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.780521</td>\n",
       "      <td>2.755124</td>\n",
       "      <td>0.147054</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.762532</td>\n",
       "      <td>2.755749</td>\n",
       "      <td>0.146810</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: qkv, run number: 4 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.935236</td>\n",
       "      <td>3.755149</td>\n",
       "      <td>0.029053</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.818629</td>\n",
       "      <td>3.347004</td>\n",
       "      <td>0.070068</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.618211</td>\n",
       "      <td>2.998031</td>\n",
       "      <td>0.125081</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.433629</td>\n",
       "      <td>2.918816</td>\n",
       "      <td>0.143555</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.288664</td>\n",
       "      <td>2.872820</td>\n",
       "      <td>0.142985</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.176811</td>\n",
       "      <td>2.847718</td>\n",
       "      <td>0.143880</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.090097</td>\n",
       "      <td>2.820101</td>\n",
       "      <td>0.145833</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.020623</td>\n",
       "      <td>2.806956</td>\n",
       "      <td>0.144043</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.963481</td>\n",
       "      <td>2.793230</td>\n",
       "      <td>0.143962</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.916457</td>\n",
       "      <td>2.783855</td>\n",
       "      <td>0.143555</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.877833</td>\n",
       "      <td>2.773443</td>\n",
       "      <td>0.144206</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.844596</td>\n",
       "      <td>2.765182</td>\n",
       "      <td>0.143555</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.817419</td>\n",
       "      <td>2.761025</td>\n",
       "      <td>0.144531</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.795714</td>\n",
       "      <td>2.758921</td>\n",
       "      <td>0.144450</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.778091</td>\n",
       "      <td>2.759239</td>\n",
       "      <td>0.145426</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv, run number: 0 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.149708</td>\n",
       "      <td>4.092676</td>\n",
       "      <td>0.009888</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4.086964</td>\n",
       "      <td>3.850610</td>\n",
       "      <td>0.009766</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.958410</td>\n",
       "      <td>3.506741</td>\n",
       "      <td>0.014893</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.794441</td>\n",
       "      <td>3.243526</td>\n",
       "      <td>0.080078</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.640262</td>\n",
       "      <td>3.109349</td>\n",
       "      <td>0.107788</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.512066</td>\n",
       "      <td>3.024459</td>\n",
       "      <td>0.126221</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.407069</td>\n",
       "      <td>2.973682</td>\n",
       "      <td>0.129272</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.320776</td>\n",
       "      <td>2.962277</td>\n",
       "      <td>0.124512</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.250237</td>\n",
       "      <td>2.943954</td>\n",
       "      <td>0.130249</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>3.191982</td>\n",
       "      <td>2.914243</td>\n",
       "      <td>0.128662</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>3.142815</td>\n",
       "      <td>2.914509</td>\n",
       "      <td>0.130859</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>3.101278</td>\n",
       "      <td>2.904391</td>\n",
       "      <td>0.131104</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>3.066841</td>\n",
       "      <td>2.910216</td>\n",
       "      <td>0.130615</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>3.037988</td>\n",
       "      <td>2.901922</td>\n",
       "      <td>0.129517</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>3.013702</td>\n",
       "      <td>2.899956</td>\n",
       "      <td>0.128906</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv, run number: 1 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.179647</td>\n",
       "      <td>4.208437</td>\n",
       "      <td>0.006348</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4.127063</td>\n",
       "      <td>4.014623</td>\n",
       "      <td>0.003296</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>4.018341</td>\n",
       "      <td>3.580541</td>\n",
       "      <td>0.030884</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.854714</td>\n",
       "      <td>3.181556</td>\n",
       "      <td>0.136108</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.688406</td>\n",
       "      <td>2.971349</td>\n",
       "      <td>0.144897</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.545444</td>\n",
       "      <td>2.868132</td>\n",
       "      <td>0.145752</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.428979</td>\n",
       "      <td>2.821742</td>\n",
       "      <td>0.150879</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.334018</td>\n",
       "      <td>2.797425</td>\n",
       "      <td>0.152588</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.256564</td>\n",
       "      <td>2.771930</td>\n",
       "      <td>0.150269</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>3.191433</td>\n",
       "      <td>2.764213</td>\n",
       "      <td>0.153076</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>3.137000</td>\n",
       "      <td>2.752953</td>\n",
       "      <td>0.144897</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>3.091043</td>\n",
       "      <td>2.750866</td>\n",
       "      <td>0.147461</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>3.051907</td>\n",
       "      <td>2.758791</td>\n",
       "      <td>0.148682</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>3.019778</td>\n",
       "      <td>2.749805</td>\n",
       "      <td>0.149658</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.992314</td>\n",
       "      <td>2.757463</td>\n",
       "      <td>0.146484</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv, run number: 2 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.152741</td>\n",
       "      <td>4.173870</td>\n",
       "      <td>0.010376</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4.112856</td>\n",
       "      <td>3.982709</td>\n",
       "      <td>0.011719</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>4.019513</td>\n",
       "      <td>3.648374</td>\n",
       "      <td>0.010254</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.878090</td>\n",
       "      <td>3.335905</td>\n",
       "      <td>0.049927</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.724534</td>\n",
       "      <td>3.155522</td>\n",
       "      <td>0.099487</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.587215</td>\n",
       "      <td>3.072929</td>\n",
       "      <td>0.122070</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.471417</td>\n",
       "      <td>2.999466</td>\n",
       "      <td>0.114624</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.375018</td>\n",
       "      <td>2.983562</td>\n",
       "      <td>0.111694</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.295118</td>\n",
       "      <td>2.938692</td>\n",
       "      <td>0.110107</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>3.229065</td>\n",
       "      <td>2.923923</td>\n",
       "      <td>0.112671</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>3.173625</td>\n",
       "      <td>2.911652</td>\n",
       "      <td>0.115479</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>3.126658</td>\n",
       "      <td>2.896686</td>\n",
       "      <td>0.118408</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>3.087798</td>\n",
       "      <td>2.896894</td>\n",
       "      <td>0.116211</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>3.054574</td>\n",
       "      <td>2.899699</td>\n",
       "      <td>0.120972</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>3.026970</td>\n",
       "      <td>2.894279</td>\n",
       "      <td>0.116089</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv, run number: 3 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.912407</td>\n",
       "      <td>3.839589</td>\n",
       "      <td>0.021729</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.853030</td>\n",
       "      <td>3.651201</td>\n",
       "      <td>0.018677</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.745261</td>\n",
       "      <td>3.366850</td>\n",
       "      <td>0.046509</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.610233</td>\n",
       "      <td>3.076569</td>\n",
       "      <td>0.116821</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.483793</td>\n",
       "      <td>2.974989</td>\n",
       "      <td>0.166382</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.378045</td>\n",
       "      <td>2.906971</td>\n",
       "      <td>0.162476</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.291733</td>\n",
       "      <td>2.884224</td>\n",
       "      <td>0.154541</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.221856</td>\n",
       "      <td>2.863935</td>\n",
       "      <td>0.152954</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.164289</td>\n",
       "      <td>2.854475</td>\n",
       "      <td>0.142578</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>3.116170</td>\n",
       "      <td>2.839276</td>\n",
       "      <td>0.142944</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>3.076059</td>\n",
       "      <td>2.836715</td>\n",
       "      <td>0.144897</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>3.042470</td>\n",
       "      <td>2.834708</td>\n",
       "      <td>0.138184</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>3.013755</td>\n",
       "      <td>2.831995</td>\n",
       "      <td>0.145264</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.989485</td>\n",
       "      <td>2.834831</td>\n",
       "      <td>0.142334</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.969034</td>\n",
       "      <td>2.836657</td>\n",
       "      <td>0.140625</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv, run number: 4 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.941447</td>\n",
       "      <td>3.871279</td>\n",
       "      <td>0.014160</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.900923</td>\n",
       "      <td>3.726888</td>\n",
       "      <td>0.023071</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.815817</td>\n",
       "      <td>3.425137</td>\n",
       "      <td>0.085449</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.694163</td>\n",
       "      <td>3.207912</td>\n",
       "      <td>0.158813</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.568534</td>\n",
       "      <td>3.071138</td>\n",
       "      <td>0.175171</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.457685</td>\n",
       "      <td>2.989364</td>\n",
       "      <td>0.177856</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.364174</td>\n",
       "      <td>2.957105</td>\n",
       "      <td>0.189697</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.284494</td>\n",
       "      <td>2.916910</td>\n",
       "      <td>0.182129</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.216493</td>\n",
       "      <td>2.893637</td>\n",
       "      <td>0.180908</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>3.159405</td>\n",
       "      <td>2.884601</td>\n",
       "      <td>0.165283</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>3.111225</td>\n",
       "      <td>2.868627</td>\n",
       "      <td>0.172241</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>3.069743</td>\n",
       "      <td>2.868533</td>\n",
       "      <td>0.169189</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>3.034752</td>\n",
       "      <td>2.857297</td>\n",
       "      <td>0.168091</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>3.004983</td>\n",
       "      <td>2.854383</td>\n",
       "      <td>0.171387</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.980475</td>\n",
       "      <td>2.854995</td>\n",
       "      <td>0.179321</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: qkv, run number: 0 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.740609</td>\n",
       "      <td>3.605752</td>\n",
       "      <td>0.016724</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.697051</td>\n",
       "      <td>3.443405</td>\n",
       "      <td>0.012329</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.612293</td>\n",
       "      <td>3.211377</td>\n",
       "      <td>0.045410</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.507961</td>\n",
       "      <td>3.067739</td>\n",
       "      <td>0.141479</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.409343</td>\n",
       "      <td>2.992069</td>\n",
       "      <td>0.149780</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.325052</td>\n",
       "      <td>2.940532</td>\n",
       "      <td>0.143799</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.254206</td>\n",
       "      <td>2.901382</td>\n",
       "      <td>0.139771</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.195019</td>\n",
       "      <td>2.886458</td>\n",
       "      <td>0.143066</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.145554</td>\n",
       "      <td>2.883131</td>\n",
       "      <td>0.149292</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>3.103770</td>\n",
       "      <td>2.865857</td>\n",
       "      <td>0.150635</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>3.068694</td>\n",
       "      <td>2.862343</td>\n",
       "      <td>0.140869</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>3.038926</td>\n",
       "      <td>2.863393</td>\n",
       "      <td>0.152222</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>3.013623</td>\n",
       "      <td>2.862822</td>\n",
       "      <td>0.146240</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.992594</td>\n",
       "      <td>2.858943</td>\n",
       "      <td>0.145996</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.974366</td>\n",
       "      <td>2.851898</td>\n",
       "      <td>0.146484</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: qkv, run number: 1 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.333464</td>\n",
       "      <td>4.419871</td>\n",
       "      <td>0.007935</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4.282651</td>\n",
       "      <td>4.157315</td>\n",
       "      <td>0.006836</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>4.166087</td>\n",
       "      <td>3.713555</td>\n",
       "      <td>0.025391</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.990422</td>\n",
       "      <td>3.341931</td>\n",
       "      <td>0.097534</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.806828</td>\n",
       "      <td>3.108755</td>\n",
       "      <td>0.137573</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.649133</td>\n",
       "      <td>3.015585</td>\n",
       "      <td>0.146851</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.520001</td>\n",
       "      <td>2.982120</td>\n",
       "      <td>0.136353</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.414663</td>\n",
       "      <td>2.968309</td>\n",
       "      <td>0.136353</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.329285</td>\n",
       "      <td>2.966604</td>\n",
       "      <td>0.132568</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>3.259296</td>\n",
       "      <td>2.931479</td>\n",
       "      <td>0.135132</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>3.201065</td>\n",
       "      <td>2.932196</td>\n",
       "      <td>0.138916</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>3.153008</td>\n",
       "      <td>2.920511</td>\n",
       "      <td>0.129150</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>3.111893</td>\n",
       "      <td>2.923879</td>\n",
       "      <td>0.132568</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>3.077513</td>\n",
       "      <td>2.919217</td>\n",
       "      <td>0.128174</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>3.049066</td>\n",
       "      <td>2.913331</td>\n",
       "      <td>0.132935</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: qkv, run number: 2 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.716943</td>\n",
       "      <td>3.717622</td>\n",
       "      <td>0.055176</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.676887</td>\n",
       "      <td>3.601315</td>\n",
       "      <td>0.085815</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.602292</td>\n",
       "      <td>3.435218</td>\n",
       "      <td>0.129150</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.500931</td>\n",
       "      <td>3.236333</td>\n",
       "      <td>0.136230</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.399268</td>\n",
       "      <td>3.120241</td>\n",
       "      <td>0.139038</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.308859</td>\n",
       "      <td>3.022156</td>\n",
       "      <td>0.126221</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.231594</td>\n",
       "      <td>2.970079</td>\n",
       "      <td>0.139404</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.166694</td>\n",
       "      <td>2.949872</td>\n",
       "      <td>0.133423</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.112954</td>\n",
       "      <td>2.927584</td>\n",
       "      <td>0.124023</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>3.067748</td>\n",
       "      <td>2.890906</td>\n",
       "      <td>0.128052</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>3.030402</td>\n",
       "      <td>2.888458</td>\n",
       "      <td>0.125732</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.998205</td>\n",
       "      <td>2.894860</td>\n",
       "      <td>0.126343</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.971555</td>\n",
       "      <td>2.881748</td>\n",
       "      <td>0.124268</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.949126</td>\n",
       "      <td>2.882975</td>\n",
       "      <td>0.133057</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.929951</td>\n",
       "      <td>2.884398</td>\n",
       "      <td>0.128174</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: qkv, run number: 3 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.183986</td>\n",
       "      <td>4.504689</td>\n",
       "      <td>0.010742</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4.129191</td>\n",
       "      <td>4.316061</td>\n",
       "      <td>0.008667</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>4.013652</td>\n",
       "      <td>3.918771</td>\n",
       "      <td>0.012451</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.854859</td>\n",
       "      <td>3.588094</td>\n",
       "      <td>0.085571</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.699386</td>\n",
       "      <td>3.371967</td>\n",
       "      <td>0.112549</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.566658</td>\n",
       "      <td>3.243070</td>\n",
       "      <td>0.124146</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.458364</td>\n",
       "      <td>3.192609</td>\n",
       "      <td>0.133911</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.370256</td>\n",
       "      <td>3.136729</td>\n",
       "      <td>0.141479</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.297586</td>\n",
       "      <td>3.088403</td>\n",
       "      <td>0.140747</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>3.237130</td>\n",
       "      <td>3.067645</td>\n",
       "      <td>0.140015</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>3.186712</td>\n",
       "      <td>3.053146</td>\n",
       "      <td>0.144409</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>3.144801</td>\n",
       "      <td>3.050110</td>\n",
       "      <td>0.138306</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>3.109267</td>\n",
       "      <td>3.055160</td>\n",
       "      <td>0.136719</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>3.079387</td>\n",
       "      <td>3.031859</td>\n",
       "      <td>0.140991</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>3.053948</td>\n",
       "      <td>3.040991</td>\n",
       "      <td>0.138428</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: qkv, run number: 4 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.012584</td>\n",
       "      <td>4.019592</td>\n",
       "      <td>0.012939</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.965423</td>\n",
       "      <td>3.834511</td>\n",
       "      <td>0.009521</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.873155</td>\n",
       "      <td>3.567561</td>\n",
       "      <td>0.009766</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.749596</td>\n",
       "      <td>3.347189</td>\n",
       "      <td>0.077515</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.624552</td>\n",
       "      <td>3.196726</td>\n",
       "      <td>0.133423</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.511297</td>\n",
       "      <td>3.081989</td>\n",
       "      <td>0.141479</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.414356</td>\n",
       "      <td>3.033066</td>\n",
       "      <td>0.148560</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.332404</td>\n",
       "      <td>3.007699</td>\n",
       "      <td>0.144775</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.263866</td>\n",
       "      <td>2.981076</td>\n",
       "      <td>0.148315</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>3.207618</td>\n",
       "      <td>2.974679</td>\n",
       "      <td>0.149048</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>3.160498</td>\n",
       "      <td>2.954500</td>\n",
       "      <td>0.145752</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>3.120324</td>\n",
       "      <td>2.957249</td>\n",
       "      <td>0.149170</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>3.086387</td>\n",
       "      <td>2.954738</td>\n",
       "      <td>0.154785</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>3.058395</td>\n",
       "      <td>2.952957</td>\n",
       "      <td>0.150391</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>3.034655</td>\n",
       "      <td>2.952814</td>\n",
       "      <td>0.146606</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "n_layers = 4\n",
    "num_epochs = 15\n",
    "which_task = 'numbers'\n",
    "num_runs = 5\n",
    "\n",
    "results = {}\n",
    "\n",
    "\n",
    "for lr in [0.001, 0.0001]:\n",
    "  for sequence_length in [16, 32, 64, 128]:\n",
    "\n",
    "    sl = sequence_length\n",
    "    seqs = L((tensor(nums[i:i+sl]), tensor(nums[i+1:i+sl+1]))\n",
    "            for i in range(0,len(nums)-sl-1,sl))\n",
    "    cut = int(len(seqs) * 0.8)\n",
    "    dls = DataLoaders.from_dsets(seqs[:cut], seqs[cut:],\n",
    "                                bs=bs, drop_last=True, shuffle=True)\n",
    "    xb, yb = dls.one_batch()\n",
    "    xb.shape, yb.shape\n",
    "\n",
    "\n",
    "    for attn_type in ['kv', 'qkv']:\n",
    "\n",
    "          for run_no in range(num_runs):\n",
    "          \n",
    "              print(f'{which_task}---------- lr:{lr} ----  attn_type: {attn_type}, run number: {run_no} ------------------------------')\n",
    "\n",
    "\n",
    "              learn = Learner(dls, Model7(len(vocab), n_layer=n_layers, attn_type=attn_type).to(device), loss_func=loss_func, metrics=accuracy)\n",
    "              learn.fit_one_cycle(num_epochs, lr)\n",
    "\n",
    "\n",
    "              train_losss = [x[0] for x in learn.recorder.values]\n",
    "              val_losss = [x[1] for x in learn.recorder.values]\n",
    "              accs = [x[2] for x in learn.recorder.values]\n",
    "              # times = [x[3] for x in learn.recorder.values]\n",
    "\n",
    "\n",
    "              results[(attn_type, run_no)] = {}\n",
    "              # results[(norm_type, run_no)]['time_spent'] = []  \n",
    "              results[(attn_type, run_no)]['train_loss_history'] = train_losss\n",
    "              results[(attn_type, run_no)]['val_loss_history'] = val_losss\n",
    "              results[(attn_type, run_no)]['accs_history'] = accs\n",
    "              # results[(norm_type, run_no)]['val_time_spent'] = []  \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "              if not os.path.exists(f'./results_new'):\n",
    "                os.mkdir(f'./results_new')\n",
    "\n",
    "\n",
    "              with open(f'./results_new/nlp/{n_layers}_{num_epochs}_{lr}_{sequence_length}_{which_task}.pkl', 'wb') as f:\n",
    "                pickle.dump(results, f)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4_15_0.001_32_numbers.pkl\n",
      "numbers---------- lr:0.001 ----  attn_type: kv-nopos, run number: 0 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.326803</td>\n",
       "      <td>3.020899</td>\n",
       "      <td>0.151286</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.031060</td>\n",
       "      <td>2.810314</td>\n",
       "      <td>0.144043</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.784628</td>\n",
       "      <td>2.238710</td>\n",
       "      <td>0.166667</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.407574</td>\n",
       "      <td>2.249287</td>\n",
       "      <td>0.260742</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.139791</td>\n",
       "      <td>2.305454</td>\n",
       "      <td>0.212891</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.928546</td>\n",
       "      <td>2.295595</td>\n",
       "      <td>0.258708</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.745933</td>\n",
       "      <td>2.155470</td>\n",
       "      <td>0.279541</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.582291</td>\n",
       "      <td>2.228803</td>\n",
       "      <td>0.295736</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.445122</td>\n",
       "      <td>2.167104</td>\n",
       "      <td>0.310710</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.332012</td>\n",
       "      <td>2.018934</td>\n",
       "      <td>0.333903</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.239526</td>\n",
       "      <td>1.909954</td>\n",
       "      <td>0.357096</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.163098</td>\n",
       "      <td>1.879413</td>\n",
       "      <td>0.376953</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.103504</td>\n",
       "      <td>1.829355</td>\n",
       "      <td>0.389079</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.061615</td>\n",
       "      <td>1.851871</td>\n",
       "      <td>0.392822</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.036166</td>\n",
       "      <td>1.840337</td>\n",
       "      <td>0.392090</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv-nopos, run number: 1 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.514449</td>\n",
       "      <td>3.159571</td>\n",
       "      <td>0.122965</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.093980</td>\n",
       "      <td>2.632054</td>\n",
       "      <td>0.150960</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.671101</td>\n",
       "      <td>2.065940</td>\n",
       "      <td>0.180583</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.329867</td>\n",
       "      <td>2.050670</td>\n",
       "      <td>0.181234</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.081638</td>\n",
       "      <td>2.005376</td>\n",
       "      <td>0.246826</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.850556</td>\n",
       "      <td>1.902087</td>\n",
       "      <td>0.303060</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.651235</td>\n",
       "      <td>1.922983</td>\n",
       "      <td>0.300700</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.485095</td>\n",
       "      <td>1.981045</td>\n",
       "      <td>0.322184</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.342813</td>\n",
       "      <td>1.976417</td>\n",
       "      <td>0.361165</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.217951</td>\n",
       "      <td>1.650403</td>\n",
       "      <td>0.428792</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.115291</td>\n",
       "      <td>1.595026</td>\n",
       "      <td>0.449951</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.031091</td>\n",
       "      <td>1.513872</td>\n",
       "      <td>0.477214</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.970975</td>\n",
       "      <td>1.461775</td>\n",
       "      <td>0.490723</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.927966</td>\n",
       "      <td>1.449045</td>\n",
       "      <td>0.492920</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.899196</td>\n",
       "      <td>1.450699</td>\n",
       "      <td>0.494385</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv-nopos, run number: 2 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.565277</td>\n",
       "      <td>3.007708</td>\n",
       "      <td>0.139648</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.127900</td>\n",
       "      <td>2.760802</td>\n",
       "      <td>0.169840</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.710863</td>\n",
       "      <td>2.152368</td>\n",
       "      <td>0.208984</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.345836</td>\n",
       "      <td>2.264728</td>\n",
       "      <td>0.201497</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.084075</td>\n",
       "      <td>2.116754</td>\n",
       "      <td>0.246012</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.856281</td>\n",
       "      <td>2.284535</td>\n",
       "      <td>0.281006</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.656638</td>\n",
       "      <td>2.217685</td>\n",
       "      <td>0.280436</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.497470</td>\n",
       "      <td>2.259738</td>\n",
       "      <td>0.304199</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.370443</td>\n",
       "      <td>2.101777</td>\n",
       "      <td>0.324544</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.259919</td>\n",
       "      <td>1.935146</td>\n",
       "      <td>0.377523</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.160152</td>\n",
       "      <td>1.662379</td>\n",
       "      <td>0.442546</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.080822</td>\n",
       "      <td>1.597908</td>\n",
       "      <td>0.472168</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.018188</td>\n",
       "      <td>1.602688</td>\n",
       "      <td>0.472331</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.975919</td>\n",
       "      <td>1.568255</td>\n",
       "      <td>0.481689</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.948845</td>\n",
       "      <td>1.568739</td>\n",
       "      <td>0.480306</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv-nopos, run number: 3 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.350297</td>\n",
       "      <td>2.989994</td>\n",
       "      <td>0.146240</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.010202</td>\n",
       "      <td>2.737592</td>\n",
       "      <td>0.124186</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.632411</td>\n",
       "      <td>2.153244</td>\n",
       "      <td>0.123210</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.299516</td>\n",
       "      <td>2.251835</td>\n",
       "      <td>0.153890</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.052961</td>\n",
       "      <td>2.258769</td>\n",
       "      <td>0.217611</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.832952</td>\n",
       "      <td>2.206638</td>\n",
       "      <td>0.237630</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.628786</td>\n",
       "      <td>2.005414</td>\n",
       "      <td>0.298828</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.442512</td>\n",
       "      <td>1.663507</td>\n",
       "      <td>0.385173</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.285105</td>\n",
       "      <td>1.535163</td>\n",
       "      <td>0.484212</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.159610</td>\n",
       "      <td>1.461109</td>\n",
       "      <td>0.515055</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.061361</td>\n",
       "      <td>1.439048</td>\n",
       "      <td>0.537679</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.989882</td>\n",
       "      <td>1.367275</td>\n",
       "      <td>0.573405</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.936679</td>\n",
       "      <td>1.399313</td>\n",
       "      <td>0.564290</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.899073</td>\n",
       "      <td>1.371792</td>\n",
       "      <td>0.571777</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.874529</td>\n",
       "      <td>1.368088</td>\n",
       "      <td>0.573161</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv-nopos, run number: 4 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.303746</td>\n",
       "      <td>2.969366</td>\n",
       "      <td>0.165120</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.999259</td>\n",
       "      <td>2.763401</td>\n",
       "      <td>0.141846</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.667826</td>\n",
       "      <td>2.153299</td>\n",
       "      <td>0.145426</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.340427</td>\n",
       "      <td>2.134171</td>\n",
       "      <td>0.173991</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.105963</td>\n",
       "      <td>2.288272</td>\n",
       "      <td>0.250000</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.903179</td>\n",
       "      <td>2.217633</td>\n",
       "      <td>0.277832</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.718937</td>\n",
       "      <td>2.264027</td>\n",
       "      <td>0.289307</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.558228</td>\n",
       "      <td>2.274264</td>\n",
       "      <td>0.309489</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.429033</td>\n",
       "      <td>2.326253</td>\n",
       "      <td>0.314453</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.325782</td>\n",
       "      <td>2.314677</td>\n",
       "      <td>0.321696</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.246843</td>\n",
       "      <td>2.359801</td>\n",
       "      <td>0.328451</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.185843</td>\n",
       "      <td>2.305776</td>\n",
       "      <td>0.330811</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.138252</td>\n",
       "      <td>2.313439</td>\n",
       "      <td>0.338053</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.103840</td>\n",
       "      <td>2.310084</td>\n",
       "      <td>0.335856</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.082988</td>\n",
       "      <td>2.303577</td>\n",
       "      <td>0.338053</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4_15_0.001_128_numbers.pkl\n",
      "numbers---------- lr:0.001 ----  attn_type: kv-nopos, run number: 0 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.911538</td>\n",
       "      <td>3.636730</td>\n",
       "      <td>0.029297</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.620038</td>\n",
       "      <td>3.151269</td>\n",
       "      <td>0.111694</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.368024</td>\n",
       "      <td>2.905936</td>\n",
       "      <td>0.118774</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.201782</td>\n",
       "      <td>2.875824</td>\n",
       "      <td>0.151611</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.088177</td>\n",
       "      <td>2.841595</td>\n",
       "      <td>0.137695</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.996253</td>\n",
       "      <td>2.841165</td>\n",
       "      <td>0.137329</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.898537</td>\n",
       "      <td>2.831757</td>\n",
       "      <td>0.156250</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.782883</td>\n",
       "      <td>2.611784</td>\n",
       "      <td>0.192017</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.662416</td>\n",
       "      <td>2.447053</td>\n",
       "      <td>0.175903</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.550918</td>\n",
       "      <td>2.417131</td>\n",
       "      <td>0.161011</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.452265</td>\n",
       "      <td>2.365232</td>\n",
       "      <td>0.164062</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.365634</td>\n",
       "      <td>2.426832</td>\n",
       "      <td>0.158325</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.290681</td>\n",
       "      <td>2.444514</td>\n",
       "      <td>0.152832</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.226903</td>\n",
       "      <td>2.392504</td>\n",
       "      <td>0.169067</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.173142</td>\n",
       "      <td>2.409379</td>\n",
       "      <td>0.166870</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv-nopos, run number: 1 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.603318</td>\n",
       "      <td>3.418094</td>\n",
       "      <td>0.063843</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.362487</td>\n",
       "      <td>2.954473</td>\n",
       "      <td>0.143311</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.176310</td>\n",
       "      <td>2.847312</td>\n",
       "      <td>0.153687</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.061529</td>\n",
       "      <td>2.798921</td>\n",
       "      <td>0.154175</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.981585</td>\n",
       "      <td>2.771469</td>\n",
       "      <td>0.154785</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.909956</td>\n",
       "      <td>2.629352</td>\n",
       "      <td>0.154419</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.810325</td>\n",
       "      <td>2.421798</td>\n",
       "      <td>0.154785</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.682508</td>\n",
       "      <td>2.269947</td>\n",
       "      <td>0.195557</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.556165</td>\n",
       "      <td>2.168996</td>\n",
       "      <td>0.221191</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.443837</td>\n",
       "      <td>2.090613</td>\n",
       "      <td>0.254395</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.345082</td>\n",
       "      <td>2.086943</td>\n",
       "      <td>0.246826</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.259248</td>\n",
       "      <td>2.109415</td>\n",
       "      <td>0.227295</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.186400</td>\n",
       "      <td>2.100787</td>\n",
       "      <td>0.240112</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.123866</td>\n",
       "      <td>2.070876</td>\n",
       "      <td>0.255127</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.070464</td>\n",
       "      <td>2.118068</td>\n",
       "      <td>0.236572</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv-nopos, run number: 2 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.599986</td>\n",
       "      <td>3.713569</td>\n",
       "      <td>0.010498</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.391700</td>\n",
       "      <td>3.144213</td>\n",
       "      <td>0.122681</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.194291</td>\n",
       "      <td>2.878254</td>\n",
       "      <td>0.074463</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.057229</td>\n",
       "      <td>2.887819</td>\n",
       "      <td>0.096680</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.938745</td>\n",
       "      <td>2.956925</td>\n",
       "      <td>0.121216</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.796438</td>\n",
       "      <td>2.556851</td>\n",
       "      <td>0.180298</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.647686</td>\n",
       "      <td>2.281328</td>\n",
       "      <td>0.188843</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.510306</td>\n",
       "      <td>2.268284</td>\n",
       "      <td>0.236450</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.390482</td>\n",
       "      <td>2.266792</td>\n",
       "      <td>0.230957</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.287519</td>\n",
       "      <td>2.193120</td>\n",
       "      <td>0.222046</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.198520</td>\n",
       "      <td>2.258857</td>\n",
       "      <td>0.226929</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.121477</td>\n",
       "      <td>2.266976</td>\n",
       "      <td>0.235107</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.056347</td>\n",
       "      <td>2.245295</td>\n",
       "      <td>0.230591</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.000645</td>\n",
       "      <td>2.233257</td>\n",
       "      <td>0.234253</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.953421</td>\n",
       "      <td>2.265634</td>\n",
       "      <td>0.210449</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv-nopos, run number: 3 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.787850</td>\n",
       "      <td>3.761080</td>\n",
       "      <td>0.051392</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.526503</td>\n",
       "      <td>3.174684</td>\n",
       "      <td>0.139160</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.299168</td>\n",
       "      <td>2.823334</td>\n",
       "      <td>0.107178</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.134689</td>\n",
       "      <td>2.736749</td>\n",
       "      <td>0.113403</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.991895</td>\n",
       "      <td>2.608718</td>\n",
       "      <td>0.147827</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.853780</td>\n",
       "      <td>2.433534</td>\n",
       "      <td>0.134521</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.711491</td>\n",
       "      <td>2.291702</td>\n",
       "      <td>0.217651</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.576069</td>\n",
       "      <td>2.174668</td>\n",
       "      <td>0.190674</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.456802</td>\n",
       "      <td>2.111661</td>\n",
       "      <td>0.178223</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.353856</td>\n",
       "      <td>2.061056</td>\n",
       "      <td>0.228516</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.265646</td>\n",
       "      <td>2.114656</td>\n",
       "      <td>0.205444</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.190878</td>\n",
       "      <td>2.063895</td>\n",
       "      <td>0.245117</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.127537</td>\n",
       "      <td>2.106740</td>\n",
       "      <td>0.225220</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.073941</td>\n",
       "      <td>2.109460</td>\n",
       "      <td>0.221313</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.027434</td>\n",
       "      <td>2.085246</td>\n",
       "      <td>0.231812</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv-nopos, run number: 4 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.523931</td>\n",
       "      <td>3.151382</td>\n",
       "      <td>0.023804</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.338785</td>\n",
       "      <td>2.908534</td>\n",
       "      <td>0.135864</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.178080</td>\n",
       "      <td>2.914286</td>\n",
       "      <td>0.133667</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.070565</td>\n",
       "      <td>2.860195</td>\n",
       "      <td>0.151611</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.994584</td>\n",
       "      <td>2.804325</td>\n",
       "      <td>0.147827</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.933560</td>\n",
       "      <td>2.787777</td>\n",
       "      <td>0.136353</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.870609</td>\n",
       "      <td>2.750323</td>\n",
       "      <td>0.151733</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.779977</td>\n",
       "      <td>2.586813</td>\n",
       "      <td>0.187744</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.665832</td>\n",
       "      <td>2.338782</td>\n",
       "      <td>0.230957</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.553482</td>\n",
       "      <td>2.202864</td>\n",
       "      <td>0.268311</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.449982</td>\n",
       "      <td>2.095644</td>\n",
       "      <td>0.213867</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.358071</td>\n",
       "      <td>2.152134</td>\n",
       "      <td>0.224121</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.278133</td>\n",
       "      <td>2.145291</td>\n",
       "      <td>0.267334</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.210114</td>\n",
       "      <td>2.132563</td>\n",
       "      <td>0.280762</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.152645</td>\n",
       "      <td>2.120331</td>\n",
       "      <td>0.285278</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4_15_0.001_16_numbers.pkl\n",
      "numbers---------- lr:0.001 ----  attn_type: kv-nopos, run number: 0 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.282768</td>\n",
       "      <td>2.944378</td>\n",
       "      <td>0.154785</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.799024</td>\n",
       "      <td>2.535611</td>\n",
       "      <td>0.115723</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.240597</td>\n",
       "      <td>2.166733</td>\n",
       "      <td>0.230957</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1.838774</td>\n",
       "      <td>1.900278</td>\n",
       "      <td>0.384115</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1.550144</td>\n",
       "      <td>1.727630</td>\n",
       "      <td>0.473796</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.336041</td>\n",
       "      <td>1.884349</td>\n",
       "      <td>0.469076</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.156748</td>\n",
       "      <td>1.443519</td>\n",
       "      <td>0.558350</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.011917</td>\n",
       "      <td>1.282459</td>\n",
       "      <td>0.606038</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.916687</td>\n",
       "      <td>1.245787</td>\n",
       "      <td>0.627360</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.850777</td>\n",
       "      <td>1.205704</td>\n",
       "      <td>0.649251</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.800977</td>\n",
       "      <td>1.098924</td>\n",
       "      <td>0.673747</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.761861</td>\n",
       "      <td>1.128590</td>\n",
       "      <td>0.662679</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.733204</td>\n",
       "      <td>1.062060</td>\n",
       "      <td>0.677327</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.719153</td>\n",
       "      <td>1.086671</td>\n",
       "      <td>0.671956</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.710898</td>\n",
       "      <td>1.082227</td>\n",
       "      <td>0.673014</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv-nopos, run number: 1 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.223829</td>\n",
       "      <td>2.939592</td>\n",
       "      <td>0.125326</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.753808</td>\n",
       "      <td>2.459373</td>\n",
       "      <td>0.155192</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.182755</td>\n",
       "      <td>2.215002</td>\n",
       "      <td>0.217773</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1.850078</td>\n",
       "      <td>1.924473</td>\n",
       "      <td>0.358398</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1.614736</td>\n",
       "      <td>1.666760</td>\n",
       "      <td>0.472819</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.419698</td>\n",
       "      <td>1.656270</td>\n",
       "      <td>0.479492</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.257469</td>\n",
       "      <td>1.508073</td>\n",
       "      <td>0.505615</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.130433</td>\n",
       "      <td>1.449066</td>\n",
       "      <td>0.524658</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.035733</td>\n",
       "      <td>1.309827</td>\n",
       "      <td>0.558675</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.954700</td>\n",
       "      <td>1.193779</td>\n",
       "      <td>0.593262</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.898439</td>\n",
       "      <td>1.157563</td>\n",
       "      <td>0.614990</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.861475</td>\n",
       "      <td>1.127355</td>\n",
       "      <td>0.623535</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.838570</td>\n",
       "      <td>1.125081</td>\n",
       "      <td>0.627279</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.820637</td>\n",
       "      <td>1.113458</td>\n",
       "      <td>0.628662</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.811653</td>\n",
       "      <td>1.115226</td>\n",
       "      <td>0.629883</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv-nopos, run number: 2 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.117005</td>\n",
       "      <td>2.911327</td>\n",
       "      <td>0.151042</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.671216</td>\n",
       "      <td>2.285156</td>\n",
       "      <td>0.210368</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.152912</td>\n",
       "      <td>2.292931</td>\n",
       "      <td>0.254557</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1.799848</td>\n",
       "      <td>1.857081</td>\n",
       "      <td>0.420898</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1.462384</td>\n",
       "      <td>1.418757</td>\n",
       "      <td>0.540365</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.187347</td>\n",
       "      <td>1.143973</td>\n",
       "      <td>0.610026</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.006631</td>\n",
       "      <td>0.977953</td>\n",
       "      <td>0.675130</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.877421</td>\n",
       "      <td>0.992475</td>\n",
       "      <td>0.692546</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.795365</td>\n",
       "      <td>0.933508</td>\n",
       "      <td>0.699463</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.743639</td>\n",
       "      <td>0.981660</td>\n",
       "      <td>0.716878</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.709179</td>\n",
       "      <td>0.890261</td>\n",
       "      <td>0.724202</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.686412</td>\n",
       "      <td>0.889793</td>\n",
       "      <td>0.717285</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.671892</td>\n",
       "      <td>0.906358</td>\n",
       "      <td>0.719645</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.663342</td>\n",
       "      <td>0.892868</td>\n",
       "      <td>0.722819</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.658388</td>\n",
       "      <td>0.898477</td>\n",
       "      <td>0.721517</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv-nopos, run number: 3 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.032800</td>\n",
       "      <td>2.806720</td>\n",
       "      <td>0.133626</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.502334</td>\n",
       "      <td>2.158022</td>\n",
       "      <td>0.167480</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.105410</td>\n",
       "      <td>2.437473</td>\n",
       "      <td>0.259521</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1.833931</td>\n",
       "      <td>2.109867</td>\n",
       "      <td>0.304036</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1.572255</td>\n",
       "      <td>1.817263</td>\n",
       "      <td>0.434977</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.328426</td>\n",
       "      <td>1.358280</td>\n",
       "      <td>0.540934</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.123454</td>\n",
       "      <td>1.084209</td>\n",
       "      <td>0.627197</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.993258</td>\n",
       "      <td>1.031687</td>\n",
       "      <td>0.641683</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.913542</td>\n",
       "      <td>1.034096</td>\n",
       "      <td>0.659587</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.864038</td>\n",
       "      <td>1.015102</td>\n",
       "      <td>0.664632</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.824781</td>\n",
       "      <td>1.044665</td>\n",
       "      <td>0.670410</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.799096</td>\n",
       "      <td>1.001438</td>\n",
       "      <td>0.673258</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.781785</td>\n",
       "      <td>1.010119</td>\n",
       "      <td>0.680827</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.765786</td>\n",
       "      <td>0.963678</td>\n",
       "      <td>0.691650</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.758459</td>\n",
       "      <td>0.971225</td>\n",
       "      <td>0.690186</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv-nopos, run number: 4 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.208045</td>\n",
       "      <td>2.851425</td>\n",
       "      <td>0.140299</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.591146</td>\n",
       "      <td>2.132311</td>\n",
       "      <td>0.159831</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.126476</td>\n",
       "      <td>2.260101</td>\n",
       "      <td>0.233236</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1.815717</td>\n",
       "      <td>1.920452</td>\n",
       "      <td>0.357829</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1.530086</td>\n",
       "      <td>1.510850</td>\n",
       "      <td>0.483398</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.296167</td>\n",
       "      <td>1.182478</td>\n",
       "      <td>0.624430</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.116079</td>\n",
       "      <td>1.086876</td>\n",
       "      <td>0.658284</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.990729</td>\n",
       "      <td>1.010036</td>\n",
       "      <td>0.679769</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.904319</td>\n",
       "      <td>1.027267</td>\n",
       "      <td>0.680094</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.838734</td>\n",
       "      <td>0.889403</td>\n",
       "      <td>0.715658</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>0.786901</td>\n",
       "      <td>0.889026</td>\n",
       "      <td>0.716064</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>0.754434</td>\n",
       "      <td>0.854837</td>\n",
       "      <td>0.725993</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>0.729307</td>\n",
       "      <td>0.885998</td>\n",
       "      <td>0.722575</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>0.718203</td>\n",
       "      <td>0.883859</td>\n",
       "      <td>0.720784</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>0.711980</td>\n",
       "      <td>0.880723</td>\n",
       "      <td>0.719320</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4_15_0.0001_32_numbers.pkl\n",
      "numbers---------- lr:0.0001 ----  attn_type: kv-nopos, run number: 0 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.788891</td>\n",
       "      <td>3.982994</td>\n",
       "      <td>0.007975</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.576691</td>\n",
       "      <td>3.405529</td>\n",
       "      <td>0.101969</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.317848</td>\n",
       "      <td>2.984964</td>\n",
       "      <td>0.129964</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.099138</td>\n",
       "      <td>2.772202</td>\n",
       "      <td>0.130941</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.902322</td>\n",
       "      <td>2.564286</td>\n",
       "      <td>0.151367</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.683233</td>\n",
       "      <td>2.385621</td>\n",
       "      <td>0.150716</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.480208</td>\n",
       "      <td>2.266459</td>\n",
       "      <td>0.158691</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.318814</td>\n",
       "      <td>2.200518</td>\n",
       "      <td>0.155029</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.200490</td>\n",
       "      <td>2.182054</td>\n",
       "      <td>0.198242</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.117126</td>\n",
       "      <td>2.171368</td>\n",
       "      <td>0.192139</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.058620</td>\n",
       "      <td>2.163592</td>\n",
       "      <td>0.193359</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.019323</td>\n",
       "      <td>2.159632</td>\n",
       "      <td>0.195882</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.990950</td>\n",
       "      <td>2.161445</td>\n",
       "      <td>0.205648</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.973052</td>\n",
       "      <td>2.160371</td>\n",
       "      <td>0.204102</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.961925</td>\n",
       "      <td>2.161112</td>\n",
       "      <td>0.201335</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv-nopos, run number: 1 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.772570</td>\n",
       "      <td>3.811592</td>\n",
       "      <td>0.054199</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.569444</td>\n",
       "      <td>3.369143</td>\n",
       "      <td>0.091227</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.305875</td>\n",
       "      <td>3.016083</td>\n",
       "      <td>0.137451</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.104151</td>\n",
       "      <td>2.874228</td>\n",
       "      <td>0.144694</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.953083</td>\n",
       "      <td>2.757483</td>\n",
       "      <td>0.138835</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.813450</td>\n",
       "      <td>2.668792</td>\n",
       "      <td>0.155762</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.687867</td>\n",
       "      <td>2.591168</td>\n",
       "      <td>0.156982</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.581160</td>\n",
       "      <td>2.557688</td>\n",
       "      <td>0.167643</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.488021</td>\n",
       "      <td>2.520365</td>\n",
       "      <td>0.186930</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.397196</td>\n",
       "      <td>2.465194</td>\n",
       "      <td>0.197835</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.317071</td>\n",
       "      <td>2.416253</td>\n",
       "      <td>0.203288</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.251629</td>\n",
       "      <td>2.388426</td>\n",
       "      <td>0.200033</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.203184</td>\n",
       "      <td>2.369028</td>\n",
       "      <td>0.200358</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.169646</td>\n",
       "      <td>2.362288</td>\n",
       "      <td>0.199219</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.147087</td>\n",
       "      <td>2.361224</td>\n",
       "      <td>0.198649</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv-nopos, run number: 2 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.976724</td>\n",
       "      <td>3.860344</td>\n",
       "      <td>0.009847</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.758102</td>\n",
       "      <td>3.351014</td>\n",
       "      <td>0.068685</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.441222</td>\n",
       "      <td>2.948176</td>\n",
       "      <td>0.154215</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.186257</td>\n",
       "      <td>2.800805</td>\n",
       "      <td>0.161947</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.004714</td>\n",
       "      <td>2.666316</td>\n",
       "      <td>0.163981</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.832710</td>\n",
       "      <td>2.472809</td>\n",
       "      <td>0.191569</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.651735</td>\n",
       "      <td>2.304093</td>\n",
       "      <td>0.199056</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.484442</td>\n",
       "      <td>2.215018</td>\n",
       "      <td>0.188883</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.346375</td>\n",
       "      <td>2.163038</td>\n",
       "      <td>0.201742</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.242635</td>\n",
       "      <td>2.131357</td>\n",
       "      <td>0.196289</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.167929</td>\n",
       "      <td>2.127648</td>\n",
       "      <td>0.200846</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.116857</td>\n",
       "      <td>2.123460</td>\n",
       "      <td>0.189941</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.082787</td>\n",
       "      <td>2.122324</td>\n",
       "      <td>0.197754</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.059448</td>\n",
       "      <td>2.123229</td>\n",
       "      <td>0.198568</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.044436</td>\n",
       "      <td>2.122380</td>\n",
       "      <td>0.199056</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv-nopos, run number: 3 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.966985</td>\n",
       "      <td>3.903271</td>\n",
       "      <td>0.001790</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.740236</td>\n",
       "      <td>3.388279</td>\n",
       "      <td>0.063639</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.432973</td>\n",
       "      <td>3.056093</td>\n",
       "      <td>0.145671</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.194297</td>\n",
       "      <td>2.897622</td>\n",
       "      <td>0.145020</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.026521</td>\n",
       "      <td>2.814209</td>\n",
       "      <td>0.143473</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.887059</td>\n",
       "      <td>2.658069</td>\n",
       "      <td>0.139404</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.721631</td>\n",
       "      <td>2.451015</td>\n",
       "      <td>0.154948</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.543761</td>\n",
       "      <td>2.295115</td>\n",
       "      <td>0.167399</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.393561</td>\n",
       "      <td>2.210131</td>\n",
       "      <td>0.169189</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.277745</td>\n",
       "      <td>2.170043</td>\n",
       "      <td>0.171631</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.192099</td>\n",
       "      <td>2.144767</td>\n",
       "      <td>0.171224</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.132810</td>\n",
       "      <td>2.132349</td>\n",
       "      <td>0.179443</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.091919</td>\n",
       "      <td>2.124367</td>\n",
       "      <td>0.177246</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.064260</td>\n",
       "      <td>2.122605</td>\n",
       "      <td>0.175374</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.047566</td>\n",
       "      <td>2.121540</td>\n",
       "      <td>0.175700</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv-nopos, run number: 4 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.221975</td>\n",
       "      <td>4.000714</td>\n",
       "      <td>0.014079</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.960573</td>\n",
       "      <td>3.317142</td>\n",
       "      <td>0.010010</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.574685</td>\n",
       "      <td>2.917142</td>\n",
       "      <td>0.135742</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.275273</td>\n",
       "      <td>2.791205</td>\n",
       "      <td>0.143717</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.049212</td>\n",
       "      <td>2.663363</td>\n",
       "      <td>0.150391</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.824650</td>\n",
       "      <td>2.478489</td>\n",
       "      <td>0.153727</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.612766</td>\n",
       "      <td>2.328286</td>\n",
       "      <td>0.154297</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.440183</td>\n",
       "      <td>2.232776</td>\n",
       "      <td>0.166992</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.306549</td>\n",
       "      <td>2.180776</td>\n",
       "      <td>0.197591</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.206108</td>\n",
       "      <td>2.148000</td>\n",
       "      <td>0.205892</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.132782</td>\n",
       "      <td>2.122235</td>\n",
       "      <td>0.214762</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.081527</td>\n",
       "      <td>2.121243</td>\n",
       "      <td>0.222249</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.047517</td>\n",
       "      <td>2.115918</td>\n",
       "      <td>0.221273</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.024085</td>\n",
       "      <td>2.114051</td>\n",
       "      <td>0.220378</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.008634</td>\n",
       "      <td>2.115664</td>\n",
       "      <td>0.220866</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4_15_0.0001_64_numbers.pkl\n",
      "numbers---------- lr:0.0001 ----  attn_type: kv-nopos, run number: 0 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.010540</td>\n",
       "      <td>4.018196</td>\n",
       "      <td>0.007568</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.909389</td>\n",
       "      <td>3.730285</td>\n",
       "      <td>0.021973</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.719113</td>\n",
       "      <td>3.375162</td>\n",
       "      <td>0.123291</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.513675</td>\n",
       "      <td>3.144850</td>\n",
       "      <td>0.154378</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.346413</td>\n",
       "      <td>3.022355</td>\n",
       "      <td>0.153076</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.218892</td>\n",
       "      <td>2.943650</td>\n",
       "      <td>0.153483</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.119738</td>\n",
       "      <td>2.889702</td>\n",
       "      <td>0.156576</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.042707</td>\n",
       "      <td>2.858454</td>\n",
       "      <td>0.158203</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.979326</td>\n",
       "      <td>2.831104</td>\n",
       "      <td>0.157878</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.927322</td>\n",
       "      <td>2.809915</td>\n",
       "      <td>0.160319</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.883887</td>\n",
       "      <td>2.789585</td>\n",
       "      <td>0.161865</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.847123</td>\n",
       "      <td>2.774964</td>\n",
       "      <td>0.162923</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.816857</td>\n",
       "      <td>2.767357</td>\n",
       "      <td>0.163981</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.792124</td>\n",
       "      <td>2.766000</td>\n",
       "      <td>0.164388</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.771636</td>\n",
       "      <td>2.763833</td>\n",
       "      <td>0.164958</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv-nopos, run number: 1 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.669616</td>\n",
       "      <td>3.391890</td>\n",
       "      <td>0.084473</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.573569</td>\n",
       "      <td>3.196163</td>\n",
       "      <td>0.172770</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.428375</td>\n",
       "      <td>3.032858</td>\n",
       "      <td>0.182129</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.282620</td>\n",
       "      <td>2.940175</td>\n",
       "      <td>0.169840</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.161929</td>\n",
       "      <td>2.914197</td>\n",
       "      <td>0.151286</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.065962</td>\n",
       "      <td>2.887295</td>\n",
       "      <td>0.132406</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.988480</td>\n",
       "      <td>2.864544</td>\n",
       "      <td>0.120361</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.922299</td>\n",
       "      <td>2.842011</td>\n",
       "      <td>0.109782</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.862444</td>\n",
       "      <td>2.819597</td>\n",
       "      <td>0.099935</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.806949</td>\n",
       "      <td>2.797694</td>\n",
       "      <td>0.095785</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.754762</td>\n",
       "      <td>2.773083</td>\n",
       "      <td>0.090088</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.708793</td>\n",
       "      <td>2.758979</td>\n",
       "      <td>0.086914</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.667909</td>\n",
       "      <td>2.749310</td>\n",
       "      <td>0.085938</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.635168</td>\n",
       "      <td>2.747010</td>\n",
       "      <td>0.087891</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.609679</td>\n",
       "      <td>2.745152</td>\n",
       "      <td>0.085856</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv-nopos, run number: 2 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.020282</td>\n",
       "      <td>3.931911</td>\n",
       "      <td>0.003337</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.896039</td>\n",
       "      <td>3.499726</td>\n",
       "      <td>0.016683</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.683948</td>\n",
       "      <td>3.098738</td>\n",
       "      <td>0.132487</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.477431</td>\n",
       "      <td>2.965060</td>\n",
       "      <td>0.140299</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.316898</td>\n",
       "      <td>2.922234</td>\n",
       "      <td>0.140544</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.194645</td>\n",
       "      <td>2.895104</td>\n",
       "      <td>0.149658</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.099192</td>\n",
       "      <td>2.879285</td>\n",
       "      <td>0.150879</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.022317</td>\n",
       "      <td>2.870580</td>\n",
       "      <td>0.148844</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.959471</td>\n",
       "      <td>2.872037</td>\n",
       "      <td>0.152100</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.905326</td>\n",
       "      <td>2.878828</td>\n",
       "      <td>0.147949</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.858013</td>\n",
       "      <td>2.884927</td>\n",
       "      <td>0.148438</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.817217</td>\n",
       "      <td>2.889329</td>\n",
       "      <td>0.148275</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.783373</td>\n",
       "      <td>2.890288</td>\n",
       "      <td>0.146729</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.756281</td>\n",
       "      <td>2.890193</td>\n",
       "      <td>0.148193</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.734318</td>\n",
       "      <td>2.892042</td>\n",
       "      <td>0.147786</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv-nopos, run number: 3 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.833795</td>\n",
       "      <td>3.859208</td>\n",
       "      <td>0.029948</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.747284</td>\n",
       "      <td>3.541175</td>\n",
       "      <td>0.022624</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.596215</td>\n",
       "      <td>3.162030</td>\n",
       "      <td>0.053060</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.420518</td>\n",
       "      <td>2.980672</td>\n",
       "      <td>0.109212</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.268608</td>\n",
       "      <td>2.908681</td>\n",
       "      <td>0.121012</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.150368</td>\n",
       "      <td>2.882752</td>\n",
       "      <td>0.125407</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.054502</td>\n",
       "      <td>2.863431</td>\n",
       "      <td>0.129232</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.975709</td>\n",
       "      <td>2.859876</td>\n",
       "      <td>0.132731</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.904353</td>\n",
       "      <td>2.849520</td>\n",
       "      <td>0.141032</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.839360</td>\n",
       "      <td>2.853543</td>\n",
       "      <td>0.147217</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.780902</td>\n",
       "      <td>2.849458</td>\n",
       "      <td>0.148600</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.728848</td>\n",
       "      <td>2.847091</td>\n",
       "      <td>0.151530</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.685019</td>\n",
       "      <td>2.845053</td>\n",
       "      <td>0.153402</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.649735</td>\n",
       "      <td>2.841277</td>\n",
       "      <td>0.154622</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.621285</td>\n",
       "      <td>2.841181</td>\n",
       "      <td>0.154460</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv-nopos, run number: 4 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.135225</td>\n",
       "      <td>4.099803</td>\n",
       "      <td>0.030029</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4.017800</td>\n",
       "      <td>3.773444</td>\n",
       "      <td>0.017253</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.790616</td>\n",
       "      <td>3.307823</td>\n",
       "      <td>0.097819</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.551802</td>\n",
       "      <td>3.074879</td>\n",
       "      <td>0.120524</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.366662</td>\n",
       "      <td>2.990604</td>\n",
       "      <td>0.117757</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.227599</td>\n",
       "      <td>2.952451</td>\n",
       "      <td>0.130778</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.122315</td>\n",
       "      <td>2.923857</td>\n",
       "      <td>0.132568</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.039625</td>\n",
       "      <td>2.913751</td>\n",
       "      <td>0.134359</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.972290</td>\n",
       "      <td>2.913548</td>\n",
       "      <td>0.138184</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.916328</td>\n",
       "      <td>2.910982</td>\n",
       "      <td>0.144124</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.867799</td>\n",
       "      <td>2.914658</td>\n",
       "      <td>0.144938</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.828050</td>\n",
       "      <td>2.920719</td>\n",
       "      <td>0.146403</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.794589</td>\n",
       "      <td>2.924521</td>\n",
       "      <td>0.146647</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.768259</td>\n",
       "      <td>2.921252</td>\n",
       "      <td>0.145752</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.746530</td>\n",
       "      <td>2.925096</td>\n",
       "      <td>0.146240</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4_15_0.001_64_numbers.pkl\n",
      "numbers---------- lr:0.001 ----  attn_type: kv-nopos, run number: 0 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.710739</td>\n",
       "      <td>3.298854</td>\n",
       "      <td>0.031657</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.365366</td>\n",
       "      <td>2.854093</td>\n",
       "      <td>0.154134</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.134145</td>\n",
       "      <td>2.833971</td>\n",
       "      <td>0.149333</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.948465</td>\n",
       "      <td>2.631584</td>\n",
       "      <td>0.173991</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.706468</td>\n",
       "      <td>2.193442</td>\n",
       "      <td>0.166504</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.467100</td>\n",
       "      <td>2.215027</td>\n",
       "      <td>0.242269</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.274833</td>\n",
       "      <td>2.230554</td>\n",
       "      <td>0.208984</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.117094</td>\n",
       "      <td>2.288951</td>\n",
       "      <td>0.224040</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.984900</td>\n",
       "      <td>2.237735</td>\n",
       "      <td>0.253499</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.870168</td>\n",
       "      <td>2.212727</td>\n",
       "      <td>0.258464</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.771807</td>\n",
       "      <td>2.174861</td>\n",
       "      <td>0.261637</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.688492</td>\n",
       "      <td>2.196735</td>\n",
       "      <td>0.260986</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.620371</td>\n",
       "      <td>2.219883</td>\n",
       "      <td>0.264974</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.563876</td>\n",
       "      <td>2.215259</td>\n",
       "      <td>0.267497</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.519975</td>\n",
       "      <td>2.224475</td>\n",
       "      <td>0.267904</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv-nopos, run number: 1 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.720624</td>\n",
       "      <td>3.428506</td>\n",
       "      <td>0.075195</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.328877</td>\n",
       "      <td>2.828905</td>\n",
       "      <td>0.142985</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.103181</td>\n",
       "      <td>2.747236</td>\n",
       "      <td>0.146729</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.907676</td>\n",
       "      <td>2.318624</td>\n",
       "      <td>0.172445</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.651434</td>\n",
       "      <td>2.042190</td>\n",
       "      <td>0.236735</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.436569</td>\n",
       "      <td>2.053672</td>\n",
       "      <td>0.223633</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.261246</td>\n",
       "      <td>2.053599</td>\n",
       "      <td>0.245361</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.113921</td>\n",
       "      <td>2.138531</td>\n",
       "      <td>0.246826</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.987472</td>\n",
       "      <td>2.082189</td>\n",
       "      <td>0.256429</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.879983</td>\n",
       "      <td>2.042519</td>\n",
       "      <td>0.249023</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.787094</td>\n",
       "      <td>2.068541</td>\n",
       "      <td>0.253825</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.709407</td>\n",
       "      <td>2.066758</td>\n",
       "      <td>0.257080</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.644236</td>\n",
       "      <td>2.059251</td>\n",
       "      <td>0.256185</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.593255</td>\n",
       "      <td>2.065084</td>\n",
       "      <td>0.257487</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.552456</td>\n",
       "      <td>2.054775</td>\n",
       "      <td>0.258382</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv-nopos, run number: 2 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.748298</td>\n",
       "      <td>3.284523</td>\n",
       "      <td>0.066162</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.372123</td>\n",
       "      <td>2.865279</td>\n",
       "      <td>0.130371</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.102929</td>\n",
       "      <td>2.749906</td>\n",
       "      <td>0.118896</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.831168</td>\n",
       "      <td>2.351525</td>\n",
       "      <td>0.126383</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.571876</td>\n",
       "      <td>2.098140</td>\n",
       "      <td>0.165039</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.369127</td>\n",
       "      <td>2.138177</td>\n",
       "      <td>0.189616</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.203411</td>\n",
       "      <td>2.127321</td>\n",
       "      <td>0.220622</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.060726</td>\n",
       "      <td>2.144757</td>\n",
       "      <td>0.249430</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.932947</td>\n",
       "      <td>2.058015</td>\n",
       "      <td>0.276937</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.819321</td>\n",
       "      <td>2.047759</td>\n",
       "      <td>0.275309</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.721320</td>\n",
       "      <td>2.082479</td>\n",
       "      <td>0.286296</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.637917</td>\n",
       "      <td>2.087503</td>\n",
       "      <td>0.286458</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.570271</td>\n",
       "      <td>2.088236</td>\n",
       "      <td>0.286133</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.515414</td>\n",
       "      <td>2.097181</td>\n",
       "      <td>0.285970</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.473396</td>\n",
       "      <td>2.097023</td>\n",
       "      <td>0.288737</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv-nopos, run number: 3 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.062104</td>\n",
       "      <td>3.507018</td>\n",
       "      <td>0.136475</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.577225</td>\n",
       "      <td>2.911824</td>\n",
       "      <td>0.130127</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.263597</td>\n",
       "      <td>2.770314</td>\n",
       "      <td>0.149902</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.057950</td>\n",
       "      <td>2.475074</td>\n",
       "      <td>0.171305</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.790841</td>\n",
       "      <td>2.145939</td>\n",
       "      <td>0.198242</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.552974</td>\n",
       "      <td>2.066573</td>\n",
       "      <td>0.172933</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.362830</td>\n",
       "      <td>2.112776</td>\n",
       "      <td>0.254720</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.206119</td>\n",
       "      <td>2.099994</td>\n",
       "      <td>0.257406</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.069004</td>\n",
       "      <td>2.077804</td>\n",
       "      <td>0.264648</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.949005</td>\n",
       "      <td>2.098068</td>\n",
       "      <td>0.262044</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.845914</td>\n",
       "      <td>2.080393</td>\n",
       "      <td>0.268880</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.758504</td>\n",
       "      <td>2.072529</td>\n",
       "      <td>0.251546</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.685754</td>\n",
       "      <td>2.081951</td>\n",
       "      <td>0.270020</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.627626</td>\n",
       "      <td>2.076927</td>\n",
       "      <td>0.253662</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.582317</td>\n",
       "      <td>2.086326</td>\n",
       "      <td>0.249023</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.001 ----  attn_type: kv-nopos, run number: 4 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.366551</td>\n",
       "      <td>3.142406</td>\n",
       "      <td>0.144857</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.119657</td>\n",
       "      <td>2.830427</td>\n",
       "      <td>0.150472</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.978676</td>\n",
       "      <td>2.801006</td>\n",
       "      <td>0.164876</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.852219</td>\n",
       "      <td>2.371310</td>\n",
       "      <td>0.150472</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.620749</td>\n",
       "      <td>2.171168</td>\n",
       "      <td>0.179850</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.410542</td>\n",
       "      <td>2.218636</td>\n",
       "      <td>0.251628</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.232699</td>\n",
       "      <td>2.283237</td>\n",
       "      <td>0.213135</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.075931</td>\n",
       "      <td>2.272716</td>\n",
       "      <td>0.249023</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.935475</td>\n",
       "      <td>2.203132</td>\n",
       "      <td>0.256022</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.809967</td>\n",
       "      <td>2.158603</td>\n",
       "      <td>0.258382</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.703604</td>\n",
       "      <td>2.209347</td>\n",
       "      <td>0.269775</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.614604</td>\n",
       "      <td>2.259757</td>\n",
       "      <td>0.266113</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.542451</td>\n",
       "      <td>2.254811</td>\n",
       "      <td>0.270264</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.485871</td>\n",
       "      <td>2.262680</td>\n",
       "      <td>0.270915</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.441090</td>\n",
       "      <td>2.264308</td>\n",
       "      <td>0.270915</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4_15_0.0001_16_numbers.pkl\n",
      "numbers---------- lr:0.0001 ----  attn_type: kv-nopos, run number: 0 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.906752</td>\n",
       "      <td>3.887024</td>\n",
       "      <td>0.017008</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.435829</td>\n",
       "      <td>3.056225</td>\n",
       "      <td>0.156169</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.071946</td>\n",
       "      <td>2.873890</td>\n",
       "      <td>0.181966</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.865332</td>\n",
       "      <td>2.773181</td>\n",
       "      <td>0.190023</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.624444</td>\n",
       "      <td>2.530368</td>\n",
       "      <td>0.212565</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.321834</td>\n",
       "      <td>2.290946</td>\n",
       "      <td>0.170166</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.124783</td>\n",
       "      <td>2.245562</td>\n",
       "      <td>0.174479</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.016585</td>\n",
       "      <td>2.272637</td>\n",
       "      <td>0.182699</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.951034</td>\n",
       "      <td>2.310341</td>\n",
       "      <td>0.227295</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.906161</td>\n",
       "      <td>2.294251</td>\n",
       "      <td>0.237549</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.875407</td>\n",
       "      <td>2.283726</td>\n",
       "      <td>0.239665</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.854944</td>\n",
       "      <td>2.276602</td>\n",
       "      <td>0.247721</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.840788</td>\n",
       "      <td>2.283763</td>\n",
       "      <td>0.262044</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.833098</td>\n",
       "      <td>2.277117</td>\n",
       "      <td>0.263184</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.829316</td>\n",
       "      <td>2.278829</td>\n",
       "      <td>0.263997</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv-nopos, run number: 1 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.740017</td>\n",
       "      <td>3.616447</td>\n",
       "      <td>0.070312</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.348908</td>\n",
       "      <td>3.073793</td>\n",
       "      <td>0.148926</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.001520</td>\n",
       "      <td>2.713579</td>\n",
       "      <td>0.149007</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.609661</td>\n",
       "      <td>2.327627</td>\n",
       "      <td>0.170003</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.302049</td>\n",
       "      <td>2.192929</td>\n",
       "      <td>0.200684</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.114176</td>\n",
       "      <td>2.142152</td>\n",
       "      <td>0.190348</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.010646</td>\n",
       "      <td>2.157420</td>\n",
       "      <td>0.197835</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.950693</td>\n",
       "      <td>2.173926</td>\n",
       "      <td>0.206299</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.914710</td>\n",
       "      <td>2.205755</td>\n",
       "      <td>0.216064</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.889639</td>\n",
       "      <td>2.221896</td>\n",
       "      <td>0.218831</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.865121</td>\n",
       "      <td>2.229166</td>\n",
       "      <td>0.233561</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.851107</td>\n",
       "      <td>2.234391</td>\n",
       "      <td>0.236328</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.842373</td>\n",
       "      <td>2.245298</td>\n",
       "      <td>0.245199</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.834122</td>\n",
       "      <td>2.242909</td>\n",
       "      <td>0.245768</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.832415</td>\n",
       "      <td>2.245298</td>\n",
       "      <td>0.245768</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv-nopos, run number: 2 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.561407</td>\n",
       "      <td>3.144603</td>\n",
       "      <td>0.027751</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.222528</td>\n",
       "      <td>2.811509</td>\n",
       "      <td>0.182048</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.944986</td>\n",
       "      <td>2.709849</td>\n",
       "      <td>0.154215</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.660495</td>\n",
       "      <td>2.340330</td>\n",
       "      <td>0.189860</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.311168</td>\n",
       "      <td>2.182970</td>\n",
       "      <td>0.214844</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.099788</td>\n",
       "      <td>2.181325</td>\n",
       "      <td>0.240153</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.983928</td>\n",
       "      <td>2.212196</td>\n",
       "      <td>0.247559</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.909877</td>\n",
       "      <td>2.227614</td>\n",
       "      <td>0.252767</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.862352</td>\n",
       "      <td>2.219564</td>\n",
       "      <td>0.232503</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.819545</td>\n",
       "      <td>2.189204</td>\n",
       "      <td>0.246501</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.792000</td>\n",
       "      <td>2.181141</td>\n",
       "      <td>0.268636</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.770259</td>\n",
       "      <td>2.167828</td>\n",
       "      <td>0.268392</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.754477</td>\n",
       "      <td>2.151115</td>\n",
       "      <td>0.279785</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.744431</td>\n",
       "      <td>2.152096</td>\n",
       "      <td>0.277344</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.740890</td>\n",
       "      <td>2.152387</td>\n",
       "      <td>0.276774</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv-nopos, run number: 3 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.773418</td>\n",
       "      <td>3.483919</td>\n",
       "      <td>0.063802</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.365049</td>\n",
       "      <td>2.967811</td>\n",
       "      <td>0.172933</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.026349</td>\n",
       "      <td>2.784224</td>\n",
       "      <td>0.164388</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.690516</td>\n",
       "      <td>2.362324</td>\n",
       "      <td>0.171387</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.345680</td>\n",
       "      <td>2.152173</td>\n",
       "      <td>0.171143</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.132761</td>\n",
       "      <td>2.086191</td>\n",
       "      <td>0.186442</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.022820</td>\n",
       "      <td>2.081572</td>\n",
       "      <td>0.165202</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.961240</td>\n",
       "      <td>2.113585</td>\n",
       "      <td>0.217041</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.926046</td>\n",
       "      <td>2.139958</td>\n",
       "      <td>0.210205</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.894571</td>\n",
       "      <td>2.187113</td>\n",
       "      <td>0.213867</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.871562</td>\n",
       "      <td>2.194308</td>\n",
       "      <td>0.221761</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.858491</td>\n",
       "      <td>2.215609</td>\n",
       "      <td>0.220622</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.852539</td>\n",
       "      <td>2.215555</td>\n",
       "      <td>0.228027</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.841511</td>\n",
       "      <td>2.215695</td>\n",
       "      <td>0.224854</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.840015</td>\n",
       "      <td>2.219449</td>\n",
       "      <td>0.224202</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv-nopos, run number: 4 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.963679</td>\n",
       "      <td>3.719703</td>\n",
       "      <td>0.032633</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.471178</td>\n",
       "      <td>3.013700</td>\n",
       "      <td>0.174479</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.063762</td>\n",
       "      <td>2.770408</td>\n",
       "      <td>0.156657</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.675071</td>\n",
       "      <td>2.337745</td>\n",
       "      <td>0.151937</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.306800</td>\n",
       "      <td>2.193826</td>\n",
       "      <td>0.156982</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.104560</td>\n",
       "      <td>2.173509</td>\n",
       "      <td>0.146891</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.002033</td>\n",
       "      <td>2.218451</td>\n",
       "      <td>0.172933</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.942125</td>\n",
       "      <td>2.267666</td>\n",
       "      <td>0.172770</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.900723</td>\n",
       "      <td>2.277162</td>\n",
       "      <td>0.189372</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.872398</td>\n",
       "      <td>2.265160</td>\n",
       "      <td>0.190755</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>1.844538</td>\n",
       "      <td>2.268383</td>\n",
       "      <td>0.193359</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>1.825426</td>\n",
       "      <td>2.257449</td>\n",
       "      <td>0.212972</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>1.811101</td>\n",
       "      <td>2.245159</td>\n",
       "      <td>0.213135</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.805257</td>\n",
       "      <td>2.244862</td>\n",
       "      <td>0.214844</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.799719</td>\n",
       "      <td>2.246069</td>\n",
       "      <td>0.215251</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4_15_0.0001_128_numbers.pkl\n",
      "numbers---------- lr:0.0001 ----  attn_type: kv-nopos, run number: 0 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.759986</td>\n",
       "      <td>3.835770</td>\n",
       "      <td>0.056396</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.727001</td>\n",
       "      <td>3.735206</td>\n",
       "      <td>0.059692</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.658362</td>\n",
       "      <td>3.553481</td>\n",
       "      <td>0.110596</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.570760</td>\n",
       "      <td>3.376137</td>\n",
       "      <td>0.138794</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.477776</td>\n",
       "      <td>3.262931</td>\n",
       "      <td>0.138916</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.391411</td>\n",
       "      <td>3.191194</td>\n",
       "      <td>0.133057</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.314557</td>\n",
       "      <td>3.130647</td>\n",
       "      <td>0.141357</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.246521</td>\n",
       "      <td>3.078539</td>\n",
       "      <td>0.138672</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.188156</td>\n",
       "      <td>3.072097</td>\n",
       "      <td>0.139771</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>3.137758</td>\n",
       "      <td>3.037115</td>\n",
       "      <td>0.141968</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>3.093534</td>\n",
       "      <td>3.026830</td>\n",
       "      <td>0.141968</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>3.055924</td>\n",
       "      <td>3.026901</td>\n",
       "      <td>0.136963</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>3.023267</td>\n",
       "      <td>3.002803</td>\n",
       "      <td>0.140259</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.996029</td>\n",
       "      <td>2.993584</td>\n",
       "      <td>0.140381</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.973166</td>\n",
       "      <td>3.007573</td>\n",
       "      <td>0.130005</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv-nopos, run number: 1 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.757943</td>\n",
       "      <td>3.820870</td>\n",
       "      <td>0.033081</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.715549</td>\n",
       "      <td>3.689784</td>\n",
       "      <td>0.041382</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.631794</td>\n",
       "      <td>3.418007</td>\n",
       "      <td>0.096680</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.521267</td>\n",
       "      <td>3.209004</td>\n",
       "      <td>0.136353</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.414556</td>\n",
       "      <td>3.105817</td>\n",
       "      <td>0.140747</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.322298</td>\n",
       "      <td>3.027052</td>\n",
       "      <td>0.138306</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.244085</td>\n",
       "      <td>2.995631</td>\n",
       "      <td>0.139893</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.178397</td>\n",
       "      <td>2.961170</td>\n",
       "      <td>0.145264</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.123964</td>\n",
       "      <td>2.920214</td>\n",
       "      <td>0.142090</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>3.077712</td>\n",
       "      <td>2.907954</td>\n",
       "      <td>0.141846</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>3.038662</td>\n",
       "      <td>2.906274</td>\n",
       "      <td>0.142700</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>3.004948</td>\n",
       "      <td>2.887065</td>\n",
       "      <td>0.148315</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.977268</td>\n",
       "      <td>2.882076</td>\n",
       "      <td>0.143188</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.953905</td>\n",
       "      <td>2.874168</td>\n",
       "      <td>0.151733</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.933987</td>\n",
       "      <td>2.879559</td>\n",
       "      <td>0.145142</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv-nopos, run number: 2 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.061850</td>\n",
       "      <td>3.816183</td>\n",
       "      <td>0.018799</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4.027058</td>\n",
       "      <td>3.709921</td>\n",
       "      <td>0.031006</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.948320</td>\n",
       "      <td>3.517023</td>\n",
       "      <td>0.072754</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.835595</td>\n",
       "      <td>3.318733</td>\n",
       "      <td>0.097046</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.708035</td>\n",
       "      <td>3.142534</td>\n",
       "      <td>0.112549</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.584185</td>\n",
       "      <td>2.993020</td>\n",
       "      <td>0.135376</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.471812</td>\n",
       "      <td>2.927533</td>\n",
       "      <td>0.145752</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.375878</td>\n",
       "      <td>2.888229</td>\n",
       "      <td>0.151123</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.294712</td>\n",
       "      <td>2.849843</td>\n",
       "      <td>0.149536</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>3.227080</td>\n",
       "      <td>2.830745</td>\n",
       "      <td>0.143433</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>3.169930</td>\n",
       "      <td>2.827409</td>\n",
       "      <td>0.141479</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>3.121796</td>\n",
       "      <td>2.819770</td>\n",
       "      <td>0.140625</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>3.081503</td>\n",
       "      <td>2.808002</td>\n",
       "      <td>0.145386</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>3.047799</td>\n",
       "      <td>2.818567</td>\n",
       "      <td>0.143799</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>3.018838</td>\n",
       "      <td>2.812011</td>\n",
       "      <td>0.138672</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv-nopos, run number: 3 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.678381</td>\n",
       "      <td>3.684989</td>\n",
       "      <td>0.084229</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.645197</td>\n",
       "      <td>3.546577</td>\n",
       "      <td>0.110596</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.584275</td>\n",
       "      <td>3.353870</td>\n",
       "      <td>0.138916</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.493696</td>\n",
       "      <td>3.148067</td>\n",
       "      <td>0.153809</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.395699</td>\n",
       "      <td>3.023834</td>\n",
       "      <td>0.150391</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.305520</td>\n",
       "      <td>2.939744</td>\n",
       "      <td>0.150146</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.226833</td>\n",
       "      <td>2.883859</td>\n",
       "      <td>0.148438</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.159680</td>\n",
       "      <td>2.845464</td>\n",
       "      <td>0.150024</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.101653</td>\n",
       "      <td>2.817755</td>\n",
       "      <td>0.148560</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>3.051918</td>\n",
       "      <td>2.797186</td>\n",
       "      <td>0.153442</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>3.008421</td>\n",
       "      <td>2.792378</td>\n",
       "      <td>0.148804</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.971780</td>\n",
       "      <td>2.776556</td>\n",
       "      <td>0.150879</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.940164</td>\n",
       "      <td>2.783480</td>\n",
       "      <td>0.154297</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>2.912471</td>\n",
       "      <td>2.778267</td>\n",
       "      <td>0.153931</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.889176</td>\n",
       "      <td>2.775673</td>\n",
       "      <td>0.155884</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numbers---------- lr:0.0001 ----  attn_type: kv-nopos, run number: 4 ------------------------------\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "<style>\n",
       "    /* Turns off some styling */\n",
       "    progress {\n",
       "        /* gets rid of default border in Firefox and Opera. */\n",
       "        border: none;\n",
       "        /* Needs to be in here for Safari polyfill so background images work as expected. */\n",
       "        background-size: auto;\n",
       "    }\n",
       "    progress:not([value]), progress:not([value])::-webkit-progress-bar {\n",
       "        background: repeating-linear-gradient(45deg, #7e7e7e, #7e7e7e 10px, #5c5c5c 10px, #5c5c5c 20px);\n",
       "    }\n",
       "    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {\n",
       "        background: #F44336;\n",
       "    }\n",
       "</style>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.978632</td>\n",
       "      <td>3.960791</td>\n",
       "      <td>0.003906</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.930984</td>\n",
       "      <td>3.760406</td>\n",
       "      <td>0.005615</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.830939</td>\n",
       "      <td>3.419741</td>\n",
       "      <td>0.011841</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.694021</td>\n",
       "      <td>3.156994</td>\n",
       "      <td>0.077148</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.557338</td>\n",
       "      <td>3.015375</td>\n",
       "      <td>0.121826</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.441343</td>\n",
       "      <td>2.960960</td>\n",
       "      <td>0.140503</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.345064</td>\n",
       "      <td>2.919086</td>\n",
       "      <td>0.147827</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.265698</td>\n",
       "      <td>2.898703</td>\n",
       "      <td>0.135620</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.200489</td>\n",
       "      <td>2.887181</td>\n",
       "      <td>0.145142</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>3.146066</td>\n",
       "      <td>2.866499</td>\n",
       "      <td>0.140015</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>3.100232</td>\n",
       "      <td>2.864798</td>\n",
       "      <td>0.145020</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>3.062048</td>\n",
       "      <td>2.861408</td>\n",
       "      <td>0.138306</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>3.029479</td>\n",
       "      <td>2.863043</td>\n",
       "      <td>0.143555</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>3.002032</td>\n",
       "      <td>2.860544</td>\n",
       "      <td>0.137451</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>2.979130</td>\n",
       "      <td>2.855863</td>\n",
       "      <td>0.134155</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# update models\n",
    "n_layers = 4\n",
    "num_epochs = 15\n",
    "which_task = 'numbers'\n",
    "num_runs = 5\n",
    "\n",
    "files = os.listdir('./results_new/nlp/')\n",
    "\n",
    "for file in files:\n",
    "\n",
    "    if which_task not in file: continue\n",
    "    print(file)\n",
    "\n",
    "    n_layers, num_epochs, lr, sequence_length, _ = file.split('_')[:5]\n",
    "    n_layers, num_epochs, lr, sequence_length = int(n_layers), int(num_epochs), float(lr), int(sequence_length)\n",
    "\n",
    "    with open(f'./results_new/nlp/{n_layers}_{num_epochs}_{lr}_{sequence_length}_{which_task}.pkl', 'rb') as f:\n",
    "                    results = pickle.load(f)\n",
    "            \n",
    "    \n",
    "    sl = sequence_length\n",
    "    seqs = L((tensor(nums[i:i+sl]), tensor(nums[i+1:i+sl+1]))\n",
    "            for i in range(0,len(nums)-sl-1,sl))\n",
    "    cut = int(len(seqs) * 0.8)\n",
    "    dls = DataLoaders.from_dsets(seqs[:cut], seqs[cut:],\n",
    "                                bs=bs, drop_last=True, shuffle=True)\n",
    "    xb, yb = dls.one_batch()\n",
    "    xb.shape, yb.shape\n",
    "\n",
    "\n",
    "    for attn_type in ['kv-nopos',]:\n",
    "\n",
    "          for run_no in range(num_runs):\n",
    "          \n",
    "              print(f'{which_task}---------- lr:{lr} ----  attn_type: {attn_type}, run number: {run_no} ------------------------------')\n",
    "\n",
    "\n",
    "              learn = Learner(dls, Model7(len(vocab), n_layer=n_layers, attn_type=attn_type).to(device), loss_func=loss_func, metrics=accuracy)\n",
    "              learn.fit_one_cycle(num_epochs, lr)\n",
    "\n",
    "\n",
    "              train_losss = [x[0] for x in learn.recorder.values]\n",
    "              val_losss = [x[1] for x in learn.recorder.values]\n",
    "              accs = [x[2] for x in learn.recorder.values]\n",
    "              # times = [x[3] for x in learn.recorder.values]\n",
    "\n",
    "\n",
    "              results[(attn_type, run_no)] = {}\n",
    "              # results[(norm_type, run_no)]['time_spent'] = []  \n",
    "              results[(attn_type, run_no)]['train_loss_history'] = train_losss\n",
    "              results[(attn_type, run_no)]['val_loss_history'] = val_losss\n",
    "              results[(attn_type, run_no)]['accs_history'] = accs\n",
    "              # results[(norm_type, run_no)]['val_time_spent'] = []  \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "              if not os.path.exists(f'./results_new'):\n",
    "                os.mkdir(f'./results_new')\n",
    "\n",
    "\n",
    "              with open(f'./results_new/nlp/{n_layers}_{num_epochs}_{lr}_{sequence_length}_{which_task}.pkl', 'wb') as f:\n",
    "                pickle.dump(results, f)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "executionInfo": {
     "elapsed": 4,
     "status": "ok",
     "timestamp": 1683927708413,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "jw1yndrFEwIn"
   },
   "outputs": [],
   "source": [
    "# learn.__dict__\n",
    "# plt.plot(learn.recorder.losses)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "executionInfo": {
     "elapsed": 489,
     "status": "ok",
     "timestamp": 1683934855097,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "ZZ1fUUSmI0Xb"
   },
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "unsupported operand type(s) for +: 'NoneType' and 'NoneType'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[56], line 11\u001b[0m\n\u001b[1;32m      8\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m x,y \u001b[38;5;129;01min\u001b[39;00m results[key]\u001b[38;5;241m.\u001b[39mitems():\n\u001b[1;32m      9\u001b[0m    \u001b[38;5;66;03m# import pdb; pdb.set_trace()\u001b[39;00m\n\u001b[1;32m     10\u001b[0m    \u001b[38;5;28;01mif\u001b[39;00m x \u001b[38;5;129;01min\u001b[39;00m results_avg[key[\u001b[38;5;241m0\u001b[39m]]:\n\u001b[0;32m---> 11\u001b[0m      results_avg[key[\u001b[38;5;241m0\u001b[39m]][x] \u001b[38;5;241m+\u001b[39m\u001b[38;5;241m=\u001b[39m np\u001b[38;5;241m.\u001b[39marray(y)\n\u001b[1;32m     12\u001b[0m    \u001b[38;5;28;01melse\u001b[39;00m: \n\u001b[1;32m     13\u001b[0m      results_avg[key[\u001b[38;5;241m0\u001b[39m]][x] \u001b[38;5;241m=\u001b[39m np\u001b[38;5;241m.\u001b[39marray(y)\n",
      "\u001b[0;31mTypeError\u001b[0m: unsupported operand type(s) for +: 'NoneType' and 'NoneType'"
     ]
    }
   ],
   "source": [
    "# compute the averages \n",
    "results_avg = {}\n",
    "\n",
    "for key, _ in results.keys():\n",
    "    results_avg[key] = {}\n",
    "\n",
    "for key in results.keys():\n",
    "       for x,y in results[key].items():\n",
    "          # import pdb; pdb.set_trace()\n",
    "          if x in results_avg[key[0]]:\n",
    "            results_avg[key[0]][x] += np.array(y)\n",
    "          else: \n",
    "            results_avg[key[0]][x] = np.array(y)\n",
    "    \n",
    "\n",
    "# compute the avg\n",
    "for key,val in results_avg.items():\n",
    "      for x, y in val.items():\n",
    "          y = np.array(y)\n",
    "          results_avg[key][x] = y / num_runs\n"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "executionInfo": {
     "elapsed": 2,
     "status": "ok",
     "timestamp": 1683934855335,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "sf6Jz1LkYYb-"
   },
   "source": [
    "# results_avg.keys()\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 465
    },
    "executionInfo": {
     "elapsed": 1014,
     "status": "ok",
     "timestamp": 1683934857158,
     "user": {
      "displayName": "Ali Borji",
      "userId": "01590204968742485374"
     },
     "user_tz": 420
    },
    "id": "i9Zf6A-tOn8u",
    "outputId": "81c8fd38-ffde-46fa-e313-ddfd21c6da8a"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAABNEAAAHCCAYAAADB1l/sAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAA9hAAAPYQGoP6dpAAD/30lEQVR4nOzdd1xV9R/H8ddlyFARN6g4ssStqIm4TcCdlKEiaZmmuVJxRbnQcuX+ac4MLfdIzY27AreUO2c4AFeIiMKFe39/3LxKgMLlwrkXPs/H4z6499zv+Z43N5LL536HSqvVahFCCCGEEEIIIYQQQqTLQukAQgghhBBCCCGEEEKYOimiCSGEEEIIIYQQQgjxGlJEE0IIIYQQQgghhBDiNaSIJoQQQgghhBBCCCHEa0gRTQghhBBCCCGEEEKI15AimhBCCCGEEEIIIYQQryFFNCGEEEIIIYQQQgghXkOKaEIIIYQQQgghhBBCvIYU0YQQQgghhBBCCCGEeA0pogkhhBBCCCGEEEII8RpWSgfIaRqNhjt37lCwYEFUKpXScYQQwuxptVoeP35MqVKlsLAwrc9mxo8fT1BQUIpjrq6uXLx4Md1zYmJi+Oqrr9i0aRMPHz6kXLlyzJ49m7Zt22b4uvK7RgghjMeUf88oRX7PCCGEcWX0d02eK6LduXMHFxcXpWMIIUSuc/PmTcqUKaN0jFSqVavG3r179Y+trNL/1ZeYmIiXlxclSpRgw4YNlC5dmr///htHR8dMXVN+1wghhPGZ6u8ZJcjvGSGEyB6v+12T54poBQsWBHQvjIODQ6bPV6vV7NmzB29vb6ytrY0dz+jMLS+YX2bJm/3MLbO55YWsZY6NjcXFxUX/76upsbKywsnJKUNtly1bxsOHDwkNDdW/DuXLl8/0NbPyuyav/fwoQfJmP3PLbG55wfwy5+bfM0qQ3zOmzdzygvlllrzZz9wyZzVvRn/X5Lki2vPhzg4ODgYX0ezt7XFwcDCbHyRzygvml1nyZj9zy2xuecE4mU11Osnly5cpVaoUtra2eHh4MHnyZMqWLZtm261bt+Lh4cGAAQPYsmULxYsXp1u3bowaNQpLS8t0r5GQkEBCQoL+8ePHjwGws7PDzs4uU3mtrKywt7fHzs7ObH5+zC2z5M1+5pbZ3PKC+WXOSl61Wg2Y7u8ZJWTlb5q8+j4lJ5lbXjC/zJI3+5lbZmPlfd3vmjxXRBNCCJF3uLu7ExwcjKurK5GRkQQFBdGkSRPOnj2b5qdM165dY//+/fj7+7Njxw6uXLlC//79UavVjBs3Lt3rTJ48OdXaawB79uzB3t7eoOwhISEGnackc8ssebOfuWU2t7xgfpkNyRsfH58NSYQQQojMkyKaEEKIXKtNmzb6+zVr1sTd3Z1y5cqxbt06evXqlaq9RqOhRIkSLF68GEtLS+rWrcvt27f59ttvX1lECwwMJCAgQP/4+XBwb29vg0YIhISE4OXlZRaf+oH5ZZa82c/cMptbXjC/zFnJGxsbm02phBBCiMyRIpoQQog8w9HRkUqVKnHlypU0n3d2dsba2jrF1M0qVaoQFRVFYmIi+fLlS/M8GxsbbGxsUh23trY2+I/brJyrFHPLLHmzn7llNre8YH6ZDcmbXvv58+fz7bffEhUVRa1atfjf//5H/fr10+1n/fr1jBkzhhs3bvDWW28xderUFDsva7Vaxo0bx5IlS4iJiaFRo0YsWLCAt956S9/mm2++Yfv27YSHh5MvXz5iYmJSXCM4OJiePXumef3o6GhKlCjBwYMHadGiRarnIyMjM7yGpxBCCGVIEU0IkedptVqSkpJITk42Sn9qtRorKyuePXtmtD6z26syW1paYmVllSvWoomLi+Pq1at07949zecbNWrEqlWr0Gg0+q2t//rrL5ydndMtoAkhhMh5a9euJSAggIULF+Lu7s7s2bNp1aoVly5dokSJEqnah4aG4ufnx+TJk2nfvj2rVq3Cx8eHU6dOUb16dQCmTZvG3LlzWb58ORUqVGDMmDG0atWK8+fPY2trC+h2cfb19cXDw4Pvv/8+1XW6dOlC69atUxz7+OOPefbsWapcly5dSjFaOa3cQgghTIsU0YQQeVpiYiKRkZFGXW9Fq9Xi5OTEzZs3zabw9LrM9vb2ZllIGj58OB06dKBcuXLcuXOHcePGYWlpiZ+fHwA9evSgdOnSTJ48GYB+/foxb948Bg8ezKBBg7h8+TKTJk3i888/V/LbEEII8R8zZ87k008/1Y/6WrhwIdu3b2fZsmV88cUXqdrPmTOH1q1bM2LECAAmTpxISEgI8+bNY+HChWi1WmbPns3o0aPp2LEjACtWrKBkyZJs3ryZrl27AujXvwwODk4z1383lLl37x779+9Ps+BWokQJHB0dDX4NhBBC5Dwpogkh8iyNRsP169extLSkVKlS5MuXzyhFL41GQ1xcHAUKFNCPZjJ16WXWarUkJiZy7949rl+/zltvvWU23xPArVu38PPz48GDBxQvXpzGjRtz5MgRihcvDkBERESK78fFxYXdu3czdOhQatasSenSpRk8eDCjRo1S6lsQQgjxH4mJiZw8eZLAwED9MQsLCzw9PQkLC0vznLCwsBRrVwK0atWKzZs3A3D9+nWioqLw9PTUP1+oUCHc3d0JCwvTF9Eya8WKFdjb2/PBBx+keq527dokJCRQvXp1xo8fT6NGjdLt57+7QD9fJ06tVut3L82o5+0ze56SzC2zueUF88ssebOfuWXOat6MnidFNCFEnpWYmIhGo8HFxcXgHRTTotFoSExMxNbW1mwKTq/KbGdnh7W1NX///be+jblYs2bNK58/ePBgqmMeHh4cOXIkmxIJIYTIqvv375OcnEzJkiVTHC9ZsiQXL15M85yoqKg020dFRemff34svTaG+P777+nWrVuK0WnOzs4sXLiQevXqkZCQwNKlS2nevDlHjx6lTp06afYju0DrmFtmc8sL5pdZ8mY/c8tsaN6MzkySIpoQIs8zl0KXkuQ1EkIIITInLCyMCxcu8OOPP6Y47urqiqurq/5xw4YNuXr1KrNmzUrV9jnZBdq8MptbXjC/zJI3+5lb5qzmzehO0FJEE0IIIYQQQpiNYsWKYWlpSXR0dIrj0dHR6e5u6eTk9Mr2z79GR0fj7Oycok3t2rUNyrl06VJq165N3bp1X9u2fv36/Pbbb+k+L7tA65hbZnPLC+aXWfJmP3PLbGjejJ4jQwuEEMIMNW/enCFDhigdQwghhMhx+fLlo27duuzbt09/TKPRsG/fPjw8PNI8x8PDI0V70E35ed6+QoUKODk5pWgTGxvL0aNH0+3zVeLi4li3bh29evXKUPvw8PAUxTshhBCmSUaiCSGEEEIIIcxKQEAAH330EfXq1aN+/frMnj2bJ0+e6Hfr/O/uy4MHD6ZZs2bMmDGDdu3asWbNGk6cOMHixYsBUKlUDBkyhK+//pq33nqLChUqMGbMGEqVKoWPj4/+uhERETx8+JCIiAiSk5MJDw8H4M0336RAgQL6dmvXriUpKYkPP/wwVfbZs2dToUIFqlWrxrNnz1i6dCn79+9nz5492fRqCSGEMBYpogkhhBBCCCHMSpcuXbh37x5jx44lKiqK2rVrs2vXLv3GAP/dfblhw4asWrWK0aNH8+WXX/LWW2+xefNmqlevrm8zcuRInjx5Qp8+fYiJiaFx48bs2rUrxYY6Y8eOZfny5frHbm5uABw4cIDmzZvrj3///fe8//77ODo6psqemJjIsGHDuH37Nvb29tSsWZO9e/fSokULY708QgghsokU0QzwNPmp0hGEECKF7du3061bN+bOnUvfvn2JiopK8cZ98ODBnDlzhv379ysXUgghhDCigQMHMnDgwDSfS2v3ZV9fX3x9fdPtT6VSMWHCBCZMmJBum+DgYIKDg1+bLTQ0NN3nRo4cyciRI1/bhxBCCNMja6Jl0pPEJ3x45kNqLqrJJ1s+YdGJRfwR9QdJmiSlowkh8qhVq1bh5+fHypUr+fDDD3F0dGTjxo3655OTk1m7di3+/v4KphSZsSF6A/139Cc6Lvr1jYUQQgghhBA5QkaiZdLJyJMkk8zFBxe5+OAiP4T/AEB+6/zUK1WPBmUa0KBMA9xLu+NcUBYHFcJczQybycywmQafr9VqUalU/PT+TzQv31x//OCNg3y4Sbc+SoBHAAEeAen0kDHz58/nq6++4pdffqFZs2YAdO3alVWrVukXM963bx8xMTF06tQpS9cSOSMyLpIN0Rt4FvmMNefXENg4kKENhmJnbad0NCGEEEIIkUc8fAhHjjhRqhS8/bbSaUyHFNEySa1R85b9W1x/dj3F6LMn6icc+vsQh/4+pD9WtlBZXVGttK6w5ubshq2VbVrdCiFMTGxCLLcf385yPwlJCakeP+83NiE2S31v2LCBu3fv8vvvv/P2S7/Z/P39adCgAXfu3KFUqVKsXLmSdu3apbkuizA9f0T/gQoVAHGJcXy1/ysWnljIpJaT6FajGxYqGUQuhBBCCCGyz+HD0KyZNeDOo0fJUkR7iRTRMqllhZZ8W+lbWni14OyDsxy9dZQjt49w5NYRIh5FpGgb8SiCiEcRrDu3DgBrC2tqO9WmQZkGfOL2CbWdaivwHQghMsLBxoHSBUsbfP7zkWg2VjYpjttY2ej7dbBxyFJGNzc3Tp06xbJly6hXrx4qla7w8vbbb1OxYkXWrFlDv379+PnnnzO0foswDa0rtmZBlQWE5gtlafhSNFoNN2Nv0v3n7sw5OocZ3jNoWq6p0jGFEEIIIUQuVbs2WFho0WhUHD4sH+C+TIpoBrKztqOhS0MaujTUH4t8HMnR20c5cktXVDt+5zjx6nj982qNmuN3jnP8znG8K3qnKKLdfXKXKw+v4FHGQ/+HsBBCOVmZaqnRaIiNjcXBwSHFzmAAzcs351bALWNEpGLFisyYMYPmzZtjaWnJvHnz9M/5+/uzcuVKypQpg4WFBe3atTPKNUXOcLR2ZF6beXze4HNGhIxg55WdAJy4c4Jmwc14r/J7TPWcyltF31I4qRBCCCGEyE0uXoTSpcHfX8ujR1f5+OPySOnoBSkpGpFzQWd8KvswxXMKBz8+yKMvHhHeN5yF7Rbyce2PqVKsir6te2n3FOfuvLyTRssa8c6Kdzh++3hORxdCmKlKlSpx4MABNm7cyJAhQ/TH/f39OXXqFN988w0ffPABNjY26XciTFa1EtXY4b+DPR/uoUaJGvrjP1/8marfVWXIriE8fPpQwYRCCCGEECI38feHIkXg77+he/fztG+vVTqSSZEiWjaysrCillMt+tbryw8df+D8gPP8M+ofDnx0gOL5i6doe+TWEUC36PhfD/5SIq4Qwky5urqyf/9+Vq9ezbBhwwB48803qV+/Pn/++afsypkLeFX04nTf0yztsBSnAk4AJGmSmHN0DhXnVmRm2MxU6+8JIYQQQgiRGXfvwqlTkJQEjx6psLaWAtp/yZi8HOZo65hip77nvCp6sff6XmwsbehavWuK5zRajSwkLYRI4eDBgykeV6lShejo6BTHjh49moOJRHaztLCkV51edKnehW9//5ZvQ7/ladJTYp7FMGzPMOYfn89Uz6l0qtJJlgUQQgghhBCZplbDwIGwezd4eWmUjmOSpDJjIt6v8j7n+59nW7dtWFpYpnjOb6MfPbf05EbMDWXCCSGEMBkF8hUgqEUQfw36i49qfaTfyfPaP9fwXe9Lkx+aEPk4UuGUQgghhBDC3JQuDf/7H/z1FwQF6YpoiYkQGgqPHysczkRIEc2EWFtaU96xfIpjJ+6cYN25dQSHB1Ppf5UYuGOg/HEkhBCCMg5lCPYJ5mSfk7Qo30J//MHTB6mWDBBCCCGEECIzrK1hx47ylChhRaNGcOCA0olMgxTRTNz1f67jaOsI6Hb3nH98PhXnVmRUyCgexD9QNpwQQgjFuTm7sa/HPrZ23YprUVe+9foWK4uUqzXIemlCCCGEECKzihR5Rny8btbDoUMKhzERihbRxo8fj0qlSnGrXLlyuu2XLFlCkyZNKFy4MIULF8bT05Njx47lYOKc51vNl+uDr/NVk6/Ib50fgKdJT5kWOo035r5B0MEgYhNiFU4phBBCSSqVig6uHTjb/yzt3mqX4rnwqHDKzS7Hd8e/I0mTpFBCIYQQQghhyqKidBsKvKxq1QeULavF3x+aNlUml6lRfCRatWrViIyM1N9+++23dNsePHgQPz8/Dhw4QFhYGC4uLnh7e3P79u0cTJzzHG0d+fqdr7k2+BpDGwzFxtIGgNiEWMYfGs8bc95geuh0nqqfKpxUCCGEkqwsrFJsKqDVahm+ZzjRT6IZsGMAy04vUzCdEEIIIYQwVd26QfHi4OsL8fG6Yw4Oaq5cSeKnn6BjR2XzmQrFi2hWVlY4OTnpb8WKFUu37cqVK+nfvz+1a9emcuXKLF26FI1Gw759+3IwsXJK5C/BzFYzufL5FfrU6aOfrvPg6QNGhIyg4tyKLDi+gMTkRIWTCiGEMAXPkp5RzF73e/WNwm/wUa2PFE4khBBCCCFMTVwc/PYbxMTAyZNgZ6d0ItNl9fom2evy5cuUKlUKW1tbPDw8mDx5MmXLls3QufHx8ajVaooUKZJum4SEBBISXqwFExurm/qoVqtRq9WZzvv8HEPONZaSdiWZ13oeQ92HMvHXiaw+uxotWiLjIum/oz/Tfp/G6Caj8a/ujyZZo3jezDKF1zgzJG/2y67MarUarVaLRqNBozHeFs5arVb/1Zj9ZqfXZdZoNGi1WtRqNZaWKXcQNqefpbzGztqONR+sYUiDIcSr47GxslE6khBCCCGEMDExMfDeexASAq1awUsTG8R/KFpEc3d3Jzg4GFdXVyIjIwkKCqJJkyacPXuWggULvvb8UaNGUapUKTw9PdNtM3nyZIKCglId37NnD/b29gZnDwkJMfhcY+ps1Rl3V3dWR63myKMjANx4dIPe23ozPmQ83Zy70aBQA5PJmxnmllnyZj9jZ34+EjYuLo7EROOP4HxshvtAp5c5MTGRp0+fcvjwYZL+s1hC/PPx3sJkNSjTQOkIQgghhBDCRJUpA2vXQnKyblRaWrRauHMHSpfO2WymRtEiWps2bfT3a9asibu7O+XKlWPdunX06tXrledOmTKFNWvWcPDgQWxtbdNtFxgYSEBAgP5xbGysfi01BweHTGdWq9WEhITg5eWFtbV1ps/PLv3ox4k7Jxh3aBwh13WFhlsJt5h2YxozK82kb8e+JpX3VUz1NU6P5M1+2ZX52bNn3Lx5kwIFCrzy35HM0mq1PH78mIIFC6ZYn8qUvS7zs2fPsLOzo2nTpqleq+cjfIX5uPvkLjPDZjKxxUSsLc3j3wEhhBBCCJG9LC2hUKHUx7/6CpYv120+8M8/kIExT7mW4tM5X+bo6EilSpW4cuXKK9tNnz6dKVOmsHfvXmrWrPnKtjY2NtjYpJ6+Ym1tnaU/xrN6fnbwKOfBnh57OHTjEF/t/4rfb/7Ou5Xe5Q37N0wy7+uYW2bJm/2MnTk5ORmVSoWFhQUWFsZbIvL5dMjnfSuhfPnyDBkyhCFDhmSo/esyW1hYoFKp0vxvYG4/R3ndjss76PFzDx48fUDBfAX5qulXSkcSQgghhBAm7J9/4Pl+jqGhuimfeZXiGwu8LC4ujqtXr+Ls7Jxum2nTpjFx4kR27dpFvXr1cjCd+WhWvhm/9vyVHd12MLH5RKXjCCGEMCEl85ck5lkMABMOT+DCvQvKBhJCCCGEEIrJyMoszZuDvT14eUG+fNkeyaQpWkQbPnw4hw4d4saNG4SGhvLee+9haWmJn58fAD169CAwMFDffurUqYwZM4Zly5ZRvnx5oqKiiIqKIi69Sbt5mEqlos1bbahSrEqK4weuH2DKb1P0i4gLIYTIW+qWqsvwhsMBSExOpNfWXiRrkhVOJYQQQgghlODvDxUrQv/+kN6Szj4+us0H9uyBFi1yMp3pUbSIduvWLfz8/HB1daVz584ULVqUI0eOULx4cQAiIiKIjIzUt1+wYAGJiYl88MEHODs762/Tp09X6lswK2fvnuW9te8RuC+Qftv7kaRJev1JQgiT9OTJE3r06EGBAgVwdnZmxowZNG/ePN3pm0uXLsXR0ZF9+/axePFiSpUqlWoXzm7dur12PUqRO4xrNo63irwFQNitMOYfn69wIiGEEEIIkdPUati3D65dgw0bIH/+tNvlyweygouOomuirVmz5pXPHzx4MMXjGzduZF+YPODXv3/lUcIjAG4/vq1wGiFEVowYMYJDhw6xZcsWSpQowZdffsmpU6eoXbt2qrbTpk1j2rRp7Nmzh/r161OnTh0GDRrEgQMHaNmyJQAPHz5k3759bNu2LYe/E6EEO2s7vn/3e5oGNwUgcF8g77q+S3nH8soGE0IIIYQQOebePahbF37/XTdVU6HlnM2KvER5SL+3+7Hy/ZU0dGnImk5rsLIwqX0lhDApM2fqtnouUwb+U8/n+vUXzw0alPrcjh1VVKvmQNmyqXe5DA5+ce6mTYZli4uL4/vvv2f69Om0bNmSGjVqsHz5cpKSUo8uHTVqFLNnz+bQoUPUr18fgMKFC9OmTRtWrVqlb7dhwwaKFi1Ki7w+PjsPaVKuCf3r9QcgXh3Pp798KlP9hRBCCCHykFKl4MABePAApk3L+Hlp/NmRZ0gRLY/pVqMbv/b8lfz5Uo7T1Gg16ZwhRN4UG6vbgeb2bUhISPlccvKL5/75J/W59+/DnTsW3L6duoj25MmLczOyiGdarl69SmJiIu7u7vpjRYoUwdXVNUW7GTNmsGTJEn777TeqVauW4jl/f382btxIwr/f3OrVq3n//fcV201UKGOy52RcHFwA2HttL8HhwcoGEkIIIYQQOa5gQShd+tVtkpJg2DDdyDUvr5zJZYrkr6U8yEKV8j/7zUc3cVvkxu8RvyuUSAjT4+Cg+0VSujTY2KR8ztLyxXOFC6c+t1gxKFVKQ+nSqUf15M//4lx7+2wK/68mTZqQnJzMunXrUj3XoUMHtFot27dv5+bNm/z666/4+vpmbyBhchxsHFjUfpH+ccCeACIfR77iDCGEEEIIkRdZWcGWLXDqFISGwtOnSidShhTR8riYZzG0XdWWP6P/pOWKlmw4v0HpSEKYhIAAuHVLd2vePOVzFSq8eO5//0t97pYtWs6diyUiInUR7eOPX5z7/vuGZatYsSLW1tYcPXpUf+yff/7hr7/+StGufv367Ny5k0mTJqXagMXW1pb333+flStXsnr1alxdXalVq5ZhgYRZa/NWG7rX7A7ofif039FfpnUKIYQQQuRyGgMmozVrpvtauTLcuWPcPOZCimh5nIXKAucCzgAkJCfQeX1nZoXNUjiVEOJVChQoQK9evRgxYgT79+/n7NmzfPzxx2lOxWzYsCE7duwgKCiI2bNnp3jO39+f7du3s2zZMrp165ZD6XPW+PHjUalUKW6VK1fO0Llr1qxBpVLh4+OTvSFNwKxWsyiRvwQAmy9uZuOFjQonEkIIIYQQ2al3b2jQAMaNg0ePMnZOUJBu/bQ//oCKFbM3n6mSIloe52DjwPZu2+lZuycAWrQE7AlgyK4hJGuSFU4nhEjPt99+S5MmTejQoQOenp40btyYunXrptm2cePGbN++ndGjR/O/l4bOvfPOOxQpUoRLly7h5+eXU9FzXLVq1YiMjNTffvvtt9eec+PGDYYPH06TJk1yIKHyitoXZV6befrHA3YM4EH8AwUTCSGEEEKI7KLVws6dcPQoTJ8OtrYZO69MGShSJHuzmTrZnlFgbWnN9+9+T7lC5Rh/aDwAc47OIeJRBCvfX4mdtZ2yAYUQqRQoUIAff/yRH3/8UX9s+/bt+vs3btxI0b5p06bExcWlOGZhYcGdf8dhazQaYmNjsy+wgqysrHBycspw++TkZPz9/QkKCuLXX38lJiYm+8KZkA+qfoBPZR82X9zM3Sd3CdgTwHKf5UrHEkIIIYQQRnb3LhQtClFR0KJF6jWgRfqkiCYAUKlUjGs+DpdCLvT5pQ/J2mR+vvgzLVe0ZKvfVorZF1M6ohBCGOTy5cuUKlUKW1tbPDw8mDx5MmXLlk23/YQJEyhRogS9evXi119/zdA1EhIS9DudAvqCpFqtRq1WZyrv8/aZPc8Y5njP4cD1AzxKeMSKP1bgW9mXVhVbvfY8JTMbQvJmP3PLbG55wfwyZyWvuXyPQghhLkqWhLNndes059LP0bONFNFECp+4fULpgqX5YP0HxCXGEXYrjIbfN2Sn/04qFsmjk56FEGbL3d2d4OBgXF1diYyMJCgoiCZNmnD27FkKFiyYqv1vv/3G999/T3h4eKauM3nyZIKCglId37NnD/YGbsMaEhJi0HlZ1b1Ed+bd1E3t/GTTJ8ytPBc7y4yNSFYqs6Ekb/Yzt8zmlhfML7MheePj47MhiRBCiDJlMn9OdDQEB8PBg+DtDUOHGjuVaZMimkil1ZutOPzxYdqtakdkXCSXH17G43sPtnXbRv3S9ZWOJ4RIx8GDB5WOYHLatGmjv1+zZk3c3d0pV64c69ato1evXinaPn78mO7du7NkyRKKFcvc6NvAwEACAgL0j2NjY3FxccHb2xsHB4dM9aVWqwkJCcHLywtra+tMnWsMbbRtuLD6Avtu7OPtcm/T5J0m+k0H0qN05sySvNnP3DKbW14wv8xZyZtblxsQQghz9OQJfPGF7n5SkhTRhADAzdmNI72P0GZlG87fO8+9+Hs0D27Omg/W8K7ru0rHE0IIgzg6OlKpUiWuXLmS6rmrV69y48YNOnTooD+m+XfvbysrKy5dukTFdLYhsrGxwSaNxSSsra0N+uP26VNL4uOtKVZMmT+Ml7y7hLBbYfhV90OlUmX4PEO/X6VI3uxnbpnNLS+YX2ZD8prT9yeEELldhQq6EWy3bsG1a6DRgEUe2rIyD32rIrPKFirLbz1/o1m5ZgA8TXrKe2vf47vj3ymcTAghDBMXF8fVq1dxdnZO9VzlypU5c+YM4eHh+tu7775LixYtCA8Px8XFJUcyajQwZ04dGja04uLFHLlkKhUKV6BbjW6ZKqAJIYQQQgjT9+WX8O67MH++YeuhqVSwfDlcvAhXruStAhpIEU28RmG7wuz+cDd+1f0A0Gg1DNgxgFEho9BoNQqnE8I4tFqt0hFMnrm+RsOHD+fQoUPcuHGD0NBQ3nvvPSwtLfHz0/2b1qNHDwIDAwGwtbWlevXqKW6Ojo4ULFiQ6tWrky9fvhzJPHOmBUeOlOLyZRXe3pCYmCOXfS11sizsLYQQQghh7jZuhF9+gcGDwdC3+O+8A66uuoJaXiNFNPFaNlY2/PT+T3zR6Av9sWmh0wjYHfCKs4Qwfc+nh8iCxa/3/DUytyk1t27dws/PD1dXVzp37kzRokU5cuQIxYsXByAiIoLIyEiFU6bUqZOG8uUfoVJpWbgQcqh290q/XPqFt/73FqciTykdRQghhBBCGOiffyAmRnffwwMKFVI0jlmSNdFEhlioLJjsOZmyhcoycOdACuQrwCdunygdS4gssbS0xNHRkbt37wJgb29vlOlrGo2GxMREnj17hoWZjG9OL7NWqyU+Pp67d+/i6OiIpaWlgikzb82aNa98/nWbMQQHBxsvTAZVqABTpvyKlVVr2rZV/tf0pgub6LSuEwCfbPmE458ex9rSvIqpQgghhBACCheGyEj44w94+lTpNOZJ+Xfnwqz0e7sfZRzKYGdtR82SNZWOI0SWOTk5AegLacag1Wp5+vQpdnZ2ZrOm1OsyOzo66l8rkf1sbZNp2zb1+PoVK6BhQ3jzzZzL0qFSB2qWrMmf0X/iXNCZ2IRYitoXzbkAQgghhBDCaCwswM0t6/3cuQN79sChQ7qpobVrZ71PcyBFNJFpHVw7vL6REGZCpVLh7OxMiRIlUKuNs+aTWq3m8OHDNG3a1GymP74qs7W1tdmNQMuNQkKgZ09wcIA1a6BVq5y5rrWlNcveXcaF+xfwr+FvNoVhIYQQQgiRfbZuhX79dPerVJEimhAZptVqGXtgLNVLVKdL9S5KxxHCIJaWlkYrFFlaWpKUlIStra3ZFNHMMXNeotVCUJBu586YGPj995wrogHULVWXuqXq5twFhRBCCCGESWve/MX9339XLEaOM4/FeoTJUier6f5zd77+9Wt6bO7BbxG/KR1JCCFyHZUKduzQbUfevj2MH690ImSHZiGE4ubPn0/58uWxtbXF3d2dY8eOvbL9+vXrqVy5Mra2ttSoUYMdO3akeF6r1TJ27FicnZ2xs7PD09OTy5cvp2jzzTff0LBhQ+zt7XF0dEzzOiqVKtXtv2t0Hjx4kDp16mBjY8Obb76pyBqcQoi8Zc4c6N0b1q8HY+yr5uoKU6fC4cOwbl3W+zMXUkQTWWJlYYWNpQ2gK6idv3de4URCCJE7OTjAzz/D2rW6tSxepsnhetaOyzuo9l01rv9zPWcvLIQQ/1q7di0BAQGMGzeOU6dOUatWLVq1apXuGqehoaH4+fnRq1cvTp8+jY+PDz4+Ppw9e1bfZtq0acydO5eFCxdy9OhR8ufPT6tWrXj27Jm+TWJiIr6+vvR7PocpHT/88AORkZH6m4+Pj/6569ev065dO1q0aEF4eDhDhgyhd+/e7N69O2svihBCvMJPP8H330PnzvDoUdb7U6lg5Eho0gRsbLLen7mQIprIEpVKxcL2C/Gp7MOmLpvoU7eP0pGEECLXsrAAe/uUx27ehJo14TUbjRpNcHgw7Va14+L9i/TZ1getNvUGCEIIkd1mzpzJp59+Ss+ePalatSoLFy7E3t6eZcuWpdl+zpw5tG7dmhEjRlClShUmTpxInTp1mDdvHqAbhTZ79mxGjx5Nx44dqVmzJitWrODOnTts3rxZ309QUBBDhw6lRo0ar8z3fEOe5zdbW1v9cwsXLqRChQrMmDGDKlWqMHDgQD744ANmzZqV9RdGCCHSEBcH5/8d71KzJjg7K5vHnMmaaCLLrC2t+bnLz0rHEEKIPOfpU/DxgXPnwNMTNmzQPc5O71d5n7EHxnIz9iZ7r+0lODyYD6t/mL0XFUKIlyQmJnLy5EkCAwP1xywsLPD09CQsLCzNc8LCwggICEhxrFWrVvoC2fXr14mKisLT01P/fKFChXB3dycsLIyuXbtmKuOAAQPo3bs3b7zxBp999hk9e/bUb8wSFhaW4jrPswwZMiTd/hISEkhISNA/jo2NBXQbA2V2Y6Tn7Y21oVJOMLfM5pYXzC+z5M0cGxuIjITfflORmAhq9es/BFU6c2ZlNW9Gz5Mimsg2R28dpV6pelhayK5+QgiRHRIToUQJ3f1y5aBp0+y/poONAwvbL6TdqnYABOwJoGX5ltl/YSGE+Nf9+/dJTk6mZMmSKY6XLFmSixcvpnlOVFRUmu2joqL0zz8/ll6bjJowYQLvvPMO9vb27Nmzh/79+xMXF8fnn3/+yiyxsbE8ffoUOzu7VH1OnjyZoKCgVMf37NmD/X+HKGdQSEiIQecpydwym1teML/Mkjfznq+1m1Gvyxwfb8X580W4dKkIfn4XUy07ktMMfY3jM7hQnBTRRLaYf2w+n+/6nMHug5nZaqbScYQQIlcqVAi2bYNx46BLFyhSJGeu2/attnxY80N++vMnYp7FMGjXIHra9cyZiwshhIkbM2aM/r6bmxtPnjzh22+/1RfRDBEYGJhiJF1sbCwuLi54e3vj4OCQqb7UajUhISF4eXmZzY7c5pbZ3PKC+WWWvNkvo5m7dLHk5591lbORI9+gVq2cSphSVl/j5yN8X0eKaMLoLt2/xOBdg9FoNcw6MosKjhUY5D5I6VhCCJErWVrC11+nPp6QoFv7ws0te647u9Vsdl/Zzb34e2z9ayuu5V1pR7vsuZgQQrykWLFiWFpaEh0dneJ4dHQ0Tk5OaZ7j5OT0yvbPv0ZHR+P80mJB0dHR1K5dO0t53d3dmThxIgkJCdjY2KSbxcHBIc1RaAA2NjbYpLFyt7W1tcF/kGflXKWYW2Zzywvml1nyZr/XZW7eXLf5FUBoqDX16uVMrvQY+hpn9BzZWEAYnWsxVxa0W6B/PHjXYLZc3KJgIiGEyHs+/xzc3WHp0uzpv6h9Uea1nad/vPTWUuLVRtgvXQghXiNfvnzUrVuXffv26Y9pNBr27duHh4dHmud4eHikaA+6KT/P21eoUAEnJ6cUbWJjYzl69Gi6fWZUeHg4hQsX1hfBXpdFCCGMaeVKCAzUbUKVmGj8/r28YMAAWL8eunUzfv+mRopoIlt8WvdTAhvrFnvVosVvox/Hbh9TOJUQQuQNP/8MixeDWg2DBsHff2fPdXyr+tK+UnsA/kn6h3nH573mDCGEMI6AgACWLFnC8uXLuXDhAv369ePJkyf07KmbWt6jR48UGw8MHjyYXbt2MWPGDC5evMj48eM5ceIEAwcOBHQ7zg8ZMoSvv/6arVu3cubMGXr06EGpUqXweWnHloiICMLDw4mIiCA5OZnw8HDCw8OJi4sD4JdffmHp0qWcPXuWK1eusGDBAiZNmsSgQS9mZXz22Wdcu3aNkSNHcvHiRb777jvWrVvH0KFDc+CVE0LkNT/8AFOmQIsWcP268fuvUgXmzYMPPoBixYzfv6mRIprINl+/8zV+1f0AeJr0lA6rO3D9n2z4v1YIIUQK7dvD4MG6+0uW6DYdyA4qlYqpnlOxUOneTkw/Mp1/nv6TPRcTQoiXdOnShenTpzN27Fhq165NeHg4u3bt0i/YHxERQWRkpL59w4YNWbVqFYsXL6ZWrVps2LCBzZs3U716dX2bkSNHMmjQIPr06cPbb79NXFwcu3btwtbWVt9m7NixuLm5MW7cOOLi4nBzc8PNzY0TJ04AuulA8+fPx8PDg9q1a7No0SJmzpzJuHHj9H1UqFCB7du3ExISQq1atZgxYwZLly6lVatW2f2yCSHymGfP4PffdffLlYNKlZTNkxvImmgi21ioLPih4w/cfnybw38f5u6Tu7Rd1ZbQT0IpbFdY6XhCCJFrWVvD7Nnw4Ydk+7oUVYtXpXuN7iz/czkxz2KY8tsUpnpNzd6LCiEEMHDgQP1Isv86ePBgqmO+vr74+vqm259KpWLChAlMmDAh3TbBwcEEBwen+3zr1q1p3bp1us8/17x5c06fPv3adkIIkRW2tnDjBoSE6GYoqFRKJzJ/MhJNZCsbKxt+7vIzrkVdAbh4/yLvrX2PhKQEhZMJIUTul1YB7dYt+HfWkdGMbToWa5VuMda5x+ZyK/aWcS8ghBBCCCEMUrKk7oPVntm8kXpyMpw8Cb/8kr3XUZoU0US2K2JXhJ3+OymRvwQAh/4+xCdbP0Gj1SicTAgh8pY1a6BGDRg+3Lj9uji40LZYWwCeJT1jwqH0R3EIIYQQQojcRauFN97QfYDbu7fucW4lRTSRIyoUrsAvfr9gZ6XbtnvVmVWM2T9G4VRCCJF3REVBr14QEwOLFsHOncbtv1PJTjjYOACw7PQyLt2/ZNwLCCGEEEIIk6RSwfMlJu/ehYsXlc2TnaSIJnJM/dL1Wd1pNSp0E7En/TaJJSeXKJxKCCHyBicnmDVLd79rV2jQwLj9O1g5MKzBMACStcl8tf8r415ACCGEEEJk2M6duvd+58/nzMiwjh3B11e3U2du3qVTimgiR3Ws3JHZrWfrH/fb3o9dV3YpF0gIIfKQTz+Ffftg9WoonA37u3z+9uc4FXACYOOFjRy7fcz4FxFCCCGEEK+1dCkEBEC1anDqVPZfr08fWLcOBgyA4sWz/3pKkSKayHGfu3/OEPchgG60gu96X8KjwhXNJIQQeYFKBe+8k33958+Xn7FNxwJQMF9BLj+4nH0XE0IIIYQQaUpK0n1wClC0KNSurWicXMVK6QAib5ruPZ2/H/3Nzxd/Ji4xjnar2nG091HKOJRROpoQQuQpz57BP/+As7Nx+utdpzd3n9yl/9v9KZ4/F38MKYQQQghhoiwt4fffYfduSEzUPRbGIUU0oQhLC0t+ev8n3ln+DkdvH6Wuc10K22bD3CIhhBDp+uMP8PcHBwc4fBisjPCuwNrSmnHNx2W9IyGEEEIIYRCVSjeNs1o1Za5/8yYkJMCbbypz/ewk0zkNEBubT+kIuYK9tT1b/bYS1DyIn7v8TP58+ZWOJIQQeYZGA926wblzEBYGU6cqnUgIIYQQQpiza9egYkUoWxbGj1c6TfaQIlomxcXBxx+3plIlK76SjceyrET+EoxtNhZLCxlfKoQQOcnCQrfgrIUF1Kql21EpOzx8+pBvDn9DYnJi9lxACCGEEEKYhDJlICpKd//QoZzZFTSnyXTOTDp+XIVGo+LGDXjwIPXzK1bodqJo0CB7dj7LCyIeRRDyIIS2tFU6ihBC5GoeHrrtz5s1Axsb4/e/9uxa+m7ry6OERxS2K0z/t/sb/yJCCCGEEELvyBG4cgW8vaFEiZy9dr58uk2sYmN17y/Vat2x3ESKaAaoVu0+V68WpWFDVYrjGg0MHgwxMVCyJERG6uYii4w7FXmKdivbEfUkitZXW9O+cnulIwkhRK7m7Z19fVcoXIFHCY8AmPb7NPrW7Ssjj4UQQgghstGiRRAcrLsfGqr70DQnbd2au+sgMp0zk1q00PLNN79z/34SnTunfO7iRV0BDXQj0f77gzNsGLRvD5Mm6SqzIrWQqyFEPdGN//zywJdotBqFEwkhRN6i1cKdO8bpq37p+nSu1plPan/Crz1/lQKaEEIIIUQ20mphzx7dfTs7cHPL+Qy5uYAGMhLNYPnygbV1ymOlS8Patbpqb716qc/ZsUNXaNuzB4YOTfncnTu6dWmcnLIvszkY0WgEh/8+zO2o22z3246FSuq8QgiRUx48gL59dVuinzkDxYplvc/VnVbLv+VCCCGEEDlAq4WVK2H3bnj6FGxtlU6U+0gRzYgKFYLOnUk1Qg0gPh7u39fdr1NHVxV+2fz5MG0afPwxjB0LLi7ZHtckWags+MnnJw6EHKBE/hyewC2EEHlcQABs3Ki737cvbNiQ9U8TpYAmhBBCCJEzLCygeXPdzRQ8eaJ7L2lvr3QS45F3tjnE3h7u3oVLl2Du3NTPh4ZCUpJup7Rbt3I+nykpkK8AliqZ8iOEEDltyhQoWhSKFIFu3bJnOH5cYhz34+8bv2MhhBBCCGES9u3TrcXm6Kj7UDY3kSJaDlKpoFIlqF8/9XPNmoGDA7Rpk3rhv9y4LWxmPIh/gN9GP67/c13pKEIIkas5O8OmTbqpnJ06GbfvxORE5h2bR8W5FRkRMsK4nQshhBBCCJNhaanbJTQpCQ4dUjqNcSlaRBs/fjwqlSrFrXLlyq88Z/369VSuXBlbW1tq1KjBjh07ciht9ho/Hm7cgO++S/3cJ5/AkCEQFZXDoUzAubvncFvkxpqza/Bd70tCUoLSkYQQIldr2hRKlTJ+v08SnzDmwBjuPrnLij9WcO7uOeNfRAghhBAijzp/Xrf++tOnSicBd3ewsYHKlaFcOaXTGJfiI9GqVatGZGSk/vbbb7+l2zY0NBQ/Pz969erF6dOn8fHxwcfHh7Nnz+Zg4uxTuDCUL5/y2NmzsHw5zJmj2/EzOVmRaIpxKeSCjZUNACcjTzJszzCFEwkhzElmP6xZsmQJTZo0oXDhwhQuXBhPT0+OHTuWg4lN030jzL4sbFeYUY1GAaDRavhq/1dZ71QIIYQQQgCweDG0aqVbluPXX5XNYmcHkZFw4YJuzffcRPEimpWVFU5OTvpbsVdsBTZnzhxat27NiBEjqFKlChMnTqROnTrMmzcvBxPnrD/+eLGjxuef64ZF5iUONg5s8N2AjaWukDb/+HzWnl2rcCohhDnJzIc1Bw8exM/PjwMHDhAWFoaLiwve3t7cvn07BxObjqQk+Ppr3SeIp09nvb/P3T+nVEHdMLctl7YQejM0650KIYQQQgh279Z9VauhRg1ls4BukFBupPjunJcvX6ZUqVLY2tri4eHB5MmTKVu2bJptw8LCCAgISHGsVatWbN68Od3+ExISSEh4MQUwNjYWALVajVqtznTe5+cYcq4hOnfWTa2ZP9+CXr00vHzZZ89g5kwLPvtMQ5EiaZ+f03mN4b+ZqxatypxWc/hsx2cA9P6lN9WKVcO1qKtiGV9mbq+xueUF88tsbnkha5lN/ft8/mFNRqxcuTLF46VLl7Jx40b27dtHjx49siOeSVu0CMaM0d3/8EM4cSL17tKZYW9tz7hm4+i7rS8AX+z9gkMfH0KVHTsYCCGEEELkEVotTJigK6TFxuoW9BfZQ9Eimru7O8HBwbi6uhIZGUlQUBBNmjTh7NmzFCxYMFX7qKgoSpYsmeJYyZIliXrFYmGTJ08mKCgo1fE9e/Zgn4V9VkNCQgw+1xAeHnDwYMpj27a9wdKlNZg2TUP//uE0bnwn3fNzOq8xvJy5pLYkzQs35+A/B4lLjKP98vZMqzQNGwsb5QL+h7m9xuaWF8wvs7nlBcMyx8fHZ0MS48nMhzX/FR8fj1qtpkh6n1T8y5gf2JhSEfbjj2HpUiv+/BN8fDRotSk/zHkuM5m7V+/O9NDpXH54mV8jfuWXi7/Q5s02Rk7+aqb0GmeEueUF88tsbnnB/DLn5g9rhBBCaSoV+PrqbqZIowELxedBGoeiRbQ2bV68aa5Zsybu7u6UK1eOdevW0atXL6NcIzAwMMXotdjYWP30HAcHh0z3p1arCQkJwcvLC2tra6NkNERyMnz+ue4/X3y8NV261KZWrdqp2plK3sxIL3PzxOY0DG7IhfsX+PvZ3+xU7WRx28UKJtUxt9fY3PKC+WU2t7yQtczPC0amKLMf1vzXqFGjKFWqFJ6enq9slx0f2JhKEbZXrwI8eWJN5cr/8LpIGc38nsN7THs4DYDBWweT7JqMhSrn31mZymucUeaWF8wvs7nlBfPLnBs/rBFCCJG29eth5UrdGm3nzkEGJ4eYNMWnc77M0dGRSpUqceXKlTSfd3JyIjo6OsWx6OjoV07TsbGxwcYm9Wgla2vrLP1xm9Xzs8raGg4fhm++gbg4qFcvZZZbt3RDOJ9/60rnNcR/MztaO7Kx80bqLalHvDqe4D+CaVa+GR/X/li5kC8xt9fY3PKC+WU2t7xgWGZT/h6z8mHNlClTWLNmDQcPHsT2+eKU6TDmBzZ5oQjbRtuGfcH7OBl5khvPbvCo3CP8q/vnQFIdc3uNzS0vmF9mc8sL5pc5t35YI4QQIn2nTsGWLbr7hw5Bly7K5jEGkyqixcXFcfXqVbp3757m8x4eHuzbt48hQ4boj4WEhODh4ZFDCU1LuXK6HTi02tTP9esHR47AsGEWVKyYS8ZNAlWKV2FR+0V0/1n3M9J/e3/qOtelRkkTWDlRCGHyXvdhzXPTp09nypQp7N27l5o1a7623+z4wMaUi7CPHoGDg27qwMsyk3mq51Q8f9SN8As6HES3mt3IZ5nP2FFfyZRf47SYW14wv8zmlhfML3Nu+7BGCCGUFhEBDx9CzZqmN2WyeXOYMkW3Y2hMjNJpjEPRl3j48OEcOnSIGzduEBoaynvvvYelpSV+fn4A9OjRg8DAQH37wYMHs2vXLmbMmMHFixcZP348J06cYODAgUp9Cybhv3/EHDsG27bB/fvw3XcWqZ43dx/W/JA+dfoA8DTpKb7rfXmc8FjhVEIIc/D8wxpnZ+d020ybNo2JEyeya9cu6tWrl4PpzMMvv8Bbb8GqVVnrp+UbLfF6wwuAGzE3WHRikRHSCSGEEELkLUuXgpsblCqlG+1lSpo2hT//hHv3oG9fpdMYh6JFtFu3buHn54erqyudO3emaNGiHDlyhOLFiwMQERFBZGSkvn3Dhg1ZtWoVixcvplatWmzYsIHNmzdTvXp1pb4Fk1SsGPj764prX3yhwdpao3Qko5vTZg61nWoDcOnBJfps64M2rSF5Qog8LbMf1kydOpUxY8awbNkyypcvT1RUFFFRUcTFxSn1LZiUEyfg3Xd1b4QGDNB98pkVUzyn6O9PPDxRPhARQgghhMik3bt1X6OjoWJFZbP8l50d1KhheiPkskLR6Zxr1qx55fMH/7sdJeDr64uvqW45YSLeeAN++gm++grKltWwd++L57Ra3aYEViY1kTfzbK1sWe+7nrqL6xKbEMuas2toWrYp/d7up3Q0IYQJef5hzYMHDyhevDiNGzdO9WGNxUu/1RcsWEBiYiIffPBBin7GjRvH+PHjczK6SapXDz78UPc7pmlTeM1Sca9Vx7kOXap1Ye25tdyLv8fMsJmMaz7OOGGFEEIIIfKAnj2hZEndh5xlyiidJvcz81KKeJUqVeC/O4LPmaObirN+vW5esjl7s8ib/NDxBzqt6wTAkN1DqF+6PnVL1VU4mRDCVGT2w5obN25kX5hc4n//Ay8v6N499XIChpjYYiIbL2wkSZPE9LDp9H+7P8XzF896x0IIIYQQecBnn+luMjErZ+SiQXXidXbsgGHDYP9+8PDQ7epp7t6v8j5D3IcAkJiciO96X2KexSiaSQghcjNHR+jRwzgFNIC3ir5Fb7feAMQlxvHNr98Yp2MhhBBCiDzElNdCX7MG+vQBHx+lk2SdFNHyEEdHKFpUd9/XFwoUUDSO0Uz1mop7aXdA9wfY1YdXFU4khBB5i1oNarXhbynGNhuLvbU9AAtOLOBGzA0jJRNCCCGEEEqbPh2WLIEtW+DBA6XTZI0U0fKQhg11O3eOGgUTJiidxnjyWeZjne86Orp2JPyzcJnOKYQQOejiRWjSxJKVKysb3IdzQWf9qOJi9sW4/s91I6UTQgghhMidduyA8HClU2RMs2a6r9bWcOaMslmyStZEy2PKl4cpU1Ifv3YNypUDS8scj2QUZQuVZXPXzUrHEEKIPCU2Vrc8QEyMBX/8UZHr15OpVMmwvkY2GkkRuyL0f7s/dtZ2xg0qhBBCCJGLqNXQty/cugWenrB9O+TLp3Sq9PXuDW3b6t432tsrnSZrZCSaICJCN0rt3Xd1fxDlFlqtlieJT5SOIYQQuZaDAwwcqLtfpMgzbt0yfDGOQraFGNZwmBTQhBBCCCFeY+NGXQENwM7OtAtooNv0sGVL8y+ggRTR8jyNBjp1guho3XDQESOUTmQcMc9ieG/te3Ra1wmNVqN0HCGEyLWGDYN585L57ru9NGki20IJIYQQQmS3Dh3gu+/grbcgIEDpNMq4fx9+/BH694etW3PuulJEy+MsLGDaNChcGN58EyZNUjpR1mm1WtqubMuWS1vYfXU3k3+drHQkIYTItRwdoU8fDdbWxi2gXf/nOhMPTUQr+7ULIYQQQqSQPz/066dbm/b5emO52dOnkJiY8tiFC7od4xcskCKayGEtWsDRo7Bt24vdO82ZSqXi63e+RoWKwraFqe1UW+lIQgghMmF66HRc57ky9uBYtv21Tek4QgghhBAmycICVIavppGjEhNh714YMwYWLszYOZs3g7s7FCqkO/dl9eqB1b+r/B89atSoryRFNAHohoG6uqY8lpio24ZWY4azId+p8A4/dPyB031P065SO6XjCCFEnvHoEaxdm7U+KhauiFqjBuCbX78xQiohhBBCCKGk2Fjw8oKvv4bvv39xPDERjh2DuXMhKSnlOc+fU6shNDTlc3Z2uimte/emfi47ye6cIk1arW5u8fffw+7dsHy5bsioOfmo9kdKRxBCiDxlyRILRo+Gf/6BSpXAzc2wfnwq+9D2rba8XeptBtUfZNyQQgghhBBm6ptvwNoaPv1UtySTOSlWDKpXh7Nn4dQpePwYChaETz6BlSt1bZo0Sfn+sWFD3dcqVdL+fj/99MV9tTr7sr9MimgiTeHh8MMPuvvbtsGlS1CnjqKRskyr1XLs9jHcy7grHUUIIXIltVpXQAMYPx62bDGsH5VKxfZu242WSwghhBDC3MXEwOTJ8OQJzJgBERFgY6N0qsz56ivdTLdmzXQFNID69V8U0UJDUxbRypSBhw9Nq2Ao0zlFmtzcdMUzBwcIDjb/AtrjhMd029SNBt83YM/VPUrHEUKIXKlXLw3ly+sWeZ05U+k0Qojcbv78+ZQvXx5bW1vc3d05duzYK9uvX7+eypUrY2trS40aNdixY0eK57VaLWPHjsXZ2Rk7Ozs8PT25fPlyijbffPMNDRs2xN7eHkdHx1TX+OOPP/Dz88PFxQU7OzuqVKnCnDlzUrQ5ePAgKpUq1S0qKsqwF0IIkSfs2wfx8br7nTqZXwENoGtX6NYNSpd+caxFC/D3h/nzoU2b1OeYUgENpIgmXqFNG7h2TfeDbu5Wn13NmrNrAOjxcw/uPrmrcCIhhMh9bGzg3DndEgAVKxq378cJj43boRDCrK1du5aAgADGjRvHqVOnqFWrFq1ateLu3bTf44WGhuLn50evXr04ffo0Pj4++Pj4cPbsWX2badOmMXfuXBYuXMjRo0fJnz8/rVq14tmzZ/o2iYmJ+Pr60q9fvzSvc/LkSUqUKMFPP/3EuXPn+OqrrwgMDGTevHmp2l66dInIyEj9rUSJEll8VYQQuVmnTroZYgMGwODBSqcxnho14KefdMtJvfGG0mleT4po4pXS2q3zp590iwFqtTmfx1C96/Sm9ZutAYh+Es1Hmz9CozXDHROEEMLE2dsbt7/ouGi+2PsFpWeW5veI343buRDCbM2cOZNPP/2Unj17UrVqVRYuXIi9vT3Lli1Ls/2cOXNo3bo1I0aMoEqVKkycOJE6deroi1tarZbZs2czevRoOnbsSM2aNVmxYgV37txh8+bN+n6CgoIYOnQoNWrUSPM6n3zyCXPmzKFZs2a88cYbfPjhh/Ts2ZNNmzalaluiRAmcnJz0NwsL+dNMCPFqb70F8+al3hRQ5BxZE01kSmgo9Oql2yXjwgXdaAMrM/gpslBZENwxmFoLaxH9JJpdV3Yx58gchnoMVTqaEELkas8/cDF0+/WdV3Yy9fepAEz+bTLbum0zUjIhhLlKTEzk5MmTBAYG6o9ZWFjg6elJWFhYmueEhYUREBCQ4lirVq30BbLr168TFRWFp6en/vlChQrh7u5OWFgYXbMwNePRo0cUKVIk1fHatWuTkJBA9erVGT9+PI0aNUq3j4SEBBISEvSPY2NjAVCr1agzuZr28/aZPU9J5pbZ3PKC+WWWvNnP3DJnNW9GzzOD8ocwJadO6QpooFsI0NJS2TyZUbJASVa8t4JWP7UCYNTeUTQr34w6zma+4JsQQpiogwdhzBjdzdvbsD78a/gz7uA4Ih5FsP3ydv6I+oNaTrWMmlMIYV7u379PcnIyJUuWTHG8ZMmSXLx4Mc1zoqKi0mz/fB2y519f1cYQoaGhrF27lu3bX2yW4uzszMKFC6lXrx4JCQksXbqU5s2bc/ToUeqksxDx5MmTCQoKSnV8z5492Bs4BDgkJMSg85RkbpnNLS+YX+a8kFettsDaWrlZVHnhNQaIf77g3GtIEU1kysCBukUAg4Phf/8zfGSBUrwrejOi4Qi+Df0WtUZN1w1dOdX3FAXyFVA6mhBC5Cr79sHzAR2jR4OXl2G/M6wtrRnRcASDdg4CYMrvU1jdabURkwohRPY4e/YsHTt2ZNy4cXi/9EmCq6srri/NxWrYsCFXr15l1qxZ/Pjjj2n2FRgYmGIkXWxsLC4uLnh7e+Pg4JCpXGq1mpCQELy8vLC2ts7kd6UMc8tsbnnB/DLnpbydO1sSEwODB2to00ZLTs38zkuvMbwY4fs6UkQTmfbee+DjY34FtOe+fudrDtw4wIk7J7j88DKDdg7ih44/KB1LCCFylRYtdAvFnjkDjx9DdDQ4ORnWVy+3Xkw8PJG7T+6y7tw6JjSfwFtF3zJuYCGE2ShWrBiWlpZER0enOB4dHY1TOv/QODk5vbL986/R0dE4OzunaFO7du1MZzx//jwtW7akT58+jB49+rXt69evz2+//Zbu8zY2NtiksRWftbW1wX/cZuVcpZhbZnPLC+aXObfnvXYNtm4FjQYuXbLgxg3I6W83t7/GL5+XEbJ6pTDIfwtojx/D+vXKZMmsfJb5WN1ptX70WXB4MKvOrFI4lRBC5C4WFjBzJqxYAWfPGl5AA7CztmOI+xAANFoN036fZpyQQgizlC9fPurWrcu+ffv0xzQaDfv27cPDwyPNczw8PFK0B92Un+ftK1SogJOTU4o2sbGxHD16NN0+03Pu3DlatGjBRx99xDfffJOhc8LDw1MU74QQAuDOnRc7ng8cCPnyKZtHSBFNGEF0NDRvDp07Qzoj0E3Om0XeZEG7BfrHn237jGv/XFMwkRBC5D6entC9u3HWz+z/dn8cbHRTlpb/sZzbsbez3qkQwmwFBASwZMkSli9fzoULF+jXrx9PnjyhZ8+eAPTo0SPFxgODBw9m165dzJgxg4sXLzJ+/HhOnDjBwIEDAVCpVAwZMoSvv/6arVu3cubMGXr06EGpUqXw8fHR9xMREUF4eDgREREkJycTHh5OeHg4cXFxgG4KZ4sWLfD29iYgIICoqCiioqK4d++evo/Zs2ezZcsWrly5wtmzZxkyZAj79+9nwIABOfDKCSHMSePGcPGibjRa375KpxEgRTRhBOvX6zYcABg+HP59D2HyPqz5Id1rdgfgceJj/Db6oU42j51HhBAirylkW4iBb+v+2FVr1MwIm6FwIiGEkrp06cL06dMZO3YstWvXJjw8nF27duk3BoiIiCAyMlLfvmHDhqxatYrFixdTq1YtNmzYwObNm6levbq+zciRIxk0aBB9+vTh7bffJi4ujl27dmFra6tvM3bsWNzc3Bg3bhxxcXG4ubnh5ubGiRMnANiwYQP37t3jp59+wtnZWX97++239X0kJiYybNgwatSoQbNmzfjjjz/Yu3cvLVu2zO6XTQhhhiwsoEMHKFpU6SQCpIgmjGDAAN2tTBk4cAAKmNEa/fPbzufNIm8CcOz2McYeGKtwIiGEyL3u3YOTJw0/f3CDwdhZ2QGw6OQi7sffN1IyIYQ5GjhwIH///TcJCQkcPXoUd3d3/XMHDx4kODg4RXtfX18uXbpEQkICZ8+epW3btimeV6lUTJgwgaioKJ49e8bevXupVKlSijbBwcFotdpUt+bNmwMwfvz4NJ+/ceOGvo+RI0dy5coVnj59yoMHDzhw4AAtWrQw6msjhBAie0gRTWSZSgVz5sCJE1C1qtJpMqegTUFWd1qNlYVuj42pv09l77W9CqcSQojcJSkJxoyBN96Arl11jw1RIn8JetfpDUC8Op65R+caMaUQQgghhGm4cAH++UfpFCItUkQTRmFpCf+Onk8hMTHns2RWvVL1mNxyMgBatHT/uTv3ntx7zVlCCCEyysoKQkN10/2vXNFtNmCo4Q2H6z/4+N+x/xGbkLHtyIUQQgghzIFWCx99BC4uMGgQPH2qdCLxMimiiWwzcya4u8OjR0oneb0AjwC8K3oDEBUXRc8tPdFqtQqnEkKI3GPiRN2W7P36QatWhvdTtlBZ/XqWMc9iWHRikZESCiGEEEIoLzQUjh+HJ0/g11/hpWUZhQmQIprIFlOnwrBhEB4OPj6mPyLNQmXBcp/llMhfgkI2hfio1keoVCqlYwkhRK7RsCHcvAnffQelS2etr1GNRqFC92/0jLAZPEt6ZoSEQgghhBDKK1NGt+a4vT0EBOiWTxKmQ4poIlv4+LzYPaRFC93oA1PnVMCJTZ038cdnf+BbzVfpOEIIkeukNe3fEK7FXPmg6gcARD+J5ofTPxinYyGEEEIIhZUrB/Pm6T587NpV6TTiv6SIJrKFqyvs2AFLlsDYseZTPW9UthHlHMspHUMIIcRrBDYO1N+fFjqNJI2BuxUIIYQQQpigIkUgXz6lU4j/kiKayDb160Pv3kqnyLqwm2FKRxBCiFxFq4Xt23Vro8XFGdaHm7Mbrd9sDcCNmBtsPL/RiAmFEEIIIYRITYpoIkddvQpr1yqdImMeJzzmo80f0XBZQ9acXaN0HCGEyDXGjoX27WHPHt10BUN92fhLqpeozsr3V9KpaifjBRRCCCGEyGEbN8K2baDRKJ1EvIoU0USOOXVKt7B0t27wyy9Kp3m9HZd3sOKPFQD03daXe0/uKZxICCFyB39/sPj3Hci+fbqRaYZoUq4Jf372J91qdMPKwsp4AYUQQgghclBSkm4TgQ4doFo13c6cwjRJEU3kmPXr4e5dXWV9wgTTr7B3rtaZbjW6UTBfQRa0W0Dx/MWVjiSEELlC5cq60WgbN+pGo2Vl3UzZSVkIIYQQ5m77doiI0N2vWBHy51c2j0ifFNFEjvnmG/Dzg0aNYPfuF6MQTJVKpWJBuwWc7nuabjW6KR1HCGGA8ePHo1KpUtwqV678ynPWr19P5cqVsbW1pUaNGuzYsSOH0uYt48bB++8bf+OZxORE43YohBBCCJHNOnSArVuheXMYOlTpNOJVZO6DyDEWFhAcDMnJYGendJqMcbBxwMHGQekYQogsqFatGnv37tU/trJK/1dfaGgofn5+TJ48mfbt27Nq1Sp8fHw4deoU1atXz4m4wkDHbx9n8m+TiU2IZW+Pva8/QQghhBDCRFhY6AppHToonUS8jomPBRK5Tb58qQtoGo3hu7MpYeP5jdyPv690DCFEBllZWeHk5KS/FStWLN22c+bMoXXr1owYMYIqVaowceJE6tSpw7ysrH4vMuT2bXjwwLBzNVoN/pv8+fniz+y7vo8jt44YN5wQQgghhBDISDShMLUaPvkEbtzQrYtjyiPUniQ+4fOdn7MsfBkdKnVgS9ctshaPEGbg8uXLlCpVCltbWzw8PJg8eTJly5ZNs21YWBgBAQEpjrVq1YrNmze/8hoJCQkkJCToH8fGxgKgVqtRq9WZyvu8fWbPU1JWMj94AJMnW7BokQX9+mmYNs2wBTOHNxhO3x19cS7gzJ1Hd1CXTD+Lub3G5pYXzC+zueUF88uclbzm8j0KIYTI/aSIJhTVqxf89JPu/ocfwoYNyuZ5ldiEWH75S7et6C9//cL84/MZWH+gwqmEEK/i7u5OcHAwrq6uREZGEhQURJMmTTh79iwFCxZM1T4qKoqSJUumOFayZEmioqJeeZ3JkycTFBSU6viePXuwt7c3KHtISIhB5ynJkMwPH9qwcKEXiYkqvvsOatbcT5EizzLdTxFNEQa5DKJp4aZYXrFkx5XXr2Vnbq+xueUF88tsbnnB/DIbkjc+Pj4bkgghhPLGjIGYGBg8GN58U+k0IiOkiCYUNWQI/PyzbkRa9+7GX2DamJwLOhPsE0y7Ve0AGL5nOE3KNqFq0aoKJxNCpKdNmzb6+zVr1sTd3Z1y5cqxbt06evXqZbTrBAYGphjBFhsbi4uLC97e3jg4ZG5dRbVaTUhICF5eXlhbWxstY3bKauYzZ2DhQi39+kHbtu9QpIhhOTrSMUPtzO01Nre8YH6ZzS0vmF/mrOR9PrpXCCFyk0ePYM4cePwYfvgBoqKgQAGlU4nXkSKaUFSdOrB5s26ttCZNlE7zem3fassQ9yHMPjqbhOQEum7sStjHYUrHEkJkkKOjI5UqVeLKlStpPu/k5ER0dHSKY9HR0Tg5Ob2yXxsbG2xsbFIdt7a2NviP26ycqxRDM48eDSNHgpOTJWBp/GDpMLfX2NzygvllNre8YH6ZDclrTt+fEEJk1MmTuk33APz9pYBmLmRjAaG4li3No4D23BTPKdR2qg3AxfsXGb53uLKBhBAZFhcXx9WrV3F2dk7zeQ8PD/bt25fiWEhICB4eHjkRL88qUgReU6fMtOv/XGfN2TXG7VQIIYQQwkjeeQdu3oQpU2DoUKXTiIySIpowSX/9VZhffzXNuZ02Vjas6bQGe2vdOkffh3/P7zG/K5xKCJGW4cOHc+jQIW7cuEFoaCjvvfcelpaW+Pn5AdCjRw8CAwP17QcPHsyuXbuYMWMGFy9eZPz48Zw4cYKBA2X9Q3MyYPsA3vrfW3y0+SMiH0cqHUcIIYQQIk1FisCoUVC5stJJREZJEU2YnB9+UPHll43o0sWSiAil06TNtZgr89rM0z+eHzGfvx/9rWAiIURabt26hZ+fH66urnTu3JmiRYty5MgRihcvDkBERASRkS+KLA0bNmTVqlUsXryYWrVqsWHDBjZv3kz16tWV+hbyHI0G1q7VLbRrqAL5CpCsTSYxOZGZYTONF04IIYQQQuRpUkQTJkWjgU2bLEhKsuT+fRVTpyqdKH0f1/6YrtW7AhCviafHlh4kaZIUTiWEeNmaNWu4c+cOCQkJ3Lp1izVr1lCxYkX98wcPHiQ4ODjFOb6+vly6dImEhATOnj1L27Ztczh13qXVgqcndO0K33wDFy4Y1s9Qj6HYWOrWqFt4ciEPnz40YkohhBBCCMPFxOj+7hXmSYpowqRYWMCKFck4OcUxYEAys2crnSh9KpWKhe0WUr5QeQDCboUx4dAEZUMJIYQZU6mgdWvdfa0W1hi4pJlTASd6uel2X41LjGPesXmvOUMIIYQQImf06QNVqsCCBZCQoHQakVlSRBMmp3BhmDHjELNmaTD1zZgK2RbiR58fsfj3f6Vvfv2GQzcOKZxKCCHM14AB0K4d7N4N48cb3s+IRiOwVOl2+pxzdA5xiXHGCSiEEEIIYaAbN2DjRvjrLwgKUjqNMITJFNGmTJmCSqViyJAhr2w3e/ZsXF1dsbOzw8XFhaFDh/Ls2bOcCSlyTP785jMt0r20O37OukXKNVoNH/78IQ/iHyicSgghzFP+/LBtG3h760amGaq8Y3m61egGwMOnD1l8crGREgohhBBCGObxY2jaVHd/wACwsVE2j8g8kyiiHT9+nEWLFlGzZs1Xtlu1ahVffPEF48aN48KFC3z//fesXbuWL7/8MoeSCqXcuwddusCtW0onSdv7Jd6nWdlmANyKvUXvX3qj1WoVTiWEEHnbF42/0N+fETaDhCSZMyGEEEII5dSoAQcOwKlT0L+/0mmEIRQvosXFxeHv78+SJUsoXLjwK9uGhobSqFEjunXrRvny5fH29sbPz49jx47lUFqhhHPnoG5dWLcOOnUyzXnjlipLgjsGU8SuCACbL25m0clFCqcSQojc4e5dw86rWrwq71V+D4A7j++w4o8VRkwlhBBCCGEYNzcoWlTpFMIQVkoHGDBgAO3atcPT05Ovv/76lW0bNmzITz/9xLFjx6hfvz7Xrl1jx44ddO/ePd1zEhISSHip6hIbGwuAWq1GrVZnOu/zcww5VwnmlhdSZy5aFCwtrQAVERFa/voricqVFQz4H89zlrAtwZJ2S+i0oRMAQ3cPxau8F2ULlVUyXiq54WfC1JlbXshaZnP6PoV5uX4dJk2C5cvhyBGoUyfzfQQ2DuTniz8DMPX3qfR062nklEIIIYQQIq9QtIi2Zs0aTp06xfHjxzPUvlu3bty/f5/GjRuj1WpJSkris88+e+V0zsmTJxOUxop9e/bswd7e3uDsISEhBp+rBHPLCykzf/65A8HB1Rky5BTXrj3j2jUFg6UjJCQESyxpW6wt+x/u51PnTznz2xnOqs4qHS1N5v4zYQ7MLS8Yljk+Pj4bkggBO3fC0qW6+0FBsGVL5vt4u/TbeL7hyd5re7n6z1XWn1vPB5U/MG5QIYQQQohXOH0aIiPzKx1DGIFiRbSbN28yePBgQkJCsLW1zdA5Bw8eZNKkSXz33Xe4u7tz5coVBg8ezMSJExkzZkya5wQGBhIQEKB/HBsbi4uLC97e3jg4OGQ6t1qtJiQkBC8vL6xNfetIzC8vpJ954ECAdxTLlZ7/5m2hbsGtx7d4q8hbSkdLU276mTBV5pYXspb5+QhfIYytVy/dSLTHj6F2bdBqDdts4MvGX7L32l4AJv02ifdd3zduUCGEEEKIdGi1MGiQJcePt2TnTi0rVkChQkqnEoZSrIh28uRJ7t69S52X5mYkJydz+PBh5s2bR0JCApaWlinOGTNmDN27d6d3794A1KhRgydPntCnTx+++uorLCxSL/FmY2ODTRpbXlhbW2fpj9usnp/TzC0vZCxzYiLky5dDgV7jeV5ra2uq2ldVOs5r5dafCVNibnnBsMzm9j0K82FjA5s2wVtvwWuWTX2l5uWb06BMA47cOsLZu2fZcWUHFsovCyuEEEKIPGDbNjh2TPe+4+pVMGAsjzAhir2DbNmyJWfOnCE8PFx/q1evHv7+/oSHh6cqoIFuytB/C2XP28lOiHlLcjJ88QW0aKErpJm6LRe3cCvWRLcWFUIIE1a/ftYKaAAqlYrAxoH6x1NDp8r7BiGEEELkCG9v+PbbZAoUSGTChGSDRtUL06HYSLSCBQtSvXr1FMfy589P0aJF9cd79OhB6dKlmTx5MgAdOnRg5syZuLm56adzjhkzhg4dOqRZdBO5V58+sGyZ7v7gwbBggbJ50vNU/ZRhe4ax4MQCmpVrxr4e+7C0kJ9VIYTIae0rtad6ieqcvXuWo7ePctb2LO1op3QsIYQQQuRyNjYweLCGUqVCaN/eW+k4IotMei5DREQEkZGR+sejR49m2LBhjB49mqpVq9KrVy9atWrFokWLFEwplNC/v+4fI0tLqFpVN8/cFD1Nesovf/0CwKG/D7HpwiaFEwkhhPlKStLt1Pnnn5k/10JlkWI02oa7G4yYTAghhBDi1fLnT5JRaLmAortz/tfBgwdf+djKyopx48Yxbty4nAslTFLduro/pJycoFkzpdOkr4hdEVa+v5J2q9ox03smH1SVHeGEEMIQFy5Ahw66tUR8fODnnzPfR+dqnRlzYAzX/rlGoiaReHU8haxlZV8hhBBCGJ9aDbJ0cO5jUkU0ITKjSxelE2RM03JNiRgSQWG7LC7qI4QQeViFChAfr7u/eTNcvqzbcCAzrCysmNN6DvlU+Yg/F4+9tb3RcwohhBBC3LgBjRrBqFHw2WeG7S4uTJNJT+cUIrOuXDHNjQakgCaEEFljawuBgeDpCb/+mvkC2nPtK7WnRfkWqOTdrBBCCCGyyZdfwp07uvW7Z8xQOo0wJimiiVxj82aoUweGDVM6yett+2sbq86sUjqGEEKYlYEDISQEGjdWOokQQgghRNqSk8Hq3zl/RYvq1vMWuYcU0USucOcOdO0Kjx/DvHmwwUTXi9ZoNQzeOZgOqzvw6S+fcun+JaUjCSGE2TD24DGtVkvI1RDORJ8xbsdCCCGEyLMsLWHFCjhxAr7/HgrJ8qu5ihTRRK5QqhTMn6+737UrtGmjbJ70WKgseJr0FIB4dTxdN3YlISlB4VRCCGG+kpMNO+9e4j2arWiG90/efLn/S+OGEkIIIUSeV7cudOyodAphbFJEE7lGr16wZw+sWgX58yudJn2zWs2icrHKAIRHhfPF3i8UTiSEEObn4kX48EPo1s2w8x2tHLkVewvQTbEPjwo3XjghhBBCCJErSRFN5CpeXqa/80n+fPlZ02kN+SzzATD76Gx2XN6hcCohhDAfSUm6DQZWroR16+CMAbMxrS2sGdVwFFWLV2XtB2upWbKm8YMKIbLV/PnzKV++PLa2tri7u3Ps2LFXtl+/fj2VK1fG1taWGjVqsGNHyvdfWq2WsWPH4uzsjJ2dHZ6enly+fDlFm2+++YaGDRtib2+Po6NjmteJiIigXbt22NvbU6JECUaMGEFSUlKKNgcPHqROnTrY2Njw5ptvEhwcnOnvXwhhWjZtgt9+UzqFyG5SRBO52uPHsHy50ilSq+VUi+le0/WPP978MZGPIxVMJIQQ5sPKCoYP190vWhSuXTOsn15uvTjT7wydq3XGQiVviYQwJ2vXriUgIIBx48Zx6tQpatWqRatWrbh7926a7UNDQ/Hz86NXr16cPn0aHx8ffHx8OHv2rL7NtGnTmDt3LgsXLuTo0aPkz5+fVq1a8ezZM32bxMREfH196devX5rXSU5Opl27diQmJhIaGsry5csJDg5m7Nix+jbXr1+nXbt2tGjRgvDwcIYMGULv3r3ZvXu3kV4dIUROe/QI+vSBJk3ggw90H/iJ3EneMYpc6/JlaNAAPv4YfvpJ6TSpDaw/kPaV2gNwL/4eH23+CI1Wo3AqIYQwD337wsyZcOOG4euNWFlYSfFMCDM1c+ZMPv30U3r27EnVqlVZuHAh9vb2LFu2LM32c+bMoXXr1owYMYIqVaowceJE6tSpw7x58wDdKLTZs2czevRoOnbsSM2aNVmxYgV37txh8+bN+n6CgoIYOnQoNWrUSPM6e/bs4fz58/z000/Url2bNm3aMHHiRObPn09iYiIACxcupEKFCsyYMYMqVaowcOBAPvjgA2bNmmXcF0kIkWMWL4YHD3T3bWxe7M4pch/5TytyrUOH4Px53f0RI6BTJ7CzUzbTy1QqFcveXUathbWIjIsk5FoIM0JnMKLRCKWjCSGEybOzg6FDjd9vYnKifrq9EMI0JSYmcvLkSQIDA/XHLCws8PT0JCwsLM1zwsLCCAgISHGsVatW+gLZ9evXiYqKwtPTU/98oUKFcHd3JywsjK5du2YoW1hYGDVq1KBkyZIprtOvXz/OnTuHm5sbYWFhKa7zvM2QIUPS7TchIYGEhBebUcXGxgKgVqtRq9UZyvbc8/aZPU9J5pbZ3PKC+WU2tbwDB0LBgiqmTLFk/Pgk/hvL1PJmhLllzmrejJ6X6SLarl27KFCgAI0bNwZ0axEsWbKEqlWrMn/+fAoXLpzZLoXIFr17w7Fj8PvvsHmzaRXQniuevzg/vvcjXj96oUXLl/u/pHn55rxd+m2lowkhRJ5y8s5Jgg4FoVKp2NJ1i9JxhBCvcP/+fZKTk1MUqgBKlizJxYsX0zwnKioqzfZRUVH6558fS69NRqR3nZevkV6b2NhYnj59il0ab1onT55MUFBQquN79uzB3t4+w/leFhISYtB5SjK3zOaWF8wvsynldXbWjZI/f/7FYI7/MqW8GWVumQ3NGx8fn6F2mS6ijRgxgqlTpwJw5swZhg0bRkBAAAcOHCAgIIAffvghs10KkW3+9z9ITISCBZVOkr6Wb7RkVKNRTPl9CkmaJPw2+nG672kK2phwaCGEMDEJCXDgALRunflzkzRJvLf2PW7G3gTgVOQp6jjXMXJCIYQwXGBgYIqRdLGxsbi4uODt7Y2Dg0Om+lKr1YSEhODl5YW1tbWxo2YLc8tsbnnB/DJL3uxnbpmzmvf5CN/XyXQR7fr161StWhWAjRs30r59eyZNmsSpU6do27ZtZrsTIlvZ2Ohupm5Ciwnsv7GfY7ePcfWfqwzYMYAV761QOpYQQpiFH3+EL7+E27fhzz+hevXMnW9lYcUXjb9gwI4BAAQdCpLRaEKYsGLFimFpaUl0dHSK49HR0Tg5OaV5jpOT0yvbP/8aHR2Ns7Nzija1a9fOcDYnJ6dUu4Q+v+7L10ori4ODQ5qj0ABsbGywSeNNrbW1tcF/3GblXKWYW2Zzywvml1nJvFotPHyo2+Qoo8zt9QXzy2xo3oyek+nVdPPly6cf5rZ37168vb0BKFKkSIYrd0IoRauFuXNh506lk6RkbWnN6k6rKZhPN/rsxz9/5Kc/TXA3BCGEMEF378KtW7p/4ydONKyPXm69KF2wNABbL23lVOQpIyYUQhhTvnz5qFu3Lvv27dMf02g07Nu3Dw8PjzTP8fDwSNEedFN+nrevUKECTk5OKdrExsZy9OjRdPtM7zpnzpxJsUtoSEgIDg4O+oEIr8sihDAPW7dChQowaRJkcCagyAUyXURr3LgxAQEBTJw4kWPHjtGuXTsA/vrrL8qUKWP0gEIYy9On0LkzDB4MH34IERFKJ0rpjcJvsLD9Qv3jftv7cfXhVQUTCSGEefjsMyhZEjp0gJEjDevDxsqGL5t8qX884dAEI6UTQmSHgIAAlixZwvLly7lw4QL9+vXjyZMn9OzZE4AePXqk2Hhg8ODB7Nq1ixkzZnDx4kXGjx/PiRMnGDhwIKDb8GnIkCF8/fXXbN26lTNnztCjRw9KlSqFj4+Pvp+IiAjCw8OJiIggOTmZ8PBwwsPDiYuLA8Db25uqVavSvXt3/vjjD3bv3s3o0aMZMGCAfiTZZ599xrVr1xg5ciQXL17ku+++Y926dQzNjt1ShBDZQq3Wved4/Bi++grMbNkwkQWZLqLNmzcPKysrNmzYwIIFCyhdWvep7c6dO2ltyEIkQuQQW1vd+migG3a7Y4eyedLSrUY3etTqAUBcYhx+G/1ITE5UOJUQyouJiVE6gjBh+fPrFvDduhXq1jW8n5dHo225tIXTkaeNlFAIYWxdunRh+vTpjB07ltq1axMeHs6uXbv0C/ZHREQQGRmpb9+wYUNWrVrF4sWLqVWrFhs2bGDz5s1Uf2n+98iRIxk0aBB9+vTh7bffJi4ujl27dmFra6tvM3bsWNzc3Bg3bhxxcXG4ubnh5ubGiRMnALC0tGTbtm1YWlri4eHBhx9+SI8ePZgw4UVhvkKFCmzfvp2QkBBq1arFjBkzWLp0Ka1atcrul00IYSTPnsE774CFBTRuDO++q3QikVMyvSZa2bJl2bZtW6rjs2bNMkogIbKLSgXBweDpCePGme4/dPPazCP0ZihXHl7h+J3jrD27lu61uisdS4gcM3XqVMqXL0+XLl0A6Ny5Mxs3bsTJyYkdO3ZQq1YthRMKU1SkSNb7sLGyIbBxIAN36kamBB0KYnPXzVnvWAiRLQYOHKgfSfZfBw8eTHXM19cXX1/fdPtTqVRMmDAhRcHrv4KDgwkODn5lrnLlyrHjNZ/WNm/enNOnpVAvhLkqWBAWLIBBg0Cj0f2tKfKGTI9EO3XqFGfOnNE/3rJlCz4+Pnz55ZckJsqIGWHaCheG48dNt4AGUNCmIGs6raFAvgLMbzufD2t+qHQkIXLUwoULcXFxAXRrxISEhLBz507atGnDiBEjFE4ncrtedWQ0mhBCCCEypmrVzG9oJMxbpotoffv25a+//gLg2rVrdO3aFXt7e9avX89IQxciESIHWWT6pz7n1S1VlxuDb9D/7f6o5GMNkcdERUXpi2jbtm2jc+fOeHt7M3LkSI4fP56lvqdMmaJf9+ZVZs+ejaurK3Z2dri4uDB06FCePXuWpWuLnBMeDu+/DxcuZP5cWytbAhu/WEdpwmFZG00IIYQQQuhkupzw119/6bd5Xr9+PU2bNmXVqlUEBwezceNGY+cTItv9+iu0bavbeMCUFLXPxF7JQuQihQsX5ubNmwDs2rULT09PALRaLcnJyQb3e/z4cRYtWkTNmjVf2W7VqlV88cUXjBs3jgsXLvD999+zdu1avvzyy1eeJ0zDpk3g5gY//5yFnTrr9KJUwVIAbL64WUajCSGEEALQLQu0e7fSKYSSMl1E02q1aDQaAPbu3Uvbtm0BcHFx4f79+8ZNJ0Q2++EHaNECdu7UzWc3ZTsv72TcgXFKxxAi273//vt069YNLy8vHjx4QJs2bQA4ffo0b775pkF9xsXF4e/vz5IlSyhcuPAr24aGhtKoUSO6detG+fLl8fb2xs/Pj2PHjhl0bZGzWrWCYsV093/7Df7dMC9TZDSaEEIIIf7rxAmYMAFat4Zu3ZROI5SS6SJavXr1+Prrr/nxxx85dOgQ7dq1A+D69ev63XCEMBf168O/u41z44ZulxVTNP7geNquasuEwxPYeF5GfIrcbdasWQwcOJCqVasSEhJCgQIFAIiMjKR///4G9TlgwADatWunH9X2Kg0bNuTkyZP6otm1a9fYsWOH/kMjYdry54fJk+F//4O//oJ/f3wyrXed3ilGo4VHhRsvpBBCCCHMzst7ijRooFgMobBM7845e/Zs/P392bx5M1999ZV+VMCGDRto2LCh0QMKkZ2qVYNFi3R/aI0bB5aWSidKW8n8LwrUWy5toVPVTgqmESJ7WVtbM3z48FTHhw4dalB/a9as4dSpUxleT61bt27cv3+fxo0bo9VqSUpK4rPPPnvldM6EhAQSEhL0j2NjYwFQq9Wo1epM5X3ePrPnKcnUMn/00Yv7aUXKSF5LLBnpMZIhe4YAMP7AeNZ/sN6YMTPM1F7fjDC3zOaWF8wvc1bymsv3KITI3ebOhcaNYdky+OwzpdMIpWS6iFazZs0Uu3M+9+2332JpqhUIIV7hQzPY/PKzep8RdiuMWiVrEeARoHQcIbLV8uXLKVasmH6k88iRI1m8eDFVq1Zl9erVlCtXLsN93bx5k8GDBxMSEoKtrW2Gzjl48CCTJk3iu+++w93dnStXrjB48GAmTpzImDFj0jxn8uTJBAUFpTq+Z88e7O3tM5z3ZSEhIQadpyRzy/y6vKU0pShiXYSH6ods+WsL8zbM4w37N3IoXWrm9vqC+WU2t7xgfpkNyRsfH58NSYQQInMsLKBrV91N5F2ZLqI9d/LkSS78u+1V1apVqVOnjtFCCaG0pCTdqDRT2RhTpVKx3Ge57NQp8oRJkyaxYMECAMLCwpg/fz6zZs1i27ZtDB06lE2bNmW4r5MnT3L37t0Uv6OSk5M5fPgw8+bNIyEhIdUHQGPGjKF79+707t0bgBo1avDkyRP69OnDV199hUUaW/wGBgYSEPCiwB0bG4uLiwve3t44ODhk6vtXq9WEhITg5eWFtbV1ps5ViqlnfvJEdytRQvc4M3lvl7zN0BDdKMhDHGJg24HZHTcVU39902Jumc0tL5hf5qzkfT66VwghhFBapotod+/epUuXLhw6dAhHR0cAYmJiaNGiBWvWrKF48eLGzihEjrp9+8UnDAMGKJ3mhbQKaEmaJKwsDK6FC2GSbt68qV8qYPPmzXTq1Ik+ffrQqFEjmjdvnqm+WrZsmWr0dM+ePalcuTKjRo1KcwR1fHx8qkLZ83ZarTbN69jY2GDzfIHFl1hbWxv8x21WzlWKqWV++hTmz4dp03QbDvz4Y8rnM5L3s/qfMS1sGpFxkWz9ayu3n9ymvGP57Av9Cqb2+maEuWU2t7xgfpkNyWtO358QIneJjYWYGChbVukkwlRkemOBQYMGERcXx7lz53j48CEPHz7k7NmzxMbG8vnnn2dHRiFyzIMHUKeObke3oUPBlDfj23dtH5XnVebS/UtKRxHCqAoUKMCDBw8A3XRILy8vAGxtbXn69Gmm+ipYsCDVq1dPccufPz9FixalevXqAPTo0YPAwBc7MXbo0IEFCxawZs0arl+/TkhICGPGjKFDhw6ybIGZSUqCKVPg3j1YtQouGfDPpa2VLV82+ZL3Kr/Hqb6nFCugCSGEECLnTZkCrq4QGKgrqAmR6SEsu3btYu/evVSpUkV/rGrVqsyfPx9vb2+jhhMipxUtCv7+MGsWODmZznTO/9p6aSs+a3zQosV3vS9Heh/B3tqwdZeEMDVeXl707t0bNzc3/vrrL/2umOfOnaN8+fJGv15ERESKkWejR49GpVIxevRobt++TfHixenQoQPffPON0a8tslfBgjBsGHz1Ffj6gqGDWQa8PYCB9XN+GqcQQgghlBMRofu78NkzmDkT+vSBTK7SIXKhTBfRNBpNmkOqra2t0Wg0RgklhJKmToV8+WDECF1RzRS1rNCSKsWrcP7eec7cPcOgHYP4vuP3SscSwijmz5/P6NGjuXnzJhs3bqTov/8jnjx5Ej8/vyz3f/DgwVc+trKyYty4cYwbNy7L1xLKGzgQOnaEqlUN70PWoxRCCCHyHgcH3fuIuXNh0CCoUEHpRMIUZLqI9s477zB48GBWr15NqVKlALh9+zZDhw6lZcuWRg8oRE6zttYN2zVl+fPlZ4PvBuotqUe8Op5l4ctoWq4pH9X+SOloQmSZo6Mj8+bNS3U8rd0vhXidggWzVkBLS7ImmUcJjyhiV8S4HQshhBDCZDg6wrffQv/+uvtCgAFros2bN4/Y2FjKly9PxYoVqVixIhUqVCA2Npb//e9/2ZFRCJPw7xJNJqNK8Sosar9I/7jf9n6cvXtWwURCGE9MTAwzZsygd+/e9O7dm1mzZvHo0SOlY4k8LlmTzKozq6j2XTX6/NJH6ThCCCGEyAEVKkDhwkqnEKYi00U0FxcXTp06xfbt2xkyZAhDhgxhx44dnDp1ijJlymRHRiEUlZCgG75bvTpERSmdJqUPa37Ip3U+BeBp0lN81/sSlxincCohsubEiRNUrFiRWbNm6TewmTlzJhUrVuTUqVNKxxNmLjQU5szJ9NsfABKTExm2ZxiXHlxi44WN/Bn9p5HTCZG7TZkyhW+//TbV8WnTpuHr66tAIiGEECJzDHoXqVKp8PLyYtCgQQwaNAhPT09j5xLCZHzxBcybpyugdesGyclKJ0ppTus51CpZC4CL9y/y2bbP0Gq1CqcSwnBDhw7l3Xff5caNG2zatIlNmzZx/fp12rdvz5AhQ5SOJ8xY167QqBF88YUFkZH5M32+nbUdXzT6AoAmZZuQrDGxXwhCmLjz58/TunXrVMfbtGnD4cOHFUgkhBApqdXQoQNs3AjyJ5VIS4bWRJs7d26GO/z8888NDiOEKQoMhHXrdNM5/fzAwrABDNnGztqOdb7rqLe4Ho8TH7PyzEqalWvGp3U/VTqaEAY5ceIES5Yswcrqxa8oKysrRo4cSb169RRMJsxdjRqwdi0kJ6vYubM8vXplvo8+dftQs2RNmpdvLhsOCJFJz549I1++fKmOW1tbExsbq0AiIYRIaelS2LZNdxs4EGTFKvFfGSqizZo1K0OdqVQqKaKJXKdECdiwAWxtwc1N6TRpq1S0EkvfXUqXDV0AGLRzEG+XfpvaTrWVDSaEARwcHIiIiKBy5copjt+8eZOCBQsqlErkBoMGwc8/Q9++SRQpch4ol+k+7KztaFGhhfHDCZEHlCtXjvXr11O9evUUx9esWUNVY+8AIoQQBjh06MV9f3/lcgjTlaEi2vXr17M7hxAmzcND6QSv17laZw7/fZj5x+eTkJyA73pfTvY5iYONg9LRhMiULl260KtXL6ZPn07Dhg0B+P333xkxYgR+fn4KpxPmzMEBTpwAtVrLjh0yR0OInObr68ukSZO4ceMG77zzDgD79u1j9erVrF+/XuF0QggBq1dD9+7w22/QoIHSaYQpylARTQiR2qVLup1aTGk2zwzvGRy5dYSTkSe58vAKvbf2Zu0Ha2XKkTAr06dPR6VS0aNHD5KSkgDdVJ9+/foxZcoUhdMJ8YJWq2XP1T2E3QpjfPPxSscRwuTVr1+fDRs2MG3aNDZs2ICdnR01a9Zk7969NGvWTOl4QgiBSgXt2uluQqRFimhCGGDFCvjsM+jTB9LYZEoxNlY2rPddj9siNx4lPGL9+fU0O96MAfUHKB1NiAzLly8fc+bMYfLkyVy9ehWAihUrYm9vr3AykRs9e6abrm+I99e9z+aLmwHoVKUTNUrWMF4wIXKptm3b0rFjR6VjCCGEEAYxsSXShTB9f/8Nn34KT5/CnDmwc6dpjfKqULgCwT7B+sdDdw/lxJ0TygUSwkD29vbUqFGDGjVqSAFNGN2TJ1ZMnGhBmTLw11+G9dG0bFP9/QmHJxgpmRC51+XLlzl27Fiq40ePHuXECXmvIoRQjuzEKTJKRqIJkUnlysHs2dC/v66Y1ry5lgMHlE6Vkk9lH4Y2GMqsI7NQa9T4rvflVJ9TFLYrrHQ0IdL0/vvvZ7jtpk2bsjGJyCt27y7PihWWAHz9tW6EcWb1rdeXqb9PJfpJNBvOb+Ds3bNUL1H99ScKkUctXryYMmXKpDp++/Ztpk6dytGjRxVIJYTI67Ra3RrYNWvCRx9Bo0ZKJxKmTEaiCWGAzz6Dgwdh8WKws1M6TdqmeE6hQRndapiuRV3RaDUKJxIifYUKFcrwTQhjaN36BkWKaLG01E3nNOQTaHtre0Y1GqV/POGQjEYT4lVu3ryJWxpbnbu5uXH+/HkFEgkhBBw/DkePwpIl8OWXSqcRps6gkWgxMTEcO3aMu3fvotGk/MO8R48eRgkmhClTqcDU17/NZ5mPtR+sZe3ZtQxrOAwLldTMhen64YcflI4g8hh7+yR++CGZ6tWteOMNw/t5eTTa+vPrZTSaEK9gbW3N3bt3cXV1TXE8MjISKyuZICOEUMalS5A/Pzx5ohuJJsSrZPq31S+//IK/vz9xcXE4ODik2PXv+W5qQuRFMTEQHg5eXkoneaFsobKMaDRC6RhCCGGS2rTRYm2dtT6ej0YL2BMAwMTDE1n7wVojpBMi96lduzajR49m69at+pHFMTExfPnll3iZ0hsoIUSe0r07vP8+bNoEsu+JeJ1MD00ZNmwYn3zyCXFxccTExPDPP//obw8fPsyOjEKYvGvXCtGggRXvvqsrpJmye0/uEZsUq3QMIYTINfrW60vJ/CUBWH9ONxpNCJHaxx9/zK1btyhXrhwtWrSgRYsWVKhQgaioKGbMmKF0PCFEHpY/v66Y5uCgdBJh6jJdRLt9+zaff/657JQmxEv273fh2jUVz55Bnz6mu7vLr3//ytvfv82ciDmyRpoQQrxEq4W9e+Hy5cyf+/LaaFq0fL7zc7Sm+otACAUVLVqUkydPMm3aNKpWrUrdunWZM2cOZ86cwcXFRel4QgghxGtluojWqlUr2YJaiP/46KPz1K2r4e23Yf163ZpppuZZ0jO6bOjCnbg7nIw9yXcnvlM6khBCmIRr16BJE910/HHjDOujb72+lHcsD8CBGwf48c8fjRdQiFwkf/78NG7cmA4dOtC0aVMcHR3ZuXMnW7duVTqaECKPSUw03cEPwnRluojWrl07RowYwfjx49m4cSNbt25NcTPUlClTUKlUDBky5JXtYmJiGDBgAM7OztjY2FCpUiV27Nhh8HWFMAZraw2bNyfz669QrpzSadJma2VLsE8wKlTUKFCDrtW6Kh1JCCFMQokSukWFAdaseXE/M+yt7fmu7YsPJwJ2B3A//r6REgqRO0RFRVG3bl2qV69Ou3bt8PHx4b333tPfhBAiJy1YAG++CUFBEB2tdBphLjK9scCnn34KwIQJqbdxV6lUJCcnZzrE8ePHWbRoETVr1nxlu8TERLy8vChRogQbNmygdOnS/P333zg6Omb6mkIYW8mSZHmB6uzmXdGbX7r+wrMLzyhmX0zpOELozZ07N8NtP//882xMIvKiAgVg5Ej44QcYO1b3htoQbd5qQ+dqnVl3bh0Pnj5gZMhIlnVcZtywQpixpUuXUr58efbv30+FChU4evQoDx8+ZNiwYUyfPl3peEKIPGb5ct1o9PHjwddX9/ecEK+T6SKaRmPcdZTi4uLw9/dnyZIlfP31169su2zZMh4+fEhoaCjW/1Yrypcvb9Q8QhiLRgOzZoG/Pzg5KZ3mBe83vNlxUUZvCtMya9asDLVTqVRSRBPZYvBgCAgAS8us9TO71Wx2XdlFbEIsP4T/QI9aPWhevrlRMgph7i5dusTChQspVqwYFhYWWFpa0rhxYyZPnsznn3/O6dOnlY4ohMgjnjyBokV1y/DUrQtVqyqdSJiLTE/nNLYBAwbQrl07PD09X9t269ateHh4MGDAAEqWLEn16tWZNGmSQaPfhMhODx9Chw4wfDh07QpJSUonSt+9J/cIOhgkGw0IRV2/fj1Dt2vXrikdVeRS+fJlvYAG4FzQmSktp+gff7btMxKSErLesRC5gEajoWDBggAUK1aMO3fuAFCuXDkuGTKPWgghDJQ/P4SEwN9/w8KFSqcR5iRDI9Hmzp1Lnz59sLW1fe2Um8yMEFizZg2nTp3i+PHjGWp/7do19u/fj7+/Pzt27ODKlSv0798ftVrNuHRWAk5ISCAh4cWb19jYWADUajVqtTrDWZ97fo4h5yrB3PKC+WVOK+/Tp/DHH1aAisOHtezfn0yLFqaxauXLecOjw/Hd4Mvfj/4mKTmJsU3HKpwubbnhZ8LUZSWzOX2fQuSEvvX6suLPFRy5dYRLDy4x5bcpjGtu4I4FQuQiZcuW5c8//6RSpUq4u7szbdo08uXLx+LFi3njjTeUjieEyINcXHQ3ITIqQ0W0WbNm4e/vj62t7Sun3GRmms3NmzcZPHgwISEh2NraZugcjUZDiRIlWLx4MZaWltStW5fbt2/z7bffpltEmzx5MkFBQamO79mzB3t7+wxdNy0hISEGn6sEc8sL5pf5v3kHDizCt9/WY/Dg0zx9eg9T2/8iJCSE07GnufnoJgBf//Y1SbeSaODYQOFk6TP3nwlzYEjm+Ph4o+e4desWW7duJSIigsTExBTPzZw50+jXE+Jl9+7B9Om6r8sMWNLMQmXB4vaLqbO4DkmaJJacWsKoxqOwtcrY+x0hcqvOnTvrl4aZMGEC7du3p0mTJhQtWpS1a9cqnE4IIYR4vQwV0a5fv57m/aw4efIkd+/epU6dOvpjycnJHD58mHnz5pGQkIDlf+ZVODs7Y21tneJ4lSpViIqKIjExkXz58qW6TmBgIAEBAfrHsbGxuLi44O3tjYODQ6Zzq9VqQkJC8PLy0q/LZsrMLS+YX+b08rZtCwMGgK3t2wqmS+3lvG2t22J3xI4v9n8BwLw78+ji1YVqxaspnDKl3PIzYcqykvn5CF9j2bdvH++++y5vvPEGFy9epHr16ty4cQOtVpvid4YQ2UGjAQ8PuHpVt05KQABUr575fmqUrMFwj+HcfXKXaV7TpIAmBODm5kbbtm0BePPNN7l48SIPHz6kcOHCqFQqhdMJIfKKBw9066EJYYhMbyxgLC1btuTMmTMpjvXs2ZPKlSszatSoVAU0gEaNGrFq1So0Gg0WFrrl3P766y+cnZ3TLKAB2NjYYGNjk+q4tbV1lv64zer5Oc3c8oL5ZU4rrynHf553ZOOR/HnvT1adWUVcYhwfbPiA458ep7BdYaUjppIbfiZMnSGZjf09BgYGMnz4cIKCgihYsCAbN26kRIkS+Pv707p1a6NeS4j/srCA/v1h2DDdv+HHjxtWRAOY1HKSFAaEeI0iRYooHUEIkYc8eqSbvlmvnm7AQ5cuSicS5sagIpoxptkULFiQ6v95V5o/f36KFi2qP96jRw9Kly7N5MmTAejXrx/z5s1j8ODBDBo0iMuXLzNp0iTZqU2YjZ07Yf9++PZbpZO8oFKpWNJhCRfuXeB01Gmu/nOVrhu7sqPbDiwtjLDKthCZdOHCBVavXg2AlZUVT58+pUCBAkyYMIGOHTvSr18/hROK3O6zzyAqSrdjZ+nShvcjBTQhhBDCtKxbp1u/+tdfoXZtKaKJzMt0ES0np9lEREToR5wBuLi4sHv3boYOHUrNmjUpXbo0gwcPZtSoUUa9rhDZYdIkGD0atFqoVg0+/ljpRC/YW9uzuetm6i2ux734e+y5uofAfYFM85qmdDSRB+XPn1//AY2zszNXr16lWjXdFOP79+8rGU3kEfb2MC0b/vmLjotmy6Ut9Knbx/idCyGEEOK1ChSAypXh4kXo0UPpNMIcZbqIlp3TbA4ePPjKxwAeHh4cOXIkS9cRQgllyugKaAC7d5tWEQ2gbKGybOi8gZYrWpKkSeLb0G9xc3LDr4af0tFEHtOgQQN+++03qlSpQtu2bRk2bBhnzpxh06ZNNGhguhtfCPEqP/7xI5/v+pyYZzFUKlqJ5uWbKx1JCCGEyHP8/KBrVwgP141EEyKzLF7fJKULFy7Q49+S7X+n2UydOtXoAYXILXr00K2zM2UKrFypdJq0NS3XlNmtZusf99rai9ORp5ULJPKkmTNn4u7uDkBQUBAtW7Zk7dq1lC9fnu+//17hdCIv0mrh6NGs9fFE/YSYZzEAfLnvy6yHEkIwf/58ypcvj62tLe7u7hw7duyV7devX0/lypWxtbWlRo0a7PjP1ularZaxY8fi7OyMnZ0dnp6eXL58OUWbhw8f4u/vj4ODA46OjvTq1Yu4uDj98+PHj0elUqW65c+fX98mODg41fO2trL5iBA5RaUCNzfdVyEyK9NFtLSm2Twn02yEeLV582DUKN3C1aaq/9v96eXWC4CnSU/xWevD3Sd3FU4l8pI33niDmjVrArrfOQsXLuTPP/9k48aNlCtXTuF0Iq8JCYG6daFBA/jjD8P76VO3Dw1dGtKjVg+2dN1ivIBC5FFr164lICCAcePGcerUKWrVqkWrVq24ezft9yyhoaH4+fnRq1cvTp8+jY+PDz4+Ppw9e1bfZtq0acydO5eFCxdy9OhR8ufPT6tWrXj27Jm+jb+/P+fOnSMkJIRt27Zx+PBh+vR5MUV7+PDhREZGprhVrVoVX1/fFHkcHBxStPn777+N/AoJIYTIDpn+U/75NBtAP83mm2++4ZNPPpFpNkK8Rlqfdjyf4mkqVCoV89vOp0EZ3f/PEY8i6Ly+M+pktcLJRF7Ru3fvNKfzC6GECxfg9L8DcoOCDO/HQmXBvh77WO6znOL5ixsnnBB52MyZM/n000/p2bMnVatWZeHChdjb27Ns2bI028+ZM4fWrVszYsQIqlSpwsSJE6lTpw7z5s0DdKPQZs+ezejRo+nYsSM1a9ZkxYoV3Llzh82bNwO6GTm7du1i6dKluLu707hxY/73v/+xZs0a7ty5A0CBAgVwcnLS36Kjozl//jy9evVKkUelUqVoV7Jkyex7sYQQREZCbKzSKURukOkimkyzEcJ4rl8HDw/4ty5tMmysbNjYeSPOBZwBOPT3IQJ2ByicSuQV9+7do3Xr1ri4uDBixAj+yMrwHyGy6NNPwdkZ6tSBTz7JWl+2VjJdSwhjSExM5OTJk3h6euqPWVhY4OnpSVhYWJrnhIWFpWgP0KpVK33769evExUVlaJNoUKFcHd317cJCwvD0dGRevXq6dt4enpiYWHB0XTmfC9dupRKlSrRpEmTFMfj4uIoV64cLi4udOzYkXPnzmXiFRBCZNbo0eDkBP7+uoKaEIbK1MYCycnJ3Lp1K9U0GyFE5p0/D40aQUwMdO6sG+lgSh9ClipYip+7/EzT4KYkJicy7/g83Jzd+MQti39FCvEaW7Zs4Z9//mH9+vWsWrWKmTNnUrlyZfz9/enWrRvly5c3uO8pU6YQGBjI4MGDmT17drrtYmJi+Oqrr9i0aRMPHz6kXLlyzJ49m7Zt2xp8bWGe7OwgLAzKljX+2in3ntzjwv0LNC3X1LgdC5HL3b9/n+Tk5FSjt0qWLMnFixfTPCcqKirN9lFRUfrnnx97VZsSJUqkeN7KyooiRYro27zs2bNnrFy5ki+++CLFcVdXV5YtW0bNmjV59OgR06dPp2HDhpw7d44yZcqkmT8hIYGEhAT949h/h9So1WrU6szNFnjePrPnKcncMptbXjC/zJnJGx8P69db8fSpil9+0fLdd0nk9Ldpbq8vmF/mrObN6HmZKqJZWlri7e3NhQsXcHR0NCSXEOJflSrpFrQ8cEC31fI//5hWEQ3AvYw7C9st5JOtusJZv+39qFq8qn6qpxDZpXDhwvTp04c+ffpw69YtVq9ezbJlyxg7dixJSUkG9Xn8+HEWLVqk/yAoPYmJiXh5eVGiRAk2bNhA6dKl+fvvv+X3Xh6WHUvx/XD6B4aHDEeFiosDL1LMvpjxLyKEUNTPP//M48eP+eijj1Ic9/DwwMPDQ/+4YcOGVKlShUWLFjFx4sQ0+5o8eTJBacwp37NnD/b29gblCwkJMeg8JZlbZnPLC+aXOSN5Y2Pz0aSJK7/+Wob69SM5eDA8+4Olw9xeXzC/zIbmjY+Pz1C7TBXRAKpXr861a9eoUKFCpkMJIV6wsoLVq2HcOJg2DRwclE6Utp5uPTkVeYp5x+fhaOtIsiZZ6UgiD1Gr1Zw4cYKjR49y48YNg9eMiYuLw9/fnyVLlvD111+/su2yZct4+PAhoaGhWFtbA2Rp9JsQaQm5FsLDpw8BGL5nOME+wcoGEsKMFCtWDEtLS6Kjo1Mcj46OxsnJKc1znq9Pll7751+jo6NxdnZO0aZ27dr6Nv/duCApKYmHDx+med2lS5fSvn371/7usra2xs3NjStXrqTbJjAwkICAF0trxMbG4uLigre3Nw6ZfBOpVqsJCQnBy8tL/3vO1JlbZnPLC+aXObN5u3aFhAR4/LgUxYqVyoGEKZnb6/v/9u47vKnyf+P4O91lb1qgSJW99yggCmWD4EAZAiKCPwVlKCAKqICyERBlKQoKgoKiICAVGSoVSqEs2SJ7yiiztE1+f5wvgdCWlrbpScr9uq5eJGfeCWme5pPnPA+4X+a05o1O4aB5911EGzlyJG+++SYjRoygWrVqDtM1A/f9Ji7yICtYENzhiuiJTSdisVgYWHcgRXIkfpmBSHpas2YN8+fPZ/HixVitVp566imWLVtGw4YNU3W8Xr160bJlS0JDQ5Mtov3000/UqVOHXr168eOPP5I/f346duzIoEGD8PT0TNX5JfM4fhxGj4Zu3Yxx0lJrYtOJrDiwgos3LjJn2xy6VurK48GPp19QkUzMx8eHatWqsXr1atq2bQuA1Wpl9erV9O7dO9F96tSpw+rVq+nbt699WVhYmL1HWHBwMAEBAaxevdpeNIuOjmbjxo288sor9mNcvHiRyMhIqlWrBsBvv/2G1Wq1jxl9y6FDh1izZg0//fRTso8nPj6eHTt23HPIAF9fX3x9fRMs9/b2TvWH27TsaxZ3y+xuecH9Mt9PXm9v4wogM7nb8wvulzm1eVO6z30X0W69uT/xxBNY7hgcxGazYbFYiI9XLxWRtLDZ4MoVyJ7d7CS3eXt6M6X5FLNjyAOicOHCnD9/nmbNmjFz5kxat26d6AeHlFqwYAFbtmwhIiIiRdv/888//Pbbb3Tq1Inly5dz4MABXn31VWJjY3n33XcT3Udj1bhX5tTm/eMPC82bexITY+HQISs//JD6v3ny+ublw8c/5NUVrwLw8rKXiXwpMtHJB9zt+QX3y+xuecH9Mqclb2L79O/fn65du1K9enVq1qzJpEmTuHr1Kt26dQOgS5cuFC5cmFGjRgHQp08fGjRowIQJE2jZsiULFixg8+bNzJw5EzBmy+zbty8jR46kRIkSBAcHM3ToUAoVKmQv1JUpU4ZmzZrRo0cPpk+fTmxsLL1796Z9+/YUKuTYs2X27NkEBgbSvHnzBNmHDx9O7dq1KV68OBcvXmTcuHEcPnyYl1566b6fGxERyVj3XURbs2aNM3KICMagl716wfbt8Oef4OfCE7lZbVYijkdQq0it5DcWuQ/vvfce7dq1S5cxyI4ePUqfPn0ICwvDL4W/UFarlQIFCjBz5kw8PT2pVq0ax48fZ9y4cUkW0TRWjcHdMt9v3thYD7JlCyUmxp/Vq63Mm7ea3Lljkt8xCQG2AEpnLc2eq3vYf34/Peb0oENgh3TL6wrcLbO75QX3y5yavImNU/Pcc89x9uxZhg0bxqlTp6hcuTIrV660Xzp55MgRPDw87NuHhIQwf/58hgwZwttvv02JEiVYsmQJ5cuXt28zcOBArl69Ss+ePbl48SL16tVj5cqVDu3HvHnz6N27N40aNcLDw4Onn36aKVMcv2i0Wq18+eWXvPDCC4n2YL5w4QI9evTg1KlT5M6dm2rVqrFhwwbKli1738+NiCTNZoN166B+fdDFBJJe7ruIFhwcTFBQkEMvNDB6oh09ejTdgok8iDp3hu+/N2737eu6l3pGx0TT+YfO/LzvZ37t8iuPFXvM7EiSifTo0SPdjhUZGcmZM2eoesd1d/Hx8axfv56pU6cSExOT4ANOYGAg3t7eDsvLlCnDqVOnuHnzJj4+PgnOo7Fq3CtzWvJevGhhz554+ve3kT9/ozRnKXamGDVn1yTOGsf3Z7/nnTbvUDpf6XTLaxZ3y+xuecH9Mqclb1Lj1PTu3TvJyzfXrl2bYFm7du1o165dkuexWCwMHz6c4cOHJ7lNnjx5mD9//j3zenh43PNz0UcffcRHH310z2OISNpFRsLjj0NgILz3HvTsaXYiyQxSVUQ7efJkgumdz58/T3BwsC7nFEmDYcNg+XLjm5JHHzU7TdJmbJ7BT3uNMT46LO7AwdcPksU7db1tRJypUaNG7Nixw2FZt27dKF26dJJjnNWtW5f58+djtVrtvRj27dtHYGBgogU00Fg1t7hb5tTkvX21Vfp8pV2lcBUGhAxg1B+jiLXG0vuX3qztujbBl5Xgfs8vuF9md8sL7pc5NXnd6fGJiOuYM8f49+RJuKNjqkia3PdL6dbYZ3e7cuVKii+VEZHEVaoE8+fDpk3QsaPZaZLWv05/mjzShFx+uZjTdo4KaOKysmfPTvny5R1+smbNSt68ee2X8HTp0oXBgwfb93nllVc4f/48ffr0Yd++ffz88898+OGH9OrVy6yHIZnckEeH8HDuhwFYf3g9X0Z9aW4gERGRTKBlS2jTBnLkgHt0QhW5LynuiXbrMhWLxcLQoUMdxniJj49n48aN9plsRCT1nnzS7ATJ8/TwZMHTC/jv+n8Uz1Pc7DgiaXL3uDlBQUH88ssv9OvXj4oVK1K4cGH69OnDoEGDTEwprio+Hi5cgHz5Un+MLN5Z+LTFpzSb1wyAN8PepFXJVuTPmj+dUoqIiDx4mjUzfq5ehaxZzU4jmUWKi2hbt24FjJ5oO3bscLikxcfHh0qVKvHmm2+mf0IRYds2KF/etQbEzO2fm9z+uc2OIXLf7h4nJ7Fxc+rUqcNff/2VMYHELVmt8O23MHw4BAfDzz+n7XhNizelQ/kOfLPzG85fP8+bYW8yp+2c9AkrIiLyAFMBTdJTiotot2bl7NatG5MnT77vgZJF5P7ZbMbkAq+/Dm++Cf+bpd0l2Ww2JoRPoHyB8jQr3szsOCIiThUXB4MGwZEjsHs3bNwItdI4WfHEphNZcWAFF29cZO62uXSt1JWGwQ3TJ7CIiIiIpNl9j4n2xRdfqIAmkkH27oXXXjM+rI0eDatXm50ocTFxMTz/w/MMCBvAc4ueY8fpHcnvJCLixnx84J13jNt164LXfU/VlFBAtgDGhI6x3/+/Zf/HjbgbaT+wiIjIA+TPP2HnTrNTSGalOSpEXFjp0jBhgnG7Xz9o0MDcPEnx9vTmWuw1AKJjomk+rzlHLyU9tbuISGbwwgvGlxu//w7VqqXPMV+q+hIhQSEA7D+/n1G/u3AXZBERERfUty9UqGC0zdHRZqeRzEZFNBEX9/rr8McfMHFi+vR0cAYPiwdfP/k1NQrVAOD45eM0n9ecizcumhtMRMSJfHygYUNIZNLyVPOweDCj1Qy8PIw3/FF/jGLPuT3pdwIREZFM7O+/YfPm2/d1EZ2kNxXRRFycxWJcKuTqsvpkZVnHZTyS+xEAdp3dRdsFbYmJizE5mYiIeylfoDwDQgYAEGuNZd7OeSYnEhERcQ+FCsGUKUYvtK5dzU4jmZGKaCJu6PRp+Owzs1MkVCBrAVY+v5J8WfIBsO7wOrou6YrVZjU5mYiI8/3zz+1L8NNq6KNDqVe0HgufWcjwBsPT56AiIiKZXK5cxpjSmzdD795mp5HMSEU0ETezaZPxzUqPHvDtt2anSah4nuIs67AMfy9/ABbuWsjAsIEmpxIRca5334WSJY2ZlP/4I+3H8/f2Z/0L63m23LNY0vN6URERkQeEh6od4gR6WYm4mY0b4fhx4/bQocbMna6mVpFafNvuWzwsxlvMhPAJTPprkrmhRESc6KGHID7euD1xYvocU8UzEREREdeiIpqIm+nd27i+v149WLfOdScbaFWyFdNbTrff7/9Lf77b9Z2JiUREnKdLF6hRA4YMgXlOGsJs7va5TP5rsnMOLiIi4saio42x0M6eNTuJZHYu+vFbRJJiscD06Ub3ZB8fs9PcW49qPTgafZQR60dgw8bzPzxPwWwFefShR82OJiKSrry8YMMG53yxYbPZmHZ0Gr9E/YKHxYNKAZV4rNhj6X8iERERN7VoEfTpA2+8AVOnwssvm51IMiv1RBNxQ35+rl9Au+X9x96nW+VuANyMv0mbBW3YdWaXyalERNKfs3oGWywWcnjlAMBqsxJ2MMw5JxIREXFTc+ca/8bFQZUq5maRzE1FNJFMIC4O+veHBQvMTpKQxWJhRqsZNCveDICLNy7SfF5zjkcfNzmZiIhzRUfDzp3pc6z2Ae1pXbI1X7b5kg8afZA+BxUREckkPvkEBg6Ehg2N4RVEnEVFNBE3d/06NG0KH30EL74I27aZnSghb09vvmv3HVUDqwJwNPooLea34NKNSyYnExFJfzYbLFwIpUtD27bG+3RaeVo8WfT0IrpW7pr2g4mIiGQy5crBmDGwerUx/I2Is6iIJuLm/PygaFHjdlxc+vV6SG/ZfLLxc8efCc4VDMD209v5fOvnJqcSEXGOTz+Fkyfh4EEYPTp9jpnYbJ3XYq+lz8FFREREJFkqoom4OYsFpk2D1q1hzRro1MnsREkLyBbAyudXktc/L0MfHUq/2v3MjiQiku4sFqOI5uVlvDd36+ac83y761uCJwez4/QO55xARERERBxodk6RTMDPD376yewUKVMyb0l299pN/qz5zY4iIuI05crB9u1Qpoxzjv/trm95btFzADz17VNs7rGZnH45nXMyERERFzVjhgeFCkHLluDra3YaeRCoJ5pIJnb1qtkJEpdYAU3jo4lIZuOsAhpA65KtqRJgTD924PwBui7pitVmdd4JRUREXExMjAfvvOPB009DqVIQH292InkQqIgmkkktWgTBwRAVZXaS5K37dx0PT3mY73d/b3YUERGnsVrhxo30OZa/tz+Ln11Mbr/cAPy490fG/jk2fQ4uIiLiBiIjCxIdbYwX+vjj4OlpciB5IKiIJpIJ/fADtGsHZ8/Ck0/CuXNmJ0ra9tPbafJ1E85fP0/HxR2JOB5hdiQRkXS3YwfUrw9vvpl+xwzOHcy8p+ZhwfgA8c5v77D6n9XpdwIREREXVqvWSX7+OY6OHeHFF81OIw8KFdFEMqHmzaFmTeN2/fqQNau5ee6lQoEKtC/fHoCGwQ0pk9+J1z+JiJjgyhXjvXjDBmPCgYh0/K6geYnmvNvgXQCsNivtF7fn6KWj6XcCERERF+XpCY0b25g3z2hnRTKCimgimZCfHyxebHxYmzMH/P3NTpQ0i8XCZ60/46OmH/Fj+x/J5pPN7EgiIukqWzYYMsS4Xbw4xMam7/GHNhhKixItADh37RzPfPcMMXEx6XsSEREREVERTSSzKlIEXnkFLBazkyTP29ObvrX74u3pbXYUERGn6NMHPv7YmLEzJCR9j+1h8eCrJ7+iWK5iAGw6vol+v/RL35OIiIi4EJvN7ATyoFIRTeQBcukS7N1rdoqUORZ9jN7Le3Mz/qbZUURE0szbG3r3NnoKO0Me/zx8/+z3+HkZJ5i2eRpzt811zslERERMsn8/PPMMdOniyb59uVVMEwB2n93N++vfx5YBLwgV0UQeEHv2QK1a0KqVF9HRPmbHuaedZ3ZS5/M6fBLxCT2W9siQN0MREXdXJbAKn7b41H7/5WUvs+3UNhMTiYiIpK9Dh4xhaxYu9GDgwEdZtMgNLrsRpwo/Gk69L+rxwR8f8N3p75x+PhXRRB4QffoYvdAOH7bw6aeVzI5zT9Ex0Zy7ZkwpOnfbXIatG2ZyIhGR9HXhgnHJfVhY+h63W5Vu9KzaE4AbcTd46tunuHD9QvqeREREJAPcvGm0l3dq3BgqVIBcuWx06rSbZ57Rl+0PsqV7l9JobiPOXz8PwF+X/nL6uLAqook8IGbPhoAAKF/eRteuu8yOc08hQSHMf2o+FoxvlsZsGMOKcytMTiUikj7++QdKl4bp0+HVV+HGjfQ9/uTmk6leqLpxrgv/0GVJF6w2a/qeRERExEmuXIExYyA4GAYNclxnscCCBfDPP3G0a7fPLcZ/Fuf4fMvnPLnwSa7HXQegUbFGfFD8A3y9fJ16XhXRRB4QhQsbPR5+/z2OwMBrZsdJ1pNlnuTj5h/b7888NpN5O+eZmEhEJH0UKwalShm3T56ELVvS9/h+Xn4sareIvP55AVi2bxmjfh+VvicRERFxEqsVPvwQTpyAuXPhzBnH9WXLGjNfy4PJZrPxwfoPeGnpS8Tb4gHoUL4DPz73I/6e/k4/v4poIg+Q8uUha1azU6Rcr5q9GFTX+PrJho3uS7szb7sKaSLi3jw8YNo0ePZZY7zK9J6tE+ChXA/xzdPf2Hv0Dl0zlHX/rkv/E4mIiKTRf/853s+RA3r0MHqdNW9u9EwTAYi3xtN7eW+GrBliX9avdj++fuprfDwzZtxvFdFEHmBWK7z3Xvr3gkhPoxqN4pVqrwBgtVnpsqQL83fMNzmViEjalCsHCxdCkSLOO0fjRxoz4vERAHSq2IkahWs472QiIiL36aefoH59qFED4uIc1735pjGe8w8/wMMPm5NPXMuNuBs8t+g5Pt18exKlcY3HMbHpRDwsGVfaUhFN5AF14wZ06gTvvw+tWsHRo2YnSpzFYmFSk0k0y9sMMAppnX/ozIKdC0xOJiLi+gbXH8xP7X9ibtu5ZPHOYnYcERERu08/hT/+MGbc/OEHx3UBAVCihDm5xDW9tvw1Fu9eDICXhxdz287lzZA3MzyHimgiDyibDQ4fNm6fPg2//25unnuxWCz0LNKTnlWMGeesNiudvu/Ewp0LTU4mIpI+rFb47DO4eDF9j+th8aB1qdZYNPKyiIiYKLH27c3/1T/KlgV/5w9lJW5uaIOhFMpeiKzeWVnaYSmdK3U2JYeKaCIPKH9/+PFHqFgRliyBjh3NTnRvHhYPpjSbwsvVXgZuF9K+3fWtyclERNJm715jXLQePWDIkOS3T6uD5w/yw+4fkt9QREQkjXbvhu7djZ5lf//tuK5RI/jtN9ixw7gyRlzLuWvniLHGmB3DrmjOoqzstJI1XdfQrHgz03K4TBFt9OjRWCwW+vbtm6LtFyxYgMVioW3btk7NJZKZ5c9vjIfWurXZSVLGw+LBpy0/pUfVHgDE2+LpuLgj3+36zuRk4i7U1ogr8vc3PkCAMeHAv/8671xL9y6l2sxqdFjcgS0nXXhATBERyRRWroTZsyEmBiZOdFxnscDjjxsT7ojruBZ7jX4r+1FkchFe2PkCA34dwNmrZzM8x9aTW7kWe81hWYWCFUwf49UlXq4RERHMmDGDihUrpmj7f//9lzfffJP69es7OZlI5ufpmXDZhQsZnyOlPCweTG81nZeqvAQYhbQOizuw+O/FJicTV6e2RlxV0aLw7rtQpozxjXyxYs4718oDK7kUc4mY+Bje+e0d551IREQEoxdajhzGT2Cg2WkkOeFHw6kyowqTNk7CarNy3XqdmVtmZviwED/v+5m6s+vy3KLniLPGJb9DBjK9iHblyhU6derErFmzyJ07d7Lbx8fH06lTJ95//30e1jQdIukuLMyYAWexC9ekPCwezGg9gxcrvwgYhbT2i9uzZM8Sc4OJy1JbI66uXz+IioIGDZx7nolNJ1KrcC2eKfsM3z6jy+FFRMS5cuSAZcvg2DEYMcLsNJKUG3E3GBQ2iHpf1GPff/sA8PX0xdviTZeKXciXJZ/D9nvO7cFqszolS3RMNF2WdOF63HWW7VvGlI1TnHKe1PIyO0CvXr1o2bIloaGhjBw5Mtnthw8fToECBejevTu/p2Ak9JiYGGJibl/HGx0dDUBsbCyxsbH3nffWPqnZ1wzulhfcL3Nmyrt1K7Ro4UVcnIXnn7cRFBRHlSoZnTChpDJ/2vxT4q3xzNk+h+w+2QnMEugS/w/u9pqAtGV2h8fpTm3Ng/b6MYOr5rVYILFI6ZnXAw+WPbeMHL45sFgsTnsOXPU5Toq75QX3y5zZ2xkRSZo69bu2yBORdFnShb/P3h60rkahGnzW6jO2/rmVhvUaOmwfHRNNrc9qEZgtkLfqvcULlV9I1zw5fHOw+NnFNP26KU+WfpJeNXql6/HTytQi2oIFC9iyZQsREREp2v6PP/7g888/JyoqKsXnGDVqFO+//36C5atWrSJLltRP9R4WFpbqfc3gbnnB/TJnhrw2G9SrV5W1a4OoWPEUhw5FcvJkvAnpEpdY5jaWNvyX7z8ez/M4p7aeYvnW5SYkS5y7vSYgdZmvXbuW/EYmcte25kF5/ZjJ1fMeP56NwoWv2O+7et7EuFtmd8sL7pc5M7YzIiLu6Gb8TT5Y/wEf/P4B8TbjM5+3hzfvPfYeA+sOxBZv45D3IQKyBTjs9/mWz4mOiSY6Jpo/jvyR7kU0gMeKPUZ493AqB1TGw2L6BZQOTCuiHT16lD59+hAWFoafn1+y21++fJnOnTsza9Ys8uXLl+z2twwePJj+/fvb70dHRxMUFESTJk3IkSPHfeeOjY0lLCyMxo0b4+3tfd/7ZzR3ywvulzmz5Q0Nhdmz4+nZMx+enk1NSJhQcplb4VrT+bjbawLSlvlWrytX5I5tzYP2+jGDq+f97z94+21PvvjCg+XL42jQ4KZT8x6NPsprK19jStMpFM1ZNF2O6erP8d3cLS+4X2ZntDOffPIJ48aN49SpU1SqVImPP/6YmjVrJnmc7777jqFDh/Lvv/9SokQJxowZQ4sWLezrbTYb7777LrNmzeLixYvUrVuXadOmUaJECfs258+f57XXXmPp0qV4eHjw9NNPM3nyZLJlywYY42kGBwcnOHd4eDi1a9dOcRYRdzd2rDH2coMGULWqJg9wNdtPb6frkq5EnYqyL6sSUIU5bedQoWAFAGLjE+8FXCmgEg0easC6w+voX6e/w7qrN69yLPoYpfKVSnGWG3E3mLd9Hi9WedFh7LWqgVXv4xFlHNOKaJGRkZw5c4aqVW8/MfHx8axfv56pU6cSExOD5x0jnh88eJB///2X1ndMI2i1Gtfgenl5sXfvXh555JEE5/H19cXX1zfBcm9v7zT9wZHW/TOau+UF98ucWfJ6e8PrrwMkMuOAyVL6HNtsNvr90o9GwY1oXcq8qUfd7TUBqcvsyo/RnduaB+X1YyZXzbtyJXzxhXG7Tx8vIiNtgHPyRhyPoPm85vx3/T/+u/4f67utx8fTJ92O76rPcVLcLS+4X+b0amcWLlxI//79mT59OrVq1WLSpEk0bdqUvXv3UqBAgQTbb9iwgQ4dOjBq1ChatWrF/Pnzadu2LVu2bKF8+fIAjB07lilTpjBnzhyCg4MZOnQoTZs25e+//7Z/EdOpUydOnjxJWFgYsbGxdOvWjZ49ezJ//nyH8/3666+UK1fOfj9v3rz3lUXEndlsMGECnDkDOXMaXw6Ja4izxjHmjzG8v+59Yq1GkczLw4t36r/DO/Xfwdsz+ffnhsENaRjckP3/7adE3hIO676M+pLeK3rTumRr3qjzBo8+9Og9JyW4dOMSbRe2Ze2/azkWfYx3H3s3bQ8wA5hWD27UqBE7duwgKirK/lO9enU6depEVFSUw4cagNKlSyfY/oknnuDxxx8nKiqKoKAgkx6JSOZ39Sp8+qnRILo6q81Kr+W9mLxxMk9/+zTL9i0zO5KYSG2NuKMuXaBuXWMw5tdfN77ccJbieYqTw9foLbnx+Ebe+OUN551MJB1NnDiRHj160K1bN8qWLcv06dPJkiULs2fPTnT7yZMn06xZMwYMGECZMmUYMWIEVatWZerUqYDxBdykSZMYMmQIbdq0oWLFisydO5cTJ06wZMkSAHbv3s3KlSv57LPPqFWrFvXq1ePjjz9mwYIFnDhxwuF8efPmJSAgwP5zZyEwuSwi7m7vXqOABlCvntEjTcy3++xuQj4PYciaIfYCWrn85dj40kbee+y9FBXQ7nR3AS3eGs/EvyYCsHTfUh6b8xg1ZtXgmx3fJNqr7cTlEzz65aOs/XctAGM3jOVY9LH7f2AZzLSeaNmzZ0/wTUvWrFnJmzevfXmXLl0oXLgwo0aNws/PL8H2uXLlAtA3NiJOdPIktGoFW7bAxYvw9ttmJ7o3m83GpZhLgDFr56Ubl0xOJGZSWyPuyMMD5swBf38oVCjxyQbSS27/3Cx6dhEhn4cQEx/D1Iip1C1al/bl2zvvpCJpdPPmTSIjIxk8eLB9mYeHB6GhoYSHhye6T3h4uMNl9wBNmza1F8gOHTrEqVOnCA0Nta/PmTMntWrVIjw8nPbt2xMeHk6uXLmoXr26fZvQ0FA8PDzYuHEjTz75pH35E088wY0bNyhZsiQDBw7kiSeeSHGWxGgCG/fK7G55IX0zP/ywMWHZ7797UKSIjdjY9P8m3t2eY7PzTo2YyuDfBhMTb7yPeFg8eKP2GwyrPwxfL99Ec91v5pi4GF6u+jJTI6ZyNPooAJEnI+n4fUfe+vUtetfozYuVXySHbw72/reXVgtacfjSYQDy+efjx+d+pKB/wVQ/R2l9jlO6n+mzc97LkSNH8NDF0yKm2rjRKKABjBsHPXpA/vzmZroXTw9P5rSdg4fFg+bFm9OxQkezI4mLU1sjriiRq4adpmpgVT5u/jE9l/UE4KWfXqJSwUqUyV8m40KI3Idz584RHx9PwYIFHZYXLFiQPXv2JLrPqVOnEt3+1KlT9vW3lt1rm7svFfXy8iJPnjz2bbJly8aECROoW7cuHh4eLF68mLZt27JkyRJ7IS25LInRBDYGd8vsbnkhfTMX/d8wm8udOO+Xuz3HZuWNOh1lL6AV9i3M60Vfp9T1UqxetTrZfe8ncylK8VHwR/x58U9+PPMj/1z/B4Aj0UcYuHog7615j8fyPMbvF37ncvxlAAr4FODdh97lbNRZlkel/cWS2uc4pZPYuFQRbe3atfe8f7cvv/zSaVlExNC2LYwebVzOuXy5axfQbvHy8OKrJ78yO4a4KLU14q5u3HDepZ0vVX2JP47+wdxtc7kae5Wnv32aTT02kc0nm3NOKJJJ5cuXz6GXWY0aNThx4gTjxo1z6I12vzSBjXtldre84H6Zlff+NLE2Yf/X+6lZqCbDGwzH39s/2X3SkvkJnmC0bTTrDq9j0qZJLD9gFMeuWa+x/NztQlnFAhVZ2n4pgdkC7+8BpXNeSPlkaS5VRBMR1zRwILz8Mvzvqja39cXWLyicozBNHmlidhQRkRSzWuGnnx6mVy8vNm0yLvFMbxaLhWktp7H15FZ2nNnB7nO76bm0J/OemnfPAYFFzJAvXz48PT05ffq0w/LTp08TEBCQ6D4BAQH33P7Wv6dPnyYwMNBhm8qVK9u3OXNroKf/iYuL4/z580meF6BWrVoOPSOSy5IYTWBjcLfM7pYX3C+z8ib078V/WfvvWl6o/MLt8+LN2hfWpmryoLRkblyiMY1LNGb32d189NdHzN02194j7rFij7HkuSXk9MuZqmMnJbV5U7qPrl8RkWRZLIkX0OLjMzxKqn2+5XNe/OlF2ixoQ9hB9+r2LSIPtrFjPZg9uwLHj1t44QWjqOYMWbyzsOjZRWT3yQ7ANzu/Ydrmac45mUga+Pj4UK1aNVavvn0ZktVqZfXq1dSpUyfRferUqeOwPRiX/NzaPjg4mICAAIdtoqOj2bhxo32bOnXqcPHiRSIjI+3b/Pbbb1itVmrVqpVk3qioKIfCXHJZRNzZ77/D11/D0aNmJ3kwzd46mwrTKtD9p+5sPLbRYV16zr59v8rkL8PM1jM53PcwY0LHMDZ0LCs7rUz3AlpGUBFNRFJlyhQIDYU7xrh1WTabjV8O/gLAjbgbPLHgCX7951eTU4mIpEyPHlby5LkOQPnyzv0Co2Teksxuc3t2w74r+7Lp+CbnnVAklfr378+sWbOYM2cOu3fv5pVXXuHq1at069YNMCaNuXPigT59+rBy5UomTJjAnj17eO+999i8eTO9e/cGjN6Yffv2ZeTIkfz000/s2LGDLl26UKhQIdq2bQtAmTJlaNasGT169GDTpk38+eef9O7dm/bt21Pof11E58yZwzfffMOePXvYs2cPH374IbNnz+a1115LcRYRdzZjBnTubIyHFhFhdpoHz8nLJ7ly8wpWm5Vha4eZHSeBgtkKMrDuQAbUHYCvV8Lete5ARTQRuW/vvQd9+sDatfDii2BL/wl30pXFYmHeU/N4srQxa9aNuBu0/qY1q/9JfiBNERGz5c0L/fptYfnyOCZOdN64aLc8U/YZ+tXuB0CsNZZ237Xjv2v/OfekIvfpueeeY/z48QwbNozKlSsTFRXFypUr7QP2HzlyhJMnT9q3DwkJYf78+cycOZNKlSqxaNEilixZ4jDz8sCBA3nttdfo2bMnNWrU4MqVK6xcuRI/Pz/7NvPmzaN06dI0atSIFi1aUK9ePWbOnOmQbcSIEVSrVo1atWrx448/snDhQntxL6VZRNyRzWZ8PgDIkgX+dyW0ZKBB9QZRo1ANulfpznftvjM7TqakMdFE5L61agVjx8L16xAcbDSYrj5kjrenNwueWcBzi55jyZ4l9kLaso7LaBjc0Ox4IiL3VKHCOUJDM+4bizGhY9h4fCMbjm7gyKUjdP6hM8s6LsPDou9fxXX07t07yd5biU0a065dO9q1a5fk8SwWC8OHD2f48OFJbpMnTx7mz5+f5PquXbvStWvXpEOnMIuIu5o7F9atg2vXnP+lz4MszhrHor8X8dexv5jUbJJ9uZeHF+teWJeiiQMkdfSXkIjct+rV4Ztv4PPPYeRI8HCTdxIfTx8WPrOQNqXaAHA97jqt5rdi+X4nzrstIuIkzryc3tvTm2+f+Zb8WYwpmVccWMGHv3/ovBOKiIjbs1igYUN4/30YN87sNJlTdEw0E8Mn8siUR+iwuAOTN05m26ltDtuogOZcbvLRV0RcTZs2xqWc7sbH04dv231L65KtAaOQ9sQ3TzAzcmYye4qIuI7ffoMSJWC1E69KL5yjMN88/Q0WjK7Gw9YM03iSIiIiJjh66SgDVg0g6KMg3lj1BkcuHbGv++5vXbaZkVREE5F0c/gw7Nljdork+Xj68F2772hX1riMIt4Wz8vLXuad1e9gc/UB3kTkgbd+PTRqZMx81rUrnD/vvHM1ergRwx83Lm2zYaPD4g4ciz7mvBOKiIiI3ZaTW+j0fScenvIw48PHEx0TbV/XskRLfuvyGyMeH2FiwgePimgiki42b4ZataBZM/eY0trXy5cFzyzgjTpv2Jd9+MeHdP6hMzfjb5qYTETk3urVM4poAKVLw00nv2W9Xf9tWpRoAUA2n2xcuH7BuScUERG3c/YshIXB1atmJ3F/VpuVn/f9TMM5Dak2sxrzd8wnzhoHgK+nLz2q9uDvV/9mWcdlPB78OBZXH5w6k1ERTUTSzGaDvn3h9GmjN9obbyS7i0vwsHgwvsl4pjSbYr9cad6OeTT7uhkXb1w0N5yISBI8PGDOHJg0CVatgoAAJ5/P4sFXT35Fj6o9iOwZSYWCFZx7QhERcTsrVkCTJpArF8yaZXYa93Qj7gafbfmM8p+Wp9U3rVjz7xr7unxZ8jHs0WEc7nuYma1nUiZ/GROTPtg0O6eIpJnFAosWQf36kC8fzHSz4cVeq/UaRXIUoeP3HbkRd4M1/66h3ux6rOi0gqCcQWbHExFJoHBh6NMn486Xxz8PM1u72Zu7iIhkmHXrjH/j4owxOyXlzl07x6cRn/JJxCecuXrGYV2JPCV4o84bdKnURRMGuAgV0UQkXQQEwJo1xrdP2bKZneb+PVnmSdZ0XUPrb1pz7to5dp3dRe3Pa/Nzx5+pHFDZ7HgiIsmKjzf+9fTMmPPFWeM4e/UsgdkDM+aEIiLisp5+Gry9ITzcGOJFUuarbV/x8rKXuR533WF5/aL1eTPkTVqVbIWHRRcQuhL9b4hIuilSJGEBLT4erl9PfHtXU7tIbcK7h1M8T3EATlw+wbSIaSanEhFJ3r//wmOPwahRGXO+U1dOETo3lCZfN+HqTQ2AIyLyoGvRAqZPh23bwF8dplKsSmAVewHN0+LJc+WeY9NLm1jfbT1PlHpCBTQXpJ5oIuI0sbHQuTNcugRLloCvr9mJklc8T3E2vLiBJxY8gb+XPx+3+NjsSCIi93TxIlStChcuGD0AmjSBmjWde87nv3+edYeNa3deW/Eas9vMdu4JRURE3FicNY4/LvzBzb03aVe+nX15+QLleabsMxTJXoQ+tftQLFcx80JKiqiIJiJO8+KLsHChcbtrV1iwwNw8KZU/a35+6/IbN+Nv4uPpY3YcEZF7ypULeveGESMgKAisVuef8+PmH1NjVg1y+eXipaovOf+EIiIiburk5ZM0n9ecbae3UTy6OE+VfQpPj9tjL3z7zLeaYdONqG+giDhNz55Gd25fX6NHmjvx9/Ynp19Oh2V7z+2lz4o+xMbHmpRKRCRxQ4fCe+9BVBTUru3885XJX4afOvzElpe3EBIU4vwTioiIy/r774z5Asdd5c+aH28PbwAOXDjA6kOrHdargOZeVEQTEaepXx9+/BGWL4eWLc1Okzanr5ym+bzmTNk0hVbftCI6JtrsSCIidt7e8O67kDNn8tuml4bBDSmQtUDGnVBERFzOyZNQrhzkywdvv212Gtfk5eHF560/p7h/cVZ0WEHjhxubHUnSQEU0EXGqxo2hYUOzU6TdttPbOHH5BGAU1ERE3EFsBnacjbfGM2/7PKw2dUcQEXlQrDOGx+TCBXNzuIp4azzjN4wn4niEw/Iy+cowruQ4GgU3Us8zN6cimohkuBUr4IMPzE5xf5o80oRfu/xK5YDKLO+0nBy+OcyOJCKSpGvX4NVXoU0bsNmcf75z187RYn4Lnv/hecb+Odb5JxQREZdQoAA8+STkyQMNGpidxlz7/9vPo18+yoCwAXRd0pUbcTcc1qt4ljloYgERyVCLF0OHDkbvCD8/eOMNsxOlXL2i9YjsGZlgqul4a7zD4KAiImZr1QrWrDFuf/op9Orl3PNFnYoi7GAYAO/89g7VClZz7glFRMQlNGxo/FitGfOljSuy2qxM3TSVt359i+tx1wHYc24Pv/7zK61KtjI5naQ39UQTkQx17Njty4siItyvsb27gHY55jIhs0P4attXJiUSEUlo4EDj31uTuzhb6MOhDGswDDA+TDy/5HnOx553/olFRMQleHiA5wP4nfKhC4doNLcRfVb2sRfQHsn9COteWKcCWialnmgikqH69IHLl+Gff2DWLHDnXs2x8bG0+64dm45vosvxLhy5dIS362tEVRExX7Nm8NFHxr+lS2fMOYc+OpTwY+GsOriKM9fOMPrQaEKvhBKUOyhjAoiIiGQQm83GrC2zeGPVG1y5ecW+vFeNXowJHUNWn6wmphNnUk80Eclw77wDn3/u/t9WWSwWHs79sP3+kDVDeHnZy8RZ40xMJSJi6Ns34wpoAJ4ensx7ah5FchQBYN+1fVT7rBorD6zMuBAiIpJhrl0zO4E5jkUfo/m85ry87GV7Aa1ozqL82vlXpraYqgJaJqcimohkOIslYQ+0ixfhjz9MiZNqXh5efNLiE0Y3Gm1fNmvLLJ7+7mmux183MZmISOLinFzjz5clHz+2/5HAbIEAnL12lubzmjNg1QBuxt907slFRCRD1a4NpUrBK6+43xAtqWGz2fgy6kvKf1qeXw7+Yl/+UpWX2PHKDho93MjEdJJRVEQTEdOdO2cMSNq48e1pst2FxWJhUL1BzH9qPj6ePgCsOLiCdw68w4nLJ0xOJyJisNlg5kyoUAEuXHDuuaoGVmVz981Uz1Hdvmx8+Hjqza7HwfMHnXtyERHJEOfOwY4dsG+fMc6xOw/RkhInL5+kzYI2dPuxG5diLgFQKHshlndczqwnZpHDN4fJCSWjqIgmIqYbOxa2boUbN+DFF29PPOBOOlTowKrnV5HLLxcA/1z/hxqf1+DXf341N5iICDB8OLz8MuzZ4/yZOgHyZ83PO8HvMD50PN4e3gBEnIigyowqfLPjG+cHEBERpzp7FkJCwNsbGjQwO43z2Gw2vtnxDeWnlWfpvqX25V0qdWHnKztpXqK5ienEDCqiiYjpRo6EFi0gMBCWLjUaY3fUoFgD/nzxTx7K+RBgXMbU5KsmvLf2PeKt8SanE5EH2YsvQq5cxu3s2Z1/WScYPXVfr/k6f730FyXylADg8s3LdPy+Iy/++CJXb151fggREXGKMmXgzz+NIVkGDzY7jXOcvXqWdt+1o+P3HTl/3ZhxumDWgix5bglz2s4ht39ukxOKGVREExHT+fjAokUQHg5ly5qdJm3K5i9LeLdwqmavCoANG++ve5+mXzfl9JXTJqcTkQdVUBDMng0//ggzZoBXBs7PXjWwKpE9I+lcsbN92RdRX1BtZjWiTkVlXBAREUl3WbJAvnxmp3COGZEzWLx7sf3+c+WeY+erO2lTuo2JqcRsKqKJiEvw94eHHkq4/KobdlTIlyUfQx4ewsjHRuJhMd5mVx9aTZUZVVj3r5sN+iYimcaTT8ITT5hz7uy+2Zn75Fzmtp1LVm9j1rK9/+2l1me1WPz34mT2FhERyXgD6w6kYsGK5PXPy7fPfMuCZxaQL0smrRhKiqmIJiIua/x4qFQJTrjh+PweFg8Ghgzkty6/2WepO3nlJA3nNmTU76Ow2qwmJxQRgfgMvtK8c6XObH15K1UDjd66/l7+VCtULWNDiIiIJCLO6jjWgY+nD9+1+45dr+6iXbl2JqUSV6Mimoi4pE8+gQED4OBBaNbMixs3PM2OlCoNijVg68tbCX04FACrzcrbv71Nq/mtOHftnMnpRORBFh4O5ctDZGTGnrdE3hJseHED/Wr3Y1brWRTLVSxjA4iISJq88w7Urw9DhsCZM2anSR9/HvmTsp+UZe+5vQ7LS+YtScFsBU1KJa5IRTQRcUlt20JwsHG7Y0crfn7uOzB/wWwFWdlpJe81eA8Lxvzfm45v4nrsdZOTiciDavVqqFfPmK3zqafg+PGMPb+vly8Tm05M8M3+5ZjLfBT+UYLeACIi4jp++QX++AM++AA8MkFFYf3h9YR+Fcr+8/tp+nVTTl4+aXYkcWGZ4CUvIplR4cLGh7wZM+Ctt9z/0kdPD0/efexdVnVeRcGsBfnqya8IyhlkdqwHzujRo7FYLPTt2zfJbWbNmkX9+vXJnTs3uXPnJjQ0lE2bNmVcSJEMUL8+VDWuqKRCBdcZFLr3it70X9WfhnMaciz6mNlxRETkLrGxcOWKcbt8eddpP9KiUsFKlMpbCoBS+UqR3Te7yYnElamIJiIuKzgYevZMuNzqxjW10IdDOfj6QZqXaO6wPDommgvXL5iU6sEQERHBjBkzqFix4j23W7t2LR06dGDNmjWEh4cTFBREkyZNOJ7RXXVEnMjHB376CV5/HRYvBl9fsxPBzjM7mbd9HgBRp6K4EXfD5EQiInI3b2+jF/OxY/DFF2anSR85/XKyotMK3qjzBks7LCWbTzazI4kLUxFNRNzKrl3GZAM7dpidJPWy+mR1uG+z2Xjpp5eoOrMqEccjTEqVuV25coVOnToxa9YscufOfc9t582bx6uvvkrlypUpXbo0n332GVarldWrV2dQWpGMERgIkye7RgENoHyB8qx7YR1FcxZleqvpFM9T3OxIIiKShMKFoXp1s1OkTrw1PsGX14HZAxnfZDw+nj4mpRJ34WV2ABGRlDp0CJo0MWbrbNAA1qwxCmrubmbkTL77+zsAWn3TikN9DpHFO4vJqTKXXr160bJlS0JDQxk5cuR97Xvt2jViY2PJkydPktvExMQQExNjvx8dHQ1AbGwssbGx93W+W9vf735mcrfMypu0q1ehXz9P3nsvnkKFUn+c1GauGViTbT22kdUnq8O+V29e5fjl45TMWzL1oe7B3V4T4H6Z05LXXR6jiLi+67HX6fh9R45cOsLarmt16abcNxXRRMRt5MgBRYoYRbRHHrk98YC7a1GiBXWK1CH8WDgzWs1QAS2dLViwgC1bthARkbpefoMGDaJQoUKEhoYmuc2oUaN4//33EyxftWoVWbKk7v8zLCwsVfuZyd0yK6+j69c9GTGiNn//nY+wsKuMGPEnefLEJL/jPaRX5k+OfML6i+vpWbgnDfM0xGKxpMtx7+Zurwlwv8ypyXvt2jUnJBGRB82F6xd4YsET/HHkDwDaL27Psg7LnNamSOakIpqIuI28eeHXX+HNN43ZgHLkMDtR+gjKGcS6F9ax4sAKnij1hNlxMpWjR4/Sp08fwsLC8PPzu+/9R48ezYIFC1i7du099x88eDD9+/e334+OjraPpZbjPl+osbGxhIWF0bhxY7y9ve87sxncLbPyJu7YMbh+3fjT8MqVbJQt28g++cD9Ss/MS/ctJSzKKLx8fPRjzmQ/w5SmU8jtf+9Ls++Hu70mwP0ypyXvrd69ImKu9ethyhTjipA2baBoUbMTpdyx6GM0+7oZu87uAiCbTzb61OqjAprcNxXRRMStZM9uzNh5N5sN3LkN9Pb0TlBAs9lsvLzsZWoUqsFLVV9SI58KkZGRnDlzhqp3VALi4+NZv349U6dOJSYmBk9Pz0T3HT9+PKNHj+bXX39NdjICX19ffBMZWMrb2zvVH27Tsq9Z3C2z8joKDjYuk3/ySZg500L16mk/V3pkblayGT2q9mDWllkALPx7IeuPrGdm65m0KtkqzRnv5G6vCXC/zKnJ606PTyQzW7nSmIxm8WLIn999imh/n/2bZl8342j0UQDyZ8nPik4rqFaomsnJxB1pYgERcXtxcfDcczB1qtlJ0tfnWz9n1pZZ9FzWk84/dObKzStmR3I7jRo1YseOHURFRdl/qlevTqdOnYiKikqygDZ27FhGjBjBypUrqe6uo+aKpMJDD0FkpGsNFp3FOwszW8/k22e+JadvTgBOXjlJ629a03VJV81sLCKSQTZvvn27QQPzctyPDUc3UG92PXsB7eHcD7Oh+wYV0CTVVEQTEbdms0HPnvDdd/DaazB2rNmJ0s/ec3vtt+ftmEeNWTXYeWaniYncT/bs2SlfvrzDT9asWcmbNy/ly5cHoEuXLgwePNi+z5gxYxg6dCizZ8+mWLFinDp1ilOnTnHlioqY8mBIrNPr4sXw338Zn+VO7cq1Y9eru2hevLl92dxtcyk/rTw/7/vZxGQiIg+GFStgyxb47DNjhmdX99Pen2g0txEXbhhftlQNrMqGFzdo9mdJExXRRMTt3WrEvb0hmavu3Mq4JuNY+MxCsvsYswbtObeHmrNqMn3zdKw2q8npMo8jR45w8uRJ+/1p06Zx8+ZNnnnmGQIDA+0/48ePNzGliHlmzYJnnoHQUDh/3twshXMU5ueOPzP7idnk8DXGGzxx+QStvmlFtx+7cfHGRXMDiohkYp6eUKUKdO9udpLkfbblM55c+CQ34m4AEPpwKGu7rqVgtoImJxN3pzHRRMStWSzGJAO5ckGxYtCsmdmJ0tez5Z6lSkAV2n3Xjm2nt3E97jqv/PwK83fMZ1brWZTKV8rsiG5n7dq197z/77//ZlgWEVd35Qrcmng2KgrmzoW+fc1MBBaLhW5VuhH6cCg9lvbgl4O/APBl1JeEHQxjVutZNC/RPJmjiIhIZmSz2fjg9w8YumaofVmH8h34su2X+Hj6mJhMMgv1RBORTGHAAGjXLuFymy3js6S3EnlLEN49nJervWxf9vuR36k4vSIfrP+Am/E3TUwnIplZtmywejUEBBjvs336mJ3otqCcQazotILPWn9m75V2/PJxWsxvQfcfu3PpxiWTE4qISEaKt8bTa3kvhwJav9r9+Pqpr1VAk3SjIpqIZFpLlxrTb1+7ZnaStPP39md6q+mEdQ7j4dwPA3Az/iZD1gyh+szqbDq+yeSEIpJZlSpl9EIbM8b1ZkG2WCx0r9qdna/spMkjTezLZ0fNpvy08vxy4BcT04mIZA7Xr0Pv3vDtt3D6tNlpEncj7gbPLnqWaZun2ZeNazyOiU0n4mFR2UPSj15NIpIprV1r9ExbuhSaNjUuScoMQh8OZccrOxgQMsD+B8GOMzuo/Vlt+q3spxk8RcQpChZMWEA7fhyio83Jc7egnEGs7LSSma1m2seRPBZ9jGbzmjHpr0nmhhOn+eSTTyhWrBh+fn7UqlWLTZvu/YXSd999R+nSpfHz86NChQosX77cYb3NZmPYsGEEBgbi7+9PaGgo+/fvd9jm/PnzdOrUiRw5cpArVy66d+/uMPHM2rVradOmDYGBgWTNmpXKlSszb948h2N8+eWXWCwWhx8/P780PhsizrNxI3zyCTz3HNwxF5PLuHjjIk2/bsr3u78HwMvDi7lt5/JmyJsmJ5PMyGWKaKNHj8ZisdD3HgNtzJo1i/r165M7d25y585NaGhoso2liDyYPD3B19e4HRQEWbKYmyc9ZfHOwtjGY4noEUHlgMoA2LAxaeMkyn9anpUHVpobUEQyvSNH4NFHoXlzuHzZ7DQGi8VCj2o92PnqTkIfDgUgq3dW2pRqY3IycYaFCxfSv39/3n33XbZs2UKlSpVo2rQpZ86cSXT7DRs20KFDB7p3787WrVtp27Ytbdu2ZefO27Nejx07lilTpjB9+nQ2btxI1qxZadq0KTdu3LBv06lTJ3bt2kVYWBjLli1j/fr19OzZ0+E8FStWZPHixWzfvp1u3brRpUsXli1b5pAnR44cnDx50v5z+PDhdH6GRNLPunW3bzdoYF6OxNhsNprPa876w+sB431/aYeldK7U2eRkklm5RBEtIiKCGTNmUDGZafXWrl1Lhw4dWLNmDeHh4QQFBdGkSROOHz+eQUlFxF3Urw9r1kDXrjBnDni4xLtd+qoaWJVNL21iTOgY/LyMb7APXzpM83nN6fxDZ85fN3kaPRHJlGw2Y7bOf/6BDRugVy+zEzkqmrMoq55fxYxWM5jSfArBuYPNjiROMHHiRHr06EG3bt0oW7Ys06dPJ0uWLMyePTvR7SdPnkyzZs0YMGAAZcqUYcSIEVStWpWpU6cCxgfxSZMmMWTIENq0aUPFihWZO3cuJ06cYMmSJQDs3r2blStX8tlnn1GrVi3q1avHxx9/zIIFCzhx4gQAb7/9NiNGjCAkJIRHHnmEPn360KxZM77//nuHPBaLhYCAAPtPwYKaMVBcV//+sGIFvPUWNGpkdhpHFouFYY8Ow9PiSb4s+VjTdQ3NimeymcbEpZg+O+eVK1fo1KkTs2bNYuTIkffc9u6u0J999hmLFy9m9erVdOnSxZkxRcQNVa0KX35pdgrn8vb0ZmDdgTxV5il6Lu3Jmn/XALDq4CpsmWFWBRFxORYLzJwJDRtCvnwwapTZiRKyWCz0rNYzwfIrN6/Qan4rhjw6xN5bTdzPzZs3iYyMZPAd15V5eHgQGhpKeHh4ovuEh4fTv39/h2VNmza1F8gOHTrEqVOnCA29/brImTMntWrVIjw8nPbt2xMeHk6uXLmoXr26fZvQ0FA8PDzYuHEjTz75ZKLnvnTpEmXKlHFYduXKFR566CGsVitVq1blww8/pFy5ckk+5piYGGJiYuz3o/93LXVsbCyxsbFJ7peYW9vf735mcrfM7pYX7p3Zz88ont0qoLnCw7ozb2ixUL5q+xWVClaiRJ4SLvm8Z7bXhCtKa96U7md6Ea1Xr160bNmS0NDQZItod7t27RqxsbHkyZMnyW3Ss8G5td+d/7o6d8sL7pdZeZ0vPTNfuADPP+/JqFHxJNP5NdXMeI4fyv4QKzusZM72OQxcPZCJjSeSwztHijOkJbM7vZZEJH1Urgy//QYFCkChQmanSbmBYQNZd3gd675ax+Rmk3m91utmR5JUOHfuHPHx8Ql6bxUsWJA9e/Ykus+pU6cS3f7UqVP29beW3WubAgUKOKz38vIiT5489m3u9u2339qvurmlVKlSzJ49m4oVK3Lp0iXGjx9PSEgIu3btokiRIokeZ9SoUbz//vsJlq9atYosqRyzIiwsLFX7mcndMrtbXnC/zLfyZiEL+w/tZz/7k9nDXO72/IL7ZU5t3mspnI3O1CLaggUL2LJlCxEREanaf9CgQRQqVMjhG6O7OaPBgQfnhWQmd8usvM6X1sw3bnjy7rsh7N2bhw0b4nj33b8oWfJCOqVLyIznuAAFmFJ8ClkPZWX5v7cHTL4Qe4E/L/5J83zN8bR4Jrl/ajKntMERkcylcuWEy2w2uHnTNS+hj42PZd9/+wBjbMmWJVqanEgyuzVr1tCtWzdmzZrl0MusTp061KlTx34/JCSEMmXKMGPGDEaMGJHosQYPHuzQky46Oto+tE2OHDnuK1dsbCxhYWE0btwYb2/v+3xU5nC3zO6WF9wrc49lPahTuA75T+SnWZNmLp8X3Ov5vcXdMqc1b3QKZ0syrYh29OhR+vTpQ1hYWKpmoxk9ejQLFixg7dq199w/PRscePBeSGZwt8zK63zplfnCBfjoI6OAlCWLNy1a1KFkyfRKeZsrPscdvu/A4uOLibJG8VXbr3gk9yMO69OSOaUNjohkbjYb9OsHu3bB4sVmp0nI29ObVZ1XMX3zdLw8vHgkzyPJ7yQuKV++fHh6enL69GmH5adPnyYgICDRfQICAu65/a1/T58+TWBgoMM2lf9XMQ4ICEgwcUFcXBznz59PcN5169bRunVrPvroo2SHnfH29qZKlSocOHAgyW18fX3xvTVj0l37pvZvjbTsaxZ3y+xueSFh5q++gqJFoVYt47JOs/1y4BfmbJ/DnO1zqJmjJq29W7vVc5wZXhOuLrV5U7qPaUW0yMhIzpw5Q9WqVe3L4uPjWb9+PVOnTiUmJgZPz8R7S4wfP57Ro0fz66+/JjsZgTManPTYP6O5W15wv8zK63xpzVygAPz6K7zwAgwZYqFcOec+fld5jnee2cn3e4wBjf+99C/5suVLMldqMrvCYxQR8w0eDJMnG7effdaT//s/c/MkxsPiwas1Xk2w/OrNqww9MJSoP6LoXLkzD+d+2IR0klI+Pj5Uq1aN1atX07ZtWwCsViurV6+md+/eie5Tp04dVq9eTd++fe3LwsLC7D3CgoODCQgIYPXq1faiWXR0NBs3buSVV16xH+PixYtERkZSrVo1AH777TesViu1atWyH3ft2rW0atWKMWPGOMzcmZT4+Hh27NhBixYt7vepEHGq2Fj4v/+Da9fgkUfgHnXeDPPrP7/ab9fPXd/EJPKgMq2I1qhRI3bs2OGwrFu3bpQuXZpBgwYlWUAbO3YsH3zwAb/88ovDoJ4iIimRLRssWpRweXy80YvCy/SRItNf+QLl+b3b77y09CWG1B9Cviz5zI4kIpnQE0/AJ5/A1avwzDNWLBazE6XckLVD2HFlBzvW7+C99e9Ru0htOlXoxLPlnqVA1gLJ7i8Zr3///nTt2pXq1atTs2ZNJk2axNWrV+nWrRsAXbp0oXDhwoz638wXffr0oUGDBkyYMIGWLVuyYMECNm/ezMyZMwFjMoq+ffsycuRISpQoQXBwMEOHDqVQoUL2Ql2ZMmVo1qwZPXr0YPr06cTGxtK7d2/at29Pof8NDrhmzRpatWpFnz59ePrpp+1jpfn4+NjHcR4+fDi1a9emePHiXLx4kXHjxnH48GFeeumljHwKRZK1ebNRQAOoXdvcLLeMazKOliVbMjdqLvWoZ3YceQCZ9nExe/bslC9f3mFZ1qxZyZs3r3353Y3fmDFjGDZsGPPnz6dYsWL2Rilbtmxky5YtYx+AiGQqb78NUVGwYAHkzm12mvRXt2hdtv3fNrw9HHuNnbl6hlG/j+Kduu+YlExEMouQEFi5Ev75B9q3t7F8efL7uILY+Fj+ufCPw7K/jv3FX8f+ou/KvoQ+HEqnCp1oW7ot2X2zm5RS7vbcc89x9uxZhg0bxqlTp6hcuTIrV660Twxw5MgRPO4YnC8kJIT58+czZMgQ3n77bUqUKMGSJUscPo8MHDiQq1ev0rNnTy5evEi9evVYuXKlw9Ax8+bNo3fv3jRq1AgPDw+efvpppkyZYl8/Z84crl27xqhRo+yfYQAaNGjA2rVrAbhw4QI9evTg1KlT5M6dm2rVqrFhwwbKli3rrKdLJFVKljRmul+3Dpo3NzvNbY8Ve4y6heuy3F0aGslUXLrPxd2N37Rp07h58ybPPPOMw3bvvvsu7733XganE5HMYt48GDvWuF23rlFM8/ExNZJT+HgmfFB9V/blm53fsHDXQt4pokKaiKRN3brGz92T9l66BDlzmpMpOd6e3vz43I98ueRLzhY8y4JdC9hxxrhaIt4Wzy8Hf+GXg7/g7+XPE6WeoGOFjjQr3izR91TJWL17907y8s1bBas7tWvXjnbt2iV5PIvFwvDhwxk+fHiS2+TJk4f58+cnuf7LL7/kyy+/THI9wEcffcRHH310z21EXEHevNC1q/EjIgaXKqLd3djdff/ff//NsCwi8uAoUgTy5YNz5+DVVzNnAS0xRy8dZem+pQDky5KPAN/EB2MWEUmLdeugTRuYPRueesrsNEkr4FOAF+q8wNuPvs2O0zuYv2M+83fO58ilIwBcj7vOwl0LWbhrIbn9ctOubDs6VexEvaL18LC44HSkIiKZxInLJ/jjyB+0K9sOizuNFSCZklp8EXngNWgAERHwwQfQq5fZaTJOUM4gdr26i9YlWzOjxQy8LC71vYqIZAKHDhmFs0uX4OmnISzM7EQpU6FgBUaFjuJQn0P83u13Xqn+Cnn989rXX7hxgZlbZtLgywY8NOkhBoYNZMfpHfc4ooiIpNawNcN4btFzhMwOYd9/+8yOIw84FdFERIBixYxx0e7+cmvTptsDqmZGRXMW5acOP1G9kCZqEZH0V6gQtGxp3G7WDB5/3Nw898vD4kG9ovX4tOWnnHzjJMs6LKNjhY5k8c5i3+ZY9DHGbRjHx5s+NjGpiEj62rYNdu82Jt4y0/bT25m9dTYAf5/9mzz+ecwNJA88FdFERJKwcyc0agSPPgrHjpmdRkTE/fj6wpw5MG0aLFzo3jMge3t607JkS+Y9NY/Tb55m3lPzaFGiBZ4WY0b5ThU6OWx/4foFPtn0CWevnjUjrohImrzzDpQtCwEBcOKEeTkGhg3EhlHJe6f+O5plXkynIpqISCKsVnj+ebhyBSIj4cMPzU4kIuKeLBb4v/+DHDkcl585A/v3m5MprbL5ZKNjhY783PFnTr5xklmtZ1H/ofoO2yz6exG9V/Sm4PiCfLvrW5OSiojcv/h4+P332/cDA83J8csBY2IXgIdyPsTrtV43J4jIHdz4+0AREefx8ICvvjIGw86VC8aPNzuRiEjmcf268f66bx98/70xNqW7yp81Py9VfSnB8vk7jRkcbdgokaeEw7qwg2H0/aUv1QtVp1pgNaoFVqNyQGWy+mTNkMwiIvdy86bRE23dOqMnmhlj+cdb4xkQNsB+f1SjUfh5+WV8EJG7qIgmIpKEChWMMdFu3IAsWZLfXkREUmb4cPjrL+N2t26wZ0/mmxl5crPJzN8xnzX/rqFcgXIO6zYd38TfZ//m77N/M3fbXMAYf61MvjJUK1SN6oHVqVbIKKzdOf6aiEhG8PeHgQONH7PM2TaHHWeMCVtqFKrBc+WfMy+MyB1URBMRuYd8iQy7cPkyDBoEI0dCHo1tKiJy3956C7ZuhT/+MHqiZbYCGkDFghWpWLBiouvOXz+Pl4cXcdY4+zKrzcqus7vYdXZXgsKavceaCmsi8gC4evMqQ34bYr8/vsl4PCwaiUpcg4poIiL3wWqFzp3hxx9h1SpYuhTKlDE7lYiIe8mZE5YtMyZwqVzZ7DQZb0LTCXzY6EN2nNnB5hObiTwRSeTJSHac2ZFkYW3OtjmAUVgrm78sVQpWobGtsVkPQUTEaSaET+DklZMAtC3dlkcfetTkRCK3qYgmInIfDh+G8HDj9n//gaenuXlERNyVl1fiBbSpU+GZZ4xxeDIzXy9fqheqTvVC1e3LbsTdYMfpHUSejDSKaycj2XlmZ4LC2s4zO9l5ZietKrZyOObUTVNZ9PciSuQpwYC6AyiZt2SGPR4Am82G5Y7BkyJPRDIjcgbHoo/h4+FDtyzdMjSPiNy/ixeNsYHz5jXn/Ccvn2Tsn2MB8PLwYkzoGHOCiCRBRTQRkfsQHAwREfD00/DBB1AyYz+fiIhkajNnwmuvwbhxRk+1ChXMTpSx/Lz8qFG4BjUK17Avu1VYu1VUu1VYC8gagK+Hr8P+ESciWHd4HesOr6N3zd4O61bsX8G4DeMonqc4JfKUoETeEhTPU5xHcj+Cv7d/stkux1zmWPQxSucr7VAom7F5Bh9v+phj0cdY1nEZ9YrWs687deUUs7bMAiCnb066lVERTcTVff21B/37Q/nyMGUKPP54xp7/3bXvcjX2KgD/V+3/MvzLAJHkqIgmInKfihaFjRuNb+nuZLNBTIx6p4mIpEZMDIz5X4eDI0eMiQcetCJaYpIqrB08d5ADGw84bHv26ln77eJ5ijus23pqK2v+XcOaf9ckOEdQjiB7ce2RPI9wLfYax6KPcSz6GEejj3Is+hjRMdEAnHnzDPmz5rfve+XmFXad3QXAsehjDsctkqOIQ+Yb8Tfu9+GLSAZbv94oku/cacxQn5F2ntnJ51s/ByCHbw6GNRiWsQFEUkBFNBGRVLi7gAYwfjx88w18913G5xERcXe+vvDnn/DEE1C/PvToYXYi1+Xn5UfJvCU5gGMRbXmn5Vy8cZFDFw6R1Serw7rj0ceTPN7R6KMcjT6aaIHtbseijzkU0YrkKIKvpy9BOYMSbFsqXykie0YSlCOInN45WbFiRbLHFxFzVali49gxOHQIKiY+N4rTDAwbiNVmBeDtem87vNeIuAoV0URE0sGKFcaMnTYb1K3rxYQJensVEblfAQGwbl3mnK0zo+Tyy0WVwCoJln/S8hNGNhzJgfMH2H9+P/v/28+BCweMf88f4L/r/yV6PD8vP4rkKGL/8fPyc1j/TNlneLbcsw6XeN65b9XAqgDExsamw6MTEWcbPNjKsGGe3LiRsVdXhB0MY8UBo9BeNGdR+tTuk3EnF7kP+pQnIpIOihY1xkv75x/o0cNKtmxxye8kIiIJ+CcyPFdUlDEO5eefQ44cGR4p08jtnzvBpaG3nL9+ngPnD3DowiGyeGchKGcQRXIUIa9/3kQLZLd4emgMA5HMyM8v+W3SS7w1njfD3rTfH9VoVIKCvYirUBFNRCQdlCsHmzbB9Onw5ptWVq40O5GISOZw4gS0agXHj8OePfDLL1CokNmpMp88/nmoWbgmNQvXNDuKiDxgvt7+NdtPbwegeqHqtC/f3uREIklLZFQfERFJjbx54Z13Eo6X9uOPGidNRCS1jh6F69eN29myQe7c5uYREcmMbDaIjU2616kztSrZir61+uLt4c34xuPxsKhMIa5Lr04RESc6fhy6dYNnn4UXXoCbN81OJCLiXmrVMmbqbNwYlixJ/HJPERFJm717oVOnFjRr5snChRl77rxZ8vJRs4843PcwDYo1yNiTi9wnFdFERJzoq6/gwgXj9vXr4O1tbh4REXdUogSsWgUFCzouj4yEpk1h+3ZzcomIZBa//+7BzZte/PabB4cPm5MhMHugOScWuQ8qoomIONGgQTBnDpQpA9OmwT3GZpYMMHr0aCwWC3379r3ndt999x2lS5fGz8+PChUqsHz58owJKCIpZrPBG28YxbXKlSEszOxEIiLuy2qFfPmuAdAggzqDXYu9ljEnEklHKqKJiDiRxQJdusCOHZAnj+O63buNrvOSMSIiIpgxYwYVK1a853YbNmygQ4cOdO/ena1bt9K2bVvatm3Lzp07MyipiKTE6dNw5Ihx+5FHMu5Dn4hIZvTyy1ZmzQpj795YqlVz/vl2ndlFkYlFGP3HaK7HXnf+CUXSiYpoIiIZwNPT8f6NG/Dcc1ClijGjp81mTq4HxZUrV+jUqROzZs0idzKjkk+ePJlmzZoxYMAAypQpw4gRI6hatSpTp07NoLQikhIBAcaXEePGwaRJ4OPjuH73bqNnhYiIpIzFAsHB4OXl/HMN/HUgF25cYPDqwcyInOH8E4qkkwz49RARkbt99JHROw3g00+NyQd8fc3NlJn16tWLli1bEhoaysiRI++5bXh4OP3793dY1rRpU5YsWZLkPjExMcTExNjvR0dHAxAbG0tsbOx9Zb21/f3uZyZ3y6y8zpdRmT08oE+fW+e8vfz8eahb14uiRWHs2Hgef/ze31ToOXa+tOR1l8co4q7i4yEuLuPOFxsfS5HsRfCweFA4e2FervZyxp1cJI1URBMRMcHrr8PRozB7NsybpwKaMy1YsIAtW7YQERGRou1PnTpFwbtGLy9YsCCnTp1Kcp9Ro0bx/vvvJ1i+atUqsmTJcn+B/yfMDQd4crfMyut8ZmWePbscFy4U58IFGD36BNevb03RfnqOnS81ea9d07hJIs7wyy8wYYIxA/JXX2XcwL3ent7MaD2D12u9zumrp/H31rTL4j5URBMRMUHWrEYPtLfegqJFHdddumR0o8+a1ZxsmcnRo0fp06cPYWFh+Pn5Oe08gwcPdui9Fh0dTVBQEE2aNCFHjhz3dazY2FjCwsJo3Lgx3m4ynau7ZVZe5zM7c9asFo4etbFnD3z+eSBFitx7xjez86aGu2VOS95bvXtF5P7ZbLBvH4SHQ6dOjjPFR0ffnpQlPNxCnToZm61cgXKUo1zGnlQkjVREExEx0d0FNJsNevaErVuNHmo1apiTK7OIjIzkzJkzVK1a1b4sPj6e9evXM3XqVGJiYvC8a8C6gIAATp8+7bDs9OnTBAQEJHkeX19ffBPpTujt7Z3qD7dp2dcs7pZZeZ3PrMyhobB5s3HZfHCw4/n//BN+/x369oW7a+t6jp0vNXnd6fGJuJoePeDzz43bFSrgMGnAraJZYGDCcSVFJHGaWEBExIXMmwfffgv790OrVnBdkxWlSaNGjdixYwdRUVH2n+rVq9OpUyeioqISFNAA6tSpw+rVqx2WhYWFUSejv54VkTTx8IBKlRyXWa1G8WzwYChdGv75x5RoIiLp6vvv4bXXjL8d73bn++CGDY7rihSBw4fh+HEYOtT5M7FM2TiFjzd+zM34m04/l4izqCeaiIgLqVnT6H0WEQGffAL+GiIiTbJnz0758uUdlmXNmpW8efPal3fp0oXChQszatQoAPr06UODBg2YMGECLVu2ZMGCBWzevJmZM2dmeH4RSV8REbBli3E7Rw546CFz84iI3I+YGDh9OuGVDB9/DGvXGrePH4fChW+ve/RRo7gWEgINGyY85t3HcpZTV07x9uq3uRp7lWmbp7H15a34emlQYHE/6okmIuJCSpY0LjVatAieecZxndVqXO4p6evIkSOcPHnSfj8kJIT58+czc+ZMKlWqxKJFi1iyZEmCYpyIuJ9atSAqCpo2hfHj4c7OqDYbLFpk4exZ0+KJiCTq5k2oW9co/rdrl3B9SIjxr8UC27c7rqtUCZYuNXrgljNx+LH31r7H1dirADxW7DEV0MRtqSeaiIiL8faGp59OuHzMGIiMhOnTIV++jM+VWay99VVtEvcB2rVrR7vE/koVEbdXoQKsXJlw+dGj2Xn9deNP4969jZ4dIiIZJS7OKIBt2ADFijlemunjA+fPG8W0rVuN4T7uvFqhSxdo0MD4oiBnzgyPnqy/z/7NrC2zAMjuk533HnvP3EAiaaAimoiIG4iMhGHDjD+wwsPh4MGEA2KLiEjqbd1awH774YcTrr812UuJEkZvDxGRtLDZHN9L9u+/Peh/mzYJxzcLCYH4eGMygOhoxyJaqVLGj6saGDYQq80Yc21wvcEUyFogmT1EXJcu5xQRcQNnzxpd+AG6d1cBTUQkvVWocI4334yncmXjcs87nT4Nzz9vfEht29aMdCKSGWzdCt26GRObfP2147pSpSB3buP2hg0Jh/CYMQP27YM5c6BgwYzJmx5W/7Oan/f/DECRHEXoW7uvuYFE0khFNBERN9CsGezcCa+/Dm+95bjOZoP//jMnl4hIZvHww5f48EMrW7dC2bKO68LCbt9ObEyhr76Cv/4yeomIiERHQ1RUfq5ccVx+4QJ8+SXs3WuMgXsnDw8YNAgmT4blyxMe08sNryGz2qy8Gfam/f6HDT/E31uzZol7c8NfRRGRB1NgoPGH1d3mz4devWDECHjlFff8I0tExJWFhBjjUv7yC7Ro4bju+nXo2RNu3DAG8I6KMiWiiLiIDz+EIUO8sNlCqFkzjubNb6+rWdMolnl6wrVrCfcdNCjjcjrDjbgbRJ6IZMPRDYQfC2fD0Q2cvnoagCoBVehUsZPJCUXSTh+1RETcWHQ0vPkmXLpk9FIrXx4ef9zsVCIimcvDD8PAgcbP3X7/3SiggTFm2t3mzDEKbaVK6f1ZJLPYuBHWrIEtW+Cbbxxn+i1WDGw2Y7Cz8HCLQxEtWzbYvBnKlMkcQ3OcuXqGdf+usxfMtpzcQqw1NtFtxzcZj4dFF8KJ+1MRTUTEjcXFGZd6fvmlMQitPqCJiGSsypVh1ixYtcp4H76T1QoDBhjjWhYvbgwcfqdly4yJYooXh0cfhezZMyy2iKSAzWZ8UZkrl+PyceNg8WLj9tChxqy/t4SEQOXKNgID/6VevaAEx6xSxXl5nSk2PhaLxYKXx+0SwoKdC+izsk+S++T0zUntIrXpWKEjDYMbZkRMEadTEU1ExI3lyQNffGFcShQYmHD9b79B/frg7Z3x2UREHgQFCsBLLxk/d9u2zSiggVEou9tXX8G33xq39+1zLKLt3w8//ACPPGJcAhaU8LO4iDhRly7G31F+fnDggOO6kJDbRbS//nIsohUrBps2xbF8+XYef7xIhuV1lu93f8/kjZOJOB7ByudX8uhDj9rXhQSFOGxbKm8pQoJCqFOkDiFBIZTJX0a9zyTTURFNRCQTqFMn4bJt26BxY2MGqE8+gccey/BYIiIPtBIl4Ndfjd5mBQokXH/woPGvhwc89JDjuvDw2+MjTZwI/frdXmezwfjxHsTE5Kd+feMLFRGzxcfDP//AzZvGZYt3v6YPH8Y+0H6pUo5juF66BMeOGbcDAiBvXsd9d+2CI0eyc/Cg8XfNnY4fh4sXjdvFi4Ov7+11V64Y5wXIly/hrJZ79sDu3cY+d493+M8/xrEBzpxx/B1u3dronRYSAiVLJvWMpL+YuBj+ufAPVpuVR/I8gp/X7WtCL964yPHo46k+9uUbl/nl3C+E3Aghv3d+h+OuP7wegA1HNzgU0SoVrMQ79d+hTpE61C5Sm7xZ8iY4rkhmoyKaiEgm9frrxqVEf/9tfEuqIpqISMbKlg0aNTJ+EvPJJ8aH+DNnwMfHcd2dPV/u7sV2/jy8/bYnEMKPP9rYujVdY4ukyh9/3P5bo3nzhDNM9uoFP/9s3D53zrFQtmIFdOhg3J40CfrcdYVgjRpexMU1pGpVK5GRjuuGDDGGtQCjIHZnkW3Tptu/f2+/DR984Lhvw4Zw8iTUq5ewiFanjvGFZK1axu/cnUW0EiWMn4zWaG4j/jxqTOsZ9XIUlQIq2df9uOdHXvjxhTSfo9mxZjxR5gn7/Vu9zYrlKoaPp+MblbenNyMbjkzzOUXciYpoIiKZ1Pjx0Lu3MZ36nT0YRETENdSqZfwkpksXY/DxgwcTjqF0qwcbQEiIFfB0WP9//2f0CqpfHzp21KzNkn42boTPP4cNG2DCBGja9Pa66tWNAfbj483Ll1oREUYPujuL2cOGwejRjpMGOFOcNY7tp7fbZ7b89+K//Pninw7b1ChUw15Ec5a/jv/lUEQrmbckJ/qfIDB7IuOGiDyA1KSKiGRSNWoYlwOdOOF4aQPA998bPSSaNDEnm4iI3Fvx4omPo3Zr3dy5cXzzzVFatXIcLC0+HubPh8uXYelS6NzZcd/oaOP93yMTDFP0ySefMG7cOE6dOkWlSpX4+OOPqVmzZpLbf/fddwwdOpR///2XEiVKMGbMGFrc0f3IZrPx7rvvMmvWLC5evEjdunWZNm0aJe7ocnT+/Hlee+01li5dioeHB08//TSTJ08mW7Zs9m22b99Or169iIiIIH/+/Lz22msMvGtq1+SymO3qVaOwVL++YxHp0CFjIg2AP/90LKJlzQp9+xqXVt45RtgtTZoYl2pCwr9LHn4Yunc3bpcrl3DfF16w8u+/x6hbN+EYY48+ejtjzpyO6wIDbx+3WrWEx+3QwRg3tnbthOucPdHHf9f+I/xYOOFHw9lwbAObjm/iWuw1h22OXjpKUM7bv+PNijfj9NXTZPHOQh5/x+u4S+QtQfcq3VOdx2azYTtjo12Zdg7LPSweKqCJ3EFFNBGRTMzDA4rc9ffm+fPw8svGpRRPPWVMzW6xmJNPRETuX5480L69jRw5thMa6vgmf+AAXL9u3K5fP+H7+2uvGcW1unVhxgwoVCiDQqezhQsX0r9/f6ZPn06tWrWYNGkSTZs2Ze/evRRIZAC6DRs20KFDB0aNGkWrVq2YP38+bdu2ZcuWLZQvXx6AsWPHMmXKFObMmUNwcDBDhw6ladOm/P333/j5GWNPderUiZMnTxIWFkZsbCzdunWjZ8+ezJ8/H4Do6GiaNGlCaGgo06dPZ8eOHbz44ovkypWLnj17pjiLmQYNMnqZxcdDVBRUun3FICH/G0fe0xP++y/hvuPHJ33c119Pel3NmsZPUj791Mry5VG0aFGIu3tedutm/CSmTBn47LOkjzthQtLr0pPVZmX32d1sOLqBDcc2EH40nL3/7b3nPv5e/uz7b59DEa1p8aY0Ld400e1DgkISDPR/P2JjY1m+fDnlC5j/GhRxZSqiiYg8YGbPNgpoYHz76uMDsbHmZhIRkfRRqpQxSPvGjUbPoLv9/rtxmX9YWMLB2/fuhaNHjcHXS5d2nNn5wgVjxlCbzfhypnDhhMeNjYUsWRL26tm5Ew4fTt9vayZOnEiPHj3o9r/qyfTp0/n555+ZPXs2b731VoLtJ0+eTLNmzRgwYAAAI0aMICwsjKlTpzJ9+nRsNhuTJk1iyJAhtGnTBoC5c+dSsGBBlixZQvv27dm9ezcrV64kIiKC6tWrA/Dxxx/TokULxo8fT6FChZg3bx43b95k9uzZ+Pj4UK5cOaKiopg4caK9iJZcloyw+W8P1kWe4dhhf94dd9JhnSVbHuLjjZ5Hi345gVfgBfs6my/M/SkLpctfJ0tWGzfikh7cPjB7oENvqZi4GA6cv2uayxSKjYvlyPUj7Dq7i3zZ8lEkh2PxeM+5PcRb4/Hx9KFEXsfByg5fPMyVm1dSdd47lcpXCi+P2x+fz149y5mrZwAomrMo2X1vd127cvMKUZejiPw9kk0nNvHXsb+4FHPpnscvmrOow8yWlQpWwttT06uLuBoV0UREHjD9+xuD4w4ffu9vjEVExD1lyQKPP55weUyM0asoOhrKlk14Sd3s2TB2rHH76FHHnszr10PbtsbtDz+EwYMd923RwpgJsWxZYybFO33yCXzzjSdz5qTpYdndvHmTyMhIBt8RwsPDg9DQUMLDwxPdJzw8nP79+zssa9q0KUuWLAHg0KFDnDp1itDQUPv6nDlzUqtWLcLDw2nfvj3h4eHkypXLXkADCA0NxcPDg40bN/Lkk08SHh7Oo48+is8dg2s1bdqUMWPGcOHCBXLnzp1slsTExMQQExNjvx8dHQ0YvYdi7/ObsNjYWD75vBgXDhqV0O/yl4Xsp29vcLw6FJgNQRsYuXseI6f9nvAgW4x/IrpHUKng7a5qi3ct5qVlLwEwtdlUelbtaV934L8DlJ+Rxl5Oe+H5Cs8zu/Vsh8V1Z9fl/PXzFM9TnL//72+Hda/+/CrLD9w1y0EqnOx70mH2yWkR03h33bsA/NDuB1qWaGlfF34knPcOvgcH7z6KwdvDmyoBVYxZLQvXpnaR2hTOfldl2gqx1oz7lvPW6+h+X09mUV7nc7fMac2b0v1URBMRecB4eBgDVnfqlHGD5YqIiPl8feGHH4zeZBcuJFz/+x21EpvNcd2dl4XevQ5uj7GW1LrElqfWuXPniI+Pp2DBgg7LCxYsyJ49exLd59SpU4luf+rUKfv6W8vutc3dl4p6eXmRJ08eh22Cg4MTHOPWuty5cyebJTGjRo3i/fffT7B81apVZMmSJcn9kpIzOIYLB/83cNmxOlBmye2VhTfDqxVTdJw//viD4/7H7fe3n99uv71z506Wn7pdvDp+4zjp4fix4yy/a+rP2JvGh99rV68lWHfmzJl0OW/Yr2Hk8Mphv7/v1D777c2bN2PZf/uXZPvl7Q775vLKRemspSmVtRSls5TmkSyP4OPhAzeBQ7Dt0Da2sS1dcqZVWFiY2RHui/I6n7tlTm3ea9euJb8RKqKJiDywVEATEXkwWSzGuGp369vXmGHx+nWjN9udHn7YGE/Nw8OYuOZu/foZsxsmMhwZrVpBUJA1XbI/qAYPHuzQey06OpqgoCCaNGlCjhw57rFnQrGxsdTe8imBj3xBgTL/kKtIHjw8kxhULBlP1H+CoBy3x+zKfSw3l7ddBuDp8k/T4KEG9nWnrpxis+/mVJ3HarVy/PhxChcuTEhQCC0qO07C0NWrK1djr1IgawFaPOa47p+If6hwJpGZDu5Tq8atyOZzewIJDoDPXqPHYdvqbR165AWdCmLTpU08WetJ6j1Uj2I5i2Fx8QFoY2NjCQsLo3Hjxnh7u/5lpMrrfO6WOa15b/XwTY6KaCIiIiIiwrPPGj+JKV8epkxJet/33kt6XfPmEBpqZXnar6gDIF++fHh6enL69GmH5adPnybg1vSPdwkICLjn9rf+PX36NIGBgQ7bVK5c2b7N3b2a4uLiOH/+vMNxEjvPnedILktifH198b37+lvA29s7VR8W21ctQYsWLdL9g/GjwY/yaPCjia4Lyh3E7LazE12XnFuD3ieV+eOWHye5b7+Qfqk6Z3LalGlDmzJtEl1XIaACLxV5iRaV0v85drbUvqbMorzO526ZU5s3pfu4zOTWo0ePxmKx0Ldv33tu991331G6dGn8/PyoUKFCgu66IiIiIiKSefn4+FCtWjVWr15tX2a1Wlm9ejV16tRJdJ86deo4bA/GJT+3tg8ODiYgIMBhm+joaDZu3Gjfpk6dOly8eJHIyEj7Nr/99htWq5VatWrZt1m/fr3D2DphYWGUKlWK3LlzpyiLiIi4LpcookVERDBjxgwqVrz3tfe3poPu3r07W7dupW3btrRt25adO3dmUFIRERERETFb//79mTVrFnPmzGH37t288sorXL161T5bZ5cuXRwmHujTpw8rV65kwoQJ7Nmzh/fee4/NmzfTu3dvAPuX+SNHjuSnn35ix44ddOnShUKFCtH2fzMqlClThmbNmtGjRw82bdrEn3/+Se/evWnfvj2FChUCoGPHjvj4+NC9e3d27drFwoULmTx5ssOlmMllERER12V6Ee3KlSt06tSJWbNm2b+dScqd00GXKVOGESNGULVqVaZOnZpBaUVERERExGzPPfcc48ePZ9iwYVSuXJmoqChWrlxpH7D/yJEjnDx50r59SEgI8+fPZ+bMmVSqVIlFixaxZMkSype/PVvkwIEDee211+jZsyc1atTgypUrrFy5Ej8/P/s28+bNo3Tp0jRq1IgWLVpQr149Zs6caV+fM2dOVq1axaFDh6hWrRpvvPEGw4YNo2fPnveVRUREXJPpY6L16tWLli1bEhoaysiRI++5bWqmgxYRERERkcynd+/eSfbeWrt2bYJl7dq1o127dkkez2KxMHz4cIYPH57kNnny5GH+/Pn3zFWxYkV+v3Oq00Qkl0VERFyTqUW0BQsWsGXLFiIiIlK0fWqmg46JiSEmJsZ+/9aMC7GxsQ5jFaTUrX1Ss68Z3C0vuF9m5XU+d8vsbnkhbZnd6XGKiIiIiIiklmlFtKNHj9KnTx/CwsIcukint1GjRvH+++8nWL5q1Sqy3D13930ICwtLS6wM5255wf0yK6/zuVtmd8sLqct87do1JyQRERERERFxLaYV0SIjIzlz5gxVq1a1L4uPj2f9+vVMnTqVmJgYPD09HfZJzXTQgwcPdrgENDo6mqCgIJo0aUKOHDnuO3dsbCxhYWE0btzYLaZ5dbe84H6Zldf53C2zu+WFtGW+1cNXREREREQkMzOtiNaoUSN27NjhsKxbt26ULl2aQYMGJSigwe3poPv27Wtfltx00L6+vvj6+iZY7u3tnaYPt2ndP6O5W15wv8zK63zultnd8kLqMrvbYxQREREREUkN04po2bNnTzADTdasWcmbN699eZcuXShcuDCjRo0CjOmgGzRowIQJE2jZsiULFixg8+bNDjPiiIiIiIiIiIiIpDcPswPcS2qmphYREREREREREUlvps7Oebe7p6JOzdTUIiIiIiIiIiIi6c2le6KJiIiIiIiIiIi4AhXRREREREREREREkqEimoiIiIiIiIiISDJcaky0jGCz2QCIjo5O1f6xsbFcu3aN6OhovL290zOaU7hbXnC/zMrrfO6W2d3yQtoy33o/vfX+Kmlrax60148ZlNf53C2zu+UF98usdiZ9qZ1xbe6WF9wvs/I6n7tlTmvelLY1D1wR7fLlywAEBQWZnEREJHO5fPkyOXPmNDuGS1BbIyKS/tTO3KZ2RkTEOZJrayy2B+wrHavVyokTJ8iePTsWi+W+94+OjiYoKIijR4+SI0cOJyRMX+6WF9wvs/I6n7tldre8kLbMNpuNy5cvU6hQITw8NEoApK2tedBeP2ZQXudzt8zulhfcL7PamfSldsa1uVtecL/Myut87pY5rXlT2tY8cD3RPDw8KFKkSJqPkyNHDrd4Id3ibnnB/TIrr/O5W2Z3ywupz6yeAY7So615kF4/ZlFe53O3zO6WF9wvs9qZ9KF2xj24W15wv8zK63zuljkteVPS1uirHBERERERERERkWSoiCYiIiIiIiIiIpIMFdHuk6+vL++++y6+vr5mR0kRd8sL7pdZeZ3P3TK7W15wz8yZlTv+X7hbZuV1PnfL7G55wf0yu1vezMwd/y/cLbO75QX3y6y8zudumTMq7wM3sYCIiIiIiIiIiMj9Uk80ERERERERERGRZKiIJiIiIiIiIiIikgwV0URERERERERERJKhIpqIiIiIiIiIiEgyVES7D5988gnFihXDz8+PWrVqsWnTJrMjJWnUqFHUqFGD7NmzU6BAAdq2bcvevXvNjpVio0ePxmKx0LdvX7Oj3NPx48d5/vnnyZs3L/7+/lSoUIHNmzebHStR8fHxDB06lODgYPz9/XnkkUcYMWIErjS3yPr162ndujWFChXCYrGwZMkSh/U2m41hw4YRGBiIv78/oaGh7N+/35yw3DtvbGwsgwYNokKFCmTNmpVChQrRpUsXTpw44ZJ57/Z///d/WCwWJk2alGH5xOAubY3amYyhdiZ9qZ1xPrU1rk/tTMZxh7bGndoZcP22xt3aGXC/tsbsdkZFtBRauHAh/fv3591332XLli1UqlSJpk2bcubMGbOjJWrdunX06tWLv/76i7CwMGJjY2nSpAlXr141O1qyIiIimDFjBhUrVjQ7yj1duHCBunXr4u3tzYoVK/j777+ZMGECuXPnNjtaosaMGcO0adOYOnUqu3fvZsyYMYwdO5aPP/7Y7Gh2V69epVKlSnzyySeJrh87dixTpkxh+vTpbNy4kaxZs9K0aVNu3LiRwUkN98p77do1tmzZwtChQ9myZQvff/89e/fu5YknnjAhqSG55/eWH374gb/++otChQplUDK5xZ3aGrUzzqd2Jv2pnXE+tTWuTe1MxnGHtsbd2hlw/bbG3doZcL+2xvR2xiYpUrNmTVuvXr3s9+Pj422FChWyjRo1ysRUKXfmzBkbYFu3bp3ZUe7p8uXLthIlStjCwsJsDRo0sPXp08fsSEkaNGiQrV69embHSLGWLVvaXnzxRYdlTz31lK1Tp04mJbo3wPbDDz/Y71utVltAQIBt3Lhx9mUXL160+fr62r755hsTEjq6O29iNm3aZANshw8fzphQ95BU3mPHjtkKFy5s27lzp+2hhx6yffTRRxme7UHmzm2N2pn0p3bGudTOOJ/aGtejdiZjuEtb427tjM3mXm2Nu7UzNpv7tTVmtDPqiZYCN2/eJDIyktDQUPsyDw8PQkNDCQ8PNzFZyl26dAmAPHnymJzk3nr16kXLli0dnmtX9dNPP1G9enXatWtHgQIFqFKlCrNmzTI7VpJCQkJYvXo1+/btA2Dbtm388ccfNG/e3ORkKXPo0CFOnTrl8NrImTMntWrVcqvfQ4vFQq5cucyOkiir1Urnzp0ZMGAA5cqVMzvOA8fd2xq1M+lP7UzGUjuTMdTWmEftTMZxl7bG3doZcO+2JjO0M+D6bY2z2xmvdD9iJnTu3Dni4+MpWLCgw/KCBQuyZ88ek1KlnNVqpW/fvtStW5fy5cubHSdJCxYsYMuWLURERJgdJUX++ecfpk2bRv/+/Xn77beJiIjg9ddfx8fHh65du5odL4G33nqL6OhoSpcujaenJ/Hx8XzwwQd06tTJ7GgpcurUKYBEfw9vrXNlN27cYNCgQXTo0IEcOXKYHSdRY8aMwcvLi9dff93sKA8kd25r1M44h9qZjKV2JmOorTGP2pmM4U5tjbu1M+DebY27tzPgHm2Ns9sZFdEeAL169WLnzp388ccfZkdJ0tGjR+nTpw9hYWH4+fmZHSdFrFYr1atX58MPPwSgSpUq7Ny5k+nTp7tko/Ptt98yb9485s+fT7ly5YiKiqJv374UKlTIJfNmJrGxsTz77LPYbDamTZtmdpxERUZGMnnyZLZs2YLFYjE7jrgZtTPOoXZGUsod2hlQWyOp5w7tDLhfW+Nu7QyorTGTO7Q1GdHO6HLOFMiXLx+enp6cPn3aYfnp06cJCAgwKVXK9O7dm2XLlrFmzRqKFClidpwkRUZGcubMGapWrYqXlxdeXl6sW7eOKVOm4OXlRXx8vNkREwgMDKRs2bIOy8qUKcORI0dMSnRvAwYM4K233qJ9+/ZUqFCBzp07069fP0aNGmV2tBS59bvmbr+Htxqbw4cPExYW5rLf2Pz++++cOXOGokWL2n8HDx8+zBtvvEGxYsXMjvdAcNe2Ru2M86idyVhqZ5xPbY251M44n7u1Ne7WzoB7tzXu2s6A+7Q1GdHOqIiWAj4+PlSrVo3Vq1fbl1mtVlavXk2dOnVMTJY0m81G7969+eGHH/jtt98IDg42O9I9NWrUiB07dhAVFWX/qV69Op06dSIqKgpPT0+zIyZQt27dBNNs79u3j4ceesikRPd27do1PDwcf+U9PT2xWq0mJbo/wcHBBAQEOPweRkdHs3HjRpf9PbzV2Ozfv59ff/2VvHnzmh0pSZ07d2b79u0Ov4OFChViwIAB/PLLL2bHeyC4W1ujdsb51M5kLLUzzqe2xlxqZ5zP3doad2tnwL3bGndsZ8C92pqMaGd0OWcK9e/fn65du1K9enVq1qzJpEmTuHr1Kt26dTM7WqJ69erF/Pnz+fHHH8mePbv9GuucOXPi7+9vcrqEsmfPnmB8g6xZs5I3b16XHfegX79+hISE8OGHH/Lss8+yadMmZs6cycyZM82OlqjWrVvzwQcfULRoUcqVK8fWrVuZOHEiL774otnR7K5cucKBAwfs9w8dOkRUVBR58uShaNGi9O3bl5EjR1KiRAmCg4MZOnQohQoVom3bti6XNzAwkGeeeYYtW7awbNky4uPj7b+HefLkwcfHx6XyFi1aNEGD6O3tTUBAAKVKlcroqA8sd2pr1M44n9qZ9Kd2xtzMamvMp3bGudytrXG3dgZcv61xt3Ymucyu2NaY3s6k2zyfD4CPP/7YVrRoUZuPj4+tZs2atr/++svsSEkCEv354osvzI6WYq48HfQtS5cutZUvX97m6+trK126tG3mzJlmR0pSdHS0rU+fPraiRYva/Pz8bA8//LDtnXfescXExJgdzW7NmjWJvm67du1qs9mMaaGHDh1qK1iwoM3X19fWqFEj2969e10y76FDh5L8PVyzZo3L5U1Mek8HLSnjLm2N2pmMoXYmfamdMTdzYtTWZDy1MxnL1dsad2pnbDbXb2vcrZ1JLrMrtjVmtzMWm81mu+/Km4iIiIiIiIiIyANEY6KJiIiIiIiIiIgkQ0U0ERERERERERGRZKiIJiIiIiIiIiIikgwV0URERERERERERJKhIpqIiIiIiIiIiEgyVEQTERERERERERFJhopoIiIiIiIiIiIiyVARTSSTWbt2LRaLhYsXL5odRUREMiG1MyIi4kxqZ8SVqYgmIiIiIiIiIiKSDBXRREREREREREREkqEimkg6s1qtjBo1iuDgYPz9/alUqRKLFi0CbndN/vnnn6lYsSJ+fn7Url2bnTt3Ohxj8eLFlCtXDl9fX4oVK8aECRMc1sfExDBo0CCCgoLw9fWlePHifP755w7bREZGUr16dbJkyUJISAh79+517gMXEZEMoXZGREScSe2MyD3YRCRdjRw50la6dGnbypUrbQcPHrR98cUXNl9fX9vatWtta9assQG2MmXK2FatWmXbvn27rVWrVrZixYrZbt68abPZbLbNmzfbPDw8bMOHD7ft3bvX9sUXX9j8/f1tX3zxhf0czz77rC0oKMj2/fff2w4ePGj79ddfbQsWLLDZbDb7OWrVqmVbu3atbdeuXbb69evbQkJCzHg6REQknamdERERZ1I7I5I0FdFE0tGNGzdsWbJksW3YsMFheffu3W0dOnSwNwi3GgibzWb777//bP7+/raFCxfabDabrWPHjrbGjRs77D9gwABb2bJlbTabzbZ3714bYAsLC0s0w61z/Prrr/ZlP//8sw2wXb9+PV0ep4iImEPtjIiIOJPaGZF70+WcIunowIEDXLt2jcaNG5MtWzb7z9y5czl48KB9uzp16thv58mTh1KlSrF7924Adu/eTd26dR2OW7duXfbv3098fDxRUVF4enrSoEGDe2apWLGi/XZgYCAAZ86cSfNjFBER86idERERZ1I7I3JvXmYHEMlMrly5AsDPP/9M4cKFHdb5+vo6NDyp5e/vn6LtvL297bctFgtgjG8gIiLuS+2MiIg4k9oZkXtTTzSRdFS2bFl8fX05cuQIxYsXd/gJCgqyb/fXX3/Zb1+4cIF9+/ZRpkwZAMqUKcOff/7pcNw///yTkiVL4unpSYUKFbBaraxbty5jHpSIiLgMtTMiIuJMamdE7k090UTSUfbs2XnzzTfp168fVquVevXqcenSJf78809y5MjBQw89BMDw4cPJmzcvBQsW5J133iFfvny0bdsWgDfeeIMaNWowYsQInnvuOcLDw5k6dSqffvopAMWKFaNr1668+OKLTJkyhUqVKnH48GHOnDnDs88+a9ZDFxGRDKB2RkREnEntjEgyzB6UTSSzsVqttkmTJtlKlSpl8/b2tuXPn9/WtGlT27p16+yDZC5dutRWrlw5m4+Pj61mzZq2bdu2ORxj0aJFtrJly9q8vb1tRYsWtY0bN85h/fXr1239+vWzBQYG2nx8fGzFixe3zZ4922az3R6I88KFC/btt27dagNshw4dcvbDFxERJ1M7IyIizqR2RiRpFpvNZjOtgifygFm7di2PP/44Fy5cIFeuXGbHERGRTEbtjIiIOJPaGXnQaUw0ERERERERERGRZKiIJiIiIiIiIiIikgxdzikiIiIiIiIiIpIM9UQTERERERERERFJhopoIiIiIiIiIiIiyVARTUREREREREREJBkqoomIiIiIiIiIiCRDRTQREREREREREZFkqIgmIiIiIiIiIiKSDBXRREREREREREREkqEimoiIiIiIiIiISDJURBMREREREREREUnG/wMYS5yoItBMRQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 1500x500 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "fig, axs = plt.subplots(1,3, figsize=(15, 5)) #, sharex=True, sharey=True)\n",
    "\n",
    "\n",
    "colors = ['g-.', 'b:', 'k', 'c', 'k--', 'r', 'r--']\n",
    "\n",
    "idx = 0\n",
    "for key in results_avg.keys():\n",
    "      tt = results_avg[key]['train_loss_history']\n",
    "\n",
    "      axs[0].plot(np.arange(len(tt)), tt, colors[idx], lw=2)\n",
    "      axs[0].grid('on')\n",
    "\n",
    "      axs[0].set(xlabel='epoch', ylabel='train loss')\n",
    "\n",
    "      idx += 1\n",
    "\n",
    "      axs[0].legend(list(results_avg.keys()))         \n",
    "\n",
    "\n",
    "idx = 0\n",
    "for key in results_avg.keys():\n",
    "      tt = results_avg[key]['val_loss_history']\n",
    "\n",
    "      axs[1].plot(np.arange(len(tt)), tt, colors[idx], lw=2)\n",
    "      axs[1].grid('on')\n",
    "\n",
    "      axs[1].set(xlabel='epoch', ylabel='val loss')\n",
    "\n",
    "      idx += 1\n",
    "\n",
    "\n",
    "\n",
    "idx = 0\n",
    "for key in results_avg.keys():\n",
    "      tt = results_avg[key]['accs_history']\n",
    "\n",
    "      axs[2].plot(np.arange(len(tt)), tt, colors[idx], lw=2)\n",
    "      axs[2].grid('on')\n",
    "\n",
    "      axs[2].set(xlabel='epoch', ylabel='acc')\n",
    "\n",
    "      idx += 1\n",
    "\n",
    "\n",
    "\n",
    "# fig.savefig(f'drive/MyDrive/KV_Transformer_Res/{d_dim}_{d_depth}_{h_heads}_{num_epochs}_{lr}_{sequence_length}_{which_task}_avg.png', dpi=150, format=\"png\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 131
    },
    "id": "fKWn5Hcr_S-v",
    "outputId": "63cfb40e-e2ca-454f-fd5d-24760757527c"
   },
   "outputs": [
    {
     "ename": "IndentationError",
     "evalue": "unexpected indent (42598641.py, line 3)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  Cell \u001b[0;32mIn[45], line 3\u001b[0;36m\u001b[0m\n\u001b[0;31m    2.737723\t2.343315\t0.247721\t00:02\u001b[0m\n\u001b[0m    ^\u001b[0m\n\u001b[0;31mIndentationError\u001b[0m\u001b[0;31m:\u001b[0m unexpected indent\n"
     ]
    }
   ],
   "source": [
    "sqrt\n",
    "\n",
    "\t2.737723\t2.343315\t0.247721\t00:02\n",
    "1\t1.794846\t1.445698\t0.539795\t00:02\n",
    "2\t1.171195\t0.925104\t0.687500\t00:01\n",
    "3\t0.850798\t0.861772\t0.723470\t00:01\n",
    "4\t0.702100\t0.805322\t0.733317\t00:01\n",
    "5\t0.623310\t0.807370\t0.741618\t00:01\n",
    "6\t0.582024\t0.810705\t0.735758\t00:01\n",
    "7\t0.563333\t0.833307\t0.731608\t00:01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "wJQzwWZo_TAI"
   },
   "outputs": [],
   "source": [
    "std\n",
    "\n",
    "0\t2.775209\t2.337709\t0.254883\t00:03\n",
    "1\t1.897800\t1.862743\t0.461182\t00:02\n",
    "2\t1.245429\t0.887715\t0.720133\t00:02\n",
    "3\t0.881179\t0.891836\t0.739421\t00:02\n",
    "4\t0.688354\t0.765689\t0.749268\t00:03\n",
    "5\t0.590414\t0.779182\t0.760905\t00:02\n",
    "6\t0.539071\t0.702758\t0.777425\t00:02\n",
    "7\t0.512280\t0.695016\t0.777262\t00:02"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "TkIH6mJ8_TDm"
   },
   "outputs": [],
   "source": [
    "layer norm\n",
    "\t\n",
    "0\t2.751513\t2.254431\t0.218750\t00:02\n",
    "1\t1.864145\t1.572920\t0.518636\t00:02\n",
    "2\t1.217126\t0.903293\t0.705566\t00:02\n",
    "3\t0.851127\t0.727705\t0.767253\t00:03\n",
    "4\t0.661995\t0.690153\t0.781738\t00:02\n",
    "5\t0.563049\t0.709912\t0.791016\t00:01\n",
    "6\t0.510370\t0.683305\t0.788818\t00:04\n",
    "7\t0.483554\t0.673274\t0.792643\t00:04\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "LYAaSfMTEROZ"
   },
   "outputs": [],
   "source": [
    "# dir(learn)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "RIEWLGhhERQu"
   },
   "outputs": [],
   "source": [
    "# learn.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "HliNp9FhERVf",
    "outputId": "b0bac7de-e5f1-4cbb-b6ab-4bba17e76377"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    }
   ],
   "source": [
    "learn.logger()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Z2vVQdCQERWr"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qr3RzX32_TtG"
   },
   "source": [
    "# --- SKIP ------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BhJJhPm0sshk"
   },
   "source": [
    "## Bonus - Generation example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "eiWQ3obRJJGb",
    "outputId": "4cc1032d-7874-45a2-a77a-8fe2b88bc800"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
     ]
    }
   ],
   "source": [
    "#hide\n",
    "from google.colab import drive\n",
    "\n",
    "drive.mount('/content/drive')\n",
    "path = Path('/content/drive/MyDrive/char_model')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "HP9Gvq3V93eH",
    "outputId": "c6fbe234-3bec-443d-b953-6c9eb3223331"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ls: cannot access '/content/drive/MyDrive/char_model': No such file or directory\n"
     ]
    }
   ],
   "source": [
    "!ls /content/drive/MyDrive/char_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YfpGVLJusshk"
   },
   "source": [
    "Learning to predict numbers is great, but let's try something more entertaining. We can train a language model to generate texts. For example let's try to generate some text in style of Lewis Carroll. For this we'll fit a language model on \"Alice in Wonderland\" and \"Through the looking glass\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Jw0D7mxmsshl"
   },
   "outputs": [],
   "source": [
    "#collapse-hide\n",
    "def parse_txt(fns):\n",
    "    txts = []\n",
    "    for fn in fns:\n",
    "        with open(fn) as f:\n",
    "            tmp = ''\n",
    "            for line in f.readlines():\n",
    "                line = line.strip('\\n')\n",
    "                if line:\n",
    "                    tmp += ' ' + line\n",
    "                elif tmp:\n",
    "                    txts.append(tmp.strip())\n",
    "                    tmp = ''\n",
    "    return txts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 287
    },
    "id": "7mnVP6BQsshm",
    "outputId": "88b974f8-7ee0-440a-ea5d-9a5d527aa7dc"
   },
   "outputs": [
    {
     "ename": "FileNotFoundError",
     "evalue": "ignored",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-75-552ed8e8b137>\u001b[0m in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mtexts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mparse_txt\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0;34m'11-0.txt'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpath\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0;34m'12-0.txt'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-74-7ebc88e45473>\u001b[0m in \u001b[0;36mparse_txt\u001b[0;34m(fns)\u001b[0m\n\u001b[1;32m      3\u001b[0m     \u001b[0mtxts\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mfn\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mfns\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 5\u001b[0;31m         \u001b[0;32mwith\u001b[0m \u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfn\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      6\u001b[0m             \u001b[0mtmp\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m             \u001b[0;32mfor\u001b[0m \u001b[0mline\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreadlines\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: '/content/drive/MyDrive/char_model/11-0.txt'"
     ]
    }
   ],
   "source": [
    "texts = parse_txt([path/'11-0.txt', path/'12-0.txt'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 166
    },
    "id": "gMz1ZbZwsshm",
    "outputId": "d035c3ac-1c23-4459-871b-18c76e4fa73e"
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "ignored",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-53-60fb2285b6d2>\u001b[0m in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mlen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtexts\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'texts' is not defined"
     ]
    }
   ],
   "source": [
    "len(texts)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "n5VahB4ksshm",
    "outputId": "4569f1fe-98e4-4106-e6ea-220e127a6544"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['\\ufeffCHAPTER I. Down the Rabbit-Hole',\n",
       " 'Alice was beginning to get very tired of sitting by her sister on the bank, and of having nothing to do: once or twice she had peeped into the book her sister was reading, but it had no pictures or conversations in it, “and what is the use of a book,” thought Alice “without pictures or conversations?”']"
      ]
     },
     "execution_count": 55,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "texts[0:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_BqN4XyzObLT"
   },
   "outputs": [],
   "source": [
    "#collapse-hide\n",
    "class CharTokenizer(Transform):\n",
    "    \"Simple charecter level tokenizer\"\n",
    "    def __init__(self, vocab=None):\n",
    "        self.vocab = ifnone(vocab, ['', 'xxbos', 'xxeos'] + list(string.printable))\n",
    "        self.c2i = defaultdict(int, [(c,i) for i, c in enumerate(self.vocab)])\n",
    "    def encodes(self, s, add_bos=False, add_eos=False):\n",
    "        strt = [self.c2i['xxbos']] if add_bos else []\n",
    "        end = [self.c2i['xxeos']] if add_eos else []\n",
    "        return LMTensorText(strt + [self.c2i[c] for c in s] + end)\n",
    "    def decodes(self, s, remove_special=False):\n",
    "        return TitledStr(''.join([self.decode_one(i) for i in s]))\n",
    "    def decode_one(self, i):\n",
    "        if i == 2: return '\\n'\n",
    "        elif i == 1: return ''\n",
    "        else: return self.vocab[i]\n",
    "    @property\n",
    "    def vocab_sz(self):\n",
    "        return len(self.vocab)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "X1bLTtE5AVMX"
   },
   "outputs": [],
   "source": [
    "tok = CharTokenizer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7am4mUFzIVTo"
   },
   "outputs": [],
   "source": [
    "def add_bos_eos(x:list, bos_id=1, eos_id=2):\n",
    "    return [bos_id] + x + [eos_id]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "UR8swikDsshn"
   },
   "outputs": [],
   "source": [
    "nums = [add_bos_eos(tok(t.lower()).tolist()) for t in texts]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Mo6sLS2Osshn",
    "outputId": "bece15b5-efd6-41c0-d90c-5c294f2a0efa"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1779"
      ]
     },
     "execution_count": 60,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(nums)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Eb9DALvisshn"
   },
   "outputs": [],
   "source": [
    "all_nums = []\n",
    "for n in nums: all_nums.extend(n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "VXNIdREpssho",
    "outputId": "04cab237-a396-4afa-f77b-75b002656699"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 0, 15, 20, 13, 28, 32, 17, 30, 97, 21, 78, 97, 16, 27]"
      ]
     },
     "execution_count": 62,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_nums[:15]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "AIAY-_vbssho",
    "outputId": "c1e96196-2370-4a72-bb66-a659c1524368"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "chapter i. down the rabbit-hole\n",
      "alice was beginning to get very tired of sitting by her sister on\n"
     ]
    }
   ],
   "source": [
    "print(tok.decode(all_nums[:100]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Gog7MW3Rsshp",
    "outputId": "e49dbf8e-8e81-474e-947e-906f21f39433"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([8, 512]), torch.Size([8, 512]))"
      ]
     },
     "execution_count": 64,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sl = 512\n",
    "seqs = L((tensor(all_nums[i:i+sl]), tensor(all_nums[i+1:i+sl+1]))\n",
    "         for i in range(0,len(all_nums)-sl-1,sl))\n",
    "cut = int(len(seqs) * 0.8)\n",
    "dls = DataLoaders.from_dsets(seqs[:cut], seqs[cut:], device='cuda',\n",
    "                             bs=8, drop_last=True, shuffle=True)\n",
    "xb, yb = dls.one_batch()\n",
    "xb.shape, yb.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 302
    },
    "id": "8mdKK9iMsshr",
    "outputId": "9852dbb3-34c6-4ed0-9bf1-fa10d4b7a43c"
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "SuggestedLRs(lr_min=0.07585775852203369, lr_steep=0.6309573650360107)"
      ]
     },
     "execution_count": 65,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXiU9b338fd3spIAAZKwBUiUVQRFjbihgEvrrk9bra211S7W1tZ67HL0nD491vOc0/Yca+vSitTWutS60OpxpepxAbWgQRFBUREDhC0hIfue+T5/zCSNMUACmTWf13XNlZn7vmfmM7nCfPndv+U2d0dERAQgEOsAIiISP1QURESki4qCiIh0UVEQEZEuKgoiItJFRUFERLqkxjpAf+Xl5XlRUVGsY4iIJJRVq1btcvf8fR2XcEWhqKiIkpKSWMcQEUkoZrapL8fp9JGIiHRRURARkS4qCiIi0kVFQUREuqgoiIhIFxUFERHpoqIgIpIAnlm3gw3l9RF/HxUFEZE4Fww6V97/Bn95oyzi76WiICIS52qa2mjrcPKHZkT8vVQURETiXEV9CwD5w1QUREQGvYo6FQUREQlLiqJgZtPNbHW3W62ZXd3jGDOzW8xsg5mtMbMjI5VHRCRRRbMoRGyVVHd/D5gDYGYpwFbgkR6HnQFMDd+OAW4P/xQRkbCK+hYyUgMMy4j8wtbROn10CvChu/dcuvU84B4PWQGMMLNxUcokIpIQKupayB+WgZlF/L2iVRQuAv7cy/YCYEu3x2XhbSIiEtZZFKIh4kXBzNKBc4GHD+A1LjezEjMrqaioGLhwIiIJoKKuJSpzFCA6LYUzgDfcfWcv+7YCE7s9nhDe9jHuvtjdi929OD9/n1eTExFJKhX1SdRSAL5A76eOAB4DvhwehXQsUOPu26OQSUQkIbR1BKlqaI1aUYhoV7aZZQOnAd/stu0KAHdfBDwFnAlsABqByyKZR0Qk0VTWtwLRGY4KES4K7t4A5PbYtqjbfQeujGQGEZFE1jVHIYn6FEREZD9V1DcD0WspqCiIiMSxaM5mBhUFEZG41lkU8nT6SEREKupaGJ6ZSmZaSlTeT0VBRCSORXOOAqgoiIjEtWgucQEqCiIicS1UFDKj9n4qCiIicSya6x6BioKISNxqaGmnobWDvGHpUXtPFQURkTi1qz66s5lBRUFEJG5Fe+IaqCiIiMQtFQUREelSUa+iICIiYRV1LQQMcrNVFEREBr2KuhZGZWeQErCovaeKgohInIr2bGZQURARiVvRXvcIVBREROJWtGczg4qCiEhcCgadXcnWUjCzEWa2xMzWm9m7ZnZcj/0LzKzGzFaHbz+JZB4RkURR09RGW4dHvSikRvj1bwaWuvvnzCwdyOrlmOXufnaEc4iIJJRYzFGACBYFM8sBTgIuBXD3VqA1Uu8nIpJMumYzJ1GfwkFABXCXmb1pZneaWXYvxx1nZm+Z2dNmdmgE84iIJIxYLHEBkS0KqcCRwO3ufgTQAFzb45g3gEJ3Pxy4FXi0txcys8vNrMTMSioqKiIYWUQkPiRjUSgDytx9ZfjxEkJFoou717p7ffj+U0CameX1fCF3X+zuxe5enJ+fH8HIIiLxoaK+hfTUAMMzI931+3ERKwruvgPYYmbTw5tOAd7pfoyZjTUzC9+fG85TGalMIiKJYmdtM2OGZxD+ioyaSJeg7wJ/Co882ghcZmZXALj7IuBzwLfMrB1oAi5yd49wJhGRuLeztpkxUbw2c6eIFgV3Xw0U99i8qNv+24DbIplBRCQR7axtYeb44VF/X81oFhGJM+4es5aCioKISJypa2mnsbWDsTnRHXkEKgoiInGnvLYZgDHD1VIQERn0dtSE5iioKIiICDvVUhARkU47uoqC+hRERAa98tpmhmWmkpUe3dnMoKIgIhJ3dtQ2MzYGp45ARUFEJO7srG2JSX8CqCiIiMSd0LpHKgoiIoNeMOiU17XEpJMZVBREROJKZUMrHUFnbI5aCiIig17nHIXRMVj3CFQURETiSmdRUEtBRERiOnENVBREROLKztoWzCB/qIqCiMigt7OmmbyhGaSmxObrWUVBRCSO7KyL3WxmUFEQEYkrO2qaY9afABEuCmY2wsyWmNl6M3vXzI7rsd/M7BYz22Bma8zsyEjmERGJd6GJa7FrKUR6Cb6bgaXu/jkzSweyeuw/A5gavh0D3B7+KSIy6LS0d1DV0BrTohCxloKZ5QAnAb8HcPdWd6/ucdh5wD0esgIYYWbjIpVJRCSeldeGrriWrH0KBwEVwF1m9qaZ3Wlm2T2OKQC2dHtcFt4mIjLodM1mTtI+hVTgSOB2dz8CaACu3Z8XMrPLzazEzEoqKioGMqOISNzY2dlSiNFsZohsUSgDytx9ZfjxEkJForutwMRujyeEt32Muy9292J3L87Pz49IWBGRWOu6NnOM1j2CCBYFd98BbDGz6eFNpwDv9DjsMeDL4VFIxwI17r49UplEROLZztpm0lMDjMhKi1mGSI8++i7wp/DIo43AZWZ2BYC7LwKeAs4ENgCNwGURziMiErdCF9fJwMxiliGiRcHdVwPFPTYv6rbfgSsjmUFEJFHsqG2O6akj0IxmEZG4UV7bwpgYdjKDioKISFxwd7UUREQkpL6lncbWDsbmxG6OAqgoiIjEhe01nRfXUUtBRGTQ21TZCEBhbs+FH6JLRUFEJA5sqmwAoHBUz3VDo0tFQUQkDmyuamRYZmpMJ66BioKISFzYVNlIYW5WTCeugYqCiEhc2FzVSOGo2PYngIqCiEjMdQSdst2NTMqNbX8CqCiIiMTctuom2jo85p3MoKIgIhJzm6tCw1EnqSiIiEjnHAWdPhIRETZVNZCWYozLGRLrKCoKIiKxtqWqkYkjs0gJxHY4KqgoiIjE3KbK+Bh5BCoKIiIx5e5srmyMi5FHoKIgIhJTuxvbqGtpZ1KMF8LrpKIgIhJD8bIQXicVBRGRGOqco1AYJ30KqZF8cTMrBeqADqDd3Yt77F8A/A/wUXjTX939hkhmEhGJJ51zFCbGSUuhT0XBzLKBJncPmtk0YAbwtLu39eHpC9191172L3f3s/uSQ0Qk2WyqbGTM8Awy01JiHQXo++mjZUCmmRUAzwCXAH+MVCgRkcFic1VDXKyO2qmvRcHcvRH4DPBbd78AOLQPz3PgGTNbZWaX7+GY48zsLTN72sx6fU0zu9zMSsyspKKioo+RRUTiXzzNUYB+FAUzOw64GHgyvK0vbZ157n4kcAZwpZmd1GP/G0Chux8O3Ao82tuLuPtidy929+L8/Pw+RhYRiW9NrR2U17XEzcgj6HtRuBq4DnjE3deZ2cHAC/t6krtvDf8sBx4B5vbYX+vu9eH7TwFpZpbXj/wiIglry+74WQivU586mt39JeAlADMLALvc/aq9PSfcOR1w97rw/U8BN/Q4Ziyw093dzOYSKlKV/f8YIiKJp3PkUWGcTFyDPrYUzOx+Mxse/nJfC7xjZj/cx9PGAC+b2VvAa8CT7r7UzK4wsyvCx3wOWBs+5hbgInf3/fsoIiKJJd4mrkHf5ynMdPdaM7sYeBq4FlgF/PeenuDuG4HDe9m+qNv924Db+pVYRCRJbK5qZFhmKiOy0mIdpUtf+xTSzCwNOB94LDw/Qf+jFxE5AJurGpk0Kguz2C+Z3amvReEOoBTIBpaZWSFQG6lQIiKDwdbdTUwYGfsL63TXp6Lg7re4e4G7n+khm4CFEc4mIpK03J1t1U2MH5GARcHMcszsps4JZGb2S0KtBhER2Q+1Te00tHZQkIhFAfgDoYXtLgzfaoG7IhVKRCTZlVWHhqPGW1Ho6+ijye7+2W6Pf2pmqyMRSERkMNhW3QyQmKePgCYzm9f5wMxOAJoiE0lEJPltqw59hcZbUehrS+EK4B4zywk/3g18JTKRRESS37bqJtJTA+Rmp8c6ysf0dZmLt4DDzWx4+HGtmV0NrIlkOBGRZFVW3UTBiCEEAvEzRwH6eTnO8AJ2nfMTrolAHhGRQSE0HDUz1jE+4UCu0Rxf5U1EJIFsq25ifE589SfAgRUFLXMhIrIfWtuDlNe1UBBns5lhH30KZlZH71/+BsTfpxERSQA7appxj7+RR7CPouDuw6IVRERksNgaHo4abxPX4MBOH4mIyH6I1zkKoKIgIhJ1nS2FcTnJNfpIRET2w7bqJvKGZpCZlhLrKJ+goiAiEmVbq5soiMM5CtD3ZS4SXnltM+u215IWCJCaYqSlGGOGZ1IwYkhcXfVIRJLf1uomZoyNz3E8ES0KZlZKaMntDqDd3Yt77DfgZuBMoBG41N3fiESW10qr+M79b35i+9jhmRxVNJKjC0dy6swxTBgZPxfQFpHk03lxnZOnj451lF5Fo6Ww0N137WHfGcDU8O0Y4PbwzwF3wuQ8/vrt42nvcNo7grQFndJdDZRs2k1JaRVPrtnO9Y+/w1GFIznnsHEUF42itqmN3Y1tVDe1khowsjNSyc5IJSsthUDACBiYGWmBABlpATJSA6SnBgg6dHQ47cEgaSkBcrLSGJaRqhaJiLC7sY3mtmBcjjyC2J8+Og+4x90dWGFmI8xsnLtvH+g3GpmdzsgeqxHOn5bPV44vAqB0VwNPvr2dx9/axvWPvzPQb0/AYPiQNNJSAgQMUszISEshf2gG+cNCt2ljhjFn4gimjRlKaoq6e0SS0dbd8TscFSJfFBx4xswcuMPdF/fYXwBs6fa4LLztY0XBzC4HLgeYNGlSRIIW5WVz5cIpXLlwCu/vrGNjRT05Q9IZlZ3OiKw02oNOQ0s79S3tNLV2EHTHHTrc6ehwmts7aGkL0toRJMWMlICRmmK0tAepbWqjurGNmqY22oNBgkEIutPU1kFFXQvvbq/lxfeaaWjtAGBIWgqzCoYzOX8oRXnZFOVmM2V0NoW52aSpWIgktM7hqBPicIkLiHxRmOfuW81sNPCsma1392X9fZFwMVkMUFxcHPE1l6aNGca0MdHtBHJ3Nlc1snpLNW9urmbt1hqee3cnu+pbu45JSzEOzhvK1DFDmTp6GFNGh+4X5maRkRp/Q9tE5JPieeIaRLgouPvW8M9yM3sEmAt0LwpbgYndHk8Ibxt0zIzC3FBr4Lw5BV3ba5vbKN3VwIbyet7fWc8HO+t4q6yaJ9/ejnvncyF/aAbjRwyhYOQQDhk7jMMnjuCwCSPIGZIWo08kIr3ZVt1EZlqAkVnx+W8zYkXBzLKBgLvXhe9/Crihx2GPAd8xswcIdTDXRKI/IZENz0zjsAmhL/jumlo7+LCing8r6vloVwPbq5vZWt3E2q01PLnmH7/CKaOHcnTRKI45aBRzDxoVt/87ERkstoYvrhOvA08i2VIYAzwS/uCpwP3uvtTMrgBw90XAU4SGo24gNCT1sgjmSSpD0lOYVZDDrIKcT+yraWrj7bIaVm/ZzapNu3lizTb+/NpmAApzszhhSh7zpuRx3MG5n+h8F5HICl1cJ37/cxaxouDuG4HDe9m+qNt9B66MVIbBKmdIGvOm5jFvah4AHUFn/Y5aVm6s4tUPK3ls9TbuX7mZgMFxk3M557DxnD5rLCOyVCBEIm1rdTOHjBse6xh7FOshqRIFKQHj0PE5HDo+h6/OO4i2jiBryqp5YX0FT6zZxrV/fZsfP7qWk6blc87h4zj1kDEMy4zP850iiay5rYNd9S2Ds6Ug8SstJcBRhaM4qnAU3//UNNZureWxt7by5JrtPL++nIzUAAunj+YzRxawcMZoDYMVGSDba5qB+LyOQicVhUHOzJg9IYfZE3K47oxDeGPzbp5Ys50n1mxn6bod5A1N5/w5BVx49MSoD9MVSSb1Le3c/WopEL/DUQHMPbEutVxcXOwlJSWxjpH02juCvPR+BQ+XlPG/63fS1uHMLRrFl44r5PRDx5KeqtaDSF+0dwR5qKSMm559n131LZw/Zzw3XnB41FctMLNVPdef6/U4FQXZl8r6Fv7yRhn3rdjM5qpG8oamc8mxRXzl+EJ1TovsRUfQ+cLiFbxWWsXRRSP5lzMP4YhJI2OSRUVBBlww6CzfsIu7Xy3l+fXlZKencPGxhXx93kGMHh6fa8OLxNJ9Kzbx40fX8tNzD+XLxxXGdG5CX4uC+hSkzwIBY/60fOZPy2f9jlpuf/FD7ly+kT++WsplxxfxrQWT1XIQCatubOXGZ97jmINGxbwg9IdODMt+mTF2ODdfdAQv/GABZx82jsXLN3LSf73Aopc+pLmtI9bxRGLul8+8T21TG9efe2jCFARQUZADVJibzU0XzuGpq06kuGgUP396PQtvfJElq8oIBhPr1KTIQHlnWy1/WrmJS44tjOuJar1RUZABcci44fzh0qN54PJjGT0sgx88/BZn3/oyL3+wp+sriSQnd+f6x9YxIiuda06bHus4/aaiIAPq2INzeeTbJ3DzRXOobW7jS79fyTfvLaFsd2Oso4lExTPv7OS10ip++Onp5MTpSqh7o6IgAy4QMM6bU8D/fn8+Pzp9Osve38WpN73Ebc9/QEu7+hskuZWUVpGRGuDC4on7PjgOqShIxGSkpvDtBVN47vvzWTh9NDc+8z6n/3q5TilJUiutbKQwN4uUQOJ0LnenoiARVzBiCLd/6Sju/upcgu586fcr+d4Db1Je1xzraCIDblNlA4W52bGOsd9UFCRq5k/L529Xn8RVp0zl6bd3cMovX+K+FZs0SkmSRjDobKpspCg3K9ZR9puKgkRVZloK15w2jaVXn8jsghx+/OhaPrfoVd7bURfraCIHbGddMy3tQbUURPrr4Pyh/Onrx/DLCw7no10NnHXLcv5r6Xpa24Oxjiay30p3hUbZFakoiPSfmfHZoybwv99fwPlHFPDbFz/kM7e/wocV9bGOJrJfNlU2AKHL3iYqFQWJuVHZ6dx4weHccclRbN3dxNm3vMwDr20m0RZrFCmtbCQtxeL6egn7EvGiYGYpZvammT3Ry75LzazCzFaHb1+PdB6JX58+dCxLrz6JIwtHcO1f3+Yb95Swo0YjlCRxbKpsYOKoxB2OCtFpKXwPeHcv+x909znh251RyCNxbMzwTO796jH8+KxDeHnDLk676SX+tFIjlCQxlFY2JnR/AkS4KJjZBOAsQF/20meBgPH1Ew/mb1efxKyCHP71kbV84Xcr2FKlpTIkfrl7eI5C4vYnQORbCr8GfgTsbUjJZ81sjZktMbNe54Wb2eVmVmJmJRUVFREJKvGnMDeb+79xDL/47Gze2VbLmTcv59E3t8Y6lkivKupbaGztUEthT8zsbKDc3Vft5bDHgSJ3Pwx4Fri7t4PcfbG7F7t7cX5+fgTSSrwyMz5/9CSe+t6JTB87jKsfXM33HniT2ua2WEcT+ZhNlaGWrFoKe3YCcK6ZlQIPACeb2X3dD3D3SndvCT+8EzgqgnkkgU0clcUDlx/LNadN44k12zn9V8t4dYPWUJL4UborNBxVLYU9cPfr3H2CuxcBFwHPu/uXuh9jZuO6PTyXvXdIyyCXmhLgqlOmsuSK48hIS+GLd67kp4+v05XeJC5sqmwkJWAUjEzc4agQg3kKZnaDmZ0bfniVma0zs7eAq4BLo51HEs8Rk0by1FUn8pXjCrnrlVLOumU5b27eHetYMsiVVjYwYeQQ0lISe/qXJdoEoeLiYi8pKYl1DIkTyz+o4EdL1rCztplvnHgw/3TaNDLTUmIdSwahc259mZHZ6dzz1bmxjtIrM1vl7sX7Oi6xS5oMeidOzedv/3QSFxZP5I5lGznzluWs2lQV61gyyLg7pZUNCb06aicVBUl4wzPT+PlnD+Per82lpS3I5xb9nR8/+jY1TRqhJNGxu7GNuub2hF4dtZOKgiSNzlbDZccfxP0rN3PqTS/x+FvbtIaSRFxpZefII7UUROLK0IxUfnLOTP7nynmMHZ7Jd//8Jpfe9bpmQ0tE/WN1VLUUROLS7Ak5PHrlCfzk7JmUlFZx2q9e4o6XPqStQ9drkIFXuqsRM5g4KrGHo4KKgiSxlIDx1XkH8ew185k3JZ+fPb2ec297hbe2VMc6miSZTZUNjM8ZQkZq4o98U1GQpDd+xBB+9+WjWPSlI6lqaOH8377C9Y+to76lPdbRJEmUVjZSlJf4/QmgoiCDhJlx+qxxPHvNfC45tpC7/17KaTe9xDPrdsQ6miSB0Oqoid+fACoKMsgMz0zjhvNm8ZdvHU/OkDQuv3cVX7/7dcp2qyNa9s8z63awu7GNQ8YOi3WUAaGiIIPSkZNG8vh353HdGTN4ZUMlp970Ere/+CGt7eqIlr7bUdPMj/6yhlkFw/n80ZNiHWdAqCjIoJWWEuCb8yfz3Pfnc+LUfH6xdD1n3bKcFRsrYx1NEkAw6Hz/4dW0tAW5+aIjSE9Njq/T5PgUIgegYMQQfvflYu78cjFNbR1ctHgF1zy4moq6ln0/WQat3y3fyCsbKvm3c2YyOX9orOMMGBUFkbBTZ47h2X+az3cWTuHxNds4+cYXWbzsQ1ratTS3fNzbZTX899/e44xZY/n80b1eMDJhqSiIdDMkPYUffHo6S68+iaMPGsV/PrWeT/1qGX9bt0PLZQgA1Y2tfPv+VeQPy+Bnn5mNmcU60oBSURDpxeT8ofzh0qO5+6tzQ30P967iwjv+zuulWoF1MOsIOlc9sJqdNS389uIjGZGVHutIA05FQWQv5k/LZ+n3TuTfz59FaWUjFyz6O5fd9RrvbKuNdTSJgV8/9z7L3q/g+nMP5YhJI2MdJyJUFET2ITUlwCXHFrLshwv559NnsGrTbs66dTnXPLSabdVNsY4nUfLMuh3c+vwGPl88kS/MTa5+hO505TWRfqppbOO3L27grldLMeCr8w7iivmTyRmSFutoEiFbqho58+blHJSfzUPfPC4hr+6nK6+JREhOVhrXnXkIz39/PmfOHsftL37IvF88z03PvEd1Y2us48kAc3f+5ZG3Cbrz24uPTMiC0B8RLwpmlmJmb5rZE73syzCzB81sg5mtNLOiSOcRGSgTRmbxq8/P4cmr5nHC5DxueX4DJ/z8eX7+9Hp21WuOQ7L4yxtbWf7BLv75jBlMGJkci97tTTRaCt8D3t3Dvq8Bu919CvAr4BdRyCMyoA4dn8OiS45i6dUnsnDGaO5YFmo53PD4O+yoaY51PDkAFXUt/PsT71BcOJIvHVMY6zhREdGiYGYTgLOAO/dwyHnA3eH7S4BTLNkG/cqgMWPscG774pE8d818zpo9nrv/XspJ//UC//LI27ryW4L66ePraGrt4OefnU0gMDi+miLdUvg18CNgT6uMFQBbANy9HagBcnseZGaXm1mJmZVUVFREKqvIgJicP5RfXng4L/5gAZ8rnsCSkjIW3Pgi1zy4mg921sU6nvTRc+/s5Ik12/nOyVOYMjo5VkDti4gVBTM7Gyh391UH+lruvtjdi929OD8/fwDSiUTexFFZ/Of/mc2yHy3ksuOLeHrtDk771TK+9sfXeXXDLs2QjmPba5q49q9rmD5mGFfMnxzrOFEVyZbCCcC5ZlYKPACcbGb39ThmKzARwMxSgRxAS1RKUhmbk8mPz57JK9eezPdOmcrqLdV88c6VnHnLyzxUsoXmNq2tFE+a2zq44t5VNLcF+c3FybP6aV9FZZ6CmS0AfuDuZ/fYfiUw292vMLOLgM+4+4V7ey3NU5BE19zWwWOrt/H7lz/ivZ115AxJ43NHTeDiYyZxcBKttpmI3J0fLVnDw6vKuOOSo/j0oWNjHWnA9HWeQmo0wnRnZjcAJe7+GPB74F4z2wBUARdFO49ItGWmpXDh0RO5oHgCKzZWcd/KTdz9aim/f/kjjp+cy8XHFHLazDGD7n+o8eC+FZt4eFUZV508JakKQn9oRrNIHCiva+ah17fw59e2sLW6ibyh6VxQPJEvzp3ExFHJPzY+Hrz2URVf/N0K5k/L53dfLk660UZ9bSmoKIjEkY6gs+z9Cv60cjPPr9+JAwunj+aSYws5aVo+KUn2RRUvtlQ1ct5vXmHEkDQeufKEpFyyJG5PH4nInqUEjIUzRrNwxmi2VjfxwGub+fNrW7jsj69TMGIInw+fdhqXMyTWUZNGfUs737inhPaOIHd+pTgpC0J/qKUgEuda24P8bd0OHnh9M69sqCRgsGD6aD5/9EROnjGatBT1PeyvYNC5/N5VvPBeOX+87GhOnJq8Q97VUhBJEumpAc45fDznHD6eTZUNPFSyhYdLynh+fTl5QzP47JEFXFA8YVBNsBooNz7zHs+9u5Prz5mZ1AWhP9RSEElA7R1BXnyvggdLtvD8+nI6gs7MccM5d06oeBSM0OmlfVm6djtX3PcGX5g7kf/8P8l3Wc2e1NEsMkiU1zXz5Jrt/M/qbazeUg3AEZNGcOascZw+a6xGL/Xiw4p6zrvtFSaPHspD3zyWjNTkXg4bVBREBqVNlQ08sWY7T729nXXhS4bOLsjh04eO4fRZY3WKCWhsbef837zCrvpWnvjuPMYPklaVioLIILe5spGn1m5n6dodXS2Ig/Oz+dTMsZw2cwxHTByRdGPx98Xd+d4Dq3lizTbu+eoxzJuaF+tIUaOiICJddtQ088w7O1i6dgcrP6qiI+jkDc3glBmjmT89nxOm5CX9UMwtVY38+rkP+MsbZfzw09O5cuGUWEeKKhUFEelVTWMbL75fzrPv7OSl9yqoa2knJWDMmTiCE6fmMW9KHodPHJE0Q123VDXymxc2sGRVGQEzvnJ8IdedccigayWpKIjIPrV3BFm9pZqX3q9g2fsVrNlagztkp6dw7MG5nDQtn/nT8inKy4511H4rr23mluc/4IHXthAIGF+cO4lvzj940E78U1EQkX6rbmxlxcZKXt6wi+Uf7GJTZeiKcUW5WRw/JY/ZBTnMLshh2phhcbtgX1VDK3cu38gfXvmI9g7nC3Mn8Z2TpzBmeGaso8WUioKIHLDSXQ289H4FL75XTsmm3dQ1twOQlmLkD81gZHY6o7LTyR+awYRRWRSOymJSbhYjs9LJSk9hSFoKQ9JTyEyL3JBPd+e9nXU8v76cF9aXs2rTboIO580ZzzWnTaMwN/FaOZGgoiAiA8rd2VzVyNtba1i3rZaKuhaqGlqpao/nehIAAAlQSURBVGilvLaZ7bXN7OnrZFhmKqOHZTB6WCZ5wzIYmZXGyKx0coemc8i44cwuyOlT4QgGnfK6FjZVNrCmrIbXSqsoKa1id2MbALMKhnPyjDGcNXsc08dq+G13WuZCRAaUmVGYm01hbjZnHzb+E/tb24NsrW5ic1Uj1Y2tNLd10NTaQUNrBxV1LZTXNVNe28LarTXsbmylOvxFDpAaMGaMG0ZRbjZtHUFa2oO0tAVpDwZpDzrBoNPQ2sGWqkZa2v9xyfei3CxOPWQMRxeNYv70/EF/imggqCiIyIBITw1wUF42B/WxU7q9I8iu+lbe3lrD6i27eXNzNWu31pCRmkJmWoD01ABpKQEy04yAGWNzAiyYlk9hbhaTcrM5ZOwwRqsIDDgVBRGJidSUAGNzMhmbk8lpM8fEOo6ExefwARERiQkVBRER6RKxomBmmWb2mpm9ZWbrzOynvRxzqZlVmNnq8O3rkcojIiL7Fsk+hRbgZHevN7M04GUze9rdV/Q47kF3/04Ec4iISB9FrCh4aAJEffhhWviWWJMiREQGmYj2KZhZipmtBsqBZ919ZS+HfdbM1pjZEjObuIfXudzMSsyspKKiIpKRRUQGtYgWBXfvcPc5wARgrpnN6nHI40CRux8GPAvcvYfXWezuxe5enJ+v66iKiERKVEYfuXs18AJweo/tle7eEn54J3BUNPKIiEjvItanYGb5QJu7V5vZEOA04Bc9jhnn7tvDD88F3t3X665atWqXmW0CcoCa8OZ93e/8mQfs2o+P0/01+7p/X9sSIfPeHg905v3Ju7+Ze9uWKJnj4e9iTxn3lX2wZI7Xv+XCfQUHQotcReIGHAa8CawB1gI/CW+/ATg3fP9nwDrgLUItiRn9eP3Ffb3f7WfJfn6Wxf3dv69tiZB5b48HOvP+5N3fzHvYlhCZ4+Hvoi9/C4M5cyL+LXe/RXL00RrgiF62/6Tb/euA6/bzLR7vx/3u2w70vfq6f1/bEiHz3h4PdOb9ydvb9r5k3tPn6K9YZI6Hv4ue2xLhb7nnNv0t70HCLZ19IMysxPuwdGw8UeboSLTMiZYXlDlaDjTzYFvmYnGsA+wHZY6ORMucaHlBmaPlgDIPqpaCiIjs3WBrKYiIyF6oKIiISBcVBRER6aKiEGZmJ5rZIjO708xejXWevjCzgJn9h5ndamZfiXWevjCzBWa2PPy7XhDrPH1hZtnhtbfOjnWWvjCzQ8K/3yVm9q1Y5+kLMzvfzH5nZg+a2adinacvzOxgM/u9mS2JdZY9Cf/t3h3+3V7cl+ckRVEwsz+YWbmZre2x/XQze8/MNpjZtXt7DXdf7u5XAE+whzWYBtJAZAbOI7SuVBtQFqms3bINRObO1XMziXDmAcoL8M/AQ5FJ+XED9Lf8bvhv+ULghEjmDWcbiMyPuvs3gCuAz0cybzjbQGTe6O5fi2zST+pn9s8AS8K/23P79AYHMvMtXm7AScCRwNpu21KAD4GDgXRCs6ZnArMJffF3v43u9ryHgGGJkBm4Fvhm+LlLEiRzIPy8McCfEiDvacBFwKXA2YnwOw4/51zgaeCLiZI5/LxfAkcmWOaI/9s7gOzXAXPCx9zfl9eP5EV2osbdl5lZUY/Nc4EN7r4RwMweAM5z958BvZ4GMLNJQI2710UwLjAwmc2sDGgNP+yIXNqQgfo9h+0GMiKRs9MA/Y4XANmE/oE1mdlT7h6M58zh13kMeMzMngTuj1Te8HsNxO/ZgJ8DT7v7G5HMCwP+txxV/clOqDU+AVhNH88MJUVR2IMCYEu3x2XAMft4zteAuyKWaN/6m/mvwK1mdiKwLJLB9qJfmc3sM8CngRHAbZGN1qt+5XX3f4XQpWOBXZEsCHvR39/xAkKnDTKApyKabM/6+7f8XeBUIMfMprj7okiG24P+/p5zgf8AjjCz68LFI1b2lP0W4DYzO4s+LoORzEWh39z932KdoT/cvZFQIUsY7v5XQsUsobj7H2Odoa/c/UXgxRjH6Bd3v4XQF1jCcPdKQn0gccvdG4DL+vOcpOho3oOtQPcruU0Ib4tnyhx5iZYXlDlaEjFzpwHLnsxF4XVgqpkdZGbphDoLH4txpn1R5shLtLygzNGSiJk7DVz2aPaaR7A3/s/Adv4xNPNr4e1nAu8T6pX/11jnVGblVeb4uCVi5mhl14J4IiLSJZlPH4mISD+pKIiISBcVBRER6aKiICIiXVQURESki4qCiIh0UVGQpGBm9VF+vwG55oaFri9RY2arzWy9md3Yh+ecb2YzB+L9RXpSURDphZntdV0wdz9+AN9uubvPAY4AzjazfV0D4XxCq7aKDDgVBUlaZjbZzJaa2SoLXe1tRnj7OWa20szeNLPnzGxMePv1Znavmb0C3Bt+/Acze9HMNprZVd1euz78c0F4/5Lw//T/FF4GGjM7M7xtlZndYmZP7C2vuzcRWuK4IPz8b5jZ62b2lpn9xcyyzOx4QtdK+O9w62Lynj6nyP5QUZBkthj4rrsfBfwA+G14+8vAse5+BPAA8KNuz5kJnOruXwg/nkFoqe+5wL+ZWVov73MEcHX4uQcDJ5hZJnAHcEb4/fP3FdbMRgJT+ccy6H9196Pd/XDgXULLGbxKaE2bH7r7HHf/cC+fU6TftHS2JCUzGwocDzwc/o87/OOiPhOAB81sHKGrVH3U7amPhf/H3ulJd28BWsysnNAV43peRvQ1dy8Lv+9qoIjQJUc3unvna/8ZuHwPcU80s7cIFYRfu/uO8PZZZvb/CF17Yijwt35+TpF+U1GQZBUAqsPn6nu6FbjJ3R8LX5Dm+m77Gnoc29Ltfge9/5vpyzF7s9zdzzazg4AVZvaQu68G/gic7+5vhS/ys6CX5+7tc4r0m04fSVJy91rgIzO7AEKXezSzw8O7c/jHWvNfiVCE94CDu102cZ8Xow+3Kn4O/HN40zBge/iU1cXdDq0L79vX5xTpNxUFSRZZZlbW7XYNoS/Sr4VPzawjdM1aCLUMHjazVcCuSIQJn4L6NrA0/D51QE0fnroIOClcTP4vsBJ4BVjf7ZgHgB+GO8ons+fPKdJvWjpbJELMbKi714dHI/0G+MDdfxXrXCJ7o5aCSOR8I9zxvI7QKas7YpxHZJ/UUhARkS5qKYiISBcVBRER6aKiICIiXVQURESki4qCiIh0UVEQEZEu/x8rE/KEqRPQCAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light",
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = Model6(tok.vocab_sz, 512, 6, p_emb=0.1, p_ff=0.1, tie_weights=True)\n",
    "learn = Learner(dls, model, loss_func=CrossEntropyLossFlat(), metrics=[accuracy, perplexity]).to_native_fp16()\n",
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "id": "sMLTtpR-sshr",
    "outputId": "017301bc-5a63-4883-b0a9-ca28dcac71b8"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.207689</td>\n",
       "      <td>3.014779</td>\n",
       "      <td>0.187012</td>\n",
       "      <td>20.384583</td>\n",
       "      <td>00:06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2.957462</td>\n",
       "      <td>2.648416</td>\n",
       "      <td>0.258105</td>\n",
       "      <td>14.131640</td>\n",
       "      <td>00:06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2.645952</td>\n",
       "      <td>2.427977</td>\n",
       "      <td>0.287435</td>\n",
       "      <td>11.335924</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2.499318</td>\n",
       "      <td>2.395460</td>\n",
       "      <td>0.292887</td>\n",
       "      <td>10.973244</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2.436253</td>\n",
       "      <td>2.394616</td>\n",
       "      <td>0.288965</td>\n",
       "      <td>10.963985</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>2.402833</td>\n",
       "      <td>2.364455</td>\n",
       "      <td>0.295703</td>\n",
       "      <td>10.638234</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>2.387243</td>\n",
       "      <td>2.367871</td>\n",
       "      <td>0.290381</td>\n",
       "      <td>10.674637</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>2.375951</td>\n",
       "      <td>2.384834</td>\n",
       "      <td>0.285010</td>\n",
       "      <td>10.857258</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>2.372440</td>\n",
       "      <td>2.380138</td>\n",
       "      <td>0.276270</td>\n",
       "      <td>10.806398</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>2.355436</td>\n",
       "      <td>2.329239</td>\n",
       "      <td>0.305892</td>\n",
       "      <td>10.270120</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10</td>\n",
       "      <td>2.314718</td>\n",
       "      <td>2.250078</td>\n",
       "      <td>0.331901</td>\n",
       "      <td>9.488480</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>11</td>\n",
       "      <td>2.231658</td>\n",
       "      <td>2.096768</td>\n",
       "      <td>0.385319</td>\n",
       "      <td>8.139816</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12</td>\n",
       "      <td>2.145322</td>\n",
       "      <td>1.989301</td>\n",
       "      <td>0.413477</td>\n",
       "      <td>7.310425</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>13</td>\n",
       "      <td>1.998541</td>\n",
       "      <td>1.862125</td>\n",
       "      <td>0.444971</td>\n",
       "      <td>6.437402</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>14</td>\n",
       "      <td>1.880289</td>\n",
       "      <td>1.772380</td>\n",
       "      <td>0.465283</td>\n",
       "      <td>5.884840</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>15</td>\n",
       "      <td>1.777548</td>\n",
       "      <td>1.735309</td>\n",
       "      <td>0.482080</td>\n",
       "      <td>5.670681</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>16</td>\n",
       "      <td>1.692429</td>\n",
       "      <td>1.649408</td>\n",
       "      <td>0.501937</td>\n",
       "      <td>5.203897</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>17</td>\n",
       "      <td>1.616076</td>\n",
       "      <td>1.616357</td>\n",
       "      <td>0.513688</td>\n",
       "      <td>5.034715</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>18</td>\n",
       "      <td>1.548247</td>\n",
       "      <td>1.586346</td>\n",
       "      <td>0.522575</td>\n",
       "      <td>4.885864</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>19</td>\n",
       "      <td>1.484927</td>\n",
       "      <td>1.550523</td>\n",
       "      <td>0.532633</td>\n",
       "      <td>4.713936</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>20</td>\n",
       "      <td>1.430124</td>\n",
       "      <td>1.512773</td>\n",
       "      <td>0.543213</td>\n",
       "      <td>4.539303</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>21</td>\n",
       "      <td>1.375972</td>\n",
       "      <td>1.500666</td>\n",
       "      <td>0.545199</td>\n",
       "      <td>4.484674</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>22</td>\n",
       "      <td>1.324876</td>\n",
       "      <td>1.491262</td>\n",
       "      <td>0.552637</td>\n",
       "      <td>4.442699</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>23</td>\n",
       "      <td>1.276637</td>\n",
       "      <td>1.469852</td>\n",
       "      <td>0.557975</td>\n",
       "      <td>4.348590</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>24</td>\n",
       "      <td>1.229700</td>\n",
       "      <td>1.477631</td>\n",
       "      <td>0.560189</td>\n",
       "      <td>4.382551</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>25</td>\n",
       "      <td>1.186633</td>\n",
       "      <td>1.459370</td>\n",
       "      <td>0.562956</td>\n",
       "      <td>4.303250</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>26</td>\n",
       "      <td>1.139820</td>\n",
       "      <td>1.467342</td>\n",
       "      <td>0.564225</td>\n",
       "      <td>4.337692</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>27</td>\n",
       "      <td>1.092420</td>\n",
       "      <td>1.476885</td>\n",
       "      <td>0.566960</td>\n",
       "      <td>4.379283</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>28</td>\n",
       "      <td>1.050866</td>\n",
       "      <td>1.486061</td>\n",
       "      <td>0.567350</td>\n",
       "      <td>4.419652</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>29</td>\n",
       "      <td>1.006091</td>\n",
       "      <td>1.505293</td>\n",
       "      <td>0.568506</td>\n",
       "      <td>4.505473</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>30</td>\n",
       "      <td>0.957875</td>\n",
       "      <td>1.528569</td>\n",
       "      <td>0.567106</td>\n",
       "      <td>4.611572</td>\n",
       "      <td>00:07</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No improvement since epoch 25: early stopping\n"
     ]
    }
   ],
   "source": [
    "#collapse_output\n",
    "learn.fit_one_cycle(50, 5e-4, cbs=EarlyStoppingCallback(patience=5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rIkhquVDJS2I"
   },
   "source": [
    "### Text generation\n",
    "\n",
    "Text generation is a big topic on it's own. One can refer to great posts [by Patrick von Platen from HuggingFace](https://huggingface.co/blog/how-to-generate) and [Lilian Weng](https://lilianweng.github.io/lil-log/2021/01/02/controllable-neural-text-generation.html) for more details on various approaches. Here I will use nucleus sampling. This method rallies on sampling from candidates compounding certain value of probability mass. Intuitively this approach should work for character level generation: when there is only one grammatically correct option for continuation we always want to select it, but when starting a new word some diversity in outputs is desirable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "SwN6sMhqsshs"
   },
   "outputs": [],
   "source": [
    "#collapse-hide\n",
    "def expand_dim1(x):\n",
    "    if len(x.shape) == 1:\n",
    "        return x[None, :]\n",
    "    else: return x\n",
    "\n",
    "def top_p_filter(logits, top_p=0.9):\n",
    "    sorted_logits, sorted_indices = torch.sort(logits, descending=True)\n",
    "    cum_probs = torch.cumsum(F.softmax(sorted_logits, dim=-1), dim=-1)\n",
    "\n",
    "    sorted_indices_to_remove = cum_probs > top_p\n",
    "    sorted_indices_to_remove[..., 1:] = sorted_indices_to_remove[..., :-1].clone()\n",
    "    sorted_indices_to_remove[..., 0] = 0\n",
    "    indices_to_remove = sorted_indices_to_remove.scatter(1, sorted_indices, sorted_indices_to_remove)\n",
    "    logits[indices_to_remove] = float('-inf')\n",
    "    return logits\n",
    "\n",
    "@torch.no_grad()\n",
    "def generate(model, inp,\n",
    "            max_len=50,\n",
    "            temperature=1.,\n",
    "            top_k = 20,\n",
    "            top_p = 0.9,\n",
    "            early_stopping=False, #need eos_idx to work\n",
    "            eos_idx=None):\n",
    "    model.to(inp.device)\n",
    "    model.eval()\n",
    "    thresh = top_p\n",
    "    inp = expand_dim1(inp)\n",
    "    b, t = inp.shape\n",
    "    out = inp\n",
    "    for _ in range(max_len):\n",
    "        x = out\n",
    "\n",
    "        logits = model(x)[:, -1, :]\n",
    "        filtered_logits = top_p_filter(logits)\n",
    "        probs = F.softmax(filtered_logits / temperature, dim=-1)\n",
    "        sample = torch.multinomial(probs, 1)\n",
    "\n",
    "        out = torch.cat((out, sample), dim=-1)\n",
    "\n",
    "        if early_stopping and (sample == eos_idx).all():\n",
    "            break\n",
    "    return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 166
    },
    "id": "mrS9x3desshs",
    "outputId": "33987ede-7fbf-49a8-ea0f-8eef9683c969"
   },
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "ignored",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-80-9886797f8ab8>\u001b[0m in \u001b[0;36m<cell line: 1>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mout\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgenerate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlearn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtok\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Alice said '\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmax_len\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m200\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mearly_stopping\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0meos_idx\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtok\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mc2i\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'xxeos'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m: name 'tok' is not defined"
     ]
    }
   ],
   "source": [
    "out = generate(learn.model, tok('Alice said '), max_len=200, early_stopping=True, eos_idx=tok.c2i['xxeos'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "pf9y-n7Usshs",
    "outputId": "31a00d55-0dc7-48b5-e1cf-fd13033cda1b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Alice said in a minute turn, only purind his with it migut at in musible. i cant elp out my why yested it to like thought: i know i did it wish indeed it hope?\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(tok.decode(out[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8ildBoOl5GsX"
   },
   "source": [
    "Our relatively simple model learned to generate mostly grammatically plausible text, but it's not entirely coherent. But it would be too much to ask from the model to learn language from scratch by \"reading\" only two novels (however great those novels are).  To get more from the model let's feed it larger corpus of data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mlwC--kfKpqa"
   },
   "source": [
    "### Pretraining on larger dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "DPuzCtAQszwS",
    "outputId": "1347266d-3201-4d87-ce69-65352b256f9e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[K     |████████████████████████████████| 163kB 14.0MB/s \n",
      "\u001b[K     |████████████████████████████████| 17.7MB 207kB/s \n",
      "\u001b[K     |████████████████████████████████| 245kB 54.7MB/s \n",
      "\u001b[?25h"
     ]
    }
   ],
   "source": [
    "#hide\n",
    "import sys\n",
    "if 'google.colab' in sys.modules:\n",
    "    !pip install -Uqq datasets\n",
    "\n",
    "from datasets import load_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wmOSmo6N5GsX"
   },
   "source": [
    "For this purpose I will use a sample from [bookcorpus dataset](https://huggingface.co/datasets/bookcorpus). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 237,
     "referenced_widgets": [
      "2d2bb02afe8c4309a87354fd7858e864",
      "89f78d2a56c14ceaa6b1ebba54c4feb2",
      "8ecda540586c4760bf94ea3eae0767d2",
      "201aa25469944b0b80c9363fcc4fc95f",
      "298d60fa5cd8441f86187130f3729e73",
      "7977691d32f54349a98abf6008504c3e",
      "fa3423ee18214e89a9ff45554a5a2d21",
      "55f2f9482e3042bf81cb51aa64defd18",
      "0ba48d9e012f457c99d9e050ee66ce19",
      "0ff15dad1c594db7b6ed85b2d4261a63",
      "0a60355694ac41c3bb36988061280cd0",
      "9542a60fecc74375b59c2ef4b667c624",
      "a2628b3a684444b9b3aca1c36e111516",
      "db53b231459c49aa9d8fcef70617ef0b",
      "cbd787e3c17c402299963144601fb8f0",
      "ec9d69f76b464d54ad54fd49738e0420",
      "1f5d679f52bf417dac756ae3e85f864a",
      "cbe70f7f0c7c4bd99bf858a1d72d63c7",
      "4468101700ae4bccb66bba2dd10a13d9",
      "13c5eb342bfd4489b067effccb6d2118",
      "a865a148857c4d1c962381692bd59857",
      "abfe640017214179908de35ea38474ce",
      "493d11cd156a4bfcb2bdc25aa21802fe",
      "9fe8214a7b1941039aadf375b90373f6",
      "0022be0dc7aa4f4f9d947589308153f8",
      "5a0205b7d88a42a68d0f8d40265add8b",
      "b5fe978e2b204b5a986feb53c241d943",
      "f61af757678d40e7b6c470ca779a0e1b",
      "7571cf9691ac419fb79dad6dfaa531d1",
      "febdc99b2c074afb8e1b11aa35e7fcdf",
      "789151104986470ea6b887996b5833b5",
      "2c46976452b8428da3f74d1d37267a55"
     ]
    },
    "id": "kSB3GygHt1om",
    "outputId": "cc02ad34-ee92-4b5a-9e55-5032d4a93e8e"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2d2bb02afe8c4309a87354fd7858e864",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=1689.0, style=ProgressStyle(description…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0ba48d9e012f457c99d9e050ee66ce19",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=946.0, style=ProgressStyle(description_…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Downloading and preparing dataset bookcorpus/plain_text (download: 1.10 GiB, generated: 4.52 GiB, post-processed: Unknown size, total: 5.62 GiB) to /root/.cache/huggingface/datasets/bookcorpus/plain_text/1.0.0/af844be26c089fb64810e9f2cd841954fd8bd596d6ddd26326e4c70e2b8c96fc...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1f5d679f52bf417dac756ae3e85f864a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Downloading', max=1179510242.0, style=ProgressStyle(descr…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0022be0dc7aa4f4f9d947589308153f8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=1.0, bar_style='info', max=1.0), HTML(value='')))"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset bookcorpus downloaded and prepared to /root/.cache/huggingface/datasets/bookcorpus/plain_text/1.0.0/af844be26c089fb64810e9f2cd841954fd8bd596d6ddd26326e4c70e2b8c96fc. Subsequent calls will reuse this data.\n"
     ]
    }
   ],
   "source": [
    "#hide_ouput\n",
    "dataset = load_dataset(\"bookcorpus\", split='train')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 225
    },
    "id": "XMSG4_7X2kFN",
    "outputId": "20ad4748-8c38-41c3-f1da-bdd86597f8e1"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>the half-ling book one in the fall of igneeria series kaylee soderburg copyright 2013 kaylee soderburg all rights reserved .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>isbn : 1492913731 isbn-13 : 978-1492913733 for my family , who encouraged me to never stop fighting for my dreams chapter 1 summer vacations supposed to be fun , right ?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>i wish i had a better answer to that question .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>starlings , new york is not the place youd expect much to happen .</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>its a small quiet town , the kind where everyone knows your name .</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                        text\n",
       "0                                               the half-ling book one in the fall of igneeria series kaylee soderburg copyright 2013 kaylee soderburg all rights reserved .\n",
       "1  isbn : 1492913731 isbn-13 : 978-1492913733 for my family , who encouraged me to never stop fighting for my dreams chapter 1 summer vacations supposed to be fun , right ?\n",
       "2                                                                                                                            i wish i had a better answer to that question .\n",
       "3                                                                                                         starlings , new york is not the place youd expect much to happen .\n",
       "4                                                                                                         its a small quiet town , the kind where everyone knows your name ."
      ]
     },
     "execution_count": 65,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(dataset[:10_000_000])\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "PKU8ohEgGWM5"
   },
   "outputs": [],
   "source": [
    "df['len'] = df['text'].str.len()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7xiHw1ljaESl"
   },
   "outputs": [],
   "source": [
    "cut = int(len(df)*0.8)\n",
    "splits = range_of(df)[:cut], range_of(df[cut:])\n",
    "tfms = Pipeline([ColReader('text'), tok])\n",
    "dsets = Datasets(df, tfms=tfms, dl_type=LMDataLoader, splits=splits)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "BAqaPvL479Rn"
   },
   "outputs": [],
   "source": [
    "#collapse\n",
    "@patch\n",
    "def create_item(self:LMDataLoader, seq):\n",
    "    if seq>=self.n: raise IndexError\n",
    "    sl = self.last_len if seq//self.bs==self.n_batches-1 else self.seq_len\n",
    "    st = (seq%self.bs)*self.bl + (seq//self.bs)*self.seq_len\n",
    "    txt = self.chunks[st : st+sl+1]    \n",
    "    return LMTensorText(txt[:-1]),txt[1:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "BWHxN6xtq9NF",
    "outputId": "3bfbe27b-749e-44e0-dd40-8ec1a9db0862"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 13min 29s, sys: 13 s, total: 13min 42s\n",
      "Wall time: 13min 35s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "dl_kwargs = [{'lens':df['len'].values[splits[0]]}, {'val_lens':df['len'].values[splits[1]]}]\n",
    "dls = dsets.dataloaders(bs=32, seq_len=512, dl_kwargs=dl_kwargs, shuffle_train=True, num_workers=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 313
    },
    "id": "SKnflwtnsFaT",
    "outputId": "c2f43ed3-8460-4288-bd4e-1bd68afc2575"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>text_</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>im sorry .`` ah , this is katrice , the rowan queen , coming toward us . ''i said ill tell you !but she still did n't understand why he was a slave here .the sprites had come to believe by now that they were a forgotten people .thats what happened .well have to take him with us for a ways and then let him go .nick stifled the fear in him that was trying to take over .crouching down behind the stone balusters , with every nerve tingling , valeria glared down at the stealthy figure .there did seem to be shado</td>\n",
       "      <td>m sorry .`` ah , this is katrice , the rowan queen , coming toward us . ''i said ill tell you !but she still did n't understand why he was a slave here .the sprites had come to believe by now that they were a forgotten people .thats what happened .well have to take him with us for a ways and then let him go .nick stifled the fear in him that was trying to take over .crouching down behind the stone balusters , with every nerve tingling , valeria glared down at the stealthy figure .there did seem to be shadow</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>habitants were genuine , hard working , proud and tough .sergeant colon 's view of the world was certainly changing .'we did n't want any part of it , ' the grandfather continued , heedless of his company , 'but maybe that 's just how the rhega are destined to die ... not by our own hands , our own fights .she looked enquiring , but no one took any notice of her .more land would n't save his people-they needed something else .the other got off a single shot of his .45 caliber pistol before he was clawed acr</td>\n",
       "      <td>abitants were genuine , hard working , proud and tough .sergeant colon 's view of the world was certainly changing .'we did n't want any part of it , ' the grandfather continued , heedless of his company , 'but maybe that 's just how the rhega are destined to die ... not by our own hands , our own fights .she looked enquiring , but no one took any notice of her .more land would n't save his people-they needed something else .the other got off a single shot of his .45 caliber pistol before he was clawed acro</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "dls.show_batch(max_n=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 302
    },
    "id": "HdEyccfops-k",
    "outputId": "8c15043d-2aee-47d1-a46d-bae586292adf"
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "SuggestedLRs(lr_min=0.06309573650360108, lr_steep=0.5248074531555176)"
      ]
     },
     "execution_count": 71,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxc1X338c9Pu0b7asuS5QUbr9jGFpCwFTCkhDiBkJCSbiQPwcnTPlmevJo2fbqkS9KkrzZpS9I2kJKlJJAGBwKEPZAACQSQjR3vGCwv2vdtRtJoOc8fMxLCeJEszdy5o+/79dJLM3fmzvlpPNLX555zzzXnHCIiIgApXhcgIiKJQ6EgIiITFAoiIjJBoSAiIhMUCiIiMkGhICIiE9K8LmAqSktL3eLFi70uQ0TEV7Zv397unCubzj6+CIXFixdTW1vrdRkiIr5iZkenu48OH4mIyASFgoiITFAoiIjIBIWCiIhMUCiIiMiEmIWCmX3bzFrNbM+kbTeZ2V4zGzOzmli1LSIiZyeWPYXvAteesG0PcCPwXAzbFRHxvZ6BYZ7Y20x7/1Bc241ZKDjnngM6T9i23zl3MFZtiogki0MtfXz87u3sbeyNa7sJO6ZgZlvNrNbMatva2rwuR0QkrrpCwwAUBdLj2m7ChoJz7k7nXI1zrqasbFpnaYuI+F5XMAxAUSAjru0mbCiIiMxlXaFoKOQoFERE5rzOUJiM1BRyMlLj2m4sp6TeC7wIrDCzejO71czeb2b1wDuBR8zsiVi1LyLiZ93BYQoD6ZhZXNuN2SqpzrkPn+KhB2LVpohIsugMhSmO86Ej0OEjEZGE1B0KUxjnmUegUBARSUidQfUUREQkqjs0TGGcp6OCQkFEJOGMjTm6QmGKFQoiItI3OMKYQ2MKIiISmXkEaExBREQmnc2sw0ciIjKx7pF6CiIi4tUKqaBQEBFJOOopiIjIhK5QmLQUIy8zZisRnZJCQUQkwXSFwhQGMuK+GB4oFEREEk5XcNiT8QRQKIiIJJzOUNiT8QRQKIiIJJzuUFg9BRERiegMDntyNjMoFEREEopzLnothSQLBTP7tpm1mtmeSduKzewpMzsU/V4Uq/ZFRPyob2iEkTHnyQqpENuewneBa0/Y9nngaefccuDp6H0REYnqDkbOZvZihVSIYSg4554DOk/YfD3wvejt7wE3xKp9ERE/8nKFVIj/mMI851xT9HYzMO9UTzSzrWZWa2a1bW1t8alORMRj4yukJt2Ywpk45xzgTvP4nc65GudcTVlZWRwrExHxzvi6R3Olp9BiZhUA0e+tcW5fRCSheblCKsQ/FB4CbonevgV4MM7ti4gktK5gmBSD/KwkCwUzuxd4EVhhZvVmdivwFeAaMzsEXB29LyIiUeOL4aWkxH8xPICYrcvqnPvwKR7aHKs2RUT8rsvDJS5AZzSLiCSUyAqp3gwyg0JBRCShdHm4QiooFEREEooOH4mICBBZDK8rOKyegoiIQCg8Snh0TGMKIiICneNnMysURESkO+TtCqmgUBARSRher5AKCgURkYTR7fEKqaBQEBFJGJ0er5AKCgURkYTRFRrGDAqyNaYgIjLndQXDFGSnk+rRYnigUBARSRiRs5m9O3QECgURkYTh9RIXoFAQEUkYXq+QCgoFEZGE0do3SGlupqc1KBRERBLA0Mgo7f1hFhRme1qHJ6FgZp82sz1mttfMPuNFDSIiiaS5ZxCABYVZntYR91Aws7XAbcCFwHpgi5kti3cdIiKJpKF7AGBO9hRWAS8550LOuRHgWeBGD+oQEUkYTd3jPYW5Fwp7gMvMrMTMAsB1wEIP6hARSRiN0Z5CRYG3h4/S4t2gc26/mf0j8CQQBHYCoyc+z8y2AlsBqqur41qjiEi8NfYMUpKTQVZ6qqd1eDLQ7Jy7yzm3yTl3OdAFvHaS59zpnKtxztWUlZXFv0gRkThq6hmgwuNBZvBu9lF59Hs1kfGEe7yoQ0QkUTR2D1BR4O14Anhw+Cjqx2ZWAgwDf+yc6/aoDhGRhNDUPcjF55R6XYY3oeCcu8yLdkVEElHv4DB9QyOeDzKDzmgWEfFcokxHBYWCiIjnGnvGT1xTT0FEZM5rTJCzmUGhICLiuabuQVJTjPI89RREROa8xu4B5udneXoZznEKBRERjzX2DCTEzCNQKIiIeK6pZ5CKBBhPAIWCiIinxsYcTd2DCTHzCBQKIiKe6giGCY+OsSABlrgAhYKIiKcSaToqKBRERDzV1JMY11EYp1AQEfFQY3SJi0r1FEREpLF7gKz0FAoD6V6XAigUREQ81dQzyILCbMy8P3ENFAoiIp5q6B5ImJlHoFAQEfFUUwKdzQwKBRERzwyPjtHaN5Qw01HBu2s0/18z22tme8zsXjNLnJgUEYmT5p5BnEuM6yiMi3somFkl8Cmgxjm3FkgFbo53HSIiXmvqSZwrro3z6vBRGpBtZmlAAGj0qA4REc+8eeLaHA4F51wD8M/AMaAJ6HHOPRnvOkREvFbflTiX4RznxeGjIuB6YAmwAMgxs98/yfO2mlmtmdW2tbXFu0wRkZg70h6kPC+TQEaa16VM8OLw0dVAnXOuzTk3DNwPXHzik5xzdzrnapxzNWVlZXEvUkQk1o50BFlcmuN1GW/hRSgcA95hZgGLnMK3GdjvQR0iIp6qaw+xpGSOh4Jz7iVgG7AD2B2t4c541yEi4qW+wWHa+4dYVBrwupS38ORAlnPuC8AXvGhbRCQRHO0IAainICIiUNceBNCYgoiIRGYeASxWT0FEROo6gszPzyI7I9XrUt5CoSAi4oEj7UEWJ9ggMygUREQ8caQjxJIEG08AhYKISNz1DAzTGQwn3HgCTDEUzCzHzFKit881s/eZWWJcUFRExGeOJOjMI5h6T+E5ICu67PWTwB8A341VUSIiyexIRyQU/Hz4yJxzIeBG4D+cczcBa2JXlohI8qprD2IG1cX+HWg2M3sn8HvAI9FtiTWPSkTEJ460B1lQkE1WeuL9GZ1qKHwG+HPgAefcXjNbCvw8dmWJiCSvuo5QQk5HhSmufeScexZ4FiA64NzunPtULAsTEUlWR9qDbFlX4XUZJzXV2Uf3mFm+meUAe4B9Zva52JYmIpJ8uoJhegaGE3KQGaZ++Gi1c64XuAF4jMhV0/4gZlWJiCSpuo7EXPNo3FRDIT16XsINwEPRK6a52JUlIpKcEvkcBZh6KNwBHAFygOfMbBHQG6uiRESSVV17kJQEnY4KUx9ovh24fdKmo2Z2ZWxKEhFJXnXtQSqLsslIS8xVhqY60FxgZl8zs9ro11eJ9BpERGQajnQEE3Y8AaZ++OjbQB/woehXL/Cds2nQzFaY2c5JX71m9pmzeS0RET9xznGkPTFXRx031Ws0n+Oc+8Ck+39rZjvPpkHn3EFgA4CZpQINwANn81oiIn7S1jdE/9BIQofCVHsKA2Z26fgdM7sEGJiF9jcDbzjnjs7Ca4mIJLS9jZH5Oasq8j2u5NSm2lP4BPDfZlYQvd8F3DIL7d8M3HuyB8xsK7AVoLq6ehaaEhHx1u6GHgDWLEjcUJhST8E5t8s5tx5YB6xzzp0PXDWThs0sA3gfcN8p2rzTOVfjnKspKyubSVMiIglhd0MPS0tzyMtK3MvRTGtOlHOuN3pmM8BnZ9j2u4EdzrmWGb6OiIgv7GnoYW1lwZmf6KGZTJS1Gbb9YU5x6EhEJNm09w/R1DPIeUkcCme9zEV0Yb1rgPtn0L6IiG/siY4nJHpP4bQDzWbWx8n/+BuQfbaNOueCQMnZ7i8i4jfjobCmMnEHmeEMoeCcy4tXISIiyWx3Qw+LSwLkJ/AgM8zs8JGIiEzRnobehD90BAoFEZGY6wyGaegeSPhBZlAoiIjE3PhJawoFERGZNMisUBARmfP2NPSwqCRAQXZiDzKDQkFEJOZ2N/SwdkHi9xJAoSAiElNdwTD1XQO+mHkECgURkZja0+ifQWZQKIiIxNTuieUtEvtM5nEKBRGRGNrb0MvC4mwKAxlelzIlCgURkRg62NLHyvn+6CWAQkFEJGbCI2McaQ9y7rxcr0uZMoWCiEiMHOkIMjLmWF7un7VFFQoiIjHyWksfAMvVUxARkUMt/aQYnFOmUBARmfMOtfZRXRwgKz3V61KmzJNQMLNCM9tmZgfMbL+ZvdOLOkREYulQSz/LfDSeAN71FP4NeNw5txJYD+z3qA4RkZgIj4xR57OZR3CGy3HGgpkVAJcDHwFwzoWBcLzrEBGJpYmZRz4LBS96CkuANuA7Zvaqmf2XmeWc+CQz22pmtWZW29bWFv8qRURm4FBLP4CvpqOCN6GQBmwE/tM5dz4QBD5/4pOcc3c652qcczVlZWXxrlFEZEZea+nDfDbzCLwJhXqg3jn3UvT+NiIhISKSNF5v7ae6OEB2hn9mHoEHoeCcawaOm9mK6KbNwL541yEiEkuvtfSxvNxfvQTwYKA56pPAD8wsAzgMfNSjOkREZt3waGTm0dWr53ldyrR5EgrOuZ1AjRdti4jE2pH28TWP/NdT0BnNIiKz7FBrZObRufP8NfMIFAoiIrPOrzOPQKEgIjLrDrX2s7DIfzOPQKEgIjLrDvl05hEoFEREZtX4zKPlPhxPAIWCiMisOtoRZHjUnzOPQKEgIjKrXq7rAmDFfPUURETmtFB4hNufPsT6qgJWV+R7Xc5ZUSiIiMySbz1XR3PvIH+5ZTUpKeZ1OWdFoSAiMgtaegf55rNvcN1587lgcbHX5Zw1hYKIyCz45ycOMjrm+LNrV3pdyox4tSBeXLze2k97/xC5mWmRr6w08rLSyEzz3wklIpK49jT0sG1HPbddtpRFJW+7ZpivJHUofPeFOr7/62Nv256VnkJBdjr5WekEMlLJSk8lkJFKICONQEYqOZmR7w4YHhljZMzhnIuGSjp5WZGQyUpPJTs9leyMyP65mWnkZCp4ROaaLz+2n8LsdP74ymVelzJjSR0KWy87h+vWVtA3NEL/4Aj9QyP0DgzTOzhMz8AwvQMjDAyPMjA8SkcwzLHOEKHwKMGhEULhUVLMSEs10lNTcM7RPzTCmJta2+PBU5CdTl5W+kRvJS8rjaKcDIoDGRTlZFCSm0FpTialeRkU52QoTER8pr4rxK9e7+BPr11BQXa61+XMWFKHQnVJgOqSwKy9nnOOUHiUvsERguERBodHGRwejQZJJEz6h0boi4ZOz8Aw3aFh+odG6A6FOd4Vom8wcnt49OTpUpyTwbz8LCoKsphfkEVlYTZVRdlUFmZTXRygLC8TM3/OahBJRs8caAXg2jXzPa5kdiR1KMw2MyMneohoJpxz9A2N0NkfpiMYpqN/iPb+MO39Q7T0DtLcM0hz7yA7j3fTGQy/Zd/s9FQWlQRYVBJgxfx8Vs3PY1VFPtXFAd9OgRPxs5/tb2VpaQ5Lfbgi6skoFDxgZuRnRcY0FpeeflAqFB6hsXuA410DHO8McbQjxNGOIIda+3lqX8vE4azczDRWVeSxZkEBaxbks35hIcvKchUUIjHUPzTCr9/o4A/fucjrUmaNJ6FgZkeAPmAUGHHO6SpspxDISGNZeR7Lyt9+yvxAeJTXWvrY39TLvqZe9jb28qPa44TCowDkZaaxobqQ86uLeMeSYjYuKiIrXWMWIrPll4faCI+OsXmV/y67eSpe9hSudM61e9i+72VnpLJ+YSHrFxZObBsdc9S1B9l5vJtXj3Xx6rFuvvHMIW53kJGawobqQi5dVsqVK8pZsyBfPQmRGXh6fyv5WWnULC7yupRZo8NHSSY1xVhWnsuy8lw+uKkKgL7BYV450smvD3fy4hsd/MvPXuNrT71GWV4mV64o412r53Pp8lL1IkSmYWzM8fODrVyxopz01OQ5D9irUHDAk2bmgDucc3ee+AQz2wpsBaiuro5zecklLyudq1bO46qVkS5ue/8Qzx5s45mDrTy2p5kf1daTm5nGVSvLue68+VyxolwBIXIGO+u7ae8Ps3lVudelzCqvQuFS51yDmZUDT5nZAefcc5OfEA2KOwFqamqmeHaATEVpbiYf2FTFBzZVER4Z48XDHTy2u4kn97Xw0K5G8jLTuHbtfK7fUMk7zykhVYeYRN7m6f0tpKYYV5yrUJgx51xD9HurmT0AXAg8d/q9JBYy0lL4rXPL+K1zy/jiDZGAeHBnI4/taea+7fWU52Xy/vMruXFjlW/XhxeJhaf3t1KzqIiCgP9PWJss7qFgZjlAinOuL3r7XcDfxbsOebu01BQuW17GZcvL+OINa3nmQCv372jgrl/Wccdzh1mzIJ8Pbqri+g2VFOdkeF2uiGfqu0IcaO7j/13n78XvTsaLnsI84IHoWblpwD3Oucc9qENOIys9levOq+C68yro6B/i4V2N/HhHA3/78D7+4dH9bF45jw9uquK3VpQl1SCbyFSMn8WcTFNRx8U9FJxzh4H18W5Xzl5JbiYfuWQJH7lkCQeae7mvtp6fvNrA43ubKc3N5IYNC/hgTRUr5/vzSlMi0/Xo7iaWluVwTpKcxTyZOZf4Y7g1NTWutrbW6zJkkuHRMX5xsI1t24/zzIFWhkcd51UWcFNNFdevr0y646wi45p6Brj4K8/w6c3L+czV53pdzmmZ2fbpnhys8xTkrKSnpnDN6nlcs3oencEwD+5s4L7aev76wb188ZH9/Paa+dx8wULeubREJ8hJUvnpriacg/etX+B1KTGhUJAZK87J4KOXLOGjlyyJXGxkez0PvNrAw7saqS4O8DsXLOSDm6qYl5/ldakiM/bgrgbWVRUkzQJ4J1IoyKxaW1nA2soCPv/ulTyxt5l7Xz7GPz1xkK899RqbV5bz4YuquXx5mc59EF96vbWfPQ29/OV7VnldSswoFCQmstJTuX5DJddvqKSuPcgPXznGttp6ntzXQmVhNjfVVHFTzUIqC7O9LlVkyh7a1YgZvDdJDx2BBpoljsIjYzy1r4UfvnKMX74eWQvx0mWlfKhmIdesnqelNSShOee46qvPUlGQxT23vcPrcqZEA82S0DLSUnjPugres66C450h7ttez7ba43zy3lfJy0pjy7oK3n9+FRcsLtLV5STh7G7ooa49yMcvX+p1KTGlUBBPLCwO8NlrzuXTm5fz68Md/HhHPQ/ubOTel49TWZjNlvUVvG/9AlZX5CsgJCE8uLOR9FTj3WsrvC4lphQK4qnUFOOSZaVcsqyUv79+hCf2NvPQrkbuer6OO549zNKyHLasW8B711WwfJ7WXhJvjI45Ht7VyBUrypP+HByFgiSMnMw0btxYxY0bq+gMhnl8TzMP7Wrg688c4vanD7FiXh5b1lWwZf0ClpzhMqYis+nZ11pp7Rvi+g3JO8A8TgPNkvBaewd5dHcTP/1NE7VHuwBYW5nPlnULeM95FSwsDnhcoSS7D33zReq7Qvzic1eSkeaftb7OZqBZoSC+0tg9wKO7m3j4N03sOt4NwHmVBdHF++azqEQ9CJld24928YH/fIG/2rKaWy9d4nU506JQkDnleGeIx/Y08cju5omAWDk/j3etmc9vr5mnQWqZFbf9dy0v13XywuevIifTX0fcFQoyZ9V3hXh8TzNP7m3hlaOdOAeVhdlsXlXOVSvLecfSEp0HIdP2emsfV3/tOT511TI++64VXpczbTpPQeasqqIAH7tsKR+7bCnt/UP8bF8LP9vfyo9qj/PfLx4lOz2VS5eXcvWqcq5cWU55ntZhkjO749nDZKWncMvFi70uJW4UCpJ0SnMzufnCam6+sJrB4VFePNzB0/tbeGZ/K0/tawFgfVUBV6yIBMS6ygKt5Cpv09QzwE92NvC7F1ZTkpvpdTlxo1CQpJaVnsqVK8q5ckU57nrHgeY+nt7fwtMHWrn9mUP829OHKMnJ4LLlpdFLkZZSrtVcBbjr+TrGHHzssuQ+g/lEnoWCmaUCtUCDc26LV3XI3GFmrKrIZ1VFPv/nquV0BsM8f6iNnx9o5Zevt/OTnY0ArJiXx8XLSrjknFIuWlpMXlZyn6wkb/eTVxu461d1fGBj1Zyb8uzZQLOZfRaoAfLPFAoaaJZYGxtz7G/u5flD7fzyUDuvHOlkaGSM1BTjvMoCLllWwiXLStlYXaQB6yT3yG+a+OS9O7hoSQnf+egFvv739s3sIzOrAr4HfAn4rEJBEs3g8CivHuvmhTfaeeGNDnYe72Z0zJGZlsLG6iIuWlrMO5aWsGFhoa//aMhbPbm3mT/6wQ7Ory7ke//rQgIZ/j7C7qdQ2AZ8GcgD/kShIImub3CYl+s6+dXrHbxU18G+pl6cg4zUFNZVFbBpcREXLCpm06IiinIyvC5XzsLPD7Sy9e5a1iwo4O5bL0yKw4a+CAUz2wJc55z7IzO7glOEgpltBbYCVFdXbzp69Ghc6xQ5nZ6BYWqPdPJSXSe1RzrZ3dDD8Gjkd2lpWQ6bqovYtKiIjYuKWFaWq9lNCe7nB1r5+N3bWTE/j+9/7CIKsv0fCOCfUPgy8AfACJAF5AP3O+d+/1T7qKcgiW5weJRdx7vZfqyLHUe72H60i67QMAB5mWlsqC7k/OoiNiwsYH1V4Zya4pjonjnQwifu3hEJhFsvSqpVUH0RCm9p/DQ9hckUCuI3zjkOtwd59Vg3rx7rYsexbg429zIW/XVbWJzNuspC1lTmc15lAWsXFOiwkweSORBAZzSLJAwz45yyXM4py+WDm6oACIVH2F3fw676bnYe7+Y3Dd08srtpYp/KwmxWVeSzZkE+qxdEvlcWZmv9phhwzvH9l47xdw/vZeX8/KQMhLOltY9EPNQTGmZPYw+7G3rY19jL3sYeDrcHGf+1LMhOZ3X03IqVFXmsnJ/HufPyNONpBgaHR/mLB/bw4x31XLmijH+9+fykGUM4kXoKIj5TEEifuPLcuFB4hAPNfext7GVfYy/7Gnu45+WjDA6PAZBisKgkhxXz8lgRDYnl83JZXJLjq7X+vXC8M8Qnvr+dvY29fHrzcj69ebkmAZxAoSCSYAIZaWysLmJjddHEttExx9GOIAeb+9jf3MfB5l4OtvTxxL7miV5FaoqxqCTAOWW5LCvPjR6+ymFpWW7S/k94Ol54vZ0/vmcHI2OOu26pYfOqeV6XlJAUCiI+kJpiLC3LZWlZLu8+780Lxw+ER3mjrZ/XWyd9tfXz8wOtjIy9eWi4NDeDpaW5LCnNYVFpgCUlOSwuzWFRScD3J2idiXOO7/zqCF96dD9LSnP41h/W6HKup5HcnwaRJJedkcraygLWVha8Zfvw6BjHOkO80drP4fYgdW1BDrf38/SBVtr7h97y3Hn5mSwqyWFRcYBFJQGqo7eriwMUBtJ9PdAdCo/w1w/uZdv2eq5ZPY9/+Z0N5PrsQjnxpndHJAmlp6ZMzH46Ud/gMEc7QtS1BznaEeRIR4gj7UF+8VobbX1vDYy8zDSqigNUF2dTWRigqiibyqJsqoqyqSoKJOxhqbExx092NvCPjx+gpXdI4wfToFAQmWPystJP2ruAyP+sj3WGONoR4nhn9KtrgDfagjz3WjsDw6MnvFYalYXZVBZms6Awm4rCLBYUZFNRkEVFQTbzCjLJTIvfTCnnHC/XdfIPjx1g1/Fu1lUV8O+/u5GaxcVxq8HvFAoiMiGQkcbK+fmsnJ//tsecc3SFhqnvCtHQNUB91wD1XSHquwZo7Blk+7EuuqNncU9WkpNBRWEW8/OzmV+Qyfz8LMrzsijPz6QsL5OSnEyKctJnFB4tvYPcv6OBbduP80ZbkPK8TL5603ref36legfTpFAQkSkxM4pzMijOyWBdVeFJnxMKj9DYPUhzzyCNPQM0dQ/S3DtIc08kQF450knPwNuDAyA3M41ARipZ6alkpaeQnZ5KdkYqORlpZGdEAmNweJSB4VEGh8cIhUcj98OjtPYNMubggsVFbL18Ke9dvyDpB9BjRe+aiMyaQEYay8ojU2JPZXB4lLa+IVp6B2nvH6IjGKazP0xHMMxAeJTBkegf++ExBsIjNPcOMhAeBYOstEhQZKWnUBTIIDsjlUB6KhWFWdywoZLFmlU0YwoFEYmrrPRUFhYH5twVzfxCpz+KiMgEhYKIiExQKIiIyASFgoiITFAoiIjIBIWCiIhMUCiIiMgEhYKIiEzwxeU4zawNOAoUAD3RzWe6Pf69FGg/i2Ynv+ZUHz/TtkSs+WTbT3f/xFonbzubuuNZ8+Tb+nxM/XF9PqYvUT4fi5xzZdOq3Dnnmy/gzqnenvS9dqZtTfXxM21LxJpPtv1090+sdaZ1x7Nmr99rfT70+Ujkz8f4l98OHz08jduTt820rak+fqZtiVjzybaf7v7Jap1J3fGsefJtfT6m/rg+H9Pnx88H4JPDRzNhZrXOuRqv65gOP9YM/qxbNcePH+ueizX7radwNu70uoCz4MeawZ91q+b48WPdc67mpO8piIjI1M2FnoKIiEyRQkFERCYoFEREZMKcDgUzu8zMvmlm/2VmL3hdz1SYWYqZfcnMvm5mt3hdz1SZ2RVm9nz0/b7C63qmysxyzKzWzLZ4XctUmNmq6Hu8zcz+t9f1TIWZ3WBm3zKz/zGzd3ldz1SZ2VIzu8vMtnldy+lEP8Pfi77Hv3em5/s2FMzs22bWamZ7Tth+rZkdNLPXzezzp3sN59zzzrlPAD8FvhfLeqO1zbhm4HqgChgG6mNV62SzVLcD+oEs4lD3LNUM8GfAj2JT5VvN0md6f/Qz/SHgkljWG61tNmr+iXPuNuATwO/Est5J9c1G3Yedc7fGttKTm2b9NwLbou/x+8744jM5883LL+ByYCOwZ9K2VOANYCmQAewCVgPnEfnDP/mrfNJ+PwLy/FAz8Hng49F9t/nlvQZSovvNA37gk5qvAW4GPgJs8UPN0X3eBzwG/K5fao7u91Vgo18+05P2i8vv4Qzq/3NgQ/Q595zptdPwKefcc2a2+ITNFwKvO+cOA5jZD4HrnXNfBk7a/TezaqDHOdcXw3KB2anZzOqBcPTuaOyqfdNsvddRXUBmLOqcbJbe6yuAHCK/WANm9qhzbiyRa46+zkPAQ2b2CHBPrOqNtjUb77MBXwEec87tiGW942b5Mx1306mfSM+8CtjJFI4O+TYUTqESOHVkL2wAAAQsSURBVD7pfj1w0Rn2uRX4TswqOrPp1nw/8HUzuwx4LpaFncG06jazG4HfBgqBb8S2tFOaVs3Oub8AMLOPAO2xDITTmO77fAWRwwWZwKMxrezUpvuZ/iRwNVBgZsucc9+MZXGnMd33ugT4EnC+mf15NDy8dKr6bwe+YWbvYQpLYSRbKEybc+4LXtcwHc65EJEg8xXn3P1EAs13nHPf9bqGqXLO/QL4hcdlTItz7nYif7h8xTnXQWQcJKE554LAR6f6fN8ONJ9CA7Bw0v2q6LZE5seawZ91q+b48GPN4N+6x81K/ckWCq8Ay81siZllEBkkfMjjms7EjzWDP+tWzfHhx5rBv3WPm5364z1qPouj7/cCTbw5NfPW6PbrgNeIjML/hdd1+r1mv9atmlVzMtYdj/q1IJ6IiExItsNHIiIyAwoFERGZoFAQEZEJCgUREZmgUBARkQkKBRERmaBQEN8ys/44tzcr19ywyLUlesxsp5kdMLN/nsI+N5jZ6tloX+R0FAoiUWZ22rXAnHMXz2JzzzvnNgDnA1vM7EzXPriByGqtIjGlUJCkYmbnmNnjZrbdIld6Wxnd/l4ze8nMXjWzn5nZvOj2vzGzu83sV8Dd0fvfNrNfmNlhM/vUpNfuj36/Ivr4tuj/9H8QXf4ZM7suum27md1uZj89Xb3OuQEiSxpXRve/zcxeMbNdZvZjMwuY2cVErpHwT9HexTmn+jlFZkqhIMnmTuCTzrlNwJ8A/xHd/kvgHc6584EfAn86aZ/VwNXOuQ9H768kssz3hcAXzCz9JO2cD3wmuu9S4BIzywLuAN4dbb/sTMWaWRGwnDeXQb/fOXeBc249sJ/I8gUvEFnD5nPOuQ3OuTdO83OKzMicXzpbkoeZ5QIXA/dF/+MOb17Qpwr4HzOrIHJVqrpJuz4U/R/7uEecc0PAkJm1Erla3ImXEH3ZOVcfbXcnsJjI5UYPO+fGX/teYOspyr3MzHYRCYR/dc41R7evNbMvErnuRC7wxDR/TpEZUShIMkkBuqPH6k/0deBrzrmHohei+ZtJjwVPeO7QpNujnPz3ZCrPOZ3nnXNbzGwJ8Gsz+5FzbifwXeAG59yu6MV9rjjJvqf7OUVmRIePJGk453qBOjO7CSKXeTSz9dGHC3hzbflbYlTCQWDppMsknvEi9NFexVeAP4tuygOaooesfm/SU/uij53p5xSZEYWC+FnAzOonfX2WyB/SW6OHZvYSuUYtRHoG95nZdqA9FsVED0H9EfB4tJ0+oGcKu34TuDwaJn8FvAT8Cjgw6Tk/BD4XHSg/h1P/nCIzoqWzRWaRmeU65/qjs5H+HTjknPsXr+sSmSr1FERm123Rgee9RA5Z3eFxPSLTop6CiIhMUE9BREQmKBRERGSCQkFERCYoFEREZIJCQUREJigURERkwv8HSW2lLVmnQMsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light",
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = Model6(tok.vocab_sz, 512, 8, p_emb=0.1, p_ff=0.1, tie_weights=True)\n",
    "learn = Learner(dls, model, loss_func=CrossEntropyLossFlat(), metrics=[accuracy, perplexity]).to_native_fp16()\n",
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "VHM-Re1EJhIi",
    "outputId": "8fbea906-d9ac-47af-cc8e-a3a470d7909a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<fastai.learner.Learner at 0x7fa5a807e0f0>"
      ]
     },
     "execution_count": 72,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn = learn.load(path/'char_bookcorpus_10m')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 87
    },
    "id": "6Dluttl_PRb4",
    "outputId": "b6f169fb-5d48-4a03-891d-a808272e7cd9"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.065701</td>\n",
       "      <td>1.064358</td>\n",
       "      <td>0.667121</td>\n",
       "      <td>2.898976</td>\n",
       "      <td>4:23:52</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(1, 1e-4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "_4dknAAELfJs",
    "outputId": "ec915db9-e669-46ff-dad2-cddb044e9d8a"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Path('/content/drive/MyDrive/char_model/char_bookcorpus_10m.pth')"
      ]
     },
     "execution_count": 74,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.save(path/'char_bookcorpus_10m')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "asOLDrwgPdUT"
   },
   "source": [
    "### Finetune on Carrolls' books\n",
    "\n",
    "Finally we can finetune the pretrained bookcorpus model on Carroll's books. This will determine the style of generated text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Z1KYZykgLoKg"
   },
   "outputs": [],
   "source": [
    "sl = 512\n",
    "seqs = L((tensor(all_nums[i:i+sl]), tensor(all_nums[i+1:i+sl+1]))\n",
    "         for i in range(0,len(all_nums)-sl-1,sl))\n",
    "cut = int(len(seqs) * 0.8)\n",
    "dls = DataLoaders.from_dsets(seqs[:cut], seqs[cut:], device='cuda',\n",
    "                             bs=16, drop_last=True, shuffle=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "FPsgcxwfFaDU"
   },
   "outputs": [],
   "source": [
    "model = Model6(tok.vocab_sz, 512, 8, p_emb=0.1, p_ff=0.1, tie_weights=True)\n",
    "learn = Learner(dls, model, loss_func=CrossEntropyLossFlat(), metrics=[accuracy, perplexity]).to_native_fp16()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 303
    },
    "id": "DBKwx9A5LoC7",
    "outputId": "7f1901ab-4932-464f-b3a4-028e67447a6a"
   },
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "SuggestedLRs(lr_min=0.03019951581954956, lr_steep=0.25118863582611084)"
      ]
     },
     "execution_count": 74,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAELCAYAAADURYGZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxddZ3/8dcne9qkSdukW9KN7qULbVN2kMXKKiC4oIDIsLiN4k+HwVFn0BlHcZxRBxlBBEQUQZayCIKAlK1Aabov6V5om6bN0jZL2+yf3x/3FkNI2qTk5Nyb+34+HvfBved+zz3vHJL76Tnf7zlfc3dERCRxJYUdQEREwqVCICKS4FQIREQSnAqBiEiCUyEQEUlwKgQiIgkusEJgZhlm9raZrTCzNWb2gw7afNPM1prZSjP7m5mNDiqPiIh0zIK6jsDMDOjv7nVmlgq8Dtzo7m+1aXMmsMjdD5jZl4Ez3P0zh/vcvLw8HzNmTCCZRUT6qiVLllS6e35H76UEtVGPVJi66MvU6MPbtVnQ5uVbwJVH+twxY8ZQXFzcUzFFRBKCmb3b2XuB9hGYWbKZLQfKgRfcfdFhml8LPBtkHhER+aBAC4G7t7j7cUAhcLyZTeuonZldCRQBP+3k/RvMrNjMiisqKoILLCKSgHpl1JC77wMWAOe2f8/MPgp8F7jI3Rs6Wf8udy9y96L8/A5PcYmIyFEKctRQvpnlRp9nAvOAde3azAJ+TaQIlAeVRUREOhdYZzEwHPidmSUTKTgPu/vTZvbvQLG7P0XkVFAW8EhkkBHb3P2iADOJiEg7QY4aWgnM6mD5v7V5/tGgti8iIl2jK4tFROLAc6t3sX3PgUA+W4VARCTG7W9o5usPLuP+N98J5PNVCEREYtzrmyppbGnlzMlDAvl8FQIRkRi3YF052ekpzB0zKJDPVyEQEYlh7s6C9eWcNjGP1ORgvrJVCEREYtianTXsrmngzEnBnBYCFQIRkZi2YF3kWtszVAhERBLTS+vLmVmYQ352emDbUCEQEYlRVXUNLN++L7DRQoeoEIiIxKhXNlTgDmepEIiIJKaX1pWTl5XOtBE5gW5HhUBEJAY1t7Ty6oYKzpyUT1KSBbotFQIRkRi05N291NQ3B35aCFQIRERi0kvry0lNNk6dkBf4tlQIRERi0MJNlcwZPZDsjNTAt6VCICISY+qbWli/q5ZZowb2yvZUCEREYsz6XbU0tTgzCoIdLXRIkHMWZ5jZ22a2wszWmNkPOmiTbmZ/MrNNZrbIzMYElUdEJF6sLK0GYHphnBcCoAE4y91nAscB55rZie3aXAvsdffxwM+BnwSYR0QkLqzeUc3AfqkU5Gb2yvYCKwQeURd9mRp9eLtmFwO/iz5/FDjborPYi4gkqpWl1UwvzKW3vg4D7SMws2QzWw6UAy+4+6J2TQqA7QDu3gxUA4ODzCQiEsvqm1rYsLu21/oHIOBC4O4t7n4cUAgcb2bTjuZzzOwGMys2s+KKioqeDSkiEkNKympoaXWm9ZVCcIi77wMWAOe2e6sUGAlgZilADlDVwfp3uXuRuxfl5+cHHVdEJDSroh3FM3qpoxiCHTWUb2a50eeZwDxgXbtmTwFXR59/EnjJ3dv3I4iIJIyVO6rJy0pjeE5Gr20zJcDPHg78zsySiRSch939aTP7d6DY3Z8C7gF+b2abgD3A5QHmERGJeatLq5lWkNNrHcUQYCFw95XArA6W/1ub5/XAp4LKICISTw42RjqKPzZ1aK9uV1cWi4jEiLVl1bQ6TC/M7dXtqhCIiMSIlTt6v6MYVAhERGLGqtJq8rPTGTqg9zqKQYVARCRmrNpR3asXkh2iQiAiEgP2NzSzqaKu124015YKgYhIDFhbVoM7TNcRgYhIYjrUUaxCICKSoNaUVjMkO50hvdxRDCoEIiIxYW1ZDVNHDAhl2yoEIiIha2xuZXNFHVOGqxCIiCSkTeV1NLW4CoGISKIqKasBYOrw7FC2r0IgIhKykrIa0lOSGDO4fyjbVyEQEQlZya4aJg3LJiU5nK9kFQIRkRC5OyVltUwZFk7/AKgQiIiEqry2gT37G5kSUv8AqBCIiIRqbbSjOKwRQxDsnMUjzWyBma01szVmdmMHbXLM7M9mtiLa5pqg8oiIxKJDI4Ymh1gIgpyzuBn4lrsvNbNsYImZveDua9u0+Sqw1t0/bmb5wHoze8DdGwPMJSISM0rKainIzSQnMzW0DIEdEbh7mbsvjT6vBUqAgvbNgGyLzNKcRWQC++agMomIxJqSsppQTwtBL/URmNkYIhPZL2r31u3AFGAnsAq40d1bO1j/BjMrNrPiioqKgNOKiPSO+qYWtlTUhdpRDL1QCMwsC3gM+Ia717R7+xxgOTACOA643cw+UBrd/S53L3L3ovz8/KAji4j0ig27a2n1cDuKIeBCYGapRIrAA+4+v4Mm1wDzPWITsBWYHGQmEZFYURIDI4Yg2FFDBtwDlLj7zzpptg04O9p+KDAJ2BJUJhGRWFJSVku/tGRGD+oXao4gRw2dAlwFrDKz5dFl3wFGAbj7ncB/APeZ2SrAgJvdvTLATCIiMaOkLHJriaQkCzVHYIXA3V8n8uV+uDY7gY8FlUFEJFZFbi1Rw4UzR4QdRVcWi4iEYWd1PTX1zaH3D4AKgYhIKEp2hjsHQVsqBCIiISh+dy+pyaYjAhGRRPXmlipmFubSLy3IMTtdo0IgItLLauqbWLVjHyePGxx2FECFQESk1y3euodWhxNVCEREEtObm6tIS0li9qiBYUcBVAhERHrdm1uqmD0ql4zU5LCjACoEIiK9at+BRtaW1XDyuLywo7xHhUBEpBe9tWUP7nBSjPQPgAqBiEivemtLFZmpycwszA07yntUCEREetGbm6soGjOQtJTY+fqNnSQiIn1cZV0D63fXcuIxsXNaCFQIRER6zVtbqgBi5kKyQ1QIRER6yZubq8hKT2F6QU7YUd5HhUBEpJe8uaWKuWMGkpIcW1+9sZVGRKSP2l1Tz5aK/TE1bPSQIOcsHmlmC8xsrZmtMbMbO2l3hpktj7Z5Jag8IiJhemNzZBbeWLqQ7JAg73/aDHzL3ZeaWTawxMxecPe1hxqYWS7wK+Bcd99mZkMCzCMiEpqFm6rI7ZfK1BiYf6C9wI4I3L3M3ZdGn9cCJUBBu2afA+a7+7Zou/Kg8oiIhMXdeWNTJScdMzj0ieo70it9BGY2BpgFLGr31kRgoJm9bGZLzOzznax/g5kVm1lxRUVFsGFFRHrYO1UH2Fldz8njY++0EPRCITCzLOAx4BvuXtPu7RRgDnABcA7wr2Y2sf1nuPtd7l7k7kX5+flBRxYR6VELN0X6B06JwY5iCLaPADNLJVIEHnD3+R002QFUuft+YL+ZvQrMBDYEmUtEpDe9sbmS4TkZjM3rH3aUDgU5asiAe4ASd/9ZJ82eBE41sxQz6wecQKQvQUSkT2htdd7cXMXJ4/KIfC3GniCPCE4BrgJWmdny6LLvAKMA3P1Ody8xs+eAlUArcLe7rw4wk4hIr1pbVsPeA02cMj42TwtBgIXA3V8Hjlj+3P2nwE+DyiEiEqZD1w+cEqMdxaAri0VEArVwUxXj8vszdEBG2FE6pUIgIhKQxuZW3t66J6aPBkCFQEQkMMu37+NgU0tM3laiLRUCEZGALNxUSZLBSTE2EU17KgQiIgF5Y3Ml0wpyyOmXGnaUw1IhEBEJQHltPcu27ePUGO8fABUCEZFA/P7Nd2lx5zNzR4Yd5YhUCEREetjBxhb+8Na7zJsylNGDY/O2Em2pEIiI9LD5y3aw90AT1512TNhRukSFQESkB7W2Ove8vpUZhTnMHTMw7DhdokIgItKDXt5QzpaK/Vx76tiYvclceyoEIiI96O7XtjI8J4Pzpw8PO0qXqRCIiPSQNTureWNzFV84eQypyfHz9Ro/SUVEYtw9r2+lX1oylx8/Kuwo3dKlQmBm/c0sKfp8opldFJ19TEREgMq6Bp5eUcYn5xSSkxlfX49dPSJ4FcgwswLgeSITztwXVCgRkXjzp8XbaWxp5fMnjQk7Srd1tRCYux8ALgV+5e6fAo4NLpaISPxobmnlD2+9y6nj8xg/JCvsON3W5UJgZicBVwDPRJclH2GFkWa2wMzWmtkaM7vxMG3nmlmzmX2yi3lERGLGiyXllFXXc9VJo8OOclS6OlXlN4B/AR539zVmdgyw4AjrNAPfcvelZpYNLDGzF9x9bdtGZpYM/ITIKScRkbjz+7feoSA3k7MnDwk7ylHpUiFw91eAVwCincaV7v71I6xTBpRFn9eaWQlQAKxt1/RrwGPA3O5FFxEJ36byWhZuquKmcyaREkdDRtvq6qihP5rZADPrD6wG1prZTV3diJmNAWYBi9otLwA+AdxxhPVvMLNiMyuuqKjo6mZFRAL3+zffJS05icvj4C6jnelq+Zrq7jXAJcCzwFgiI4eOyMyyiPyL/xvRz2jrF8DN7t56uM9w97vcvcjdi/Lz87sYWUQkWHUNzTy2tJQLZw5ncFZ62HGOWlf7CFKj1w1cAtzu7k1m5kdaKbrOY8AD7j6/gyZFwEPR+3HkAeebWbO7P9HFXCIioXl48XbqGprjcshoW10tBL8G3gFWAK+a2Wig/b/u38ci3+73ACXu/rOO2rj72Dbt7wOeVhEQkXiwYvs+fvLcOk4eN5jjRuaGHedD6Wpn8W3AbW0WvWtmZx5htVOInD5aZWbLo8u+A4yKfuad3cwqIhITyqoPcv39xeRnp/PLz84KO86H1qVCYGY5wC3A6dFFrwD/DlR3to67vw50+R6s7v6FrrYVEQnLgcZmrr+/mAONLfz+2hPium/gkK52Ft8L1AKfjj5qgN8GFUpEJBa1tjrfengFa3fW8MvPzmLSsOywI/WIrvYRjHP3y9q8/kGb0z0iIgnhFy9u4NnVu/jeBVM4M04vHutIV48IDprZqYdemNkpwMFgIomIxJ4/r9jJbS9t4tNFhVx76tgjrxBHunpE8CXg/mhfAcBe4OpgIomIxJaVO/bxT4+s4Pgxg/jhJdPjZgrKrurqqKEVwEwzGxB9XWNm3wBWBhlORCRsu2vquf7+YvKy0rnjytmkpcTnbSQOp1s/kbvXtLk6+JsB5BERiRkHG1u44f5iauubufvqoj4xQqgjXT011JG+dWwkItJGU0srX/3jUlaWVvPrK+cwZfiAsCMF5sMc4xzxFhMiIvGotdW56ZEVvLSunB9eMo2PHTss7EiBOuwRgZnV0vEXvgGZgSQSEQmRu/PvT6/lieU7uemcSVxxQnxONtMdhy0E7t43rpYQEemi//3bRu574x2uO3UsXzljXNhxesWH6SMQEekz3J2fv7CB217axCfnFPLdC6b0uWGinVEhEJGE5+78x9Ml3LtwK58pGsmPLu171wocjgqBiCS0llbnu4+v4qHF27nmlDH86wVTSUpKnCIAKgQiksBaW51/emQFjy8r5Wtnjeeb8yYm1JHAISoEIpKwfvLcOh5fVso3503k62dPCDtOaPretdIiIl1w38Kt/PrVLVx14mi+dtb4sOOESoVARBLOc6t38YOn1zJv6lC+f9GxCXk6qK3ACoGZjTSzBWa21szWmNmNHbS5wsxWmtkqM3vDzGYGlUdEBGDJu3u48aFlzCzM5bbLZ5GcYB3DHQmyj6AZ+Ja7LzWzbGCJmb3g7mvbtNkKfMTd95rZecBdwAkBZhKRBLZ8+z6+cO9iRuRmcs/VRWSmJYcdKSYEVgjcvQwoiz6vNbMSoABY26bNG21WeQsoDCqPiCS2lTv2cdU9ixjYP40Hrusbcw33lF7pIzCzMcAsYNFhml0LPNvJ+jeYWbGZFVdUVPR8QBHp01aXVnPl3YvIyUzlwRtOZESubpXWVuCFwMyygMeAb7SZy6B9mzOJFIKbO3rf3e9y9yJ3L8rPzw8urIj0OSVlNVxx9yKyM1J58PoTKVAR+IBAryMws1QiReABd5/fSZsZwN3Aee5eFWQeEUksZdUHuea3i8lMTeahG05k5KB+YUeKSYEVAouMx7oHKHH3n3XSZhQwH7jK3TcElUVEEk9tfRPX/HYxdQ3NPPKlk1QEDiPII4JTgKuAVWa2PLrsO8AoAHe/E/g3YDDwq+g43mZ3Lwowk4gkgKaWVr7ywFI2ltfx2y/M7dOzi/WEIEcNvc4RprN09+uA64LKICKJx9353uOreW1jJf912QxOn6h+xSPRlcUi0qf879828qfi7XztrPF8eu7IsOPEBRUCEekzHlj0Lr94cSOXzS7km/Mmhh0nbqgQiEif8NzqXfzrE6s5c1I+t16WWBPLfFgqBCIS9xZtqeLrDy1j5shc/u+K2aQm66utO7S3RCSubdxdy3X3FzNyYCb3Xj2XfmmaZqW7VAhEJG5V1DZwzX2LSU9J5nf/cDwD+6eFHSkuqRCISFyqb2rh+vuLqaxr4J6riygcqAvGjpaOoUQk7rS2Ot96eAUrduzjjivmMHNkbtiR4pqOCEQk7vz0+fU8s6qM75w3hXOnDQs7TtxTIRCRuPJw8XbueHkznzthFNedNjbsOH2CCoGIxI23tlTx3cdXcdqEPH6guYZ7jAqBiMSFrZX7+dIfljB6cH9u/5yuFehJ2pMiEvP2HWjkH+5bTJIZ9149l5zM1LAj9SkqBCIS01pana89uIzSvQe566o5jBqsYaI9TcNHRSSm/fKljby2sZIfXzqdojGDwo7TJ+mIQERi1usbK/nfv23k0lkFXK5bSgcmsEJgZiPNbIGZrTWzNWZ2YwdtzMxuM7NNZrbSzGYHlae9qroG3t66p7c2JyLdtKu6nhsfWsb4/Cx++IlpGiEUoCCPCJqBb7n7VOBE4KtmNrVdm/OACdHHDcAdAeZ5n28+vILP/uYtymvqe2uTItJFzS2tfO3BpRxsauGOK2frRnIBC6wQuHuZuy+NPq8FSoCCds0uBu73iLeAXDMbHlSmQ17fWMkrGypoaXWeWF4a9OZEpJvuXbiVxe/s5ceXTmf8kOyw4/R5vdJHYGZjgFnAonZvFQDb27zewQeLRY9qbXV+9JcSCgdmMqMwh0eKd+DuQW5SRLqhobmFu1/byinjB3PxcYF+HUhU4IXAzLKAx4BvuHvNUX7GDWZWbGbFFRUVHyrP48tKWVtWw03nTOLyuaPYWF7Hyh3VH+ozRaTnPLl8J+W1Ddxw+riwoySMQAuBmaUSKQIPuPv8DpqUAm2HAhRGl72Pu9/l7kXuXpSfn3/UeeqbWvif59czozCHj88YwYUzh5OeksSjS3Yc9WeKSM9pbXV+8+oWJg/L5vQJeWHHSRhBjhoy4B6gxN1/1kmzp4DPR0cPnQhUu3tZUJnuXbiVndX1/Mt5U0hKMgZkpHLutGE8tWIn9U0tQW1WRLro5Q3lbCyv44sfOUajhHpRkEcEpwBXAWeZ2fLo43wz+5KZfSna5i/AFmAT8BvgK0GFqapr4I4Fmzl78hBOGjf4veWfnFNI9cEmXizZHdSmRaSL7nxlCyNyMrhwxoiwoySUwMZkufvrwGFLukd6ab8aVIa2XttYSX1zC98+b/L7lp88Lo/hORk8umSHfvlEQrR8+z7e3rqH710wRTeU62UJs7cvmVXA6zefxYSh7x+KlpxkXDq7gFc3VLC7k2sK3J2X1u1m+54DvRFVJCHd9epmsjNSuPz4UWFHSTgJUwgAhg7I6HD5ZbMLafXIiKKO/M/zG/iH+4o57b8W8Mk73uAPb73L3v2NQUYVSSjbqg7w3OpdXHniaLLSdfFYb0uoQtCZY/KzOH7MIG7720aeXrnzfe/9+pXN3L5gE5+aU8g/nzuJ6oNNfO+J1Rz/oxf50u+X8LeS3TS1tIaUXKRvmL9sBw58/qTRYUdJSCq9Ubd9dhZfeWAJ//jHZSzbto9vnzeZR5fs4MfPruPCGcO59bIZJCcZX/7IONaW1TB/aSlPLi/luTW7yMtK57I5BVx76liGZHd81CEiHXN3nly+kxPHDmZ4TmbYcRKSxdtVtUVFRV5cXBzIZzc2t/Kjv5Rw3xvvMHlYNut31/KRifncdVURaSkfPHhqamnl5fUVPFK8nRdLdpOWksSVJ4zmix8ZR352eiAZRfqalTv2cdHtC7n10unqHwiQmS1x96KO3tMRQRtpKUl8/6JjmTUql28/toq5owdxxxVzOiwCAKnJScybOpR5U4eytXI/t7+0iXsXbuUPi97l8rmj+MzckUwZPqDDdesamlmxfR9L3t3LxvI6RuRkcEx+f8blZzFhaLZmYJKE8cSynaQlJ3He9MBvMyad0BFBJ/bubyQrI6Xbw9i2Vu7nly9t5OkVZTS2tDKzMIdPFY1kUP80Nu6uY1NFHRt317Jhdy2tDmYwIieTitoGGqN9DWkpSVx36li+fMY4sjNUEKTvaml1Tvzx35g9KpdfX9XhP1alhxzuiECFICB79jfyxLJSHi7ezrpdtUDkS78gN5PxQ7KYWZjLnNEDOW5ULgMyUmlpdXbsPcCWiv08ubyUJ5bvJC8rjf83byLnTxvOxvI61u2qYf2uWuoamklOMpLNSDKjpr6Jqv2N7NnfSF19M6MG92PS0GwmDs1iTF5/+qenkJmaTL+0ZPKz03VLX4kZr2+s5Mp7FvGrK2Zzvo4IAqVCECJ3p6SsllZ3xuVnkZmW3KX1Vmzfx38+U8Lb77x/8pyczFRy+0UKx6FHdkYKg7PSGdw/jcy0ZN6tOsCGXbXUNjR/4HNTkozZowdy+oQ8TpuQz6hB/Wh1pzX6azC4fxpJSbq0X3rHTY+s4LnVu1j8vY+Skdq1vw05OioEccrdebGknM0VdUwals2UYQMYOiC9S/dgcXfKquvZvucAB5paqG9s4UBjC5sq6nhtYwWrSzu+EeyAjBSOGzWQ40bmMnV4NnsPNFG69yA79h7gQGMLx43K5YSxg5hekNtp34lIV9Q3tTD3hy9yzrRh/PenZoYdp89TZ3GcMrNIZzRDj2rdEbmZjMj94HC8m8+dTFVdA29srqKqrgEzI8ki52vX765j2ba93P7SxveOEpKTjGEDMkhLSeL5tZF7MmWkJjG9IIfxQ7KZMCSL8UOyGNgvjVZ3WtxxdzJSk8lOTyUrI4V+ack0tzoNTS00NLdyoLGFyrqGyKO2geQk47iRA5k8PFu3F0gQC9aVU9vQzCWacyB0KgQJanBWOh+f2fm9lfY3NLO5oo5B/dMYNiCDlOiXc2VdA8Xv7OGtLXtYXVrNs6vLePBAU4/lSk+JFJjJw7MZObAfhQP7MXJQJoP6pzEgM5WstBSduuojnlheSn52+vtuAinhUCGQDvVPT2FGYe4HludlpXPutOGcOy3SsefuVO1vZFN5HXX1zSQlET3CMA42trC/oZm6hmb2NzaTmpREemoS6SlJZKQmk5eVHn2kUd/cyvJt+1i2bS9Lt+3lzyvKqD74wQJjBrmZqZw5aQifnFPIiccMVmGIQ+W19SxYV8GVJ44mWf//QqdCIB+Kmb33hf5hFeRmcsGMv48cqak/1D9xkL0HGqk52ERNfTM79x3kr2t2MX9ZKQW5mVwyawQfnTKUmYW5Kgpx4vaXNtHqrltKxAgVAolZAzJSGTA8tcOL8n54yTSeX7ubx5bs4I6XN/N/CzYzuH8aH5mYz9lThnLm5HwNk41R26oO8MdF2/jM3JGMyesfdhxBhUDiVEZqMhfNHMFFM0ewd38jr26sYMG6chasL2f+slIyUpM4a/IQzp8+nDMnDaG/7mgZM37+4gZSko2vnz0h7CgSpb8OiXsD+6dx8XEFXHxcAS2tzuJ39vCXVWU8u3oXf1m1i9RkY/aogZw2IY9TJ+QzvSBH56VDsm5XDU8sL+WLp4/r9Lbw0vsCu47AzO4FLgTK3X1aB+/nAH8ARhEpSP/t7r890ucm0nUE8uEcKgovr6/gtY0VrNkZuXYiPzudc48dxnnThnH82EHvjYiS4F33u8Us2rqH1//5LHL66fYpvSmUC8rM7HSgDri/k0LwHSDH3W82s3xgPTDM3Q8744sKgRytyroGXt9YyfNrd/HSunLqm1oZ1D+Nc44dygXTR3DiMSoKQVry7h4uu+NNbjpnEl89c3zYcRJOKBeUufurZjbmcE2AbItcJpsF7AE+eE8EkR6Sl5XOJbMKuGRWAQcam3llfQV/Wb2LJ5fv5MG3tzOofxrnThvGpbMKmDN6YJeu4Jau+/kLG8nLSueaU8aEHUXaCbOP4HbgKWAnkA18xt011Zf0in5pKZw3fTjnTR9OfVMLL68v5+mVZTy+tJQ/LtrGmMH9uHR2IZfOLqBwYL+w48a9mvom3thcyVfPHK/RXDEozP8j5wDLgbOAccALZvaau3/gJjhmdgNwA8CoUZq4QnpWRmryexfJ1TU08+yqMh5buoOfvbCBn72wgePHDuITswo4f9pwndc+Sou37qHV4eRxeWFHkQ6EWQiuAW71SCfFJjPbCkwG3m7f0N3vAu6CSB9Br6aUhJKVnsKnikbyqaKRbN9zgCeXlzJ/WSn/Mn8Vtzy5hjMm5XPJrALOmjxEd8vshjc2V5GWksSsUR+8Wl3CF2Yh2AacDbxmZkOBScCWEPOIvM/IQf34x7Mm8NUzx7O6tIbHl5Xy55U7eX7tbrLTUzhn2jDOnz6Mk8flqSgcwZubq5gzaqD2U4wKrBCY2YPAGUCeme0AbgFSAdz9TuA/gPvMbBVgwM3uXhlUHpGjZWZML8xhemEO371gCm9uruLJ5aU8t3oXjy7ZQWZqMqdPzGPe1GF8dMoQcvulhR05puzd38jashq+NW9i2FGkE0GOGvrsEd7fCXwsqO2LBCE5yTh1Qh6nTsjjh5+Yxltb9vDC2l28uLacv67ZTUqScfL4PC6YPoyPTR3GwP4qCou2VgHoLqMxTBPTiPQAd2fljuro1cxlbNtzADOYOCSb2aNzmTVqICeOHcyowYk3AumWJ1fzcPEOVtzyMU1mFCJNTCMSMDNj5shcZo7M5eZzJ7FmZw1/Kyln6ba9PLOyjAff3g7AjMIcLpo5go/PHJEwt1h4Y3MVc8cOUhGIYSoEIj3MzJhWkMO0ghwAWludLZV1LFhXwVMrdvLDZ0r4z7+UcOBq9EMAAAvZSURBVOr4PD5/0hjOmjykz977qKK2gY3ldVw6uzDsKHIYKgQiAUtKMsYPyWb8kGyuP/0YtlTU8eTynfxp8Xauv7+YgtxMrjppNJ8uGsmgPtan8OaWSP/AyeofiGk6VhPpZcfkZ/H/5k3k9ZvP5I4rZlM4MJNbn13HCT96ka8+sJRXNlTQ0hpffXedeXNzFdnpKRw74oNzSkjs0BGBSEhSkpPeu83F+l21PLR4G48vK+WZVWUU5GZy+dyRfOb4kQzJjt++hLe2VOkOr3FA/3dEYsCkYdnc8vFjWfSds7n9c7MYm9ef/3lhA6fc+hJfe3AZi7ZUEW8j/MqqD7K1cr+GjcYBHRGIxJD0lGQunDGCC2eMYEtFHQ8s2sYjxdv584qdFORmcuHM4Xx8xgiOHTEg5u+O+uZmXT8QL3QdgUiMO9jYwrOry/jzip28trGS5lZnXH5/PjlnJJfNLmBIDA5DbW11rrlvMSt27GPp9+aR1EdHRcUTXUcgEscy05Kjt8QuZO/+Rp5dvYvHl+3gJ8+t47+fX89HJuZz2exCzpo8hMy02LiXz63PreOVDRV89/wpKgJxQIVAJI4M7J/G504YxedOGMWWijoeXbKDR5fs4KV15fRLS2be1KFcOGMEp0/MIz0lnKLw24VbuevVLXz+pNFcd9rYUDJI9+jUkEica25pZdHWPTy9cifPrt7FvgNNZGekRIvCcE4dn99rV/U+u6qMr/xxKfOmDOWOK+f02Qvl4lEocxYHRYVApHNNLa28vqmSZ1aW8fyaXdTUNzMgI4V5U4dxwYxhgRaF4nf28Lm7FzFtxAD+eP2JuuV0jFEhEElAjc2tLNxUydMry3h+7S5q65vfO1K4aOYIThmfR2oPje/fvucAF//fQnIyU3nsyyf3uSuk+wJ1FoskoLSUJM6cPIQzJw+hsXk6CzdV8syqyJHC/KWlDOqfxvnTh3HRzAKKRg886k7duoZmrvtdMc0trdx9dZGKQBzSEYFIgmlobuHVDZU8ubyUF0t2U9/UyvCcDC6cMZyPzxzB9IKcLl+j0NLqfPH3xSxYX8F918zltAn5AaeXo6UjAhF5T3pKZHTRvKlD2d/QzIslu/nzijLue+MdfvPaVobnZHDC2EEcP3Ywx48dxDF5/Ts8Wmhtdf7ruXW8WFLODy46VkUgjgU5VeW9wIVAubtP66TNGcAviExhWenuHwkqj4h8UP/0FC4+roCLjyug+kATf12zi1c2VrBwcxVPLN8JQGqyMSQ7g+E5GeRnp1N9sInSfQcp21dPY0srV5wwis+fNDrkn0Q+jMBODZnZ6UAdcH9HhcDMcoE3gHPdfZuZDXH38iN9rk4NiQTP3Xmn6gCLt+5ha9V+dlfXU1ZdT3ltPbn90hiRm8mI3AzG5WfxiVkFPdbpLMEJ5dSQu79qZmMO0+RzwHx33xZtf8QiICK9w8wYm9efsXn9w44ivSDMMj4RGGhmL5vZEjP7fIhZREQSVpidxSnAHOBsIBN408zecvcN7Rua2Q3ADQCjRo3q1ZAiIn1dmEcEO4C/uvt+d68EXgVmdtTQ3e9y9yJ3L8rP18gEEZGeFGYheBI41cxSzKwfcAJQEmIeEZGEFOTw0QeBM4A8M9sB3EJkmCjufqe7l5jZc8BKoBW4291XB5VHREQ6FuSooc92oc1PgZ8GlUFERI5Mg39FRBKcCoGISIKLu5vOmVkFsA+obrM4p83rjp4f+m8eUHmUm277ud15v6PlneVt+7qjNkeb/0jZD9ems3wdvT7cvofg8h/tvm//Wvu+69mO9P7R7vu2z2Pp77a72ds+j5V9n+vuHQ+7dPe4ewB3dfa6o+dt/lvcU9vs6vsdLe8sb0eZeyL/kbJ3J//R7vsg8x/tvu9iZu37Xtz3HeWPhb/b7mbvjd+dD7Pv2z/i9dTQnw/zuqPn7dv3xDa7+n5HyzvL2/b14dp0V1fW72r+vrTv27/Wvj9yhq6+f7T7vu3zWMrf3exd2faRBLnv3yfuTg19GGZW7J3cdCkeKH944jk7xHf+eM4O8ZE/Xo8IjtZdYQf4kJQ/PPGcHeI7fzxnhzjIn1BHBCIi8kGJdkQgIiLtqBCIiCQ4FQIRkQSnQhBlZqeZ2Z1mdreZvRF2nu4ysyQz+08z+6WZXR12nu4wszPM7LXo/j8j7DxHw8z6m1mxmV0YdpbuMLMp0f3+qJl9Oew83WVml5jZb8zsT2b2sbDzdJeZHWNm95jZo2Hm6BOFwMzuNbNyM1vdbvm5ZrbezDaZ2bcP9xnu/pq7fwl4GvhdkHnb64n8wMVAIdBEZK6HXtFD2Z3I/NYZ9GJ26LH8ADcDDweTsmM99HtfEv29/zRwSpB52+uh/E+4+/XAl4DPBJm3vR7Kv8Xdrw02aRcc7RV7sfQATgdmA6vbLEsGNgPHAGnACmAqMJ3Il33bx5A26z0MZMdbfuDbwBej6z4aZ9mTousNBR6Iw30/D7gc+AJwYTxlj65zEfAs8Ll42/dt1vsfYHYc5++1v9mOHmFOVdlj3P1VMxvTbvHxwCZ33wJgZg8BF7v7j4EOD9/NbBRQ7e61Acb9gJ7IH53zoTH6siW4tO/XU/s+ai+QHkTOzvTQvj8D6E/kD/6gmf3F3VuDzA09t+/d/SngKTN7BvhjcIk/sN2e2PcG3Ao86+5Lg038fj38ux+qPlEIOlEAbG/zegeRWdAO51rgt4El6p7u5p8P/NLMTiMy7WeYupXdzC4FzgFygduDjdYl3crv7t8FMLMvAJW9UQQOo7v7/gzgUiIF+C+BJuua7v7efw34KJBjZuPd/c4gw3VBd/f/YOA/gVlm9i/RgtHr+nIh6DZ3vyXsDEfL3Q8QKWRxx93nEylkcc3d7ws7Q3e5+8vAyyHHOGrufhtwW9g5jpa7VxHp3whVn+gs7kQpMLLN68LosngRz/njOTvEd/54zg7KH4q+XAgWAxPMbKyZpRHpzHsq5EzdEc/54zk7xHf+eM4Oyh+OMHuqe7D3/kGgjL8Pnbw2uvx8YAORXvzvhp2zL+aP5+zxnj+esyt/bD100zkRkQTXl08NiYhIF6gQiIgkOBUCEZEEp0IgIpLgVAhERBKcCoGISIJTIZA+wczqenl7PTJnRXQuhmozW25m68zsv7uwziVmNrUnti8CKgQiHTKzw96Hy91P7sHNvebuxwGzgAvN7EjzAlxC5E6nIj1ChUD6LDMbZ2bPmdkSi8yANjm6/ONmtsjMlpnZi2Y2NLr8+2b2ezNbCPw++vpeM3vZzLaY2dfbfHZd9L9nRN9/NPov+geit0bGzM6PLltiZreZ2dOHy+vuB4HlRO5giZldb2aLzWyFmT1mZv3M7GQi8wf8NHoUMa6zn1Okq1QIpC+7C/iau88B/gn4VXT568CJ7j4LeAj45zbrTAU+6u6fjb6eTOQW2ccDt5hZagfbmQV8I7ruMcApZpYB/Bo4L7r9/COFNbOBwAT+fhvx+e4+191nAiVEbmHwBpF719zk7se5++bD/JwiXaLbUEufZGZZwMnAI9F/oMPfJ70pBP5kZsOJzCK1tc2qT0X/ZX7IM+7eADSYWTmRWdTaT6f5trvviG53OTCGyNSbW9z90Gc/CNzQSdzTzGwFkSLwC3ffFV0+zcx+SGSehizgr938OUW6RIVA+qokYF/03Ht7vwR+5u5PRSdm+X6b9/a3a9vQ5nkLHf/NdKXN4bzm7hea2VjgLTN72N2XA/cBl7j7iuikN2d0sO7hfk6RLtGpIemT3L0G2Gpmn4LIlIZmNjP6dg5/v0f81QFFWA8c02YqwyNOrB49ergVuDm6KBsoi56OuqJN09roe0f6OUW6RIVA+op+ZrajzeObRL48r42edlkDXBxt+30ip1KWAJVBhImeXvoK8Fx0O7VAdRdWvRM4PVpA/hVYBCwE1rVp8xBwU7Szexyd/5wiXaLbUIsExMyy3L0uOoro/4CN7v7zsHOJtKcjApHgXB/tPF5D5HTUr0POI9IhHRGIiCQ4HRGIiCQ4FQIRkQSnQiAikuBUCEREEpwKgYhIglMhEBFJcP8fi+10XRER8RUAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light",
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn = learn.load(path/'char_bookcorpus_10m')\n",
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 398
    },
    "id": "3OUpAAMVF7Nx",
    "outputId": "4fd1e032-fff5-47d1-80af-5cbd56e415c7"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.744969</td>\n",
       "      <td>1.472124</td>\n",
       "      <td>0.618443</td>\n",
       "      <td>4.358481</td>\n",
       "      <td>00:08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1.442924</td>\n",
       "      <td>1.128739</td>\n",
       "      <td>0.658796</td>\n",
       "      <td>3.091756</td>\n",
       "      <td>00:08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>1.248169</td>\n",
       "      <td>1.068535</td>\n",
       "      <td>0.670881</td>\n",
       "      <td>2.911111</td>\n",
       "      <td>00:08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1.134621</td>\n",
       "      <td>1.048366</td>\n",
       "      <td>0.675764</td>\n",
       "      <td>2.852986</td>\n",
       "      <td>00:08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1.058302</td>\n",
       "      <td>1.044972</td>\n",
       "      <td>0.678554</td>\n",
       "      <td>2.843319</td>\n",
       "      <td>00:08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.004934</td>\n",
       "      <td>1.036241</td>\n",
       "      <td>0.680333</td>\n",
       "      <td>2.818603</td>\n",
       "      <td>00:08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>0.966509</td>\n",
       "      <td>1.042076</td>\n",
       "      <td>0.679461</td>\n",
       "      <td>2.835096</td>\n",
       "      <td>00:08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.938987</td>\n",
       "      <td>1.040590</td>\n",
       "      <td>0.680507</td>\n",
       "      <td>2.830886</td>\n",
       "      <td>00:08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>0.920266</td>\n",
       "      <td>1.035050</td>\n",
       "      <td>0.681728</td>\n",
       "      <td>2.815248</td>\n",
       "      <td>00:08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>0.907935</td>\n",
       "      <td>1.037062</td>\n",
       "      <td>0.681257</td>\n",
       "      <td>2.820917</td>\n",
       "      <td>00:08</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(10, 1e-4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RmpE5ZBb5Gsa"
   },
   "source": [
    "As you see pretraining model on large corpus followed by finetuning helped to reduce validation loss from arount 1.53 to 1.037 and improve accuracy in predicting next character to 68% (compared to 56.7% before). Let's see how it effects sampled text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "S1znWGrSGdBi"
   },
   "outputs": [],
   "source": [
    "out = generate(learn.model, tok('Alice said '), max_len=200, early_stopping=True, eos_idx=tok.c2i['xxeos'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "g58gqpjSGdBn",
    "outputId": "fa861c38-f437-490e-bd86-625e338e4366"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Alice said what you want is, why, if you cant see a little rather from people to bed their moment, when birds began drinking from behind, and offering the cart to say something, and dripping off a strange mou\n"
     ]
    }
   ],
   "source": [
    "#collapse-hide\n",
    "print(tok.decode(out[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "4E3SG-5uGWHD",
    "outputId": "9e984d1d-a21b-46e1-f5f5-381103124cba"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Path('/content/drive/MyDrive/char_model/char_alice.pth')"
      ]
     },
     "execution_count": 80,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#hide\n",
    "learn.save(path/'char_alice')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ZDKztD0VKor4"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "19342813113834066795298816"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "16**21"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "toc-showcode": false,
  "toc-showmarkdowntxt": false,
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "0022be0dc7aa4f4f9d947589308153f8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_b5fe978e2b204b5a986feb53c241d943",
       "IPY_MODEL_f61af757678d40e7b6c470ca779a0e1b"
      ],
      "layout": "IPY_MODEL_5a0205b7d88a42a68d0f8d40265add8b"
     }
    },
    "0a60355694ac41c3bb36988061280cd0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: ",
      "description_tooltip": null,
      "layout": "IPY_MODEL_db53b231459c49aa9d8fcef70617ef0b",
      "max": 946,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_a2628b3a684444b9b3aca1c36e111516",
      "value": 946
     }
    },
    "0ba48d9e012f457c99d9e050ee66ce19": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_0a60355694ac41c3bb36988061280cd0",
       "IPY_MODEL_9542a60fecc74375b59c2ef4b667c624"
      ],
      "layout": "IPY_MODEL_0ff15dad1c594db7b6ed85b2d4261a63"
     }
    },
    "0ff15dad1c594db7b6ed85b2d4261a63": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "13c5eb342bfd4489b067effccb6d2118": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9fe8214a7b1941039aadf375b90373f6",
      "placeholder": "​",
      "style": "IPY_MODEL_493d11cd156a4bfcb2bdc25aa21802fe",
      "value": " 1.18G/1.18G [03:47&lt;00:00, 5.19MB/s]"
     }
    },
    "1f5d679f52bf417dac756ae3e85f864a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_4468101700ae4bccb66bba2dd10a13d9",
       "IPY_MODEL_13c5eb342bfd4489b067effccb6d2118"
      ],
      "layout": "IPY_MODEL_cbe70f7f0c7c4bd99bf858a1d72d63c7"
     }
    },
    "201aa25469944b0b80c9363fcc4fc95f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_55f2f9482e3042bf81cb51aa64defd18",
      "placeholder": "​",
      "style": "IPY_MODEL_fa3423ee18214e89a9ff45554a5a2d21",
      "value": " 3.49k/? [07:00&lt;00:00, 8.30B/s]"
     }
    },
    "298d60fa5cd8441f86187130f3729e73": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "2c46976452b8428da3f74d1d37267a55": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2d2bb02afe8c4309a87354fd7858e864": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_8ecda540586c4760bf94ea3eae0767d2",
       "IPY_MODEL_201aa25469944b0b80c9363fcc4fc95f"
      ],
      "layout": "IPY_MODEL_89f78d2a56c14ceaa6b1ebba54c4feb2"
     }
    },
    "4468101700ae4bccb66bba2dd10a13d9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_abfe640017214179908de35ea38474ce",
      "max": 1179510242,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_a865a148857c4d1c962381692bd59857",
      "value": 1179510242
     }
    },
    "493d11cd156a4bfcb2bdc25aa21802fe": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "55f2f9482e3042bf81cb51aa64defd18": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5a0205b7d88a42a68d0f8d40265add8b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7571cf9691ac419fb79dad6dfaa531d1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "789151104986470ea6b887996b5833b5": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "7977691d32f54349a98abf6008504c3e": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "89f78d2a56c14ceaa6b1ebba54c4feb2": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8ecda540586c4760bf94ea3eae0767d2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: ",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7977691d32f54349a98abf6008504c3e",
      "max": 1689,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_298d60fa5cd8441f86187130f3729e73",
      "value": 1689
     }
    },
    "9542a60fecc74375b59c2ef4b667c624": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ec9d69f76b464d54ad54fd49738e0420",
      "placeholder": "​",
      "style": "IPY_MODEL_cbd787e3c17c402299963144601fb8f0",
      "value": " 1.67k/? [00:00&lt;00:00, 3.96kB/s]"
     }
    },
    "9fe8214a7b1941039aadf375b90373f6": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a2628b3a684444b9b3aca1c36e111516": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "a865a148857c4d1c962381692bd59857": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "abfe640017214179908de35ea38474ce": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b5fe978e2b204b5a986feb53c241d943": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "info",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_febdc99b2c074afb8e1b11aa35e7fcdf",
      "max": 1,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_7571cf9691ac419fb79dad6dfaa531d1",
      "value": 1
     }
    },
    "cbd787e3c17c402299963144601fb8f0": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "cbe70f7f0c7c4bd99bf858a1d72d63c7": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "db53b231459c49aa9d8fcef70617ef0b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ec9d69f76b464d54ad54fd49738e0420": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f61af757678d40e7b6c470ca779a0e1b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2c46976452b8428da3f74d1d37267a55",
      "placeholder": "​",
      "style": "IPY_MODEL_789151104986470ea6b887996b5833b5",
      "value": " 74004228/0 [10:05&lt;00:00, 122513.28 examples/s]"
     }
    },
    "fa3423ee18214e89a9ff45554a5a2d21": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "febdc99b2c074afb8e1b11aa35e7fcdf": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}