HW4_Ulyanova_3.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "brf4zRkH9S6t"
      },
      "source": [
        "# Задание 1: сгенерировать датасет.\n",
        "\n",
        "Вам предстоит обучить микро-GPT2 модель для генерации текста. Но для начала вам\n",
        "нужно сделать датасет!\n",
        "Датасет должен состоять из текстов вашего любого писателя. Требование к датасету:\n",
        "\n",
        "*   Датасет должен быть одним txt-файлом.\n",
        "*   Выложите датасет куда-нибудь в открытый доступ и загрузите в этот ноутбук с помощью wget\n",
        "*   Датасет должен содержать по меньшей мере 500 тысяч символов. Лучше -- миллион.\n",
        "\n",
        "\n",
        "Советы:\n",
        "* www.lib.ru -- удобный сайт с txt текстов русских писателей.\n",
        "* Моделировать поэзию чуть веселее, чем прозу.\n",
        "\n",
        "# Задание 2: оптимизировать модель\n",
        "Ниже приведен код обучения микро-GPT2. Прежде, чем обучать, нужно немного оптимизировать код.  Объедините `Head` и `MultiHeadAttention` в один класс, который обрабатывает все головы параллельно, рассматривая головы как еще одну размерность. Это поможет избавиться от цикла в `MultiHeadAttention` и ускорить процесс обучения. Проверьте, что ваша имплементация идентична той, что приведена ниже.\n",
        "# Задание 3: обучите модель на ваших текстах\n",
        "Используйте те параметры, которые указаны ниже и обучите модель на вашем датасете. Обучение в колабе может занять около часа. Если результаты будут выглядеть неубедительно, учите подольше. Если есть доступ к хорошей GPU, можете немного увеличить размер трансформера.\n",
        "# Задание 4: сложение\n",
        "Обучите GPT выполнять сложение двух чисел, то есть $a+b=c$. Возможно, вам будет полезно предсказать цифры числа $c$ в обратном порядке, поскольку типичный алгоритм сложения (который, как мы надеемся, GPT выучит) выполняется справа налево. Возможно, вы захотите модифицировать dataloader, чтобы он просто предоставлял случайные задачи и пропускал генерацию train и val. Возможно, вы захотите замаскировать loss на входных позициях a+b, которые просто определяют задачу, используя y=-1 в таргетах (см. CrossEntropyLoss ignore_index). Работает ли сложение? Постройте график точности сложения в зависимости от количества цифр в примере. Для этого сгенерируйте по 100 примеров с 2-мя цифрам, 3-мя и т.д.\n",
        "# Задание 5: калькулятор\n",
        "Если ваш трасформер научился складывать (хотя бы до какого-то порядка), научите его умножать, делить и вычитать. Попробуйте обучить простой калькулятор (без скобок), в котором по одному разу может встречать каждый из знаков: *, /, +, -. Это не очень простая задача :) Возможно, вы захотите добавить в датасет chain of thoughts (https://arxiv.org/abs/2201.11903), а не только ответ.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jQSoeF2b4vix"
      },
      "source": [
        "# Генерация стихов"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "UG77Iy4zvQds"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "import torch.nn as nn\n",
        "from torch.nn import functional as F"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "r19j7PyxzjRD",
        "outputId": "8aa8dbd1-1eb3-4f0b-ae8b-e2b100df7102"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<torch._C.Generator at 0x117b93290>"
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# # hyperparameters\n",
        "# batch_size = 64\n",
        "# block_size = 256\n",
        "# max_iters = 5000\n",
        "# eval_interval = 500\n",
        "# learning_rate = 3e-4\n",
        "# device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
        "# eval_iters = 200\n",
        "# n_embd = 384\n",
        "# n_head = 6 # D = 384 // 6\n",
        "# n_layer = 6\n",
        "# dropout = 0.2\n",
        "# # ------------\n",
        "\n",
        "# torch.manual_seed(1337)\n",
        "\n",
        "# hyperparameters\n",
        "batch_size = 512\n",
        "block_size = 256\n",
        "max_iters = 5000\n",
        "eval_interval = 500\n",
        "learning_rate = 3e-4\n",
        "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
        "eval_iters = 200\n",
        "n_embd = 1600\n",
        "n_head = 12 # D = 384 // 6\n",
        "n_layer = 48\n",
        "dropout = 0.2\n",
        "# ------------\n",
        "\n",
        "torch.manual_seed(1337)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "fsFmeu0czmJ-",
        "outputId": "5198dea2-31e8-4cd5-97d7-d1c8b214ca62"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "--2024-05-29 09:44:58--  https://raw.githubusercontent.com/marulyanova/NLP_6sem/main/dataset_poetry_mac.txt\n",
            "Распознаётся raw.githubusercontent.com (raw.githubusercontent.com)… 185.199.108.133, 185.199.109.133, 185.199.110.133, ...\n",
            "Подключение к raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... соединение установлено.\n",
            "HTTP-запрос отправлен. Ожидание ответа… 200 OK\n",
            "Длина: 677788 (662K) [text/plain]\n",
            "Сохранение в: «dataset_poetry_mac.txt»\n",
            "\n",
            "dataset_poetry_mac. 100%[===================>] 661,90K  1,64MB/s    за 0,4s    \n",
            "\n",
            "2024-05-29 09:44:59 (1,64 MB/s) - «dataset_poetry_mac.txt» сохранён [677788/677788]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!wget https://raw.githubusercontent.com/marulyanova/NLP_6sem/main/dataset_poetry_mac.txt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "Clm_Lf82fF7r"
      },
      "outputs": [],
      "source": [
        "import re\n",
        "\n",
        "with open('dataset_poetry_mac.txt', 'r', encoding = 'MACCYRILLIC') as file:\n",
        "    lines = file.readlines()\n",
        "\n",
        "new_lines = []\n",
        "for line in lines:\n",
        "\n",
        "    # убрать пробелы в начале строки, оставить только русские буквы, убрать строки, где только цифры (года написания стихов), названия стихов\n",
        "\n",
        "    line = line.lstrip()\n",
        "    line = re.sub(r'[a-zA-Z]', '', line)\n",
        "    if line.isdigit():\n",
        "        continue\n",
        "    if line.isupper():\n",
        "            line = '*\\n'\n",
        "    new_lines.append(line)\n",
        "\n",
        "with open('dataset_poetry_mac_modified.txt', 'w', encoding = 'MACCYRILLIC') as file:\n",
        "    file.writelines(new_lines)\n",
        "\n",
        "\n",
        "with open('dataset_poetry_mac_modified.txt', 'r', encoding = 'MACCYRILLIC') as f:\n",
        "    text = f.read()\n",
        "\n",
        "chars = sorted(list(set(text)))\n",
        "vocab_size = len(chars)\n",
        "stoi = { ch:i for i,ch in enumerate(chars) }\n",
        "itos = { i:ch for i,ch in enumerate(chars) }\n",
        "encode = lambda s: [stoi[c] for c in s]\n",
        "decode = lambda l: ''.join([itos[i] for i in l])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NMF8BOorhHoN",
        "outputId": "207a98d9-00e7-43c9-f2f2-8be5abc5b7c9"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "['Ты опять упрекнула меня,',\n",
              " 'Что я с музой моей раздружился,',\n",
              " 'Что заботам текущего дня',\n",
              " 'И забавам его подчинился.',\n",
              " 'Для житейских расчетов и чар',\n",
              " 'Не расстался б я с музой моею,',\n",
              " 'Но бог весть, не погас ли тот дар,',\n",
              " 'Что, бывало, дружил меня с нею?',\n",
              " 'Но не брат еще людям поэт,',\n",
              " 'И тернист его путь, и непрочен,',\n",
              " 'Я умел не бояться клевет,',\n",
              " 'Не был ими я сам озабочен;',\n",
              " 'Но я знал, чье во мраке ночном',\n",
              " 'Надрывалося сердце с печали',\n",
              " 'И на чью они грудь упадали свинцом,',\n",
              " 'И кому они жизнь отравляли.',\n",
              " 'И пускай они мимо прошли,',\n",
              " 'Надо мною ходившие грозы,',\n",
              " 'Знаю я, чьи молитвы и слезы',\n",
              " 'Роковую стрелу отвели...',\n",
              " 'Да и время ушло,- я устал']"
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "text[:600].split('\\n')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cN5eHXFyicFT",
        "outputId": "8acc12ed-4768-4ee3-ae53-9999acdf0d30"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "626213"
            ]
          },
          "execution_count": 14,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "len(text) # длина датасета 600к символов"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZqfdB6HehhWa",
        "outputId": "9b6d8050-a5d2-4dcc-e4a9-2419cc70e940"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'\\n': 0,\n",
              " ' ': 1,\n",
              " '!': 2,\n",
              " '\"': 3,\n",
              " \"'\": 4,\n",
              " '(': 5,\n",
              " ')': 6,\n",
              " '*': 7,\n",
              " ',': 8,\n",
              " '-': 9,\n",
              " '.': 10,\n",
              " '0': 11,\n",
              " '1': 12,\n",
              " '2': 13,\n",
              " '3': 14,\n",
              " '4': 15,\n",
              " '5': 16,\n",
              " '6': 17,\n",
              " '7': 18,\n",
              " '8': 19,\n",
              " '9': 20,\n",
              " ':': 21,\n",
              " ';': 22,\n",
              " '<': 23,\n",
              " '>': 24,\n",
              " '?': 25,\n",
              " '@': 26,\n",
              " '[': 27,\n",
              " ']': 28,\n",
              " '_': 29,\n",
              " '{': 30,\n",
              " '}': 31,\n",
              " 'А': 32,\n",
              " 'Б': 33,\n",
              " 'В': 34,\n",
              " 'Г': 35,\n",
              " 'Д': 36,\n",
              " 'Е': 37,\n",
              " 'Ж': 38,\n",
              " 'З': 39,\n",
              " 'И': 40,\n",
              " 'Й': 41,\n",
              " 'К': 42,\n",
              " 'Л': 43,\n",
              " 'М': 44,\n",
              " 'Н': 45,\n",
              " 'О': 46,\n",
              " 'П': 47,\n",
              " 'Р': 48,\n",
              " 'С': 49,\n",
              " 'Т': 50,\n",
              " 'У': 51,\n",
              " 'Ф': 52,\n",
              " 'Х': 53,\n",
              " 'Ц': 54,\n",
              " 'Ч': 55,\n",
              " 'Ш': 56,\n",
              " 'Щ': 57,\n",
              " 'Ъ': 58,\n",
              " 'Ы': 59,\n",
              " 'Ь': 60,\n",
              " 'Э': 61,\n",
              " 'Ю': 62,\n",
              " 'Я': 63,\n",
              " 'а': 64,\n",
              " 'б': 65,\n",
              " 'в': 66,\n",
              " 'г': 67,\n",
              " 'д': 68,\n",
              " 'е': 69,\n",
              " 'ж': 70,\n",
              " 'з': 71,\n",
              " 'и': 72,\n",
              " 'й': 73,\n",
              " 'к': 74,\n",
              " 'л': 75,\n",
              " 'м': 76,\n",
              " 'н': 77,\n",
              " 'о': 78,\n",
              " 'п': 79,\n",
              " 'р': 80,\n",
              " 'с': 81,\n",
              " 'т': 82,\n",
              " 'у': 83,\n",
              " 'ф': 84,\n",
              " 'х': 85,\n",
              " 'ц': 86,\n",
              " 'ч': 87,\n",
              " 'ш': 88,\n",
              " 'щ': 89,\n",
              " 'ъ': 90,\n",
              " 'ы': 91,\n",
              " 'ь': 92,\n",
              " 'э': 93,\n",
              " 'ю': 94,\n",
              " 'я': 95,\n",
              " 'ё': 96}"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "stoi"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "HIhreXFwzvDw"
      },
      "outputs": [],
      "source": [
        "data = torch.tensor(encode(text), dtype=torch.long)\n",
        "n = int(0.9 * len(data))\n",
        "train_data = data[:n]\n",
        "val_data = data[n:]\n",
        "\n",
        "def get_batch(split):\n",
        "    data = train_data if split == 'train' else val_data\n",
        "    ix = torch.randint(len(data) - block_size, (batch_size,))\n",
        "    x = torch.stack([data[i:i+block_size] for i in ix])\n",
        "    y = torch.stack([data[i+1:i+block_size+1] for i in ix])\n",
        "    x, y = x.to(device), y.to(device)\n",
        "    return x, y"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "id": "_J1208XQzxqQ"
      },
      "outputs": [],
      "source": [
        "@torch.no_grad()\n",
        "def estimate_loss():\n",
        "    out = {}\n",
        "    model.eval()\n",
        "    for split in ['train', 'val']:\n",
        "        losses = torch.zeros(eval_iters)\n",
        "        for k in range(eval_iters):\n",
        "            X, Y = get_batch(split)\n",
        "            logits, loss = model(X, Y)\n",
        "            losses[k] = loss.item()\n",
        "        out[split] = losses.mean()\n",
        "    model.train()\n",
        "    return out"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "y1k8yJp_IyEd"
      },
      "outputs": [],
      "source": [
        "class MultiHeadAttention_Parallel(nn.Module):\n",
        "\n",
        "    def __init__(self, num_heads, head_size):\n",
        "        super().__init__()\n",
        "        self.key = nn.Linear(n_embd, head_size * num_heads, bias = False)\n",
        "        self.query = nn.Linear(n_embd, head_size * num_heads, bias = False)\n",
        "        self.value = nn.Linear(n_embd, head_size * num_heads, bias = False)\n",
        "        self.register_buffer('tril', torch.tril(torch.ones(block_size, block_size)))\n",
        "\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "        self.num_heads = num_heads\n",
        "        self.head_size = head_size\n",
        "\n",
        "        self.proj = nn.Linear(head_size * num_heads, n_embd)\n",
        "\n",
        "    def forward(self, x):\n",
        "        # input of size (B, T, C)\n",
        "        # output of size (B, T, D * num_heads)\n",
        "\n",
        "        B, T, C = x.shape\n",
        "        k = self.key(x).reshape(B, T, self.num_heads, self.head_size).transpose(1, 2) # (B, head_size, T, num_heads)\n",
        "        q = self.query(x).reshape(B, T, self.num_heads, self.head_size).transpose(1, 2) # (B, head_size, T, num_heads)\n",
        "        v = self.value(x).reshape(B, T, self.num_heads, self.head_size).transpose(1, 2) # (B, head_size, T, num_heads)\n",
        "\n",
        "        masks = self.tril[:T, :T].unsqueeze(0).unsqueeze(0)\n",
        "        wei = q @ k.transpose(-2, -1) * k.shape[-1] ** -0.5\n",
        "        wei = wei.masked_fill(self.tril[:T, :T] == 0, float('-inf'))\n",
        "        wei = F.softmax(wei, dim =- 1)\n",
        "        wei = self.dropout(wei)\n",
        "        out = wei @ v\n",
        "\n",
        "        out = out.transpose(1, 2).contiguous().view(B, T, self.head_size * self.num_heads)\n",
        "        out = self.dropout(self.proj(out))\n",
        "\n",
        "        return out"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "id": "QP5ka09oz6HK"
      },
      "outputs": [],
      "source": [
        "class FeedFoward(nn.Module):\n",
        "    \"\"\" a simple linear layer followed by a non-linearity \"\"\"\n",
        "\n",
        "    def __init__(self, n_embd):\n",
        "        super().__init__()\n",
        "        self.net = nn.Sequential(\n",
        "            nn.Linear(n_embd, 4 * n_embd),\n",
        "            nn.ReLU(),\n",
        "            nn.Linear(4 * n_embd, n_embd),\n",
        "            nn.Dropout(dropout),\n",
        "        )\n",
        "\n",
        "    def forward(self, x):\n",
        "        return self.net(x)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "id": "in5r2XmXz8N9"
      },
      "outputs": [],
      "source": [
        "class Block(nn.Module):\n",
        "    \"\"\" Transformer block: communication followed by computation \"\"\"\n",
        "\n",
        "    def __init__(self, n_embd, n_head):\n",
        "        super().__init__()\n",
        "        head_size = n_embd // n_head\n",
        "        self.sa = MultiHeadAttention_Parallel(n_head, head_size)\n",
        "        self.ffwd = FeedFoward(n_embd)\n",
        "        self.ln1 = nn.LayerNorm(n_embd)\n",
        "        self.ln2 = nn.LayerNorm(n_embd)\n",
        "\n",
        "    def forward(self, x):\n",
        "        x = x + self.sa(self.ln1(x))\n",
        "        x = x + self.ffwd(self.ln2(x))\n",
        "        return x"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "id": "bqk8eOkHz-V2"
      },
      "outputs": [],
      "source": [
        "class GPTLanguageModel(nn.Module):\n",
        "\n",
        "    def __init__(self):\n",
        "        super().__init__()\n",
        "        self.token_embedding_table = nn.Embedding(vocab_size, n_embd)\n",
        "        self.position_embedding_table = nn.Embedding(block_size, n_embd)\n",
        "        self.blocks = nn.Sequential(*[Block(n_embd, n_head=n_head) for _ in range(n_layer)])\n",
        "        self.ln_f = nn.LayerNorm(n_embd)\n",
        "        self.lm_head = nn.Linear(n_embd, vocab_size)\n",
        "\n",
        "        self.apply(self._init_weights)\n",
        "\n",
        "    def _init_weights(self, module):\n",
        "        if isinstance(module, nn.Linear):\n",
        "            torch.nn.init.normal_(module.weight, mean=0.0, std=0.02)\n",
        "            if module.bias is not None:\n",
        "                torch.nn.init.zeros_(module.bias)\n",
        "        elif isinstance(module, nn.Embedding):\n",
        "            torch.nn.init.normal_(module.weight, mean=0.0, std=0.02)\n",
        "\n",
        "    def forward(self, idx, targets=None):\n",
        "        B, T = idx.shape\n",
        "\n",
        "        tok_emb = self.token_embedding_table(idx) # (B,T,C)\n",
        "        pos_emb = self.position_embedding_table(torch.arange(T, device=device)) # (T,C)\n",
        "        x = tok_emb + pos_emb # (B,T,C)\n",
        "        x = self.blocks(x) # (B,T,C)\n",
        "        x = self.ln_f(x) # (B,T,C)\n",
        "        logits = self.lm_head(x) # (B,T,vocab_size)\n",
        "\n",
        "        if targets is None:\n",
        "            loss = None\n",
        "        else:\n",
        "            B, T, C = logits.shape\n",
        "            logits = logits.view(B*T, C)\n",
        "            targets = targets.view(B*T)\n",
        "            loss = F.cross_entropy(logits, targets)\n",
        "\n",
        "        return logits, loss\n",
        "\n",
        "    def generate(self, idx, max_new_tokens):\n",
        "        for _ in range(max_new_tokens):\n",
        "            idx_cond = idx[:, -block_size:]\n",
        "            logits, loss = self(idx_cond)\n",
        "            logits = logits[:, -1, :]\n",
        "            probs = F.softmax(logits, dim=-1)\n",
        "            idx_next = torch.multinomial(probs, num_samples=1)\n",
        "            idx = torch.cat((idx, idx_next), dim=1)\n",
        "        return idx"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "k7wE3_FD0Coy",
        "outputId": "a539f18d-b241-4884-9fed-694785d6a714"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "1474.822497 M parameters\n"
          ]
        }
      ],
      "source": [
        "model = GPTLanguageModel()\n",
        "m = model.to(device)\n",
        "print(sum(p.numel() for p in m.parameters())/1e6, 'M parameters')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NxB-pom00Fdk",
        "outputId": "bcc5a0c8-524a-4e6f-956a-128e42985d6e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "step 0: train loss 4.6937, val loss 4.7020\n",
            "step 500: train loss 2.3368, val loss 2.4008\n",
            "step 1000: train loss 1.8596, val loss 2.0421\n",
            "step 1500: train loss 1.5944, val loss 1.9295\n",
            "step 2000: train loss 1.4269, val loss 1.9088\n",
            "step 2500: train loss 1.2806, val loss 1.9353\n",
            "step 3000: train loss 1.1455, val loss 1.9942\n",
            "step 3500: train loss 1.0095, val loss 2.0591\n",
            "step 4000: train loss 0.8757, val loss 2.1618\n",
            "step 4500: train loss 0.7514, val loss 2.2879\n",
            "step 4999: train loss 0.6358, val loss 2.3948\n"
          ]
        }
      ],
      "source": [
        "try:\n",
        "  optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)\n",
        "\n",
        "  for iter in range(max_iters):\n",
        "\n",
        "      if iter % eval_interval == 0 or iter == max_iters - 1:\n",
        "          losses = estimate_loss()\n",
        "          print(f\"step {iter}: train loss {losses['train']:.4f}, val loss {losses['val']:.4f}\")\n",
        "\n",
        "      xb, yb = get_batch('train')\n",
        "      logits, loss = model(xb, yb)\n",
        "      optimizer.zero_grad(set_to_none=True)\n",
        "      loss.backward()\n",
        "      optimizer.step()\n",
        "\n",
        "except KeyboardInterrupt: 0"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xlA07ptw46Di"
      },
      "source": [
        "## Результаты генерации"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "I6brfGxe0KVY",
        "outputId": "ee6eb46f-f649-4997-cc56-3bf99218dd86"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "Влад у небес, мать Боюся, хоть и вдохновенных,\n",
            "Не сели одинких гор\n",
            "Заковарным в ином племенье дань\n",
            "За кустером, улетела, молчалая\n",
            "И спящей в беседе ядов шинки\n",
            "И средь них гордела невежен;\n",
            "Играть сопутного Россиянки -\n",
            "Кричит на тихом в капичке.\n",
            "И, моя лил твой был крыток,\n",
            "Скажи, даже слышу в лед,\n",
            "Воспросами до творца кручи,\n",
            "И ярый и жених думает обнимает.\n",
            "Питомец мудрец был у обним,\n",
            "Кроме сколько в зеркалье сметливой,\n",
            "Держа пал сухуду прожает,\n",
            "На ветвями тверых законел.\n",
            "Однажды утомлен, везде пос\n"
          ]
        }
      ],
      "source": [
        "# generate from the model\n",
        "context = torch.zeros((1, 1), dtype=torch.long, device=device)\n",
        "print(decode(m.generate(context, max_new_tokens = 500)[0].tolist()))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ap47V2p4tq-v",
        "outputId": "41241d65-dd17-4860-9393-8060a38b23f2"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "И звезда горда потерянь!\n",
            "Но не закрылся мне, роскоши судь\n",
            "Спокойной рукою спиной!\n",
            "Мною на радостью в немиле, -\n",
            "Мною старальной для страстью овладеленный.\n",
            "Он был влюблен от ранний серы ус,\n",
            "И все, для клякою нанесенный,\n",
            "И как божий остынет пыла ть и лечу.\n",
            "*\n",
            "Когда зареют волнует ковровь -\n",
            "Уже зарею пусть вам обиды,\n",
            "Сна гром колпака и заснает.\n",
            "Он вас не увидит штуненья;\n",
            "Волнуясь ночи потомству находит\n",
            "Не вдруг за кричи ночный темнотой,\n",
            "Ужели тлполеды кровавой?\n",
            "Все тихо, видит он молчат.\n",
            "Пред сомным,\n"
          ]
        }
      ],
      "source": [
        "context = torch.zeros((1, 1), dtype=torch.long, device=device)\n",
        "print(decode(m.generate(context, max_new_tokens = 500)[0].tolist()))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0rHWEcQQIyEg"
      },
      "source": [
        "----"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vMFlToBSIyEg"
      },
      "source": [
        "# Сложение"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GpIbzz0qIyEh"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import random\n",
        "import torch.nn.functional as F\n",
        "from torch.utils.data import DataLoader, TensorDataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lInEqeU-U8FD"
      },
      "source": [
        "### Создание датасета\n",
        "Если имеем пример 'abc+xyz=nmk', то для задачи предсказывания следующего токена, в датасете создадим четыре записи:\n",
        "\n",
        "1) context 'abc+xyz=' target 'abc+xyz=k'\n",
        "\n",
        "2) context 'abc+xyz=k' target 'abc+xyz=km'\n",
        "\n",
        "3) context 'abc+xyz=km' target 'abc+xyz=kmn'\n",
        "\n",
        "4) context 'abc+xyz=kmn' target 'abc+xyz=kmnE' (E - end of sequence)\n",
        "\n",
        "Более короткий пример '1+1=2':\n",
        "\n",
        "1) context '1+1=' target '1+1=2'\n",
        "\n",
        "1) context '1+1=2' target '1+1=2E'\n",
        "\n",
        "Максимальная длина последовательности = 12, так как max = трехзначное + трехзначное = трехзначноеЕ\n",
        "\n",
        "Таргет (результат сложения) перевёрнут.\n",
        "\n",
        "Паддинги для дополнения до максимальной длины добавляем вперёд последовательности (примера)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "lGL4UNHbvXdt",
        "outputId": "eedf19d1-a7ae-4a97-e6bb-5c1e6312022a"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "dataset"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-0c1281ef-51f8-4a23-a82f-a13a96ef6d2e\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>context</th>\n",
              "      <th>target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>........7+5=</td>\n",
              "      <td>.......7+5=2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>.......7+5=2</td>\n",
              "      <td>......7+5=21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>......7+5=21</td>\n",
              "      <td>.....7+5=21E</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>......25+53=</td>\n",
              "      <td>.....25+53=8</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>.....25+53=8</td>\n",
              "      <td>....25+53=87</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>773492</th>\n",
              "      <td>.298+502=008</td>\n",
              "      <td>298+502=008E</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>773505</th>\n",
              "      <td>....280+417=</td>\n",
              "      <td>...280+417=7</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>773506</th>\n",
              "      <td>...280+417=7</td>\n",
              "      <td>..280+417=79</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>773507</th>\n",
              "      <td>..280+417=79</td>\n",
              "      <td>.280+417=796</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>773508</th>\n",
              "      <td>.280+417=796</td>\n",
              "      <td>280+417=796E</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>179125 rows × 2 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-0c1281ef-51f8-4a23-a82f-a13a96ef6d2e')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-0c1281ef-51f8-4a23-a82f-a13a96ef6d2e button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-0c1281ef-51f8-4a23-a82f-a13a96ef6d2e');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-8db81910-bcc8-42ab-8992-d4fd6f347336\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-8db81910-bcc8-42ab-8992-d4fd6f347336')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-8db81910-bcc8-42ab-8992-d4fd6f347336 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "             context        target\n",
              "0       ........7+5=  .......7+5=2\n",
              "1       .......7+5=2  ......7+5=21\n",
              "2       ......7+5=21  .....7+5=21E\n",
              "3       ......25+53=  .....25+53=8\n",
              "4       .....25+53=8  ....25+53=87\n",
              "...              ...           ...\n",
              "773492  .298+502=008  298+502=008E\n",
              "773505  ....280+417=  ...280+417=7\n",
              "773506  ...280+417=7  ..280+417=79\n",
              "773507  ..280+417=79  .280+417=796\n",
              "773508  .280+417=796  280+417=796E\n",
              "\n",
              "[179125 rows x 2 columns]"
            ]
          },
          "execution_count": 28,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "def add_padding(context, max_len, pad_token = '.'):\n",
        "    return pad_token * (max_len - len(context)) + context\n",
        "\n",
        "NUM_SAMPLES = 100000\n",
        "context, target = [], []\n",
        "\n",
        "def create_sample(x, y):\n",
        "  global context\n",
        "  global target\n",
        "\n",
        "  a, b = random.randint(x, y), random.randint(x, y)\n",
        "  if a + b >= 1000:\n",
        "    return\n",
        "  task = str(a) + '+' + str(b) + '='\n",
        "  ans = str(a + b)[::-1]\n",
        "  for i in range(len(ans)):\n",
        "    context += [add_padding(task + ans[:i], 12)]\n",
        "    target += [add_padding(task + ans[: i + 1], 12)]\n",
        "  context += [add_padding(task + ans, 12)]\n",
        "  target += [add_padding(task + ans + 'E', 12)]\n",
        "\n",
        "for i in range(NUM_SAMPLES):\n",
        "  create_sample(0, 10)\n",
        "  create_sample(10, 100)\n",
        "  create_sample(100, 1000)\n",
        "\n",
        "dataset = pd.DataFrame({'context': context, 'target': target}).drop_duplicates()\n",
        "dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-z3If_gb3qTW"
      },
      "outputs": [],
      "source": [
        "chars = ['.'] + [str(i) for i in range(0, 10)] + ['+', '=', 'E']\n",
        "vocab_size = len(chars)\n",
        "stoi = { ch:i for i,ch in enumerate(chars) }\n",
        "itos = { i:ch for i,ch in enumerate(chars) }\n",
        "encode = lambda s: [stoi[c] for c in s]\n",
        "decode = lambda l: ''.join([itos[i] for i in l])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "j12PKOlQ3yis",
        "outputId": "2e936cdd-685d-4230-8e4b-223e319b05c0"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'.': 0,\n",
              " '0': 1,\n",
              " '1': 2,\n",
              " '2': 3,\n",
              " '3': 4,\n",
              " '4': 5,\n",
              " '5': 6,\n",
              " '6': 7,\n",
              " '7': 8,\n",
              " '8': 9,\n",
              " '9': 10,\n",
              " '+': 11,\n",
              " '=': 12,\n",
              " 'E': 13}"
            ]
          },
          "execution_count": 30,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "stoi"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "oQ9Am9SZ32Pd",
        "outputId": "0d2ee445-1bdd-490a-a7cc-51858bbab3eb"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "dataset"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-7580255f-ad90-474f-8233-3f297ec3c4c9\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>context</th>\n",
              "      <th>target</th>\n",
              "      <th>context_encoded</th>\n",
              "      <th>target_encoded</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>........7+5=</td>\n",
              "      <td>.......7+5=2</td>\n",
              "      <td>[0, 0, 0, 0, 0, 0, 0, 0, 8, 11, 6, 12]</td>\n",
              "      <td>[0, 0, 0, 0, 0, 0, 0, 8, 11, 6, 12, 3]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>.......7+5=2</td>\n",
              "      <td>......7+5=21</td>\n",
              "      <td>[0, 0, 0, 0, 0, 0, 0, 8, 11, 6, 12, 3]</td>\n",
              "      <td>[0, 0, 0, 0, 0, 0, 8, 11, 6, 12, 3, 2]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>......7+5=21</td>\n",
              "      <td>.....7+5=21E</td>\n",
              "      <td>[0, 0, 0, 0, 0, 0, 8, 11, 6, 12, 3, 2]</td>\n",
              "      <td>[0, 0, 0, 0, 0, 8, 11, 6, 12, 3, 2, 13]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>......25+53=</td>\n",
              "      <td>.....25+53=8</td>\n",
              "      <td>[0, 0, 0, 0, 0, 0, 3, 6, 11, 6, 4, 12]</td>\n",
              "      <td>[0, 0, 0, 0, 0, 3, 6, 11, 6, 4, 12, 9]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>.....25+53=8</td>\n",
              "      <td>....25+53=87</td>\n",
              "      <td>[0, 0, 0, 0, 0, 3, 6, 11, 6, 4, 12, 9]</td>\n",
              "      <td>[0, 0, 0, 0, 3, 6, 11, 6, 4, 12, 9, 8]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>773492</th>\n",
              "      <td>.298+502=008</td>\n",
              "      <td>298+502=008E</td>\n",
              "      <td>[0, 3, 10, 9, 11, 6, 1, 3, 12, 1, 1, 9]</td>\n",
              "      <td>[3, 10, 9, 11, 6, 1, 3, 12, 1, 1, 9, 13]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>773505</th>\n",
              "      <td>....280+417=</td>\n",
              "      <td>...280+417=7</td>\n",
              "      <td>[0, 0, 0, 0, 3, 9, 1, 11, 5, 2, 8, 12]</td>\n",
              "      <td>[0, 0, 0, 3, 9, 1, 11, 5, 2, 8, 12, 8]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>773506</th>\n",
              "      <td>...280+417=7</td>\n",
              "      <td>..280+417=79</td>\n",
              "      <td>[0, 0, 0, 3, 9, 1, 11, 5, 2, 8, 12, 8]</td>\n",
              "      <td>[0, 0, 3, 9, 1, 11, 5, 2, 8, 12, 8, 10]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>773507</th>\n",
              "      <td>..280+417=79</td>\n",
              "      <td>.280+417=796</td>\n",
              "      <td>[0, 0, 3, 9, 1, 11, 5, 2, 8, 12, 8, 10]</td>\n",
              "      <td>[0, 3, 9, 1, 11, 5, 2, 8, 12, 8, 10, 7]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>773508</th>\n",
              "      <td>.280+417=796</td>\n",
              "      <td>280+417=796E</td>\n",
              "      <td>[0, 3, 9, 1, 11, 5, 2, 8, 12, 8, 10, 7]</td>\n",
              "      <td>[3, 9, 1, 11, 5, 2, 8, 12, 8, 10, 7, 13]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>179125 rows × 4 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-7580255f-ad90-474f-8233-3f297ec3c4c9')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-7580255f-ad90-474f-8233-3f297ec3c4c9 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-7580255f-ad90-474f-8233-3f297ec3c4c9');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-543ffa5a-7c4d-4414-9828-ced81b9ae972\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-543ffa5a-7c4d-4414-9828-ced81b9ae972')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-543ffa5a-7c4d-4414-9828-ced81b9ae972 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "             context        target                          context_encoded  \\\n",
              "0       ........7+5=  .......7+5=2   [0, 0, 0, 0, 0, 0, 0, 0, 8, 11, 6, 12]   \n",
              "1       .......7+5=2  ......7+5=21   [0, 0, 0, 0, 0, 0, 0, 8, 11, 6, 12, 3]   \n",
              "2       ......7+5=21  .....7+5=21E   [0, 0, 0, 0, 0, 0, 8, 11, 6, 12, 3, 2]   \n",
              "3       ......25+53=  .....25+53=8   [0, 0, 0, 0, 0, 0, 3, 6, 11, 6, 4, 12]   \n",
              "4       .....25+53=8  ....25+53=87   [0, 0, 0, 0, 0, 3, 6, 11, 6, 4, 12, 9]   \n",
              "...              ...           ...                                      ...   \n",
              "773492  .298+502=008  298+502=008E  [0, 3, 10, 9, 11, 6, 1, 3, 12, 1, 1, 9]   \n",
              "773505  ....280+417=  ...280+417=7   [0, 0, 0, 0, 3, 9, 1, 11, 5, 2, 8, 12]   \n",
              "773506  ...280+417=7  ..280+417=79   [0, 0, 0, 3, 9, 1, 11, 5, 2, 8, 12, 8]   \n",
              "773507  ..280+417=79  .280+417=796  [0, 0, 3, 9, 1, 11, 5, 2, 8, 12, 8, 10]   \n",
              "773508  .280+417=796  280+417=796E  [0, 3, 9, 1, 11, 5, 2, 8, 12, 8, 10, 7]   \n",
              "\n",
              "                                  target_encoded  \n",
              "0         [0, 0, 0, 0, 0, 0, 0, 8, 11, 6, 12, 3]  \n",
              "1         [0, 0, 0, 0, 0, 0, 8, 11, 6, 12, 3, 2]  \n",
              "2        [0, 0, 0, 0, 0, 8, 11, 6, 12, 3, 2, 13]  \n",
              "3         [0, 0, 0, 0, 0, 3, 6, 11, 6, 4, 12, 9]  \n",
              "4         [0, 0, 0, 0, 3, 6, 11, 6, 4, 12, 9, 8]  \n",
              "...                                          ...  \n",
              "773492  [3, 10, 9, 11, 6, 1, 3, 12, 1, 1, 9, 13]  \n",
              "773505    [0, 0, 0, 3, 9, 1, 11, 5, 2, 8, 12, 8]  \n",
              "773506   [0, 0, 3, 9, 1, 11, 5, 2, 8, 12, 8, 10]  \n",
              "773507   [0, 3, 9, 1, 11, 5, 2, 8, 12, 8, 10, 7]  \n",
              "773508  [3, 9, 1, 11, 5, 2, 8, 12, 8, 10, 7, 13]  \n",
              "\n",
              "[179125 rows x 4 columns]"
            ]
          },
          "execution_count": 31,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "dataset['context_encoded'] = dataset['context'].apply(lambda x: [stoi[i] for i in x])\n",
        "dataset['target_encoded'] = dataset['target'].apply(lambda x: [stoi[i] for i in x])\n",
        "dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TFpzD85D4Q-M"
      },
      "outputs": [],
      "source": [
        "train_dataset = TensorDataset(torch.tensor(np.stack(dataset['context_encoded'].values)[:int(len(dataset) * 0.8)]),\n",
        "                              torch.tensor(np.stack(dataset['target_encoded'].values)[:int(len(dataset) * 0.8)]))\n",
        "\n",
        "val_dataset = TensorDataset(torch.tensor(np.stack(dataset['context_encoded'].values)[int(len(dataset) * 0.8):]),\n",
        "                            torch.tensor(np.stack(dataset['target_encoded'].values)[int(len(dataset) * 0.8):]))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aeHG_kK64-Ci",
        "outputId": "04b23ac5-3297-416d-a6bf-b8d61b57b738"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<torch._C.Generator at 0x78bbc43c9610>"
            ]
          },
          "execution_count": 33,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# hyperparameters\n",
        "batch_size = 64\n",
        "block_size = 12\n",
        "max_iters = 5000\n",
        "eval_interval = 500\n",
        "learning_rate = 3e-4\n",
        "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
        "eval_iters = 200\n",
        "n_embd = 384\n",
        "n_head = 6 # D = 384 // 6\n",
        "n_layer = 6\n",
        "dropout = 0.2\n",
        "# ------------\n",
        "\n",
        "torch.manual_seed(1337)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-VDMOur65JeS"
      },
      "outputs": [],
      "source": [
        "model = GPTLanguageModel().to(device)\n",
        "m = model.to(device)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tfRz5f2v9APo",
        "outputId": "e624d2d3-019a-47be-9525-33df51978d76"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "0/5000. Train loss: 2.2926676273345947 , Valid loss: 2.0744266510009766\n",
            "50/5000. Train loss: 1.5435444116592407 , Valid loss: 1.4898895025253296\n",
            "100/5000. Train loss: 0.9131597876548767 , Valid loss: 0.7749669551849365\n",
            "150/5000. Train loss: 0.8540360927581787 , Valid loss: 0.6814400553703308\n",
            "200/5000. Train loss: 0.7346890568733215 , Valid loss: 0.6416792869567871\n",
            "250/5000. Train loss: 0.6901510953903198 , Valid loss: 0.5935299396514893\n",
            "300/5000. Train loss: 0.6483660936355591 , Valid loss: 0.6080003380775452\n",
            "350/5000. Train loss: 0.6532577872276306 , Valid loss: 0.5704285502433777\n",
            "400/5000. Train loss: 0.635216236114502 , Valid loss: 0.545965850353241\n",
            "450/5000. Train loss: 0.6237342357635498 , Valid loss: 0.544535219669342\n",
            "500/5000. Train loss: 0.6006761789321899 , Valid loss: 0.5591723918914795\n",
            "550/5000. Train loss: 0.5621727705001831 , Valid loss: 0.5241012573242188\n",
            "600/5000. Train loss: 0.5966814160346985 , Valid loss: 0.5412322282791138\n",
            "650/5000. Train loss: 0.5804938673973083 , Valid loss: 0.5401427745819092\n"
          ]
        }
      ],
      "source": [
        "try:\n",
        "  optimizer = torch.optim.AdamW(model.parameters(), lr = learning_rate)\n",
        "\n",
        "  for iter in range(max_iters):\n",
        "\n",
        "      train_loss, val_loss = 0, 0\n",
        "\n",
        "      ''' Если прогонять каждую эпоху весь датасет, это будет очень долго.\n",
        "       Поэтому в каждой эпохе шаффлю лоадер (чтобы модель видела разные данные) и беру первые 50 батчей на обучение и вторые 50 на валидацию.'''\n",
        "      batches = 0\n",
        "      train_loader = DataLoader(train_dataset, batch_size = 64, shuffle = True)\n",
        "      val_loader = DataLoader(val_dataset, batch_size = 64, shuffle = True)\n",
        "\n",
        "      for context, target in train_loader:\n",
        "\n",
        "          context, target = context.to(device), target.to(device)\n",
        "          logits, _ = model(context)\n",
        "\n",
        "          B, T, C = logits.shape\n",
        "          logits = logits.view(B * T, C)\n",
        "\n",
        "          '''Нам не нужно в таргете предсказывать сам пример, то есть всё перед знаком = включительно.\n",
        "           Создадим маску, которая будет занулять (а 0 в stoi это точка-паддинг) лишнюю часть и в кросс-энтропии будем игнорировать этот паддинг'''\n",
        "          mask = torch.zeros_like(target, dtype = torch.bool, device=device)\n",
        "          for i in range(target.size(0)):\n",
        "            index_eq = ((target[i] == 12).int()).nonzero(as_tuple=True)[0] # stoi['='] = 12\n",
        "            mask[i, :index_eq + 1] = True\n",
        "            masked_targets = target.masked_fill(mask, 0)\n",
        "            masked_targets = masked_targets.view(B * T)\n",
        "\n",
        "            loss = F.cross_entropy(logits, masked_targets, ignore_index = 0)\n",
        "\n",
        "          train_loss += loss\n",
        "\n",
        "          optimizer.zero_grad(set_to_none = True)\n",
        "          loss.backward()\n",
        "          optimizer.step()\n",
        "\n",
        "          batches += 1\n",
        "          if batches > 50:\n",
        "            break\n",
        "\n",
        "      for context, target in val_loader:\n",
        "          context, target = context.to(device), target.to(device)\n",
        "          with torch.no_grad():\n",
        "              logits, _ = model(context)\n",
        "              B, T, C = logits.shape\n",
        "\n",
        "              logits = logits.view(B * T, C)\n",
        "\n",
        "              mask = torch.zeros_like(target, dtype = torch.bool, device=device)\n",
        "              for i in range(target.size(0)):\n",
        "                index_eq = (target[i] == 12).nonzero(as_tuple=True)[0]\n",
        "                mask[i, :index_eq + 1] = True\n",
        "                masked_targets = target.masked_fill(mask, 0)\n",
        "                masked_targets = masked_targets.view(B * T)\n",
        "\n",
        "                loss = F.cross_entropy(logits, masked_targets, ignore_index = 0)\n",
        "\n",
        "              val_loss += loss\n",
        "\n",
        "          batches += 1\n",
        "          if batches > 100:\n",
        "            break\n",
        "\n",
        "      if iter % 50 == 0:\n",
        "        print(f'{iter}/{max_iters}. Train loss: {train_loss / 50} , Valid loss: {val_loss / 50}')\n",
        "except KeyboardInterrupt: 0"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Zg7fBcxEzZ1W"
      },
      "outputs": [],
      "source": [
        "def inference(sample):\n",
        "  samle_str = '.' * (12 - len(sample)) + sample\n",
        "  context = torch.tensor([encode(samle_str)], dtype = torch.int32)\n",
        "  context = context.to(device)\n",
        "  res = decode(m.generate(context, max_new_tokens = 15)[0].tolist()).replace('.', '')\n",
        "  res = res[res.index('=') + 1 :res.index('E')]\n",
        "  return res"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mWacGKuDXhxH"
      },
      "source": [
        "### Сгенерируем по 100 наборов сложения одно, двух, трехзначных чисел. График accuracy\n",
        "\n",
        "Интересный момент, что точность на двухзначных числах больше"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 448
        },
        "id": "JtabycfaBdzi",
        "outputId": "938b8b68-9f17-4f5a-9f59-5ceebb40fa91"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<BarContainer object of 3 artists>"
            ]
          },
          "execution_count": 36,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAje0lEQVR4nO3de1Bc5cHH8R9LZDeIEBPMktA1OzFqghqwIIiXms6s4qUZ43jB1AruKB0vtJnu6BiaBEy8bNRIsRVF06CtlwnVRlsnKW3ckZlqsFTQVtsYq20CRncDathIptCy5/2j0827DZAcID5Cvp+ZM5M9+5xznh2P5JuzZ9kky7IsAQAAGOIwPQEAAHB0I0YAAIBRxAgAADCKGAEAAEYRIwAAwChiBAAAGEWMAAAAo4gRAABg1BTTEzgcsVhMH3/8sY477jglJSWZng4AADgMlmVp3759mj17thyO4a9/TIgY+fjjj+XxeExPAwAAjEJXV5e+9rWvDfv8hIiR4447TtJ/Xkx6errh2QAAgMMRjUbl8Xjif48PZ0LEyH/fmklPTydGAACYYA51iwU3sAIAAKOIEQAAYBQxAgAAjCJGAACAUcQIAAAwihgBAABGESMAAMAoYgQAABhFjAAAAKOIEQAAYBQxAgAAjCJGAACAUcQIAAAwihgBAABGTTE9AeBo512+2fQUYNjOtZeZngJgFFdGAACAUcQIAAAwihgBAABGESMAAMAoYgQAABhFjAAAAKOIEQAAYBQxAgAAjCJGAACAUcQIAAAwihgBAABGESMAAMCoUcVIfX29vF6vXC6XioqK1NbWNuL4uro6nXrqqZo6dao8Ho9+8IMf6J///OeoJgwAACYX2zHS1NSkQCCgmpoadXR0KDc3VyUlJdqzZ8+Q45977jktX75cNTU12r59uzZs2KCmpib98Ic/HPPkAQDAxGc7Rmpra1VRUSG/36+cnBw1NDQoNTVVjY2NQ47ftm2bzj33XH3729+W1+vVRRddpKVLlx7yagoAADg62IqRgYEBtbe3y+fzHdiBwyGfz6fW1tYhtznnnHPU3t4ej4+///3v2rJliy699NJhj9Pf369oNJqwAACAyWmKncE9PT0aHByU2+1OWO92u/Xee+8Nuc23v/1t9fT06LzzzpNlWfr3v/+tm2++ecS3aYLBoFavXm1nagAAYII64p+maWlp0X333adHH31UHR0d2rRpkzZv3qy777572G2qqqrU29sbX7q6uo70NAEAgCG2roxkZmYqOTlZkUgkYX0kElFWVtaQ26xatUrXX3+9brrpJknSGWecob6+Pn33u9/VihUr5HAc3ENOp1NOp9PO1AAAwARl68pISkqK8vPzFQqF4utisZhCoZCKi4uH3Gb//v0HBUdycrIkybIsu/MFAACTjK0rI5IUCARUXl6ugoICFRYWqq6uTn19ffL7/ZKksrIyZWdnKxgMSpIWL16s2tpanXnmmSoqKtIHH3ygVatWafHixfEoAQAARy/bMVJaWqru7m5VV1crHA4rLy9Pzc3N8ZtaOzs7E66ErFy5UklJSVq5cqV2796tE044QYsXL9a99947fq8CAABMWEnWBHivJBqNKiMjQ729vUpPTzc9HWBceZdvNj0FGLZz7WWmpwAcEYf79zffTQMAAIwiRgAAgFHECAAAMIoYAQAARhEjAADAKGIEAAAYRYwAAACjiBEAAGAUMQIAAIwiRgAAgFHECAAAMIoYAQAARhEjAADAKGIEAAAYRYwAAACjiBEAAGAUMQIAAIwiRgAAgFHECAAAMIoYAQAARhEjAADAKGIEAAAYRYwAAACjiBEAAGAUMQIAAIwiRgAAgFHECAAAMIoYAQAARhEjAADAKGIEAAAYRYwAAACjRhUj9fX18nq9crlcKioqUltb27BjFy1apKSkpIOWyy67bNSTBgAAk4ftGGlqalIgEFBNTY06OjqUm5urkpIS7dmzZ8jxmzZt0ieffBJf3n33XSUnJ+vqq68e8+QBAMDEZztGamtrVVFRIb/fr5ycHDU0NCg1NVWNjY1Djp8+fbqysrLiy9atW5WamkqMAAAASTZjZGBgQO3t7fL5fAd24HDI5/OptbX1sPaxYcMGXXvttTr22GOHHdPf369oNJqwAACAyclWjPT09GhwcFButzthvdvtVjgcPuT2bW1tevfdd3XTTTeNOC4YDCojIyO+eDweO9MEAAATyJf6aZoNGzbojDPOUGFh4Yjjqqqq1NvbG1+6urq+pBkCAIAv2xQ7gzMzM5WcnKxIJJKwPhKJKCsra8Rt+/r6tHHjRq1Zs+aQx3E6nXI6nXamBgAAJihbV0ZSUlKUn5+vUCgUXxeLxRQKhVRcXDzits8//7z6+/v1ne98Z3QzBQAAk5KtKyOSFAgEVF5eroKCAhUWFqqurk59fX3y+/2SpLKyMmVnZysYDCZst2HDBi1ZskQzZswYn5kDAIBJwXaMlJaWqru7W9XV1QqHw8rLy1Nzc3P8ptbOzk45HIkXXHbs2KHXXntNv/vd78Zn1gAAYNJIsizLMj2JQ4lGo8rIyFBvb6/S09NNTwcYV97lm01PAYbtXMtvpMbkdLh/f/PdNAAAwChiBAAAGEWMAAAAo4gRAABgFDECAACMIkYAAIBRtn/PyGTDxyrBxyoBwCyujAAAAKOIEQAAYBQxAgAAjCJGAACAUcQIAAAwihgBAABGESMAAMAoYgQAABhFjAAAAKOIEQAAYBQxAgAAjCJGAACAUcQIAAAwihgBAABGESMAAMAoYgQAABhFjAAAAKOIEQAAYBQxAgAAjCJGAACAUcQIAAAwihgBAABGESMAAMAoYgQAABg1qhipr6+X1+uVy+VSUVGR2traRhy/d+9e3XbbbZo1a5acTqdOOeUUbdmyZVQTBgAAk8sUuxs0NTUpEAiooaFBRUVFqqurU0lJiXbs2KGZM2ceNH5gYEAXXnihZs6cqRdeeEHZ2dnatWuXpk2bNh7zBwAAE5ztGKmtrVVFRYX8fr8kqaGhQZs3b1ZjY6OWL19+0PjGxkZ99tln2rZtm4455hhJktfrHdusAQDApGHrbZqBgQG1t7fL5/Md2IHDIZ/Pp9bW1iG3+fWvf63i4mLddtttcrvdOv3003XfffdpcHBw2OP09/crGo0mLAAAYHKyFSM9PT0aHByU2+1OWO92uxUOh4fc5u9//7teeOEFDQ4OasuWLVq1apUeeugh3XPPPcMeJxgMKiMjI754PB470wQAABPIEf80TSwW08yZM/XEE08oPz9fpaWlWrFihRoaGobdpqqqSr29vfGlq6vrSE8TAAAYYuuekczMTCUnJysSiSSsj0QiysrKGnKbWbNm6ZhjjlFycnJ83YIFCxQOhzUwMKCUlJSDtnE6nXI6nXamBgAAJihbV0ZSUlKUn5+vUCgUXxeLxRQKhVRcXDzkNueee64++OADxWKx+Lr3339fs2bNGjJEAADA0cX22zSBQEDr16/Xz372M23fvl233HKL+vr64p+uKSsrU1VVVXz8Lbfcos8++0zLli3T+++/r82bN+u+++7TbbfdNn6vAgAATFi2P9pbWlqq7u5uVVdXKxwOKy8vT83NzfGbWjs7O+VwHGgcj8ej3/72t/rBD36ghQsXKjs7W8uWLdOdd945fq8CAABMWLZjRJIqKytVWVk55HMtLS0HrSsuLtYbb7wxmkMBAIBJju+mAQAARhEjAADAKGIEAAAYRYwAAACjiBEAAGAUMQIAAIwiRgAAgFHECAAAMIoYAQAARhEjAADAKGIEAAAYRYwAAACjiBEAAGAUMQIAAIwiRgAAgFHECAAAMIoYAQAARhEjAADAKGIEAAAYRYwAAACjiBEAAGAUMQIAAIwiRgAAgFHECAAAMIoYAQAARhEjAADAKGIEAAAYRYwAAACjiBEAAGAUMQIAAIwiRgAAgFGjipH6+np5vV65XC4VFRWpra1t2LFPPfWUkpKSEhaXyzXqCQMAgMnFdow0NTUpEAiopqZGHR0dys3NVUlJifbs2TPsNunp6frkk0/iy65du8Y0aQAAMHnYjpHa2lpVVFTI7/crJydHDQ0NSk1NVWNj47DbJCUlKSsrK7643e4xTRoAAEwetmJkYGBA7e3t8vl8B3bgcMjn86m1tXXY7b744gvNmTNHHo9Hl19+uf7yl7+MeJz+/n5Fo9GEBQAATE62YqSnp0eDg4MHXdlwu90Kh8NDbnPqqaeqsbFRv/rVr/TMM88oFovpnHPO0UcffTTscYLBoDIyMuKLx+OxM00AADCBHPFP0xQXF6usrEx5eXm64IILtGnTJp1wwgl6/PHHh92mqqpKvb298aWrq+tITxMAABgyxc7gzMxMJScnKxKJJKyPRCLKyso6rH0cc8wxOvPMM/XBBx8MO8bpdMrpdNqZGgAAmKBsXRlJSUlRfn6+QqFQfF0sFlMoFFJxcfFh7WNwcFDvvPOOZs2aZW+mAABgUrJ1ZUSSAoGAysvLVVBQoMLCQtXV1amvr09+v1+SVFZWpuzsbAWDQUnSmjVrdPbZZ2vevHnau3evHnzwQe3atUs33XTT+L4SAAAwIdmOkdLSUnV3d6u6ulrhcFh5eXlqbm6O39Ta2dkph+PABZfPP/9cFRUVCofDOv7445Wfn69t27YpJydn/F4FAACYsJIsy7JMT+JQotGoMjIy1Nvbq/T09HHdt3f55nHdHyaenWsvM3p8zkGYPgeBI+Vw//7mu2kAAIBRxAgAADCKGAEAAEYRIwAAwChiBAAAGEWMAAAAo4gRAABgFDECAACMIkYAAIBRxAgAADCKGAEAAEYRIwAAwChiBAAAGEWMAAAAo4gRAABgFDECAACMIkYAAIBRxAgAADCKGAEAAEYRIwAAwChiBAAAGEWMAAAAo4gRAABgFDECAACMIkYAAIBRxAgAADBqiukJAADM8i7fbHoKMGzn2suMHp8rIwAAwChiBAAAGEWMAAAAo0YVI/X19fJ6vXK5XCoqKlJbW9thbbdx40YlJSVpyZIlozksAACYhGzHSFNTkwKBgGpqatTR0aHc3FyVlJRoz549I263c+dO3X777Tr//PNHPVkAADD52I6R2tpaVVRUyO/3KycnRw0NDUpNTVVjY+Ow2wwODuq6667T6tWrNXfu3DFNGAAATC62YmRgYEDt7e3y+XwHduBwyOfzqbW1ddjt1qxZo5kzZ+rGG28c/UwBAMCkZOv3jPT09GhwcFButzthvdvt1nvvvTfkNq+99po2bNigt99++7CP09/fr/7+/vjjaDRqZ5oAAGACOaKfptm3b5+uv/56rV+/XpmZmYe9XTAYVEZGRnzxeDxHcJYAAMAkW1dGMjMzlZycrEgkkrA+EokoKyvroPEffvihdu7cqcWLF8fXxWKx/xx4yhTt2LFDJ5100kHbVVVVKRAIxB9Ho1GCBACAScpWjKSkpCg/P1+hUCj+8dxYLKZQKKTKysqDxs+fP1/vvPNOwrqVK1dq3759evjhh4cNDKfTKafTaWdqAABggrL93TSBQEDl5eUqKChQYWGh6urq1NfXJ7/fL0kqKytTdna2gsGgXC6XTj/99ITtp02bJkkHrQcAAEcn2zFSWlqq7u5uVVdXKxwOKy8vT83NzfGbWjs7O+Vw8ItdAQDA4RnVt/ZWVlYO+baMJLW0tIy47VNPPTWaQwIAgEmKSxgAAMAoYgQAABhFjAAAAKOIEQAAYBQxAgAAjCJGAACAUcQIAAAwihgBAABGESMAAMAoYgQAABhFjAAAAKOIEQAAYBQxAgAAjCJGAACAUcQIAAAwihgBAABGESMAAMAoYgQAABhFjAAAAKOIEQAAYBQxAgAAjCJGAACAUcQIAAAwihgBAABGESMAAMAoYgQAABhFjAAAAKOIEQAAYBQxAgAAjCJGAACAUcQIAAAwalQxUl9fL6/XK5fLpaKiIrW1tQ07dtOmTSooKNC0adN07LHHKi8vT08//fSoJwwAACYX2zHS1NSkQCCgmpoadXR0KDc3VyUlJdqzZ8+Q46dPn64VK1aotbVVf/7zn+X3++X3+/Xb3/52zJMHAAATn+0Yqa2tVUVFhfx+v3JyctTQ0KDU1FQ1NjYOOX7RokW64oortGDBAp100klatmyZFi5cqNdee23MkwcAABOfrRgZGBhQe3u7fD7fgR04HPL5fGptbT3k9pZlKRQKaceOHfrGN75hf7YAAGDSmWJncE9PjwYHB+V2uxPWu91uvffee8Nu19vbq+zsbPX39ys5OVmPPvqoLrzwwmHH9/f3q7+/P/44Go3amSYAAJhAbMXIaB133HF6++239cUXXygUCikQCGju3LlatGjRkOODwaBWr179ZUwNAAAYZitGMjMzlZycrEgkkrA+EokoKytr2O0cDofmzZsnScrLy9P27dsVDAaHjZGqqioFAoH442g0Ko/HY2eqAABggrB1z0hKSory8/MVCoXi62KxmEKhkIqLiw97P7FYLOFtmP/ldDqVnp6esAAAgMnJ9ts0gUBA5eXlKigoUGFhoerq6tTX1ye/3y9JKisrU3Z2toLBoKT/vOVSUFCgk046Sf39/dqyZYuefvppPfbYY+P7SgAAwIRkO0ZKS0vV3d2t6upqhcNh5eXlqbm5OX5Ta2dnpxyOAxdc+vr6dOutt+qjjz7S1KlTNX/+fD3zzDMqLS0dv1cBAAAmrFHdwFpZWanKysohn2tpaUl4fM899+iee+4ZzWEAAMBRgO+mAQAARhEjAADAKGIEAAAYRYwAAACjiBEAAGAUMQIAAIwiRgAAgFHECAAAMIoYAQAARhEjAADAKGIEAAAYRYwAAACjiBEAAGAUMQIAAIwiRgAAgFHECAAAMIoYAQAARhEjAADAKGIEAAAYRYwAAACjiBEAAGAUMQIAAIwiRgAAgFHECAAAMIoYAQAARhEjAADAKGIEAAAYRYwAAACjiBEAAGAUMQIAAIwiRgAAgFGjipH6+np5vV65XC4VFRWpra1t2LHr16/X+eefr+OPP17HH3+8fD7fiOMBAMDRxXaMNDU1KRAIqKamRh0dHcrNzVVJSYn27Nkz5PiWlhYtXbpUr776qlpbW+XxeHTRRRdp9+7dY548AACY+GzHSG1trSoqKuT3+5WTk6OGhgalpqaqsbFxyPHPPvusbr31VuXl5Wn+/Pn66U9/qlgsplAoNObJAwCAic9WjAwMDKi9vV0+n+/ADhwO+Xw+tba2HtY+9u/fr3/961+aPn36sGP6+/sVjUYTFgAAMDnZipGenh4NDg7K7XYnrHe73QqHw4e1jzvvvFOzZ89OCJr/FQwGlZGREV88Ho+daQIAgAnkS/00zdq1a7Vx40a9+OKLcrlcw46rqqpSb29vfOnq6voSZwkAAL5MU+wMzszMVHJysiKRSML6SCSirKysEbddt26d1q5dq1deeUULFy4ccazT6ZTT6bQzNQAAMEHZujKSkpKi/Pz8hJtP/3szanFx8bDbPfDAA7r77rvV3NysgoKC0c8WAABMOraujEhSIBBQeXm5CgoKVFhYqLq6OvX19cnv90uSysrKlJ2drWAwKEm6//77VV1dreeee05erzd+b0laWprS0tLG8aUAAICJyHaMlJaWqru7W9XV1QqHw8rLy1Nzc3P8ptbOzk45HAcuuDz22GMaGBjQVVddlbCfmpoa3XXXXWObPQAAmPBsx4gkVVZWqrKycsjnWlpaEh7v3LlzNIcAAABHCb6bBgAAGEWMAAAAo4gRAABgFDECAACMIkYAAIBRxAgAADCKGAEAAEYRIwAAwChiBAAAGEWMAAAAo4gRAABgFDECAACMIkYAAIBRxAgAADCKGAEAAEYRIwAAwChiBAAAGEWMAAAAo4gRAABgFDECAACMIkYAAIBRxAgAADCKGAEAAEYRIwAAwChiBAAAGEWMAAAAo4gRAABgFDECAACMIkYAAIBRxAgAADCKGAEAAEaNKkbq6+vl9XrlcrlUVFSktra2Ycf+5S9/0ZVXXimv16ukpCTV1dWNdq4AAGASsh0jTU1NCgQCqqmpUUdHh3Jzc1VSUqI9e/YMOX7//v2aO3eu1q5dq6ysrDFPGAAATC62Y6S2tlYVFRXy+/3KyclRQ0ODUlNT1djYOOT4s846Sw8++KCuvfZaOZ3OMU8YAABMLrZiZGBgQO3t7fL5fAd24HDI5/OptbV13CbV39+vaDSasAAAgMnJVoz09PRocHBQbrc7Yb3b7VY4HB63SQWDQWVkZMQXj8czbvsGAABfLV/JT9NUVVWpt7c3vnR1dZmeEgAAOEKm2BmcmZmp5ORkRSKRhPWRSGRcb051Op3cXwIAwFHC1pWRlJQU5efnKxQKxdfFYjGFQiEVFxeP++QAAMDkZ+vKiCQFAgGVl5eroKBAhYWFqqurU19fn/x+vySprKxM2dnZCgaDkv5z0+tf//rX+J93796tt99+W2lpaZo3b944vhQAADAR2Y6R0tJSdXd3q7q6WuFwWHl5eWpubo7f1NrZ2SmH48AFl48//lhnnnlm/PG6deu0bt06XXDBBWppaRn7KwAAABOa7RiRpMrKSlVWVg753P8GhtfrlWVZozkMAAA4CnwlP00DAACOHsQIAAAwihgBAABGESMAAMAoYgQAABhFjAAAAKOIEQAAYBQxAgAAjCJGAACAUcQIAAAwihgBAABGESMAAMAoYgQAABhFjAAAAKOIEQAAYBQxAgAAjCJGAACAUcQIAAAwihgBAABGESMAAMAoYgQAABhFjAAAAKOIEQAAYBQxAgAAjCJGAACAUcQIAAAwihgBAABGESMAAMAoYgQAABhFjAAAAKOIEQAAYNSoYqS+vl5er1cul0tFRUVqa2sbcfzzzz+v+fPny+Vy6YwzztCWLVtGNVkAADD52I6RpqYmBQIB1dTUqKOjQ7m5uSopKdGePXuGHL9t2zYtXbpUN954o9566y0tWbJES5Ys0bvvvjvmyQMAgInPdozU1taqoqJCfr9fOTk5amhoUGpqqhobG4cc//DDD+viiy/WHXfcoQULFujuu+/W17/+dT3yyCNjnjwAAJj4ptgZPDAwoPb2dlVVVcXXORwO+Xw+tba2DrlNa2urAoFAwrqSkhK99NJLwx6nv79f/f398ce9vb2SpGg0ame6hyXWv3/c94mJ5UicV3ZwDoJzEKYdqXPwv/u1LGvEcbZipKenR4ODg3K73Qnr3W633nvvvSG3CYfDQ44Ph8PDHicYDGr16tUHrfd4PHamCxyWjDrTM8DRjnMQph3pc3Dfvn3KyMgY9nlbMfJlqaqqSriaEovF9Nlnn2nGjBlKSkoyOLPJJxqNyuPxqKurS+np6aang6MQ5yBM4xw8cizL0r59+zR79uwRx9mKkczMTCUnJysSiSSsj0QiysrKGnKbrKwsW+Mlyel0yul0JqybNm2ananCpvT0dP4nhFGcgzCNc/DIGOmKyH/ZuoE1JSVF+fn5CoVC8XWxWEyhUEjFxcVDblNcXJwwXpK2bt067HgAAHB0sf02TSAQUHl5uQoKClRYWKi6ujr19fXJ7/dLksrKypSdna1gMChJWrZsmS644AI99NBDuuyyy7Rx40a9+eabeuKJJ8b3lQAAgAnJdoyUlpaqu7tb1dXVCofDysvLU3Nzc/wm1c7OTjkcBy64nHPOOXruuee0cuVK/fCHP9TJJ5+sl156Saeffvr4vQqMmtPpVE1NzUFviwFfFs5BmMY5aF6SdajP2wAAABxBfDcNAAAwihgBAABGESMAAMAoYgTDuuuuu5SXl2d6GjiKeb1e1dXVjTgmKSlpxK+XAMaipaVFSUlJ2rt377BjnnrqqUP+Lix+no6MGAEAAEYRI0exnTt38uv1YdRdd92lG264wfQ0cJTzer1qaWkxPY2jGjEyyTQ3N+u8887TtGnTNGPGDH3rW9/Shx9+OKZ9Pv744/J4PEpNTdU111wT/xZlHH0OdX7deeedOuWUU5Samqq5c+dq1apV+te//jWmY+7bt09Lly7Vscceq+zsbNXX1x805pNPPtEll1yiqVOnau7cuXrhhRfGdEyYM9Zz7MMPP9Tll18ut9uttLQ0nXXWWXrllVfGPK/XX39dCxculMvl0tlnn6133333oDEvvfSSTj75ZLlcLpWUlKirq+ugMfw8HRoxMsn09fUpEAjozTffVCgUksPh0BVXXKFYLDaq/X3wwQf6xS9+oZdfflnNzc166623dOutt47zrDFRHOr8Ou644/TUU0/pr3/9qx5++GGtX79eP/rRj8Z0zAcffFC5ubl66623tHz5ci1btkxbt25NGLNq1SpdeeWV+tOf/qTrrrtO1157rbZv3z6m48KMsZ5jX3zxhS699FKFQiG99dZbuvjii7V48WJ1dnaOaV533HGHHnroIf3xj3/UCSecoMWLFydE0P79+3Xvvffq5z//uV5//XXt3btX1157bcI++Hk6AguTWnd3tyXJeueddw567h//+Ic10ilQU1NjJScnWx999FF83W9+8xvL4XBYn3zyyRGZLyaWkc4vy7KsBx980MrPzx92+5qaGqu8vHzY5+fMmWNdfPHFCetKS0utSy65JP5YknXzzTcnjCkqKrJuueWWw3gF+Kob6zlmWZZ12mmnWT/5yU+GfX7OnDnWq6++OuRzr776qiXJ2rhxY3zdp59+ak2dOtVqamqyLMuynnzySUuS9cYbb8THbN++3ZJk/eEPf7Asi5+nh8KVkUnmb3/7m5YuXaq5c+cqPT1dXq9XkuL/KjjttNOUlpamtLQ0nXbaaZIUf5yWlqZLLrkkYX8nnniisrOz44+Li4sVi8W0Y8eOL+cF4SvlUOdXU1OTzj33XGVlZSktLU0rV65M+Bfp73//+4Tz7b777tOzzz6bsO7ZZ59NOOb/fqlmcXHxQVc9DmcMJoaxnmNffPGFbr/9di1YsEDTpk1TWlqatm/fnjDm5ptvTjjnOjs7dckllySs+1///xybPn26Tj311IRzbMqUKTrrrLPij+fPn69p06YljOHn6fBsfzcNvtoWL16sOXPmaP369Zo9e7ZisZhOP/10DQwMSJK2bNkSv7S4e/duLVq0SG+//XZ8+6lTp5qYNiaIkc6v1tZWXXfddVq9erVKSkqUkZGhjRs36qGHHopvX1BQkHC+/fjHP9bu3bt1//33x9f993uucHQa6zl2++23a+vWrVq3bp3mzZunqVOn6qqrror/DJSkNWvW6Pbbb48/XrRoke6//34VFRV9qa8VBxAjk8inn36qHTt2aP369Tr//PMlSa+99lrCmDlz5sT/PGXKf/7zz5s3b9h9dnZ26uOPP9bs2bMlSW+88YYcDodOPfXU8Z4+vuIOdX5t27ZNc+bM0YoVK+Lrdu3albCPqVOnJpxv06dPVzQaHfEcfOONNw56vGDBgoPWlZWVJTw+88wzbbw6fBWMxzn2+uuv64YbbtAVV1wh6T9XSnbu3JkwZubMmZo5c2b88ZQpU5SdnX3I8/DEE0+UJH3++ed6//33E87Df//733rzzTdVWFgoSdqxY4f27t2bMIafp8MjRiaR448/XjNmzNATTzyhWbNmqbOzU8uXLx/TPl0ul8rLy7Vu3TpFo1F9//vf1zXXXKOsrKxxmjUmikOdXyeffLI6Ozu1ceNGnXXWWdq8ebNefPHFMR/39ddf1wMPPKAlS5Zo69atev7557V58+aEMc8//7wKCgp03nnn6dlnn1VbW5s2bNgw5mPjyzUe59jJJ5+sTZs2afHixUpKStKqVatGfQP//7dmzRrNmDFDbrdbK1asUGZmppYsWRJ//phjjtH3vvc9/fjHP9aUKVNUWVmps88+Ox4nEj9PR2T6phWMr61bt1oLFiywnE6ntXDhQqulpcWSZL344osHjT2cG1hzc3OtRx991Jo9e7blcrmsq666yvrss8+O4CvAV9mhzq877rjDmjFjhpWWlmaVlpZaP/rRj6yMjIxh93c4N7CuXr3auvrqq63U1FQrKyvLevjhhxPGSLLq6+utCy+80HI6nZbX643fWIiJZ6zn2D/+8Q/rm9/8pjV16lTL4/FYjzzyiHXBBRdYy5YtG/aYh3MD68svv2yddtppVkpKilVYWGj96U9/io958sknrYyMDOuXv/ylNXfuXMvpdFo+n8/atWtXfAw/T0eWZFmWZTKGAADA0Y1P0wAAAKOIEQAAYBQxAgAAjCJGAACAUcQIAAAwihgBAABGESMAAMAoYgQAABhFjAAAAKOIEQAAYBQxAgAAjCJGAACAUf8HEbCd1GZWZWAAAAAASUVORK5CYII=",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "NUM_SAMPLES = 100\n",
        "\n",
        "def get_predictions(NUM_SAMPLES, x, y):\n",
        "  i = 0\n",
        "  accuracy = 0\n",
        "  while i < NUM_SAMPLES:\n",
        "    a, b = random.randint(x, y), random.randint(x, y)\n",
        "    if a + b >= 1000:\n",
        "      i += 1\n",
        "      continue\n",
        "\n",
        "    task = str(a) + '+' + str(b) + '='\n",
        "    ans = a + b\n",
        "    res = int(inference(task)[::-1])\n",
        "    accuracy += int(ans == res)\n",
        "    i += 1\n",
        "\n",
        "  return accuracy / NUM_SAMPLES\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "plt.bar(['a+b', 'aa+bb', 'aaa+bbb'], [get_predictions(NUM_SAMPLES, 0, 9), get_predictions(NUM_SAMPLES, 10, 99), get_predictions(NUM_SAMPLES, 100, 999)])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NSwDe3xVVLBd"
      },
      "source": [
        "# Калькулятор"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xHLDHjzWYBPn"
      },
      "source": [
        "Датасет по тому же принципу, что и для сложения"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "UUkmQqCyMNQ7",
        "outputId": "a13f2d43-8658-4b4c-cee5-0893b6a5c460"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"dataset\",\n  \"rows\": 54556,\n  \"fields\": [\n    {\n      \"column\": \"context\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 54556,\n        \"samples\": [\n          \".....98+68=66\",\n          \"....79*52=801\",\n          \".......39+27=\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"target\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 54556,\n        \"samples\": [\n          \"....98+68=661\",\n          \"...79*52=8014\",\n          \"......39+27=6\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe",
              "variable_name": "dataset"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-1eb3909f-47ad-434b-99be-0251f903edf7\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>context</th>\n",
              "      <th>target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>.........6+2=</td>\n",
              "      <td>........6+2=8</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>........6+2=8</td>\n",
              "      <td>.......6+2=8E</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>.........9-9=</td>\n",
              "      <td>........9-9=0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>........9-9=0</td>\n",
              "      <td>.......9-9=0E</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>.........9/9=</td>\n",
              "      <td>........9/9=1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1783558</th>\n",
              "      <td>.....54+19=37</td>\n",
              "      <td>....54+19=37E</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1842792</th>\n",
              "      <td>.......80+68=</td>\n",
              "      <td>......80+68=8</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1842793</th>\n",
              "      <td>......80+68=8</td>\n",
              "      <td>.....80+68=84</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1842794</th>\n",
              "      <td>.....80+68=84</td>\n",
              "      <td>....80+68=841</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1842795</th>\n",
              "      <td>....80+68=841</td>\n",
              "      <td>...80+68=841E</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>54556 rows × 2 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-1eb3909f-47ad-434b-99be-0251f903edf7')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-1eb3909f-47ad-434b-99be-0251f903edf7 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-1eb3909f-47ad-434b-99be-0251f903edf7');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-568f1005-664f-4073-9f55-547211bdc439\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-568f1005-664f-4073-9f55-547211bdc439')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-568f1005-664f-4073-9f55-547211bdc439 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "               context         target\n",
              "0        .........6+2=  ........6+2=8\n",
              "1        ........6+2=8  .......6+2=8E\n",
              "2        .........9-9=  ........9-9=0\n",
              "3        ........9-9=0  .......9-9=0E\n",
              "4        .........9/9=  ........9/9=1\n",
              "...                ...            ...\n",
              "1783558  .....54+19=37  ....54+19=37E\n",
              "1842792  .......80+68=  ......80+68=8\n",
              "1842793  ......80+68=8  .....80+68=84\n",
              "1842794  .....80+68=84  ....80+68=841\n",
              "1842795  ....80+68=841  ...80+68=841E\n",
              "\n",
              "[54556 rows x 2 columns]"
            ]
          },
          "execution_count": 8,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "def add_padding(context, max_len, pad_token = '.'):\n",
        "    return pad_token * (max_len - len(context)) + context\n",
        "\n",
        "NUM_SAMPLES = 700000\n",
        "context, target = [], []\n",
        "\n",
        "def create_sample(x, y, operation):\n",
        "  global context\n",
        "  global target\n",
        "\n",
        "  a, b = random.randint(x, y), random.randint(x, y)\n",
        "  if a < b: # Чтобы не возникало отрицательных чисел при вычитании :)\n",
        "    c = a\n",
        "    a = b\n",
        "    b = a\n",
        "\n",
        "  task = str(a) + operation + str(b) + '='\n",
        "  if operation == '+':\n",
        "    ans = str(a + b)[::-1]\n",
        "  if operation == '-':\n",
        "    ans = str(a - b)[::-1]\n",
        "  if operation == '*':\n",
        "    ans = str(a * b)[::-1]\n",
        "  if operation == '/':\n",
        "    if b == 0:\n",
        "      return\n",
        "    ans = str(a // b)[::-1] # Деление будет целочисленное\n",
        "\n",
        "  for i in range(len(ans)):\n",
        "    context += [add_padding(task + ans[:i], 13)]\n",
        "    target += [add_padding(task + ans[: i + 1], 13)]\n",
        "  context += [add_padding(task + ans, 13)]\n",
        "  target += [add_padding(task + ans + 'E', 13)]\n",
        "\n",
        "for i in range(NUM_SAMPLES):\n",
        "  create_sample(0, 9, '+')\n",
        "  create_sample(0, 9, '-')\n",
        "  create_sample(0, 9, '/')\n",
        "  create_sample(0, 9, '*')\n",
        "  create_sample(10, 99, '+')\n",
        "  create_sample(10, 99, '-')\n",
        "  create_sample(10, 99, '/')\n",
        "  create_sample(10, 99, '*')\n",
        "\n",
        "dataset = pd.DataFrame({'context': context, 'target': target}).drop_duplicates()\n",
        "dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HryznAzHT9qA",
        "outputId": "22816116-3d76-4505-c54c-593097d08e4f"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'.': 0,\n",
              " '0': 1,\n",
              " '1': 2,\n",
              " '2': 3,\n",
              " '3': 4,\n",
              " '4': 5,\n",
              " '5': 6,\n",
              " '6': 7,\n",
              " '7': 8,\n",
              " '8': 9,\n",
              " '9': 10,\n",
              " '+': 11,\n",
              " '-': 12,\n",
              " '*': 13,\n",
              " '/': 14,\n",
              " '=': 15,\n",
              " 'E': 16}"
            ]
          },
          "execution_count": 9,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "chars = ['.'] + [str(i) for i in range(0, 10)] + ['+', '-', '*', '/', '=', 'E']\n",
        "vocab_size = len(chars)\n",
        "stoi = { ch:i for i,ch in enumerate(chars) }\n",
        "itos = { i:ch for i,ch in enumerate(chars) }\n",
        "encode = lambda s: [stoi[c] for c in s]\n",
        "decode = lambda l: ''.join([itos[i] for i in l])\n",
        "stoi"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "rTg1I-GKUB8W",
        "outputId": "24c30925-9f76-42d5-f232-bc158430723c"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"dataset\",\n  \"rows\": 54556,\n  \"fields\": [\n    {\n      \"column\": \"context\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 54556,\n        \"samples\": [\n          \".....98+68=66\",\n          \"....79*52=801\",\n          \".......39+27=\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"target\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 54556,\n        \"samples\": [\n          \"....98+68=661\",\n          \"...79*52=8014\",\n          \"......39+27=6\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"context_encoded\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"target_encoded\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe",
              "variable_name": "dataset"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-babe946e-96c8-401c-b814-d00ed3e5ab17\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>context</th>\n",
              "      <th>target</th>\n",
              "      <th>context_encoded</th>\n",
              "      <th>target_encoded</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>.........6+2=</td>\n",
              "      <td>........6+2=8</td>\n",
              "      <td>[0, 0, 0, 0, 0, 0, 0, 0, 0, 7, 11, 3, 15]</td>\n",
              "      <td>[0, 0, 0, 0, 0, 0, 0, 0, 7, 11, 3, 15, 9]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>........6+2=8</td>\n",
              "      <td>.......6+2=8E</td>\n",
              "      <td>[0, 0, 0, 0, 0, 0, 0, 0, 7, 11, 3, 15, 9]</td>\n",
              "      <td>[0, 0, 0, 0, 0, 0, 0, 7, 11, 3, 15, 9, 16]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>.........9-9=</td>\n",
              "      <td>........9-9=0</td>\n",
              "      <td>[0, 0, 0, 0, 0, 0, 0, 0, 0, 10, 12, 10, 15]</td>\n",
              "      <td>[0, 0, 0, 0, 0, 0, 0, 0, 10, 12, 10, 15, 1]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>........9-9=0</td>\n",
              "      <td>.......9-9=0E</td>\n",
              "      <td>[0, 0, 0, 0, 0, 0, 0, 0, 10, 12, 10, 15, 1]</td>\n",
              "      <td>[0, 0, 0, 0, 0, 0, 0, 10, 12, 10, 15, 1, 16]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>.........9/9=</td>\n",
              "      <td>........9/9=1</td>\n",
              "      <td>[0, 0, 0, 0, 0, 0, 0, 0, 0, 10, 14, 10, 15]</td>\n",
              "      <td>[0, 0, 0, 0, 0, 0, 0, 0, 10, 14, 10, 15, 2]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1783558</th>\n",
              "      <td>.....54+19=37</td>\n",
              "      <td>....54+19=37E</td>\n",
              "      <td>[0, 0, 0, 0, 0, 6, 5, 11, 2, 10, 15, 4, 8]</td>\n",
              "      <td>[0, 0, 0, 0, 6, 5, 11, 2, 10, 15, 4, 8, 16]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1842792</th>\n",
              "      <td>.......80+68=</td>\n",
              "      <td>......80+68=8</td>\n",
              "      <td>[0, 0, 0, 0, 0, 0, 0, 9, 1, 11, 7, 9, 15]</td>\n",
              "      <td>[0, 0, 0, 0, 0, 0, 9, 1, 11, 7, 9, 15, 9]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1842793</th>\n",
              "      <td>......80+68=8</td>\n",
              "      <td>.....80+68=84</td>\n",
              "      <td>[0, 0, 0, 0, 0, 0, 9, 1, 11, 7, 9, 15, 9]</td>\n",
              "      <td>[0, 0, 0, 0, 0, 9, 1, 11, 7, 9, 15, 9, 5]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1842794</th>\n",
              "      <td>.....80+68=84</td>\n",
              "      <td>....80+68=841</td>\n",
              "      <td>[0, 0, 0, 0, 0, 9, 1, 11, 7, 9, 15, 9, 5]</td>\n",
              "      <td>[0, 0, 0, 0, 9, 1, 11, 7, 9, 15, 9, 5, 2]</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1842795</th>\n",
              "      <td>....80+68=841</td>\n",
              "      <td>...80+68=841E</td>\n",
              "      <td>[0, 0, 0, 0, 9, 1, 11, 7, 9, 15, 9, 5, 2]</td>\n",
              "      <td>[0, 0, 0, 9, 1, 11, 7, 9, 15, 9, 5, 2, 16]</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>54556 rows × 4 columns</p>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-babe946e-96c8-401c-b814-d00ed3e5ab17')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-babe946e-96c8-401c-b814-d00ed3e5ab17 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-babe946e-96c8-401c-b814-d00ed3e5ab17');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-7d9f06e4-95f6-4975-9143-4811c7c5384a\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-7d9f06e4-95f6-4975-9143-4811c7c5384a')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-7d9f06e4-95f6-4975-9143-4811c7c5384a button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "               context         target  \\\n",
              "0        .........6+2=  ........6+2=8   \n",
              "1        ........6+2=8  .......6+2=8E   \n",
              "2        .........9-9=  ........9-9=0   \n",
              "3        ........9-9=0  .......9-9=0E   \n",
              "4        .........9/9=  ........9/9=1   \n",
              "...                ...            ...   \n",
              "1783558  .....54+19=37  ....54+19=37E   \n",
              "1842792  .......80+68=  ......80+68=8   \n",
              "1842793  ......80+68=8  .....80+68=84   \n",
              "1842794  .....80+68=84  ....80+68=841   \n",
              "1842795  ....80+68=841  ...80+68=841E   \n",
              "\n",
              "                                     context_encoded  \\\n",
              "0          [0, 0, 0, 0, 0, 0, 0, 0, 0, 7, 11, 3, 15]   \n",
              "1          [0, 0, 0, 0, 0, 0, 0, 0, 7, 11, 3, 15, 9]   \n",
              "2        [0, 0, 0, 0, 0, 0, 0, 0, 0, 10, 12, 10, 15]   \n",
              "3        [0, 0, 0, 0, 0, 0, 0, 0, 10, 12, 10, 15, 1]   \n",
              "4        [0, 0, 0, 0, 0, 0, 0, 0, 0, 10, 14, 10, 15]   \n",
              "...                                              ...   \n",
              "1783558   [0, 0, 0, 0, 0, 6, 5, 11, 2, 10, 15, 4, 8]   \n",
              "1842792    [0, 0, 0, 0, 0, 0, 0, 9, 1, 11, 7, 9, 15]   \n",
              "1842793    [0, 0, 0, 0, 0, 0, 9, 1, 11, 7, 9, 15, 9]   \n",
              "1842794    [0, 0, 0, 0, 0, 9, 1, 11, 7, 9, 15, 9, 5]   \n",
              "1842795    [0, 0, 0, 0, 9, 1, 11, 7, 9, 15, 9, 5, 2]   \n",
              "\n",
              "                                       target_encoded  \n",
              "0           [0, 0, 0, 0, 0, 0, 0, 0, 7, 11, 3, 15, 9]  \n",
              "1          [0, 0, 0, 0, 0, 0, 0, 7, 11, 3, 15, 9, 16]  \n",
              "2         [0, 0, 0, 0, 0, 0, 0, 0, 10, 12, 10, 15, 1]  \n",
              "3        [0, 0, 0, 0, 0, 0, 0, 10, 12, 10, 15, 1, 16]  \n",
              "4         [0, 0, 0, 0, 0, 0, 0, 0, 10, 14, 10, 15, 2]  \n",
              "...                                               ...  \n",
              "1783558   [0, 0, 0, 0, 6, 5, 11, 2, 10, 15, 4, 8, 16]  \n",
              "1842792     [0, 0, 0, 0, 0, 0, 9, 1, 11, 7, 9, 15, 9]  \n",
              "1842793     [0, 0, 0, 0, 0, 9, 1, 11, 7, 9, 15, 9, 5]  \n",
              "1842794     [0, 0, 0, 0, 9, 1, 11, 7, 9, 15, 9, 5, 2]  \n",
              "1842795    [0, 0, 0, 9, 1, 11, 7, 9, 15, 9, 5, 2, 16]  \n",
              "\n",
              "[54556 rows x 4 columns]"
            ]
          },
          "execution_count": 10,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "dataset['context_encoded'] = dataset['context'].apply(lambda x: [stoi[i] for i in x])\n",
        "dataset['target_encoded'] = dataset['target'].apply(lambda x: [stoi[i] for i in x])\n",
        "dataset"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1Zfvtq-dUOPg"
      },
      "outputs": [],
      "source": [
        "train_dataset = TensorDataset(torch.tensor(np.stack(dataset['context_encoded'].values)[:int(len(dataset) * 0.8)]),\n",
        "                              torch.tensor(np.stack(dataset['target_encoded'].values)[:int(len(dataset) * 0.8)]))\n",
        "\n",
        "val_dataset = TensorDataset(torch.tensor(np.stack(dataset['context_encoded'].values)[int(len(dataset) * 0.8):]),\n",
        "                            torch.tensor(np.stack(dataset['target_encoded'].values)[int(len(dataset) * 0.8):]))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eIW_WIy6Uij1"
      },
      "outputs": [],
      "source": [
        "# hyperparameters\n",
        "batch_size = 64\n",
        "block_size = 13\n",
        "max_iters = 5000\n",
        "eval_interval = 500\n",
        "learning_rate = 3e-4\n",
        "device = 'cuda' if torch.cuda.is_available() else 'cpu'\n",
        "eval_iters = 200\n",
        "n_embd = 384\n",
        "n_head = 6 # D = 384 // 6\n",
        "n_layer = 6\n",
        "dropout = 0.2\n",
        "# ------------\n",
        "\n",
        "torch.manual_seed(1337)\n",
        "\n",
        "model = GPTLanguageModel().to(device)\n",
        "m = model.to(device)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vwTfwIeGUj4G",
        "outputId": "7217938f-0b12-42d9-dde7-e82f7745a8de"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "0/5000. Train loss: 2.2532362937927246 , Valid loss: 2.0120201110839844\n",
            "50/5000. Train loss: 1.3832560777664185 , Valid loss: 1.3684803247451782\n",
            "100/5000. Train loss: 1.0406014919281006 , Valid loss: 1.0072081089019775\n",
            "150/5000. Train loss: 0.6329367160797119 , Valid loss: 0.6519628763198853\n",
            "200/5000. Train loss: 0.5978934168815613 , Valid loss: 0.5829275250434875\n",
            "250/5000. Train loss: 0.5503133535385132 , Valid loss: 0.5501938462257385\n",
            "300/5000. Train loss: 0.5367287993431091 , Valid loss: 0.536878228187561\n",
            "350/5000. Train loss: 0.4794479012489319 , Valid loss: 0.5101238489151001\n",
            "400/5000. Train loss: 0.433556467294693 , Valid loss: 0.4591020345687866\n",
            "450/5000. Train loss: 0.39261892437934875 , Valid loss: 0.40951722860336304\n",
            "500/5000. Train loss: 0.37071171402931213 , Valid loss: 0.3846268057823181\n",
            "550/5000. Train loss: 0.3064945340156555 , Valid loss: 0.3440845310688019\n",
            "600/5000. Train loss: 0.2610361874103546 , Valid loss: 0.30954113602638245\n",
            "650/5000. Train loss: 0.23788310587406158 , Valid loss: 0.27103278040885925\n",
            "700/5000. Train loss: 0.21019317209720612 , Valid loss: 0.299137681722641\n",
            "750/5000. Train loss: 0.18781274557113647 , Valid loss: 0.2764294445514679\n",
            "800/5000. Train loss: 0.17534205317497253 , Valid loss: 0.2936801612377167\n",
            "850/5000. Train loss: 0.1570039838552475 , Valid loss: 0.30613014101982117\n"
          ]
        }
      ],
      "source": [
        "try:\n",
        "  optimizer = torch.optim.AdamW(model.parameters(), lr = learning_rate)\n",
        "\n",
        "  for iter in range(max_iters):\n",
        "\n",
        "      train_loss, val_loss = 0, 0\n",
        "      batches = 0\n",
        "      train_loader = DataLoader(train_dataset, batch_size = 64, shuffle = True)\n",
        "      val_loader = DataLoader(val_dataset, batch_size = 64, shuffle = True)\n",
        "\n",
        "      for context, target in train_loader:\n",
        "\n",
        "          context, target = context.to(device), target.to(device)\n",
        "          logits, _ = model(context)\n",
        "          B, T, C = logits.shape\n",
        "          logits = logits.view(B * T, C)\n",
        "\n",
        "          mask = torch.zeros_like(target, dtype = torch.bool, device=device)\n",
        "          for i in range(target.size(0)):\n",
        "            index_eq = ((target[i] == 15).int()).nonzero(as_tuple=True)[0] # stoi['='] = 15\n",
        "            mask[i, :index_eq + 1] = True\n",
        "            masked_targets = target.masked_fill(mask, 0)\n",
        "            masked_targets = masked_targets.view(B * T)\n",
        "\n",
        "            loss = F.cross_entropy(logits, masked_targets, ignore_index = 0)\n",
        "\n",
        "          train_loss += loss\n",
        "\n",
        "          optimizer.zero_grad(set_to_none = True)\n",
        "          loss.backward()\n",
        "          optimizer.step()\n",
        "\n",
        "          batches += 1\n",
        "          if batches > 50:\n",
        "            break\n",
        "\n",
        "      for context, target in val_loader:\n",
        "          context, target = context.to(device), target.to(device)\n",
        "          with torch.no_grad():\n",
        "              logits, _ = model(context)\n",
        "              B, T, C = logits.shape\n",
        "\n",
        "              logits = logits.view(B * T, C)\n",
        "\n",
        "              mask = torch.zeros_like(target, dtype = torch.bool, device=device)\n",
        "              for i in range(target.size(0)):\n",
        "                index_eq = (target[i] == 15).nonzero(as_tuple=True)[0]\n",
        "                mask[i, :index_eq + 1] = True\n",
        "                masked_targets = target.masked_fill(mask, 0)\n",
        "                masked_targets = masked_targets.view(B * T)\n",
        "\n",
        "                loss = F.cross_entropy(logits, masked_targets, ignore_index = 0)\n",
        "\n",
        "              val_loss += loss\n",
        "\n",
        "          batches += 1\n",
        "          if batches > 100:\n",
        "            break\n",
        "\n",
        "      if iter % 50 == 0:\n",
        "        print(f'{iter}/{max_iters}. Train loss: {train_loss / 50} , Valid loss: {val_loss / 50}')\n",
        "except KeyboardInterrupt: 0"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OxO_P2KsYd3M"
      },
      "source": [
        "## График accuracy для калькулятора\n",
        "\n",
        "Точность для а+б возрасла более, чем на 0.1, в сравнении в результатами сложения. Точность аа+бб, наоборот, уменьшилась.\n",
        "\n",
        "Во всех экспериментах модели было сложнее всего выучить умножение двузначных чисел аа*бб, что вполне логично, так как и для нас на бумаге этот вид примера окажется сложнее, чем остальные представленные.\n",
        "\n",
        "В других экспериментах точность вычитания была чуть выше, но всё равно не доходила до сложения"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 448
        },
        "id": "1ImpNh2JUCtl",
        "outputId": "62346cae-974f-40ad-8637-4b58840f0730"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<BarContainer object of 8 artists>"
            ]
          },
          "execution_count": 14,
          "metadata": {},
          "output_type": "execute_result"
        },
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAhg0lEQVR4nO3de1RVZf7H8Q8XOYgKpiheIumipZnaT9MwK6ehtDEaZ02TWZPKNM5osrKolpoKmpNoJmFFMZmunJlMyzJrMMooauUlJy+tLt4LMQu8lRgZJDy/P1qd5gwXOSB8Bd+vtVirs9l7n+c5O/B99tmHE+CccwIAADASaD0AAABwZiNGAACAKWIEAACYIkYAAIApYgQAAJgiRgAAgCliBAAAmCJGAACAqWDrAdREeXm5vvrqK7Vq1UoBAQHWwwEAADXgnNOxY8fUqVMnBQZWff6jUcTIV199pejoaOthAACAWti3b5/OPvvsKr/fKGKkVatWkn6aTHh4uPFoAABATRQVFSk6Otr773hVGkWM/PzSTHh4ODECAEAjc7JLLLiAFQAAmCJGAACAKWIEAACYIkYAAIApYgQAAJgiRgAAgCliBAAAmCJGAACAKWIEAACYIkYAAIApYgQAAJgiRgAAgCliBAAAmCJGAACAqWDrAViLmZxlPYRay5szzHoIAADUGWdGAACAKWIEAACYIkYAAIApYgQAAJgiRgAAgCliBAAAmCJGAACAKWIEAACYIkYAAIApYgQAAJgiRgAAgCliBAAAmCJGAACAKWIEAACYIkYAAIApYgQAAJgiRgAAgCliBAAAmCJGAACAKWIEAACYIkYAAIApYgQAAJgiRgAAgCliBAAAmCJGAACAKWIEAACYIkYAAIApYgQAAJgiRgAAgCliBAAAmCJGAACAKWIEAACYIkYAAIApYgQAAJgiRgAAgCliBAAAmCJGAACAKWIEAACYqlWMZGRkKCYmRqGhoRowYIA2btxY7frp6em68MIL1bx5c0VHR+uee+7RDz/8UKsBAwCApsXvGFm+fLmSkpKUkpKizZs3q3fv3hoyZIgOHDhQ6fpLly7V5MmTlZKSom3btmnRokVavny5HnjggToPHgAANH5+x0haWprGjh2rhIQE9ejRQ5mZmQoLC9PixYsrXX/dunW64oordOuttyomJkbXXXedRo4cedKzKQAA4MzgV4yUlpZq06ZNiouL+2UHgYGKi4vT+vXrK91m4MCB2rRpkzc+Pv/8c61evVq/+c1vqryfkpISFRUV+XwBAICmKdiflQ8dOqSysjJFRUX5LI+KitL27dsr3ebWW2/VoUOHNGjQIDnndOLECY0bN67al2lSU1M1c+ZMf4YGAAAaqXp/N01ubq5mz56tJ598Ups3b9bLL7+srKwszZo1q8ptpkyZoqNHj3q/9u3bV9/DBAAARvw6MxIZGamgoCAVFhb6LC8sLFSHDh0q3Wb69Om6/fbb9ec//1mSdMkll6i4uFh/+ctfNHXqVAUGVuwhj8cjj8fjz9AAAEAj5deZkZCQEPXt21c5OTneZeXl5crJyVFsbGyl23z//fcVgiMoKEiS5Jzzd7wAAKCJ8evMiCQlJSVp9OjR6tevn/r376/09HQVFxcrISFBkjRq1Ch17txZqampkqT4+HilpaXp0ksv1YABA7R7925Nnz5d8fHx3igBAABnLr9jZMSIETp48KCSk5NVUFCgPn36KDs723tRa35+vs+ZkGnTpikgIEDTpk3T/v371a5dO8XHx+uhhx46dbMAAACNVoBrBK+VFBUVKSIiQkePHlV4ePgp3XfM5KxTur+GlDdnmPUQAACoUk3//eazaQAAgCliBAAAmCJGAACAKWIEAACYIkYAAIApYgQAAJgiRgAAgCliBAAAmCJGAACAKWIEAACYIkYAAIApYgQAAJgiRgAAgCliBAAAmCJGAACAKWIEAACYIkYAAIApYgQAAJgiRgAAgCliBAAAmCJGAACAKWIEAACYIkYAAIApYgQAAJgiRgAAgCliBAAAmCJGAACAKWIEAACYCrYeABpGzOQs6yHUSt6cYdZDAADUM86MAAAAU8QIAAAwRYwAAABTxAgAADBFjAAAAFPECAAAMEWMAAAAU8QIAAAwRYwAAABTxAgAADBFjAAAAFPECAAAMEWMAAAAU8QIAAAwRYwAAABTxAgAADBFjAAAAFPECAAAMEWMAAAAU8QIAAAwRYwAAABTxAgAADBFjAAAAFPECAAAMEWMAAAAU8QIAAAwRYwAAABTxAgAADBFjAAAAFPECAAAMEWMAAAAU8QIAAAwRYwAAABTxAgAADBFjAAAAFPECAAAMEWMAAAAU7WKkYyMDMXExCg0NFQDBgzQxo0bq13/22+/1YQJE9SxY0d5PB5169ZNq1evrtWAAQBA0xLs7wbLly9XUlKSMjMzNWDAAKWnp2vIkCHasWOH2rdvX2H90tJSXXvttWrfvr1WrFihzp07a+/evWrduvWpGD8AAGjk/I6RtLQ0jR07VgkJCZKkzMxMZWVlafHixZo8eXKF9RcvXqwjR45o3bp1atasmSQpJiambqMGAABNhl8v05SWlmrTpk2Ki4v7ZQeBgYqLi9P69esr3ebVV19VbGysJkyYoKioKPXs2VOzZ89WWVlZlfdTUlKioqIiny8AANA0+RUjhw4dUllZmaKionyWR0VFqaCgoNJtPv/8c61YsUJlZWVavXq1pk+frvnz5+tvf/tblfeTmpqqiIgI71d0dLQ/wwQAAI1Ivb+bpry8XO3bt9fTTz+tvn37asSIEZo6daoyMzOr3GbKlCk6evSo92vfvn31PUwAAGDEr2tGIiMjFRQUpMLCQp/lhYWF6tChQ6XbdOzYUc2aNVNQUJB3Wffu3VVQUKDS0lKFhIRU2Mbj8cjj8fgzNAAA0Ej5dWYkJCREffv2VU5OjndZeXm5cnJyFBsbW+k2V1xxhXbv3q3y8nLvsp07d6pjx46VhggAADiz+P0yTVJSkhYuXKglS5Zo27ZtGj9+vIqLi73vrhk1apSmTJniXX/8+PE6cuSIJk6cqJ07dyorK0uzZ8/WhAkTTt0sAABAo+X3W3tHjBihgwcPKjk5WQUFBerTp4+ys7O9F7Xm5+crMPCXxomOjtYbb7yhe+65R7169VLnzp01ceJETZo06dTNAgAANFp+x4gkJSYmKjExsdLv5ebmVlgWGxurDRs21OauAABAE8dn0wAAAFPECAAAMEWMAAAAU8QIAAAwRYwAAABTxAgAADBFjAAAAFPECAAAMEWMAAAAU8QIAAAwRYwAAABTxAgAADBFjAAAAFPECAAAMEWMAAAAU8QIAAAwRYwAAABTxAgAADBFjAAAAFPECAAAMBVsPQAAqErM5CzrIdRK3pxh1kMAGhXOjAAAAFPECAAAMEWMAAAAU8QIAAAwRYwAAABTxAgAADBFjAAAAFPECAAAMEWMAAAAU8QIAAAwRYwAAABTxAgAADBFjAAAAFPECAAAMEWMAAAAU8QIAAAwRYwAAABTxAgAADBFjAAAAFPECAAAMEWMAAAAU8QIAAAwRYwAAABTxAgAADBFjAAAAFPECAAAMEWMAAAAU8QIAAAwRYwAAABTxAgAADAVbD0A4FSKmZxlPYRayZszzK/1z5R5AjgzcGYEAACYIkYAAIApYgQAAJgiRgAAgCliBAAAmCJGAACAKWIEAACY4u+MAABwivA3gGqHMyMAAMAUMQIAAEwRIwAAwBQxAgAATBEjAADAFDECAABM1SpGMjIyFBMTo9DQUA0YMEAbN26s0XbLli1TQECAhg8fXpu7BQAATZDfMbJ8+XIlJSUpJSVFmzdvVu/evTVkyBAdOHCg2u3y8vJ033336corr6z1YAEAQNPjd4ykpaVp7NixSkhIUI8ePZSZmamwsDAtXry4ym3Kysp02223aebMmTrvvPPqNGAAANC0+BUjpaWl2rRpk+Li4n7ZQWCg4uLitH79+iq3e/DBB9W+fXvdcccdtR8pAABokvz6c/CHDh1SWVmZoqKifJZHRUVp+/btlW7z/vvva9GiRdq6dWuN76ekpEQlJSXe20VFRf4MEwAANCL1+m6aY8eO6fbbb9fChQsVGRlZ4+1SU1MVERHh/YqOjq7HUQIAAEt+nRmJjIxUUFCQCgsLfZYXFhaqQ4cOFdbfs2eP8vLyFB8f711WXl7+0x0HB2vHjh06//zzK2w3ZcoUJSUleW8XFRURJAAANFF+xUhISIj69u2rnJwc79tzy8vLlZOTo8TExArrX3TRRfr44499lk2bNk3Hjh3TggULqgwMj8cjj8fjz9AAAEAj5VeMSFJSUpJGjx6tfv36qX///kpPT1dxcbESEhIkSaNGjVLnzp2Vmpqq0NBQ9ezZ02f71q1bS1KF5QAA4Mzkd4yMGDFCBw8eVHJysgoKCtSnTx9lZ2d7L2rNz89XYCB/2BUAANSM3zEiSYmJiZW+LCNJubm51W777LPP1uYuAQBAE8UpDAAAYIoYAQAApogRAABgihgBAACmiBEAAGCKGAEAAKaIEQAAYIoYAQAApogRAABgihgBAACmiBEAAGCKGAEAAKaIEQAAYIoYAQAApogRAABgihgBAACmiBEAAGAq2HoAAICmL2ZylvUQaiVvzjDrIZwRODMCAABMESMAAMAUMQIAAEwRIwAAwBQxAgAATBEjAADAFDECAABMESMAAMAUMQIAAEwRIwAAwBQxAgAATBEjAADAFDECAABMESMAAMAUMQIAAEwRIwAAwBQxAgAATBEjAADAFDECAABMESMAAMAUMQIAAEwRIwAAwBQxAgAATBEjAADAFDECAABMESMAAMAUMQIAAEwRIwAAwBQxAgAATBEjAADAFDECAABMESMAAMAUMQIAAEwRIwAAwBQxAgAATBEjAADAFDECAABMESMAAMAUMQIAAEwRIwAAwBQxAgAATBEjAADAFDECAABMESMAAMAUMQIAAEwRIwAAwBQxAgAATBEjAADAFDECAABMBddmo4yMDM2bN08FBQXq3bu3Hn/8cfXv37/SdRcuXKh//OMf+uSTTyRJffv21ezZs6tcHwDOJDGTs6yHUGt5c4ZZDwFNhN9nRpYvX66kpCSlpKRo8+bN6t27t4YMGaIDBw5Uun5ubq5Gjhypd955R+vXr1d0dLSuu+467d+/v86DBwAAjZ/fMZKWlqaxY8cqISFBPXr0UGZmpsLCwrR48eJK13/uued05513qk+fPrrooov0zDPPqLy8XDk5OXUePAAAaPz8ipHS0lJt2rRJcXFxv+wgMFBxcXFav359jfbx/fff68cff1SbNm2qXKekpERFRUU+XwAAoGnyK0YOHTqksrIyRUVF+SyPiopSQUFBjfYxadIkderUySdo/ldqaqoiIiK8X9HR0f4MEwAANCIN+m6aOXPmaNmyZVq5cqVCQ0OrXG/KlCk6evSo92vfvn0NOEoAANCQ/Ho3TWRkpIKCglRYWOizvLCwUB06dKh220ceeURz5szRW2+9pV69elW7rsfjkcfj8WdoAACgkfLrzEhISIj69u3rc/HpzxejxsbGVrndww8/rFmzZik7O1v9+vWr/WgBAECT4/ffGUlKStLo0aPVr18/9e/fX+np6SouLlZCQoIkadSoUercubNSU1MlSXPnzlVycrKWLl2qmJgY77UlLVu2VMuWLU/hVAAAQGPkd4yMGDFCBw8eVHJysgoKCtSnTx9lZ2d7L2rNz89XYOAvJ1yeeuoplZaW6qabbvLZT0pKimbMmFG30QMAgEavVn+BNTExUYmJiZV+Lzc31+d2Xl5ebe4CAACcIfhsGgAAYIoYAQAApogRAABgihgBAACmiBEAAGCKGAEAAKaIEQAAYIoYAQAApogRAABgihgBAACmiBEAAGCKGAEAAKaIEQAAYIoYAQAApogRAABgihgBAACmiBEAAGCKGAEAAKaIEQAAYIoYAQAApogRAABgihgBAACmiBEAAGCKGAEAAKaIEQAAYIoYAQAApogRAABgihgBAACmiBEAAGCKGAEAAKaIEQAAYIoYAQAApogRAABgihgBAACmiBEAAGCKGAEAAKaIEQAAYIoYAQAApogRAABgihgBAACmiBEAAGCKGAEAAKaIEQAAYIoYAQAApogRAABgihgBAACmiBEAAGCKGAEAAKaIEQAAYIoYAQAApogRAABgihgBAACmiBEAAGCKGAEAAKaIEQAAYIoYAQAApogRAABgihgBAACmiBEAAGCKGAEAAKaIEQAAYIoYAQAApogRAABgihgBAACmiBEAAGCKGAEAAKZqFSMZGRmKiYlRaGioBgwYoI0bN1a7/osvvqiLLrpIoaGhuuSSS7R69epaDRYAADQ9fsfI8uXLlZSUpJSUFG3evFm9e/fWkCFDdODAgUrXX7dunUaOHKk77rhDW7Zs0fDhwzV8+HB98skndR48AABo/PyOkbS0NI0dO1YJCQnq0aOHMjMzFRYWpsWLF1e6/oIFCzR06FDdf//96t69u2bNmqX/+7//0xNPPFHnwQMAgMYv2J+VS0tLtWnTJk2ZMsW7LDAwUHFxcVq/fn2l26xfv15JSUk+y4YMGaJXXnmlyvspKSlRSUmJ9/bRo0clSUVFRf4Mt0bKS74/5ftsKP48Ho11nv4ec+Z5emOeFTXWOUpnxjz5f/bU7Nc5V+16fsXIoUOHVFZWpqioKJ/lUVFR2r59e6XbFBQUVLp+QUFBlfeTmpqqmTNnVlgeHR3tz3CbvIh06xHUvzNhjhLzbGqYZ9NxJsxRqv95Hjt2TBEREVV+368YaShTpkzxOZtSXl6uI0eOqG3btgoICDAcWc0VFRUpOjpa+/btU3h4uPVw6g3zbFqYZ9NxJsxRYp6nO+ecjh07pk6dOlW7nl8xEhkZqaCgIBUWFvosLywsVIcOHSrdpkOHDn6tL0kej0cej8dnWevWrf0Z6mkjPDy8Uf2PU1vMs2lhnk3HmTBHiXmezqo7I/Izvy5gDQkJUd++fZWTk+NdVl5erpycHMXGxla6TWxsrM/6krRmzZoq1wcAAGcWv1+mSUpK0ujRo9WvXz/1799f6enpKi4uVkJCgiRp1KhR6ty5s1JTUyVJEydO1NVXX6358+dr2LBhWrZsmT788EM9/fTTp3YmAACgUfI7RkaMGKGDBw8qOTlZBQUF6tOnj7Kzs70Xqebn5ysw8JcTLgMHDtTSpUs1bdo0PfDAA+ratateeeUV9ezZ89TN4jTk8XiUkpJS4eWmpoZ5Ni3Ms+k4E+YoMc+mIsCd7P02AAAA9YjPpgEAAKaIEQAAYIoYAQAApogRIzNmzFCfPn2sh1HvYmJilJ6eXu06AQEB1X48wOkoNzdXAQEB+vbbb62HUq/GjBmj4cOHV7vO4MGDdffddzfIeOpLXl6eAgICtHXrVuuh1Kua/N6pyTEHTjViBMAZJTc3VzNmzLAeRr0bM2aM8vLyrIfR4Hii94vG9ESPGKknPz/TaupmzJihMWPGWA8DOKnMzEwdOHDAe7u0tFTz58/Xjz/+aDiqU+vIkSPKyMjw+VCyPXv26LnnnjMcVf07fvy4WrRood27d1sPpV4tWbJEgwYNsh5GvSBG/JCdna1BgwapdevWatu2rW644Qbt2bOnTvv8+9//rujoaIWFhenmm2/2fkJxfTnZHCZNmqRu3bopLCxM5513nqZPn17nX9bHjh3TyJEj1aJFC3Xu3FkZGRkV1vn66691/fXXq3nz5jrvvPO0YsWKOt2nv2p7bNeuXatevXopNDRUl19+uT755JMGGG3l6vPYzpw5U+3atVN4eLjGjRun0tJSn++fOHFCiYmJioiIUGRkpKZPn37ST+msD9U9BtHR0brxxhu1cuVKffrpp7rmmmskyedJw/bt2zVw4ECFhoaqZ8+eevfdd0+rOUjVH8fQ0FDt379fQ4cO1ZdffqnMzEyNGTNG5557rs991OT3zsmO+elkzZo16tKliy644ALrodSrVatW6cYbb7QeRr0gRvxQXFyspKQkffjhh8rJyVFgYKB+97vfqby8vFb72717t1544QW99tprys7O1pYtW3TnnXee4lH7OtkcWrVqpWeffVafffaZFixYoIULF+rRRx+t033OmzdPvXv31pYtWzR58mRNnDhRa9as8Vln+vTp+v3vf6+PPvpIt912m2655RZt27atTvfrj9oe2/vvv1/z58/Xf/7zH7Vr107x8fFmz7Tr69jm5ORo27Ztys3N1fPPP6+XX365wqdqL1myRMHBwdq4caMWLFigtLQ0PfPMM/Uyz+pU9xgMGzZMb7zxht58802tXr1ajz/+uO69914FB//ytx/vv/9+3XvvvdqyZYtiY2MVHx+vw4cPnzZzkKo/jmFhYZo9e7YmTpyo3NxcbdiwQW+//bYGDhzo3X9Nfu/U5Jg3lJo8UajsH+mGfqJXnVPxROGHH37Qm2++6TPPxvBEr8Ycau3gwYNOkvv4448rfO+LL75w1T28KSkpLigoyH355ZfeZa+//roLDAx0X3/9db2MtzLVzcE55+bNm+f69u1b5fYpKSlu9OjRVX6/S5cubujQoT7LRowY4a6//nrvbUlu3LhxPusMGDDAjR8/vgYzqB8ne1zeeecdJ8ktW7bMu+zw4cOuefPmbvny5Q01zGrV9dg659zo0aNdmzZtXHFxsXfZU0895Vq2bOnKysqcc85dffXVrnv37q68vNy7zqRJk1z37t1PwSzq5r8fg9dff91dfvnl7q677nI33XSTGzRokEtPT3cnTpzw/rzOmTPHu+2PP/7ozj77bDd37lzDGfh3HI8fP+6mT5/urrvuOvfrX//aTZo0yV111VXugw8+cM7V7PdOTY55Q1qxYoV76aWX3K5du9yWLVtcfHy8u+SSS7xjKSsrc+3bt3fr1q1zzv00xxYtWrhrrrnGbdmyxb377rvuggsucLfeemuDj72mc5g1a5Zbu3at++KLL9yrr77qoqKiKvx/9+9//9t169bNe7tLly6uVatWLjU11e3YscM99thjLigoyL355pvedSS5tm3buoULF7odO3a4adOmuaCgIPfZZ581zMT9QIz4YefOne6WW25x5557rmvVqpVr0aKFk+SysrKcc8716NHDtWjRwrVo0cKFhYU5Sd7bLVq08PlHOSUlxZ177rk++//222+dJJebm2s2h2XLlrmBAwe6qKgo16JFC+fxeFy7du2827/33ns+c2rWrJkLDg72Wfavf/3Lu36XLl3czJkzfcaQnp7uYmJivLcluSVLlvisc/fdd7vBgwfXx0NQqeoel6FDh3rn1qNHD+fcLzGyd+9en/306dPHzZgxo8HG/d/qcmz37t3rcwwfeugh59xP/zD96le/8rmfrVu3OkkuLy/POfdTjCQkJPis88orr7jg4GB34sSJ+p62j+oegyeffNIVFha6d955x6WkpLiSkhI3b948V1pa6o2Rd99912d/w4cPd2PGjDlt5uBc9cfx8OHD7oknnnDl5eVu9OjR7osvvnC7d+92//znP51zNfu9U5Njbul/42zt2rWuffv23n/YT5cnetWpzROFsWPHuvvuu897u7E+0auK359NcyaLj49Xly5dtHDhQnXq1Enl5eXq2bOn97XU1atXe0+t7d+/X4MHD/Z5q2Dz5s0thu2jujmsX79et912m2bOnKkhQ4YoIiJCy5Yt0/z5873b9+vXz2dOjz32mPbv36+5c+d6l/38OUWNSXWPyzPPPKPjx49Lkpo1a2Y80qrV5dh26tTJ57i2adPGaBZ1U91jMH78eEnSZ599JumnTyG/7777LIdbqbocxzZt2mjChAk++zv//PN1/vnnW0zllNi1a5eSk5P1wQcf6NChQ96Xq/Lz89WzZ0+tWrVKN9xwg89nop1zzjnq3Lmz93ZsbKzKy8u1Y8cOdejQ4bSbw/Lly/XYY49pz549+u6773TixAmFh4d7t3fO6bXXXtMLL7zgs9/Y2NgKt//3HTaVrXM6voWdGKmhw4cPa8eOHVq4cKGuvPJKSdL777/vs06XLl28//3z69DVXVCVn5+vr776Sp06dZIkbdiwQYGBgbrwwgtP9fAlnXwO69atU5cuXTR16lTvsr179/rso3nz5j5zatOmjYqKiqqd54YNGyrc7t69e4Vlo0aN8rl96aWX+jG72jvZ4/Lfv9T+14YNG3TOOedIkr755hvt3LmzwtwaQl2PbXBwcJXH8KOPPtLx48e9Mb1hwwa1bNlS0dHR3nU++OADn202bNigrl27KigoqO6Tq6Ga/IxKP/1dlMGDB1e6jw0bNuiqq66S9NNFuZs2bVJiYmK9jfl/nYqf0Z89++yzlS6vye+dmhzzhnKyJ4Gvvvqq5syZ0+Dj8kddnwRu3LhRJ06c8Ln2p6khRmrorLPOUtu2bfX000+rY8eOys/P1+TJk+u0z9DQUI0ePVqPPPKIioqKdNddd+nmm2+ut3I/2Ry6du2q/Px8LVu2TJdddpmysrK0cuXKOt/v2rVr9fDDD2v48OFas2aNXnzxRWVlZfms8+KLL6pfv34aNGiQnnvuOW3cuFGLFi2q833XRF2O7YMPPqi2bdsqKipKU6dOVWRkpMkfjKrPY1taWqo77rhD06ZNU15enlJSUpSYmOjzTDQ/P19JSUn661//qs2bN+vxxx/3+WXaEE7Fz2hGRoa6du2q7t2769FHH9U333yjP/3pT/U04ooa4me0Jr93anLMG8LJ4mzXrl3au3evrr32Wp/tGvqJXnVORWCuWrVKw4YNqxD3p/sTPb9Yv07UmKxZs8Z1797deTwe16tXL5ebm+skuZUrV1ZYtyYXsPbu3ds9+eSTrlOnTi40NNTddNNN7siRI/U4g5PP4f7773dt27Z1LVu2dCNGjHCPPvqoi4iIqHYeJ7uAdebMme4Pf/iDCwsLcx06dHALFizwWUeSy8jIcNdee63zeDwuJiamwS8C9efYOvfLNSOvvfaau/jii11ISIjr37+/++ijjxp03P/tVB9b5366fuC3v/2tS05O9m47duxY98MPP3jXufrqq92dd97pxo0b58LDw91ZZ53lHnjgAZ8LWhuKv8fxZz//vC5dutT179/fhYSEuB49eri33367YQb+X+rjOP6sJr93anLMG0pZWZlr27at++Mf/+h27drlcnJy3GWXXeZ9PObNm+fi4+N9tvn5Ata4uDi3detW995777lu3bq5W265pcHHX5M5rFq1ygUHB7vnn3/e7d692y1YsMC1adPG55hefPHF7qWXXvLZb5cuXVx4eLibO3eu27Fjh3viiSdcUFCQy87O9q4jyUVGRrpFixa5HTt2uOTkZBcYGOg+/fTThpp+jREjAIDTVnVxNmjQILdw4UKf9a2e6FWnLoG5e/du5/F43Hfffeezz8byRK+mApwz+MtEAADUwaFDh9SxY0d9+eWXjfKi+ZpKS0vTW2+9pdWrV1sPpV7xR88AAI3OkSNHlJaW1qRDRJLOPvtsTZkyxXoY9Y4zIwAAwBRnRgAAgCliBAAAmCJGAACAKWIEAACYIkYAAIApYgQAAJgiRgAAgCliBAAAmCJGAACAqf8HJYdPKy+AuaQAAAAASUVORK5CYII=",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "NUM_SAMPLES = 100\n",
        "\n",
        "def inference(sample):\n",
        "  samle_str = '.' * (13 - len(sample)) + sample\n",
        "  context = torch.tensor([encode(samle_str)], dtype = torch.int32)\n",
        "  context = context.to(device)\n",
        "  res = decode(m.generate(context, max_new_tokens = 17)[0].tolist()).replace('.', '')\n",
        "  res = res[res.index('=') + 1 :res.index('E')]\n",
        "  return res\n",
        "\n",
        "def get_predictions(NUM_SAMPLES, x, y, operation):\n",
        "  i = 0\n",
        "  accuracy = 0\n",
        "  while i < NUM_SAMPLES:\n",
        "    a, b = random.randint(x, y), random.randint(x, y)\n",
        "\n",
        "    task = str(a) + operation + str(b) + '='\n",
        "    if operation == '+':\n",
        "      ans = a + b\n",
        "    if operation == '-':\n",
        "      ans = a - b\n",
        "    if operation == '*':\n",
        "      ans = a * b\n",
        "    if operation == '/':\n",
        "      if b == 0:\n",
        "        i += 1\n",
        "        continue\n",
        "      ans = a // b\n",
        "\n",
        "    res = int(inference(task)[::-1])\n",
        "    accuracy += int(ans == res)\n",
        "    i += 1\n",
        "\n",
        "  return accuracy / NUM_SAMPLES\n",
        "\n",
        "plt.bar(['a+b', 'aa+bb', 'a-b', 'aa-bb', 'a*b', 'aa*bb', 'a/b', 'aa/bb'],\n",
        "        [get_predictions(NUM_SAMPLES, 0, 9, '+'), get_predictions(NUM_SAMPLES, 10, 99, '+'),\n",
        "         get_predictions(NUM_SAMPLES, 0, 9, '-'), get_predictions(NUM_SAMPLES, 10, 99, '-'),\n",
        "         get_predictions(NUM_SAMPLES, 0, 9, '*'), get_predictions(NUM_SAMPLES, 10, 99, '*'),\n",
        "         get_predictions(NUM_SAMPLES, 0, 9, '/'), get_predictions(NUM_SAMPLES, 10, 99, '/')],)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.8"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}