interactive.ipynb

{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "FU-1GTZT28li"
      },
      "source": [
        "# The tuned lens 🔎\n",
        "A tuned lens allows us to peak at the iterative computations that a transformer is using the compute the next token.\n",
        "\n",
        "A lens into a transformer with n layers allows you to replace the last $m$ layers of the model with an [affine transformation](https://pytorch.org/docs/stable/generated/torch.nn.Linear.html) (we call these affine translators).\n",
        "\n",
        "This essentially skips over these last few layers and lets you see the best prediction that can be made from the model's representations, i.e. the residual stream, at layer $n - m$. Since the representations may be rotated, shifted, or stretched from layer to layer it's useful to train the len's affine translators specifically on each layer. This training is what differentiates this method from simpler approaches that decode the residual stream of the network directly using the unembedding layer i.e. the logit lens. We explain this process along with more applications of the method in [the paper](ttps://arxiv.org/abs/2303.08112).\n",
        "\n",
        "You can find the complete set of pretrained lenses on [the hugging face space](https://huggingface.co/spaces/AlignmentResearch/tuned-lens/tree/main/lens).\n",
        "\n",
        "## Usage\n",
        "Since the tuned lens produces a distribution of predictions to visualize it's output we need to we need to provide a summary statistic to plot.  The default is simply [entropy](https://en.wikipedia.org/wiki/Entropy_(information_theory)), but you can also choose the [cross entropy](https://en.wikipedia.org/wiki/Cross_entropy) with the target token, or the [KL divergence](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence) between the model's predictions and the tuned lens' predictions. You can also hover over a token to see more of the distribution i.e. the top 10 most probable tokens and their probabilities.\n",
        "\n",
        "## Examples\n",
        "Some interesting examples you can try.\n",
        "\n",
        "### Copy paste:\n",
        "```\n",
        "Copy: A!2j!#u&NGApS&MkkHe8Gm!#\n",
        "Paste: A!2j!#u&NGApS&MkkHe8Gm!#\n",
        "```\n",
        "\n",
        "### Trivial in-context learning\n",
        "```\n",
        "inc 1 2\n",
        "inc 4 5\n",
        "inc 13 \n",
        "```\n",
        "\n",
        "#### Addition\n",
        "```\n",
        "add 1 1 2\n",
        "add 3 4 7\n",
        "add 13 2 \n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "W9cRsIdK2-Jm"
      },
      "outputs": [],
      "source": [
        "# !pip install tuned-lens\n",
        "# !pip install nbformat --upgrade"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "FGd2YmyD28lk",
        "scrolled": true
      },
      "outputs": [],
      "source": [
        "# import torch\n",
        "# from tuned_lens.nn.lenses import TunedLens, LogitLens\n",
        "# from transformers import AutoModelForCausalLM, AutoTokenizer\n",
        "\n",
        "\n",
        "# device = torch.device('cpu')\n",
        "# model_id_pythia = 'gpt2-large'\n",
        "# cache_dir_pythia = \"./models/\" + model_id_pythia\n",
        "# # To try a diffrent modle / lens check if the lens is avalible then modify this code\n",
        "# model = AutoModelForCausalLM.from_pretrained(\n",
        "#     pretrained_model_name_or_path=model_id_pythia,\n",
        "#     cache_dir=cache_dir_pythia,\n",
        "#     device_map=device,\n",
        "#     )\n",
        "# model = model.to(device)\n",
        "# tokenizer = AutoTokenizer.from_pretrained(\n",
        "#     pretrained_model_name_or_path=model_id_pythia,\n",
        "#     cache_dir=cache_dir_pythia,\n",
        "#     )\n",
        "# tuned_lens = TunedLens.from_model_and_pretrained(model)\n",
        "# tuned_lens = tuned_lens.to(device)\n",
        "# logit_lens = LogitLens.from_model(model)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "DL8f4i2828lm",
        "scrolled": true
      },
      "outputs": [],
      "source": [
        "# from tuned_lens.plotting import PredictionTrajectory\n",
        "# import ipywidgets as widgets\n",
        "# from plotly import graph_objects as go\n",
        "\n",
        "\n",
        "# def make_plot(lens, text, layer_stride, statistic, token_range):\n",
        "#     input_ids = tokenizer.encode(text)\n",
        "#     targets = input_ids[1:] + [tokenizer.eos_token_id]\n",
        "\n",
        "#     if len(input_ids) == 0:\n",
        "#         return widgets.Text(\"Please enter some text.\")\n",
        "    \n",
        "#     if (token_range[0] == token_range[1]):\n",
        "#         return widgets.Text(\"Please provide valid token range.\")\n",
        "#     pred_traj = PredictionTrajectory.from_lens_and_model(\n",
        "#         lens=lens,\n",
        "#         model=model,\n",
        "#         input_ids=input_ids,\n",
        "#         tokenizer=tokenizer,\n",
        "#         targets=targets,\n",
        "#     ).slice_sequence(slice(*token_range))\n",
        "\n",
        "#     return getattr(pred_traj, statistic)().stride(layer_stride).figure(\n",
        "#         title=f\"{lens.__class__.__name__} ({model.name_or_path}) {statistic}\",\n",
        "#     )\n",
        "\n",
        "# style = {'description_width': 'initial'}\n",
        "# statistic_wdg = widgets.Dropdown(\n",
        "#     options=[\n",
        "#         ('Entropy', 'entropy'),\n",
        "#         ('Cross Entropy', 'cross_entropy'),\n",
        "#         ('Forward KL', 'forward_kl'),\n",
        "#     ],\n",
        "#     description='Select Statistic:',\n",
        "#     style=style,\n",
        "# )\n",
        "# text_wdg = widgets.Textarea(\n",
        "#     description=\"Input Text\",\n",
        "#     value=\"it was the best of times, it was the worst of times\",\n",
        "# )\n",
        "# lens_wdg = widgets.Dropdown(\n",
        "#     options=[('Tuned Lens', tuned_lens), ('Logit Lens', logit_lens)],\n",
        "#     description='Select Lens:',\n",
        "#     style=style,\n",
        "# )\n",
        "\n",
        "# layer_stride_wdg = widgets.BoundedIntText(\n",
        "#     value=2,\n",
        "#     min=1,\n",
        "#     max=10,\n",
        "#     step=1,\n",
        "#     description='Layer Stride:',\n",
        "#     disabled=False\n",
        "# )\n",
        "\n",
        "# token_range_wdg = widgets.IntRangeSlider(\n",
        "#     description='Token Range',\n",
        "#     min=0,\n",
        "#     max=1,\n",
        "#     step=1,\n",
        "#     style=style,\n",
        "# )\n",
        "\n",
        "\n",
        "# def update_token_range(*args):\n",
        "#     token_range_wdg.max = len(tokenizer.encode(text_wdg.value))\n",
        "\n",
        "# update_token_range()\n",
        "\n",
        "# token_range_wdg.value = [0, token_range_wdg.max]\n",
        "# text_wdg.observe(update_token_range, 'value')\n",
        "\n",
        "# interact = widgets.interact.options(manual_name='Run Lens', manual=True)\n",
        "\n",
        "# plot = interact(\n",
        "#     make_plot,\n",
        "#     text=text_wdg,\n",
        "#     statistic=statistic_wdg,\n",
        "#     lens=lens_wdg,\n",
        "#     layer_stride=layer_stride_wdg,\n",
        "#     token_range=token_range_wdg,\n",
        "# )"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {},
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/Users/s2011847/repos/tuned-lens/tuned_lens/nn/lenses.py:280: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don't have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.\n",
            "  state = th.load(ckpt_path, **th_load_kwargs, map_location=device)\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "f514f3873bce4ab2aadb997e2fa8d8d2",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "interactive(children=(Dropdown(description='Select Lens:', options=(('Tuned Lens', TunedLens(\n",
              "  (unembed): Une…"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.plotly.v1+json": {
              "config": {
                "plotlyServerURL": "https://plot.ly"
              },
              "data": [
                {
                  "colorbar": {
                    "title": {
                      "side": "right",
                      "text": "Entropy (nats)"
                    }
                  },
                  "colorscale": [
                    [
                      0,
                      "rgb(5,48,97)"
                    ],
                    [
                      0.1,
                      "rgb(33,102,172)"
                    ],
                    [
                      0.2,
                      "rgb(67,147,195)"
                    ],
                    [
                      0.3,
                      "rgb(146,197,222)"
                    ],
                    [
                      0.4,
                      "rgb(209,229,240)"
                    ],
                    [
                      0.5,
                      "rgb(247,247,247)"
                    ],
                    [
                      0.6,
                      "rgb(253,219,199)"
                    ],
                    [
                      0.7,
                      "rgb(244,165,130)"
                    ],
                    [
                      0.8,
                      "rgb(214,96,77)"
                    ],
                    [
                      0.9,
                      "rgb(178,24,43)"
                    ],
                    [
                      1,
                      "rgb(103,0,31)"
                    ]
                  ],
                  "customdata": [
                    [
                      [
                        "       ,",
                        " 13.65%",
                        "     _is",
                        "  5.30%",
                        "       .",
                        "  5.10%",
                        "    _and",
                        "  2.89%",
                        "    _was",
                        "  2.22%",
                        "      's",
                        "  1.60%",
                        "     _to",
                        "  1.09%",
                        "    _has",
                        "  1.08%",
                        "       >",
                        "  1.02%",
                        "      \\n",
                        "  0.99%"
                      ],
                      [
                        "      _a",
                        " 10.89%",
                        "    _the",
                        "  2.63%",
                        "   _also",
                        "  2.13%",
                        "   _used",
                        "  1.99%",
                        "    _not",
                        "  1.76%",
                        " _prepa…",
                        "  1.33%",
                        " _treat…",
                        "  1.29%",
                        " _enter…",
                        "  1.26%",
                        " _estim…",
                        "  1.02%",
                        "     _in",
                        "  1.00%"
                      ],
                      [
                        "   _same",
                        "  1.36%",
                        "  _right",
                        "  0.85%",
                        " _former",
                        "  0.77%",
                        " _condi…",
                        "  0.69%",
                        " _police",
                        "  0.67%",
                        " _remai…",
                        "  0.65%",
                        "   _fact",
                        "  0.58%",
                        " _netwo…",
                        "  0.53%",
                        " _probl…",
                        "  0.53%",
                        "   _East",
                        "  0.52%"
                      ],
                      [
                        "       .",
                        "  4.89%",
                        "       ,",
                        "  4.27%",
                        " _surpr…",
                        "  2.19%",
                        "     _of",
                        "  2.03%",
                        "     _to",
                        "  1.72%",
                        " _probl…",
                        "  1.42%",
                        "     _in",
                        "  1.10%",
                        "     _on",
                        "  1.05%",
                        "   _fair",
                        "  0.90%",
                        "    over",
                        "  0.82%"
                      ],
                      [
                        "    _the",
                        " 17.64%",
                        "      _a",
                        "  6.37%",
                        "   _this",
                        "  1.72%",
                        "    _his",
                        "  1.18%",
                        "  _those",
                        "  1.01%",
                        "  _their",
                        "  0.91%",
                        "      \\n",
                        "  0.75%",
                        "   _that",
                        "  0.72%",
                        "    _all",
                        "  0.66%",
                        "    _our",
                        "  0.58%"
                      ],
                      [
                        "       ,",
                        " 17.56%",
                        "       .",
                        "  8.71%",
                        "     _of",
                        "  6.01%",
                        "   _with",
                        "  2.40%",
                        "    _and",
                        "  2.09%",
                        "     _in",
                        "  1.86%",
                        "      _a",
                        "  1.65%",
                        "     _on",
                        "  1.63%",
                        "      \\n",
                        "  1.24%",
                        "    _the",
                        "  1.20%"
                      ],
                      [
                        "    _and",
                        " 10.23%",
                        "      _a",
                        "  3.02%",
                        "    _the",
                        "  2.91%",
                        "      \\n",
                        "  2.50%",
                        "    _but",
                        "  1.66%",
                        "     _or",
                        "  1.58%",
                        "      _I",
                        "  1.55%",
                        "  _which",
                        "  1.42%",
                        "     _it",
                        "  1.39%",
                        "     _he",
                        "  1.23%"
                      ],
                      [
                        "     _is",
                        " 11.97%",
                        "       .",
                        "  6.63%",
                        "    _was",
                        "  5.42%",
                        "       ,",
                        "  5.18%",
                        "      's",
                        "  3.89%",
                        "    _has",
                        "  3.62%",
                        "     âĢĻ",
                        "  2.70%",
                        "    _can",
                        "  1.34%",
                        "      .\"",
                        "  1.23%",
                        " _should",
                        "  1.13%"
                      ],
                      [
                        "      _a",
                        " 10.89%",
                        "    _the",
                        "  2.63%",
                        "   _also",
                        "  2.13%",
                        "   _used",
                        "  1.99%",
                        "    _not",
                        "  1.76%",
                        " _prepa…",
                        "  1.33%",
                        " _treat…",
                        "  1.29%",
                        " _enter…",
                        "  1.26%",
                        " _estim…",
                        "  1.02%",
                        "     _in",
                        "  1.00%"
                      ],
                      [
                        "   _same",
                        "  1.36%",
                        "  _right",
                        "  0.85%",
                        " _former",
                        "  0.77%",
                        " _condi…",
                        "  0.69%",
                        " _police",
                        "  0.67%",
                        " _remai…",
                        "  0.65%",
                        "   _fact",
                        "  0.58%",
                        " _netwo…",
                        "  0.53%",
                        " _probl…",
                        "  0.53%",
                        "   _East",
                        "  0.52%"
                      ],
                      [
                        "       ,",
                        "  4.79%",
                        "     _of",
                        "  4.75%",
                        "       .",
                        "  4.43%",
                        "      _a",
                        "  2.51%",
                        "    _now",
                        "  1.94%",
                        "     _on",
                        "  1.73%",
                        "      .\"",
                        "  1.16%",
                        "   _that",
                        "  1.04%",
                        "    _one",
                        "  0.98%",
                        "    _out",
                        "  0.98%"
                      ],
                      [
                        "    _the",
                        " 17.64%",
                        "      _a",
                        "  6.37%",
                        "   _this",
                        "  1.72%",
                        "    _his",
                        "  1.18%",
                        "  _those",
                        "  1.01%",
                        "  _their",
                        "  0.91%",
                        "      \\n",
                        "  0.75%",
                        "   _that",
                        "  0.72%",
                        "    _all",
                        "  0.66%",
                        "    _our",
                        "  0.58%"
                      ],
                      [
                        "       ,",
                        " 17.56%",
                        "       .",
                        "  8.71%",
                        "     _of",
                        "  6.01%",
                        "   _with",
                        "  2.40%",
                        "    _and",
                        "  2.09%",
                        "     _in",
                        "  1.86%",
                        "      _a",
                        "  1.65%",
                        "     _on",
                        "  1.63%",
                        "      \\n",
                        "  1.24%",
                        "    _the",
                        "  1.20%"
                      ]
                    ],
                    [
                      [
                        "       k",
                        " 11.54%",
                        "       \"",
                        "  8.67%",
                        "       g",
                        "  5.29%",
                        "       _",
                        "  2.79%",
                        "     _is",
                        "  2.36%",
                        "      (\"",
                        "  1.88%",
                        "      \":",
                        "  1.69%",
                        "      ia",
                        "  1.60%",
                        "       u",
                        "  1.59%",
                        "     iff",
                        "  1.59%"
                      ],
                      [
                        "      _a",
                        "  5.93%",
                        "      \\n",
                        "  3.07%",
                        "   _also",
                        "  2.91%",
                        "       .",
                        "  2.22%",
                        "     _in",
                        "  1.99%",
                        "    _too",
                        "  1.89%",
                        "   _used",
                        "  1.64%",
                        "       ,",
                        "  1.45%",
                        "      _\"",
                        "  1.44%",
                        "     _to",
                        "  1.41%"
                      ],
                      [
                        "      \\n",
                        "  1.06%",
                        "  _first",
                        "  0.85%",
                        "      _\"",
                        "  0.78%",
                        "   _same",
                        "  0.65%",
                        "    _pre",
                        "  0.55%",
                        "   _last",
                        "  0.53%",
                        "   _best",
                        "  0.50%",
                        "   _fact",
                        "  0.48%",
                        " _police",
                        "  0.47%",
                        "   _fair",
                        "  0.46%"
                      ],
                      [
                        "       .",
                        "  4.42%",
                        "     _of",
                        "  3.21%",
                        "    _and",
                        "  2.55%",
                        " _perfo…",
                        "  2.38%",
                        "    _bet",
                        "  2.05%",
                        "  _thing",
                        "  1.87%",
                        "       s",
                        "  1.75%",
                        "     _in",
                        "  1.45%",
                        "       ,",
                        "  1.44%",
                        "   _time",
                        "  1.43%"
                      ],
                      [
                        "    _the",
                        "  9.00%",
                        "      _a",
                        "  3.10%",
                        "    _our",
                        "  2.49%",
                        "   _your",
                        "  1.00%",
                        "   _this",
                        "  1.00%",
                        "    _all",
                        "  0.82%",
                        "  _their",
                        "  0.82%",
                        "    _any",
                        "  0.80%",
                        " _things",
                        "  0.73%",
                        "     _my",
                        "  0.68%"
                      ],
                      [
                        "    _and",
                        " 12.86%",
                        "       .",
                        "  9.86%",
                        "       ,",
                        "  5.11%",
                        "     _in",
                        "  2.90%",
                        "   _that",
                        "  2.55%",
                        "      _a",
                        "  2.38%",
                        "    _for",
                        "  2.30%",
                        "     _of",
                        "  2.30%",
                        "   _when",
                        "  2.20%",
                        "   _with",
                        "  2.20%"
                      ],
                      [
                        "    _and",
                        "  4.78%",
                        "     _or",
                        "  3.00%",
                        "      \\n",
                        "  2.51%",
                        "      _a",
                        "  2.05%",
                        "    _the",
                        "  1.52%",
                        "    _but",
                        "  1.06%",
                        "     _it",
                        "  1.02%",
                        "   _when",
                        "  0.89%",
                        "     _in",
                        "  0.82%",
                        " _becau…",
                        "  0.76%"
                      ],
                      [
                        "     _is",
                        " 13.40%",
                        "    _was",
                        "  5.30%",
                        "       .",
                        "  4.70%",
                        "    _has",
                        "  4.54%",
                        "       ,",
                        "  3.73%",
                        "      's",
                        "  3.53%",
                        " _should",
                        "  2.78%",
                        "  _seems",
                        "  2.39%",
                        "     _in",
                        "  2.02%",
                        "     âĢĻ",
                        "  1.83%"
                      ],
                      [
                        "      _a",
                        " 12.52%",
                        "   _also",
                        "  3.73%",
                        "    _the",
                        "  2.41%",
                        "  _still",
                        "  1.62%",
                        "    _not",
                        "  1.56%",
                        "    _too",
                        "  1.46%",
                        "   _that",
                        "  1.44%",
                        "   _just",
                        "  1.19%",
                        "     _so",
                        "  1.15%",
                        "     _in",
                        "  1.11%"
                      ],
                      [
                        "      \\n",
                        "  0.87%",
                        "   _same",
                        "  0.86%",
                        "  _first",
                        "  0.81%",
                        "   _best",
                        "  0.73%",
                        "      _\"",
                        "  0.71%",
                        "   _fact",
                        "  0.62%",
                        "   _last",
                        "  0.56%",
                        " _police",
                        "  0.54%",
                        "   _fair",
                        "  0.53%",
                        "  _right",
                        "  0.50%"
                      ],
                      [
                        "     _of",
                        "  4.53%",
                        "       .",
                        "  3.89%",
                        "    _and",
                        "  3.32%",
                        "       ,",
                        "  3.28%",
                        " _probl…",
                        "  2.31%",
                        "  _thing",
                        "  2.17%",
                        "    _bet",
                        "  2.16%",
                        " _perfo…",
                        "  2.09%",
                        "   _time",
                        "  2.00%",
                        "      .\"",
                        "  1.55%"
                      ],
                      [
                        "    _the",
                        " 18.08%",
                        "      _a",
                        "  6.02%",
                        "    _our",
                        "  2.08%",
                        "   _this",
                        "  1.39%",
                        "  _their",
                        "  1.39%",
                        "  _those",
                        "  0.95%",
                        "    _any",
                        "  0.95%",
                        "    _all",
                        "  0.94%",
                        "   _them",
                        "  0.83%",
                        " _things",
                        "  0.80%"
                      ],
                      [
                        "       .",
                        " 13.78%",
                        "    _and",
                        " 11.39%",
                        "       ,",
                        "  9.45%",
                        "   _when",
                        "  3.06%",
                        "   _that",
                        "  2.96%",
                        "      _a",
                        "  2.59%",
                        "     _in",
                        "  2.47%",
                        "     _of",
                        "  2.11%",
                        "     _or",
                        "  1.77%",
                        "   _with",
                        "  1.63%"
                      ]
                    ],
                    [
                      [
                        "       _",
                        "  4.26%",
                        "     aba",
                        "  3.42%",
                        "    ness",
                        "  2.00%",
                        "       .",
                        "  1.59%",
                        "     _is",
                        "  1.52%",
                        "       k",
                        "  1.14%",
                        "    ania",
                        "  1.14%",
                        "       ,",
                        "  0.92%",
                        "    rate",
                        "  0.92%",
                        "       -",
                        "  0.91%"
                      ],
                      [
                        "      _a",
                        "  7.70%",
                        "   _also",
                        "  4.06%",
                        "     _to",
                        "  1.73%",
                        "   _made",
                        "  1.38%",
                        " _becau…",
                        "  1.36%",
                        "    _the",
                        "  1.32%",
                        "   _that",
                        "  1.15%",
                        "     _in",
                        "  1.09%",
                        "   _when",
                        "  0.94%",
                        "    _not",
                        "  0.84%"
                      ],
                      [
                        "  _right",
                        "  1.07%",
                        "   _best",
                        "  0.93%",
                        "   _same",
                        "  0.76%",
                        "   _case",
                        "  0.74%",
                        "   _fact",
                        "  0.61%",
                        " _favour",
                        "  0.59%",
                        "    _pre",
                        "  0.57%",
                        "   _last",
                        "  0.57%",
                        "    _job",
                        "  0.51%",
                        "      \\n",
                        "  0.47%"
                      ],
                      [
                        " _possi…",
                        "  6.97%",
                        "     _of",
                        "  4.33%",
                        "    _bet",
                        "  3.85%",
                        " _perfo…",
                        "  2.28%",
                        "       .",
                        "  2.27%",
                        "       ,",
                        "  2.09%",
                        "    _one",
                        "  1.95%",
                        " _outco…",
                        "  1.94%",
                        "    _and",
                        "  1.79%",
                        " _result",
                        "  1.65%"
                      ],
                      [
                        "    _the",
                        "  7.63%",
                        "    _our",
                        "  5.26%",
                        "   _your",
                        "  3.93%",
                        "      _a",
                        "  2.41%",
                        "    _all",
                        "  2.19%",
                        "  _being",
                        "  1.95%",
                        " _career",
                        "  1.81%",
                        "     _it",
                        "  1.64%",
                        "   _good",
                        "  1.56%",
                        "  _doing",
                        "  1.51%"
                      ],
                      [
                        "    _and",
                        " 19.79%",
                        "       .",
                        " 16.27%",
                        "   _when",
                        "  8.66%",
                        "       ,",
                        "  7.78%",
                        "    _for",
                        "  3.56%",
                        "     âĢĶ",
                        "  3.44%",
                        "   _that",
                        "  2.83%",
                        "  _being",
                        "  2.46%",
                        "     _in",
                        "  2.24%",
                        "     _of",
                        "  2.11%"
                      ],
                      [
                        "    _and",
                        "  4.52%",
                        "      _a",
                        "  2.99%",
                        "     _it",
                        "  1.94%",
                        "    _the",
                        "  1.81%",
                        "      \\n",
                        "  1.70%",
                        "     _or",
                        "  1.41%",
                        " _career",
                        "  1.13%",
                        "   _when",
                        "  0.98%",
                        "      _I",
                        "  0.92%",
                        "     _he",
                        "  0.78%"
                      ],
                      [
                        "     _is",
                        "  9.48%",
                        "    _was",
                        "  5.86%",
                        "    _has",
                        "  5.46%",
                        "       .",
                        "  3.33%",
                        "     âĢĻ",
                        "  3.25%",
                        "      's",
                        "  2.87%",
                        "     _in",
                        "  2.05%",
                        "       ,",
                        "  2.03%",
                        "    _may",
                        "  1.84%",
                        "    _and",
                        "  1.81%"
                      ],
                      [
                        "      _a",
                        " 12.85%",
                        "   _also",
                        "  3.88%",
                        "    _the",
                        "  2.02%",
                        "   _just",
                        "  2.00%",
                        "  _still",
                        "  1.65%",
                        "    _not",
                        "  1.40%",
                        "     _in",
                        "  1.31%",
                        "   _that",
                        "  1.24%",
                        "   _made",
                        "  1.22%",
                        "       ,",
                        "  0.94%"
                      ],
                      [
                        "  _right",
                        "  1.46%",
                        "   _best",
                        "  1.38%",
                        "   _fact",
                        "  0.90%",
                        "   _same",
                        "  0.88%",
                        "   _last",
                        "  0.78%",
                        "    _job",
                        "  0.60%",
                        "  _other",
                        "  0.58%",
                        "   _case",
                        "  0.55%",
                        " _respo…",
                        "  0.53%",
                        "  _first",
                        "  0.53%"
                      ],
                      [
                        "     _of",
                        "  6.04%",
                        "    _and",
                        "  4.06%",
                        "       ,",
                        "  3.48%",
                        "       .",
                        "  2.28%",
                        "    _one",
                        "  2.20%",
                        " _perfo…",
                        "  2.19%",
                        " _possi…",
                        "  2.06%",
                        "    _son",
                        "  1.91%",
                        "      .\"",
                        "  1.89%",
                        "   _time",
                        "  1.80%"
                      ],
                      [
                        "    _the",
                        " 12.04%",
                        "   _them",
                        "  4.71%",
                        "  _those",
                        "  3.75%",
                        "    _our",
                        "  2.60%",
                        "    _all",
                        "  2.20%",
                        " _things",
                        "  1.98%",
                        "   _this",
                        "  1.93%",
                        "   _that",
                        "  1.82%",
                        "   _your",
                        "  1.69%",
                        "     _it",
                        "  1.63%"
                      ],
                      [
                        "       .",
                        " 15.74%",
                        "     âĢĶ",
                        "  9.91%",
                        "    _and",
                        "  9.82%",
                        "   _when",
                        "  8.35%",
                        "       ,",
                        "  6.58%",
                        "  _being",
                        "  3.73%",
                        "   _that",
                        "  3.60%",
                        "    _for",
                        "  2.35%",
                        "     _in",
                        "  2.31%",
                        "   _with",
                        "  2.16%"
                      ]
                    ],
                    [
                      [
                        "     ual",
                        " 11.74%",
                        "      \\n",
                        "  6.30%",
                        "       \"",
                        "  5.62%",
                        "     ify",
                        "  2.91%",
                        "     _is",
                        "  2.15%",
                        "       .",
                        "  1.44%",
                        "      in",
                        "  1.27%",
                        "     ics",
                        "  1.22%",
                        "       '",
                        "  1.13%",
                        "       )",
                        "  1.02%"
                      ],
                      [
                        "      _a",
                        " 14.33%",
                        "  _super",
                        "  5.45%",
                        "   _made",
                        "  2.79%",
                        "    _not",
                        "  2.53%",
                        "   _used",
                        "  2.34%",
                        "    _the",
                        "  1.90%",
                        "   _just",
                        "  1.60%",
                        "   _also",
                        "  1.39%",
                        "      _\"",
                        "  1.33%",
                        "     _he",
                        "  1.21%"
                      ],
                      [
                        "  _right",
                        "  2.89%",
                        "   _last",
                        "  1.17%",
                        " _police",
                        "  1.08%",
                        "  _super",
                        "  0.84%",
                        " _favour",
                        "  0.78%",
                        "   _same",
                        "  0.75%",
                        "   _best",
                        "  0.75%",
                        "  _first",
                        "  0.68%",
                        " _source",
                        "  0.67%",
                        "    _pre",
                        "  0.59%"
                      ],
                      [
                        "    _bet",
                        "  5.01%",
                        "  _thing",
                        "  4.71%",
                        "     _of",
                        "  3.13%",
                        " _outco…",
                        "  2.93%",
                        " _perfo…",
                        "  2.82%",
                        "       ,",
                        "  2.62%",
                        " _result",
                        "  2.41%",
                        "    _and",
                        "  2.26%",
                        " _possi…",
                        "  1.98%",
                        "       -",
                        "  1.55%"
                      ],
                      [
                        "    _all",
                        "  7.77%",
                        "    _the",
                        "  6.98%",
                        "      _a",
                        "  4.15%",
                        "    _one",
                        "  2.99%",
                        "   _when",
                        "  2.68%",
                        "   _your",
                        "  2.37%",
                        "   _what",
                        "  2.32%",
                        "    _our",
                        "  2.27%",
                        " _compa…",
                        "  2.20%",
                        " _Chris…",
                        "  1.92%"
                      ],
                      [
                        "    _for",
                        " 22.59%",
                        "       ,",
                        " 14.26%",
                        "    _and",
                        " 12.72%",
                        "       .",
                        "  7.54%",
                        "     _to",
                        "  6.53%",
                        "   _when",
                        "  3.47%",
                        "   _that",
                        "  3.29%",
                        "       :",
                        "  2.57%",
                        "     _in",
                        "  2.06%",
                        "      .\"",
                        "  1.29%"
                      ],
                      [
                        "    _and",
                        " 14.34%",
                        "      _a",
                        "  3.10%",
                        "   _when",
                        "  2.96%",
                        "      _I",
                        "  2.75%",
                        "    _but",
                        "  2.41%",
                        "     _or",
                        "  2.41%",
                        "     _it",
                        "  2.21%",
                        "    _the",
                        "  2.02%",
                        "     _in",
                        "  1.77%",
                        "     _so",
                        "  1.62%"
                      ],
                      [
                        "     _is",
                        " 14.70%",
                        "      's",
                        " 13.80%",
                        "    _was",
                        " 10.27%",
                        "    _has",
                        "  7.49%",
                        "     âĢĻ",
                        "  6.44%",
                        "   _took",
                        "  2.40%",
                        "  _would",
                        "  1.76%",
                        "    _had",
                        "  1.72%",
                        "  _seems",
                        "  1.36%",
                        "   _will",
                        "  1.12%"
                      ],
                      [
                        "      _a",
                        " 11.45%",
                        "   _also",
                        "  3.75%",
                        "    _not",
                        "  2.93%",
                        "    _the",
                        "  2.73%",
                        "   _just",
                        "  2.51%",
                        "   _that",
                        "  2.29%",
                        "  _super",
                        "  1.75%",
                        "     _so",
                        "  1.67%",
                        " _actua…",
                        "  1.57%",
                        "  _still",
                        "  1.47%"
                      ],
                      [
                        "   _last",
                        "  4.22%",
                        "  _right",
                        "  3.76%",
                        "   _best",
                        "  1.70%",
                        "   _same",
                        "  1.04%",
                        "  _first",
                        "  0.95%",
                        "      \\n",
                        "  0.95%",
                        "   _fact",
                        "  0.95%",
                        "   _time",
                        "  0.79%",
                        " _respo…",
                        "  0.73%",
                        " _police",
                        "  0.68%"
                      ],
                      [
                        "     _of",
                        "  8.95%",
                        "       ,",
                        "  6.92%",
                        "       .",
                        "  6.50%",
                        "    _and",
                        "  4.84%",
                        "      .\"",
                        "  4.58%",
                        " _possi…",
                        "  2.96%",
                        "  _thing",
                        "  2.54%",
                        " _perfo…",
                        "  2.06%",
                        "    _for",
                        "  1.73%",
                        "     _in",
                        "  1.32%"
                      ],
                      [
                        "   _them",
                        " 12.41%",
                        "    _all",
                        " 11.83%",
                        "    _the",
                        "  9.59%",
                        "   _your",
                        "  4.30%",
                        "      _a",
                        "  3.30%",
                        "     _my",
                        "  2.73%",
                        "  _their",
                        "  2.39%",
                        "    _his",
                        "  2.33%",
                        "    _one",
                        "  1.78%",
                        "    _our",
                        "  1.67%"
                      ],
                      [
                        "       .",
                        " 15.32%",
                        "       ,",
                        " 14.80%",
                        "    _for",
                        "  8.14%",
                        "    _and",
                        "  7.25%",
                        "   _when",
                        "  6.76%",
                        "     _to",
                        "  5.15%",
                        "   _that",
                        "  2.10%",
                        "     _in",
                        "  1.81%",
                        "     âĢĶ",
                        "  1.74%",
                        " _becau…",
                        "  1.63%"
                      ]
                    ],
                    [
                      [
                        "      \\n",
                        " 14.64%",
                        "       .",
                        "  1.66%",
                        "    _the",
                        "  1.58%",
                        "    _The",
                        "  1.22%",
                        "       \"",
                        "  0.82%",
                        "       ,",
                        "  0.78%",
                        "       -",
                        "  0.77%",
                        "      _a",
                        "  0.76%",
                        "        ",
                        "  0.66%",
                        "       _",
                        "  0.58%"
                      ],
                      [
                        "      _a",
                        " 20.40%",
                        "    _the",
                        "  2.90%",
                        " _actua…",
                        "  2.56%",
                        "     _so",
                        "  2.40%",
                        "       ,",
                        "  2.37%",
                        "   _held",
                        "  2.10%",
                        "     _an",
                        "  1.80%",
                        "    _not",
                        "  1.76%",
                        "   _said",
                        "  1.76%",
                        "  _found",
                        "  1.36%"
                      ],
                      [
                        "   _last",
                        "  5.26%",
                        "   _best",
                        "  3.95%",
                        "      \\n",
                        "  3.48%",
                        "  _third",
                        "  3.36%",
                        "  _right",
                        "  2.35%",
                        "  _first",
                        "  2.04%",
                        "   _same",
                        "  1.96%",
                        "   _Lord",
                        "  1.14%",
                        " _second",
                        "  1.08%",
                        "   _text",
                        "  1.05%"
                      ],
                      [
                        "  _thing",
                        " 13.03%",
                        "     _of",
                        "  7.38%",
                        "  _place",
                        "  4.02%",
                        "    _for",
                        "  3.89%",
                        "   _time",
                        "  3.73%",
                        "       ,",
                        "  3.28%",
                        "    _bet",
                        "  3.21%",
                        "    _and",
                        "  2.66%",
                        " _possi…",
                        "  2.20%",
                        "    _way",
                        "  2.03%"
                      ],
                      [
                        "    _all",
                        " 31.71%",
                        "   _both",
                        " 19.10%",
                        "    _the",
                        " 11.33%",
                        "    _his",
                        "  6.95%",
                        "    _our",
                        "  3.08%",
                        "  _their",
                        "  1.80%",
                        "   _when",
                        "  1.77%",
                        "      _a",
                        "  1.73%",
                        "   _East",
                        "  1.29%",
                        " _course",
                        "  0.81%"
                      ],
                      [
                        "       ,",
                        " 44.70%",
                        "    _and",
                        " 13.27%",
                        "       .",
                        " 12.21%",
                        "    _for",
                        "  3.39%",
                        "     _to",
                        "  3.26%",
                        "   _that",
                        "  2.63%",
                        "   _when",
                        "  2.37%",
                        "     _on",
                        "  1.29%",
                        " _during",
                        "  1.12%",
                        "       :",
                        "  0.90%"
                      ],
                      [
                        "    _and",
                        " 12.56%",
                        "      _I",
                        "  8.54%",
                        "    _the",
                        "  4.61%",
                        "      \\n",
                        "  2.79%",
                        "     _he",
                        "  2.72%",
                        "     _it",
                        "  2.45%",
                        "   _when",
                        "  2.38%",
                        "    _but",
                        "  2.36%",
                        "      _a",
                        "  2.32%",
                        "   _they",
                        "  2.28%"
                      ],
                      [
                        "     _is",
                        " 21.04%",
                        "    _was",
                        " 16.15%",
                        "  _seems",
                        "  9.34%",
                        "    _has",
                        "  5.91%",
                        "      's",
                        "  5.22%",
                        "     âĢĻ",
                        "  3.39%",
                        " _becom…",
                        "  1.96%",
                        "  _tells",
                        "  1.76%",
                        "   _took",
                        "  1.70%",
                        "  _doesn",
                        "  1.64%"
                      ],
                      [
                        "      _a",
                        " 11.30%",
                        "    _not",
                        "  4.34%",
                        "   _easy",
                        "  3.21%",
                        "    _the",
                        "  2.87%",
                        "     _so",
                        "  2.66%",
                        " _actua…",
                        "  2.52%",
                        "   _just",
                        "  2.17%",
                        "   _also",
                        "  2.17%",
                        "  _never",
                        "  2.14%",
                        "       ,",
                        "  2.09%"
                      ],
                      [
                        "   _last",
                        " 11.81%",
                        "   _best",
                        "  7.99%",
                        "  _thing",
                        "  3.25%",
                        "      \\n",
                        "  2.70%",
                        "   _only",
                        "  2.40%",
                        "  _first",
                        "  2.31%",
                        "  _right",
                        "  2.22%",
                        "   _very",
                        "  2.14%",
                        " _easie…",
                        "  2.13%",
                        "   _same",
                        "  1.77%"
                      ],
                      [
                        "     _of",
                        "  9.85%",
                        "       ,",
                        "  9.79%",
                        "  _thing",
                        "  9.54%",
                        " _night…",
                        "  6.58%",
                        "       .",
                        "  5.67%",
                        " _possi…",
                        "  5.23%",
                        "    _and",
                        "  4.63%",
                        "   _when",
                        "  3.17%",
                        "     _he",
                        "  2.23%",
                        "      .\"",
                        "  2.17%"
                      ],
                      [
                        "    _all",
                        " 51.96%",
                        "    _the",
                        " 15.00%",
                        "   _both",
                        "  2.93%",
                        "   _when",
                        "  2.37%",
                        " _every…",
                        "  1.72%",
                        "      _a",
                        "  1.69%",
                        "    _his",
                        "  1.51%",
                        "   _them",
                        "  1.22%",
                        "     _my",
                        "  0.99%",
                        "    _our",
                        "  0.94%"
                      ],
                      [
                        "       ,",
                        " 43.10%",
                        "       .",
                        " 19.82%",
                        "    _and",
                        "  5.94%",
                        "   _when",
                        "  4.17%",
                        "     _to",
                        "  3.25%",
                        "     âĢĶ",
                        "  2.36%",
                        "   _that",
                        "  2.20%",
                        "    _for",
                        "  1.88%",
                        "  _being",
                        "  1.51%",
                        "     _it",
                        "  0.95%"
                      ]
                    ],
                    [
                      [
                        "      \\n",
                        " 13.07%",
                        "    _The",
                        "  1.66%",
                        "    _the",
                        "  1.22%",
                        "       .",
                        "  0.99%",
                        "      _a",
                        "  0.98%",
                        "        ",
                        "  0.84%",
                        "    _any",
                        "  0.64%",
                        "       >",
                        "  0.60%",
                        "       -",
                        "  0.60%",
                        "       \"",
                        "  0.52%"
                      ],
                      [
                        "      _a",
                        " 17.91%",
                        "       ,",
                        "  3.62%",
                        "    _the",
                        "  3.29%",
                        "     _so",
                        "  2.57%",
                        "     _as",
                        "  1.58%",
                        "   _just",
                        "  1.53%",
                        "     _he",
                        "  1.47%",
                        "   _made",
                        "  1.18%",
                        "     _in",
                        "  1.17%",
                        "   _also",
                        "  1.16%"
                      ],
                      [
                        "   _last",
                        "  7.72%",
                        "   _best",
                        "  2.32%",
                        "      \\n",
                        "  1.83%",
                        "  _right",
                        "  1.58%",
                        "   _same",
                        "  1.41%",
                        "  _first",
                        "  1.35%",
                        " _double",
                        "  1.21%",
                        "      _a",
                        "  1.13%",
                        "    _one",
                        "  1.08%",
                        " _absol…",
                        "  1.06%"
                      ],
                      [
                        " _perfo…",
                        " 12.01%",
                        "  _place",
                        " 10.36%",
                        "   _time",
                        "  7.27%",
                        "       ,",
                        "  6.75%",
                        "    _for",
                        "  6.45%",
                        "     _of",
                        "  5.88%",
                        "  _thing",
                        "  5.52%",
                        "    _and",
                        "  5.39%",
                        " _perfo…",
                        "  2.05%",
                        "     _in",
                        "  1.71%"
                      ],
                      [
                        "    _the",
                        " 27.29%",
                        "   _both",
                        " 11.50%",
                        "    _all",
                        "  8.88%",
                        "   _when",
                        "  7.35%",
                        "   _time",
                        "  3.42%",
                        "      _a",
                        "  3.06%",
                        "   _East",
                        "  2.08%",
                        "   _guys",
                        "  1.81%",
                        "    _his",
                        "  1.61%",
                        "       ,",
                        "  1.49%"
                      ],
                      [
                        "       ,",
                        " 25.34%",
                        "    _and",
                        " 22.41%",
                        "       .",
                        " 13.39%",
                        "   _when",
                        "  7.93%",
                        "    _for",
                        "  4.22%",
                        "     _to",
                        "  2.71%",
                        "    ,âĢĿ",
                        "  1.67%",
                        "    .âĢĿ",
                        "  1.65%",
                        "      .\"",
                        "  0.93%",
                        "     âĢĶ",
                        "  0.82%"
                      ],
                      [
                        "    _and",
                        " 12.07%",
                        "      _I",
                        "  8.79%",
                        "     _he",
                        "  6.11%",
                        "      \\n",
                        "  3.63%",
                        "    _the",
                        "  2.96%",
                        "   _they",
                        "  2.91%",
                        "   _when",
                        "  2.84%",
                        "      _a",
                        "  2.34%",
                        "    _but",
                        "  1.60%",
                        "     _as",
                        "  1.52%"
                      ],
                      [
                        "     _is",
                        " 14.34%",
                        "    _was",
                        " 14.32%",
                        "     âĢĻ",
                        "  9.27%",
                        "      's",
                        "  8.58%",
                        "    _has",
                        "  4.14%",
                        "  _seems",
                        "  2.98%",
                        " _becom…",
                        "  1.29%",
                        "   _took",
                        "  1.25%",
                        "   _goes",
                        "  1.02%",
                        " _should",
                        "  0.95%"
                      ],
                      [
                        "      _a",
                        " 13.19%",
                        "    _the",
                        "  5.95%",
                        "   _also",
                        "  3.80%",
                        "   _easy",
                        "  2.73%",
                        "   _when",
                        "  2.58%",
                        "   _just",
                        "  2.18%",
                        "       ,",
                        "  1.97%",
                        "    _too",
                        "  1.71%",
                        "     _no",
                        "  1.65%",
                        "    _not",
                        "  1.62%"
                      ],
                      [
                        "   _last",
                        " 17.76%",
                        "   _best",
                        "  9.18%",
                        "   _only",
                        "  5.95%",
                        " _easie…",
                        "  3.79%",
                        "  _first",
                        "  1.80%",
                        " _harde…",
                        "  1.28%",
                        "  _tears",
                        "  1.08%",
                        " _perfe…",
                        "  1.05%",
                        "   _dead",
                        "  0.73%",
                        " _medium",
                        "  0.71%"
                      ],
                      [
                        "     _of",
                        " 60.93%",
                        "       ,",
                        "  7.59%",
                        "    _and",
                        "  5.31%",
                        "       .",
                        "  3.62%",
                        "     _in",
                        "  2.45%",
                        "   _time",
                        "  1.63%",
                        "    _for",
                        "  1.43%",
                        "   _when",
                        "  1.33%",
                        "   _that",
                        "  1.33%",
                        "     _on",
                        "  1.12%"
                      ],
                      [
                        "   _time",
                        " 48.28%",
                        "  _times",
                        " 37.28%",
                        "   _when",
                        "  3.03%",
                        " _things",
                        "  1.51%",
                        "    _all",
                        "  1.33%",
                        " _season",
                        "  1.03%",
                        "   _days",
                        "  0.48%",
                        " _events",
                        "  0.43%",
                        "    _day",
                        "  0.41%",
                        "   _guys",
                        "  0.27%"
                      ],
                      [
                        "       ,",
                        " 50.42%",
                        "       .",
                        " 19.46%",
                        "   _when",
                        "  5.71%",
                        "    _and",
                        "  5.51%",
                        "     âĢĶ",
                        "  2.98%",
                        "     _to",
                        "  2.25%",
                        "    _for",
                        "  0.98%",
                        "       ;",
                        "  0.83%",
                        "      .\"",
                        "  0.81%",
                        " _becau…",
                        "  0.78%"
                      ]
                    ],
                    [
                      [
                        "      \\n",
                        " 11.62%",
                        "    _The",
                        "  1.42%",
                        "        ",
                        "  0.92%",
                        "       \"",
                        "  0.90%",
                        "        ",
                        "  0.84%",
                        "       [",
                        "  0.73%",
                        "       -",
                        "  0.67%",
                        "       .",
                        "  0.61%",
                        "       (",
                        "  0.59%",
                        " _career",
                        "  0.54%"
                      ],
                      [
                        "      _a",
                        " 17.55%",
                        "       ,",
                        "  4.21%",
                        "    _the",
                        "  4.04%",
                        "   _just",
                        "  1.63%",
                        "   _when",
                        "  1.57%",
                        "     _as",
                        "  1.41%",
                        "    _not",
                        "  1.31%",
                        "     _in",
                        "  1.23%",
                        "   _over",
                        "  1.16%",
                        "  _found",
                        "  1.15%"
                      ],
                      [
                        "   _last",
                        "  8.23%",
                        "  _right",
                        "  3.25%",
                        "      \\n",
                        "  1.71%",
                        "      _a",
                        "  1.55%",
                        "   _only",
                        "  1.54%",
                        "   _best",
                        "  1.37%",
                        "      _P",
                        "  1.23%",
                        " _immed…",
                        "  1.09%",
                        "  _thing",
                        "  1.06%",
                        " _perfe…",
                        "  1.01%"
                      ],
                      [
                        "  _thing",
                        " 12.14%",
                        "     _of",
                        "  6.27%",
                        " _perfo…",
                        "  5.50%",
                        "    _for",
                        "  5.35%",
                        "    _and",
                        "  4.87%",
                        "       ,",
                        "  4.42%",
                        "       .",
                        "  3.24%",
                        "       -",
                        "  2.87%",
                        "   _time",
                        "  2.51%",
                        " _resou…",
                        "  2.34%"
                      ],
                      [
                        "    _the",
                        " 19.44%",
                        "   _hell",
                        " 14.54%",
                        "   _time",
                        "  9.86%",
                        "      _a",
                        "  6.84%",
                        "    _all",
                        "  6.35%",
                        "   _when",
                        "  5.09%",
                        "      \\n",
                        "  4.84%",
                        "   _both",
                        "  3.68%",
                        "       ,",
                        "  1.98%",
                        "    _our",
                        "  1.55%"
                      ],
                      [
                        "       ,",
                        " 24.01%",
                        "       .",
                        " 18.64%",
                        "    _and",
                        " 18.32%",
                        "   _when",
                        "  3.61%",
                        "    ,âĢĿ",
                        "  2.43%",
                        "     _to",
                        "  2.24%",
                        "    _for",
                        "  2.13%",
                        "       !",
                        "  1.65%",
                        "     _it",
                        "  1.41%",
                        "    _too",
                        "  1.10%"
                      ],
                      [
                        "    _and",
                        " 12.28%",
                        "   _when",
                        "  5.75%",
                        "      _I",
                        "  5.29%",
                        "      \\n",
                        "  3.56%",
                        "    _but",
                        "  3.26%",
                        "    _the",
                        "  3.09%",
                        "   _they",
                        "  2.80%",
                        "      _a",
                        "  2.18%",
                        "    _too",
                        "  1.93%",
                        "     _it",
                        "  1.63%"
                      ],
                      [
                        "    _was",
                        " 16.73%",
                        "      's",
                        " 10.77%",
                        "     âĢĻ",
                        "  6.21%",
                        "     _is",
                        "  4.90%",
                        "    _has",
                        "  2.07%",
                        "  _seems",
                        "  1.95%",
                        "   _took",
                        "  1.76%",
                        " _should",
                        "  1.50%",
                        "  _could",
                        "  1.42%",
                        "      \\n",
                        "  1.39%"
                      ],
                      [
                        "      _a",
                        " 12.24%",
                        "    _the",
                        "  9.26%",
                        "    _not",
                        "  2.16%",
                        "   _last",
                        "  1.99%",
                        "   _when",
                        "  1.88%",
                        "   _also",
                        "  1.68%",
                        "    _too",
                        "  1.66%",
                        "     _to",
                        "  1.64%",
                        "   _that",
                        "  1.56%",
                        "       .",
                        "  1.52%"
                      ],
                      [
                        "   _last",
                        " 32.34%",
                        "   _best",
                        " 25.97%",
                        " _easie…",
                        "  3.68%",
                        "  _early",
                        "  1.30%",
                        "   _only",
                        "  1.01%",
                        " _bigge…",
                        "  0.88%",
                        " _harde…",
                        "  0.88%",
                        " _middle",
                        "  0.84%",
                        "  _great",
                        "  0.70%",
                        "    _one",
                        "  0.52%"
                      ],
                      [
                        "     _of",
                        " 52.15%",
                        "   _time",
                        " 21.43%",
                        "       .",
                        "  4.38%",
                        "       ,",
                        "  3.68%",
                        "    _and",
                        "  2.01%",
                        "      _I",
                        "  1.52%",
                        "  _thing",
                        "  1.17%",
                        "   _last",
                        "  0.70%",
                        "    _for",
                        "  0.67%",
                        "      .\"",
                        "  0.61%"
                      ],
                      [
                        "  _times",
                        " 56.90%",
                        "   _time",
                        " 41.84%",
                        "   _days",
                        "  0.35%",
                        "   times",
                        "  0.14%",
                        " _things",
                        "  0.12%",
                        " _events",
                        "  0.07%",
                        "  _games",
                        "  0.07%",
                        " _season",
                        "  0.06%",
                        "    _day",
                        "  0.06%",
                        "   _life",
                        "  0.03%"
                      ],
                      [
                        "       ,",
                        " 49.48%",
                        "       .",
                        " 19.91%",
                        "    _and",
                        "  4.22%",
                        "    ,âĢĿ",
                        "  3.37%",
                        "     âĢĶ",
                        "  2.73%",
                        "   _when",
                        "  1.71%",
                        " _becau…",
                        "  1.23%",
                        "      _I",
                        "  1.14%",
                        "    .âĢĿ",
                        "  1.09%",
                        "    _too",
                        "  1.05%"
                      ]
                    ],
                    [
                      [
                        "      \\n",
                        " 11.87%",
                        "       .",
                        "  2.26%",
                        "    _The",
                        "  1.39%",
                        "   }}})$",
                        "  1.21%",
                        "       ,",
                        "  1.20%",
                        "       )",
                        "  1.11%",
                        "     _is",
                        "  0.97%",
                        "      _a",
                        "  0.92%",
                        "        ",
                        "  0.91%",
                        "     _He",
                        "  0.91%"
                      ],
                      [
                        "      _a",
                        " 23.39%",
                        "    _the",
                        "  2.49%",
                        "       ,",
                        "  1.97%",
                        "    _not",
                        "  1.66%",
                        "   _made",
                        "  1.60%",
                        "   _also",
                        "  1.56%",
                        "     _an",
                        "  1.37%",
                        "    _non",
                        "  1.24%",
                        "   _over",
                        "  1.21%",
                        "   _when",
                        "  1.09%"
                      ],
                      [
                        "   _last",
                        "  5.80%",
                        "  _right",
                        "  2.01%",
                        "   _only",
                        "  1.67%",
                        "    _one",
                        "  1.63%",
                        "      _a",
                        "  1.35%",
                        "   _same",
                        "  1.25%",
                        " _immed…",
                        "  1.11%",
                        "   _best",
                        "  1.09%",
                        " _fourth",
                        "  1.03%",
                        "      \\n",
                        "  1.00%"
                      ],
                      [
                        "  _thing",
                        " 12.97%",
                        "   _time",
                        "  5.72%",
                        "       ,",
                        "  4.38%",
                        "     _of",
                        "  4.37%",
                        "       .",
                        "  3.02%",
                        "    _and",
                        "  2.85%",
                        "      .\"",
                        "  2.72%",
                        "    _for",
                        "  2.56%",
                        "       -",
                        "  2.54%",
                        " _resou…",
                        "  2.49%"
                      ],
                      [
                        "    _the",
                        " 25.57%",
                        "   _both",
                        " 21.35%",
                        "    _all",
                        "  9.35%",
                        "      _a",
                        "  4.24%",
                        "   _time",
                        "  2.88%",
                        "  _times",
                        "  2.66%",
                        "   _them",
                        "  2.53%",
                        "   _hell",
                        "  1.46%",
                        "    _his",
                        "  1.34%",
                        "  _their",
                        "  1.33%"
                      ],
                      [
                        "       ,",
                        " 39.31%",
                        "       .",
                        " 15.30%",
                        "    _and",
                        "  8.41%",
                        "     _to",
                        "  2.19%",
                        "    _for",
                        "  2.10%",
                        "      .\"",
                        "  1.89%",
                        "     _in",
                        "  1.89%",
                        "   _when",
                        "  1.49%",
                        "    ,âĢĿ",
                        "  1.37%",
                        "     _on",
                        "  1.01%"
                      ],
                      [
                        "    _and",
                        " 12.92%",
                        "   _when",
                        "  9.06%",
                        "     _he",
                        "  3.65%",
                        "      _I",
                        "  3.62%",
                        "     _it",
                        "  3.24%",
                        "    _the",
                        "  2.97%",
                        "      \\n",
                        "  2.93%",
                        "     _so",
                        "  2.09%",
                        "     _at",
                        "  2.05%",
                        "      _a",
                        "  1.98%"
                      ],
                      [
                        "    _was",
                        " 50.15%",
                        "     âĢĻ",
                        "  5.63%",
                        "      's",
                        "  3.13%",
                        "   _took",
                        "  2.82%",
                        "    _had",
                        "  1.82%",
                        "     _is",
                        "  1.62%",
                        "    _did",
                        "  1.40%",
                        "  _could",
                        "  1.34%",
                        "   _came",
                        "  1.23%",
                        " _stayed",
                        "  0.93%"
                      ],
                      [
                        "      _a",
                        " 20.40%",
                        "    _the",
                        " 12.83%",
                        "    _not",
                        "  5.05%",
                        "   _also",
                        "  3.17%",
                        "    _all",
                        "  1.81%",
                        "   _that",
                        "  1.74%",
                        "     _to",
                        "  1.68%",
                        "   _last",
                        "  1.65%",
                        "    _one",
                        "  1.49%",
                        "    _too",
                        "  1.30%"
                      ],
                      [
                        "   _last",
                        " 31.59%",
                        "   _best",
                        " 21.69%",
                        "  _great",
                        "  4.01%",
                        "  _right",
                        "  1.56%",
                        "   _good",
                        "  1.48%",
                        " _middle",
                        "  1.37%",
                        "   _most",
                        "  1.34%",
                        "   _only",
                        "  1.10%",
                        "  _early",
                        "  1.02%",
                        "  _Grand",
                        "  0.97%"
                      ],
                      [
                        "     _of",
                        " 45.44%",
                        "       ,",
                        " 14.04%",
                        "       .",
                        " 10.13%",
                        "   _time",
                        "  3.48%",
                        "    _and",
                        "  2.68%",
                        "   _last",
                        "  1.92%",
                        "      .\"",
                        "  1.78%",
                        "     _in",
                        "  1.58%",
                        "   _that",
                        "  1.31%",
                        "    _for",
                        "  1.23%"
                      ],
                      [
                        "  _times",
                        " 93.07%",
                        "   _time",
                        "  5.01%",
                        "   _days",
                        "  0.66%",
                        "  _games",
                        "  0.29%",
                        " _seaso…",
                        "  0.26%",
                        "   times",
                        "  0.12%",
                        " _season",
                        "  0.08%",
                        "  _downs",
                        "  0.03%",
                        "   _when",
                        "  0.03%",
                        "  _heart",
                        "  0.02%"
                      ],
                      [
                        "       ,",
                        " 56.89%",
                        "       .",
                        " 21.37%",
                        "    ,âĢĿ",
                        "  2.94%",
                        "    _and",
                        "  2.04%",
                        "     âĢĶ",
                        "  1.59%",
                        "    .âĢĿ",
                        "  1.55%",
                        "      .\"",
                        "  1.25%",
                        "   _that",
                        "  1.12%",
                        "    _too",
                        "  0.64%",
                        "     _in",
                        "  0.60%"
                      ]
                    ],
                    [
                      [
                        "      \\n",
                        "  9.52%",
                        "       .",
                        "  2.17%",
                        "       =",
                        "  1.39%",
                        "      00",
                        "  1.28%",
                        "       ,",
                        "  1.27%",
                        "      _I",
                        "  1.14%",
                        "    _The",
                        "  1.07%",
                        "     ati",
                        "  1.01%",
                        "        ",
                        "  0.83%",
                        "        ",
                        "  0.83%"
                      ],
                      [
                        "      _a",
                        " 16.65%",
                        "    _the",
                        "  3.32%",
                        "       ,",
                        "  2.65%",
                        "   _also",
                        "  2.09%",
                        "      \\n",
                        "  1.93%",
                        "    _not",
                        "  1.29%",
                        "     _an",
                        "  1.16%",
                        "   _when",
                        "  1.05%",
                        "    _non",
                        "  0.98%",
                        "   _just",
                        "  0.93%"
                      ],
                      [
                        "   _last",
                        "  6.78%",
                        "   _only",
                        "  3.39%",
                        "   _same",
                        "  3.25%",
                        "   _fact",
                        "  2.67%",
                        " _result",
                        "  1.89%",
                        "    _one",
                        "  1.85%",
                        " _probl…",
                        "  1.34%",
                        "  _right",
                        "  1.30%",
                        " _diffe…",
                        "  1.21%",
                        " _expre…",
                        "  0.99%"
                      ],
                      [
                        "  _thing",
                        " 28.57%",
                        "     _of",
                        "  3.00%",
                        "   _time",
                        "  2.78%",
                        "       ,",
                        "  2.44%",
                        "    _and",
                        "  2.26%",
                        "      .\"",
                        "  2.07%",
                        "       .",
                        "  1.96%",
                        " _perfo…",
                        "  1.72%",
                        "       -",
                        "  1.72%",
                        " _perfo…",
                        "  1.64%"
                      ],
                      [
                        "    _the",
                        " 34.28%",
                        "     _it",
                        " 12.63%",
                        "    _all",
                        "  8.82%",
                        "   _both",
                        "  7.56%",
                        "      _a",
                        "  3.86%",
                        "    _his",
                        "  2.84%",
                        "   _them",
                        "  2.45%",
                        "  _times",
                        "  1.79%",
                        "  _their",
                        "  1.63%",
                        "      \\n",
                        "  1.12%"
                      ],
                      [
                        "       ,",
                        " 39.71%",
                        "       .",
                        " 15.04%",
                        "    _and",
                        " 10.60%",
                        "    _for",
                        "  3.64%",
                        "   _when",
                        "  1.77%",
                        "     _to",
                        "  1.57%",
                        "      .\"",
                        "  1.54%",
                        "      \\n",
                        "  1.26%",
                        "     _in",
                        "  1.19%",
                        "   _that",
                        "  0.94%"
                      ],
                      [
                        "    _and",
                        " 13.02%",
                        "    _the",
                        "  7.87%",
                        "     _it",
                        "  7.66%",
                        "   _when",
                        "  5.21%",
                        "      _I",
                        "  4.19%",
                        "      _a",
                        "  3.44%",
                        "     _he",
                        "  3.26%",
                        "      \\n",
                        "  3.20%",
                        "    _was",
                        "  1.61%",
                        "     _so",
                        "  1.56%"
                      ],
                      [
                        "    _was",
                        " 37.93%",
                        "     âĢĻ",
                        " 15.09%",
                        "   _took",
                        "  5.22%",
                        "      's",
                        "  4.18%",
                        "     _is",
                        "  2.24%",
                        "    _had",
                        "  1.68%",
                        " _worked",
                        "  1.11%",
                        "  _could",
                        "  1.09%",
                        "    _did",
                        "  0.97%",
                        "    _has",
                        "  0.96%"
                      ],
                      [
                        "    _the",
                        " 34.04%",
                        "      _a",
                        "  8.53%",
                        "   _also",
                        "  4.41%",
                        "       ,",
                        "  2.09%",
                        "      \\n",
                        "  2.03%",
                        "    _one",
                        "  1.92%",
                        "       .",
                        "  1.49%",
                        "    _all",
                        "  1.34%",
                        "   _when",
                        "  1.20%",
                        "   _that",
                        "  1.13%"
                      ],
                      [
                        "   _best",
                        " 22.89%",
                        "   _last",
                        " 17.84%",
                        "   _most",
                        "  6.65%",
                        " _easie…",
                        "  3.62%",
                        "  _other",
                        "  2.72%",
                        "  _great",
                        "  2.50%",
                        "   _more",
                        "  2.19%",
                        "  _right",
                        "  1.84%",
                        "   _fact",
                        "  1.76%",
                        " _others",
                        "  1.52%"
                      ],
                      [
                        "     _of",
                        " 36.94%",
                        "       ,",
                        " 29.34%",
                        "       .",
                        " 11.79%",
                        "    _and",
                        "  3.82%",
                        "      .\"",
                        "  2.01%",
                        "   _time",
                        "  1.37%",
                        "    _for",
                        "  0.91%",
                        "   _when",
                        "  0.76%",
                        "      _I",
                        "  0.65%",
                        "      \\n",
                        "  0.62%"
                      ],
                      [
                        "  _times",
                        " 98.01%",
                        "   _time",
                        "  0.83%",
                        "   times",
                        "  0.55%",
                        "   _days",
                        "  0.15%",
                        "    _the",
                        "  0.05%",
                        "      _a",
                        "  0.03%",
                        "    _all",
                        "  0.02%",
                        "   _when",
                        "  0.02%",
                        "  _games",
                        "  0.02%",
                        "   _reck",
                        "  0.02%"
                      ],
                      [
                        "       ,",
                        " 68.11%",
                        "       .",
                        " 13.16%",
                        "    .âĢĿ",
                        "  4.04%",
                        "    _and",
                        "  2.81%",
                        "    ,âĢĿ",
                        "  2.44%",
                        "      .\"",
                        "  1.85%",
                        "     âĢĶ",
                        "  0.74%",
                        "    _for",
                        "  0.50%",
                        "      ._",
                        "  0.43%",
                        "    _âĢĶ",
                        "  0.29%"
                      ]
                    ],
                    [
                      [
                        "      \\n",
                        " 10.56%",
                        "       .",
                        "  2.34%",
                        "       ,",
                        "  1.32%",
                        "       )",
                        "  1.11%",
                        "    _The",
                        "  1.08%",
                        "   _that",
                        "  0.97%",
                        "        ",
                        "  0.91%",
                        "       =",
                        "  0.84%",
                        "     _to",
                        "  0.82%",
                        "     _He",
                        "  0.81%"
                      ],
                      [
                        "      _a",
                        " 12.90%",
                        "       ,",
                        "  3.00%",
                        "    _the",
                        "  2.75%",
                        "   _also",
                        "  2.53%",
                        "      \\n",
                        "  1.73%",
                        "   _when",
                        "  0.93%",
                        "    _all",
                        "  0.90%",
                        "     _so",
                        "  0.90%",
                        "    _not",
                        "  0.86%",
                        "     _in",
                        "  0.77%"
                      ],
                      [
                        "   _last",
                        "  5.37%",
                        "   _only",
                        "  3.36%",
                        "    _one",
                        "  2.19%",
                        "   _same",
                        "  1.47%",
                        "  _first",
                        "  1.41%",
                        " _respo…",
                        "  1.40%",
                        " _result",
                        "  1.31%",
                        " _easie…",
                        "  1.06%",
                        "   _fact",
                        "  1.06%",
                        "  _right",
                        "  0.96%"
                      ],
                      [
                        "  _thing",
                        " 14.88%",
                        "       ,",
                        "  3.53%",
                        "     _of",
                        "  3.04%",
                        "    _bet",
                        "  2.42%",
                        "      .\"",
                        "  2.19%",
                        "    _and",
                        "  1.81%",
                        "       .",
                        "  1.76%",
                        "      _I",
                        "  1.61%",
                        " _perfo…",
                        "  1.58%",
                        " _perfo…",
                        "  1.39%"
                      ],
                      [
                        "    _the",
                        " 29.92%",
                        "    _all",
                        " 12.18%",
                        "   _both",
                        "  6.89%",
                        "    _his",
                        "  5.15%",
                        "     _it",
                        "  4.09%",
                        "   _them",
                        "  4.00%",
                        "  _times",
                        "  3.26%",
                        "      _a",
                        "  2.87%",
                        "  _those",
                        "  1.66%",
                        "  _their",
                        "  1.46%"
                      ],
                      [
                        "       ,",
                        " 38.51%",
                        "       .",
                        " 16.85%",
                        "    _and",
                        "  7.31%",
                        "    _for",
                        "  4.86%",
                        "      .\"",
                        "  2.08%",
                        "   _when",
                        "  1.73%",
                        "      \\n",
                        "  1.69%",
                        "     _to",
                        "  1.69%",
                        "    ,âĢĿ",
                        "  1.04%",
                        "       ;",
                        "  1.00%"
                      ],
                      [
                        "    _and",
                        " 10.21%",
                        "    _the",
                        "  5.33%",
                        "      \\n",
                        "  4.84%",
                        "     _it",
                        "  4.07%",
                        "   _when",
                        "  3.83%",
                        "     _he",
                        "  3.42%",
                        "      _I",
                        "  3.26%",
                        "      _a",
                        "  3.26%",
                        "    _but",
                        "  2.73%",
                        "     _so",
                        "  2.13%"
                      ],
                      [
                        "    _was",
                        " 29.03%",
                        "     âĢĻ",
                        "  9.64%",
                        "   _took",
                        "  6.75%",
                        "     _is",
                        "  2.72%",
                        "      's",
                        "  2.70%",
                        "   _also",
                        "  1.48%",
                        " _should",
                        "  1.41%",
                        "    _has",
                        "  1.37%",
                        "    _did",
                        "  1.25%",
                        "      \\n",
                        "  1.11%"
                      ],
                      [
                        "    _the",
                        " 24.11%",
                        "      _a",
                        "  9.75%",
                        "   _also",
                        "  5.35%",
                        "    _all",
                        "  2.62%",
                        "      \\n",
                        "  2.04%",
                        "   _fair",
                        "  1.83%",
                        "       ,",
                        "  1.57%",
                        "    _one",
                        "  1.21%",
                        "   _easy",
                        "  1.18%",
                        "   _when",
                        "  0.83%"
                      ],
                      [
                        " _easie…",
                        " 10.45%",
                        "   _best",
                        "  8.28%",
                        "   _last",
                        "  6.20%",
                        "   _most",
                        "  5.34%",
                        "  _other",
                        "  5.13%",
                        "   _only",
                        "  4.27%",
                        " _others",
                        "  3.19%",
                        " _bigge…",
                        "  2.16%",
                        " _harde…",
                        "  2.00%",
                        "  _times",
                        "  1.92%"
                      ],
                      [
                        "       ,",
                        " 32.66%",
                        "     _of",
                        " 25.39%",
                        "       .",
                        " 17.05%",
                        "      .\"",
                        "  3.87%",
                        "    _and",
                        "  1.97%",
                        "    .âĢĿ",
                        "  1.29%",
                        "      _I",
                        "  1.25%",
                        "      \\n",
                        "  1.20%",
                        "   _when",
                        "  1.01%",
                        "       ;",
                        "  0.65%"
                      ],
                      [
                        "  _times",
                        " 99.71%",
                        "   times",
                        "  0.12%",
                        "   _days",
                        "  0.09%",
                        " _places",
                        "  0.02%",
                        " _stori…",
                        "  0.01%",
                        "  _games",
                        "  0.01%",
                        " _momen…",
                        "  0.00%",
                        " _revie…",
                        "  0.00%",
                        " _things",
                        "  0.00%",
                        " _situa…",
                        "  0.00%"
                      ],
                      [
                        "       ,",
                        " 58.90%",
                        "       .",
                        " 19.17%",
                        "      .\"",
                        "  5.30%",
                        "    .âĢĿ",
                        "  2.91%",
                        "    _and",
                        "  2.09%",
                        "    ,âĢĿ",
                        "  1.87%",
                        "    _for",
                        "  0.87%",
                        "      .'",
                        "  0.78%",
                        "    _âĢĵ",
                        "  0.70%",
                        "      ,\"",
                        "  0.58%"
                      ]
                    ],
                    [
                      [
                        "      \\n",
                        "  9.18%",
                        "       .",
                        "  2.58%",
                        "       ,",
                        "  2.16%",
                        "   }}})$",
                        "  1.24%",
                        "       )",
                        "  1.15%",
                        "    _The",
                        "  0.91%",
                        "        ",
                        "  0.83%",
                        "        ",
                        "  0.73%",
                        "   _that",
                        "  0.73%",
                        "      _a",
                        "  0.71%"
                      ],
                      [
                        "      _a",
                        "  7.59%",
                        "    _the",
                        "  1.79%",
                        "       ,",
                        "  1.59%",
                        "   _also",
                        "  1.57%",
                        "    _not",
                        "  1.26%",
                        " _actua…",
                        "  1.07%",
                        "     _so",
                        "  0.83%",
                        "   _that",
                        "  0.80%",
                        "    _all",
                        "  0.78%",
                        "      \\n",
                        "  0.74%"
                      ],
                      [
                        "   _last",
                        "  6.62%",
                        "   _only",
                        "  4.07%",
                        " _easie…",
                        "  2.14%",
                        "  _first",
                        "  2.02%",
                        " _respo…",
                        "  1.96%",
                        "    _one",
                        "  1.69%",
                        "   _same",
                        "  1.48%",
                        " _result",
                        "  1.44%",
                        "   _best",
                        "  1.32%",
                        " _begin…",
                        "  1.02%"
                      ],
                      [
                        "  _thing",
                        " 17.39%",
                        "     _of",
                        "  3.09%",
                        "       ,",
                        "  2.17%",
                        "      .\"",
                        "  2.17%",
                        "    _and",
                        "  1.55%",
                        " _choice",
                        "  1.54%",
                        " _option",
                        "  1.45%",
                        "    _job",
                        "  1.35%",
                        "       .",
                        "  1.35%",
                        "     _in",
                        "  0.94%"
                      ],
                      [
                        "    _the",
                        " 16.50%",
                        "  _times",
                        " 10.41%",
                        "    _all",
                        "  8.88%",
                        "   _both",
                        "  7.97%",
                        "     _it",
                        "  3.98%",
                        "   _them",
                        "  1.70%",
                        " _optio…",
                        "  1.68%",
                        "   _days",
                        "  1.58%",
                        "      _a",
                        "  1.47%",
                        "   _ones",
                        "  1.39%"
                      ],
                      [
                        "       ,",
                        " 29.66%",
                        "       .",
                        " 17.28%",
                        "    _and",
                        " 14.17%",
                        "    _for",
                        "  4.20%",
                        "      .\"",
                        "  2.22%",
                        "     _to",
                        "  1.60%",
                        "    ,âĢĿ",
                        "  1.34%",
                        "       :",
                        "  1.30%",
                        "       ;",
                        "  1.08%",
                        "      \\n",
                        "  0.99%"
                      ],
                      [
                        "    _and",
                        " 11.52%",
                        "    _the",
                        "  6.05%",
                        "     _it",
                        "  4.05%",
                        "    _but",
                        "  3.03%",
                        "   _when",
                        "  2.43%",
                        "     _so",
                        "  2.38%",
                        "      _a",
                        "  2.30%",
                        "      \\n",
                        "  2.03%",
                        "     _he",
                        "  1.99%",
                        "      _I",
                        "  1.85%"
                      ],
                      [
                        "    _was",
                        " 24.37%",
                        "   _took",
                        "  5.02%",
                        "     âĢĻ",
                        "  4.77%",
                        "     _is",
                        "  4.39%",
                        "      's",
                        "  2.39%",
                        "    _did",
                        "  1.97%",
                        "    _has",
                        "  1.92%",
                        " _worked",
                        "  1.75%",
                        " _should",
                        "  1.61%",
                        "   _didn",
                        "  1.43%"
                      ],
                      [
                        "    _the",
                        " 34.65%",
                        "      _a",
                        " 10.33%",
                        "   _also",
                        "  4.18%",
                        "    _all",
                        "  1.66%",
                        " _always",
                        "  1.01%",
                        "       ,",
                        "  0.91%",
                        "   _fair",
                        "  0.89%",
                        "     _on",
                        "  0.86%",
                        "      \\n",
                        "  0.80%",
                        "     _in",
                        "  0.73%"
                      ],
                      [
                        "   _best",
                        " 27.28%",
                        "   _most",
                        " 19.78%",
                        " _harde…",
                        " 10.72%",
                        " _easie…",
                        "  9.19%",
                        "   _last",
                        "  6.47%",
                        "  _worst",
                        "  3.81%",
                        " _bigge…",
                        "  1.88%",
                        "   _only",
                        "  1.38%",
                        "   _toug",
                        "  1.19%",
                        "    _saf",
                        "  1.01%"
                      ],
                      [
                        "     _of",
                        " 38.48%",
                        "       ,",
                        " 21.99%",
                        "       .",
                        " 15.38%",
                        "      .\"",
                        "  3.82%",
                        "    _and",
                        "  2.45%",
                        "    .âĢĿ",
                        "  0.87%",
                        "   _when",
                        "  0.74%",
                        "     _in",
                        "  0.71%",
                        "  _thing",
                        "  0.70%",
                        "     ...",
                        "  0.55%"
                      ],
                      [
                        "  _times",
                        " 97.96%",
                        "   _days",
                        "  1.44%",
                        "   times",
                        "  0.06%",
                        " _things",
                        "  0.06%",
                        " _condi…",
                        "  0.03%",
                        "   _time",
                        "  0.02%",
                        " _stori…",
                        "  0.02%",
                        " _circu…",
                        "  0.02%",
                        "    _the",
                        "  0.02%",
                        " _situa…",
                        "  0.02%"
                      ],
                      [
                        "       ,",
                        " 40.43%",
                        "       .",
                        " 23.50%",
                        "      .\"",
                        "  7.80%",
                        "    _and",
                        "  4.29%",
                        "    .âĢĿ",
                        "  3.50%",
                        "      .'",
                        "  2.65%",
                        "    ,âĢĿ",
                        "  1.74%",
                        "     ...",
                        "  1.50%",
                        "    _for",
                        "  1.06%",
                        "      ,\"",
                        "  1.01%"
                      ]
                    ],
                    [
                      [
                        "      \\n",
                        "  5.21%",
                        "       .",
                        "  2.39%",
                        "       _",
                        "  2.29%",
                        "       ,",
                        "  2.25%",
                        "   }}})$",
                        "  1.84%",
                        "       >",
                        "  0.91%",
                        "      =\"",
                        "  0.86%",
                        "       -",
                        "  0.74%",
                        "    _and",
                        "  0.73%",
                        "       (",
                        "  0.69%"
                      ],
                      [
                        "      _a",
                        " 10.92%",
                        "   _also",
                        "  2.18%",
                        "    _the",
                        "  2.14%",
                        "       ,",
                        "  1.35%",
                        "    _not",
                        "  1.33%",
                        " _actua…",
                        "  1.04%",
                        "   _that",
                        "  0.94%",
                        "     _in",
                        "  0.93%",
                        "   _told",
                        "  0.83%",
                        "     _so",
                        "  0.82%"
                      ],
                      [
                        "   _last",
                        "  4.49%",
                        "  _first",
                        "  2.26%",
                        "   _only",
                        "  2.19%",
                        " _result",
                        "  1.76%",
                        "    _one",
                        "  1.72%",
                        "   _best",
                        "  1.65%",
                        "   _same",
                        "  1.62%",
                        " _easie…",
                        "  1.42%",
                        " _respo…",
                        "  1.31%",
                        "  _right",
                        "  1.15%"
                      ],
                      [
                        "  _thing",
                        " 11.78%",
                        "     _of",
                        "  2.72%",
                        "       ,",
                        "  1.80%",
                        "    _day",
                        "  1.76%",
                        "       .",
                        "  1.76%",
                        " _perfo…",
                        "  1.57%",
                        "    _job",
                        "  1.51%",
                        "    _and",
                        "  1.49%",
                        " _resou…",
                        "  1.30%",
                        " _option",
                        "  1.26%"
                      ],
                      [
                        "    _the",
                        " 15.65%",
                        "  _times",
                        " 12.30%",
                        "   _both",
                        "  9.53%",
                        "    _all",
                        "  7.54%",
                        "     _it",
                        "  3.02%",
                        " _things",
                        "  2.27%",
                        "   _ones",
                        "  1.72%",
                        "   _days",
                        "  1.47%",
                        "  _those",
                        "  1.40%",
                        "    _his",
                        "  1.26%"
                      ],
                      [
                        "       ,",
                        " 31.06%",
                        "       .",
                        " 22.10%",
                        "    _and",
                        " 13.16%",
                        "    _for",
                        "  5.32%",
                        "     _to",
                        "  1.93%",
                        "     _in",
                        "  1.68%",
                        "   _when",
                        "  1.63%",
                        "       :",
                        "  1.15%",
                        "       ;",
                        "  1.12%",
                        " _becau…",
                        "  1.00%"
                      ],
                      [
                        "    _and",
                        " 20.01%",
                        "    _the",
                        "  8.08%",
                        "     _it",
                        "  4.07%",
                        "    _but",
                        "  4.02%",
                        "      _a",
                        "  3.05%",
                        "      _I",
                        "  2.53%",
                        "     _so",
                        "  2.01%",
                        "   _when",
                        "  1.87%",
                        "     _he",
                        "  1.54%",
                        "      \\n",
                        "  1.54%"
                      ],
                      [
                        "    _was",
                        " 31.42%",
                        "   _took",
                        "  7.09%",
                        "     _is",
                        "  5.21%",
                        "   _gave",
                        "  1.89%",
                        "      's",
                        "  1.65%",
                        "    _did",
                        "  1.51%",
                        "   _didn",
                        "  1.44%",
                        "    _has",
                        "  1.42%",
                        "   _made",
                        "  1.13%",
                        "   _also",
                        "  1.08%"
                      ],
                      [
                        "    _the",
                        " 38.33%",
                        "      _a",
                        " 14.09%",
                        "   _also",
                        "  3.11%",
                        " _always",
                        "  1.33%",
                        "   _fair",
                        "  1.25%",
                        "    _all",
                        "  0.73%",
                        "       ,",
                        "  0.72%",
                        "     _in",
                        "  0.65%",
                        "   _only",
                        "  0.64%",
                        "   _when",
                        "  0.62%"
                      ],
                      [
                        "   _best",
                        " 42.86%",
                        "   _most",
                        "  5.86%",
                        "   _last",
                        "  4.22%",
                        " _easie…",
                        "  4.10%",
                        " _harde…",
                        "  4.04%",
                        "  _worst",
                        "  2.19%",
                        "   _only",
                        "  2.17%",
                        "   _toug",
                        "  1.86%",
                        "    _saf",
                        "  1.78%",
                        " _bigge…",
                        "  1.55%"
                      ],
                      [
                        "     _of",
                        " 42.04%",
                        "       ,",
                        " 22.29%",
                        "       .",
                        " 18.75%",
                        "    _and",
                        "  2.39%",
                        "    _day",
                        "  1.02%",
                        "   _when",
                        "  0.99%",
                        "   _days",
                        "  0.68%",
                        "     _in",
                        "  0.64%",
                        "  _thing",
                        "  0.50%",
                        "   _time",
                        "  0.49%"
                      ],
                      [
                        "  _times",
                        " 94.82%",
                        "   _days",
                        "  2.82%",
                        " _things",
                        "  0.40%",
                        "    _the",
                        "  0.20%",
                        "  _terms",
                        "  0.12%",
                        "      \\n",
                        "  0.09%",
                        " _situa…",
                        "  0.06%",
                        "  _games",
                        "  0.06%",
                        " _circu…",
                        "  0.05%",
                        "       ,",
                        "  0.05%"
                      ],
                      [
                        "       ,",
                        " 50.90%",
                        "       .",
                        " 30.16%",
                        "    _and",
                        "  4.87%",
                        "    _for",
                        "  1.63%",
                        "      .\"",
                        "  1.51%",
                        "       ;",
                        "  1.05%",
                        "      \\n",
                        "  0.53%",
                        "     _in",
                        "  0.45%",
                        "     _to",
                        "  0.37%",
                        "   _when",
                        "  0.36%"
                      ]
                    ],
                    [
                      [
                        "       ,",
                        "  5.38%",
                        "       .",
                        "  5.12%",
                        "     _is",
                        "  4.88%",
                        "      's",
                        "  3.70%",
                        "     âĢĻ",
                        "  3.59%",
                        "       -",
                        "  2.61%",
                        "    _was",
                        "  2.12%",
                        "       _",
                        "  1.83%",
                        "       :",
                        "  0.80%",
                        "    _and",
                        "  0.77%"
                      ],
                      [
                        "      _a",
                        " 10.32%",
                        "    _not",
                        "  4.81%",
                        "    _the",
                        "  4.81%",
                        "     _an",
                        "  1.95%",
                        "   _also",
                        "  1.44%",
                        "       ,",
                        "  1.30%",
                        "     _in",
                        "  1.09%",
                        "   _just",
                        "  0.97%",
                        "   _that",
                        "  0.86%",
                        "   _only",
                        "  0.69%"
                      ],
                      [
                        "  _first",
                        "  7.10%",
                        "   _only",
                        "  4.80%",
                        "   _most",
                        "  3.96%",
                        "   _same",
                        "  3.39%",
                        "   _last",
                        "  2.72%",
                        "   _best",
                        "  2.24%",
                        " _result",
                        "  1.26%",
                        "    _one",
                        "  1.02%",
                        " _second",
                        "  0.92%",
                        "   _time",
                        "  0.79%"
                      ],
                      [
                        "     _of",
                        "  5.70%",
                        "  _thing",
                        "  4.87%",
                        "       .",
                        "  3.64%",
                        "       ,",
                        "  2.13%",
                        "       -",
                        "  2.08%",
                        "      _I",
                        "  1.78%",
                        "     _in",
                        "  1.57%",
                        "   _game",
                        "  1.36%",
                        "    _way",
                        "  1.29%",
                        "   _part",
                        "  1.28%"
                      ],
                      [
                        "    _the",
                        " 34.42%",
                        "  _times",
                        " 15.13%",
                        "   _them",
                        "  4.63%",
                        "    _all",
                        "  4.62%",
                        "     _it",
                        "  3.00%",
                        "   _both",
                        "  2.11%",
                        "    _his",
                        "  1.75%",
                        " _things",
                        "  1.23%",
                        "  _their",
                        "  1.20%",
                        "    _its",
                        "  1.02%"
                      ],
                      [
                        "       .",
                        " 33.86%",
                        "       ,",
                        " 22.67%",
                        "    _and",
                        "  8.32%",
                        "    _for",
                        "  7.30%",
                        "     _to",
                        "  1.91%",
                        "     _in",
                        "  1.40%",
                        "      .\"",
                        "  1.37%",
                        "       !",
                        "  1.23%",
                        "       ;",
                        "  1.19%",
                        "   _when",
                        "  1.19%"
                      ],
                      [
                        "    _and",
                        " 24.83%",
                        "    _but",
                        " 14.86%",
                        "    _the",
                        "  8.46%",
                        "     _it",
                        "  3.30%",
                        "      _I",
                        "  1.79%",
                        "     _he",
                        "  1.69%",
                        "   _with",
                        "  1.49%",
                        "     _as",
                        "  1.46%",
                        "   _when",
                        "  1.16%",
                        "     _we",
                        "  0.96%"
                      ],
                      [
                        "    _was",
                        " 49.94%",
                        "      's",
                        "  4.95%",
                        "     âĢĻ",
                        "  4.81%",
                        "     _is",
                        "  4.61%",
                        "   _wasn",
                        "  2.66%",
                        "    _had",
                        "  1.50%",
                        "   _didn",
                        "  1.21%",
                        "   _took",
                        "  1.17%",
                        "  _would",
                        "  1.14%",
                        " _seemed",
                        "  0.92%"
                      ],
                      [
                        "    _the",
                        " 40.88%",
                        "      _a",
                        "  8.04%",
                        "   _also",
                        "  3.94%",
                        "    _not",
                        "  1.10%",
                        "     _an",
                        "  1.05%",
                        " _always",
                        "  0.98%",
                        "   _easy",
                        "  0.90%",
                        "   _just",
                        "  0.87%",
                        "    _all",
                        "  0.82%",
                        "   _good",
                        "  0.73%"
                      ],
                      [
                        "   _best",
                        " 32.24%",
                        "  _worst",
                        " 25.11%",
                        "   _most",
                        "  9.21%",
                        "   _only",
                        "  2.57%",
                        "   _last",
                        "  1.71%",
                        " _harde…",
                        "  1.70%",
                        "  _other",
                        "  0.95%",
                        " _bigge…",
                        "  0.74%",
                        "   _time",
                        "  0.74%",
                        " _great…",
                        "  0.61%"
                      ],
                      [
                        "     _of",
                        " 52.97%",
                        "       .",
                        " 19.50%",
                        "       ,",
                        "  8.20%",
                        "   _time",
                        "  1.74%",
                        "      .\"",
                        "  1.53%",
                        "    _and",
                        "  1.22%",
                        "      \\n",
                        "  0.66%",
                        "     _in",
                        "  0.56%",
                        "   _when",
                        "  0.56%",
                        "       ;",
                        "  0.56%"
                      ],
                      [
                        "  _times",
                        " 95.99%",
                        "    _the",
                        "  0.52%",
                        "   _days",
                        "  0.25%",
                        "      \\n",
                        "  0.20%",
                        " _seaso…",
                        "  0.16%",
                        " _things",
                        "  0.13%",
                        "  _years",
                        "  0.10%",
                        " _momen…",
                        "  0.07%",
                        " _people",
                        "  0.07%",
                        " _places",
                        "  0.07%"
                      ],
                      [
                        "       .",
                        " 46.71%",
                        "       ,",
                        " 25.48%",
                        "      .\"",
                        "  5.95%",
                        "    _and",
                        "  3.45%",
                        "    _for",
                        "  2.58%",
                        "    .âĢĿ",
                        "  1.66%",
                        "       ;",
                        "  1.25%",
                        "      ,\"",
                        "  1.04%",
                        "     ...",
                        "  0.56%",
                        "       !",
                        "  0.52%"
                      ]
                    ]
                  ],
                  "hoverlabel": {
                    "bgcolor": "rgb(42, 42, 50)",
                    "font": {
                      "family": "Monospace"
                    }
                  },
                  "hovertemplate": "%{customdata[0]}%{customdata[1]}<br>%{customdata[2]}%{customdata[3]}<br>%{customdata[4]}%{customdata[5]}<br>%{customdata[6]}%{customdata[7]}<br>%{customdata[8]}%{customdata[9]}<br>%{customdata[10]}%{customdata[11]}<br>%{customdata[12]}%{customdata[13]}<br>%{customdata[14]}%{customdata[15]}<br>%{customdata[16]}%{customdata[17]}<br>%{customdata[18]}%{customdata[19]}<br><extra></extra>",
                  "text": [
                    [
                      ",",
                      "_a",
                      "_same",
                      ".",
                      "_the",
                      ",",
                      "_and",
                      "_is",
                      "_a",
                      "_same",
                      ",",
                      "_the",
                      ","
                    ],
                    [
                      "k",
                      "_a",
                      "\\n",
                      ".",
                      "_the",
                      "_and",
                      "_and",
                      "_is",
                      "_a",
                      "\\n",
                      "_of",
                      "_the",
                      "."
                    ],
                    [
                      "_",
                      "_a",
                      "_right",
                      "_possi…",
                      "_the",
                      "_and",
                      "_and",
                      "_is",
                      "_a",
                      "_right",
                      "_of",
                      "_the",
                      "."
                    ],
                    [
                      "ual",
                      "_a",
                      "_right",
                      "_bet",
                      "_all",
                      "_for",
                      "_and",
                      "_is",
                      "_a",
                      "_last",
                      "_of",
                      "_them",
                      "."
                    ],
                    [
                      "\\n",
                      "_a",
                      "_last",
                      "_thing",
                      "_all",
                      ",",
                      "_and",
                      "_is",
                      "_a",
                      "_last",
                      "_of",
                      "_all",
                      ","
                    ],
                    [
                      "\\n",
                      "_a",
                      "_last",
                      "_perfo…",
                      "_the",
                      ",",
                      "_and",
                      "_is",
                      "_a",
                      "_last",
                      "_of",
                      "_time",
                      ","
                    ],
                    [
                      "\\n",
                      "_a",
                      "_last",
                      "_thing",
                      "_the",
                      ",",
                      "_and",
                      "_was",
                      "_a",
                      "_last",
                      "_of",
                      "_times",
                      ","
                    ],
                    [
                      "\\n",
                      "_a",
                      "_last",
                      "_thing",
                      "_the",
                      ",",
                      "_and",
                      "_was",
                      "_a",
                      "_last",
                      "_of",
                      "_times",
                      ","
                    ],
                    [
                      "\\n",
                      "_a",
                      "_last",
                      "_thing",
                      "_the",
                      ",",
                      "_and",
                      "_was",
                      "_the",
                      "_best",
                      "_of",
                      "_times",
                      ","
                    ],
                    [
                      "\\n",
                      "_a",
                      "_last",
                      "_thing",
                      "_the",
                      ",",
                      "_and",
                      "_was",
                      "_the",
                      "_easie…",
                      ",",
                      "_times",
                      ","
                    ],
                    [
                      "\\n",
                      "_a",
                      "_last",
                      "_thing",
                      "_the",
                      ",",
                      "_and",
                      "_was",
                      "_the",
                      "_best",
                      "_of",
                      "_times",
                      ","
                    ],
                    [
                      "\\n",
                      "_a",
                      "_last",
                      "_thing",
                      "_the",
                      ",",
                      "_and",
                      "_was",
                      "_the",
                      "_best",
                      "_of",
                      "_times",
                      ","
                    ],
                    [
                      ",",
                      "_a",
                      "_first",
                      "_of",
                      "_the",
                      ".",
                      "_and",
                      "_was",
                      "_the",
                      "_best",
                      "_of",
                      "_times",
                      "."
                    ]
                  ],
                  "texttemplate": "<b>%{text}</b>",
                  "type": "heatmap",
                  "x": [
                    "it",
                    "_was‌",
                    "_the‌‌",
                    "_best‌‌‌",
                    "_of‌‌‌‌",
                    "_times‌‌‌‌‌",
                    ",‌‌‌‌‌‌",
                    "_it‌‌‌‌‌‌‌",
                    "_was‌‌‌‌‌‌‌‌",
                    "_the‌‌‌‌‌‌‌‌‌",
                    "_worst‌‌‌‌‌‌‌‌‌‌",
                    "_of‌‌‌‌‌‌‌‌‌‌‌",
                    "_times‌‌‌‌‌‌‌‌‌‌‌‌"
                  ],
                  "y": [
                    "0",
                    "1",
                    "2",
                    "3",
                    "4",
                    "5",
                    "6",
                    "7",
                    "8",
                    "9",
                    "10",
                    "11",
                    "output"
                  ],
                  "z": [
                    [
                      5.8909220695495605,
                      6.26378870010376,
                      7.4228973388671875,
                      6.6046013832092285,
                      6.288013935089111,
                      5.280069351196289,
                      6.3639421463012695,
                      5.17836856842041,
                      6.26378870010376,
                      7.4228973388671875,
                      6.297543525695801,
                      6.288013935089111,
                      5.280069351196289
                    ],
                    [
                      5.261128902435303,
                      6.205988883972168,
                      7.5773468017578125,
                      6.344673156738281,
                      6.883652210235596,
                      4.9236650466918945,
                      7.11436128616333,
                      4.984077453613281,
                      5.959990501403809,
                      7.533729076385498,
                      5.861849784851074,
                      6.098382949829102,
                      4.5414018630981445
                    ],
                    [
                      6.8805108070373535,
                      6.60089635848999,
                      7.571177005767822,
                      5.770522594451904,
                      5.869997501373291,
                      3.6200361251831055,
                      7.1237406730651855,
                      5.424977779388428,
                      6.0533833503723145,
                      7.473552703857422,
                      5.90798807144165,
                      5.539000511169434,
                      3.807018995285034
                    ],
                    [
                      5.744678020477295,
                      5.629146099090576,
                      7.321230888366699,
                      5.963696002960205,
                      5.469607353210449,
                      3.232449531555176,
                      5.67760705947876,
                      4.178005218505859,
                      5.705913066864014,
                      6.9584784507751465,
                      5.311422824859619,
                      4.786301136016846,
                      3.821739673614502
                    ],
                    [
                      6.841712951660156,
                      5.199743747711182,
                      6.3767547607421875,
                      4.937988758087158,
                      3.0058770179748535,
                      2.3771533966064453,
                      5.092628479003906,
                      3.6610565185546875,
                      5.378477573394775,
                      5.538779258728027,
                      4.402218341827393,
                      2.590442419052124,
                      2.425248384475708
                    ],
                    [
                      6.9633378982543945,
                      5.646194934844971,
                      6.51641321182251,
                      4.228825569152832,
                      3.657759428024292,
                      2.910261631011963,
                      5.30303430557251,
                      4.371655464172363,
                      5.396872520446777,
                      5.31761360168457,
                      2.113765239715576,
                      1.5849642753601074,
                      2.0486090183258057
                    ],
                    [
                      7.075319766998291,
                      5.663469314575195,
                      6.578805446624756,
                      4.894145488739014,
                      3.5810599327087402,
                      3.1404457092285156,
                      5.44074821472168,
                      4.742762088775635,
                      5.379448413848877,
                      3.555778741836548,
                      2.072054862976074,
                      0.7814419269561768,
                      2.166154623031616
                    ],
                    [
                      6.6494646072387695,
                      5.507911205291748,
                      6.729212284088135,
                      5.1333112716674805,
                      3.4127867221832275,
                      3.1131510734558105,
                      5.148497581481934,
                      3.255575180053711,
                      4.585875511169434,
                      3.6767690181732178,
                      2.5892908573150635,
                      0.34824851155281067,
                      1.8469456434249878
                    ],
                    [
                      6.32597017288208,
                      5.850934982299805,
                      6.339789390563965,
                      4.740157127380371,
                      3.3291640281677246,
                      2.9597792625427246,
                      4.904228210449219,
                      3.4538159370422363,
                      4.102084636688232,
                      3.935969829559326,
                      2.260867118835449,
                      0.14705528318881989,
                      1.4576054811477661
                    ],
                    [
                      6.488999843597412,
                      6.40374231338501,
                      6.8499979972839355,
                      5.850010871887207,
                      3.7260982990264893,
                      3.1229333877563477,
                      5.6737799644470215,
                      4.319716930389404,
                      4.8548173904418945,
                      4.953128814697266,
                      2.4418447017669678,
                      0.025771891698241234,
                      1.662613868713379
                    ],
                    [
                      6.709041118621826,
                      7.070559024810791,
                      6.671853542327881,
                      5.9567060470581055,
                      4.690244197845459,
                      3.3718810081481934,
                      6.02152156829834,
                      4.6320319175720215,
                      4.448858737945557,
                      2.927260398864746,
                      2.439546585083008,
                      0.13974793255329132,
                      2.36719012260437
                    ],
                    [
                      6.579579830169678,
                      6.749608516693115,
                      6.908118724822998,
                      6.286174774169922,
                      4.780987739562988,
                      3.035792589187622,
                      5.1085381507873535,
                      4.344996929168701,
                      3.98771595954895,
                      3.4110937118530273,
                      2.114767074584961,
                      0.35380133986473083,
                      1.8159735202789307
                    ],
                    [
                      6.978684425354004,
                      6.589774131774902,
                      6.962846755981445,
                      6.757388114929199,
                      3.9679970741271973,
                      2.8335583209991455,
                      4.289145469665527,
                      3.0917325019836426,
                      4.078311920166016,
                      3.2848899364471436,
                      2.131364345550537,
                      0.38901424407958984,
                      2.112548828125
                    ]
                  ],
                  "zmax": 7.5773468017578125,
                  "zmin": 0.025771891698241234
                }
              ],
              "layout": {
                "template": {
                  "data": {
                    "bar": [
                      {
                        "error_x": {
                          "color": "#2a3f5f"
                        },
                        "error_y": {
                          "color": "#2a3f5f"
                        },
                        "marker": {
                          "line": {
                            "color": "#E5ECF6",
                            "width": 0.5
                          },
                          "pattern": {
                            "fillmode": "overlay",
                            "size": 10,
                            "solidity": 0.2
                          }
                        },
                        "type": "bar"
                      }
                    ],
                    "barpolar": [
                      {
                        "marker": {
                          "line": {
                            "color": "#E5ECF6",
                            "width": 0.5
                          },
                          "pattern": {
                            "fillmode": "overlay",
                            "size": 10,
                            "solidity": 0.2
                          }
                        },
                        "type": "barpolar"
                      }
                    ],
                    "carpet": [
                      {
                        "aaxis": {
                          "endlinecolor": "#2a3f5f",
                          "gridcolor": "white",
                          "linecolor": "white",
                          "minorgridcolor": "white",
                          "startlinecolor": "#2a3f5f"
                        },
                        "baxis": {
                          "endlinecolor": "#2a3f5f",
                          "gridcolor": "white",
                          "linecolor": "white",
                          "minorgridcolor": "white",
                          "startlinecolor": "#2a3f5f"
                        },
                        "type": "carpet"
                      }
                    ],
                    "choropleth": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "type": "choropleth"
                      }
                    ],
                    "contour": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "contour"
                      }
                    ],
                    "contourcarpet": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "type": "contourcarpet"
                      }
                    ],
                    "heatmap": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "heatmap"
                      }
                    ],
                    "heatmapgl": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "heatmapgl"
                      }
                    ],
                    "histogram": [
                      {
                        "marker": {
                          "pattern": {
                            "fillmode": "overlay",
                            "size": 10,
                            "solidity": 0.2
                          }
                        },
                        "type": "histogram"
                      }
                    ],
                    "histogram2d": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "histogram2d"
                      }
                    ],
                    "histogram2dcontour": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "histogram2dcontour"
                      }
                    ],
                    "mesh3d": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "type": "mesh3d"
                      }
                    ],
                    "parcoords": [
                      {
                        "line": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "parcoords"
                      }
                    ],
                    "pie": [
                      {
                        "automargin": true,
                        "type": "pie"
                      }
                    ],
                    "scatter": [
                      {
                        "fillpattern": {
                          "fillmode": "overlay",
                          "size": 10,
                          "solidity": 0.2
                        },
                        "type": "scatter"
                      }
                    ],
                    "scatter3d": [
                      {
                        "line": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatter3d"
                      }
                    ],
                    "scattercarpet": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattercarpet"
                      }
                    ],
                    "scattergeo": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattergeo"
                      }
                    ],
                    "scattergl": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattergl"
                      }
                    ],
                    "scattermapbox": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scattermapbox"
                      }
                    ],
                    "scatterpolar": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatterpolar"
                      }
                    ],
                    "scatterpolargl": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatterpolargl"
                      }
                    ],
                    "scatterternary": [
                      {
                        "marker": {
                          "colorbar": {
                            "outlinewidth": 0,
                            "ticks": ""
                          }
                        },
                        "type": "scatterternary"
                      }
                    ],
                    "surface": [
                      {
                        "colorbar": {
                          "outlinewidth": 0,
                          "ticks": ""
                        },
                        "colorscale": [
                          [
                            0,
                            "#0d0887"
                          ],
                          [
                            0.1111111111111111,
                            "#46039f"
                          ],
                          [
                            0.2222222222222222,
                            "#7201a8"
                          ],
                          [
                            0.3333333333333333,
                            "#9c179e"
                          ],
                          [
                            0.4444444444444444,
                            "#bd3786"
                          ],
                          [
                            0.5555555555555556,
                            "#d8576b"
                          ],
                          [
                            0.6666666666666666,
                            "#ed7953"
                          ],
                          [
                            0.7777777777777778,
                            "#fb9f3a"
                          ],
                          [
                            0.8888888888888888,
                            "#fdca26"
                          ],
                          [
                            1,
                            "#f0f921"
                          ]
                        ],
                        "type": "surface"
                      }
                    ],
                    "table": [
                      {
                        "cells": {
                          "fill": {
                            "color": "#EBF0F8"
                          },
                          "line": {
                            "color": "white"
                          }
                        },
                        "header": {
                          "fill": {
                            "color": "#C8D4E3"
                          },
                          "line": {
                            "color": "white"
                          }
                        },
                        "type": "table"
                      }
                    ]
                  },
                  "layout": {
                    "annotationdefaults": {
                      "arrowcolor": "#2a3f5f",
                      "arrowhead": 0,
                      "arrowwidth": 1
                    },
                    "autotypenumbers": "strict",
                    "coloraxis": {
                      "colorbar": {
                        "outlinewidth": 0,
                        "ticks": ""
                      }
                    },
                    "colorscale": {
                      "diverging": [
                        [
                          0,
                          "#8e0152"
                        ],
                        [
                          0.1,
                          "#c51b7d"
                        ],
                        [
                          0.2,
                          "#de77ae"
                        ],
                        [
                          0.3,
                          "#f1b6da"
                        ],
                        [
                          0.4,
                          "#fde0ef"
                        ],
                        [
                          0.5,
                          "#f7f7f7"
                        ],
                        [
                          0.6,
                          "#e6f5d0"
                        ],
                        [
                          0.7,
                          "#b8e186"
                        ],
                        [
                          0.8,
                          "#7fbc41"
                        ],
                        [
                          0.9,
                          "#4d9221"
                        ],
                        [
                          1,
                          "#276419"
                        ]
                      ],
                      "sequential": [
                        [
                          0,
                          "#0d0887"
                        ],
                        [
                          0.1111111111111111,
                          "#46039f"
                        ],
                        [
                          0.2222222222222222,
                          "#7201a8"
                        ],
                        [
                          0.3333333333333333,
                          "#9c179e"
                        ],
                        [
                          0.4444444444444444,
                          "#bd3786"
                        ],
                        [
                          0.5555555555555556,
                          "#d8576b"
                        ],
                        [
                          0.6666666666666666,
                          "#ed7953"
                        ],
                        [
                          0.7777777777777778,
                          "#fb9f3a"
                        ],
                        [
                          0.8888888888888888,
                          "#fdca26"
                        ],
                        [
                          1,
                          "#f0f921"
                        ]
                      ],
                      "sequentialminus": [
                        [
                          0,
                          "#0d0887"
                        ],
                        [
                          0.1111111111111111,
                          "#46039f"
                        ],
                        [
                          0.2222222222222222,
                          "#7201a8"
                        ],
                        [
                          0.3333333333333333,
                          "#9c179e"
                        ],
                        [
                          0.4444444444444444,
                          "#bd3786"
                        ],
                        [
                          0.5555555555555556,
                          "#d8576b"
                        ],
                        [
                          0.6666666666666666,
                          "#ed7953"
                        ],
                        [
                          0.7777777777777778,
                          "#fb9f3a"
                        ],
                        [
                          0.8888888888888888,
                          "#fdca26"
                        ],
                        [
                          1,
                          "#f0f921"
                        ]
                      ]
                    },
                    "colorway": [
                      "#636efa",
                      "#EF553B",
                      "#00cc96",
                      "#ab63fa",
                      "#FFA15A",
                      "#19d3f3",
                      "#FF6692",
                      "#B6E880",
                      "#FF97FF",
                      "#FECB52"
                    ],
                    "font": {
                      "color": "#2a3f5f"
                    },
                    "geo": {
                      "bgcolor": "white",
                      "lakecolor": "white",
                      "landcolor": "#E5ECF6",
                      "showlakes": true,
                      "showland": true,
                      "subunitcolor": "white"
                    },
                    "hoverlabel": {
                      "align": "left"
                    },
                    "hovermode": "closest",
                    "mapbox": {
                      "style": "light"
                    },
                    "paper_bgcolor": "white",
                    "plot_bgcolor": "#E5ECF6",
                    "polar": {
                      "angularaxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      },
                      "bgcolor": "#E5ECF6",
                      "radialaxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      }
                    },
                    "scene": {
                      "xaxis": {
                        "backgroundcolor": "#E5ECF6",
                        "gridcolor": "white",
                        "gridwidth": 2,
                        "linecolor": "white",
                        "showbackground": true,
                        "ticks": "",
                        "zerolinecolor": "white"
                      },
                      "yaxis": {
                        "backgroundcolor": "#E5ECF6",
                        "gridcolor": "white",
                        "gridwidth": 2,
                        "linecolor": "white",
                        "showbackground": true,
                        "ticks": "",
                        "zerolinecolor": "white"
                      },
                      "zaxis": {
                        "backgroundcolor": "#E5ECF6",
                        "gridcolor": "white",
                        "gridwidth": 2,
                        "linecolor": "white",
                        "showbackground": true,
                        "ticks": "",
                        "zerolinecolor": "white"
                      }
                    },
                    "shapedefaults": {
                      "line": {
                        "color": "#2a3f5f"
                      }
                    },
                    "ternary": {
                      "aaxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      },
                      "baxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      },
                      "bgcolor": "#E5ECF6",
                      "caxis": {
                        "gridcolor": "white",
                        "linecolor": "white",
                        "ticks": ""
                      }
                    },
                    "title": {
                      "x": 0.05
                    },
                    "xaxis": {
                      "automargin": true,
                      "gridcolor": "white",
                      "linecolor": "white",
                      "ticks": "",
                      "title": {
                        "standoff": 15
                      },
                      "zerolinecolor": "white",
                      "zerolinewidth": 2
                    },
                    "yaxis": {
                      "automargin": true,
                      "gridcolor": "white",
                      "linecolor": "white",
                      "ticks": "",
                      "title": {
                        "standoff": 15
                      },
                      "zerolinecolor": "white",
                      "zerolinewidth": 2
                    }
                  }
                },
                "title": {
                  "text": "TunedLens (EleutherAI/pythia-160m-deduped) entropy",
                  "x": 0.5
                },
                "width": 1240,
                "xaxis": {
                  "title": {
                    "text": "Input"
                  }
                },
                "yaxis": {
                  "title": {
                    "text": "Layer"
                  }
                }
              }
            }
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "import torch\n",
        "from tuned_lens.nn.lenses import TunedLens, LogitLens\n",
        "from tuned_lens import TunedLens\n",
        "# from tuned_lens_example import TunedLens, LogitLens\n",
        "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
        "\n",
        "from tuned_lens.plotting import PredictionTrajectory\n",
        "import ipywidgets as widgets\n",
        "from plotly import graph_objects as go\n",
        "\n",
        "# temp = torch.load(\"/Users/s2011847/.cache/huggingface/hub/spaces--AlignmentResearch--tuned-lens/snapshots/1ac7285852a22309f571c2555efc37375d0c4cda/lens/EleutherAI/pythia-410m-deduped/params.pt\", map_location=torch.device(\"cpu\"))\n",
        "\n",
        "device = torch.device('mps')\n",
        "# model_id_pythia = 'gpt2-large'\n",
        "# model_id_pythia = \"meta-llama/Meta-Llama-3-8B-Instruct\"\n",
        "model_id_pythia = \"EleutherAI/pythia-160m-deduped\"\n",
        "if \"llama\" not in model_id_pythia: cache_dir_pythia = \"./models/\" + model_id_pythia\n",
        "# To try a diffrent modle / lens check if the lens is avalible then modify this code\n",
        "model = AutoModelForCausalLM.from_pretrained(\n",
        "    pretrained_model_name_or_path=model_id_pythia,\n",
        "    cache_dir=cache_dir_pythia,\n",
        "    device_map=device,\n",
        "    )\n",
        "model = model.to(device)\n",
        "tokenizer = AutoTokenizer.from_pretrained(\n",
        "    pretrained_model_name_or_path=model_id_pythia,\n",
        "    cache_dir=cache_dir_pythia,\n",
        "    )\n",
        "tuned_lens = TunedLens.from_model_and_pretrained(model)\n",
        "tuned_lens = tuned_lens.to(device)\n",
        "logit_lens = LogitLens.from_model(model)\n",
        "\n",
        "\n",
        "def make_plot(lens, text, layer_stride, statistic, token_range):\n",
        "    input_ids = tokenizer.encode(text)\n",
        "    targets = input_ids[1:] + [tokenizer.eos_token_id]\n",
        "\n",
        "    if len(input_ids) == 0:\n",
        "        return widgets.Text(\"Please enter some text.\")\n",
        "    \n",
        "    if (token_range[0] == token_range[1]):\n",
        "        return widgets.Text(\"Please provide valid token range.\")\n",
        "    pred_traj = PredictionTrajectory.from_lens_and_model(\n",
        "        lens=lens,\n",
        "        model=model,\n",
        "        input_ids=input_ids,\n",
        "        tokenizer=tokenizer,\n",
        "        targets=targets,\n",
        "    ).slice_sequence(slice(*token_range))\n",
        "\n",
        "    return getattr(pred_traj, statistic)().stride(layer_stride).figure(\n",
        "        title=f\"{lens.__class__.__name__} ({model.name_or_path}) {statistic}\",\n",
        "    )\n",
        "\n",
        "style = {'description_width': 'initial'}\n",
        "statistic_wdg = widgets.Dropdown(\n",
        "    options=[\n",
        "        ('Entropy', 'entropy'),\n",
        "        ('Cross Entropy', 'cross_entropy'),\n",
        "        ('Forward KL', 'forward_kl'),\n",
        "    ],\n",
        "    description='Select Statistic:',\n",
        "    style=style,\n",
        ")\n",
        "text_wdg = widgets.Textarea(\n",
        "    description=\"Input Text\",\n",
        "    value=\"it was the best of times, it was the worst of times\",\n",
        ")\n",
        "lens_wdg = widgets.Dropdown(\n",
        "    options=[('Tuned Lens', tuned_lens), ('Logit Lens', logit_lens)],\n",
        "    description='Select Lens:',\n",
        "    style=style,\n",
        ")\n",
        "\n",
        "layer_stride_wdg = widgets.BoundedIntText(\n",
        "    value=1,\n",
        "    min=1,\n",
        "    max=10,\n",
        "    step=1,\n",
        "    description='Layer Stride:',\n",
        "    disabled=False\n",
        ")\n",
        "\n",
        "token_range_wdg = widgets.IntRangeSlider(\n",
        "    description='Token Range',\n",
        "    min=0,\n",
        "    max=1,\n",
        "    step=1,\n",
        "    style=style,\n",
        ")\n",
        "\n",
        "\n",
        "def update_token_range(*args):\n",
        "    token_range_wdg.max = len(tokenizer.encode(text_wdg.value))\n",
        "\n",
        "update_token_range()\n",
        "\n",
        "token_range_wdg.value = [0, token_range_wdg.max]\n",
        "text_wdg.observe(update_token_range, 'value')\n",
        "\n",
        "interact = widgets.interact.options(manual_name='Run Lens', manual=True)\n",
        "\n",
        "plot = interact(\n",
        "    make_plot,\n",
        "    text=text_wdg,\n",
        "    statistic=statistic_wdg,\n",
        "    lens=lens_wdg,\n",
        "    layer_stride=layer_stride_wdg,\n",
        "    token_range=token_range_wdg,\n",
        ")"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.12.5"
    },
    "vscode": {
      "interpreter": {
        "hash": "ca27fca65bbd5c56c827a2643e94bc7b2b551ee6ee2fe84566f2c789012bce4f"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}