finetuning-English-GPT2-any-language-Portuguese-HuggingFace-fastaiv2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Faster than training from scratch \n",
    "# Fine-tuning the English GPT-2 in any language with Hugging Face and fastai v2 (practical case with Portuguese)\n",
    "\n",
    "> Tutorial on how to use fastai v2 over Hugging Face's Transformers and Tokenizers libraries to fine-tune an English pre-trained transformer-based language model (GPT-2) to any language other than English"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Author: [Pierre Guillou](https://www.linkedin.com/in/pierreguillou)\n",
    "- Date: July 2020 (14/07/2020)\n",
    "- Post in medium: [Faster than training from scratch - Fine-tuning the English GPT-2 in any language with Hugging Face and fastai v2 (practical case with Portuguese)](https://medium.com/@pierre_guillou/faster-than-training-from-scratch-fine-tuning-the-english-gpt-2-in-any-language-with-hugging-f2ec05c98787)\n",
    "- Fast notebook: [finetuning-English-GPT2-any-language-Portuguese-HuggingFace-fastaiv2_FAST.ipynb](https://github.com/piegu/fastai-projects/blob/master/finetuning-English-GPT2-any-language-Portuguese-HuggingFace-fastaiv2_FAST.ipynb)\n",
    "- Hugging face model page of [GPorTuguese-2](https://huggingface.co/pierreguillou/gpt2-small-portuguese): a language model for Portuguese text generation (and more NLP tasks...)\n",
    "- Other posts in medium of the GPT-2 series: \n",
    "  - [NLP & fastai | GPT-2](https://medium.com/@pierre_guillou/nlp-fastai-gpt-2-16ee145a4a28)\n",
    "  - [Byte-level BPE, an universal tokenizer but...](https://medium.com/@pierre_guillou/byte-level-bpe-an-universal-tokenizer-but-aff932332ffe)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, instead of training from scratch, we will see how to fine-tune in just over a day, on one GPU and with a little more than 1GB of training data an English pre-trained [transformer](https://arxiv.org/abs/1706.03762)-based language model to any another language. \n",
    "\n",
    "As a practical case, we fine-tune to Portuguese the [English pre-trained GPT-2](https://github.com/openai/gpt-2) by wrapping the [Transformers](https://github.com/huggingface/transformers) and [Tokenizers](https://github.com/huggingface/tokenizers) libraries of Hugging Face into [fastai v2](https://github.com/fastai/fastai2). We thus create a new language model: [GPorTuguese-2](https://huggingface.co/pierreguillou/gpt2-small-portuguese), a language model for Portuguese text generation (and more NLP tasks...)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![The 3 main steps of fine-tuning the English GPT-2 to Portuguese with Hugging Face and fastai v2 (image edited from fast.ai NLP)](images/GPT2_tf_ft_approach.png \"The 3 main steps of fine-tuning the English GPT-2 to Portuguese with Hugging Face and fastai v2 (image edited from fast.ai NLP)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Texts generated by GPorTuguese-2 on Covid-19, Netflix, Artificial Intelligence and... unicorns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Texts generated by GPorTuguese-2 on Covid-19, Netflix, Artificial Intelligence and... unicorns](images/4generatedtexts.png \"Texts generated by GPorTuguese-2 on Covid-19, Netflix, Artificial Intelligence and... unicorns\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Acknowledgment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tutorial was made possible thanks to the computing power of the [AI Lab](https://www.linkedin.com/company/ailab-unb/) (University of Brasilia) to which I am attached as an Associate Researcher in NLP and the participation of its directors in the definition of the NLP strategy, Professors [Fabricio Ataides Braz](https://www.linkedin.com/in/fabricio-braz-b356457/) and [Nilton Correia da Silva](https://www.linkedin.com/in/nilton-silva-6097853/). Thank you so much!\n",
    "\n",
    "And special thanks to Sylvain Gugger for his [tutorial on Transformers and fastai v2](https://dev.fast.ai/tutorial.transformers) which is the basis of this tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of contents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [Overview](#Overview)\n",
    "- [Texts generated by GPorTuguese-2 on Covid-19, Netflix, Artificial Intelligence and... unicorns](#Texts-generated-by-GPorTuguese-2-on-Covid-19-Netflix-Artificial-Intelligence-and...-unicorns)\n",
    "- [Acknowledgment](#Acknowledgment)\n",
    "- [Post, notebooks, Web App and model download](#Post,-notebooks,-Web-App-and-model-download)\n",
    "- [Results](#Results)\n",
    "- [About the need for language models not just in English... and how to do it in real life](#About-the-need-for-language-models-not-just-in-English...-and-how-to-do-it-in-real-life)\n",
    "  - [(option 1) Fast pipeline to localize any transformer-based model to any language](#(option-1)-Fast-pipeline-to-localize-any-transformer-based-model-to-any-language)\n",
    "  - [(option 2) Fine-tuning of an existing pre-trained model](#(option-2)-Fine-tuning-of-an-existing-pre-trained-model)\n",
    "- [Why using fastai v2 over Hugging Face libraries to fine-tune a pre-trained transformer-based language model?](#Why-using-fastai-v2-over-Hugging-Face-libraries-to-fine-tune-a-pre-trained-transformer-based-language-model?)\n",
    "  - [Tokenizers and Transformers from Hugging Face](#Tokenizers-and-Transformers-from-Hugging-Face)\n",
    "  - [fastai v2](#fastai-v2)\n",
    "- [About the choice of GPT-2](#About-the-choice-of-GPT-2)\n",
    "- [References](#References)\n",
    "  - [GPT-2](#GPT-2)\n",
    "  - [Datasets in Portuguese](#Datasets-in-Portuguese)\n",
    "  - [Hugging Face](#Hugging-Face)\n",
    "  - [Pytorch, fastai & Transformers (Hugging Face)](#Pytorch,-fastai-&-Transformers-(Hugging-Face))\n",
    "- [Main coding steps to fine-tune a Hugging Face language model with fastai v2](#Main-coding-steps-to-fine-tune-a-Hugging-Face-language-model-with-fastai-v2)\n",
    "  - [1. Initialization](#Initialization)\n",
    "  - [2. Download Wikipedia in Portuguese](#2.-Download-Wikipedia-in-Portuguese)\n",
    "  - [3. Download a GPT-2 English pre-trained model and train a GPT-2 tokenizer with a vocab in Portuguese](#3.-Download-a-GPT-2-English-pre-trained-model-and-train-a-GPT-2-tokenizer-with-a-vocab-in-Portuguese)\n",
    "  - [4. Create a fastai tokenizer and update the embeddings matrix of the GPT-2 English pre-trained model](#4.-Create-a-fastai-tokenizer-and-update-the-embeddings-matrix-of-the-GPT-2-English-pre-trained-model)\n",
    "    - [4.1 GPT2TokenizerFast (imported GPT-2 tokenizer) --> fastai Tokenizer](#4.1-GPT2TokenizerFast-(imported-GPT-2-tokenizer)--->-fastai-Tokenizer)\n",
    "    - [4.2 Change vocab embeddings (wte matrix) in the GPT-2 pre-trained model to adapt to the Portuguese vocab](#4.2-Change-vocab-embeddings-(wte-matrix)-in-the-GPT-2-pre-trained-model-to-adapt-to-the-Portuguese-vocab)\n",
    "  - [5. Create fastai v2 Datasets and Dataloaders](#5.-Create-fastai-v2-Datasets-and-Dataloaders)\n",
    "    - [5.1 fastai v2 Datasets](#5.1-fastai-v2-Datasets)\n",
    "      - [Visualize Data](#Visualize-Data)\n",
    "      - [Sample (this allows us to quickly test our code)](#Sample-(this-allows-us-to-quickly-test-our-code))\n",
    "      - [All data](#All-data)\n",
    "      - [Check datasets](#Check-datasets)\n",
    "    - [5.2 fastai v2 Dataloaders](#5.2-fastai-v2-Dataloaders)\n",
    "- [Model sharing and uploading in the Hugging Face model hub](#Model-sharing-and-uploading-in-the-Hugging-Face-model-hub)\n",
    "- [Text Generation by our Portuguese GPT-2](#Text-Generation-by-our-Portuguese-GPT-2)\n",
    "  - [Famous OpenAI generated text about unicorns](#Famous-OpenAI-generated-text-about-unicorns)\n",
    "  - [Text generation techniques]()\n",
    "    - [Text Generation techniques](#Text-generation-techniques)\n",
    "      - [(Use case 1) Top-k sampling](#(Use-case-1)-Top-k-sampling)\n",
    "      - [(Use case 2) Top-p (nucleus) sampling](#(Use-case-2)-Top-p-(nucleus)-sampling)\n",
    "    - [Text n°1 | Famous OpenAI generated text about unicorns](#Text-n°1-|-Famous-OpenAI-generated-text-about-unicorns)\n",
    "    - [Text n°2 | Recent text on the coronavirus disease (Covid-19)](#Text-n°2-|-Recent-text-on-the-coronavirus-disease-(Covid-19))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Post, notebooks, Web App and model download"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main code of the tutorial is published in this post ([Faster than training from scratch - Fine-tuning the English GPT-2 in any language with Hugging Face and fastai v2 (practical case with Portuguese)](https://medium.com/@pierre_guillou/faster-than-training-from-scratch-fine-tuning-the-english-gpt-2-in-any-language-with-hugging-f2ec05c98787)), organized by paragraph. \n",
    "\n",
    "The complete code is in this notebook. However, as this notebook is very detailed, you can use this fast notebook finetuning-English-GPT2-any-language-Portuguese-HuggingFace-fastaiv2_FAST.ipynb (nbviewer version) if you just want to execute the code without explanation.\n",
    "\n",
    "In addition, our **GPorTuguese-2 (Portuguese GPT-2 small) , a language model for Portuguese text generation (and more NLP tasks...)**, is testable online in the [Hugging face model hub](https://huggingface.co/models) with all usage information at this address: https://huggingface.co/pierreguillou/gpt2-small-portuguese"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![GPorTuguese-2 (Portuguese GPT-2 small) , a language model for Portuguese text generation (and more NLP tasks...)](images/hfmh.png \"GPorTuguese-2 (Portuguese GPT-2 small) , a language model for Portuguese text generation (and more NLP tasks...)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a little more than a day (we only used one GPU NVIDIA V100 32GB; through a Distributed Data Parallel (DDP) training mode, we could have divided by three this time to 10 hours, just with 2 GPUs), **we got a loss of 3.17, an accuracy of 37.99% and a perplexity of 23.76** (see the validation results table below and explications about perplexity at the end of the paragraph). Happy!\n",
    "\n",
    "```\n",
    "+------------+------+----------+------------+----------+-----------+\n",
    "|   after    | loss | accuracy | perplexity |   time   | cumulative|\n",
    "| ... epochs |      |   (%)    |            | by epoch |    time   |\n",
    "+------------+------+----------+------------+----------+-----------+\n",
    "|      0     | 9.95 |    9.90  |  20950.94  | 00:00:00 | 00:00:00  |\n",
    "|      1     | 3.64 |   32.52  |     38.12  |  5:48:31 |  5:48:31  |\n",
    "|      2     | 3.30 |   36.29  |     27.16  |  5:38:18 | 11:26:49  |\n",
    "|      3     | 3.21 |   37.46  |     24.71  |  6:20:51 | 17:47:40  |\n",
    "|      4     | 3.19 |   37.74  |     24.21  |  6:06:29 | 23:54:09  |\n",
    "|      5     | 3.17 |   37.99  |     23.76  |  6:16:22 | 30:10:31  |\n",
    "+------------+------+----------+------------+----------+-----------+\n",
    "                 Fine-tuning of GPT-2 into Portuguese\n",
    "               Table of training and validation results\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Validation loss and accuracy of pre-trained English GPT-2 of Hugging Face fine-tuned to Portuguese by fastai v2](images/gpt2_loss_acc_finetuned_fastaiv2.png \"Validation loss and accuracy of pre-trained English GPT-2 of Hugging Face fine-tuned to Portuguese by fastai v2\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**After a huge gain at the end of the first epoch (see validation results graph below), the validation accuracy continues to improve until the end of training but less** (it goes to nearly 40%, that is considered a good performance for a language model - check these notebooks [nn-vietnamese.ipynb](https://github.com/fastai/course-nlp/blob/master/nn-vietnamese.ipynb) and [nn-turkish.ipynb](https://github.com/fastai/course-nlp/blob/master/nn-turkish.ipynb) from Jeremy Howard of fastai).\n",
    "\n",
    "To read more about these results, read the post [\"Faster than training from scratch - Fine-tuning the English GPT-2 in any language with Hugging Face and fastai v2 (practical case with Portuguese)\"](https://medium.com/@pierre_guillou/faster-than-training-from-scratch-fine-tuning-the-english-gpt-2-in-any-language-with-hugging-f2ec05c98787)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## About the need for language models not just in English... and how to do it in real life"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even if English is today the most spoken language in the world, **the world is multilingual**. It is therefore necessary to have **natural language models trained in all existing languages**, and not just in English, since these models constitute the essential basis for the training of models capable of performing a particular task in linguistics (classification, Q&A, synthesis, entity searches, etc.).\n",
    "\n",
    "However, if it is extremely simple and free to download a language model trained in English via in particular the [Transformers library](https://huggingface.co/transformers/) of Hugging Face, it is often much more difficult to find online a model trained in another language. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (option 1) Fast pipeline to localize any transformer-based model to any language"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The easiest way to get theses language-specific language models would be to **use a pipeline of existing pre-trained transformer-based models** like the following one:\n",
    "\n",
    "![pipeline of existing pre-trained transformer-based models with a translator one at the input and output (image edited from fast.ai NLP)](images/trans_tf.png \"pipeline of existing pre-trained transformer-based models with a translator one at the input and output (image edited from fast.ai NLP)\")\n",
    "\n",
    "For example, to obtain a Portuguese GPT-2, we could download from the [Transformers](https://github.com/huggingface/transformers) library of Hugging Face the [OpenAI GPT-2 pre-trained in English](https://huggingface.co/transformers/model_doc/gpt2.html) and the [MarianMT](https://huggingface.co/transformers/model_doc/marian.html) translator (we could also use [BART](https://huggingface.co/transformers/model_doc/bart.html) or [T5](https://huggingface.co/transformers/model_doc/t5.html) for the translation) in order to create the following pipeline:\n",
    "```\n",
    "                (input) Portuguese to English (MarianMT) \n",
    "                          >> English pre-trained language model (GPT-2) \n",
    "                                    >> (output) English to Portuguese (MarianMT)\n",
    "```\n",
    "\n",
    "So, for free and with only a few lines of code, we can get any language model in any language, and even any task-oriented NLP model (classification, Q&A, synthesis, entity searches, etc.) using the same pipeline. Not bad!\n",
    "\n",
    "We will find the code of this pipeline and examples of use for text generation in the post \"**Fast pipeline to localize any transformer-based model to any language**\".\n",
    "\n",
    "However, **the problem with this simple solution is that we depend on the quality of training of 2 pre-trained NLP models, which greatly increases the risk of losing the linguistic singularities and nuances of the desired language**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### (option 2) Fine-tuning of an existing pre-trained model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Therefore, it often becomes necessary to have to train its own language model.\n",
    "\n",
    "Nevertheless, training from scratch a powerful language model like [GPT-2](https://github.com/openai/gpt-2) or [GPT-3](https://github.com/openai/gpt-3) of OpenAI, [BART](https://arxiv.org/abs/1910.13461) of Facebook or [T5](https://ai.googleblog.com/2020/02/exploring-transfer-learning-with-t5.html) of Google requires tens or even hundreds of GB of text, which is impossible or difficult to find or requires power gigantic computing that only a few companies in the world have. For example,\n",
    "- [GPT-2 Extra-Large](https://openai.com/blog/gpt-2-1-5b-release/) (1.5 billion parameters) was trained on 40GB of WebText on [32 Cloud TPU v3](https://twitter.com/teradimich/status/1096232184875233280) for 1 week ([cost of 43.008 dollars](https://twitter.com/Smerity/status/1096268294942674946))\n",
    "- [CamemBERT, the BERT in French, was trained on 38GB of raw text on 256 GPUs (32 GB Tesla V100) for 1 day](https://github.com/huggingface/transformers/issues/1356#issuecomment-561691234)\n",
    "- [RoBERTa was pre-trained for 24 hours on 1,024 (full size, 32GB) V100s](https://github.com/huggingface/transformers/issues/1356#issuecomment-536187777)... and we are not talking about T5 or GPT-3 whose [computational cost was estimated at 4.6 million of dollars](https://lambdalabs.com/blog/demystifying-gpt-3/)! (\"*We are waiting for OpenAI to reveal more details about the training infrastructure and model implementation. But to put things into perspective, GPT-3 175B model required 3.14E23 FLOPS of computing for training. Even at theoretical 28 TFLOPS for V100 and lowest 3 year reserved cloud pricing we could find, this will take 355 GPU-years and cost 4.6M dollars for a single training run.*\")\n",
    "\n",
    "![NLP models through time, with their number of parameters (Image credit: TensorFlow blog)](images/NLPmodels.png \"NLP models through time, with their number of parameters (Image credit: TensorFlow blog)\")\n",
    "\n",
    "Thus, as it is easy to download a few GB of texts from an online language corpus ([Wikipedia](https://dumps.wikimedia.org/), [OSCAR](https://oscar-corpus.com/), [Common Crawl](https://commoncrawl.org/) for example) and rent a NVIDIA V100 GPU for $1.24 an hour ([GCP](https://cloud.google.com/), [AWS](https://aws.amazon.com/), [Azur](https://azure.microsoft.com/) for example), **it is more realistic for the majority of people and organizations wishing to use a language model other than English to fine-tune on few GB of texts a model already pre-trained in English** (i.e. fine-tuning a model obtained by Transfer Learning) using Deep Learning frameworks such as [TensorFlow](https://www.tensorflow.org/)+[Keras](https://keras.io/) or [PyTorch](https://pytorch.org/)+[fastai](https://dev.fast.ai/).\n",
    "\n",
    "This tutorial show how to implement this second option and you will find examples of use for text generation in the paragraph [Text Generation by our Portuguese GPT-2](##Text-Generation-by-our-Portuguese-GPT-2) at the end of this tutorial."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Why using fastai v2 over Hugging Face libraries to fine-tune a pre-trained transformer-based language model?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### Tokenizers and Transformers from Hugging Face"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The [Tokenizers](https://github.com/huggingface/tokenizers) and [Transformers](https://huggingface.co/transformers/) library from [Hugging Face](https://huggingface.co/) are today **the most up-to-date NLP libraries (Natural Language Processing)** used all over the world.\n",
    "\n",
    "Let's copy and paste the most important information from the [Transformers documentation](https://huggingface.co/transformers/):\n",
    "\n",
    "> Transformers (formerly known as pytorch-transformers and pytorch-pretrained-bert) provides general-purpose architectures (BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet...) for Natural Language Understanding (NLU) and Natural Language Generation (NLG) with over 32+ [pretrained models](https://huggingface.co/transformers/pretrained_models.html) in 100+ languages and deep interoperability between TensorFlow 2.0 and PyTorch\".\n",
    "\n",
    "> The library was designed with [two strong goals](https://huggingface.co/transformers/quickstart.html#philosophy) in mind:\n",
    "> - be as easy and fast to use as possible\n",
    "> - provide state-of-the-art models with performances as close as possible to the original models\n",
    "\n",
    "> The library is build around [three types of classes for each model](https://huggingface.co/transformers/quickstart.html#main-concepts):\n",
    "> - **model classes** like `BertModel` which are 20+ PyTorch models (`torch.nn.Modules`) that work with the pretrained weights provided in the library. In TF2, these are `tf.keras.Model`.\n",
    "> - **configuration classes** which store all the parameters required to build a model, like `BertConfig`. You don’t always need to instantiate these your-self. In particular, if you are using a pretrained model without any modification, creating the model will automatically take care of instantiating the configuration (which is part of the model).\n",
    "> - **tokenizer classes** which store the vocabulary for each model and provide methods for encoding/decoding strings in a list of token embeddings indices to be fed to a model, like `BertTokenizer`.\n",
    "\n",
    "> All these classes can be instantiated from pretrained instances: `from_pretrained()` let you instantiate a model/configuration/tokenizer from a pretrained version either provided by the library itself or stored locally (or on a server) by the user."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "### fastai v2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "However, as written in the [Philosophy](https://huggingface.co/transformers/quickstart.html#philosophy) paragraph of the [Quickstart](https://huggingface.co/transformers/quickstart.html) page:\n",
    "> the Transformers library is NOT a modular toolbox of building blocks for neural nets. If you want to extend/build-upon the library, just use regular Python/PyTorch modules and inherit from the base classes of the library to reuse functionalities like model loading/saving.\n",
    "\n",
    "Therefore, despite of the running py files published by Hugging Face (for example, the [run_language_modeling.py](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_language_modeling.py) for fine-tuning the library models for language modeling on a text file (GPT, GPT-2, BERT, RoBERTa)), **when it comes necessary to fine-tune a pre-trained model to another language and/or to another task, we need to use regular Python/PyTorch modules in order to apply Transfer Learning and fine-tuning modern techniques, in particular if we do not have a huge new training dataset**.\n",
    "\n",
    "Here is a non-exhaustive list of these fine-tuning techniques based on Transfer Learning:\n",
    "- **Learning rate finder** (method that helps finding the best learning rate to train the model)\n",
    "- **Mixed precision training** (some of the operations will be done in FP16, others in FP32 in order to speed up the training)\n",
    "- **Gradual unfreezing** (layers groups are defined allowing to decide the layers to be trained)\n",
    "- **1cycle policy** (the 1cycle policy was introduced by Leslie N. Smith et al. in <a href=\"https://arxiv.org/abs/1708.07120\">Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates</a>. It schedules the learning rate with a cosine annealing)\n",
    "- **Differential learning rates** (a specific learning rate is setup by layers group)\n",
    "- **Distributed training** (training distributed on different GPUs in order to speed up the training)\n",
    "\n",
    "Since **fastai v2 provides all of these powerful fine-tuning techniques**, this is a primary candidate library for training transformer-based language models pre-trained with the Tokenizers and Transformers libraries of Hugging Face."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## About the choice of GPT-2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "In order to demonstrate the feasibility of fine-tuning Hugging Face models via fastai v2, we had to choose an emblematic model of the [Transformer revolution](https://arxiv.org/abs/1706.03762) in the NLP since 2017.\n",
    "\n",
    "Thus, between the GPT-2 and [BERT](https://github.com/google-research/bert) models, we chose the GPT-2 model because it has strongly influenced minds beyond the circle of Deep Learning specialists in early 2019 by [writing texts of a quality level close to that of humans](https://openai.com/blog/better-language-models/#samples). Today \"exceeded\" in number of parameters and performance by more recent models like BART, T5 and of course GPT-3 (175 billion parameters!), it remains a reference and a model used in research and applications.\n",
    "For those you want to understand better how GPT-2 works, read the following posts:\n",
    "- [The Illustrated GPT-2 (Visualizing Transformer Language Models)](http://jalammar.github.io/illustrated-gpt2/)\n",
    "- [NLP & fastai | GPT-2](https://medium.com/@pierre_guillou/nlp-fastai-gpt-2-16ee145a4a28)\n",
    "\n",
    "**About the version of GPT-2**\n",
    "\n",
    "There are 3 versions of the GPT-2 model (look at the [transformers documentation](https://huggingface.co/transformers/pretrained_models.html) for more details). Here, **we use the small version**, the one with the smallest number of weights (124 millions, not 117 as written in the original paper) but you can change the model used by changing the content of `pretrained_weights` (if it's not a GPT2 model, you'll need to change the classes used for the model and the tokenizer of course)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "**More about GPT-2**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Source: https://huggingface.co/transformers/model_doc/gpt2.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "> OpenAI GPT-2 model was proposed in [Language Models are Unsupervised Multitask Learners](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) by Alec Radford*, Jeffrey Wu*, Rewon Child, David Luan, Dario Amodei** and Ilya Sutskever**. It’s a causal (unidirectional) transformer pre-trained using language modeling on a very large corpus of ~40 GB of text data.\n",
    "\n",
    "> The abstract from the paper is the following: *GPT-2 is a large transformer-based language model with 1.5 billion parameters, trained on a dataset[1] of 8 million web pages. GPT-2 is trained with a simple objective: predict the next word, given all of the previous words within some text. The diversity of the dataset causes this simple goal to contain naturally occurring demonstrations of many tasks across diverse domains. GPT-2 is a direct scale-up of GPT, with more than 10X the parameters and trained on more than 10X the amount of data.*\n",
    "\n",
    "> Tips:\n",
    "> - GPT-2 is a model with absolute position embeddings so it’s usually advised to pad the inputs on the right rather than the left.\n",
    "> - GPT-2 was trained with a causal language modeling (CLM) objective and is therefore powerful at predicting the next token in a sequence. Leveraging this feature allows GPT-2 to generate syntactically coherent text as it can be observed in the run_generation.py example script.\n",
    "> - The PyTorch models can take the past as input, which is the previously computed key/value attention pairs. Using this past value prevents the model from re-computing pre-computed values in the context of text generation. See [reusing the past in generative models](https://huggingface.co/transformers/quickstart.html#using-the-past) for more information on the usage of this argument.\n",
    "\n",
    "> [Write With Transformer](https://transformer.huggingface.co/doc/gpt2-large) is a webapp created and hosted by Hugging Face showcasing the generative capabilities of several models. GPT-2 is one of them and is available in five different sizes: small, medium, large, xl and a distilled version of the small checkpoint: distilgpt-2.\n",
    "\n",
    ">The original code can be found [here](https://openai.com/blog/better-language-models/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### GPT-2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "- Understanding\n",
    "  - [Better Language Models and Their Implications](https://openai.com/blog/better-language-models/) (OpenAI, 02/14/2019)\n",
    "  - [The Illustrated GPT-2 (Visualizing Transformer Language Models)](http://jalammar.github.io/illustrated-gpt2/)\n",
    "  - [The Annotated GPT-2](https://amaarora.github.io/2020/02/18/annotatedGPT2.html)\n",
    "  - [Understanding the GPT-2 Source Code](https://medium.com/analytics-vidhya/understanding-the-gpt-2-source-code-part-1-4481328ee10b)\n",
    "  - [How To Make Custom AI-Generated Text With GPT-2](https://minimaxir.com/2019/09/howto-gpt2/)\n",
    "- Online Apps\n",
    "  - [Write With Transformer (distilgpt2-small, gpt2small, gpt2medium, gpt2large)](https://transformer.huggingface.co/doc/gpt2-large)\n",
    "  - [Write With DistilGPT-2](https://transformer.huggingface.co/model/distil-gpt2)\n",
    "  - [Generate custom text from an AI using GPT-2 (using the 117M default model)](https://minimaxir.com/apps/gpt2-small/)\n",
    "  - [Allen GPT2 Large Demo](https://demo.allennlp.org/next-token-lm?text=AllenNLP%20is)\n",
    "- Others papers: [The Annotated Transformer](https://nlp.seas.harvard.edu/2018/04/03/attention.html), [Layer Normalization](https://arxiv.org/abs/1607.06450)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Datasets in Portuguese"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "- Wikipedia\n",
    "  - (fastai): code from [Vietnamese ULMFiT from scratch](https://github.com/fastai/course-nlp/blob/master/nn-vietnamese.ipynb)\n",
    "  - (Hugging Face): [code from nlp](https://huggingface.co/nlp/viewer/?dataset=wikipedia&config=20200501.pt)\n",
    "- [OSCAR corpus](https://traces1.inria.fr/oscar/): code from [Find a Dataset](https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb#scrollTo=oK7PPVm2XBgr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Hugging Face"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "- Dataset\n",
    "  - [nlp](https://github.com/huggingface/nlp)\n",
    "  - [Colab tutorial](https://colab.research.google.com/github/huggingface/nlp/blob/master/notebooks/Overview.ipynb)\n",
    "  - [Online dataset explorer](https://huggingface.co/nlp/viewer)\n",
    "- Tokenizers\n",
    "  - [Tokenizers](https://github.com/huggingface/tokenizers) (github)\n",
    "  - Source code\n",
    "    - [Source code for transformers.tokenization_gpt2](https://huggingface.co/transformers/_modules/transformers/tokenization_gpt2.html)\n",
    "    - [Source code for transformers.tokenization_utils_base](https://huggingface.co/transformers/_modules/transformers/tokenization_utils_base.html)\n",
    "    - [Source code for transformers.tokenization_utils](https://huggingface.co/transformers/_modules/transformers/tokenization_utils.html)\n",
    "    - [Source code for transformers.tokenization_utils_fast](https://huggingface.co/transformers/_modules/transformers/tokenization_utils_fast.html)\n",
    "    - [classmethod from_pretrained()](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.from_pretrained): Instantiate a PreTrainedTokenizer (or a derived class) from a predefined tokenizer.\n",
    "  - [Source code for transformers.tokenization_gpt2](https://huggingface.co/transformers/_modules/transformers/tokenization_gpt2.html)\n",
    "  - [Hugging Face Tutorials - Training Tokenizer](https://www.kaggle.com/funtowiczmo/hugging-face-tutorials-training-tokenizer)\n",
    "  - [Hugging Face Introduces Tokenizers](https://medium.com/dair-ai/hugging-face-introduces-tokenizers-d792482db360)\n",
    "  - How to train a new language model from scratch using Transformers and Tokenizers (05/15/2020): [blog post](https://huggingface.co/blog/how-to-train) & [colab notebook](https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb)\n",
    "  - [HuggingFace Tokenizers Cheat Sheet](https://www.kaggle.com/debanga/huggingface-tokenizers-cheat-sheet)\n",
    "  - [Tokenizers: How machines read](https://blog.floydhub.com/tokenization-nlp/) (01/28/2020)\n",
    "  - [Byte Pair Encoding](https://leimao.github.io/blog/Byte-Pair-Encoding/) (07/19/2019)\n",
    "  - [What is a tokenizer?](https://docs.rs/tokenizers/0.10.1/tokenizers/#what-is-a-tokenizer)\n",
    "- Transformers\n",
    "  - [Transformers](https://huggingface.co/transformers/) de Hugging Face & [Transformers github](https://github.com/huggingface/transformers)\n",
    "  - [Glossary](https://huggingface.co/transformers/glossary.html)\n",
    "  - [OpenAI GPT2](https://huggingface.co/transformers/model_doc/gpt2.html#openai-gpt2)\n",
    "  - Source code\n",
    "    - [Source code for transformers.modeling_gpt2](https://huggingface.co/transformers/_modules/transformers/modeling_gpt2.html)\n",
    "    - [Source code for transformers.configuration_gpt2](https://huggingface.co/transformers/_modules/transformers/configuration_gpt2.html)\n",
    "  - [DistilBERT](https://medium.com/huggingface/distilbert-8cf3380435b5), [DistilGPT2](https://huggingface.co/distilgpt2) & [Download Model: distilgpt2](https://huggingface.co/distilgpt2)\n",
    "  - [Train a GPT-2 Text-Generating Model w/ GPU For Free](https://colab.research.google.com/drive/1VLG8e7YSEwypxU-noRNhsv5dW4NfTGce#scrollTo=H7LoMj4GA4n_) (colab notebook, 11/10/2019)\n",
    "  - How to generate text: using different decoding methods for language generation with Transformers (03/18/2020, Hugging Face): [blog post](https://huggingface.co/blog/how-to-generate) and [colab notebook](https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/02_how_to_generate.ipynb) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "### Pytorch, fastai & Transformers (Hugging Face)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "- [Sequence-to-Sequence Modeling with nn.Transformer and TorchText](https://pytorch.org/tutorials/beginner/transformer_tutorial.html#sequence-to-sequence-modeling-with-nn-transformer-and-torchtext)\n",
    "- [Fastai v2](https://dev.fast.ai) (Deep Learning library on PyTorch) & [Hugging face](https://huggingface.co/)\n",
    "- [blurr](https://ohmeow.github.io/blurr/): a library that integrates huggingface transformers with version 2 of the fastai framework\n",
    "- fastai v2\n",
    "  - Integration of the GPT2 model into fastai v2: code from [Tutorial - Transformers](https://dev.fast.ai/tutorial.transformers) and [10_nlp.ipynb](https://github.com/fastai/fastbook/blob/master/10_nlp.ipynb) (how to fine-tune an NLP model with fastai v2)\n",
    "  - FastHugs\n",
    "    - [FastHugs in the fastai forum](https://forums.fast.ai/t/fasthugs-fastai-v2-and-huggingface-transformers/63681)\n",
    "    - [FastHugs: Language Modelling with Tranformers and Fastai](https://www.ntentional.com/nlp/transformers/training%20technique/classification/2020/04/24/fasthugs_language_model.html) (04/24/2020, fastai v2)\n",
    "    - [FastHugs: Sequence Classification with Transformers and Fastai](https://www.ntentional.com/nlp/training%20technique/classification/2020/04/17/fasthugs_seq_classification.html) (04/17/2020, fastai v2)\n",
    "- fastai v1\n",
    "  - [A Tutorial to Fine-Tuning BERT with Fast AI](http://mlexplained.com/2019/05/13/a-tutorial-to-fine-tuning-bert-with-fast-ai/) (05/15/2019, fastai v1)\n",
    "  - [Fastai integration with BERT: Multi-label text classification identifying toxicity in texts](https://medium.com/@abhikjha/fastai-integration-with-bert-a0a66b1cecbe) (07/17/2019, fastai v1)\n",
    "  - [When Pytorch-transformers meets Fastai (w/ Google Colab)](https://towardsdatascience.com/best-of-two-worlds-pytorch-transformers-meets-fastai-5fd51ef34b0f) (08/26/2019, fastai v1)\n",
    "  - [Using RoBERTa with Fastai for NLP](https://medium.com/analytics-vidhya/using-roberta-with-fastai-for-nlp-7ed3fed21f6c) (09/02/2019, fastai v1)\n",
    "  - [RoBERTa with Fastai](https://www.kaggle.com/abhikjha/roberta-with-fastai) (11/14/2019, fastai v1)\n",
    "  - [Fastai with 🤗Transformers (BERT, RoBERTa, XLNet, XLM, DistilBERT)](https://towardsdatascience.com/fastai-with-transformers-bert-roberta-xlnet-xlm-distilbert-4f41ee18ecb2) (11/27/2019, fastai v1): A tutorial to implement state-of-the-art NLP models with Fastai for Sentiment Analysis ([notebook](https://www.kaggle.com/maroberti/fastai-with-transformers-bert-roberta))\n",
    "  - [RoBERTa (fastai, HuggingFace 🤗Transformers)](https://www.kaggle.com/melissarajaram/roberta-fastai-huggingface-transformers/execution) (01/17/2020, fastai v1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Main coding steps to fine-tune a Hugging Face language model with fastai v2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The 6 main steps detailed below can be summarized in 3 main ones:\n",
    "\n",
    "1. **Initialization & download** (download of Portuguese Wikipedia and GPT-2 English pre-trained model and tokenizer)\n",
    "2. **GPT-2 tokenizer with a Portuguese vocab** (train a GPT-2 tokenizer with a vocab in Portuguese, wrap it into a fastai v2 tokenizer and update the embeddings matrix of the GPT-2 English pre-trained model according to the new Portuguese vocab: keep the embeddings vectors of the common tokens between English and Portuguese vocabs)\n",
    "3. **Fine-tune on Portuguese Wikipedia the GPT-2 model with fastai v2 training functionalities**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Initialization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "nbpresent": {
     "id": "151cd18f-76e3-440f-a8c7-ffa5c6b5da01"
    }
   },
   "outputs": [],
   "source": [
    "from fastai2.text.all import *\n",
    "from nlputils_fastai2 import * \n",
    "\n",
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "nbpresent": {
     "id": "6ceb4db2-e4cf-4fe0-a393-91df4a7ed3e7"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cuda device: 0\n"
     ]
    }
   ],
   "source": [
    "gpu = 0\n",
    "torch.cuda.set_device(gpu)\n",
    "print(f'cuda device: {torch.cuda.current_device()}')\n",
    "# print(f'cuda device name: {torch.cuda.get_device_name(gpu)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'archive_path': '/storage/archive/',\n",
       " 'data_path': '/storage/data/',\n",
       " 'model_path': '/storage/models/',\n",
       " 'storage_path': '/storage/data/',\n",
       " 'version': 2}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get config of paths\n",
    "config = Config()\n",
    "config.d"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "cf070ab7-babb-4cf0-a315-401f65461dc8"
    }
   },
   "source": [
    "This will create a `{lang}wiki` folder, containing a `{lang}wiki` text file with the wikipedia contents. (For other languages, replace `{lang}` with the appropriate code from the [list of wikipedias](https://meta.wikimedia.org/wiki/List_of_Wikipedias).)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "nbpresent": {
     "id": "70da588b-8af1-4f97-97c2-c9f2d4d46e1a"
    }
   },
   "outputs": [],
   "source": [
    "lang = 'pt'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "nbpresent": {
     "id": "701ab344-0430-4f43-bbe2-337a12cae6be"
    }
   },
   "outputs": [],
   "source": [
    "# setup new path_data and create the corresponding folder\n",
    "name = f'{lang}wiki'\n",
    "data_path = config['data_path']\n",
    "path_data = data_path/name\n",
    "path_data.mkdir(exist_ok=True, parents=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "bfe49910-58e0-4be3-aba1-7733dc18cca2"
    }
   },
   "source": [
    "### 2. Download Wikipedia in Portuguese"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### By fastai"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: all the following methods come from the file nlputils_fastai2.py."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Path('/mnt/home/pierre/course-v4/nbs'),\n",
       " Path('/mnt/home/pierre/.fastai/data/ptwiki'))"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Path.cwd(), path_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/mnt/home/pierre/.fastai/data/ptwiki/ptwiki already exists; not downloading\n"
     ]
    }
   ],
   "source": [
    "get_wiki(path_data,lang)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If `get_wiki(path_data,lang)` breaks, fix the download manually no terminal:\n",
    "- mkdir -p /mnt/home/pierre/.fastai/data/ptwiki\n",
    "- cd /mnt/home/pierre/.fastai/data/ptwiki\n",
    "- wget -c https://dumps.wikimedia.org/ptwiki/latest/ptwiki-latest-pages-articles.xml.bz2\n",
    "- bzip2 -dk ptwiki-latest-pages-articles.xml.bz2\n",
    "\n",
    "And re-run `get_wiki(path_data,lang)` once the download is successful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<doc id=\"220\" url=\"https://pt.wikipedia.org/wiki?curid=220\" title=\"Astronomia\">\r\n",
      "Astronomia\r\n",
      "\r\n",
      "Astronomia é uma ciência natural que estuda corpos celestes (como estrelas, planetas, cometas, nebulosas, aglomerados de estrelas, galáxias) e fenômenos que se originam fora da atmosfera da Terra (como a radiação cósmica de fundo em micro-ondas). Preocupada com a evolução, a física, a química e o movimento de objetos celestes, bem como a formação e o desenvolvimento do universo.\r\n"
     ]
    }
   ],
   "source": [
    "!head -n4 {path_data}/{name}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/mnt/home/pierre/.fastai/data/ptwiki/docs already exists; not splitting\n"
     ]
    }
   ],
   "source": [
    "dest = split_wiki(path_data,lang)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/mnt/home/pierre/.fastai/data/ptwiki/docs/Fotografia.txt\n",
      "/mnt/home/pierre/.fastai/data/ptwiki/docs/Espadanedo (Macedo de Cavaleiros).txt\n",
      "/mnt/home/pierre/.fastai/data/ptwiki/docs/Jacques-Germain Soufflot.txt\n",
      "/mnt/home/pierre/.fastai/data/ptwiki/docs/Faculdade de Medicina da Universidade de São Paulo.txt\n",
      "/mnt/home/pierre/.fastai/data/ptwiki/docs/Escola do Teatro Bolshoi no Brasil.txt\n"
     ]
    }
   ],
   "source": [
    "dest = path_data/'docs'\n",
    "for file in dest.ls()[:5]:\n",
    "    print(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "203205 files - 193686269 tokens\n",
      "CPU times: user 1min, sys: 20.8 s, total: 1min 21s\n",
      "Wall time: 11min 16s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Size of downloaded data in the docs folder\n",
    "num_files, num_tokens = get_num_tokens(dest)\n",
    "print(f'{num_files} files - {num_tokens} tokens')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### By Hugging Face"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[nlp](https://github.com/huggingface/nlp) is a lightweight and extensible library from Hugging Face to easily share and access datasets and evaluation metrics for Natural Language Processing (NLP)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**[ WARNING ]** We did try to use it in order to download Wikipedia in Portuguese but without success. However, to help people solving this issue, we decided to leave our code in this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# source: https://huggingface.co/nlp/viewer/?dataset=wikipedia&config=20200501.pt\n",
    "# !pip install nlp\n",
    "\n",
    "# Issues\n",
    "# source: https://github.com/huggingface/nlp/issues/227\n",
    "# !pip instal apache_beam\n",
    "# !pip install dill==0.3.1.1\n",
    "# !pip install apache-beam[interactive]\n",
    "# !pip install mwparserfromhell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# source: https://github.com/huggingface/nlp\n",
    "import nlp\n",
    "from nlp import load_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Error and Warning messages when running load_dataset() with wikipedia**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/\n",
    "If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory). \n",
    "\n",
    "Example of usage: \n",
    "\t`load_dataset('wikipedia', '20200501.pt', beam_runner='DirectRunner')`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "WARNING:apache_beam.options.pipeline_options:Discarding unparseable args: ['-f', '/mnt/home/pierre/.local/share/jupyter/runtime/kernel-bc2a44af-2ceb-4b4c-ae03-ca85e5598616.json']\n",
    "WARNING:apache_beam.runners.interactive.interactive_environment:Dependencies required for Interactive Beam PCollection visualization are not available, please use: `pip install apache-beam[interactive]` to install necessary dependencies to enable all data visualization features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['aeslc', 'ai2_arc', 'allocine', 'anli', 'arcd', 'art', 'billsum', 'blended_skill_talk', 'blimp', 'blog_authorship_corpus', 'bookcorpus', 'boolq', 'break_data', 'c4', 'cfq', 'civil_comments', 'cmrc2018', 'cnn_dailymail', 'coarse_discourse', 'com_qa', 'commonsense_qa', 'compguesswhat', 'coqa', 'cornell_movie_dialog', 'cos_e', 'cosmos_qa', 'crime_and_punish', 'csv', 'definite_pronoun_resolution', 'discofuse', 'drop', 'eli5', 'empathetic_dialogues', 'eraser_multi_rc', 'esnli', 'event2Mind', 'flores', 'fquad', 'gap', 'germeval_14', 'gigaword', 'glue', 'hansards', 'hellaswag', 'imdb', 'jeopardy', 'json', 'k-halid/ar', 'kor_nli', 'lc_quad', 'lhoestq/c4', 'librispeech_lm', 'lm1b', 'math_dataset', 'math_qa', 'mlqa', 'movie_rationales', 'multi_news', 'multi_nli', 'multi_nli_mismatch', 'natural_questions', 'newsroom', 'openbookqa', 'opinosis', 'para_crawl', 'piaf', 'qa4mre', 'qangaroo', 'qanta', 'qasc', 'quarel', 'quartz', 'quoref', 'race', 'reclor', 'reddit', 'reddit_tifu', 'rotten_tomatoes', 'scan', 'scicite', 'scientific_papers', 'scifact', 'sciq', 'scitail', 'sentiment140', 'snli', 'social_i_qa', 'squad', 'squad_es', 'squad_it', 'squad_v1_pt', 'squad_v2', 'super_glue', 'ted_hrlr', 'ted_multi', 'tiny_shakespeare', 'trivia_qa', 'tydiqa', 'ubuntu_dialogs_corpus', 'webis/tl_dr', 'wiki40b', 'wiki_qa', 'wiki_snippets', 'wiki_split', 'wikihow', 'wikipedia', 'wikisql', 'wikitext', 'winogrande', 'wiqa', 'wmt14', 'wmt15', 'wmt16', 'wmt17', 'wmt18', 'wmt19', 'wmt_t2t', 'wnut_17', 'x_stance', 'xcopa', 'xnli', 'xquad', 'xsum', 'xtreme', 'yelp_polarity']\n"
     ]
    }
   ],
   "source": [
    "# Print all the available datasets\n",
    "datasets = nlp.list_datasets()\n",
    "print([dataset.id for dataset in datasets])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['bertscore', 'bleu', 'coval', 'gleu', 'glue', 'rouge', 'sacrebleu', 'seqeval', 'squad', 'squad_v2', 'xnli']\n"
     ]
    }
   ],
   "source": [
    "# List all the available metrics\n",
    "print([metric.id for metric in nlp.list_metrics()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "asdict() should be called on dataclass instances",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-15-b8ace6c227a2>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mdataclasses\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0masdict\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0;32mfor\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mvalue\u001b[0m \u001b[0;32min\u001b[0m \u001b[0masdict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdatasets\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m6\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mitems\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      5\u001b[0m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'👉 '\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mkey\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m': '\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/.conda/envs/fastai2/lib/python3.7/dataclasses.py\u001b[0m in \u001b[0;36masdict\u001b[0;34m(obj, dict_factory)\u001b[0m\n\u001b[1;32m   1061\u001b[0m     \"\"\"\n\u001b[1;32m   1062\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0m_is_dataclass_instance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1063\u001b[0;31m         \u001b[0;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"asdict() should be called on dataclass instances\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1064\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0m_asdict_inner\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobj\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdict_factory\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1065\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: asdict() should be called on dataclass instances"
     ]
    }
   ],
   "source": [
    "# You can read a few attributes of the datasets before loading them (they are python dataclasses)\n",
    "from dataclasses import asdict\n",
    "\n",
    "for key, value in asdict(datasets[6]).items():\n",
    "    print('👉 ' + key + ': ' + str(value))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# source: https://huggingface.co/nlp/viewer/?dataset=wikipedia&config=20200501.pt\n",
    "# !pip install nlp\n",
    "\n",
    "# Issues\n",
    "# source: https://github.com/huggingface/nlp/issues/227\n",
    "# !pip instal apache_beam\n",
    "# !pip install dill==0.3.1.1\n",
    "# !pip install apache-beam[interactive]\n",
    "# !pip install mwparserfromhell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "#dataset = load_dataset('wikipedia', '20200501.pt', beam_runner='DirectRunner')\n",
    "dataset = load_dataset('wikipedia', '20200501.pt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Downloading and preparing dataset wikipedia/20200501.pt (download: Unknown size, generated: Unknown size, total: Unknown size) to /mnt/home/pierre/.cache/huggingface/datasets/wikipedia/20200501.pt/1.0.0..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Informations on the dataset (description, citation, size, splits, format...)\n",
    "# are provided in `dataset.info` (as a python dataclass)\n",
    "for key, value in asdict(dataset.info).items():\n",
    "    print('👉 ' + key + ': ' + str(value))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pprint import pprint\n",
    "\n",
    "print(f\"👉Dataset len(dataset): {len(dataset)}\")\n",
    "print(\"\\n👉First item 'dataset[0]':\")\n",
    "pprint(dataset[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print the first examples in the training set\n",
    "print(dataset['train'][0])\n",
    "\n",
    "# Load a metric\n",
    "metric = nlp.load_metric('wikipedia')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "nbpresent": {
     "id": "bfe49910-58e0-4be3-aba1-7733dc18cca2"
    }
   },
   "source": [
    "#### Create text and csv files of wikipedia in Portuguese"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dest = path_data/'docs'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Text file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "get_one_clean_file(dest,lang)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### csv file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "get_one_clean_csv_file(dest,lang)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fname = f'all_texts_{lang}wiki.csv'\n",
    "df = pd.read_csv(dest.parent/fname)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# csv file \n",
    "dest.parent/fname"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Download a GPT-2 English pre-trained model and train a GPT-2 tokenizer with a vocab in Portuguese"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are following 3 steps in order to **get a GPT-2 tokenizer with the vocab in Portuguese**:\n",
    "- 3.1) **Get the pre-trained GPT-2 Tokenizer & Model (pre-training with an English corpus) from the Transformers library (Hugging Face)**: it will give us the tokenizer structure we need and the pre-trained model weights (it's better to start training our GPT-2 model in Portuguese from weights already trained even in another language than from random values)\n",
    "- 3.2) **Train a Byte-level BPE (BBPE) Tokenizer on the Portuguese Wikipedia corpus by using the Tokenizers library (Hugging Face)**: this will give us the vocabulary files in Portuguese of our GPT-2 tokenizer.\n",
    "- 3.3) **Import the tokenizer Portuguese config files into the pre-trained GPT-2 Tokenizer**: it will give us a GPT-2 tokenizer structure with the vocab in Portuguese."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.1 Get the pre-trained GPT2 Tokenizer & Model (pre-training with an English corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Firstly, will need to install the transformers library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "transformers==3.0.0\r\n"
     ]
    }
   ],
   "source": [
    "# ! pip install transformers\n",
    "!pip freeze | grep transformers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we will fine-tune the GPT2 pretrained model on wikipedia in Portuguese, we need:\n",
    "- the tokenizer (`GPT2Tokenizer` or `GPT2TokenizerFast`) to prepare the data \n",
    "- and the model (`GPT2LMHeadModel`) (since we want the GPT2 language model)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import GPT2TokenizerFast, GPT2LMHeadModel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use several versions of this GPT2 model, look at the [transformers documentation](https://huggingface.co/transformers/pretrained_models.html) for more details. Here we will use the basic version (ie, the one with the smallest number of weights) but you can change the model used by changing the content of `pretrained_weights` (if it's not a GPT2 model, you'll need to change the classes used for the model and the tokenizer of course):\n",
    "\n",
    "- **Model**\n",
    "  - OpenAI GPT-2 English model\n",
    "  - 12-layer, 768-hidden, 12-heads\n",
    "  - 117M parameters, 548 Mo\n",
    "  - Download time: about 10 minutes\n",
    "- **Tokenizer**\n",
    "  - [GPT2TokenizerFast](https://huggingface.co/transformers/model_doc/gpt2.html#gpt2tokenizerfast): Constructs a “Fast” GPT-2 BPE tokenizer (backed by HuggingFace’s tokenizers library). \n",
    "  - Peculiarities:\n",
    "    - Byte-level Byte-Pair-Encoding (BBPE)\n",
    "    - WARNING: the following information from the [HF GPT2TokenizerFast documentation](https://huggingface.co/transformers/model_doc/gpt2.html#transformers.GPT2TokenizerFast) is WRONG: *Requires a space to start the input string => the encoding methods should be called with the `add_prefix_space` flag set to `True`. Otherwise, this tokenizer encode and decode method will not conserve the absence of a space at the beginning of a string.*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Import"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of GPT2LMHeadModel were not initialized from the model checkpoint at gpt2 and are newly initialized: ['h.0.attn.masked_bias', 'h.1.attn.masked_bias', 'h.2.attn.masked_bias', 'h.3.attn.masked_bias', 'h.4.attn.masked_bias', 'h.5.attn.masked_bias', 'h.6.attn.masked_bias', 'h.7.attn.masked_bias', 'h.8.attn.masked_bias', 'h.9.attn.masked_bias', 'h.10.attn.masked_bias', 'h.11.attn.masked_bias', 'lm_head.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 13 s, sys: 922 ms, total: 13.9 s\n",
      "Wall time: 7.48 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# The GPT2 Model transformer with a language modeling head on top\n",
    "# (linear layer with weights tied to the input embeddings)\n",
    "\n",
    "# GPT2Tokenizer: https://huggingface.co/transformers/model_doc/gpt2.html#gpt2tokenizer\n",
    "# GPT2TokenizerFast: https://huggingface.co/transformers/model_doc/gpt2.html#gpt2tokenizerfast\n",
    "# GPT2LMHeadModel: https://huggingface.co/transformers/model_doc/gpt2.html#transformers.GPT2LMHeadModel\n",
    "\n",
    "pretrained_weights = 'gpt2'\n",
    "tokenizer_en = GPT2TokenizerFast.from_pretrained(pretrained_weights)\n",
    "model_en = GPT2LMHeadModel.from_pretrained(pretrained_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To correct the warning about token_pad (GPT2TokenizerFast), run the following code\n",
    "# source: https://github.com/huggingface/transformers/issues/2648#issuecomment-616177044\n",
    "tokenizer_en.pad_token = tokenizer_en.eos_token"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Check the GPT2 model config (config of GPT2LMHeadModel)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GPT2Config {\n",
       "  \"activation_function\": \"gelu_new\",\n",
       "  \"architectures\": [\n",
       "    \"GPT2LMHeadModel\"\n",
       "  ],\n",
       "  \"attn_pdrop\": 0.1,\n",
       "  \"bos_token_id\": 50256,\n",
       "  \"embd_pdrop\": 0.1,\n",
       "  \"eos_token_id\": 50256,\n",
       "  \"initializer_range\": 0.02,\n",
       "  \"layer_norm_epsilon\": 1e-05,\n",
       "  \"model_type\": \"gpt2\",\n",
       "  \"n_ctx\": 1024,\n",
       "  \"n_embd\": 768,\n",
       "  \"n_head\": 12,\n",
       "  \"n_layer\": 12,\n",
       "  \"n_positions\": 1024,\n",
       "  \"resid_pdrop\": 0.1,\n",
       "  \"summary_activation\": null,\n",
       "  \"summary_first_dropout\": 0.1,\n",
       "  \"summary_proj_to_labels\": true,\n",
       "  \"summary_type\": \"cls_index\",\n",
       "  \"summary_use_proj\": true,\n",
       "  \"task_specific_params\": {\n",
       "    \"text-generation\": {\n",
       "      \"do_sample\": true,\n",
       "      \"max_length\": 50\n",
       "    }\n",
       "  },\n",
       "  \"vocab_size\": 50257\n",
       "}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_en.config"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Check the GPT2 tokenizer config (config of GPT2TokenizerFast pre-trained in English)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using sep_token, but it is not set yet.\n",
      "Using cls_token, but it is not set yet.\n",
      "Using mask_token, but it is not set yet.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "max_model_input_sizes\n",
      "-  gpt2 1024\n",
      "-  gpt2-medium 1024\n",
      "-  gpt2-large 1024\n",
      "-  gpt2-xl 1024\n",
      "-  distilgpt2 1024\n",
      "\n",
      "model_max_length: 1024\n",
      "\n",
      "padding_side: right\n",
      "\n",
      "model_input_names: ['token_type_ids', 'attention_mask']\n",
      "\n",
      "bos_token & bos_token_id: <|endoftext|> 50256\n",
      "\n",
      "eos_token & eos_token_id: <|endoftext|> 50256\n",
      "\n",
      "unk_token & unk_token_id: <|endoftext|> 50256\n",
      "\n",
      "sep_token: None\n",
      "\n",
      "pad_token, pad_token_id & pad_token_type_id: <|endoftext|> 50256 0\n",
      "\n",
      "cls_token: None\n",
      "\n",
      "mask_token: None\n",
      "\n",
      "additional_special_tokens: []\n",
      "\n",
      "all_special_tokens & all_special_ids: ['<|endoftext|>'] [50256]\n",
      "\n",
      "---------- vocab ----------\n",
      "\n",
      "vocab_files_names: {'vocab_file': 'vocab.json', 'merges_file': 'merges.txt'}\n",
      "\n",
      "vocab_file\n",
      "-  gpt2 : https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-vocab.json\n",
      "-  gpt2-medium : https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-medium-vocab.json\n",
      "-  gpt2-large : https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-large-vocab.json\n",
      "-  gpt2-xl : https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-xl-vocab.json\n",
      "-  distilgpt2 : https://s3.amazonaws.com/models.huggingface.co/bert/distilgpt2-vocab.json\n",
      "\n",
      "merges_file\n",
      "-  gpt2 : https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-merges.txt\n",
      "-  gpt2-medium : https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-medium-merges.txt\n",
      "-  gpt2-large : https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-large-merges.txt\n",
      "-  gpt2-xl : https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-xl-merges.txt\n",
      "-  distilgpt2 : https://s3.amazonaws.com/models.huggingface.co/bert/distilgpt2-merges.txt\n",
      "\n",
      "vocab_size: 50257\n",
      "\n",
      "First 20 items of the vocab: {'Ġeuph': 40381, 'DeliveryDate': 39749, 'ĠFighting': 19098, 'Ġmandated': 28853, 'Ġcalls': 3848, 'ibly': 3193, 'Pain': 38490, 'Ġremind': 7101, 'odynamics': 44124, 'Ġelectoral': 13901, '989': 42520, 'Emb': 31567, 'pure': 37424, 'Ġ404': 32320, 'ĠFridays': 45011, 'ĠWick': 36029, 'Ġaggression': 15569, 'Ġapprehended': 41979, 'ĠGenetic': 42295, 'ĠHB': 25997}\n"
     ]
    }
   ],
   "source": [
    "# source: https://huggingface.co/transformers/_modules/transformers/tokenization_utils_fast.html\n",
    "    \n",
    "# print('short-cut-names:',tokenizer_en.short-cut-names) \n",
    "# print()  \n",
    "\n",
    "print('max_model_input_sizes')\n",
    "for k,v in tokenizer_en.max_model_input_sizes.items():\n",
    "    print('- ',k,v)\n",
    "print()\n",
    "    \n",
    "print('model_max_length:',tokenizer_en.model_max_length)\n",
    "print()\n",
    "\n",
    "for k,v in tokenizer_en.pretrained_init_configuration.items():\n",
    "    print(k,v)\n",
    "    \n",
    "print('padding_side:',tokenizer_en.padding_side)\n",
    "print()\n",
    "\n",
    "print('model_input_names:',tokenizer_en.model_input_names)\n",
    "print()\n",
    "\n",
    "print('bos_token & bos_token_id:',tokenizer_en.bos_token,tokenizer_en.bos_token_id)\n",
    "print()\n",
    "\n",
    "print('eos_token & eos_token_id:',tokenizer_en.eos_token,tokenizer_en.eos_token_id)\n",
    "print()\n",
    "\n",
    "print('unk_token & unk_token_id:',tokenizer_en.unk_token,tokenizer_en.unk_token_id)\n",
    "print()\n",
    "\n",
    "print('sep_token:',tokenizer_en.sep_token)\n",
    "print()\n",
    "\n",
    "print('pad_token, pad_token_id & pad_token_type_id:',tokenizer_en.pad_token,tokenizer_en.pad_token_id,tokenizer_en.pad_token_type_id)\n",
    "print()\n",
    "\n",
    "print('cls_token:',tokenizer_en.cls_token)\n",
    "print()\n",
    "\n",
    "print('mask_token:',tokenizer_en.mask_token)\n",
    "print()\n",
    "\n",
    "print('additional_special_tokens:',tokenizer_en.additional_special_tokens)\n",
    "print()\n",
    "\n",
    "print('all_special_tokens & all_special_ids:',tokenizer_en.all_special_tokens,tokenizer_en.all_special_ids)\n",
    "print()\n",
    "\n",
    "print('---------- vocab ----------')\n",
    "print()\n",
    "\n",
    "print('vocab_files_names:',tokenizer_en.vocab_files_names)\n",
    "print()\n",
    "\n",
    "for k,v in tokenizer_en.pretrained_vocab_files_map.items():\n",
    "    print(k)\n",
    "    for kk,vv in v.items():\n",
    "        print('- ',kk,':',vv)\n",
    "    print()\n",
    "    \n",
    "print('vocab_size:',tokenizer_en.vocab_size)\n",
    "print()\n",
    "#print(tokenizer_en.get_vocab())\n",
    "\n",
    "num = 20\n",
    "print(f'First {num} items of the vocab: {dict(itertools.islice(tokenizer_en.get_vocab().items(), 20))}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.2 Train a Byte Level BPE (BBPE) Tokenizer on the Portuguese Wikipedia"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll use the [Tokenizers](https://github.com/huggingface/tokenizers) library from Hugging Face in order to train a **Byte Level BPE (BBPE) Tokenizer** on the Portuguese Wikipedia with the objective to get the vocab files `vocab.json`, which is a list of the most frequent tokens ranked by frequency, and the `merges.txt` which is a list of merges."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tokenizers==0.8.0\r\n"
     ]
    }
   ],
   "source": [
    "# !pip install tokenizers\n",
    "!pip freeze | grep tokenizers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Byte Level BPE (BBPE) Tokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Neural Machine Translation with Byte-Level Subwords](https://arxiv.org/pdf/1909.03341.pdf]) (Facebook AI, 12/05/2019)\n",
    "\n",
    "**[Abstract]** Almost all existing machine translation models are built on top of character-based vocabularies: characters, subwords orwords. Rare characters from noisy text or character-rich languages such as Japanese and Chinese however can  unnecessarily take up vocabulary slots and limit its compactness. Representing text at the level of bytes andusing  the  256 byte set as vocabulary is a potential solution to this issue. High computational cost has however prevented it from being widely deployed or used in practice. **In this paper, we investigate byte-level subwords, specifically byte-level BPE (BBPE), which is compacter than character vocabulary and has no out-of-vocabulary tokens, but is more efficient than using pure bytes only is.** We claim that contextualizing BBPE embeddings is necessary, which can be implemented by a convolutional or recurrent layer. **Our experiments show that BBPE has comparable performance to BPE while its size is only 1/8 of that for BPE.** In the multilingual setting, BBPE maximizes vocabulary sharing across many languages and achieves better translation quality. Moreover, we show that BBPE enables transferring models between languages with non-overlapping character sets.\n",
    "\n",
    "**[Conclusion]** We proposed BBPE which builds a byte-level subword vocabulary for machine translation. It results in a much more compact vocabulary than character-based ones do without the loss of performance. In multilingual settings, the former often outperforms the latter. **BBPE does not have any out-of-vocabulary tokens, allowing us to transfer a model using BBPE between languages with non-overlapping vocabularies.** This transfer learning paradigm is actually very generic and can be applied to any languages and datasets for performance gain or training acceleration. With the same vocabulary size, BBPE segments sentences into shorter sequences than character-based methods do, leading to faster trainingand inference. Our future work includes: eliminating source-target sentence length imbalance; evaluating BBPE in one-to-many and many-to-many translation settings; exploring better segmentation algorithms for byte-level subwords."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "50257"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get GPT2 tokenizer_en vocab size\n",
    "ByteLevelBPE_tokenizer_pt_vocab_size = tokenizer_en.vocab_size\n",
    "ByteLevelBPE_tokenizer_pt_vocab_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 5h 47min 23s, sys: 43min 12s, total: 6h 30min 35s\n",
      "Wall time: 10min 52s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "['/mnt/home/pierre/.fastai/data/ptwiki/ByteLevelBPE_tokenizer_pt/vocab.json',\n",
       " '/mnt/home/pierre/.fastai/data/ptwiki/ByteLevelBPE_tokenizer_pt/merges.txt']"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "# ByteLevelBPETokenizer Represents a Byte-level BPE as introduced by OpenAI with their GPT-2 model\n",
    "from tokenizers import ByteLevelBPETokenizer\n",
    "\n",
    "ByteLevelBPE_tokenizer_pt = ByteLevelBPETokenizer()\n",
    "\n",
    "# Get list of paths to corpus files\n",
    "paths = [str(path_data/'all_texts_ptwiki.txt')]\n",
    "\n",
    "# Customize training with <|endoftext|> special GPT2 token\n",
    "ByteLevelBPE_tokenizer_pt.train(files=paths, \n",
    "                                vocab_size=ByteLevelBPE_tokenizer_pt_vocab_size, \n",
    "                                min_frequency=2, \n",
    "                                special_tokens=[\"<|endoftext|>\"])\n",
    "\n",
    "# Get sequence length max of 1024\n",
    "ByteLevelBPE_tokenizer_pt.enable_truncation(max_length=1024)\n",
    "\n",
    "# save tokenizer\n",
    "ByteLevelBPE_tokenizer_pt_rep = 'ByteLevelBPE_tokenizer_pt'\n",
    "path_to_ByteLevelBPE_tokenizer_pt_rep = path_data/ByteLevelBPE_tokenizer_pt_rep\n",
    "if not (path_to_ByteLevelBPE_tokenizer_pt_rep).exists():\n",
    "    path_to_ByteLevelBPE_tokenizer_pt_rep.mkdir(exist_ok=True, parents=True)\n",
    "ByteLevelBPE_tokenizer_pt.save_model(str(path_to_ByteLevelBPE_tokenizer_pt_rep))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have both a vocab.json, which is a list of the most frequent tokens ranked by frequency, and a merges.txt list of merges."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Check our tokenizer pre-trained in Portuguese"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the tokenizer ByteLevelBPE_tokenizer_pt\n",
    "from tokenizers import ByteLevelBPETokenizer\n",
    "\n",
    "# Get the path to ByteLevelBPE_tokenizer_pt config files\n",
    "ByteLevelBPE_tokenizer_pt_rep = 'ByteLevelBPE_tokenizer_pt'\n",
    "path_to_ByteLevelBPE_tokenizer_pt_rep = path_data/ByteLevelBPE_tokenizer_pt_rep\n",
    "\n",
    "ByteLevelBPE_tokenizer_pt = ByteLevelBPETokenizer(\n",
    "    vocab_file=f'{path_to_ByteLevelBPE_tokenizer_pt_rep}/vocab.json',\n",
    "    merges_file=f'{path_to_ByteLevelBPE_tokenizer_pt_rep}/merges.txt'\n",
    ")\n",
    "\n",
    "# Get sequence length max of 1024\n",
    "ByteLevelBPE_tokenizer_pt.enable_truncation(max_length=1024)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(50257, ['<|endoftext|>', '!', '\"', '#', '$'])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get vocab as a list\n",
    "ByteLevelBPE_tokenizer_pt_vocab = ByteLevelBPE_tokenizer_pt.get_vocab() \n",
    "ByteLevelBPE_tokenizer_pt_vocab_ls = [k for k, v in sorted(ByteLevelBPE_tokenizer_pt_vocab.items(), key=lambda item: item[1])]\n",
    "len(ByteLevelBPE_tokenizer_pt_vocab_ls),ByteLevelBPE_tokenizer_pt_vocab_ls[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([39, 1119, 298, 22317, 258, 11041, 14],\n",
       " ['G', 'osto', 'Ġdo', 'Ġqueijo', 'Ġe', 'Ġvinho', '.'],\n",
       " [(0, 1), (1, 5), (5, 8), (8, 15), (15, 17), (17, 23), (23, 24)])"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = \"Gosto do queijo e vinho.\"\n",
    "output = ByteLevelBPE_tokenizer_pt.encode(text)\n",
    "output.ids,output.tokens,output.offsets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input text: Gosto do queijo e vinho.\n",
      "tokens ids: [39, 1119, 298, 22317, 258, 11041, 14]\n",
      "back to text: Gosto do queijo e vinho.\n"
     ]
    }
   ],
   "source": [
    "back_to_text = ByteLevelBPE_tokenizer_pt.decode(ByteLevelBPE_tokenizer_pt.encode(text).ids)\n",
    "\n",
    "print('input text:', text)\n",
    "print('tokens ids:', output.ids)\n",
    "print('back to text:', back_to_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3.3 Import the tokenizer config files in Portuguese into the pre-trained GPT2 Tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the path to ByteLevelBPE_tokenizer_pt config files\n",
    "ByteLevelBPE_tokenizer_pt_rep = 'ByteLevelBPE_tokenizer_pt'\n",
    "path_to_ByteLevelBPE_tokenizer_pt_rep = path_data/ByteLevelBPE_tokenizer_pt_rep\n",
    "\n",
    "# import the pre-trained GPT2TokenizerFast tokenizer with the tokenizer_pt config files\n",
    "tokenizer_pt = GPT2TokenizerFast.from_pretrained(\n",
    "    str(path_to_ByteLevelBPE_tokenizer_pt_rep), \n",
    "    pad_token='<|endoftext|>')\n",
    "\n",
    "# Get sequence length max of 1024\n",
    "tokenizer_pt.model_max_length = 1024"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input text: Gosto do queijo e vinho.\n",
      "tokens ids: [39, 1119, 298, 22317, 258, 11041, 14]\n",
      "back to text: Gosto do queijo e vinho.\n"
     ]
    }
   ],
   "source": [
    "# Check\n",
    "text = \"Gosto do queijo e vinho.\"\n",
    "tokens_ids = tokenizer_pt.encode(text)\n",
    "back_to_text = tokenizer_pt.decode(tokenizer_pt.encode(text))\n",
    "\n",
    "print('input text:', text)\n",
    "print('tokens ids:', tokens_ids)\n",
    "print('back to text:', back_to_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Create a fastai tokenizer and update the embeddings matrix of the GPT-2 English pre-trained model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's see how we can use fastai v2 to fine-tune this model on Wikipedia in Portuguese, using all the fastai v2 training utilities.\n",
    "\n",
    "We will follow these 2 following steps:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 4.1) **GPT2TokenizerFast (imported GPT-2 tokenizer) --> fastai Tokenizer**: to process the data to train a model, we need to build a fastai tokenizer from the GPT-2 tokenizer with vocab in Portuguese.\n",
    "- 4.2) **Change vocab embeddings (wte matrix) in the GPT-2 pre-trained model to adapt to the Portuguese vocab**: as the vocab embedding matrix (wte) of the pre-trained GPT-2 model corresponds to the English vocabulary, we'll keep the embeddings vectors of the common tokens between the English and Portuguese vocab."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " First, we import all the text utilities:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai2.text.all import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.1 GPT2TokenizerFast (imported GPT-2 tokenizer) --> fastai Tokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*(text from Sylvain Gugger Transformers Tutorial)* To process this data to train a model, we need to build a `Transform` that will be applied lazily. In a fastai `Transform` you can define:\n",
    "- an `encodes` method that is applied when you call the transform (a bit like the `forward` method in a `nn.Module`)\n",
    "- a `decodes` method that is applied when you call the [decode](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.decode) method of the transform, if you need to decode anything for showing purposes (like converting ids to a text here)\n",
    "- a `setups` method that sets some inner state of the `Transform` (not needed here)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TransformersTokenizer(Transform):\n",
    "    def __init__(self, tokenizer): self.tokenizer = tokenizer\n",
    "    def encodes(self, x): \n",
    "        toks = self.tokenizer.tokenize(x)\n",
    "        return tensor(self.tokenizer.convert_tokens_to_ids(toks))\n",
    "    def decodes(self, x): return TitledStr(self.tokenizer.decode(x.cpu().numpy()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Two comments on the code above:\n",
    "- in `encodes` we don't use the [tokenizer.encode](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.encode) method since it does some additional preprocessing for the model after tokenizing and numericalizing (the aprt throwing a warning before). Here we don't need any post-processing so it's fine to skip it and we use the [tokenizer.tokenize](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.tokenize) method followed by the [tokenizer.convert_tokens_to_ids](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.convert_tokens_to_ids) one.\n",
    "- in `decodes` we return a `TitledStr` object and not just a plain string. That's a fastai class that adds a `show` method to the string, which will allow us to use all the fastai show methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Tokenizers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of GPT2LMHeadModel were not initialized from the model checkpoint at gpt2 and are newly initialized: ['h.0.attn.masked_bias', 'h.1.attn.masked_bias', 'h.2.attn.masked_bias', 'h.3.attn.masked_bias', 'h.4.attn.masked_bias', 'h.5.attn.masked_bias', 'h.6.attn.masked_bias', 'h.7.attn.masked_bias', 'h.8.attn.masked_bias', 'h.9.attn.masked_bias', 'h.10.attn.masked_bias', 'h.11.attn.masked_bias', 'lm_head.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 16.3 s, sys: 831 ms, total: 17.1 s\n",
      "Wall time: 6.94 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Load the GPT2 tokenizer in English\n",
    "from transformers import GPT2TokenizerFast, GPT2LMHeadModel\n",
    "pretrained_weights = 'gpt2'\n",
    "tokenizer_en = GPT2TokenizerFast.from_pretrained(pretrained_weights)\n",
    "model_en = GPT2LMHeadModel.from_pretrained(pretrained_weights)\n",
    "\n",
    "# To correct the warning about token_pad (GPT2TokenizerFast), run the following code\n",
    "# source: https://github.com/huggingface/transformers/issues/2648#issuecomment-616177044\n",
    "tokenizer_en.pad_token = tokenizer_en.eos_token"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the path to ByteLevelBPE_tokenizer_pt config files\n",
    "ByteLevelBPE_tokenizer_pt_rep = 'ByteLevelBPE_tokenizer_pt'\n",
    "path_to_ByteLevelBPE_tokenizer_pt_rep = path_data/ByteLevelBPE_tokenizer_pt_rep\n",
    "\n",
    "# import the pre-trained GPT2TokenizerFast tokenizer with the tokenizer_pt config files\n",
    "tokenizer_pt = GPT2TokenizerFast.from_pretrained(\n",
    "    str(path_to_ByteLevelBPE_tokenizer_pt_rep), \n",
    "    pad_token='<|endoftext|>')\n",
    "\n",
    "# Get sequence length max of 1024\n",
    "tokenizer_pt.model_max_length = 1024"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input text: Maybe, you're right\n",
      "text tokens: ['Maybe', ',', 'Ġyou', \"'re\", 'Ġright']\n",
      "text tokens_ids: tensor([13300,    11,   345,   821,   826])\n",
      "output text: Maybe, you're right\n"
     ]
    }
   ],
   "source": [
    "# Test of the class TransformersTokenizer of fastai with tokenizer_en\n",
    "tokenizer_fastai_en = TransformersTokenizer(tokenizer_en)\n",
    "text = \"Maybe, you're right\"\n",
    "tokens_ids = tokenizer_fastai_en.encodes(text)\n",
    "tokens = tokenizer_fastai_en.tokenizer.convert_ids_to_tokens(tokens_ids)\n",
    "\n",
    "print('input text:',TitledStr(text))\n",
    "print('text tokens:',TitledStr(tokens))\n",
    "print('text tokens_ids:',TitledStr(tokens_ids))\n",
    "print('output text:',TitledStr(tokenizer_fastai_en.decodes(tokens_ids)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input text: Maybe, you're right\n",
      "text tokens: ['May', 'be', ',', 'Ġyou', \"'re\", 'Ġrig', 'ht']\n",
      "text tokens_ids: tensor([24656,   609,    12, 17793, 25842,  8341,  2084])\n",
      "output text: Maybe, you're right\n"
     ]
    }
   ],
   "source": [
    "# Test of the class TransformersTokenizer of fastai with tokenizer_pt\n",
    "tokenizer_fastai_pt = TransformersTokenizer(tokenizer_pt)\n",
    "text = \"Maybe, you're right\"\n",
    "tokens_ids = tokenizer_fastai_pt.encodes(text)\n",
    "tokens = tokenizer_fastai_pt.tokenizer.convert_ids_to_tokens(tokens_ids)\n",
    "\n",
    "print('input text:',TitledStr(text))\n",
    "print('text tokens:',TitledStr(tokens))\n",
    "print('text tokens_ids:',TitledStr(tokens_ids))\n",
    "print('output text:',TitledStr(tokenizer_fastai_pt.decodes(tokens_ids)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4.2 Change vocab embeddings (wte matrix) in the GPT-2 pre-trained model to adapt to the Portuguese vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import model if needed\n",
    "# from transformers import GPT2LMHeadModel\n",
    "# pretrained_weights = 'gpt2'\n",
    "# model_en = GPT2LMHeadModel.from_pretrained(pretrained_weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Check vocabs size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(50257, 50257, 0)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer_fastai_en = TransformersTokenizer(tokenizer_en)\n",
    "old_vocab_size = tokenizer_fastai_en.tokenizer.vocab_size\n",
    "\n",
    "tokenizer_fastai_pt = TransformersTokenizer(tokenizer_pt)\n",
    "new_vocab_size = tokenizer_fastai_pt.tokenizer.vocab_size\n",
    "\n",
    "old_vocab_size,new_vocab_size,old_vocab_size-new_vocab_size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Check vocabs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(50257, ['!', '\"', '#', '$', '%', '&', \"'\", '(', ')', '*'])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer_fastai_vocab_en = tokenizer_fastai_en.tokenizer.get_vocab()\n",
    "tokenizer_fastai_vocab_ls_en = [k for k, v in sorted(tokenizer_fastai_vocab_en.items(), key=lambda item: item[1])]\n",
    "len(tokenizer_fastai_vocab_ls_en),tokenizer_fastai_vocab_ls_en[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(50257, ['<|endoftext|>', '!', '\"', '#', '$', '%', '&', \"'\", '(', ')'])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer_fastai_vocab_pt = tokenizer_fastai_pt.tokenizer.get_vocab() \n",
    "tokenizer_fastai_vocab_ls_pt = [k for k, v in sorted(tokenizer_fastai_vocab_pt.items(), key=lambda item: item[1])]\n",
    "len(tokenizer_fastai_vocab_ls_pt),tokenizer_fastai_vocab_ls_pt[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Changing vocabs and the vocab embeddings matrix (ie, setup new embeddings matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Parameter containing:\n",
       " tensor([[-0.1101, -0.0393,  0.0331,  ..., -0.1364,  0.0151,  0.0453],\n",
       "         [ 0.0403, -0.0486,  0.0462,  ...,  0.0861,  0.0025,  0.0432],\n",
       "         [-0.1275,  0.0479,  0.1841,  ...,  0.0899, -0.1297, -0.0879],\n",
       "         ...,\n",
       "         [-0.0445, -0.0548,  0.0123,  ...,  0.1044,  0.0978, -0.0695],\n",
       "         [ 0.1860,  0.0167,  0.0461,  ..., -0.0963,  0.0785, -0.0225],\n",
       "         [ 0.0514, -0.0277,  0.0499,  ...,  0.0070,  0.1552,  0.1207]],\n",
       "        requires_grad=True),\n",
       " Parameter containing:\n",
       " tensor([[-0.1101, -0.0393,  0.0331,  ..., -0.1364,  0.0151,  0.0453],\n",
       "         [ 0.0403, -0.0486,  0.0462,  ...,  0.0861,  0.0025,  0.0432],\n",
       "         [-0.1275,  0.0479,  0.1841,  ...,  0.0899, -0.1297, -0.0879],\n",
       "         ...,\n",
       "         [-0.0445, -0.0548,  0.0123,  ...,  0.1044,  0.0978, -0.0695],\n",
       "         [ 0.1860,  0.0167,  0.0461,  ..., -0.0963,  0.0785, -0.0225],\n",
       "         [ 0.0514, -0.0277,  0.0499,  ...,  0.0070,  0.1552,  0.1207]],\n",
       "        requires_grad=True),\n",
       " tensor(True))"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check atual weight of wte and lm_head and if wte = lm_head\n",
    "tens_a = model_en.transformer.wte.weight\n",
    "tens_b = model_en.lm_head.weight\n",
    "model_en.transformer.wte.weight,model_en.lm_head.weight,torch.all(tens_a.eq(tens_b))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get weights of the old wte\n",
    "old_wgts = model_en.transformer.get_input_embeddings().weight.clone().detach()\n",
    "\n",
    "# Get the mean embedding vetor of the old wte\n",
    "wgts_m = old_wgts.mean(0)\n",
    "\n",
    "# Initialize vocab size and weights of the new wte\n",
    "new_vocab_size = tokenizer_fastai_pt.tokenizer.vocab_size\n",
    "new_wgts = old_wgts.new_zeros(new_vocab_size,old_wgts.size(1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Save**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Portuguese wte matrix setup done!\n",
      "\n",
      "We kept 12948 embeddings vetors from the English one.\n",
      "We did not kept 37309 embeddings vetors from the English one (we used the old wte mean vetor).\n",
      "\n",
      "15 first tokens IN common between the 2 vocabs:\n",
      "[('ĠQuit', 40195), ('Smith', 32470), ('Ġomit', 39040), ('oc', 574), ('ym', 18252), ('Ġactual', 9443), ('ck', 911), ('ĠPremier', 16558), ('Ġeste', 987), ('ĠInd', 3438), ('Ġbol', 4203), ('phen', 35836), ('ĠParticip', 36689), ('ĠZeus', 19316), ('Ġnan', 39770)]\n",
      "\n",
      "15 first tokens NOT in common between the 2 vocabs:\n",
      "[('PSDB', 23151), ('Ġenvio', 19270), ('ĠocupaÃ§Ã£o', 5938), ('Ġdocumentada', 30011), ('Ġduros', 36706), ('visto', 44422), ('ĠSiro', 43061), ('Ġdestacavam', 47397), ('Ġarqui', 49060), ('ĠArte', 5977), ('ĠValor', 29721), ('Ġalinhados', 38446), ('ĠnÃºmeros', 4626), ('ĠpÃªnis', 31686), ('cisa', 29710)]\n"
     ]
    }
   ],
   "source": [
    "# Get the new wte keeping the embeddings vetors of tokens in common in the 2 vocabs\n",
    "# A token present in the new vocab but not in the old one gets the mean embedding vetor of the old wte\n",
    "old_vocab = tokenizer_fastai_en.tokenizer.get_vocab()\n",
    "new_vocab = tokenizer_fastai_pt.tokenizer.get_vocab()\n",
    "same_tokens_list = list()\n",
    "different_tokens_list = list()\n",
    "    \n",
    "for w,idx_new in new_vocab.items():    \n",
    "    idx_old = old_vocab.get(w, -1)\n",
    "    if idx_old>=0:\n",
    "        new_wgts[idx_new] = old_wgts[idx_old]\n",
    "        same_tokens_list.append((w,idx_new))\n",
    "    else:\n",
    "        new_wgts[idx_new] = wgts_m\n",
    "        different_tokens_list.append((w,idx_new))\n",
    "\n",
    "# setup in model the new wte\n",
    "new_wte = nn.Embedding(new_vocab_size,old_wgts.size(1))\n",
    "#new_wte.weight.data.normal_(mean=0.0, std=model.config.initializer_range)\n",
    "new_wte.weight.data = new_wgts\n",
    "model_en.transformer.set_input_embeddings(new_wte)\n",
    "print(f'Portuguese wte matrix setup done!\\n\\nWe kept {len(same_tokens_list)} embeddings vectors from the English one.\\nWe did not kept {len(different_tokens_list)} embeddings vectors from the English one (instead, we used the old wte mean vector).\\n')\n",
    "\n",
    "# Check identical tokens between the 2 vocabs               \n",
    "num = 15\n",
    "print(f'{num} first tokens IN common between the 2 vocabs:\\n{same_tokens_list[:num]}\\n')\n",
    "print(f'{num} first tokens NOT in common between the 2 vocabs:\\n{different_tokens_list[:num]}')\n",
    "\n",
    "# save new_wgts\n",
    "torch.save(new_wgts, path_data/'new_wte_wgts.pt')\n",
    "# save same_tokens_list and different_tokens_list\n",
    "torch.save(same_tokens_list, path_data/'same_tokens_list.pt')\n",
    "torch.save(different_tokens_list, path_data/'different_tokens_list.pt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Load**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Portuguese wte matrix setup done!\n",
      "\n",
      "We kept 12948 embeddings vetors from the English one.\n",
      "We did not kept 37309 embeddings vetors from the English one (we used the old wte mean vetor).\n",
      "\n",
      "15 first tokens IN common between the 2 vocabs:\n",
      "[('ĠQuit', 40195), ('Smith', 32470), ('Ġomit', 39040), ('oc', 574), ('ym', 18252), ('Ġactual', 9443), ('ck', 911), ('ĠPremier', 16558), ('Ġeste', 987), ('ĠInd', 3438), ('Ġbol', 4203), ('phen', 35836), ('ĠParticip', 36689), ('ĠZeus', 19316), ('Ġnan', 39770)]\n",
      "\n",
      "15 first tokens NOT in common between the 2 vocabs:\n",
      "[('PSDB', 23151), ('Ġenvio', 19270), ('ĠocupaÃ§Ã£o', 5938), ('Ġdocumentada', 30011), ('Ġduros', 36706), ('visto', 44422), ('ĠSiro', 43061), ('Ġdestacavam', 47397), ('Ġarqui', 49060), ('ĠArte', 5977), ('ĠValor', 29721), ('Ġalinhados', 38446), ('ĠnÃºmeros', 4626), ('ĠpÃªnis', 31686), ('cisa', 29710)]\n"
     ]
    }
   ],
   "source": [
    "# load new_wgts\n",
    "new_wgts = torch.load(path_data/'new_wte_wgts.pt')\n",
    "# load same_tokens_list and different_tokens_list\n",
    "same_tokens_list = torch.load(path_data/'same_tokens_list.pt')\n",
    "different_tokens_list = torch.load(path_data/'different_tokens_list.pt')\n",
    "                      \n",
    "# setup in model the new wte\n",
    "new_wte = nn.Embedding(new_vocab_size,old_wgts.size(1))\n",
    "new_wte.weight.data = new_wgts\n",
    "model_en.transformer.set_input_embeddings(new_wte)\n",
    "print(f'Portuguese wte matrix setup done!\\n\\nWe kept {len(same_tokens_list)} embeddings vectors from the English one.\\nWe did not kept {len(different_tokens_list)} embeddings vectors from the English one (instead, we used the old wte mean vector).\\n')\n",
    "\n",
    "# Check identical tokens between the 2 vocabs               \n",
    "num = 15\n",
    "print(f'{num} first tokens IN common between the 2 vocabs:\\n{same_tokens_list[:num]}\\n')\n",
    "print(f'{num} first tokens NOT in common between the 2 vocabs:\\n{different_tokens_list[:num]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Check"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Great! All the embeddings vetors of the 12948 common tokens are the ones of the old wte matrix :-)\n",
      "\n",
      "CPU times: user 351 ms, sys: 3.98 ms, total: 355 ms\n",
      "Wall time: 353 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Check that the embeddings vetors of the common tokens are the ones from the old wte matrix\n",
    "old_vocab = tokenizer_fastai_en.tokenizer.get_vocab()\n",
    "#new_vocab = tokenizer_fastai_pt.tokenizer.get_vocab()\n",
    "count = 0\n",
    "\n",
    "for (tok,idx) in same_tokens_list:\n",
    "    w = tokenizer_fastai_pt.tokenizer.convert_ids_to_tokens(idx)\n",
    "    tens_a = new_wgts[idx]\n",
    "    idx_old = old_vocab.get(w, -1)\n",
    "    if idx_old >= 0:\n",
    "        tens_b = old_wgts[idx_old]\n",
    "    else:\n",
    "        tens_b = wgts_m\n",
    "    if ( torch.all(tens_a.eq(tens_b)) == False) or (w != tok):\n",
    "        print('idx,tok:',idx,tok)\n",
    "        print('idx,w:',idx,w)\n",
    "        print('idx_old:',idx_old)\n",
    "        print('identical?',torch.all(tens_a.eq(tens_b)))\n",
    "        count += 1\n",
    "\n",
    "if count == 0:        \n",
    "    print(f'Great! All the embeddings vetors of the {len(same_tokens_list)} common tokens are the ones of the old wte matrix :-)\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Great! All the embeddings vetors of the 37309 NOT common tokens are the old wte mean vetor :-)\n",
      "\n",
      "CPU times: user 1.24 s, sys: 635 µs, total: 1.24 s\n",
      "Wall time: 1.23 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Check that the embeddings vetors of the NOT common tokens are the old wte mean vetor\n",
    "count = 0\n",
    "\n",
    "for (tok,idx) in different_tokens_list:\n",
    "    w = tokenizer_fastai_pt.tokenizer.convert_ids_to_tokens(idx)\n",
    "    tens_a = new_wgts[idx]\n",
    "    idx_old = old_vocab.get(w, -1)\n",
    "    if idx_old >= 0:\n",
    "        tens_b = old_wgts[idx_old]\n",
    "    else:\n",
    "        tens_b = wgts_m\n",
    "    if ( torch.all(tens_a.eq(tens_b)) == False) or (w != tok):\n",
    "        print('idx,tok:',idx,tok)\n",
    "        print('idx,w:',idx,w)\n",
    "        print('idx_old:',idx_old)\n",
    "        print('identical?',torch.all(tens_a.eq(tens_b)))\n",
    "        count += 1\n",
    "\n",
    "if count == 0:        \n",
    "    print(f'Great! All the embeddings vetors of the {len(different_tokens_list)} NOT common tokens are the old wte mean vetor :-)\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Changing lm_head weights with the new embeddings matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Linear(in_features=768, out_features=50257, bias=False)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_en.lm_head.weight = model_en.transformer.wte.weight\n",
    "model_en.lm_head"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Parameter containing:\n",
       " tensor([[ 0.0514, -0.0277,  0.0499,  ...,  0.0070,  0.1552,  0.1207],\n",
       "         [-0.1101, -0.0393,  0.0331,  ..., -0.1364,  0.0151,  0.0453],\n",
       "         [ 0.0403, -0.0486,  0.0462,  ...,  0.0861,  0.0025,  0.0432],\n",
       "         ...,\n",
       "         [-0.0025, -0.0585,  0.1174,  ...,  0.0236,  0.0039,  0.0344],\n",
       "         [ 0.1554, -0.1325,  0.3294,  ..., -0.0977,  0.0863, -0.1514],\n",
       "         [-0.0025, -0.0585,  0.1174,  ...,  0.0236,  0.0039,  0.0344]],\n",
       "        requires_grad=True),\n",
       " Parameter containing:\n",
       " tensor([[ 0.0514, -0.0277,  0.0499,  ...,  0.0070,  0.1552,  0.1207],\n",
       "         [-0.1101, -0.0393,  0.0331,  ..., -0.1364,  0.0151,  0.0453],\n",
       "         [ 0.0403, -0.0486,  0.0462,  ...,  0.0861,  0.0025,  0.0432],\n",
       "         ...,\n",
       "         [-0.0025, -0.0585,  0.1174,  ...,  0.0236,  0.0039,  0.0344],\n",
       "         [ 0.1554, -0.1325,  0.3294,  ..., -0.0977,  0.0863, -0.1514],\n",
       "         [-0.0025, -0.0585,  0.1174,  ...,  0.0236,  0.0039,  0.0344]],\n",
       "        requires_grad=True),\n",
       " tensor(True))"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check atual weight of wte and lm_head and if wte = lm_head\n",
    "tens_a = model_en.transformer.wte.weight\n",
    "tens_b = model_en.lm_head.weight\n",
    "model_en.transformer.wte.weight,model_en.lm_head.weight,torch.all(tens_a.eq(tens_b))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Create fastai v2 Datasets and Dataloaders"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5.1 fastai v2 Datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*(text from Sylvain Gugger Transformers Tutorial)* You can then group your data with this `Transform` using a `TfmdLists`. It has an s in its name because it contains the training and validation datasets. \n",
    "\n",
    "We indicate the indices of the training dataset and the validation dataset with `splits` (here, 80% of the indices randomly chosen, then all the remaining indices)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Visualize Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "203205"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lang = 'pt'\n",
    "fname = f'all_texts_{lang}wiki.csv'\n",
    "df = pd.read_csv(path_data/fname)\n",
    "len(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Fotografia (do grego φως [\"fós\"] (\"luz\"), e γραφις [\"grafis\"] (\"estilo\", \"pincel\") ou γραφη \"grafê\", e significa \"desenhar com luz e contraste\"), por definição, é essencialmente a \"técnica de criação\" de imagens por meio de exposição luminosa, fixando-as em uma superfície sensível. A primeira fotografia reconhecida remonta ao ano de 1826 e é atribuída ao francês Joseph Nicéphore Niépce. Contudo, a invenção da fotografia não é obra de um só autor, mas um processo de acúmulo de avanços por parte de muitas pessoas, trabalhando, juntas ou em paralelo, ao longo de muitos anos. Se por um lado os...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Espadanedo é uma antiga freguesia portuguesa do concelho de Macedo de Cavaleiros, com 17,90 km² de área e 188 habitantes (2011). A sua densidade populacional era 10,5 hab/km².\\nFoi extinta (agregada) pela reorganização administrativa de 2012/2013, sendo o seu território integrado na União de Freguesias de Espadanedo, Edroso, Murçós e Soutelo Mourisco.\\n\\nA antiga freguesia de S. Miguel de Espadanedo e Valongo pertenceu ao antigo concelho de Torre D. Chama, extinto a 24 de Outubro de 1855, em 1839 aparece agregada à comarca de Bragança e em 1852 à de Mirandela. Passa a pertencer definitivam...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Jacques-Germain Soufflot (Irancy, 22 de julho de 1713 — Paris, 29 de agosto de 1780) foi um arquitecto francês, iniciador do estilo arquitectónico do Neoclassicismo. O seu trabalho mais conhecido é, sem dúvida, o Panthéon (Panteão) de Paris, construído a partir de 1755, inicialmente uma igreja dedicada a Santa Genoveva.\\n\\nSoufflot nasceu em Irancy, perto de Auxerre, na França. Com 18 anos, entrou para a Academia Francesa de Roma, onde os jovens estudantes da década de 1750 se tornariam na primeira geração de criativos pleanamente neoclássicos. Ficará na Itália de 1731 a 1738. Quando volto...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>A Faculdade de Medicina da Universidade de São Paulo (FMUSP) é uma escola médica da Universidade de São Paulo. Foi fundada em 1912 com o nome de \"Faculdade de Medicina e Cirurgia de São Paulo\" por Arnaldo Vieira de Carvalho (1867-1920), médico formado em 1888 pela Faculdade de Medicina do Rio de Janeiro. Em homenagem ao ilustre fundador, a Faculdade é, ainda hoje, chamada de a \"Casa de Arnaldo\" por seus alunos e ex-alunos. Em 1925 teve seu nome alterado para \"Faculdade de Medicina de São Paulo\" e em 1934, foi incorporada à recém-criada Universidade de São Paulo, passando a ter a atual desi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>A Escola do Teatro Bolshoi no Brasil é uma tradicional escola de balé existente na cidade de Joinville, no estado de Santa Catarina. Fundada em 2000, é a única filial do Teatro Bolshoi de Moscou e possui alunos de vários estados brasileiros. Tem como missão formar artistas cidadãos, promover e difundir a arte-educação.\\n\\nA instituição foi fundada em 15 de março de 2000, é a única filial do Teatro Bolshoi. \\n\\nUm orgulho para o Brasil e para Joinville, cidade sede. A Escola do Teatro Bolshoi no Brasil, com professores russos e brasileiros, forma bailarinos com a mesma precisão, técnica e q...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      text\n",
       "0  Fotografia (do grego φως [\"fós\"] (\"luz\"), e γραφις [\"grafis\"] (\"estilo\", \"pincel\") ou γραφη \"grafê\", e significa \"desenhar com luz e contraste\"), por definição, é essencialmente a \"técnica de criação\" de imagens por meio de exposição luminosa, fixando-as em uma superfície sensível. A primeira fotografia reconhecida remonta ao ano de 1826 e é atribuída ao francês Joseph Nicéphore Niépce. Contudo, a invenção da fotografia não é obra de um só autor, mas um processo de acúmulo de avanços por parte de muitas pessoas, trabalhando, juntas ou em paralelo, ao longo de muitos anos. Se por um lado os...\n",
       "1  Espadanedo é uma antiga freguesia portuguesa do concelho de Macedo de Cavaleiros, com 17,90 km² de área e 188 habitantes (2011). A sua densidade populacional era 10,5 hab/km².\\nFoi extinta (agregada) pela reorganização administrativa de 2012/2013, sendo o seu território integrado na União de Freguesias de Espadanedo, Edroso, Murçós e Soutelo Mourisco.\\n\\nA antiga freguesia de S. Miguel de Espadanedo e Valongo pertenceu ao antigo concelho de Torre D. Chama, extinto a 24 de Outubro de 1855, em 1839 aparece agregada à comarca de Bragança e em 1852 à de Mirandela. Passa a pertencer definitivam...\n",
       "2  Jacques-Germain Soufflot (Irancy, 22 de julho de 1713 — Paris, 29 de agosto de 1780) foi um arquitecto francês, iniciador do estilo arquitectónico do Neoclassicismo. O seu trabalho mais conhecido é, sem dúvida, o Panthéon (Panteão) de Paris, construído a partir de 1755, inicialmente uma igreja dedicada a Santa Genoveva.\\n\\nSoufflot nasceu em Irancy, perto de Auxerre, na França. Com 18 anos, entrou para a Academia Francesa de Roma, onde os jovens estudantes da década de 1750 se tornariam na primeira geração de criativos pleanamente neoclássicos. Ficará na Itália de 1731 a 1738. Quando volto...\n",
       "3  A Faculdade de Medicina da Universidade de São Paulo (FMUSP) é uma escola médica da Universidade de São Paulo. Foi fundada em 1912 com o nome de \"Faculdade de Medicina e Cirurgia de São Paulo\" por Arnaldo Vieira de Carvalho (1867-1920), médico formado em 1888 pela Faculdade de Medicina do Rio de Janeiro. Em homenagem ao ilustre fundador, a Faculdade é, ainda hoje, chamada de a \"Casa de Arnaldo\" por seus alunos e ex-alunos. Em 1925 teve seu nome alterado para \"Faculdade de Medicina de São Paulo\" e em 1934, foi incorporada à recém-criada Universidade de São Paulo, passando a ter a atual desi...\n",
       "4  A Escola do Teatro Bolshoi no Brasil é uma tradicional escola de balé existente na cidade de Joinville, no estado de Santa Catarina. Fundada em 2000, é a única filial do Teatro Bolshoi de Moscou e possui alunos de vários estados brasileiros. Tem como missão formar artistas cidadãos, promover e difundir a arte-educação.\\n\\nA instituição foi fundada em 15 de março de 2000, é a única filial do Teatro Bolshoi. \\n\\nUm orgulho para o Brasil e para Joinville, cidade sede. A Escola do Teatro Bolshoi no Brasil, com professores russos e brasileiros, forma bailarinos com a mesma precisão, técnica e q..."
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Sample (this allows us to quickly test our code)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- train: 80%\n",
    "- val = 20%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_sample = df[:1000]\n",
    "\n",
    "num = int(0.8*len(df_sample))\n",
    "\n",
    "idxs = np.random.randint(0, len(df_sample), len(df_sample))\n",
    "idxs_train = idxs[:num]\n",
    "idxs_val = idxs[num:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We gather all texts in one numpy array (since it will be easier to use this way with fastai):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.46 ms, sys: 218 µs, total: 1.68 ms\n",
      "Wall time: 1.61 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "all_texts = np.concatenate([df_sample.iloc[idxs_train].text.values, df_sample.iloc[idxs_val].text.values])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 322 ms, sys: 43.1 ms, total: 365 ms\n",
      "Wall time: 33.3 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "splits = [list(idxs_train), list(idxs_val)]\n",
    "tls = TfmdLists(all_texts, TransformersTokenizer(tokenizer_pt), splits=splits, dl_type=LMDataLoader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We specify `dl_type=LMDataLoader` for when we will convert this `TfmdLists` to `DataLoaders`: we will use an `LMDataLoader` since we have a language modeling problem, not the usual fastai `TfmdDL`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### All data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- train: 80%\n",
    "- val = 20%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "# num = int(0.8*len(df))\n",
    "\n",
    "# idxs = np.random.randint(0, len(df), len(df))\n",
    "# idxs_train = idxs[:num]\n",
    "# idxs_val = idxs[num:]\n",
    "\n",
    "# save idxs train and valid\n",
    "# torch.save(idxs_train, path_data/'idxs_train.pt')\n",
    "# torch.save(idxs_val, path_data/'idxs_val.pt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load idxs train and valid\n",
    "idxs_train = torch.load(path_data/'idxs_train.pt')\n",
    "idxs_val = torch.load(path_data/'idxs_val.pt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We gather all texts in one numpy array (since it will be easier to use this way with fastai):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 42.6 ms, sys: 3.79 ms, total: 46.4 ms\n",
      "Wall time: 44.6 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "all_texts = np.concatenate([df.iloc[idxs_train].text.values, df.iloc[idxs_val].text.values])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 340 ms, sys: 31.9 ms, total: 372 ms\n",
      "Wall time: 151 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "splits = [list(idxs_train), list(idxs_val)]\n",
    "tls = TfmdLists(all_texts, TransformersTokenizer(tokenizer_pt), splits=splits, dl_type=LMDataLoader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We specify `dl_type=LMDataLoader` for when we will convert this `TfmdLists` to `DataLoaders`: we will use an `LMDataLoader` since we have a language modeling problem, not the usual fastai `TfmdDL`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Check datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a `TfmdLists` you can access to the elements of the training or validation set quite easily:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([19715,   305, 12866,  ..., 21725,    14,   199]),\n",
       " tensor([   47,  3004,   261,  6913,  7358,    12,   372,   342,  3046,   261,\n",
       "          3478,  2535,    13, 29617,    12,  3998,   347,  5926,  4067,   305,\n",
       "           787,    14,   199,   199,    47,  1326,   712,  1912,   303,  1486,\n",
       "           275,  3004,   621,   342, 23590,   307,  1162,   303, 13286, 33431,\n",
       "           261,  4266,  3925,   298,  6431,   723,  4202, 11772,    12, 47284,\n",
       "           324,   341,  3007,  1361,   475,  1055,  1938,   298,  1568,   261,\n",
       "          3699,  4384,    12,   300, 13988,    14,   315,  3182,   379,  7618,\n",
       "          1224,   305,  2582,  4060,   643,    12, 10247,   475,  5306,    14,\n",
       "           199,   199,  2704, 10553,   544,  7095,   315, 12240,  7941,    12,\n",
       "          3837,   305,  4624,   261, 29549,   305,  1668, 10052,    14,   937,\n",
       "         10130,   323,   443,  1569,   349,  5615,   300,   307,   377, 21016,\n",
       "           303, 10388,   818,   347, 32881,    12,   443,  6339,   298,  6042,\n",
       "         15992,   489,  1622,  2341,   305,  1223,    12,   391,  5760,   258,\n",
       "          2806,    14,   384,  1845,   621, 25092,   258, 18822,  4166,    12,\n",
       "           576,   379, 10553,   297,   342, 11641,   325,  3928,    12,   300,\n",
       "           307, 21016,   258,  5437,  1188,   447, 24901,   307,   377, 16987,\n",
       "            14, 14194,   788,   750,    12,  1188, 38843,   430,   375, 21697,\n",
       "           341,   259,  1868,   261, 48357,    14, 29591,   307,  5724,  9573,\n",
       "           375, 21697,  7297,   275,  6744,   261,   810,   604,   341,  5105,\n",
       "            14,   315,  3219,   621,  3124,  6913,  7358,   275,  2043,  2823,\n",
       "           298,  1568,   300,  1796,   710,    14,   199,   199,  3062,   377,\n",
       "          2862,  4426,   298,  1279,  3129,    12,   259,  4900,   305,  4607,\n",
       "           303,  5741,   391,  2074,   261,  5773,   305,  8368,  1326,    12,\n",
       "           297,   660, 17316,    12, 35051,   258, 18433,    12,  1361,   712,\n",
       "           375,  2247,   303, 12790,   341, 23511,    12,  6315,   261,  1876,\n",
       "           258,   305,  1172,    14,  2284, 48800,    12,   259,  4823,  5594,\n",
       "           373,  5278,   341,   275,   346,  3433,   569,   363,   401,   868,\n",
       "           258, 22825,  5661,  1367,   818,    12, 32711, 49209,    12,  1674,\n",
       "          2935,   261, 21346,  1106,  9792,   615,   608,   199,   199,    47,\n",
       "          5252,  3648,   363,   445,  1223,  1351,  1286,   762,     9,   621,\n",
       "           914, 31729,   661,  6139,    13,   393,    13, 46944,    14,  6329,\n",
       "            12,   259,  2794,  7624,  2424,   341, 16041,  9519,    12,  7583,\n",
       "           290,   414,  1569, 49532,   322, 31366,    12,  1205,  5365, 38605,\n",
       "           330,   266,  1278,   595,   551,  2358,   621,  3009,   307,   377,\n",
       "         19054,  4795, 22399, 10627,  7189,    12,   297,  3406,    77, 10122,\n",
       "           258,  7004,   282,  4394,    14,   199,   199,    33,  1646,  6860,\n",
       "           258,   375, 16021,   468,  9433,   793,  1125,  4297,    14,  1309,\n",
       "           307,  1173, 10470,   300,  2551, 15359,   298,  2043,   258,   450,\n",
       "          5147,  7608,   375, 32985,   261,  4453,   489, 39323, 33204,    14,\n",
       "          2044,  1705,    12,   347,  1430,  2768,  1173,  1188,  6654,  7050,\n",
       "           305,  4623,    12,   307,    12,   358,  2390,   261,   342,  2374,\n",
       "         18423,    12,  4296, 47253,    13,  1276,    12,  9782,   330,    86,\n",
       "          8949,   824,   551,   199,   199,    33,  1583,   261,  2750,   372,\n",
       "          9245,   534,  1868,   261,  7580, 28818, 25399,    12,  4037,   549,\n",
       "          2319,   579, 49568,    12,  4117,  1361,   259,  2611,  5592,   261,\n",
       "         13621,    14,   199,   199,  1220,   604,  4127, 32122,   259, 36930,\n",
       "           298,  3004,    14,   315, 17856,  2721, 18633,   261,  6913,  7358,\n",
       "          6526,   300,   867, 33193,  7016,    14,   384,  3396, 33957,   324,\n",
       "           305,  1646,   347,   787, 10103,   300,   349,  1430,  2768,  9347,\n",
       "         29455,    14,  7267,   623,   375,   604,   300,   307,   259,  8368,\n",
       "         23832,   660, 13027,   297,   275,  3004,    12, 12603,  2613,   341,\n",
       "          2611,  2310,   447,  9251,   305,   787,    12,   391,   259, 27928,\n",
       "         19106,    14,   199,   199,  3062,   375,   604,   307,   303,  4171,\n",
       "            12,   275,  3004,   379, 23111,  7032,  1768,  3958,   305,   787,\n",
       "            14,   778,  1368,  3376, 13111,  1396,   420, 25006,    14, 47557,\n",
       "          1139,  1370,   267,    12, 13567,   358, 36628,  1739,   258,  4915,\n",
       "         25459,   258,  2775,  4718,   261, 23268, 45599,   409,   275,   467,\n",
       "          4007,   258,  4275,  1979,  1179,  3883,   443,  4786,   298, 20445,\n",
       "           661,   604,  4251,    14,   199,   199,   598,  2791,    12,  6710,\n",
       "           349,  7744,  2978,    12,   259,  7336,  2589,   275,  3343,   261,\n",
       "         27374,   298,  3004,    14, 11073,   377,  3735,   307, 30954,  3826,\n",
       "          1473,   259,  7693,   420,   452,  1368,  3376,   489,  5437,    27,\n",
       "           259,  6509,   261, 14577,   261,  3349,    12,   307, 40320, 13076,\n",
       "           259, 23268,    27,   258,   259,  8237,   261,  5437,   258, 22524,\n",
       "            14,   199,   199, 21101,  1430,   261, 12770,   258,  4084,    76,\n",
       "           310,   305,   787,    14, 21514, 27281,   300, 13491,    14,   199,\n",
       "           199,  6658,  2123,   391, 12943, 12150, 23287,    14, 27050,  1056,\n",
       "           375, 18948,    12,   297,  1472,   261,   961,   708, 34943,  4482,\n",
       "            14,   199,   199,    47,   762,   258,   447, 32339,  5199,  3901,\n",
       "           261,  3699,  4384,    14,   199,   199,   598,  2265,  5576, 21352,\n",
       "           423,    14, 12937,  1496,   347,  1583,   261,  3188,    14,   199,\n",
       "           199,  2557,   342,  2424, 47930,  1397,   261, 31301,  2022,   199,\n",
       "           199, 18000,   261, 19220,   307,  8816,   325,  6016,   305,  2659,\n",
       "          7813,   261,  6032,   261,  3699,  4384,    12,   300,  2265, 20092,\n",
       "            13,  7448, 12478,    12,   261,  2440,   341,   275, 18538,  4067,\n",
       "            14,   199,   199, 36610,   300,   342,  4968, 10330,  6016,   261,\n",
       "          5565, 10569,    13,    71,  8189,    12,   261,  8300,  3752,    14,\n",
       "           199,   199,   598,  2265, 20092,    13,    39,  7622,   258, 20092,\n",
       "            13,  7448, 12478,    14, 21514, 37162,   300, 10415,    14,   199,\n",
       "           199, 49222,   261,   762,  2168,   261,  1821,   259,   787,    14,\n",
       "           199, 30330,  9447,   261,  6913,  7358,    14, 28923,   358,   712,\n",
       "          2121,  1461,   275,  3093,   258,  4208,   261,  1815,  9312,   341,\n",
       "           259,  1868,   305,  1055,  1938,    14,   199,   199, 49163,   347,\n",
       "          6608,   305,  8330,    12,   325,  6016,   305,  2659,  3265, 12005,\n",
       "            12,   300,  2265, 20092,    13,  7448, 12478,    14,   199]))"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tls.train[0],tls.valid[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "They are not the same. We can see the shape are differents:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([1327]), torch.Size([849]))"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tls.tfms(tls.train.items[0]).shape, tls.tfms(tls.valid.items[0]).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can have a look at both decodes using `show_at`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Francisco da Silveira Pinto da Fonseca Teixeira (Canelas (Peso da Régua), 1 de Setembro de 1763 — Vila Real, 27 de Maio de 1821), 1.º conde de Amarante, mais conhecido por General Silveira, foi um oficial general do Exército Português e político, que se destacou durante a Guerra Peninsular.\n",
      "\n",
      "O general Silveira (como é mais conhecido em numerosas obras) era filho de Manuel da Silveira Pinto da Fonseca e de D. Antónia da Silveira, teve um irmão, António da Silveira Pinto da Fonseca, que foi o 1º Visconde de Canelas. Casou em 16 de Abril de 1781 com D. Maria Emília Teixeira de Magalhães e Lacerda e deste matrimónio teve três filhos: Manuel da Silveira Pinto da Fonseca Teixeira, 1º Marquês de Chaves, Miguel da Silveira Pinto da Fonseca e D. Mariana da Silveira Pinto da Fonseca.\n",
      "\n",
      "Assentou praça, como Cadete, no Regimento de Cavalaria de Almeida (mais tarde Cavalaria 11) a 25 de Abril de 1780. Ali foi promovido a Alferes em 22 de Abril de 1790 e a Tenente, no Regimento de Cavalaria 6, em 17 de Dezembro de 1792. No dia 17 de Dezembro de 1799 foi promovido a Capitão e nomeado Ajudante-de-Ordens do Governador das Armas da província da Beira. Em 1801, por decreto de 6 de Março, Francisco da Silveira foi promovido ao posto de sargento-mor. Nesse mesmo ano, quando teve início a chamada Guerra das Laranjas, juntamente com outras pessoas importantes de Trás-os-Montes, participou no levantamento de um corpo de voluntários, tendo sido nomeado a 19 de Maio Comandante-Chefe das Companhias Francas de Trás-os-Montes. Em 14 de Março de 1803 foi promovido ao posto de tenente-coronel, no Regimento de Cavalaria 6.\n",
      "\n",
      "Em Dezembro de 1807, no início da Primeira Invasão Francesa encontrava-se em Aveiro. Junot tinha dado ordem para desmobilizar a maior parte do Exército Português e formar, com as melhores unidades, um corpo de tropas que foi designado Legião Lusitana (ou Legião Portuguesa). Silveira foi chamado a Coimbra para testemunhar a desmobilização dos Regimentos de Cavalaria 6, 9, 11 e 12. Pediu a sua demissão que foi aceite pelo governo de Junot e partiu para o Porto com o objectivo de chegar a bordo da esquadra britânica que se encontrava ao largo e, dessa forma, fugir para o Brasil. Não lhe sendo possível executar este plano foi para Vila Real.\n",
      "\n",
      "Quando a insurreição que tinha nascido em Espanha chegou a Portugal, foi criada no Porto a Junta Provisional do Supremo Governo do Reino, presidida pelo bispo do Porto. A revolta rapidamente se estendeu a todo o Reino, primeiro em Trás-os-Montes, onde Francisco da Silveira teve um papel preponderante na aclamação do governo legítimo de Portugal. O papel que então desempenhou mereceu o reconhecimento da Junta do Supremo Governo do Reino que decretou, a 21 de Julho, a sua promoção a coronel do regimento de Cavalaria 6.\n",
      "\n",
      "Ainda em 1808, participou na força sob comando do general Bernardim Freire de Andrade que se dirigiu para Sul ao encontro das tropas britânicas desembarcadas em Lavos. Exercia então as funções de comandante da guarda avançada deste corpo de tropas. Depois de as tropas francesas terem saído de Portugal, Silveira regressou ao Norte para participar na reorganização do Exército. Foi promovido a Brigadeiro e nomeado Governador Militar de Trás-os-Montes por carta régia de 15 de Fevereiro de 1809. \n",
      "\n",
      "Entretanto teve início a Segunda Invasão Francesa. Perante a superioridade dos franceses, Silveira foi obrigado a abandonar a praça de Chaves, tendo-se dirigido para a região de Vila Real. Quando as tropas francesas continuaram o seu avanço em direcção a Braga, deixando uma pequena guarnição em Chaves, Silveira regressou com as suas tropas, colocou cerco aquela praça que acabou por se render pouco tempo depois. Soult ocupou o Porto em finais de Março de 1809 e Silveira, com as suas tropas mal treinadas, mal equipadas, muitos deles sem armas de fogo, desenvolveu uma actividade notável na defesa da linha do Tâmega. Desta acção, o episódio mais marcante foi a defesa da ponte de Amarante. Obrigado a retirar após catorze dias de resistência, não se deixou abater e, em breve, as suas tropas, obrigavam o corpo de tropas francesas, sob comando de Loison, a retirar para Guimarães.\n",
      "\n",
      "Na ordem do dia de 21 de Maio de 1809, o brigadeiro Francisco da Silveira era promovido a Marechal de Campo em recompensa pela forma como se bateu contra os invasores franceses. A Terceira Invasão Francesa colocou-o novamente em actividade operacional em que o seu prestígio se consolidou. Das acções em que participou destaca-se o ataque ao reduto em Puebla de Sanábria (1 a 10 de Agosto de 1810) na província de Zamora junto à fronteira no norte de Portugal, nos combates em Valverde (14 de Novembro de 1810), defesa de Pinhel (31 de Dezembro de 1810) e defesa Vila da Ponte (11 de Janeiro de 1811).\n",
      "A portaria de 5 de Fevereiro de 1812 veio confirmar a sua promoção a Tenente-general com antiguidade reportada a 1 de Janeiro desse ano. O seu valor foi reconhecido tanto pelo marechal Beresford, comandante em chefe do Exército Português, como por Wellesley que comandava o Exército Aliado na luta contra os franceses. É assim que o encontramos a comandar uma divisão de infantaria portuguesa no exército de Wellesley na Batalha de Vitoria e outras acções na fase final da guerra.\n",
      "\n",
      "Após a Guerra Peninsular, que termina em 1814, Silveira voltou ao lugar de Governador das Armas da Província de Trás-os-Montes. Em 1820, quando a 24 de Agosto rebentou a revolução Liberal, foi convidado a aderir ao partido da Junta Provisória do Porto mas recusou e reuniu em Chaves as tropas da província de Trás-os-Montes com o objectivo contrário, de combater a revolução. Não foi bem sucedido nesta empresa pois essas tropas acabaram por se colocar do lado da Junta do Porto. Depois deste episódio, Silveira retirou-se para Vila Real onde veio a falecer no ano seguinte. Foi sepultado no jazigo da família, na capela do Espírito Santo em Canelas.\n",
      "\n",
      "A 13 de Maio de 1809, o Príncipe Regente agraciou-o com o título de Conde de Amarante. A sua acção durante a Segunda Invasão Francesa mereceu-lhe o maior respeito em todo o Reino. Além desta honra, o 1º Conde de Amarante foi condecorado com a medalha de sete campanhas da Guerra Peninsular, com as medalhas britânica e espanhola por acções e batalhas nesta guerra, com os graus de grã-cruz da Ordem Militar de Cristo e de comendador honorário da Ordem Militar da Torre e Espada.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "show_at(tls.train, 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5.2 fastai v2 Dataloaders"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*(text from Sylvain Gugger Transformers Tutorial)* The fastai v2 library expects the data to be assembled in a `DataLoaders` object (something that has a training and validation dataloader). We can get one by using the `dataloaders` method. We just have to specify a batch size and a sequence length. \n",
    "\n",
    "Since the GPT-2 model was trained with sequences of size 1024, we use this sequence length (it's a stateless model, so it will change the perplexity if we use less)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 6h 43min 14s, sys: 59min 46s, total: 7h 43min\n",
      "Wall time: 31min 27s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "bs,sl = 8,1024\n",
    "dls = tls.dataloaders(bs=bs, seq_len=sl)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4 µs, sys: 1e+03 ns, total: 5 µs\n",
      "Wall time: 11 µs\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# IMPOSSIBLE TO SAVE Dataloaders at the date of 07/01/2020\n",
    "# source: https://forums.fast.ai/t/how-to-save-dataloaders/73828/6\n",
    "\n",
    "# save\n",
    "# fname = 'dls_ptwiki_tokenizerGPT2.pkl'\n",
    "# torch.save(dls, path_data/fname)\n",
    "\n",
    "# load\n",
    "# dls = torch.load(path_data/fname)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that you may have to reduce the batch size depending on your GPU RAM."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In fastai v2, as soo as we have a `DataLoaders`, we can use `show_batch` to have a look at the data (here texts for inputs, and the same text shifted by one token to the right for validation)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>text_</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Leandro da Silva Nogueira, ou Buscapé, como também é conhecido (São Paulo, 11 de novembro de 1985) é um lutador brasileiro de MMA. Atualmente integra o elenco UFC na categoria peso-leve (até 70,3 kg).  \\n\\nLeandro Silva fez sua estreia no MMA profissional no Beach Fight Festival contra o lutador Viscardi Andrade e ganhou por decisão unânime.\\n\\nBuscapé lutou nas eliminatórias do TUF contra o lutador David Vieira mas perdeu por decisão majoritária. \\n\\nLeandro foi contratado para enfrentar o Ildemar Marajó no, mas Leandro perdeu por decisão unânime. Após a derrota ele foi liberado do UFC.\\n\\nApós ser demitido do UFC, Leandro lutou em alguns eventos no Brasil e também no Pancrase do Japão, vencendo todas as cinco lutas que fez.\\n\\nLeandro retornou ao UFC em 14 de Setembro de 2014 no substituindo o mexicano Efrain Escudero e enfrentou seu conterrâneo Francisco Trinaldo. Ele foi derrotado por decisão unânime em uma luta muito equilibrada.\\n\\nLeandro</td>\n",
       "      <td>andro da Silva Nogueira, ou Buscapé, como também é conhecido (São Paulo, 11 de novembro de 1985) é um lutador brasileiro de MMA. Atualmente integra o elenco UFC na categoria peso-leve (até 70,3 kg).  \\n\\nLeandro Silva fez sua estreia no MMA profissional no Beach Fight Festival contra o lutador Viscardi Andrade e ganhou por decisão unânime.\\n\\nBuscapé lutou nas eliminatórias do TUF contra o lutador David Vieira mas perdeu por decisão majoritária. \\n\\nLeandro foi contratado para enfrentar o Ildemar Marajó no, mas Leandro perdeu por decisão unânime. Após a derrota ele foi liberado do UFC.\\n\\nApós ser demitido do UFC, Leandro lutou em alguns eventos no Brasil e também no Pancrase do Japão, vencendo todas as cinco lutas que fez.\\n\\nLeandro retornou ao UFC em 14 de Setembro de 2014 no substituindo o mexicano Efrain Escudero e enfrentou seu conterrâneo Francisco Trinaldo. Ele foi derrotado por decisão unânime em uma luta muito equilibrada.\\n\\nLeandro</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>em idades mais precoces. Carleton criticou ainda mais a decisão de diminuir a votação, citando as preocupações americanas com a juventude em geral, a dependência exagerada do ensino superior e equiparando a inteligência tecnológica com a responsabilidade e a inteligência. Ele denunciou também o argumento do serviço militar, chamando-o de \"clichê\". Considerando as idades dos soldados na Guerra Civil Americana, Ele afirmou que alfabetização e educação não eram motivo para limitar o voto; Em vez disso, o senso comum e a capacidade de compreender o sistema político fundamentaram as restrições de idade para votar.\\n\\nJames J. Kilpatrick, Um colunista político, afirmou que os estados foram \"extorquidos para ratificar a Vigésima Sexta Emenda\". Em seu artigo, ele alega que ao passar a extensão de 1970 à Lei de Direitos de Voto, o Congresso efetivamente forçou os Estados a ratificar a emenda para que eles não sejam forçados a lidar financeiramente e burocraticamente</td>\n",
       "      <td>idades mais precoces. Carleton criticou ainda mais a decisão de diminuir a votação, citando as preocupações americanas com a juventude em geral, a dependência exagerada do ensino superior e equiparando a inteligência tecnológica com a responsabilidade e a inteligência. Ele denunciou também o argumento do serviço militar, chamando-o de \"clichê\". Considerando as idades dos soldados na Guerra Civil Americana, Ele afirmou que alfabetização e educação não eram motivo para limitar o voto; Em vez disso, o senso comum e a capacidade de compreender o sistema político fundamentaram as restrições de idade para votar.\\n\\nJames J. Kilpatrick, Um colunista político, afirmou que os estados foram \"extorquidos para ratificar a Vigésima Sexta Emenda\". Em seu artigo, ele alega que ao passar a extensão de 1970 à Lei de Direitos de Voto, o Congresso efetivamente forçou os Estados a ratificar a emenda para que eles não sejam forçados a lidar financeiramente e burocraticamente com</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>registra o primeiro voo intercontinental sobre o Atlântico norte com um avião de dois motores, Boeing 767 - Montreal/Tel Aviv.\\n\\nA EL AL opera o voo direto mais longo da sua história: Los Angeles-Tel Aviv: 11.265,408 quilômetros em 13 horas e 41 minutos.\\n\\nVoo histórico para Moscou.\\n\\nUm Boeing 747 da EL AL leva um recorde de 1087 passageiros - judeus etíopes voando de Addis Ababa para Israel, parte da Operação Salomão.\\n\\nO primeiro voo para Aman.\\n\\nO primeiro voo do Boeing 777 da EL AL\\n\\nAbertura de capital - Como parte do processo de privatização da companhia de aviação, esta e o Estado de Israel publicaram um prospecto, em 30 de maio de 2003, colocando a venda as ações e as opções de ações da empresa. No início de junho de 2003 a empresa abriu o seu capital e as suas ações são registradas para negociações na Bolsa de Valores de Tel Aviv.\\n\\nA companhia</td>\n",
       "      <td>o primeiro voo intercontinental sobre o Atlântico norte com um avião de dois motores, Boeing 767 - Montreal/Tel Aviv.\\n\\nA EL AL opera o voo direto mais longo da sua história: Los Angeles-Tel Aviv: 11.265,408 quilômetros em 13 horas e 41 minutos.\\n\\nVoo histórico para Moscou.\\n\\nUm Boeing 747 da EL AL leva um recorde de 1087 passageiros - judeus etíopes voando de Addis Ababa para Israel, parte da Operação Salomão.\\n\\nO primeiro voo para Aman.\\n\\nO primeiro voo do Boeing 777 da EL AL\\n\\nAbertura de capital - Como parte do processo de privatização da companhia de aviação, esta e o Estado de Israel publicaram um prospecto, em 30 de maio de 2003, colocando a venda as ações e as opções de ações da empresa. No início de junho de 2003 a empresa abriu o seu capital e as suas ações são registradas para negociações na Bolsa de Valores de Tel Aviv.\\n\\nA companhia de</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>. Em 1857, foi nomeado Director do Hospital de Febre Amarela instalado no Convento do Desterro.\\n\\nAssistiu, em 1867, como Delegado do Governo Português, ao Congresso Internacional de Oftalmologia, que se reuniu em Paris, França, e onde apresentou valiosos trabalhos sobre a Oftalmologia em Portugal. José Cândido Loureiro teve que defender os seus trabalhos em controvérsias com vários Professores, nomeadamente com o Especialista Francês Carcassonne.\\n\\nDe novo em Portugal, dedicou-se activamente ao estudo da doenças dos olhos e pugnou, com extraordinária energia, pelo estabelecimento duma Enfermaria para Oftálmicos, no Hospital Real de São José.\\n\\nFoi, ainda, Subdelegado de Saúde em Lisboa.\\n\\nTendo adoecido e, sendo dum carácter profundamente impressionável, julgou-se perdido e atirou-se da janela do seu quarto, situado no terceiro pavimento do Hospital Real de São José.\\n\\nColaborou no \"Jornal da Sociedade de Ciências Médicas\" e escreveu:\\nCartão Verde é um telejornal esportivo produzido e exibido pela TV Cultura. Idealizado por Michel Laurence, é exibido</td>\n",
       "      <td>Em 1857, foi nomeado Director do Hospital de Febre Amarela instalado no Convento do Desterro.\\n\\nAssistiu, em 1867, como Delegado do Governo Português, ao Congresso Internacional de Oftalmologia, que se reuniu em Paris, França, e onde apresentou valiosos trabalhos sobre a Oftalmologia em Portugal. José Cândido Loureiro teve que defender os seus trabalhos em controvérsias com vários Professores, nomeadamente com o Especialista Francês Carcassonne.\\n\\nDe novo em Portugal, dedicou-se activamente ao estudo da doenças dos olhos e pugnou, com extraordinária energia, pelo estabelecimento duma Enfermaria para Oftálmicos, no Hospital Real de São José.\\n\\nFoi, ainda, Subdelegado de Saúde em Lisboa.\\n\\nTendo adoecido e, sendo dum carácter profundamente impressionável, julgou-se perdido e atirou-se da janela do seu quarto, situado no terceiro pavimento do Hospital Real de São José.\\n\\nColaborou no \"Jornal da Sociedade de Ciências Médicas\" e escreveu:\\nCartão Verde é um telejornal esportivo produzido e exibido pela TV Cultura. Idealizado por Michel Laurence, é exibido</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>outros suprimentos, enquanto lealistas saqueavam assentamentos civis, especialmente em Nova Iorque, Kentucky e na Pensilvânia. Forças-tarefa de Iroqueses e lealistas lançaram-se em batalhas bem-sucedidas, como no vale de Wyoming e no vale de Cherry em 1778, provocando George Washington a ordenar uma expedição punitiva sob comando do general John Sullivan a oeste de Nova Iorque, no verão de 1779. Houve pouca luta, com os americanos incendiando terras indígenas e saqueando mantimentos, forçando os índios a fugir até Quebec ou para a região das Cataratas do Niágara.\\n\\nNo norte das regiões de Ohio e Illinois, o general americano George Rogers Clark tentou neutralizar a influência britânica nas tribos próximas a Ohio ao capturar as importantes cidades de Kaskaskia, Cahokia e Vincennes no verão de 1778. Ele foi bem-sucedido. Quando o general Henry Hamilton, comandante da guarnição britânica em Detroit, retomou Vincennes, Clark retornou com uma marcha surpresa em janeiro de 1779</td>\n",
       "      <td>suprimentos, enquanto lealistas saqueavam assentamentos civis, especialmente em Nova Iorque, Kentucky e na Pensilvânia. Forças-tarefa de Iroqueses e lealistas lançaram-se em batalhas bem-sucedidas, como no vale de Wyoming e no vale de Cherry em 1778, provocando George Washington a ordenar uma expedição punitiva sob comando do general John Sullivan a oeste de Nova Iorque, no verão de 1779. Houve pouca luta, com os americanos incendiando terras indígenas e saqueando mantimentos, forçando os índios a fugir até Quebec ou para a região das Cataratas do Niágara.\\n\\nNo norte das regiões de Ohio e Illinois, o general americano George Rogers Clark tentou neutralizar a influência britânica nas tribos próximas a Ohio ao capturar as importantes cidades de Kaskaskia, Cahokia e Vincennes no verão de 1778. Ele foi bem-sucedido. Quando o general Henry Hamilton, comandante da guarnição britânica em Detroit, retomou Vincennes, Clark retornou com uma marcha surpresa em janeiro de 1779 e</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dls.show_batch(max_n=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another way to gather the data is to preprocess the texts once and for all and only use the transform to decode the tensors to texts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 5 µs, sys: 1e+03 ns, total: 6 µs\n",
      "Wall time: 11.4 µs\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "def tokenize(text):\n",
    "    toks = tokenizer.tokenize(text)\n",
    "    return tensor(tokenizer.convert_tokens_to_ids(toks))\n",
    "\n",
    "#tokenized = [tokenize(t) for t in progress_bar(all_texts)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [],
   "source": [
    "#save tokenized\n",
    "# torch.save(tokenized, path_data/'tokenized_gpt2.pt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "# load tokenized\n",
    "tokenized_pt = torch.load(path_data/'tokenized_gpt2.pt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we change the previous `Tokenizer` like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TransformersTokenizer(Transform):\n",
    "    def __init__(self, tokenizer): self.tokenizer = tokenizer\n",
    "    def encodes(self, x): \n",
    "        return x if isinstance(x, Tensor) else tokenize(x)\n",
    "        \n",
    "    def decodes(self, x): return TitledStr(self.tokenizer.decode(x.cpu().numpy()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the `encodes` method, we still account for the case where we get something that's not already tokenized, just in case we were to build a dataset with new texts using this transform."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "tls2 = TfmdLists(tokenized_pt, TransformersTokenizer(tokenizer_pt), splits=splits, dl_type=LMDataLoader)\n",
    "dls2 = tls.dataloaders(bs=bs, seq_len=sl)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can check it still works properly for showing purposes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>text_</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>O GAP (Projeto de Proteção dos Grandes Primatas) é um movimento internacional com o objetivo de garantir direitos básicos aos Grandes Primatas: Chimpanzés, Gorilas, Orangotangos e Bonobos.\\n\\nO conceito de Direitos Animais proíbe o uso de animais não-humanos como meios para fins humanos (alimentação, roupa, companhia, entretenimento, estudo, pesquisa médica...), reconhecendo-os como fins em si. A abrangência desta idéia é uma discussão à parte: De acordo com Tom Regan devemos estendê-la a todos os animais que sejam \"sujeitos-de-uma-vida\", ou seja, que existem e sabem que existem, possuem uma vida e são sencientes. Mesmo assim, é de concordância geral que se os direitos devem ser aplicados em etapas, o primeiro e decisivo passo seria incluir os grandes primatas, uma vez que eles além de serem os nossos parentes evolutivos compartilham conosco 98,6% do DNA.\\n\\nDe acordo com essa filosofia os animais, bem como os humanos sem a capacidade do raciocínio (Argumento dos Casos</td>\n",
       "      <td>GAP (Projeto de Proteção dos Grandes Primatas) é um movimento internacional com o objetivo de garantir direitos básicos aos Grandes Primatas: Chimpanzés, Gorilas, Orangotangos e Bonobos.\\n\\nO conceito de Direitos Animais proíbe o uso de animais não-humanos como meios para fins humanos (alimentação, roupa, companhia, entretenimento, estudo, pesquisa médica...), reconhecendo-os como fins em si. A abrangência desta idéia é uma discussão à parte: De acordo com Tom Regan devemos estendê-la a todos os animais que sejam \"sujeitos-de-uma-vida\", ou seja, que existem e sabem que existem, possuem uma vida e são sencientes. Mesmo assim, é de concordância geral que se os direitos devem ser aplicados em etapas, o primeiro e decisivo passo seria incluir os grandes primatas, uma vez que eles além de serem os nossos parentes evolutivos compartilham conosco 98,6% do DNA.\\n\\nDe acordo com essa filosofia os animais, bem como os humanos sem a capacidade do raciocínio (Argumento dos Casos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>e televisão americano.\\n\\nVencedor de 5 prêmios Globo de Ouro e 7 Emmy é mais conhecido por criar e escrever as séries Nip/Tuck (com Dylan Walsh e Julian McMahon) e a comédia-drama musical Glee. É também conhecido pela série \"Popular\". Suas mais recentes séries são \"9-1-1\", \"Pose\" e \"The Politician\". É também o realizador de \"Eat Pray Love\". Ryan é abertamente \"gay\", o que é um dos assuntos mais comuns em suas séries.\\n\\nEm 13 de fevereiro de 2018, foi anunciado que Ryan assinou um contrato milionário de 5 anos com a Netflix para produzir conteúdos originais para seu catálogo.\\n\\nMurphy começou como um jornalista trabalhando para \"The Miami Herald\", \"Los Angeles Times\", \"New York Daily News\", \"Knoxville News Sentinel\" e \"Entertainment Weekly\". Ele começou a escrever roteiros no fim de 1990, quando Steven Spielberg comprou seu roteiro \"Why Can't I Be Audrey Hepburn?\".\\n\\nMurphy começou sua carreira na televisão em 1999 com</td>\n",
       "      <td>televisão americano.\\n\\nVencedor de 5 prêmios Globo de Ouro e 7 Emmy é mais conhecido por criar e escrever as séries Nip/Tuck (com Dylan Walsh e Julian McMahon) e a comédia-drama musical Glee. É também conhecido pela série \"Popular\". Suas mais recentes séries são \"9-1-1\", \"Pose\" e \"The Politician\". É também o realizador de \"Eat Pray Love\". Ryan é abertamente \"gay\", o que é um dos assuntos mais comuns em suas séries.\\n\\nEm 13 de fevereiro de 2018, foi anunciado que Ryan assinou um contrato milionário de 5 anos com a Netflix para produzir conteúdos originais para seu catálogo.\\n\\nMurphy começou como um jornalista trabalhando para \"The Miami Herald\", \"Los Angeles Times\", \"New York Daily News\", \"Knoxville News Sentinel\" e \"Entertainment Weekly\". Ele começou a escrever roteiros no fim de 1990, quando Steven Spielberg comprou seu roteiro \"Why Can't I Be Audrey Hepburn?\".\\n\\nMurphy começou sua carreira na televisão em 1999 com a</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ando de um símbolo já costumeiro.\\n\\nEm 14 de março de 1933, logo após o compromisso de Hitler como Chanceler da Alemanha, a bandeira do NSDAP foi içada ao lado das cores nacionais da Alemanha. Foi adotada como bandeira nacional exclusiva no dia 15 de setembro de 1935.\\n\\nA suástica foi usada nos distintivos e bandeiras ao longo de toda a era nazista, especialmente pelo governo e pelo exército, mas também foi usada por organizações \"populares\", como o \"Reichsbund Deutsche Jägerschaft\".\\n\\nVariantes do uso da suástica pelos nazistas:\\n\\nA suástica, usada por Hitler e pelo Partido Nazista, ainda é, na atualidade, utilizada por grupos neonazistas. Mas, para a maioria das pessoas do Ocidente, a suástica, sendo um símbolo associado às ideias e atos de nazistas, tornou-se um tabu em muitos países ocidentais.\\n\\nNa Alemanha de pós-guerra, por exemplo, o Código Penal (\"Strafgesetzbuch\") tornou criminosa a exibição de símbolos nazistas, tais como a suástica - exceto</td>\n",
       "      <td>de um símbolo já costumeiro.\\n\\nEm 14 de março de 1933, logo após o compromisso de Hitler como Chanceler da Alemanha, a bandeira do NSDAP foi içada ao lado das cores nacionais da Alemanha. Foi adotada como bandeira nacional exclusiva no dia 15 de setembro de 1935.\\n\\nA suástica foi usada nos distintivos e bandeiras ao longo de toda a era nazista, especialmente pelo governo e pelo exército, mas também foi usada por organizações \"populares\", como o \"Reichsbund Deutsche Jägerschaft\".\\n\\nVariantes do uso da suástica pelos nazistas:\\n\\nA suástica, usada por Hitler e pelo Partido Nazista, ainda é, na atualidade, utilizada por grupos neonazistas. Mas, para a maioria das pessoas do Ocidente, a suástica, sendo um símbolo associado às ideias e atos de nazistas, tornou-se um tabu em muitos países ocidentais.\\n\\nNa Alemanha de pós-guerra, por exemplo, o Código Penal (\"Strafgesetzbuch\") tornou criminosa a exibição de símbolos nazistas, tais como a suástica - exceto</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>\\nteoria da aprendizagem computacional, um cálculo é considerado viável se\\nele pode ser feito em um tempo polinomial. Existe dois tipos de resultados \\nde complexibilidade de tempo:\\nOs resultados negativos muitas vezes dependem se geralmente se acredita, mas ainda a hipóteses não comprovadas, tais como:\\n\\nHá várias abordagens diferentes da teoria de aprendizagem computacional. \\nEstas diferenças são baseadas em fazer suposições sobre a\\nInferência princípios usados para generalizar a partir de dados limitados.\\nIsto inclui diferentes definições Probabilidade (veja\\nProbabilidade de frequência, Probabilidade epistemológica) e premissas diferentes sobre a geração de amostras.\\n\\n\\nLimitações de criptografia na aprendizagem fórmulas boolean e autômatos finitos. Em Proceedings of the 21st Annual ACM Symposium sobre Teoria da Computação, páginas 433-444, de Nova York. ACM.\\nWilliam \"Bill\" Warren Bradley (nascido em 28 de julho de 1943) é um político americano e ex-jogador de basquete profissional. Ele serviu três mandatos como senador democrata de Nova Jersey. Ele concorreu sem sucesso pela nomeação do Partido</td>\n",
       "      <td>teoria da aprendizagem computacional, um cálculo é considerado viável se\\nele pode ser feito em um tempo polinomial. Existe dois tipos de resultados \\nde complexibilidade de tempo:\\nOs resultados negativos muitas vezes dependem se geralmente se acredita, mas ainda a hipóteses não comprovadas, tais como:\\n\\nHá várias abordagens diferentes da teoria de aprendizagem computacional. \\nEstas diferenças são baseadas em fazer suposições sobre a\\nInferência princípios usados para generalizar a partir de dados limitados.\\nIsto inclui diferentes definições Probabilidade (veja\\nProbabilidade de frequência, Probabilidade epistemológica) e premissas diferentes sobre a geração de amostras.\\n\\n\\nLimitações de criptografia na aprendizagem fórmulas boolean e autômatos finitos. Em Proceedings of the 21st Annual ACM Symposium sobre Teoria da Computação, páginas 433-444, de Nova York. ACM.\\nWilliam \"Bill\" Warren Bradley (nascido em 28 de julho de 1943) é um político americano e ex-jogador de basquete profissional. Ele serviu três mandatos como senador democrata de Nova Jersey. Ele concorreu sem sucesso pela nomeação do Partido</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>do culto público, um ofício, uma missa especial e uma leitura própria no Breviário (atual Liturgia das Horas). O Papa Leão XIII aprovou o uso do famoso \"Cordão de Santa Filomena\", assim como eregiu a Arquiconfraria de Santa Filomena na França. Por sua vez São Pio X estendeu a Arquiconfraria de Santa Filomena para o mundo inteiro. Assim, a popularidade dela logo se espalhou, sendo seus mais memoráveis devotos João Maria Batista Vianney, Madalena Sofia Barat, Pedro Eymard, e Pedro Chanel, todos eles santos da Igreja Católica Apostólica Romana. \\n\\nJá em 1904, uma publicação feita pelo arqueólogo Orazio Marucchi () (\"Osservazioni archeologiche sulla Iscrizione di S. Filomena dans Miscellanea di Storia Ecclesiastica, Vol. 2, 1904, pp. 365–386\") colocou em dúvida a existência histórica de Santa Filomena. Ele defendia fundamentalmente quatro postulados:\\n\\n1. As placas deveriam estar em ordem direta e não indireta (como estavam) o que constituía um indício de que</td>\n",
       "      <td>culto público, um ofício, uma missa especial e uma leitura própria no Breviário (atual Liturgia das Horas). O Papa Leão XIII aprovou o uso do famoso \"Cordão de Santa Filomena\", assim como eregiu a Arquiconfraria de Santa Filomena na França. Por sua vez São Pio X estendeu a Arquiconfraria de Santa Filomena para o mundo inteiro. Assim, a popularidade dela logo se espalhou, sendo seus mais memoráveis devotos João Maria Batista Vianney, Madalena Sofia Barat, Pedro Eymard, e Pedro Chanel, todos eles santos da Igreja Católica Apostólica Romana. \\n\\nJá em 1904, uma publicação feita pelo arqueólogo Orazio Marucchi () (\"Osservazioni archeologiche sulla Iscrizione di S. Filomena dans Miscellanea di Storia Ecclesiastica, Vol. 2, 1904, pp. 365–386\") colocou em dúvida a existência histórica de Santa Filomena. Ele defendia fundamentalmente quatro postulados:\\n\\n1. As placas deveriam estar em ordem direta e não indireta (como estavam) o que constituía um indício de que haviam</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dls2.show_batch(max_n=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Fine-tuning the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*(text from Sylvain Gugger Transformers Tutorial)* The Hugging Face model will return a tuple in outputs, with the actual predictions and some additional activations (should we want to use them is some regularization scheme). To work inside the fastai training loop, we will need to drop those using a `Callback`: we use those to alter the behavior of the training loop.\n",
    "\n",
    "Here we need to write the event `after_pred` and replace `self.learn.pred` (which contains the predictions that will be passed to the loss function) by just its first element. In callbacks, there is a shortcut that lets you access any of the underlying `Learner` attribute so we can write `self.pred[0]` instead of `self.learn.pred[0]`. That shorcut only works for read access, not write, so we have to write `self.learn.pred` on the right side (otherwise we would set a `pred` attribute in the `Callback`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DropOutput(Callback):\n",
    "    def after_pred(self): self.learn.pred = self.pred[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course we could make this a bit more complex and add some penalty to the loss using the other part of the tuple of predictions, like the `RNNRegularizer`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.1 Splitter (get layers groups)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model has 2 main layers groups: `transformer` and `lm_head`. As we can read in [The illustrated GPT2](http://jalammar.github.io/illustrated-gpt2/#model-output), the `lm_head` is a copy of the embeddings matrix `wte`. Therefore, we need to split only the `transformer` layers group to get all layers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `transformer`\n",
    "  - (wte) embeddings (tokens --> embeddings of tokens)\n",
    "  - (wpe) positionning \n",
    "  - 12 decoder blocks (attention heads)\n",
    "- `lm_head`\n",
    "  - LayerNorm\n",
    "\n",
    "Note: the last layer is a copy of the embeddings matrix wte in order to get after the softmax probability of each token in the vocab."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can create our layers groups that will allow us to use fastai v2 fine-tuning techniques:\n",
    "- Differential learning rates (a specific learning rate is setup by layers group)\n",
    "- Gradual unfreezing (layers groups are defined allowing to decide the layers to be trained)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We decided to follow the fine-tuning method showed in the notebook [10_nlp.ipynb](https://github.com/fastai/fastbook/blob/master/10_nlp.ipynb) by creating 4 layers groups: 3 layers groups of 4 decoder blocks and one embeddings groups with the wte and wpe matrices."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### GPT2 layers groups"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GPT2LMHeadModel(\n",
       "  (transformer): GPT2Model(\n",
       "    (wte): Embedding(50257, 768)\n",
       "    (wpe): Embedding(1024, 768)\n",
       "    (drop): Dropout(p=0.1, inplace=False)\n",
       "    (h): ModuleList(\n",
       "      (0): Block(\n",
       "        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (attn): Attention(\n",
       "          (c_attn): Conv1D()\n",
       "          (c_proj): Conv1D()\n",
       "          (attn_dropout): Dropout(p=0.1, inplace=False)\n",
       "          (resid_dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (mlp): MLP(\n",
       "          (c_fc): Conv1D()\n",
       "          (c_proj): Conv1D()\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (1): Block(\n",
       "        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (attn): Attention(\n",
       "          (c_attn): Conv1D()\n",
       "          (c_proj): Conv1D()\n",
       "          (attn_dropout): Dropout(p=0.1, inplace=False)\n",
       "          (resid_dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (mlp): MLP(\n",
       "          (c_fc): Conv1D()\n",
       "          (c_proj): Conv1D()\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (2): Block(\n",
       "        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (attn): Attention(\n",
       "          (c_attn): Conv1D()\n",
       "          (c_proj): Conv1D()\n",
       "          (attn_dropout): Dropout(p=0.1, inplace=False)\n",
       "          (resid_dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (mlp): MLP(\n",
       "          (c_fc): Conv1D()\n",
       "          (c_proj): Conv1D()\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (3): Block(\n",
       "        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (attn): Attention(\n",
       "          (c_attn): Conv1D()\n",
       "          (c_proj): Conv1D()\n",
       "          (attn_dropout): Dropout(p=0.1, inplace=False)\n",
       "          (resid_dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (mlp): MLP(\n",
       "          (c_fc): Conv1D()\n",
       "          (c_proj): Conv1D()\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (4): Block(\n",
       "        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (attn): Attention(\n",
       "          (c_attn): Conv1D()\n",
       "          (c_proj): Conv1D()\n",
       "          (attn_dropout): Dropout(p=0.1, inplace=False)\n",
       "          (resid_dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (mlp): MLP(\n",
       "          (c_fc): Conv1D()\n",
       "          (c_proj): Conv1D()\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (5): Block(\n",
       "        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (attn): Attention(\n",
       "          (c_attn): Conv1D()\n",
       "          (c_proj): Conv1D()\n",
       "          (attn_dropout): Dropout(p=0.1, inplace=False)\n",
       "          (resid_dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (mlp): MLP(\n",
       "          (c_fc): Conv1D()\n",
       "          (c_proj): Conv1D()\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (6): Block(\n",
       "        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (attn): Attention(\n",
       "          (c_attn): Conv1D()\n",
       "          (c_proj): Conv1D()\n",
       "          (attn_dropout): Dropout(p=0.1, inplace=False)\n",
       "          (resid_dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (mlp): MLP(\n",
       "          (c_fc): Conv1D()\n",
       "          (c_proj): Conv1D()\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (7): Block(\n",
       "        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (attn): Attention(\n",
       "          (c_attn): Conv1D()\n",
       "          (c_proj): Conv1D()\n",
       "          (attn_dropout): Dropout(p=0.1, inplace=False)\n",
       "          (resid_dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (mlp): MLP(\n",
       "          (c_fc): Conv1D()\n",
       "          (c_proj): Conv1D()\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (8): Block(\n",
       "        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (attn): Attention(\n",
       "          (c_attn): Conv1D()\n",
       "          (c_proj): Conv1D()\n",
       "          (attn_dropout): Dropout(p=0.1, inplace=False)\n",
       "          (resid_dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (mlp): MLP(\n",
       "          (c_fc): Conv1D()\n",
       "          (c_proj): Conv1D()\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (9): Block(\n",
       "        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (attn): Attention(\n",
       "          (c_attn): Conv1D()\n",
       "          (c_proj): Conv1D()\n",
       "          (attn_dropout): Dropout(p=0.1, inplace=False)\n",
       "          (resid_dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (mlp): MLP(\n",
       "          (c_fc): Conv1D()\n",
       "          (c_proj): Conv1D()\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (10): Block(\n",
       "        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (attn): Attention(\n",
       "          (c_attn): Conv1D()\n",
       "          (c_proj): Conv1D()\n",
       "          (attn_dropout): Dropout(p=0.1, inplace=False)\n",
       "          (resid_dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (mlp): MLP(\n",
       "          (c_fc): Conv1D()\n",
       "          (c_proj): Conv1D()\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "      (11): Block(\n",
       "        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (attn): Attention(\n",
       "          (c_attn): Conv1D()\n",
       "          (c_proj): Conv1D()\n",
       "          (attn_dropout): Dropout(p=0.1, inplace=False)\n",
       "          (resid_dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (mlp): MLP(\n",
       "          (c_fc): Conv1D()\n",
       "          (c_proj): Conv1D()\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "    (ln_f): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "  )\n",
       "  (lm_head): Linear(in_features=768, out_features=50257, bias=False)\n",
       ")"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#model_en = model_en.train().to('cuda')\n",
    "model_en"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Block(\n",
       "  (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "  (attn): Attention(\n",
       "    (c_attn): Conv1D()\n",
       "    (c_proj): Conv1D()\n",
       "    (attn_dropout): Dropout(p=0.1, inplace=False)\n",
       "    (resid_dropout): Dropout(p=0.1, inplace=False)\n",
       "  )\n",
       "  (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "  (mlp): MLP(\n",
       "    (c_fc): Conv1D()\n",
       "    (c_proj): Conv1D()\n",
       "    (dropout): Dropout(p=0.1, inplace=False)\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Example of a decoder block (attention head)\n",
    "model_en.transformer.h[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Split a GPT2 model in 4 groups for differential learning rates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "def splitter(model):\n",
    "    \"Split a GPT2 `model` in 3 groups for differential learning rates.\"\n",
    "    \n",
    "    # First layers group : decoder blocks from 0 to 3\n",
    "    modules = []\n",
    "    for i in range(4): modules.append(model.transformer.h[i])\n",
    "    groups = [nn.Sequential(*modules)]\n",
    "\n",
    "    # Second layers group : decoder blocks from 4 to 7\n",
    "    modules = []\n",
    "    for i in range(4,8,1): modules.append(model.transformer.h[i])\n",
    "    groups = L(groups + [nn.Sequential(*modules)])\n",
    "\n",
    "    # Third layers group : decoder blocks from 8 to 11\n",
    "    modules = []\n",
    "    for i in range(8,12,1): modules.append(model.transformer.h[i])\n",
    "    groups = L(groups + [nn.Sequential(*modules)])\n",
    "    \n",
    "    # Fourth layers group : embeddings matrices wte and wpe + LayerNorm at the model output\n",
    "    groups = L(groups + [nn.Sequential(model.transformer.wte,model.transformer.wpe,model.transformer.ln_f)])\n",
    "    \n",
    "    return groups.map(params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6.2 Learner"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*(text from Sylvain Gugger Transformers Tutorial)* Now, we are ready to create our `Learner`, which is a fastai object grouping data, model and loss function and handles model training or inference. Since we are in a language model setting, we pass accuracy and perplexity as metrics, and we need to use the callback we just defined. Lastly, we use mixed precision to save every bit of memory we can (and if you have a modern GPU, it will also make training faster)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Learner: basic class for handling the training loop\n",
    "# source: https://dev.fast.ai/learner#Learner\n",
    "learn = Learner(dls, model_en, loss_func=CrossEntropyLossFlat(),\n",
    "                splitter = splitter,\n",
    "                cbs=[DropOutput], \n",
    "                metrics=[accuracy, Perplexity()]).to_fp16()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "number of parameters groups: 4\n",
      "0 {'wd': 0.01, 'sqr_mom': 0.99, 'lr': 0.001, 'mom': 0.9, 'eps': 1e-05}\n",
      "1 {'wd': 0.01, 'sqr_mom': 0.99, 'lr': 0.001, 'mom': 0.9, 'eps': 1e-05}\n",
      "2 {'wd': 0.01, 'sqr_mom': 0.99, 'lr': 0.001, 'mom': 0.9, 'eps': 1e-05}\n",
      "3 {'wd': 0.01, 'sqr_mom': 0.99, 'lr': 0.001, 'mom': 0.9, 'eps': 1e-05}\n"
     ]
    }
   ],
   "source": [
    "# Check the number of parameters groups and the hyperparameters values\n",
    "learn.create_opt()\n",
    "print(f'number of parameters groups: {len(learn.opt.param_groups)}')\n",
    "\n",
    "# ... and the list of Learning Rates (before its atualization by the Optimizer of the function fit_one_cycle())\n",
    "for i,h in enumerate(learn.opt.hypers):\n",
    "    print(i,h)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can check how good the model is without any fine-tuning step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4h 45min 53s, sys: 1h 1min 36s, total: 5h 47min 30s\n",
      "Wall time: 53min 2s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(#3) [9.949938774108887,0.09898579120635986,20950.939453125]"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "# loss, accuracy, Perplexity() of validation dataset\n",
    "learn.validate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Loss = 9.95\n",
    "- accuracy = 0.099\n",
    "- perplexity = 20950.94"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a `Learner`, we will use during training all the **fine-tuning techniques** seen for classification model training (see the notebook [10_nlp.ipynb](https://github.com/fastai/fastbook/blob/master/10_nlp.ipynb) about \"NLP Deep Dive: RNNs\") to take advantage of the **Transfer Learning** of the GPT-2 pre-trained embeddings and model from Hugging Face Transformers:\n",
    "- **learning rate finder** (method that helps finding the best learning rate to train the model)\n",
    "- **Mixed precision training** (some of the operations will be done in FP16, others in FP32 in order to speed up the training)\n",
    "- **gradual unfreezing** (the model has 4 layers groups created by our method `splitter` : the embedding one and the 3 groups of 4 decoder blocks each)\n",
    "- **1cycle policy** with the method [fit_one_cycle()](https://dev.fast.ai/callback.schedule#Learner.fit_one_cycle) (The 1cycle policy was introduced by Leslie N. Smith et al. in <a href=\"https://arxiv.org/abs/1708.07120\">Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates</a>. It schedules the learning rate with a cosine annealing from `lr_max/div` to `lr_max` then `lr_max/div_final` (pass an array to `lr_max` if you want to use differential learning rates) and the momentum with cosine annealing according to the values in `moms`. The first phase takes `pct_start` of the training. You can optionally pass additional `cbs` and `reset_opt`.)\n",
    "- **differential learning rates** (each layers group with a learning rate different: the biggest one for the embeddings group, and the smallest one for the first 4 decoder blocks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 6.2.1 Freeze all layers but the last layers group (do not freeze `wte`, `wpe` embeddings matrices and last `LayerNorm`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GPT2LMHeadModel (Input shape: ['8 x 1024'])\n",
       "================================================================\n",
       "Layer (type)         Output Shape         Param #    Trainable \n",
       "================================================================\n",
       "Embedding            8 x 1024 x 768       38,597,376 True      \n",
       "________________________________________________________________\n",
       "Embedding            8 x 1024 x 768       786,432    True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Linear               8 x 1024 x 50257     38,597,376 True      \n",
       "________________________________________________________________\n",
       "\n",
       "Total params: 163,037,184\n",
       "Total trainable params: 77,982,720\n",
       "Total non-trainable params: 85,054,464\n",
       "\n",
       "Optimizer used: <function Adam at 0x7fce2f8dae60>\n",
       "Loss function: FlattenedLoss of CrossEntropyLoss()\n",
       "\n",
       "Model frozen up to parameter group number 3\n",
       "\n",
       "Callbacks:\n",
       "  - DropOutput\n",
       "  - ModelToHalf\n",
       "  - TrainEvalCallback\n",
       "  - Recorder\n",
       "  - ProgressCallback\n",
       "  - MixedPrecision"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.freeze()\n",
    "learn.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `learn.summary ()` method gives almost the right numbers. In fact, it counts twice the weights of the wte matrix (vocab embeddings) because they are duplicated in the weights of the output linear layer.\n",
    "\n",
    "The real numbers are:\n",
    "- Total params: 163,037,184 - 38,597,376 = **124,439,808** (about 124 millions)\n",
    "- Total trainable params: 77,982,720 - 38,597,376 = **39,385,344** (about 40 millions)\n",
    "- Total non-trainable params: **85,054,464** (about 85 millions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4min 21s, sys: 58.9 s, total: 5min 20s\n",
      "Wall time: 1min 5s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "SuggestedLRs(lr_min=0.005754399299621582, lr_steep=2.2908675418875646e-06)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXhV5bn38e+9M5KQgYQEQpgFFWSUiFLn4lwrak+ttLZap9raqp3eY9u357ztOVU7D9rW0jq1Kq2tUrEialXAAdQgICigBETDlEBCQubpfv/YG4y4AwGzp+T3ua59Za+119rrzjbuH89az3oec3dERET2F4h1ASIiEp8UECIiEpYCQkREwlJAiIhIWAoIEREJSwEhIiJhJce6gJ40cOBAHzlyZKzLEBFJGMuXL9/p7gXhXutVATFy5EhKS0tjXYaISMIws81dvaZTTCIiEpYCQkREwlJAiIhIWAoIEREJSwEhIiJhKSBERCQsBYSISAJ7Y2sNS96qjMh7KyBERBLY/cs2882/r4rIe0csIMzsbjOrMLM1ndb91MzWmdnrZjbPzHK72PccM1tvZhvM7OZI1SgikuhqG9vITo/MPc+RbEHcC5yz37qngQnuPgl4C/jO/juZWRLwW+BcYDww28zGR7BOEZGEVdPYSk6/lIi8d8QCwt2XAFX7rXvK3dtCi8uAoWF2nQ5scPeN7t4C/BWYFak6RUQSWW1TK9mJFhDdcCXwRJj1xcB7nZbLQ+tERGQ/tY2tZKf3ooAws+8BbcAD4V4Os84P8F7XmlmpmZVWVkbmSr6ISLxKyFNMXTGzy4Hzgc+5e7gv/nJgWKflocDWrt7P3ee4e4m7lxQUhB2xVkSkV3J3apvayO6XeBepP8TMzgH+E7jA3Ru62OxVYKyZjTKzVOBSYH60ahQRSRT1Le20d3jitSDMbC6wFDjKzMrN7CrgDiALeNrMVprZnaFth5jZAoDQReyvAk8Ca4GH3P2NSNUpIpKoahtbASJ2DSJiEwa5++wwq+/qYtutwHmdlhcACyJUmohIr1DbFAqIRGtBiIhIZNU0BAMi4U4xiYhIZNU2BW8r61XdXEVE5KOraVQLQkREwth3kbo3dHMVEZGes7cFkaVTTCIi0lltUytZackkBcINQPHRKSBERBJUbWNbxLq4ggJCRCRh1TRGbiRXUECIiCSs2qbWiE0WBAoIEZGEVRvBkVxBASEikrBqdYpJRETCieRcEKCAEBFJSG3tHdS3tEdsmA1QQIiIJKQ9e8dhitBd1KCAEBFJSJEehwkUECIiCWnfXBA6xSQiIp3ta0FkKCBERKST2sbIzgUBCggRkYRUE+GhvkEBISKSkPZeg9BFahER+YDaxlaSA0a/lKSIHUMBISKSgPbeRW0WmbkgQAEhIpKQapsiOxcERDAgzOxuM6swszWd1n3azN4wsw4zKznAvu+Y2WozW2lmpZGqUUQkUUV6LgiIbAviXuCc/datAS4GlnRj/9PdfYq7dxkkIiJ9VW1jZOeCgAgGhLsvAar2W7fW3ddH6pgiIn1FbVNityA+CgeeMrPlZnbtgTY0s2vNrNTMSisrK6NUnohIbEV6siCI34A40d2PBc4FrjezU7ra0N3nuHuJu5cUFBREr0IRkRhxd2ob2yJ6FzXEaUC4+9bQzwpgHjA9thWJiMSPptYOWto7+l4LwswyzSxr73PgLIIXt0VEhE4juUZwmA2IbDfXucBS4CgzKzezq8zsIjMrB2YAj5vZk6Fth5jZgtCug4AXzGwV8ArwuLsvjFSdIiKJZt84TBE+xRSx+HH32V28NC/MtluB80LPNwKTI1WXiEiiq43CZEEQh6eYRETkwN4/xaSAEBGRTqIx3SgoIEREEs77kwUl6EVqERGJjPcnC1ILQkREOqltbCUjNYmUpMh+hSsgREQSTE1ja8S7uIICQkQk4dQ2RX4cJlBAiIgknNrGtojfRQ0KCBGRhFMThZFcQQEhIpJwapt0DUJERMKIxnSjoIAQEUkoHR1OXXObAkJERD5oT3Mb7pG/ixoUECIiCSVaI7mCAkJEJKFEa5gNUECIiCSU2ihNFgQKCBGRhFLdEAyIvMzUiB9LASEikkCq6psBGJCpFoSIiHRSVR9sQQzIUAtCREQ6qW5oISs9OeJDfYMCQkQkoVTVt5AfhesPoIAQEUko1Q0tDEj0gDCzu82swszWdFr3aTN7w8w6zKzkAPueY2brzWyDmd0cqRpFRBJNVX0LeVG4/gCRbUHcC5yz37o1wMXAkq52MrMk4LfAucB4YLaZjY9QjSIiCaW6vhe0INx9CVC137q17r7+ILtOBza4+0Z3bwH+CsyKUJkiIgmlqqElKvdAQHxegygG3uu0XB5aF5aZXWtmpWZWWllZGfHiRERipbGlnabWjqh0cYX4DAgLs8672tjd57h7ibuXFBQURLAsEZHY2hW6SS4vCjfJQXwGRDkwrNPyUGBrjGoREYkb1VG8SQ7iMyBeBcaa2SgzSwUuBebHuCYRkZiramgBojMOE0S2m+tcYClwlJmVm9lVZnaRmZUDM4DHzezJ0LZDzGwBgLu3AV8FngTWAg+5+xuRqlNEJFFU10c3ICI2JZG7z+7ipXlhtt0KnNdpeQGwIEKliYgkpKooB0Q8nmISEZEwqhtaCFh05oIABYSISMKoqm9hQEYqgUC4zp49TwEhIpIgojkOEyggREQSRjTHYQIFhIhIwqiqb4nKTHJ7KSBERBJEVX1r1HowgQJCRCQhuHvwGoROMYmISGe1TW20d7haECIi8kHRvosaFBAiIglh7zhM6uYqIiIfsK8FoWsQIiLSWbTHYQIFhIhIQqjWKSYREQlnV30LqUkBMlOTonZMBYSISAKoDt1FbRadgfpAASEikhCq6lujepMcKCBERBJCdUNLVC9QgwJCRCQhVNfHaUCY2RFmlhZ6fpqZ3WBmuZEtTURE9qqK4xbEw0C7mY0B7gJGAQ9GrCoREdmnrb2Dmsb4vQbR4e5twEXAr9z960BR5MoSEZG9ahpbcY/uTXLQ/YBoNbPZwOXAv0LrojdrhYhIHxaLm+Sg+wHxRWAG8CN332Rmo4D7D7SDmd1tZhVmtqbTujwze9rM3g79HNDFvu1mtjL0mN/dX0ZEpDfaVRf9cZigmwHh7m+6+w3uPjf0pZ7l7rcdZLd7gXP2W3cz8Iy7jwWeCS2H0+juU0KPC7pTo4hIb/V+CyK6J26624tpkZllm1kesAq4x8x+caB93H0JULXf6lnAfaHn9wEXHmK9IiJ9TlV9KxC/1yBy3L0WuBi4x92nAWccxvEGufs2gNDPwi62SzezUjNbZmYKERHp0/a1IKJ8iim5u9uZWRFwCfC9CNaz13B332pmo4FnzWy1u5eF29DMrgWuBRg+fHgUShMRia6q+hYyU5NIT4neQH3Q/RbED4EngTJ3fzX0xf32YRxvRyhoCP2sCLeRu28N/dwILAKmdvWG7j7H3UvcvaSgoOAwShIRiW/Bgfqi23qA7l+k/ru7T3L3L4eWN7r7pw7jePMJdpUl9PPR/TcwswGd7toeCJwIvHkYxxIR6RVicRc1dP8i9VAzmxfqtrrDzB42s6EH2WcusBQ4yszKzewq4DbgTDN7GzgztIyZlZjZn0K7jgNKzWwV8Bxwm7srIESkz6qub4n69Qfo/jWIewgOrfHp0PJloXVndrWDu8/u4qWZYbYtBa4OPX8JmNjNukREer2qhhZGF/SP+nG7ew2iwN3vcfe20ONeQCf8RUSioKquhdyM6A9e0d2A2Glml5lZUuhxGbArkoWJiAg0tbZT39JOfrxegwCuJNjFdTuwDfgPgsNviIhIBO29ByIvMy3qx+5uL6Z33f0Cdy9w90J3v5DgTXMiIhJBVfV7AyJ+TzGF840eq0JERMJ6PyDitAXRBeuxKkREJKxEbUF4j1UhIiJhVcewBXHA+yDMbA/hg8CAfhGpSERE9qmqb8EMcvpFvwVxwIBw96xoFSIiIh9W1dBCbr8UkgLRP6v/UU4xiYhIhFXVx2YcJlBAiIjENQWEiIiEpYAQEZGwqupbFRAiIvJB7k51jOaCAAWEiEjcqm1so73DYzIXBCggRETiVtW+gfoUECIi0klVfTOggBARkf1U1bcCCggREdmPWhAiIhKWWhAiIhJWVX0z6SkBMlIPOGxexCggRETiVFV9K3kx6uIKBxnN9aMys7uB84EKd58QWpcH/A0YCbwDXOLu1WH2vRz4v6HF/3X3+yJZa09pbe/glU1VLFyznWfXVWAGw/MyGJGfwYj8TCYW5zBxaA7Z6dEfuldEEkt1QwsDYnR6CSIcEMC9wB3Anzutuxl4xt1vM7ObQ8v/2XmnUIj8N1BCcD6K5WY2P1yQxIPW9g5eKtvFv1Zt5em1O9jd0Eq/lCROOXIg6SlJvFvVwFNv7GBXaOIPgCMKMpk2YgCnHlnISWMGkpPx4cDo6AjeRVmxp5ncjBSKcjQFh0hfsiuG4zBBhAPC3ZeY2cj9Vs8CTgs9vw9YxH4BAZwNPO3uVQBm9jRwDjC3p2tsbmvn1gXrOGv8ID42ZuAh7VvT2MqPF65jwept7G5oJSstmTPHD+LsCYM5ZWwB/VKTPrh9Qyuvb9nNqvd2s/K93Sxcs52HSstJChhThuUyICOF2sY2apta2d3Qys66Zto6gvM1JQeMq08ezQ0zx8TsfKSIRFd1fQsj8zNidvxYfNMMcvdtAO6+zcwKw2xTDLzXabk8tK7HtbU7z79dyeOrt7HghpMpyOr+tH7/b/4bzF+1lfMnFXH+pCGccuRA0pKTutw+JyOFk8cWcPLYgtCxO1hVvptF6yt5YcNOttU0kZWezPC8DCYUp1CYlUZhVhoFWeksWl/BnYvLmL9yC//1yWM4+5hBmH14AhH3YKCEe01EEkssR3KF2AREd4T7dgs7B7aZXQtcCzB8+PBDPlBmWjJ3fPZYLvzti3zjoZXc98XpBLoxc9PTb+5g3oot3DhzLF8/88hDPi5AclKAaSPymDYij2+eddQBt/3EpCIuOW4Y3//nGq67fzlZackMyklncHY6uRkpVNW3sL22ie01TSQHjONH5zNjdD4fG5PPmIL+JCepP4JIImlua6euua33XqTuwg4zKwq1HoqAijDblPP+aSiAoQRPRX2Iu88B5gCUlJSEDZGDGVeUzX9/8hi+O281v19cxvWnjzng9rsbWvjuvNUcPTjroNv2pONG5vHY107ikdfKeXNrLTtqm9le20R5dQN5makcPTiLU48soKG5naUbd/H0mzsACBgUZKUxODudwaFQGZSTTlFOOlOHDWDkwMyo/Q4i0j3Ve++B6N+3AmI+cDlwW+jno2G2eRK4xcwGhJbPAr4TyaJmTx/G0o27+PlT6zluZB7TR+V1ue0P//UmVfUt3HPFcaQmR/df5ilJAT5zXPdaSlt2N7KsbBebqxrYXtPItpomNlbW81LZLvY0tQGQnhLgwWtO4NjhAw7ybiISTVWhTi29tgVhZnMJtgQGmlk5wZ5JtwEPmdlVwLvAp0PblgDXufvV7l5lZv8DvBp6qx/uvWAdwVq55aIJrC7fzQ1zV7DwppPJDfMf5tl1O3jktS3c8PExTCjOiWRJH1lxbj8+NW1o2Nfqm9t4t6qB6+5fzpX3vso/rpvBmMKsKFcoIl2pDo3kGsturhH956+7z3b3IndPcfeh7n6Xu+9y95nuPjb0syq0bam7X91p37vdfUzocU8k69wrKz2F22cfS2VdMz9euO5Dr+9uaOE7jwRPLX3142OjUVLEZKYlM64omz9fOZ3kQIAv3PUK22uaYl2WiITs7Raf31sDIhFNHJrDlSeOZO4r71H6zvuNFnfne/9cw666Fn726clRP7UUKSPyM7n3i8dR29TG5Xe/QlllHW3tHfte317TxJwlZXziN88z49Zn+M4jr/Psuh00tbbHsGqR3q+6PvYtiHjtxRRTN51xJI+/vo3vzVvDv244iZSkAI+u3Mrjr2/j22cfFfenlg7VhOIc5nx+Gpff8wozf76YlCRjeF4G/dNTeL18N+4weWgOU4blMn/lVua+8h4ZqUmMHZRFUXY6RbnpDBuQwfmTiijMTj+kY9c3t9HY2s7A/t3vXizSF+yqb8EMcvvFbtQFBUQYmWnJ/GDWBK75cyl/en4TF0wZwvcfXUPJiAFcd+oRsS4vIj42ZiALbzqF5Zur2bSznk2V9VTWNXPjzLHMmlLMqFBPp+a2dpZtrOLZtTvYuLOeDZV1PP92JfUt7dyyYC3nTSziihNHMnVYbth7MTo6nI0761i0vpJF6yt5ZVMVLe0djC/K5vSjCzjtqEImD809YAutprGV8uoGtlQ3khQwcjNSGZCRwsCsNA1hIr1GdX0LOf1SYtpF3fbeWNUblJSUeGlpaY+937V/LmXJ25UcNTibDTv28MSNpzA8hnc1xit3551dDfxl6Wb+Xvoee5rbGDqgH4Oz0ynISiO/fypV9S1srKxn0856mtuCp7DGFvbn9KMLyc1IYfH6Sko3V9Pe4SQHjCMK+nN0URYj8jOpqm9m2+4mtuxuZMvuxn09sMKZNDSHjx9dyBnjBnHMkGwA2jqc1vYOtlQ3UhaqYevuRnL6pTCwf+q+YNmbZ+7Q1Brsg17X3EZLWwdDB2QwpjCTEfmZpOieEomC6x94jbXba3n2m6dF9DhmttzdS8K+poDo2tbdjZzxi8U0tLTzk09N4pLjhvXYe/dW9c1tPPJaOS9vqmJnXTM761rYVddMbkYqowdmMrogkzGF/TlxzECGDvhg2NY0tvLShp2s3lLD+u17WLd9D1t2N5KbkcKQnH4Mye3HkNDprKED+lE8oB8dHuztsbuhhfKqRp5bX8GK94KnxZICRntH+L/v7PRk6prb6OLlLiUHjHFF2cyaMoQLpgyhMOvQTqmJdNfsOctobe/gH1/+WESPo4D4CBau2cab2/bw9TPGaviKGGht7zjkf7HvrGvmuXUVbNxZT0pSgJSAkZwUoCgnndEFmYwcmEl2esq+wRAr65qp269Vkp6SRP+0ZPqnJ5McMN6taqCsso4NFXW88PZOVpXXkBQwTh47kJlHF3LsiAEcPTibpG7chS/SHWf/cgkj8jOY84Ww3909RgEh0sM2VNQxb0U5/1yxlS27GwHITE1i0tBchudlUJSbzpCcfhw5OIvJQ3P0jws5ZMf96N/MPLqQ2z41KaLHOVBA6CK1yGEYU9ifb599NN866yjKqxtZvrma5Zureb18N8+sq2BnXfO+bYcO6McnJw/hgslDOHpwlsJCDsrdqa6P7VwQoIAQ+UjMjGF5GQzLy+DCqe8PONzc1s6OmmZefaeK+au2MmfJRn6/qIzRAzM585hBnDV+MFOH5XZrYEjpe2qb2mjr8JjeJAcKCJGISEtOYnh+BsPzM/jUtKHsqmvmiTXbefKN7dz1/Cb+sHgjBVlpnDV+EOdMGMwJo/PVO0r22XeTXAzHYQIFhEhU5PdP47ITRnDZCSOoaWxl0foKnnxjO/NWbOGBl98lOz2ZmeMGMXNcIaccWaD7Ofq4vcNsxHIkV1BAiERdTr8UZk0pZtaUYppa23n+7Z08sWYbz62rYN6KLSQHjONG5nHBlCHMmjJEMwj2QdVxMJIrKCBEYio9JYkzxw/izPGDaO9wVrxbzbPrKnjqzR1855HV3PL4Wj41bSiXnTBco+32IfuG+tY1CBGB4I19JSPzKBmZx7fPPorX3q3mL0s38+DL73LvS+9w+lEFfOnUIzh+VJ56QvVyVQ0KCBHpgpntm472++c37wuJS+csY8qwXK47dTRnjR+sXlC9VHV9C6nJATJSu57jPhrUbUIkzuX3T+NrM8fy4s0f538vnEB1QwvX3f8aZ/9qCY+u3PKB4dmld9hW00RhVlrMW4oKCJEEkZ6SxGUnjODZb57Gry+dghnc+NeVnPGLxTyxelusy5MeVFZZxxEF/WNdhgJCJNEkBYxZU4pZeOMp3HnZNNJTkvjyA6/xlQeWf+AObklMHR3Oxsp6BYSIHL5AwDhnwmAe+9pJfPvso/j3mxWc+YvFPLpyC71pjLW+ZlttE42t7RxRmBnrUhQQIokuJSnA9aeP4fEbTmJEfiY3/nUlV977Ku9VNcS6NDkMZRV1AGpBiEjPGTsoi4e//DG+f/54Xt5UxVm/XMKcJWW6iJ1gyioVECISAUkB46qTRvH0N07lxDH53LJgHRf97iW1JhJIWWUd2enJDIzxMBuggBDplYpz+/HHL5Twu88dyzu76rngjhd44e2dsS5LuqGsop4jCvvHvIsrxCggzOxGM1tjZm+Y2U1hXj/NzGrMbGXo8V+xqFMkkZkZ500sYv5XT6IgK40v3P0ydy4u0wXsOBcvXVwhBgFhZhOAa4DpwGTgfDMbG2bT5919Sujxw6gWKdKLjBqYybyvnMi5E4u47Yl1fOWB19jT1BrrsiSM2qZWKvY0992AAMYBy9y9wd3bgMXARTGoQ6TPyExL5o7ZU/nueUfz1Js7mPXbF3lrx55YlyX72VhZD8ARBbHv4gqxCYg1wClmlm9mGcB5wLAw280ws1Vm9oSZHdPVm5nZtWZWamallZWVkapZJOGZGdeecgQPXH08tY2tXPjbF5m/amusy5JO9nVxLeyjLQh3Xwv8GHgaWAisAtr22+w1YIS7TwZuB/55gPeb4+4l7l5SUFAQoapFeo8TRufz+A0nM64omxvmruA3z7yt6xJxoqyyjuSAMTwvI9alADG6SO3ud7n7se5+ClAFvL3f67XuXhd6vgBIMbOBMShVpFcalJ3O3GtO4OJji/nF02/xg8fepKNDIRFrZZV1jMjPiJvpZ2My3LeZFbp7hZkNBy4GZuz3+mBgh7u7mU0nGGS7YlCqSK+VmhzgZ/8xmbyMVP70wiaqG1r42acnx82XU19UFidjMO0Vq/kgHjazfKAVuN7dq83sOgB3vxP4D+DLZtYGNAKXutrAIj0uEDC+94lx5PVP5ScL11PT2MrvPzeNfjGeh6Avam3vYPOues4cPyjWpewTk4Bw95PDrLuz0/M7gDuiWpRIH2VmfOW0MeRlpPLdeav5/F0vc9cVx5HTLyXWpfUp71U10NrucdWCUFtSRAC4dPpw7vjssawq382lc5ZRuUdDh0dTWZx1cQUFhIh0ct7EIv50+XG8s7OeT9/5EuXVGsMpWvYO0jdaLQgRiVenHlnA/VdPp6q+hUvuXMo7O+tjXVKfUFZRR0FWWlyd2lNAiMiHTBuRx4PXnEBjazuX/GEpGyp013WkBcdgip/TS6CAEJEuTCjO4W9fmkGHw2f+sIy122pjXVKv5e5x18UVFBAicgBHDsrioS+dQEpSgNl/XMbyzVWxLqlX2lXfQk1jqwJCRBLL6IL+PPSlGeT2S2H2H1/mMY3f1OPWbQuewouXMZj2UkCIyEENz8/gka+cyKTiHL42dwW/fW6Dxm/qQQvf2EZ6SoCSEQNiXcoHKCBEpFvyMlO5/+rjuWDyEH765Hq+9ffXqW/ef5xNOVRt7R08sXo7M8cNIjMtVoNbhKeAEJFuS09J4teXTuHGmWN5ZEU55/76eV59R9clPoqlG3exq76FT04aEutSPkQBISKHxMz4+plH8rdrZ+A4l/xhKbcuWEtTa3usS0tIj63aSv+0ZE47Kv6mK1BAiMhhmT4qjyduPIVLjxvOH5Zs5IxfLGbeinING34ImtvaWbhmO2cdM4j0lPgbIFEBISKHrX9aMrdePJEHrj6enH4pfP1vq/jE7S/w3PoKXcTuhuff2kltUxufnBx/p5dAASEiPeDEMQN57Ksn8etLp1DX3MoX73mVWb99kcdf30a7WhRdeuz1reRmpHDSmPicD00BISI9IhAwZk0p5plvnMYtF02ktrGV6x98jZk/X8TcV96ltb0j1iXGlcaWdv795g7OnTA4bidpis+qRCRhpSYH+Ozxw3nmm6fx+88dS06/FL7zyGrO/MVi5q/aqmsUIc+tr6C+pT0uey/tpYAQkYhIChjnTizin9efyN1XlJCeksQNc1dw/u0v8NQb2/t8UDy2aisD+6dx/Oj8WJfSJQWEiESUmfHxowex4IaT+dVnplDX3Ma1f1nOGb9czF9febdPdo+tbWrl2XUVfGLiYJICFutyuqSAEJGoCASMC6cW8+w3T+X22VPJSE3i5kdWc/JPnuP3i8rY09Qa6xKjZuHq7TS3dXDh1OJYl3JA8XVft4j0eslJAT45eQjnTyripbJd3Lm4jB8vXMfvFm3gCzNGcMXHRlGQlRbrMiNq3ootjBqYyZRhubEu5YAUECISE2bGiWMGcuKYgawur+H3izfwu0Vl/HHJJs6fVMTnZ4xgyrBczOL3FMzh2Lq7kWWbdnHTzCPj/ndTQIhIzE0cmsPvPjeNjZV13PfSOzz82hYeWbGFicU5fH7GCC6YPCQu7zQ+HI+u3Io7XDg1fnsv7RWTaxBmdqOZrTGzN8zspjCvm5n9xsw2mNnrZnZsLOoUkegaXdCfH8yawLLvzuR/Zh1DU2s7/+cfr3PCrc9wy4K1bN6V2PNjuzvzVpQzbcQARuTH1/Si4US9BWFmE4BrgOlAC7DQzB5397c7bXYuMDb0OB74feiniPQB/dOS+fyMkVx2wgiWbaziL8ve4a4XNjFnyUYmFudwxrhBzBxXyDFDsuP+NE1nb26r5a0ddfzPhRNiXUq3xOIU0zhgmbs3AJjZYuAi4CedtpkF/NmDg7ksM7NcMyty923RL1dEYsXMmHFEPjOOyGd7TROPrCjnmbUV/OqZt/jlv9+iOLcf50wYzHkTi5g6LJdAHHcZBfjnii2kJBnnTyyKdSndEouAWAP8yMzygUbgPKB0v22Kgfc6LZeH1n0oIMzsWuBagOHDh0eiXhGJA4Nz0vnKaWP4ymlj2FnXzLPrKnjqje38Zelm7nphE4Oz0zlnwmA+MamIacMHxF1YtHc4j67cymlHFTIgMzXW5XRL1APC3dea2Y+Bp4E6YBWw/7RU4f7Lhr3t0t3nAHMASkpK+vatmSJ9xMD+aVxSMoxLSoYFbzpbW8Hjq7fx4Cvvcu9L71CYlcY5EwZzytgCpo/OIzs9Jcq4ybsAAAlvSURBVNYl81LZTir2NHNRnN/70FlMejG5+13AXQBmdgvBFkJn5cCwTstDAc2ULiIfkp2ewoVTi7lwajF1zW08s3YHT6zezkOl7/HnpZsJGEwcmsvHjsjn5DEDmTZyAGnJ0e0R9eKGndz88Gpy+qXw8aMLo3rsj8JiMWa7mRW6e4WZDQeeAma4e3Wn1z8BfJXg6afjgd+4+/SDvW9JSYmXlu5/tkpE+qKm1nZWvLubpWU7ebFsF6ve201bh5OeEmD6qHxmjM7nuJEDmDg0J2KBUd/cxq1PrOX+Ze8yemAmP79kMlOHD4jIsQ6XmS1395Jwr8XqPoiHQ9cgWoHr3b3azK4DcPc7gQUEw2ED0AB8MUZ1ikiCSk9J2neB+xtAXXMbL2/cxfNv7+TFDTv58cJ1QHD02UnFOUwelsukoTlMLM5hZH7mIV/DqG1qZUt1I5t31bNxZz2bKut5ccNOttU2cfVJo/jW2Ucl3L0cMWlBRIpaECLSXbvqmlm+uZrSzdWUvlPFG1traW4LzlmRmhygMCuNQdnpFGalkZocoK3DaW932jqcto4OWts7aG13ahtb2bK7kT1NH7yUWpiVxthB/bnpjCM5bmReLH7FbonHFoSISEzl90/jrGMGc9YxgwFoa+/g7Yo6Vm+poayijoo9zeyobeLtijra2jtIChjJgQCBgJGaZCQnBUhJMopz+zF9VB7Fuf0oHtCPEXmZjCrIpH9a4n+9Jv5vICLSA5KTAowrymZcUXasS4kbGu5bRETCUkCIiEhYCggREQlLASEiImEpIEREJCwFhIiIhKWAEBGRsBQQIiISVq8aasPMKoHdQE2n1Tmdlvc+D7duILDzMA7b+b26+/rB1u3/ek/XfbCau1NjuHUHex6Lz7qr9YlY94GWVffB6zrY64dTd2/4Lhnr7jlhj+juveoBzOlqee/zLtaV9sTxuvP6wdZ19Tv0VN0Hq7kn6j7Q5x7Nz7o31d2dv23VHd26e9N3SbhHbzzF9NgBlh87wLqeOl53Xj/Yuq5+h56quzv7ftS6D/a5H47D+ay7Wp+IdXfnb7vzc9V9aK8fTt29+rukV51i+ijMrNS7GNEwniVi3YlYM6juaFPdsdcbWxCHa06sCzhMiVh3ItYMqjvaVHeMqQUhIiJhqQUhIiJhKSBERCQsBYSIiISlgOgGMzvZzO40sz+Z2Uuxrqc7zCxgZj8ys9vN7PJY19NdZnaamT0f+rxPi3U9h8LMMs1suZmdH+tausvMxoU+63+Y2ZdjXU93mdmFZvZHM3vUzM6KdT3dZWajzewuM/tHrGvpjl4fEGZ2t5lVmNma/dafY2brzWyDmd18oPdw9+fd/TrgX8B9kaw3VNtHrhmYBRQDrUB5pGrtrIfqdqAOSCex6gb4T+ChyFT5YT30t7029Ld9CRCVrpk9VPc/3f0a4ArgMxEst3N9PVH3Rne/KrKV9qDDueMvkR7AKcCxwJpO65KAMmA0kAqsAsYDEwmGQOdHYaf9HgKyE6Fm4GbgS6F9/5EonzUQCO03CHgggeo+A7iU4BfW+YlSd2ifC4CXgM8mUt2h/X4OHJuAdUfl/8mP+kiml3P3JWY2cr/V04EN7r4RwMz+Csxy91uBsKcHzGw4UOPutREsF+iZms2sHGgJLbZHrtr39dRnHVINpEWizv310Od9OpBJ8Muh0cwWuHtHvNcdep/5wHwzexx4MHIV7zteT3zeBtwGPOHur0W24qAe/vtOCL0+ILpQDLzXabkcOP4g+1wF3BOxig7uUGt+BLjdzE4GlkSysIM4pLrN7GLgbCAXuCOypR3QIdXt7t8DMLMrgJ2RDocDONTP+zTgYoJhvCCilR3Yof59f41gqy3HzMa4+52RLO4ADvXzzgd+BEw1s++EgiRu9dWAsDDrDnjHoLv/d4Rq6a5DqtndGwiGWqwdat2PEAy3WDvkvxEAd7+350s5JIf6eS8CFkWqmENwqHX/BvhN5MrptkOtexdwXeTK6Vm9/iJ1F8qBYZ2WhwJbY1RLdyVizaC6o011R1ei1t0tfTUgXgXGmtkoM0sleHFxfoxrOphErBlUd7Sp7uhK1Lq7J9ZXySP9AOYC23i/u+dVofXnAW8R7IHwvVjXmeg1q27Vrbrjs+6P8tBgfSIiElZfPcUkIiIHoYAQEZGwFBAiIhKWAkJERMJSQIiISFgKCBERCUsBIb2amdVF+Xh/MrPxPfRe7Wa20szWmNljZpZ7kO1zzewrPXFsEUD3QUjvZmZ17t6/B98v2d3beur9DnKsfbWb2X3AW+7+owNsPxL4l7tPiEZ90vupBSF9jpkVmNnDZvZq6HFiaP10M3vJzFaEfh4VWn+Fmf3dzB4DnrLgrHeLLDgL2zozeyA0/DSh9SWh53UWnNVvlZktM7NBofVHhJZfNbMfdrOVs5TgyKGYWX8ze8bMXjOz1WY2K7TNbcARoVbHT0Pbfjt0nNfN7Ac9+DFKH6CAkL7o18Av3f044FPAn0Lr1wGnuPtU4L+AWzrtMwO43N0/HlqeCtxEcP6H0cCJYY6TCSxz98kEh1y/ptPxfx06/kEHdjOzJGAm74/x0wRc5O7HAqcDPw8F1M1AmbtPcfdvW3AqzrEE5yyYAkwzs1MOdjyRvfrqcN/St50BjA/9ox8g28yygBzgPjMbS3DI5pRO+zzt7lWdll9x93IAM1sJjARe2O84LQRnEgNYDpwZej4DuDD0/EHgZ13U2a/Tey8Hng6tN+CW0Jd9B8GWxaAw+58VeqwILfcnGBixnB9EEogCQvqiADDD3Rs7rzSz24Hn3P2i0Pn8RZ1ert/vPZo7PW8n/P9Lrf7+Rb6utjmQRnefYmY5BIPmeoJzIHwOKACmuXurmb1DcA7v/Rlwq7v/4RCPKwLoFJP0TU8BX927YGZTQk9zgC2h51dE8PjLCJ7aguDw0Afk7jXADcC3zCyFYJ0VoXA4HRgR2nQPkNVp1yeBK81s74XuYjMr7KHfQfoABYT0dhlmVt7p8Q2CX7YloQu3b/L+DF8/AW41sxcJTkYfKTcB3zCzV4AioOZgO7j7CmAVwUB5gGD9pQRbE+tC2+wCXgx1i/2puz9F8BTWUjNbDfyDDwaIyAGpm6tIlJlZBsHTR25mlwKz3X3WwfYTiTZdgxCJvmnAHaGeR7uBK2Ncj0hYakGIiEhYugYhIiJhKSBERCQsBYSIiISlgBARkbAUECIiEpYCQkREwvr/rKryNsD2i3AAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%time\n",
    "learn.lr_find()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The learning rate finder curve suggests picking a lr min of 6e-3. Let's use 2e-3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.803344</td>\n",
       "      <td>3.640777</td>\n",
       "      <td>0.325177</td>\n",
       "      <td>38.121441</td>\n",
       "      <td>5:48:31</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "learn.fit_one_cycle(1, 2e-3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAD4CAYAAAD1jb0+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de3xU9Z3/8dcnyeQGAUIIdzCgCAJCwIgoSrEoCmqxShV/unWxu3TVrsX+/FVa13qpWtatdtvdquu17dZbi/qgraCiglQFFBQhIMhFLuEaAgmX3JPv749MhtwJmUlOzuT9fDx45MyZM2c+35nhnW++c873mHMOERHxnxivCxARkZZRgIuI+JQCXETEpxTgIiI+pQAXEfGpuLZ8sh49eriMjIy2fEoREd9bvXr1Qedcet31bRrgGRkZrFq1qi2fUkTE98xsR0PrNYQiIuJTJw1wM3vezA6YWXaNdd3NbLGZbQ7+TG3dMkVEpK7m9MB/B1xeZ91c4D3n3BDgveBtERFpQycdA3fOLTOzjDqrpwOTgsu/B5YCd0ewLhERAMrKysjJyaG4uNjrUlpdYmIi/fv3JxAINGv7ln6J2cs5txfAObfXzHo2tqGZzQZmAwwcOLCFTyciHVVOTg4pKSlkZGRgZl6X02qcc+Tl5ZGTk8OgQYOa9ZhW/xLTOfe0cy7LOZeVnl7vKBgRkSYVFxeTlpYW1eENYGakpaWd0l8aLQ3w/WbWJ/ikfYADLdyPiMhJRXt4VzvVdrY0wP8C3BxcvhlY0ML9NMvW3GN8uPlgaz6FiIjvNOcwwpeB5cBQM8sxs+8B84BLzWwzcGnwdqu55PEPuOm5la35FCIiDcrPz+eJJ5445cdNmzaN/Pz8VqjohOYchXJDI3dNjnAtTdTQVs8kIlJbdYDfdttttdZXVFQQGxvb6OMWLlzY2qW17an0IiJ+M3fuXLZu3UpmZiaBQIDOnTvTp08f1qxZw4YNG7j66qvZtWsXxcXF/PCHP2T27NnAialDjh07xtSpU7nwwgv5+OOP6devHwsWLCApKSns2nwV4JWVjpiYjvFlhojU98Bf17Nhz5GI7nN43y7cd9WIRu+fN28e2dnZrFmzhqVLl3LFFVeQnZ0dOtTv+eefp3v37hQVFXHuuedy7bXXkpaWVmsfmzdv5uWXX+aZZ57huuuu47XXXuOmm24Ku3ZfzIUy7ezeAJRVVnpciYh0dOPGjat1nPZvfvMbRo8ezfjx49m1axebN2+u95hBgwaRmZkJwDnnnMP27dsjUosveuCj+3dj4bp9lFc4EnxRsYi0hqZ6ym2lU6dOoeWlS5fy7rvvsnz5cpKTk5k0aVKDx3EnJCSElmNjYykqKopILb7ogcfFVpVZXqFvM0WkbaWkpHD06NEG7ysoKCA1NZXk5GQ2btzIihUr2rQ2X/RnA7FV494aQhGRtpaWlsaECRMYOXIkSUlJ9OrVK3Tf5ZdfzlNPPcWoUaMYOnQo48ePb9PafBHgcTHqgYuId1566aUG1yckJLBo0aIG76se5+7RowfZ2aHZuLnrrrsiVpcvhlBCPfAK9cBFRKr5JMCDPfBK9cBFRKr5IsDj1AMXEanHHwEeHANXgIuInOCLAK8eA9eXmCIiJ/gkwKvHwNUDFxGp5osAPzEGrh64iLRvnTt3BmDPnj3MmDGjwW0mTZrEqlWrwn4uXwR4QGdiiojP9O3bl/nz57fqc/jkRB6diSki3rj77rs57bTTQvOB33///ZgZy5Yt4/Dhw5SVlfHQQw8xffr0Wo/bvn07V155JdnZ2RQVFTFr1iw2bNjAWWedFbG5UHwR4NU98LJyBbhIh7ZoLuxbF9l99j4bpjZ+UbGZM2cyZ86cUID/6U9/4q233uLOO++kS5cuHDx4kPHjx/Otb32r0WtaPvnkkyQnJ7N27VrWrl3L2LFjI1K6rwJcJ/KISFsbM2YMBw4cYM+ePeTm5pKamkqfPn248847WbZsGTExMezevZv9+/fTu3fvBvexbNky7rjjDgBGjRrFqFGjIlKbLwJcJ/KICNBkT7k1zZgxg/nz57Nv3z5mzpzJiy++SG5uLqtXryYQCJCRkdHgNLI1neoV55vDH19iajIrEfHQzJkzeeWVV5g/fz4zZsygoKCAnj17EggEWLJkCTt27Gjy8RMnTuTFF18EIDs7m7Vr10akLl/0wANx6oGLiHdGjBjB0aNH6devH3369OHGG2/kqquuIisri8zMTIYNG9bk42+99VZmzZrFqFGjyMzMZNy4cRGpyx8BHhwDL1WAi4hH1q078eVpjx49WL58eYPbHTt2DKi6qHH1NLJJSUm88sorEa8prCEUM/uhmWWb2XozmxOpouqKjwsGuI5CEREJaXGAm9lI4J+BccBo4EozGxKpwmqKVw9cRKSecHrgZwErnHOFzrly4APg25Epq7ZQgKsHLtIhOdcxDmA41XaGE+DZwEQzSzOzZGAaMCCM/TUqJsaIizEFuEgHlJiYSF5eXtSHuHOOvLw8EhMTm/2YFn+J6Zz70sz+HVgMHAO+AMrrbmdms4HZAAMHDmzp0xEfF6MAF+mA+vfvT05ODrm5uV6X0uoSExPp379/s7cP6ygU59xzwHMAZvYIkNPANk8DTwNkZWW1+FdofFyMxsBFOqBAIMCgQYO8LqNdCivAzaync+6AmQ0ErgHOj0xZ9QViY3QcuIhIDeEeB/6amaUBZcDtzrnDEaipQfGxMZRoCEVEJCTcIZSLIlXIycTHxeiCDiIiNfhiLhSo6oGXlld4XYaISLvhnwBXD1xEpBbfBHggVseBi4jU5JsA12GEIiK1+SbAA7E6kUdEpCbfBHhCnI4DFxGpyTcBrlPpRURq802A60xMEZHafBPg8RoDFxGpxTcBnhiIpVgBLiIS4qMAj6G4TGdiiohU81GAx1JUVhH1k7qLiDSXrwLcOV0XU0Skmq8CHKC4TAEuIgK+CvCqUks0Di4iAvgpwOPUAxcRqck/AR4cQilSD1xEBPBVgFeVqkMJRUSq+CjAq4dQFOAiIuDHANfZmCIigK8CvKrUolL1wEVEwFcBXtUDL9GFjUVEAB8GuMbARUSqhBXgZnanma03s2wze9nMEiNVWF2JcdVHoWgMXEQEwghwM+sH3AFkOedGArHAzEgVVldSvI4DFxGpKdwhlDggyczigGRgT/glNezEmZgKcBERCCPAnXO7gV8CO4G9QIFz7p2625nZbDNbZWarcnNzW15ojBEfG6MhFBGRoHCGUFKB6cAgoC/Qycxuqrudc+5p51yWcy4rPT295ZUCCbqog4hISDhDKJcAXzvncp1zZcDrwAWRKathiYFYBbiISFA4Ab4TGG9myWZmwGTgy8iU1TBdVk1E5IRwxsBXAvOBz4B1wX09HaG6GpQUiNUYuIhIUFw4D3bO3QfcF6FaTqrqyvTqgYuIgI/OxISqQwk1hCIiUsVfAR4fq8msRESCfBXgXZMCHCku97oMEZF2wVcB3i0pQH5hqddliIi0C/4K8OQABUVlVFY6r0sREfGcrwK8a1KASgdHNYwiIuKvAO+WHA9AfpGGUURE/BXgSQEA8gvLPK5ERMR7vgrw1E7BAC9SgIuI+CrAq4dQPtjU8mlpRUSiha8C/LTuyQC88PHXHlciIuK9sOZCaWtxsTF0io8lIXiBYxGRjsxXPXCA73/jdA4dL6VA4+Ai0sH5LsDHDOwGwKJ1ez2uRETEW74L8MwBVQG+Zle+x5WIiHjLdwGekhhgUI9O7M4v8roUERFP+S7AAcYP7s6KbXmaG1xEOjRfBvgFp/egrMKxavthr0sREfGMLwN8aO8UAOa8+rnHlYiIeMeXAX5mr6oAL6vQtLIi0nH5MsABArFGQVEZy7fmeV2KiIgnfBvgD199NgA3PLMC59QTF5GOx7cBft25A0LLf1y508NKRES80eIAN7OhZramxr8jZjYnksWdzCf3TAYg53BhWz6tiEi70OLJrJxzm4BMADOLBXYDb0SormbpmZLImb06s3n/sbZ8WhGRdiFSQyiTga3OuR0R2l+zBWJjeH/jAQ4d12XWRKRjiVSAzwRebugOM5ttZqvMbFVubuQvxBAfV9WEWb/7NOL7FhFpz8IOcDOLB74F/Lmh+51zTzvnspxzWenp6eE+XT2v33oBAF9ocisR6WAi0QOfCnzmnNsfgX2dMjNjWPDMzJueXelFCSIinohEgN9AI8MnbeVHl54JwIdbDlJSrgmuRKRjCCvAzSwZuBR4PTLltMyUEb05Pb0TABv2HPGyFBGRNhNWgDvnCp1zac65gkgV1FJP3HgOAN9+4mOPKxERaRu+PROzriE9O4eWH3tnk4eViIi0jagJ8JgY47xB3QFYuinyhyuKiLQ3URPgAK9+/3wA1u0uYNchnV4vItEtqgIcoFtyAICLHl3icSUiIq0r6gL8ldnjQ8srt2mucBGJXlEX4MN6d+H126rOzrz+6RXkF2qOFBGJTlEX4ABjB6aGljMfXOxhJSIirScqAxzgq4emel2CiEiritoAj4+L4Zox/QDYcuCox9WIiERe1AY4wNDgJFeXPL7M40pERCIvqgN89sTBoeWyikoPKxERibyoDnAzI8aqlifquHARiTJRHeAAn987BYC9BcVUVDqPqxERiZyoD/CuwTMzAZ5cusXDSkREIivqAxxg488vB+BIcbnHlYiIRE6HCPDEQCwj+nZh0z4dTigi0aNDBDiAc/DBV7nszi/yuhQRkYjoMAF+zdiqk3omzHvf40pERCKjwwT49y4cFFr+Yle+h5WIiERGhwlwM+Ox74wGYPpvP/K4GhGR8HWYAAe49pz+XpcgIhIxHSrAAZICsQBkzH3T40pERMLT4QL8F9ecHVr++uBxDysREQlPWAFuZt3MbL6ZbTSzL83s/EgV1lqmZ/YNLV/8y6XeFSIiEqZwe+C/Bt5yzg0DRgNfhl9S6zKzWhd7KCqt8LAaEZGWa3GAm1kXYCLwHIBzrtQ554vj8+LjYnjixrEA3P+X9R5XIyLSMuH0wAcDucALZva5mT1rZp3qbmRms81slZmtys3NDePpImvC6T0AeHXVLpZsOuBxNSIipy6cAI8DxgJPOufGAMeBuXU3cs497ZzLcs5lpaenh/F0kVVzlsJZL3zqYSUiIi0TToDnADnOuZXB2/OpCnTf+NGlZ4aWndNc4SLiLy0OcOfcPmCXmQ0NrpoMbIhIVW3kjslDuP+q4QD8z7JtHlcjInJqwj0K5V+BF81sLZAJPBJ+SW1raO8uAMxbtNHjSkRETk1YAe6cWxMc3x7lnLvaOXc4UoW1lfMGdQ8tD/23RR5WIiJyajrcmZh1xcQYj147CoCS8kq25h7zuCIRkebp8AEOcN25AzirT9VQyuTHPiDvWInHFYmInJwCPOivP5gQWj7noXc9rEREpHkU4EFxsTG1JroSEWnvFOA13DBuIFcHJ7v6wUufeVyNiEjTFOB13BK89Nrf1u5lwZrdHlcjItI4BXgdo/p3Cy1vP1joYSUiIk1TgDdg+7wrAPjVu19xpLjM42pERBqmAD+JUfe/43UJIiINUoA34o3bLggtr9p+yMNKREQapgBvxJiBqdwz7SwAZjy13ONqRETqU4A34Z8nDg4t5xeWeliJiEh9CvCT+OP3zgMg88HFLFy31+NqREROUICfxLgasxXe9uJn7Mkv8rAaEZETFOAnER8XQ9ekE5df+8Z/LPGwGhGRExTgzfDFfVP45KeTASircBwvKeeLXfkeVyUiHV2c1wX4Rc8uiWQO6MaaXfmMuO9tAAZ0T+LvP/6mx5WJSEelHvgpuGvK0Fq3dx3SeLiIeEcBfgpGD+hab11xWYUHlYiIKMBPSUpigO3zrmDDg5cxsl/VFXzufm0tlZXO48pEpCNSgLdAcnwcC26/EIAFa/Yw+KcLPa5IRDoiBXgLxcZYrduaL0VE2poCPAxbH5kWWp7x1HIef2eTh9WISEcTVoCb2XYzW2dma8xsVaSK8ovYGAvNHQ7wm/e3kDH3TQ8rEpGOJBI98Iudc5nOuawI7MuX5tW5GHL27gKPKhGRjkRDKBEwPbMfsyZkEBccF7/x2ZU4pyNTRKR1hRvgDnjHzFab2eyGNjCz2Wa2ysxW5ebmhvl07VNSfCz3XTWCzQ9PBaCgqIxBP1nIq5/u9LgyEYlm4Qb4BOfcWGAqcLuZTay7gXPuaedclnMuKz09Pcyna9/MjH/5xumh23e/to5vP/GRhxWJSDQLK8Cdc3uCPw8AbwDjIlGUn82dOozf33LiZfh8Zz7lFZUeViQi0arFAW5mncwspXoZmAJkR6owP/vGmelsq3GI4Rn3LNI0tCISceH0wHsBH5rZF8AnwJvOubciU5b/xcQYr916fuj2jrxCMua+SUFRmYdViUg0aXGAO+e2OedGB/+NcM49HMnCosE5p3Wvt270A+94UImIRCMdRtjKNv78ciYP68nUkb1D63Syj4hEgrXl8cpZWVlu1aoOd8JmyIEjxYx75L166x+9dhTXnTvAg4pExA/MbHVDJ0uqB96GenZJ5MeXD623/sevreV4SbkHFYmInynA29htk87gwekj6q2vvkybiEhzKcA98N3zM/jkp5O5Lqs/T900NrR+4bq9HlYlIn6jMfB24M21e7n9pc9Ct38+fQSPL/6Kz382xcOqRKS90Bh4O3bFqD61bt+7YD2HC8vIeuhd3sre51FVItLeKcDbiS3BibBqOnishH/542qccxSW6ktOEalNQyjtTM7hQlZtP8ycV9c0us3L/zye8YO7Y2aNbiMi0aOxIRQFeDtVXFbBd55azrpmXByi5lWBRCT6aAzcZxIDsfz1Xy/kkW+fzfmD05rcNmPumyzesL+NKhOR9kI9cB/ZsOcI6SkJnPvwu41uo964SPTREEoU2rTvKJf957J661+YdS4XD+0JwLGScvILS+mfmtzW5YlIhCjAo9iBI8W8+ukuHlv8VaPbTM/sy4I1e/jP6zO5eky/NqxORMKlMfAo1rNLIrdffEaT2yxYsweAOa+u4Z431gFwvKScrbnHdAFmEZ9SDzyK7D9SzI68Qq77n+VMOCONj7bkNetxP7r0TO6YPKSVqxORltIQSgd0+HgpR4rLSE9JIC4mhq8PHm9wzLymWyYM4mdXDQegotJx6eMfcMfkIRp2EfGQAlxCWnJBiaV3TSKjR6dWqEZETkYBLiEFRWXgoGty4JTD/OtfTKt1BmhxWQWf7TjMBWf0iHSZIhKkAJeTOlZSzsg685IP7J7MzkOFzXr8/VcNp0tSgE+3HyY1OcD4wWkUl1UwZUTvkz9YRBqlAJdmOVZSTnIglpiYE71s5xyDfrKwxfvMfuAyOifERaI8kQ5JhxFKs3ROiKsV3gBmxvZ5V/DAt+pfSQhg1oSMJvc58r63yZj7Jm9l7+W3S7bw1f6jnPfIu2TMfZNZL3xSa9t1OQWUlFeE1QaRjiLsHriZxQKrgN3OuSub2lY9cP9buG4vI/p24bS0+l9oVo+nr3/gMuLjYvhgUy7/9IdTf7+vHdufx64bDcAbn+fQp2sS408yH0xzOedwjnq/pETas1YbQjGzHwFZQBcFeMdWUVn1WYqtEY7lFZXc/MInzT4mvTH3TDuL4rIKHlv8FedmpPJfN4yld9fERrcvKCrj0bc2MueSM0lPSQitnzDvffqnJvHK7PEcOFpCz5QETcsr7V6rBLiZ9Qd+DzwM/EgBLk355OtDvLY6h5H9unDvgvW8PWciv/v4a17+ZFdom/GDu7Ni26Fm7W/qyN4kBmJ54/PdoXVvzbmINTvzmfv6utC6V2eP55FFG/nm0J786t360w1UTwBWUek4eKyETglxOOfonBCHmVFaXsmOvOMM6ZXS0qaLhKW1Anw+8AsgBbiroQA3s9nAbICBAwees2PHjhY/n0Sv8opK4mJjKK+o5Ix7FgEwoHsSP58+kn984dNWfe4B3ZOYd80obnx2ZZPb3TbpdC44vQc3PXdiu8V3TiQ9JYHMBxdzxdl9+O2NY5vYg0jLRDzAzexKYJpz7jYzm0QjAV6TeuDSHEWlFSzfdpBvDusFwL6CYmY89TH/eX0mYwem8uv3NvPr9zafdD+3TBjE8x993drl1nN6eie25h7noatHUlBUxiVn9WJo76Z771tzj7HzUGFoFsnG5B0rYW9BMSP7dY1kydLOtUaA/wL4B6AcSAS6AK87525q7DEKcImEykrHexsPcOnwXpRVVBKIjeEHL31Gv9QkXlqxk0dnjGLq2VUXis49WsL7G/fTvVMC727YzyPXnF1rjL7uiUzrH7iMT7cfYlvucR7824aI1ZyaHOBwYVmtdXdMHsLGvUd4p8bFOL64bwpdkwLsLSjieEkFK7blMT2zLymJgVr1/u/3xrFiWx7fHNaTrknxnNGzc619HzhSTGqneAKxzTvQ7HhJOTFmJMXHhtNMaSWtehy4euDiV8dKytm07wg/nr+We68czqQ6PeCDx0p4/8sDjD0tlUse/wCoCvnk+Fim/vrvbNx3FIAP776YJ5Zu5aWVO1ulzhf+8VzO6NmZix5d0ug2i354EXe/tpYRfbuEvleIj4thXEZ37r1yOL/7+GvO6JnCe1/uZ8zAbvy/y4Zx4Ggx4x5+L7SPX1xzNj95fR2/v2UcWw8cY9aEDD7aksdpackM6F41p/yXe48QF2P0S01i+M/e5vvfGMzdlw0jJsYoKCrjk68PcenwXrVq25F3nIS42Ca/eG7MvoJiuiTFkRzfcc8lUICLhKmgsIyjJWWhi2McKS5jX0ExPTon0L1TPAD/8fZGTk/vTFrnBOJijLP7d2XU/e94WbYnnrxxLD94+XP+8YIMrs7sx1X//SEAC++4iECscfBYKTc8syK0/egB3fhiVz4PXT2Sm8afxr+/tZEnl27lu+efxh+WV31vdqpXmzpaXEYgNobEQP2/KlZuy+P6p1fw9pyJ9EtNqnWiWfWFUmac059ffmc0t7/4GReckcY9b2QD8KvrR7N8ax7Tzu6Dc3DxsBO/9D/acpAbn13Juz/6Rr2/isKhMzFFPFQ91HPoeClJgViWbDrAbS9+BsDff3wxA7onszYnn+c+/Jqdhwp54FsjGNSjEzvyCrnyvz6sta/P772U837xHqXllXx276WM/fnisOs7f3Aay7eFd6hnW7ln2ll8kZPP39bu5Q+3jGPimelA1TDQsq9yuTX4ulYb2iuFTfuPNrnPP37vPLolB/jdx9uZvzrnlGtKjo+lsLT2CWjXjOnH49dnUlRaQXllZWgYrCUU4CLtTFFpBYFYI64Z49Sl5ZWc+W+LuOSsnjx787kN7uvBv63noiHpjBnYjT5dk6gMHhaZnpLA2+v3kV9YRlJ8LHExMWzYW8Bvl2wFYPW/XUJa5wQqKx2VzvH9/11NflEZB44WU1JWyR//6Tym/KrhaYinjuzN0eJyPtxyMLwXowNYeMdFDO/bpUWPVYCLSC2l5ZXszi9i0ClME7x6xyEGpCZz4GgJyzbnctukqitBFZaWM/xnVROhffXQVD7acpBJQ9PZkVfI7z7ezpxLhrBgzR7KKip56M0vAfjbv15Ij84JdEmKo7C0gh6dExh539scKyln5U8n06tLImUVlcSasXrnYb7z1PJm13l91gDuuGQIE+a93+R2P58+gnsXrK+3PiUhjqMl5QC8c+fE0C+wLQ9PZWvucXp3SWTjviP8n2dXhk5gq/bTacN4ZOHGevvc8vDUZv2ybogCXERa1eodhzmzV+eTDhWUV1RS4RwJcad2xMszy7ZRUFTGfy/Zwi+/M5qhvVK4689fhIZHnrxxLJ0T4+iSGGD0gG4AvPrpTpLj47hqdF9e/XQnYwamUlhawcdbD4Z++ew6VMjDb37J+5sOcO8VZ/EP52cAVdMu1DxLt+7tmuvv+vNaMgd0DT0W4IanV7B8Wx4Zacn83ylDuWp031Nqb00KcBERn9JshCIiUUYBLiLiUwpwERGfUoCLiPiUAlxExKcU4CIiPqUAFxHxKQW4iIhPKcBFRHxKAS4i4lMKcBERn1KAi4j4VJtOZmVmuUBLL0vfA+hIkw6rvdFN7Y1ukW7vac659Lor2zTAw2FmqxqajStaqb3RTe2Nbm3VXg2hiIj4lAJcRMSn/BTgT3tdQBtTe6Ob2hvd2qS9vhkDFxGR2vzUAxcRkRoU4CIiPtXuA9zMLjezTWa2xczmel1POMxsu5mtM7M1ZrYquK67mS02s83Bn6nB9WZmvwm2e62Zja2xn5uD2282s5u9ak9dZva8mR0ws+wa6yLWPjM7J/j6bQk+tv4lwttQI+2938x2B9/jNWY2rcZ9PwnWvsnMLquxvsHPuJkNMrOVwdfhVTOLb7vW1WdmA8xsiZl9aWbrzeyHwfVR+R430d728x4759rtPyAW2AoMBuKBL4DhXtcVRnu2Az3qrHsUmBtcngv8e3B5GrAIMGA8sDK4vjuwLfgzNbic6nXbgrVNBMYC2a3RPuAT4PzgYxYBU9the+8H7mpg2+HBz28CMCj4uY5t6jMO/AmYGVx+CrjV4/b2AcYGl1OAr4Ltisr3uIn2tpv3uL33wMcBW5xz25xzpcArwHSPa4q06cDvg8u/B66usf4PrsoKoJuZ9QEuAxY75w455w4Di4HL27rohjjnlgGH6qyOSPuC93Vxzi13VZ/2P9TYlycaaW9jpgOvOOdKnHNfA1uo+nw3+BkP9jy/CcwPPr7ma+cJ59xe59xnweWjwJdAP6L0PW6ivY1p8/e4vQd4P2BXjds5NP0CtncOeMfMVpvZ7OC6Xs65vVD1gQF6Btc31na/vSaRal+/4HLd9e3RD4JDBs9XDydw6u1NA/Kdc+V11rcLZpYBjAFW0gHe4zrthXbyHrf3AG9o/MvPxz1OcM6NBaYCt5vZxCa2bazt0fKanGr7/NLuJ4HTgUxgL/BYcH3UtNfMOgOvAXOcc0ea2rSBdb5rcwPtbTfvcXsP8BxgQI3b/YE9HtUSNufcnuDPA8AbVP1ptT/4pyPBnweCmzfWdr+9JpFqX05wue76dsU5t985V+GcqwSeoeo9hlNv70Gqhhzi6qz3lJkFqAqzF51zrwdXR+173BnjfbUAAAFbSURBVFB729N73N4D/FNgSPCb2nhgJvAXj2tqETPrZGYp1cvAFCCbqvZUfwt/M7AguPwX4LvBb/LHAwXBP0/fBqaYWWrwT7cpwXXtVUTaF7zvqJmND44dfrfGvtqN6iAL+jZV7zFUtXemmSWY2SBgCFVf2DX4GQ+OAS8BZgQfX/O180TwdX8O+NI593iNu6LyPW6sve3qPfbqG97m/qPqm+yvqPoW9x6v6wmjHYOp+vb5C2B9dVuoGgd7D9gc/Nk9uN6A3wbbvQ7IqrGvW6j6gmQLMMvrttWo62Wq/qQso6rX8b1Itg/ICv5n2Qr8N8EzidtZe/832J61wf/QfWpsf0+w9k3UOLqisc948DPzSfB1+DOQ4HF7L6TqT/y1wJrgv2nR+h430d528x7rVHoREZ9q70MoIiLSCAW4iIhPKcBFRHxKAS4i4lMKcBERn1KAi4j4lAJcRMSn/j/bhjUBDwV19gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#hide\n",
    "learn.recorder.plot_loss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In just **one epoch**, our model passed \n",
    "- from an accuracy of 9.90% to **32.52%**\n",
    "- from a perplexity of 20950.94 to **38.12**\n",
    "\n",
    "Not too bad! \n",
    "\n",
    "Now, We can pass `-2` to `freeze_to` to freeze all except the last two parameter groups."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save(path_data/'GPT2_pt_1epoch_lr2e-3')\n",
    "learn = learn.load(path_data/'GPT2_pt_1epoch_lr2e-3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "788M\t/mnt/home/pierre/.fastai/data/ptwiki/GPT2_pt_1epoch_lr2e-3.pth\r\n"
     ]
    }
   ],
   "source": [
    "# model size\n",
    "!du -hs {path_data}/'GPT2_pt_1epoch_lr2e-3.pth'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 6.2.2 Freeze all layers but the last 2 layers groups"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GPT2LMHeadModel (Input shape: ['8 x 1024'])\n",
       "================================================================\n",
       "Layer (type)         Output Shape         Param #    Trainable \n",
       "================================================================\n",
       "Embedding            8 x 1024 x 768       38,597,376 True      \n",
       "________________________________________________________________\n",
       "Embedding            8 x 1024 x 768       786,432    True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Linear               8 x 1024 x 50257     38,597,376 True      \n",
       "________________________________________________________________\n",
       "\n",
       "Total params: 163,037,184\n",
       "Total trainable params: 106,334,208\n",
       "Total non-trainable params: 56,702,976\n",
       "\n",
       "Optimizer used: <function Adam at 0x7fb385bf8e60>\n",
       "Loss function: FlattenedLoss of CrossEntropyLoss()\n",
       "\n",
       "Model frozen up to parameter group number 2\n",
       "\n",
       "Callbacks:\n",
       "  - DropOutput\n",
       "  - ModelToHalf\n",
       "  - TrainEvalCallback\n",
       "  - Recorder\n",
       "  - ProgressCallback\n",
       "  - MixedPrecision"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.freeze_to(-2)\n",
    "learn.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `learn.summary ()` method gives almost the right numbers. In fact, it counts twice the weights of the wte matrix (vocab embeddings) because they are duplicated in the weights of the output linear layer.\n",
    "\n",
    "The real numbers are:\n",
    "- Total params: 163,037,184 - 38,597,376 = **124,439,808** (about 124 millions)\n",
    "- Total trainable params: 106,334,208 - 38,597,376 = **67,736,832** (about 68 millions)\n",
    "- Total non-trainable params: **56,702,976** (about 57 millions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.453913</td>\n",
       "      <td>3.301886</td>\n",
       "      <td>0.362879</td>\n",
       "      <td>27.163816</td>\n",
       "      <td>5:38:18</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(1, slice(1e-3/(2.6**4),1e-3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO2deXgUVdbG35NOZyWQkAQIBAj7FgKEiOw7GkDBBRUV95FxG3c/wV0cNcO4z6iMzui4ITq4C4ggmyJb0BD2PUhYQyCBhKyd+/3RVd3V1VXV1Z3O0p3ze548VFfdqr7VCW/dPvfc95AQAgzDMEzgE9LQHWAYhmH8Aws6wzBMkMCCzjAMEySwoDMMwwQJLOgMwzBBQmhDvXFCQoJISUlpqLdnGIYJSDZv3nxKCJGodazBBD0lJQXZ2dkN9fYMwzABCREd0jvGIReGYZgggQWdYRgmSGBBZxiGCRIaLIbOMAzjLVVVVcjPz0d5eXlDd6XOiYiIQHJyMqxWq+lzWNAZhgkY8vPzERMTg5SUFBBRQ3enzhBCoLCwEPn5+ejUqZPp8zjkwjBMwFBeXo74+PigFnMAICLEx8d7/U2EBZ1hmIAi2MVcxpf7DDhBP1dehW9yjjR0NxiGYRodASfos77YivsW5GDH0bMN3RWGYZoYRUVFeOutt7w+b9KkSSgqKqqDHrkScIJ+rLgMAHC+srqBe8IwTFNDT9BtNpvheYsXL0ZsbGxddctBwGW5NJX4GcMwjY9Zs2Zh//796N+/P6xWK5o1a4akpCTk5ORgx44duOyyy3D48GGUl5fjvvvuw8yZMwE4rU5KSkowceJEDB8+HL/++ivatWuHb775BpGRkX7pn0dBJ6IIAGsAhEvtFwohnla16QDgAwCxACwAZgkhFvulh+r+SP9y4TyGado8+912v4dee7dtjqcv7aN7PCsrC9u2bUNOTg5WrVqFyZMnY9u2bY7Uwvfeew8tW7ZEWVkZLrjgAlx55ZWIj493ucbevXvx6aef4t1338XVV1+NL774AjNmzPBL/82M0CsAjBVClBCRFcAvRLRECLFe0eYJAJ8LId4mot4AFgNI8UsPVcgDdC6FyjBMQzNo0CCXPPE33ngDX331FQDg8OHD2Lt3r5ugd+rUCf379wcADBw4EHl5eX7rj0dBF/Yq0iXSS6v0o5ZTAaC5tN0CwFF/ddCgX3X9FgzDNGKMRtL1RXR0tGN71apVWL58OdatW4eoqCiMHj1aM488PDzcsW2xWFBWVua3/piaFCUiCxHlADgJYJkQYoOqyTMAZhBRPuyj87/oXGcmEWUTUXZBQYFPHSYp6MJyzjBMfRMTE4Nz585pHisuLkZcXByioqKwa9curF+/XrNdXWJK0IUQNiFEfwDJAAYRUaqqybUA/iuESAYwCcBHROR2bSHEO0KIDCFERmKipj+7ZzjkwjBMAxEfH49hw4YhNTUVjzzyiMuxzMxMVFdXIy0tDU8++SQGDx5c7/3zKstFCFFERKsAZALYpjh0m7QPQoh10kRqAuwjer8iT4q+tnwPhnQZ4u/LMwzDGDJ//nzN/eHh4ViyZInmMTlOnpCQgG3bnNL58MMP+7VvHkfoRJRIRLHSdiSA8QB2qZr9AWCc1KYXgAgAvsVUPPbH/u+Gg6fr4vIMwzABi5mQSxKAlUSUC2AT7DH074loDhFNkdo8BOB2ItoC4FMAN4t6mrWsqREoreBFRgzDMGayXHIBDNDY/5RieweAYf7tmjYE14VFl/zjF+w4dhY75lyMqLCAWyfFMAzjNwJu6X/OYVc/hB3H7AsLzpbxKJ1hmKZNwAl6WZXTM+GFxTsd2+Ghvt3KtiPF+Gi9bhFthmGYgCHgYhTRYRaUVtpF/Z01Bxz7bT6G7C/5xy8AgBsGd6x95xiGYRqQgBuht4vTNrGp4cR0hmEaGc2aNQMAHD16FNOmTdNsM3r0aGRnZ/vl/QJO0PVgPWcYprHStm1bLFy4sM7fJ+AEXZ3lIlNZXVOr6x4oKPHciGGYJs2jjz7q4of+zDPP4Nlnn8W4ceOQnp6Ovn374ptvvnE7Ly8vD6mp9gX2ZWVlmD59OtLS0nDNNdf41csl4GLoenbory7bg1eu6e/zdQ8UlKJzYjOfz2cYpp5ZMgs4vtW/12zTF5iYpXt4+vTpuP/++3HXXXcBAD7//HP88MMPeOCBB9C8eXOcOnUKgwcPxpQpU3RrN7z99tuIiopCbm4ucnNzkZ6e7rfuB9wIPUTnQ/ry99rVGQ3zMUuGYZimw4ABA3Dy5EkcPXoUW7ZsQVxcHJKSkvDYY48hLS0N48ePx5EjR3DixAnda6xZs8bhf56Wloa0tDS/9S/gRuhGCCF8rmgUExFUHwXDBD8GI+m6ZNq0aVi4cCGOHz+O6dOn45NPPkFBQQE2b94Mq9WKlJQUTdtcJXVVeS3ghqVG2SzT5q3z+nox4XYhLy6r8rlPDMM0HaZPn44FCxZg4cKFmDZtGoqLi9GqVStYrVasXLkShw4Zr2sZOXIkPvnkEwDAtm3bkJub67e+BZygj+vVSvfY5kNnXF7nnSpFeZVx8dZzkg/Mze9vqn3nGIYJevr06YNz586hXbt2SEpKwvXXX4/s7GxkZGTgk08+Qc+ePQ3Pv/POO1FSUoK0tDTMnTsXgwYN8lvfAi7O0DI63HMjADuPncXE138GAORlTa7LLjEM08TYutU5GZuQkIB167SjAyUl9uy5lJQUh21uZGQkFixYUCf9CrgRuln2nNCuKlJeZUNNjbmk9S2Hi3CqpMKf3WIYhqkzAk7QPbny3vHRZhwpcs3r3H38nOPcnk/+gM6PLTb1XlPfXIspkjUAwzBMYyfgBN0TP2w/jmFZK1wmT698+1f8uv8U7p7/m+nrVFTbY+9Hi41nqxmGqV+aSoF4X+4z4GLoZnngsy2ObSEErnvXta510flKxEaF6Z5f25WnDMP4n4iICBQWFiI+Pr7OUv8aA0IIFBYWIiIiwqvzglbQlcjujEq2HinGiG7ahapragTeWrW/rrvFMIyXJCcnIz8/HwUFdVLhslERERGB5ORkr84JOEGXv4X8cP8IZL72s8/XsYToP93nLt2NeatZ0BmmsWG1WtGpU6eG7kajJWBj6MlxUbU63yJ9XdMSdhZzhmECkYATdAH/TIgs2XYcZZU22KQUxgtS4vxyXYZhmIYi4ARdprbTIf/9NQ+7FbnqVTb9B0VJRTVqagSOFfvP5pJhGMbfBJyg+zNjyaKYJa+y6We1nCmtxD9W7MOQF1dgr86CJYZhmIYm4ARdRpmxdMeoLj5do7DUuQq0Whqha+V+Wi0heHX5HgDAhFfX+PReDMMwdU3ACfrU/u2w8I4hCA+1OPY9OKE77h7jvagrDbmqauwj9Aof8s//9EE2lu/Q9z9mGIapDwJO0Nu0iEBGSkuX7BSrhWAJqd2tyCGXc+XVbscGv/iT4bnLd57Anz70T5FXhmEYXwm4PHQtiAihBnnlZqi2CUx8/WfsPHbWq/OayjJkhmEaPwE3QtfDaKGQGapswmsxB4C/L91tuu3K3SexTArNVFTbcOIs+8QwDOM/AnqE/vzlqSiRQiTVOmmH7WIj3dwXtTDKcjHCG4uAW6SY/cEXJ6HHEz8AAN65YSAu6tPGp/dmGIZREtCCfv2FHR3bZTqViSpNCnV9lqBT5ryv3lPgtaBvP1qMxJhwtIrxzriHYZjgxmPIhYgiiGgjEW0hou1E9KxGm1eJKEf62UNERXXTXX3OlWsLsq8jb3+w8eBpfJ971G1/dY2zT3qFOIyY/MYvGPfy6lr1jWGY4MNMDL0CwFghRD8A/QFkEtFgZQMhxANCiP5CiP4A/gHgS/931ZiwUN+nA67JaO+2b96Mgfjl0TGa7U+XVpq67tX/Wod75v+OgnOuVY9siopJeYXnTffTViMc52pl4+ghhMCKXSd4ApdhghyPKijslEgvrdKPkTJcC+BTP/TNK6wW+61c3Kc1bh6a4tgfqpHOGGZx3ZccF+nWJjLMomsAlv7cMny68Q/TfbvxvY0ur5XxfrXYGzFy7kqkPr3UZd/UN9di+N9WALA/aLTK63226TBu/W82/pedb/q9GIYJPEwNa4nIQkQ5AE4CWCaE2KDTriOATgBW6ByfSUTZRJTtbz9jWcgGdozDQxd1d+yff/uFbm3DVaP5tftPubWxWoyzZmZ/udX0SF2dPVOtEt2zOuEimfwz55EyaxGOFJW5zBVUVNuw5XAR8s+U4cTZcqQ/twz/XLnP7fyj0qTwMa6+xDBBjSlBF0LYpHBKMoBBRJSq03Q6gIVCCM0ZSiHEO0KIDCFERmKidnEJX7FJ4YQQIpdKJt1bx7i1VRc6WX/gtFub/QWlHt9TL26vxcTXnd7tNpWgj5q70vDcL387orlfzpQBgOOSWL+ybI/pPjEME1x4FXgWQhQBWAUgU6fJdDRAuAVwmnYRkaETY1yUFSEmctbN2OmqhdkI5ShdOSkKAGfOV+GkQU56hNXzr8moGpfcy0qbdiYQwzDBgZksl0QiipW2IwGMB7BLo10PAHEA1vm7k2aQi0JbSF/cvrprKJbeP9KU9W7HltEAgIcV4Rv393R9fcZkCGb439xH5INecLcXWLL1GFJmLcKeEyVux9QYPVx+/8OedPRNjnvGDcMwwYOZEXoSgJVElAtgE+wx9O+JaA4RTVG0uxbAAtFAqRSuI3RtyR7QIQ6tmke4FZf986jObm3lrJkJvfVzxNUj7Yznl6OmRuDxr7Zix1HvV52qeelH+yrUhZs9T2Yu3nrMsV1R7ToS79e+BQAgLblFrfvEMEzjxePCIiFELoABGvufUr1+xn/d8h0iZ1Wj6DCLZht1xKW1YoHO9Ava48Ur+jpE32jis1LlzGirEThaXIZPNvyBTzaYz4JRsmzHCazddwrtYiNNxfFl3v35oGN7z/ES9FWIt+xMGWHV/jwYhgkOAnqlqJKkWLsox0eHI9JqwZ2ju2BKv7YAgPG9Wrv4pqhH6DcNTcGc73c42iqPG8Wvz1e6x6S1winecLsfXBuPFJW5CHqU9GBrEWmt9bUZhmm8BI2gzxzRGZ3io5GZ2gZEhEczezqO/fumDJe26oCMJYTwwPjueHX5HiTGhLsc6xhvj6XHRIS6LeYprTC/uMcM/opWhaseQiHSA0orR51hmOAhaNwWQy0hmNg3yW30rUWIoo0cfvnL2K747p7h6Nc+1qWtLLKhIYQv7hzicuzdnw/Usteu6PnONI/w7rl7i6JwB+C8R5ZzhglugmaE7g2yns+bkY60ZLuAh4SQS5hCJjrc/hHdNDQFrZu7mmFp5a/XhmFZmuuxcNaLZf5ayGmaW/KL3Y7ZagRCyD0MxTBM4BE0I3RvkEfofdq2QNtY92X/SiKsFuRlTcb947sjOS4Kc6b2qbN+nSoxl/boLbJUbzns6plWWV2DLo8txp8/2lwn78swTP3SJAVd9m6xWry//RuHpGimOepxdUay1+9RX8jpjT9yPVSGCQqapKC/PWMg/nndALRp4Zuf+CzFhCsATEzVz1VvHtHwmSV6i46Os7cLwwQVTVLQW0aH4ZK0tj6fT0SOVED5enoM75bg8/v4C51iTpj65lrH9qFC8znvDMM0TpqkoPsDi2ISsW87/RWYI7v514TMF2w12tkzyjz6M+frr2ITwzB1Awu6j5xT5KB3adVMt53SCKyVKsd9VPe6EfskVShJmQ2ZnXcauflFbjnvi6TKSuVVNvz5o2wcPGUfsd+/4HekzFpUJ/1kGMa/sKD7AXXBDD3aqQppfHDrIBx8cZJf+zKlX1s3v3flCH3avHWY8s+1uHdBjksb2Tpg48HTWLr9BJ76ZhsA4Oscp9AzDNO4YUH3A2GhIXjzunTHa72Y+u0jOuNvV/YFYC9xB3if/92+ZSQmpyUZ9kXtMaMuqAEA323Rdl6UM38qVNfwpuQdwzANQ5NcWORvwkJDMDktCXfPt7+WwxlqG4FxvVohPNSCay7o4PN7VVTVOPLoe7aJwa7jrkWmw0JD3FacerPkPyzUfu0qW42j0hFgtzlQ3w/DMI0LHqH7ATnksnbWWAD2NMHlD47E8gdHubSzatQ39ZYbBneEXB1v5sjOWP7gSJfj4aEhqKhyCnpxWRXeWOFelk4P+RvD738UYahi5WqJj74158qrcPcnv+HkOU6RZJi6hgXdD8je6XLsuryqBl1bxbi5G+pVSrp/fDeP7/HcZanY9Vwm7hnbFRbpwWCrES5FsP88sjPCQkNQoRih553yLh3x9eV7NfefOa+9ivVceRV+2qm/MOnr349g0dZjePh/uV71g2EY72FB9wPyCF0Wdj2TLT0yDRYmyVw3qAMirBYQEeQ5WFuNgFUxATp7Ui+EW+wx9LEvr4IQwrA0nRohBFbv0S7erecLf927G3DbB9k4XlwOIQSqVPcuP8TW6FyXYRj/wYLuB2RRNZvtoqZnm+ZY+fBowzbKwb1jhC4ErKpRv/xQOVBQio/XH/Kq7JxRGbtCHZ+ZrUfshl97T57DhFfXoNvjS1yOt4yyTxBn9vH80GIYpnbwpKgfkEMtEVYLnrqkN8b0bOX1NaLDjasJKbNhlCN0dRhHrk4EAE9+s92rPmhlw8johVxkrJYQ7Dtpr31abatBqNTJGkdpQK+6wjCMD7Cg+8hfL0tFta0G1w/u6GLydevwTj5d72yZ+UnHUEUMPaGZa+ZJWKjvX7pqDApsFHoogK3Mfd+YdxpDu9gtD+RygCzoDFP3cMjFR2YM7oibh3Uy5djYzoNFL2DPL1ejJ85y2qJWiGTjQe882nslNccVA9oB0PdjB4DPNh12s99Vopyczc4749iuURTvZhimbmFBrweW3D8C62aPNWyjDJXIqBcIycj2v/HN3BcweZsrPqF3a6RKXjRGfi62GuFi5qVGmZGpfNDIOfAhLOgMU+dwyKUeaB5h9clG9+qMZHyene+2/6ahKWgbG4GLpYnGnx5y5rsP6RKP//6aZ/o9moVbYNFJp/QGpf+XHLo5V16Fw6fPA3Cv48owjP/hEXojYt6MgVjx0Ci8dFU/AMA9Y7Tz0y0hhMxUZ/3ULonN0CXRbhDmbaZNlc19YtWIN1dqL1KqUii6PELv+8yPeHnZHgDAseIyh7gzDFM3sKA3IjJT26BzYjNMG5iMvKzJbguTzOBtZCMuKgyhXgj635fudmyfOOtc/XnFW78anrcp7wxGzF3pXecYhvEKFvRGjC9OAXqTjx1aRmnun5jaBqWqZf2tm4cjqUWE5kStzNnyKlz4wk+ax5pHWrFVoyC1Wf4oPI+UWYsw6wvn6tKi85U44yHThmGaOizojRhfYtt6Z1wmZbIouX98N8RFhyFHlb1SI4B1s8fh8Okyt3MA4EhRGT5Ym6fbh6wlu3DpP3/RPX7pP35ByqxFuqZh8rkLNh127Os/ZxkGPLcMALBk6zGkzFqEYoNJ3CNFZThQUKJ7nGGCERb0RowvmSF6D4G7x3Rx2ze+V2sA7l7n6gIZaoZlrXDExn1BXl2qZ5FQXGZcPWnemgMAgP2n9AV7WNYKjH15tY89ZJjAhAW9EePTCF3jlIV3DNFMiwyVbBuX7zzpsv/uMV29fl9f+M8vB306R86HN1gHxTBNEhb0RozFhxG61qg+Olw7O1WeDFU/NyZII/exPlgYeEJp3rX9qHucfcUufedGAHju+x2KV6zoDKOEBb0R4006oUzPNjFu+2KjjLNlhnVN0HxftXOiP1CGUxZvPe52/Nb/Zpu+Fo/QGcYVj4JORBFEtJGIthDRdiJ6Vqfd1US0Q2oz3/9dZcwQGxWGJfeNcNmX1EI7W6Wy2q6ISj/256b2cWyry9D5A3W8vjYPDT09L6t0voc31ZoYJtAxs1K0AsBYIUQJEVkB/EJES4QQ6+UGRNQNwGwAw4QQZ4jI/9/VmzAPjO/uVftIq7Zz4/s3X4DOidEY9fdVAIDOidEAgE4JzRxtLuwc79hWCqO/UF+z2iag012P6In1E19vc2zbhEAIr1NlmggeBV3YC2TK6QRW6Uf9P+l2AG8KIc5I55wE4xfysiZ7fY6eqZfa1jdCUlJ5clTNXy9LxQuLd2KDl4ZfRry4ZJfL6+qaGgC+KbpNJ+by5e9Ou4TaPDAYJtAwFUMnIgsR5QA4CWCZEGKDqkl3AN2JaC0RrSeiTJ3rzCSibCLKLijgCjZ1RbgHC92VD4/GN3cPc7xWrhRVamS/9rH47M9D/Nq3Fbtcn/U3vrcRBecqdNsfK9bOhQdc/WOUTO3X1rH9zLfbsb+gBGfLjVMhGSYYMCXoQgibEKI/gGQAg4goVdUkFEA3AKMBXAvg30QUq3Gdd4QQGUKIjMTExNr1nNElQjEkndq/rdvxTgnR6Nfe+evxtdKSmu//Mtzrc37/owgXPL9c9/ju4+d0jylH6KdKKlAkFeFQrpb9LPswxr28GhNf+9nrvjFMoOGV26IQooiIVgHIBLBNcSgfwHohRBWAg0S0G3aB3+SvjjLmiQ4PxcqHR8NqIVNe7KEKQe/eupluu0cu7uHi5aLG0zcDT+w6ftZtn9HErE0xRM/4q/2hkJc1WTNifqRIf6TPMMGCmSyXRHm0TUSRAMYD2KVq9jWAMVKbBNhDMAf821XGGzolRCM5LsrrwhJa7bu1aoap/du6jfb7tG3u8ro21ZIA4Mmvt7ntq6yu0Z38PFBQqn2hWs6Bfrguz6O1AMM0Rsz8D0wCsJKIcmEfcS8TQnxPRHOIaIrUZimAQiLaAWAlgEeEEIV102Wmvln24Ci8Pn2A2/5F97qmR9ZG0FfvKdBcGZscF4nzVdrZNn9dtNPNWAwAyAdF/7+FWzBv9X4AwFNSLdZjZ3lUzwQWZrJccgG4/W8WQjyl2BYAHpR+mABkztQ+6JygH24BtEfvvz05AemSaVaEhr2AWW56b6NmtaXLPdjyfvX7EcwY3FHVT+/eu/h8laOQyB2jnJ43epOuDNNY4ZWiDADgxiEpGN4twbBNhMYIvGW0swxeZFjt8gONsl302HfS1aBr25Fi3fH5w//bgmqNhUzXvLPOsa0sn3fmPNv1MoEFCzpjmvhmxvVKIwwSvr3Np79MIztHi//+mgehyHbJP1OmO0JfuDkfe0+6OzTuUmTSvLbc6SLZysv6rL5yvLhc09eGYbyFBZ3xCi1hHtDBLUO11hw4pTPhqYHShveZb7cberzYPFgBrNnjXB9h9IDyJ0OzfsLkN/T94xnGLFwkmqk1X9451PC4VWclqhH9kmORa7LqUaUitfH42XJUG4j2/oISpLZroXtcOU9QUw/uX6dLK8F2M4y/4BE6U2uIyDA9cs3/jfHqeu1bRuLGIR09N5Sosrkq4le/H9Ft2zzC6Ty5ek8B1h9wTcZSVm/yNJr3B8oQz/lKZ8bOibPlGPvyKi6szXgFCzrjNXOm9sG8GQNNtc3oGOdwe5w7Lc3UOT//31i016mBqoU3wtsswvml9Kb3NmL6O+t128rXTZm1yJHSCABbDhdhUe4xAEBJRTVy84s0zzeD0r++37M/OrYXbs7HgYJSzN/4h8/XZpoeLOiM19w4JAWZqW0M2wyXPNaVxl9pyc5Qx7wZA/HtPcPczpPxZtVpjRCI0SniocabKEp1jXBkxWQt2YXTpZU4X1mNqW+uxd3zfwMA3PnxZkz551o3W2BzfREOuwLA9ZvGTzvthT58KXLCNF1Y0Jk6QU5hbKvwYleORsf0TERasvtk6nUXdgCgnfMu8+VdQ/HIxT0cr3VXjGogC/TsL7d6bGurES7WA+nPLcMz3253vC4+X4XfDp0B4F4ftaLahpRZi/D68r261//3zwfxdc5RzWO//WEf9ftQ44RpwrCgM35lxmC7IKd3iMPcaWmYc5nTx00pTnKN0/7tXUV9sMKPXY/0DnG4Mj3Z8frad9frriZVI0+YfmoilGGrES4TroA9FCLTb86PKJX83dX2BKdK7CPvV5frF9NetUfbZVrpDMnzpYw3sKAzfuWi3vZQzKBOcbg6oz2aKUIhWqPue8e5FqQ2OyJVZ85oxdGbR7iHYaq9WP5ZXSNcSuYB+oW71Zk1Z0qNFyVVVtcg5w/t2HthifNcb714mKYNCzrjV0Z2T8Su5zIxsGNLt2NaBaw3Hjzj0/voCauS3GcudttXbTM/5q0RAqNfWuWyL0FncdV177pOrp4qMV71+tdFOxyjezXKW+MYOuMNLOiM39FbkKMlTsqFPGbolWR3eIw2OQmqxihHXc3Js+6irC6oLbPnhOsK1KcVsXYtth3RzrHPzS/yyVyMYQAWdKYe0Rps3jIsxatryH7tVh+LcnhTlFrOZFHSwSCdUmlBoFyUtPmQewm/33TCLSXl1dimsAEQHEVnvIAFnak3QqRYgjJcok5/VKYVTk5LcmzPHNkZr1zdD3OmOidZR3gwE9OitouFjM5Xph0ePu203jUqCqKGiBAX5TQ88/XBxTRN+K+FqTfkkItyoK72UFfKZeuYCADAwI5xeGxSL1yRnowWkc6Vnmbi6Gq+zjnqMb5txNLtx3WPHdTxn1GuPvUEkeuEr1Haox5llTZs9GNhbyZwYEFn6g1Zf5VxbGuI/p+gNdR+wvherTWPe2MPILNmT4GjXJ0v7DKocXrxa2s0LYB7tGnutu/aQe01r/Hz3gKXz6fSVoNVu7XTG5XMW70fK6V2s77MxdX/Wsdl95ogLOhMvaHllx4SQsjLmowVD43CiG4JGNXNWTx8TI9WAIChXbRz01s3j/Bb35Ja+OdayhxyGa3vEbGKsIqSN1fud8tpf2RhrsvrW97fiI/XHwJg94N/bfkeZC3ZhVvet5fwlf1pSsrt3jDr9he6LIhighcWdKbeiJGMsbQEunNiM3x024VoEeUMqQzuHI+8rMno17729rwd4429YZ66pHet3wPQTs1UuzaWV9nw9qr9LvtevqqfY1udiaOO26/cXYAnpPqr1767Hq+pwjInpOwcOSR143sb8N9f83BO42HjDUrzMKZxwoLO1CvZT4zHezdf4JdreZPe98mfLjQ8rpx8fGB8d5/7JDTMYo6qQh9Xvm1cVk8t4OrFTTJVthpDDxk5Fi8L+3mdvHczfLQuD72fWsruj40cFq14EYwAABwhSURBVHSmXkloFu63whHxzbTDFnrva0SohTA5LQlEwH3ju6F3knvc2wxqK1/AaQMgs/3oWcNrLJOMuWRsNQLvrjng1m6ahweDGm9y8NU8KRXO/mGb/qQw0/CwoDMBS+vmEZh/+4WYqOH8KOe3f3zbhbiwU0uP7o1hlhC8eV06Dr5or8hk9dB+zSPaHu8Xv7bGLQYOAF/+lm8ohspIzfwN7j4zzy/eidOllUiZtcixb0t+sZvZi2suvLzP/q9WPVVvqeLK2Y0arljEBDRDuyRg1W77atMIawjKq+yCI5t8De+W4LH4NeAu4FYPKZFtY/UnUbVE78HPtwAAdsxxtyMAYKpq0cLNh932qU97/SdnPF0Wd7lNbUboMp6+6TANC4/QmYBHHhHfPdpp9GUmuv713U4/9lCVgBst6Ll3XDfNyU8ZI7+Y5xftdNuXlzXZVGz6hcW73PaVVLhOVConSNX67Y8KTC11snOYxgELOhPwXNDJbgQ2qFNLjOtpT3U041IYqYjlqwU81KAOakp8lGPVqxZLDEIrp0srNYtqT9fJS68NjvCL9I83tgd6+Oqhw9QPLOhMwHNxnzb4/ckJuLBzvCO8YGaErkwn3HnMdaIyTCXwyhi8pxWqD/9vi+6xs+VVjhWwSpIUhUD8hSOGLn0qthqB99ceRMqsRaio9i3jRWstAdN4YEFngoK4aHsoYNpAe+GLXm09Z6koQxDtYl0FVTlC3/PXibgqw1lQwxfLAZm1+wqRV+i0COjROsbna3ni4tfWAHBm3vy6vxBvSDH2c+Wec8rPlldhU95plCnSHeujcDbjOyzoTFAxqW8S8rImuwm0Fkpx6qVKU1SGYMJCQxCqsChQ579fO6iD5vUv7OTuCQ847QPmTkvD0gdGeuynv1i775Tjns08kjJfXYOr5q1DYanTzkArz55pPLCgM00WZR67OstFPUmqnAQ9Xerq19JGx4IgRqNikky/9rG4OsP/cXMjEpqF46w0MtdbrKTkaHE5ADgyhwD3EXppRTV+NDAsY+oXFnSmyZIc57QDUAv4WVVIQhmCsagMxXLztd0UbTUCgztrj9LV7+crmX3cc/CVaHnLAK61UbVQ5tIr/dzVEZfHvtqKmR9txp4T+qZlTP3hUdCJKIKINhLRFiLaTkTParS5mYgKiChH+vlT3XSXYeoGdZbLil1258JJfe2CqRyhq4tOFOrUD+3XPhYLZg7RPFabOLySHzyMjt9QpDEOUXjoGKVdAq45649+sdWxrfalOVBgnw+oja0A4z/MjNArAIwVQvQD0B9AJhEN1mj3mRCiv/Tzb7/2kmH8QOvm+oti9AQ2ItQiHXfuU68E1dPmP43orP9+GoL6y6PO1adyZSYAiKpFZkmlTTtcovahV6M3+akWdFn4/fGNo9pWg9IKNgCrDR4FXdiRCyZapR+eGWECjuYRVs+NVIQ6DK6c/1XUWqeeUHWcayByWnnuyhBQVJgz/q4s6uEtSv2tttWguRTXf2XZHsxbvR97T5zTnOhUrnaVc/sBd6GX7QSM8vbN0vXxJejz9NKgmXgtraj2S+6/N5iKoRORhYhyAJwEsEwIsUGj2ZVElEtEC4lIc7aHiGYSUTYRZRcUeFccmGFqy3SdbBQjlmy1hzSUI2q1qClL5SmRBV1rubynkIvVQo4smeE6hanN8Hm20y7gX2sOuIzEspbswoRX1+DfPx90O8+mWO2q9G6XR+iy6MojdE8WL2rHSSM2BEm1pT5PL8Wg530vpuILpgRdCGETQvQHkAxgEBGlqpp8ByBFCJEGYDmAD3Su844QIkMIkZGYmKjVhGHqjFuHpeDgi5Pw4ITuuCAlztQ556QQgHIE2q99C5c2WjYBNw9NQai0/6eHRmGsYpQLAEfOGAtclU04snBsJkasH9+mbQ9cUe1U2vwzZaisdlfebI0i1soRelmVMwxSUwMcKixFp9mL8X3uUccI1Cg//X/ZhzE0a4VmsexDhaVYtsPVXbKwRHtOIhA5c752HvTe4lWWixCiCMAqAJmq/YVCCDmX610AA/3SO4bxI0QEIsK947rhf3cMNWzbOTHa5bVyEnFgR9fMFa2xdn9FUY4WkVb849oBLsf3nixRn+JCzuEiPH9ZX/xlbFfNEXq0Iq6eEh9lyoAMcBV4I5Q2wMoJT5sQyM0vBmD/9iL71lQbDNHXH7AL+f4C95qr415ejds/zHYJs5RX2VBaUe3VqJ6xYybLJZGIYqXtSADjAexStVF+55wCwN2BiGEaIekaviqAe7xdndmiRMs3pktiM5fX0eGhePv6dK/6Fhcdhocu6uEQUCWdFA+cvEJ3Y6+Mjua+gegxcu5Kx/Y+xcNHCOEYjVtCyCHk+06WIGXWIs36p1/8Zk+R1LIwlkM2Hyssg8uqbLj6X+swNGtFre6hIVE+oI4V19+DycwIPQnASiLKBbAJ9hj690Q0h4imSG3ulVIatwC4F8DNddNdhvEv828fjM1PjHfbn9rOPtE5a2JPAK4xZTXqBUSPTeqJvskt3NoZGXoZ0UojO+eKAckaLZ1kHzpj+vpa83bKEEq+Ijxkq7ELLgBsPVLsGMlvlt5vzvc7HG3vW/A7nv1uu+Jc/c/wSamkHgAkxoR7LAICALuPn8NDn2/BFx5y6tXcPf83XP7WWgD2bwPX/GsdTuuknvqK8l4PaHwzqSs8WqcJIXIBDNDY/5RiezaA2f7tGsPUPRFWi2YFJXmAJYc2jLzE1WmFesWrPeV+A8DAjnHYfOgMtj/r9E2/c1QXtI6JwGUD2mH8K6tx8FQpRvVIRM/sGIeNQG1QpyIaYRMCs7+056UfPFXq+HxkQVeK1zc5R13OfXPlPlyR7nwQTXr9Z833iDHp6Ch71XzxWz6uHGj8gFOyKPeYS582HDyN9OeWIS9rsulreGKL4ltVfSbtsBcmw2jg+D8oibDW6LJ183BHQWZ/8cWd7rF9InIIlvzwiQqz6ArFrucy0fPJH0y/Z5kXi4LUKYWl0rme5gQAeyz+hv9swJR+bfHyj3tw/Gy5Zjtl7nzR+UpYLSEO215bjUBldY2b62NpRbVP1r51IbbVthqXurGxUb6nnXoLL/1nGA1GSBORae3soRN5hD6+lzNb5au7huH16f3dRt56I3F/5Fe/c8NAPD6pF5JaRDpG1k9f2tulTYTVgpuHpjhee8pjL62sRt6pUkfB6XM6dgFA7dwWjxWX4+e9p/DIwlxdMQdcJ277z1mG0S+tcrx+8PMc9HrK/WHV5+mlPvXp043u5f5qi3pOg0foDNPATOybhK3PXIQYaXJUFuN0xWRj29hITO3fDoUlrqN0vciKP/5ft28ZhdtHdna5npwe+fikXsg/4z5B6il0v/dECUa/tAptW0Tg19njcMJAbM3quZkKTHp8vP6Qy+uCc87PVx3GMcNXv+cjhAhT+7dzO6Zn21Abrv/3epfXRhlA/oZH6AyjQ4zJlaXxzcLx3T3DMb5XawD6I3St4tG14ZiU1ic/UG4f2RnPTlUvEfGcCy1Pch4tLkfKrEUY/8oa3bZm72GEIkvGWyamai/U8qUfAPDAZ1tw34Icj+1SZi3CW6v2eWy3Ke+07vunzFrkFoarTw95FnSGMYGcmqj39blvcguH34vegNjf/6/l+LWyjqiMeqLW25RJPWqEcDy46gozNgL3feZZoH1h7g+7DY9/sTkfV81bhwmvrjZ9TRZ0hmlkyGELozi4fEgv5OJNNkltuWdsV5fXZtIAzWATwqM/ye5aZt6YGX1/t8Vc6MWTmF7ar62p68j8vNduWaK1SEqPDQdP44dtxzw39AMs6AxjgssG2OOvE/vqhwNuHpYCwDXOrqSuBF1rcZTS3AsALunnOYxhhse/2oYiD8Ux5HRCXzFjdWCW99e6+9TInDhbjh6Sq2U/jXUDWih9bczyyrI9uOPj37w+zxdY0BnGBN1bxyAva7LbClAlQ7skIC9rMlppFIEGXAVdmS1TWwZ1ivfYRun8OKVfW6x+ZDQm9PYcOln+4Ci3fVsOaxf08BfKCknesP5AoctkbGV1DX77Q3+B1YUv/ORYGGW0zkBJpwT7Ct1R3X3zovK1OLdZWNAZpp6QR81XpLfDW9f7z+5IWbhCiycm94Iysj97Uk90jI/Guzdm6NY9lemi8rSpD55TrDb1hunvrHeZjE19eikWb3UWALl63jq30ntyRo+WaZkWT39rX/m64WAhTpdW4owiS8ZTWuonGw6hxxM/OCyH6wJOW2SYeuKi3q3xwuV9cUV6O48FJrzB5iEtbkS3RN3URU/90PKpMSIqzNJoqhdVqoRzY95pRwxcZsEmu73wXpVfjfq+D58+j69+P+J4ffmAdkh/bhkAOFaYehrlP/vdDke/QjUcOv0BCzrD1BNEhOsu9N6T3RMXpBiPsnu0iUHeKecknnIgqWX9WxvMhi4aCtLNQXJiqxEItRB+3luA85U2/GX+724PhzCNz63awO8HcH4LqMusFw65MEyAEx3meVzWMT4K3VrZ4//Kghty7VR/YTZ0ocXDF3X3Wz/W7NEuoPPyMuO0RMCeXvrTzhO44T8b8eePNruJOQD8ur/Qsf3Kj/Zrml1A9Pelu3ULi9cWFnSGCXDMuDgSEZY9OAp5WZP9Gu4B7MWw/cE9Y7t5bKNlwavFje9t1NxvxvmwrMqG2z7INmyjDNG8scK+GMnTCF3mw3WHsPOYf9JI1bCgM0yA4il8897NGbh7TJe674gPaYbNvDTS2l9QgmpbDfolGz885CyU2vDCIu/LORw8VYoBUkzdDP4OdcmwoDNMgPLC5X0NLV/H9myNRy7u6dO1J/RujUX3DjfV9qI+bdAi0opLdGqraiGX1zPLuJdX46Uf97iUxtPi4KlSrNQosuENnynqsJpljMJATOYig7TQupoUZUFnmCaMVrRm2sBkvHRVP/Rpa19s8909w/HTQ+756DLtYiOx5emL0NmL0bEvi6w25Z02Fda45f1NXl+7LnjnxgzdY2Em7A18gQWdYZowWgkXL13Vz8Vyt29yC8MFVRbpqeDNCk9fFoNuPnRG03bg2kH+zRzyMlPTJ44W6Tta1gYWdIZpwnRtpS/UZglxFAExf46vq/vVpeLysiZjU95p3y6mQ31Y7mhlzvgDFnSGacJo5VProTcSlkfm6pGtUfKNrx7hJ89VYIrKUKtXUnOfrlWXtNaoA6uEJ0UZhvE76hTGbgYj9hev6OvyurckpOol75FSmbwl943UvE5iTLjbaD5DMjSLDnOv76omVPWkuEUyRdMjPtp7Q63a8MujY/DrrHGGbeoqqsOCzjBNmH9c61r/3Wz2yb1ju2KHlEutDnncM7Yrds7JREsdIV376FiXEfqIbglYKNVS/eDWQR7f+7tcV+tcT98y1CZk3nwr8YWW0WGOeQU9OrSMqpP3ZkFnmCZMe5WwhIaYk4SBCruBYxoTfJFhFpeRtDwCj4kIRVhoiG62SoYHGwMADodEGU/hC/XCqw9v8/zQMMvyB92/hZgJp4zq4ZtboydY0BmGcXBVRrKpds3CnaERWV7VY1KLIjVPdoTM7NMGgLEp2N+npQEABnaMw+J7R3jsi6cKR+rBsreLmvQIs4Sga6sY9/6YWLlrqaNUGhZ0hmEcGKUnKokOD8WIbgkAgMGd7aNqeXJUznpRipYc5pDbfDZzsON8NfIkZ5sWEbophB/eOghf3mUP01g9fKtIiXfNjw8PDTFckGUWvUwVMw6VZuwafIEFnWGaON/d41wRanbg2Cw8FG9MH4BpA5MxY3BH+z7JJCwuyp7DrowjW6URuTx/2q11DJ6+tA8AYHhXV2FPbdcCr1zdD1lX9NVNIRzZPRHpHeKkaxt3+uoL2uPj2y50vPbX4HhcT/8VKfEXbJ/LME2cvoryayfPVaCPiXOiwkIRFx2Gl67q59g3c1RnxESE4qqM9gBcQw/yttI6tmurZvh11lgktXCv8HRFuj30s+5AvtuxJy/p7fLaU9w/NIQwXPFtQOk2KTNvRjoqqmtw3wLPxafXPDIGryzbjdmTenlsW9+woDMMg9eu6Y/7P8vBkM6ey9kB2q6H4aEW3Dysk+O1coQeLcWt1atJ28ZGGr6PVkWlfQqnQwCw6sTQ505LQ2KzcEelqPWzx+GP0+fd6oLOntgTmalJbsUv9GjfMhKvTR/guSGAn/9vDGqEwH0LcpBTx6X7ABZ0hmFgL4ItF8I2Q4TVc764MpbcPMIehqnxsrhD99buk47lVa4VkfSMrlrFhGN0D2dYpE2LCLTR+DaQInnQmC08YRQj3/f8RJfXchZRm+badWb9DcfQGYbxGk951jJ5WZORlzUZzSPtY0e93HQ9tN5GbewVZbUgPjoM16vshD2FYiKs9uMDpZTKAe3jvOqbFnqfy8xRnWt9bTOwoDMMU+cM75qAv13ZF49P9i7uHKIxGlZXRQoJIWx+coJjclbGUzqjvKJVbqVMs3z3xgxc7sU3Fhm90bs8gVvXeBR0Ioogoo1EtIWIthPRswZtpxGRICJ930iGYQKWaQOTcddo74tmEBGuuaCDI55t/jz3fe104u7q0fH+ghLNdjKO/HnpTZSTuF0So03lkwPACgNrYS3mzUj3qr03mPl0KwCMFUKUEJEVwC9EtEQIsV7ZiIhiANwLYEMd9JNhmEaAMqulPlCPeG8d1gmzJmoX7VDLb6sY47j136f1w8s/7kbzCLsMKr8NWELIdHpjZ5O5+zKZqeYLgXiLR0EXducd+VFnlX60Zg+eAzAXwMN+6x3DMIyCpy7trXtMLcC92xq7ME7o3drF50WZLRNChDiNeP+lKqfHxoap7z9EZAGwGUBXAG8KITaojg8A0F4I8T0R6Qo6Ec0EMBMAOnTwryk9wzBNG/Vo3mzIRO/8O0Z2wb9WHwAAPDShO3q0icFFknWBL7x/ywVI1MiB9yemJkWFEDYhRH8AyQAGEVGqfIyIQgC8CuAhE9d5RwiRIYTISEysG3MahmGCC7PL9NvFRjrCJ0DtLGrDrSGIjXJWbfrLuG61EnMAGNOjFVLbtfDcsBZ4leUihCgCsApApmJ3DIBUAKuIKA/AYADf8sQowzD+4uI+rXUXEMlEWC3IfeZi/Gm4fXGTVsjELCFEpjxZZDL7tNGdrK1PPIZciCgRQJUQooiIIgGMB/A3+bgQohhAgqL9KgAPCyGy/d9dhmGaIv+6wfz48IlLeuOJS/Rj7WbwdnQ/74aBtXo/f2Emhp4E4AMpjh4C4HMpVj4HQLYQ4ts67SHDMEwDoecI2Vgxk+WSC8DNuEAI8ZRO+9G17xbDMEz9M7xrAn7Zd8rhDvmRwqUxEGAvF4ZhGIm3ZqRjx9GzDu+ZQIOX/jMMw0g0j7BisEnHycYICzrDMEyQwILOMAwTJLCgMwzDBAks6AzDMEECCzrDMEyQwILOMAwTJLCgMwzDBAks6AzDMEECCzrDMEyQwILOMAwTJLCgMwzDBAks6AzDMEECCzrDMEyQwILOMAwTJLCgMwzDBAks6AzDMEECCzrDMEyQwILOMAwTJLCgMwzDBAks6AzDMEECCzrDMEyQwILOMAwTJLCgMwzDBAks6AzDMEECCzrDMEyQwILOMAwTJLCgMwzDBAkeBZ2IIohoIxFtIaLtRPSsRps7iGgrEeUQ0S9E1LtuusswDMPoYWaEXgFgrBCiH4D+ADKJaLCqzXwhRF8hRH8AcwG84ud+MgzDMB4I9dRACCEAlEgvrdKPULU5q3gZrT7OMAzD1D0eBR0AiMgCYDOArgDeFEJs0GhzN4AHAYQBGOvPTjIMwzCeMTUpKoSwSeGUZACDiChVo82bQoguAB4F8ITWdYhoJhFlE1F2QUFBbfrNMAzDqPAqy0UIUQRgFYBMg2YLAFymc/47QogMIURGYmKiN2/NMAzDeMBMlksiEcVK25EAxgPYpWrTTfFyMoC9/uwkwzAM4xkzMfQkAB9IcfQQAJ8LIb4nojkAsoUQ3wK4h4jGA6gCcAbATXXWY4ZhGEYTM1kuuQAGaOx/SrF9n5/7xTAMw3gJrxRlGIYJEljQGYZhggQWdIZhmCCBBZ1hGCZIIPvK/gZ4Y6ICAId8PD0BwCk/dqexw/cb3PD9Bjf+vt+OQgjNhTwNJui1gYiyhRAZDd2P+oLvN7jh+w1u6vN+OeTCMAwTJLCgMwzDBAmBKujvNHQH6hm+3+CG7ze4qbf7DcgYOsMwDONOoI7QGYZhGBUs6AzDMEFCwAk6EWUS0W4i2kdEsxq6P75CRHmKwtrZ0r6WRLSMiPZK/8ZJ+4mI3pDuOZeI0hXXuUlqv5eIGpXLJRG9R0QniWibYp/f7pGIBkqf4T7pXKrfO3RF536fIaIj0u85h4gmKY7Nlvq+m4guVuzX/Bsnok5EtEH6HD4jorD6uztXiKg9Ea0kop1S8fj7pP1B+fs1uN/G9fsVQgTMDwALgP0AOsNe6m4LgN4N3S8f7yUPQIJq31wAs6TtWQD+Jm1PArAEAAEYDGCDtL8lgAPSv3HSdlxD35vifkYCSAewrS7uEcBGAEOkc5YAmNgI7/cZAA9rtO0t/f2GA+gk/V1bjP7GAXwOYLq0PQ/AnQ14r0kA0qXtGAB7pHsKyt+vwf02qt9voI3QBwHYJ4Q4IISohL060tQG7pM/mQrgA2n7AzgrP00F8KGwsx5ALBElAbgYwDIhxGkhxBkAy2BcTapeEUKsAXBatdsv9ygday6EWCfs/wM+hE6lrPpC5371mApggRCiQghxEMA+2P++Nf/GpdHpWAALpfOVn129I4Q4JoT4Tdo+B2AngHYI0t+vwf3q0SC/30AT9HYADite58P4Q23MCAA/EtFmIpop7WsthDgG2P+AALSS9uvddyB+Hv66x3bStnp/Y+QeKczwnhyCgPf3Gw+gSAhRrdrf4BBRCuw1EzagCfx+VfcLNKLfb6AJulYMLVDzLocJIdIBTARwNxGNNGird9/B9Hl4e4+Bcu9vA+gCoD+AYwBelvYHxf0SUTMAXwC4Xwhx1qipxr5guN9G9fsNNEHPB9Be8ToZwNEG6kutEEIclf49CeAr2L+KnZC+akL696TUXO++A/Hz8Nc95kvb6v2NCiHECSGETQhRA+Bd2H/PgPf3ewr2MEWoan+DQURW2MXtEyHEl9LuoP39at1vY/v9BpqgbwLQTZoNDgMwHcC3DdwnryGiaCKKkbcBXARgG+z3Is/y3wTgG2n7WwA3SpkCgwEUS19nlwK4iIjipK96F0n7GjN+uUfp2DkiGizFH29UXKvRIIubxOWw/54B+/1OJ6JwIuoEoBvsk4Caf+NSHHklgGnS+crPrt6RPvP/ANgphHhFcSgof79699vofr8NNWvs6w/ss+V7YJ8pfryh++PjPXSGfXZ7C4Dt8n3AHkf7CcBe6d+W0n4C8KZ0z1sBZCiudSvsEy77ANzS0Pemus9PYf8aWgX7yOQ2f94jgAzpP9B+AP+EtPK5kd3vR9L95Er/yZMU7R+X+r4bigwOvb9x6e9mo/Q5/A9AeAPe63DYQwK5AHKkn0nB+vs1uN9G9fvlpf8MwzBBQqCFXBiGYRgdWNAZhmGCBBZ0hmGYIIEFnWEYJkhgQWcYhgkSWNAZhmGCBBZ0hmGYIOH/AS00mrP+a81eAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#hide\n",
    "learn.recorder.plot_loss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save(path_data/'GPT2_pt_2epoch_lr1e-3')\n",
    "learn = learn.load(path_data/'GPT2_pt_2epoch_lr1e-3')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1004M\t/mnt/home/pierre/.fastai/data/ptwiki/GPT2_pt_2epoch_lr1e-3.pth\r\n"
     ]
    }
   ],
   "source": [
    "# model size\n",
    "!du -hs {path_data}/'GPT2_pt_2epoch_lr1e-3.pth'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 6.2.3 Freeze all layers but the last 3 layers groups"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GPT2LMHeadModel (Input shape: ['8 x 1024'])\n",
       "================================================================\n",
       "Layer (type)         Output Shape         Param #    Trainable \n",
       "================================================================\n",
       "Embedding            8 x 1024 x 768       38,597,376 True      \n",
       "________________________________________________________________\n",
       "Embedding            8 x 1024 x 768       786,432    True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  False     \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Linear               8 x 1024 x 50257     38,597,376 True      \n",
       "________________________________________________________________\n",
       "\n",
       "Total params: 163,037,184\n",
       "Total trainable params: 134,685,696\n",
       "Total non-trainable params: 28,351,488\n",
       "\n",
       "Optimizer used: <function Adam at 0x7fb385bf8e60>\n",
       "Loss function: FlattenedLoss of CrossEntropyLoss()\n",
       "\n",
       "Model frozen up to parameter group number 1\n",
       "\n",
       "Callbacks:\n",
       "  - DropOutput\n",
       "  - ModelToHalf\n",
       "  - TrainEvalCallback\n",
       "  - Recorder\n",
       "  - ProgressCallback\n",
       "  - MixedPrecision"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.freeze_to(-3)\n",
    "learn.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `learn.summary ()` method gives almost the right numbers. In fact, it counts twice the weights of the wte matrix (vocab embeddings) because they are duplicated in the weights of the output linear layer.\n",
    "\n",
    "The real numbers are:\n",
    "- Total params: 163,037,184 - 38,597,376 = **124,439,808** (about 124 millions)\n",
    "- Total trainable params: 134,685,696 - 38,597,376 = **96,088,320** (about 96 millions)\n",
    "- Total non-trainable params: **28,351,488** (about 28 millions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.333389</td>\n",
       "      <td>3.207390</td>\n",
       "      <td>0.374579</td>\n",
       "      <td>24.714487</td>\n",
       "      <td>6:20:51</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(1, slice(5e-4/(2.6**4),5e-4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO2deXwV1dnHf89NQha2BAgYCBg22VfDIiIiIKBYtYpKra2vb6t1qdpqrdQVta3WtrZ1X/pqW9ci1ooiWlGQyh4EwiLIFiSyhS0kQEKW8/5xZ+49d+6ZmTN3yU1uni+ffLh35szMmXvn/ubMc56FhBBgGIZhkhdfojvAMAzDxBcWeoZhmCSHhZ5hGCbJYaFnGIZJcljoGYZhkpzURHfASocOHURBQUGiu8EwDNOkWL169UEhRK5qXaMT+oKCAhQVFSW6GwzDME0KItplt45NNwzDMEkOCz3DMEySw0LPMAyT5DQ6Gz3DMIxXampqUFpaiqqqqkR3Je5kZGQgPz8faWlp2tuw0DMM0+QpLS1F69atUVBQACJKdHfihhAChw4dQmlpKbp37669HZtuGIZp8lRVVaF9+/ZJLfIAQERo37695ycXFnqGYZKCZBd5k0jOk4W+CXPgWBX+s3FforvBMEwjh4W+CTPjxeW44dXVqK2rT3RXGKZZc/ToUTz77LOet7vwwgtx9OjROPQoFBb6JsyOg8cT3QWGYWAv9HV1dY7bffjhh8jOzo5XtwKw100SwDXCGCaxzJw5E9u3b8fQoUORlpaGVq1aIS8vD2vXrsWmTZtw6aWXYvfu3aiqqsLtt9+OG264AUAw5UtlZSUuuOACjB07FkuXLkWXLl3w3nvvITMzMyb9Y6FPAuq5HCTDBHjo/Y3YtOdYTPfZv3MbPPidAbbrH3vsMWzYsAFr167FokWLMG3aNGzYsCHgAvnyyy+jXbt2OHnyJEaMGIHLL78c7du3D9nH1q1b8eabb+Kll17ClVdeiXfeeQfXXHNNTPrPQp8EsM4zTONi5MiRIX7uTz75JN59910AwO7du7F169Ywoe/evTuGDh0KADjzzDNRUlISs/6w0DdRyk/WBF4v/roMkweclsDeMEzjwWnk3VC0bNky8HrRokVYsGABli1bhqysLIwfP17pB5+enh54nZKSgpMnT8asPzwZ20Q5Jgn9i4t3JLAnDMO0bt0aFRUVynXl5eXIyclBVlYWNm/ejOXLlzdw73hE32Tx+YJBE2yjZ5jE0r59e5x99tkYOHAgMjMz0alTp8C6qVOn4vnnn8fgwYPRp08fjB49usH7x0LfREmVhL6OdZ5hEs4bb7yhXJ6eno758+cr15l2+A4dOmDDhg2B5b/4xS9i2jc23TRR5Cjo+npWeoZh7GGhb6pI2s6mG4ZhnGChb6LI0l7HI3qGYRxgoW+iyKN4HtAzDOMEC30TRRZ3oZEEYUdZJY4cPxXHHjEM01hhoW+iyCN6HdPNhD9+jp+++WU8u8QwTCOFhb6JEjKi1zTdLNl2KD6dYRjGE61atQIA7NmzB9OnT1e2GT9+PIqKimJyPBb6JIC9bhimadK5c2fMmTMn7sdhoU8w+8qrUDBzHq58fplrWyEENu/zZ+ULmYyNW+8YhtHh7rvvDslHP2vWLDz00EOYOHEihg8fjkGDBuG9994L266kpAQDBw4EAJw8eRIzZszA4MGDcdVVV8U01w1HxiaYNd8cAQCsLDns2vbNlbtxz7vr8eqPRqJrTlZgObtXMozE/JnAvvWx3edpg4ALHrNdPWPGDPzsZz/DzTffDACYPXs2PvroI/z85z9HmzZtcPDgQYwePRoXX3yxbc3X5557DllZWSguLkZxcTGGDx8es+6z0CcYL3V+N+0tBwDsPHgcXbKDBQlKj8Tuzs8wjHeGDRuGAwcOYM+ePSgrK0NOTg7y8vLw85//HIsXL4bP58O3336L/fv347TT1JlmFy9ejNtuuw0AMHjwYAwePDhm/XMVeiLKALAYQLrRfo4Q4kGbttMBvA1ghBCiiIgKAHwFYIvRZLkQ4sYY9DupuXtOMd5b9y02P3JByHJC8K7AY3iGscFh5B1Ppk+fjjlz5mDfvn2YMWMGXn/9dZSVlWH16tVIS0tDQUGBMj2xjN1oP1p0bPTVACYIIYYAGApgKhGFpV8jotYAbgOwwrJquxBiqPHHIq/BP4t2o6omvOD3ln3+NKhC+O31DMM0HmbMmIG33noLc+bMwfTp01FeXo6OHTsiLS0NCxcuxK5duxy3HzduHF5//XUAwIYNG1BcXByzvrkKvfBTabxNM/5UKvMIgMcBON+ymIgx7fhCCI6GZZhGxoABA1BRUYEuXbogLy8P3//+91FUVITCwkK8/vrr6Nu3r+P2N910EyorKzF48GA8/vjjGDlyZMz6pmWjJ6IUAKsB9ALwjBBihWX9MABdhRAfEJE1v2Z3IloD4BiA+4QQ/41Bv5sd3x4N2uEFgFjPv9bW1WPt7qMoLGhn22bP0ZNIT/Whfat02zYM05xZvz44CdyhQwcsW6b2pqus9I+dCwoKAumJMzMz8dZbb8WlX1rulUKIOiHEUAD5AEYS0UBzHRH5APwJwJ2KTfcC6CaEGAbgDgBvEFEbayMiuoGIioioqKysLJLzSHoOVwbTFwgRnvbgteXOj4Vu/HnBVkx/flnAC0jFmMc+w5m/XhDVcRiGaXg8+dELIY4CWARgqrS4NYCBABYRUQmA0QDmElGhEKJaCHHI2HY1gO0AzlDs90UhRKEQojA3NzeiE2lO1CtMN/f9e4O6sSabDfv/gYrqqPbDMEzjw1XoiSiXiLKN15kAJgHYbK4XQpQLIToIIQqEEAUAlgO42PC6yTXMPiCiHgB6A+ACpxI1EZSHOnmqzlM0LE/cMs2B5nKdR3KeOiP6PAALiagYwCoAnxi2+IeJ6GKXbccBKCaidQDmALhRCOEeGdSMOFlT53mbmrp6T5OxOvb8OHl1MUyDkJGRgUOHDiW92AshcOjQIWRkZHjaznUyVghRDGCYYvkDNu3HS6/fAfCOpx41M3rmtrJdV18vAkXAZSGurvUq9F5G/6HvN+4px7Qnv8CCO8bpH5BhGpj8/HyUlpaiOczxZWRkID8/39M2HBkbIVU1dUjxEdJSok0XZC/CdULAh/Chdk2d0MpBX1VTh50Hj6NHbkvXtl/vr1Aun7t2DwDgk00HXPfBMIkiLS0N3bt3T3Q3Gi2c1CxC+t7/Ea7QSETmhjyC/nD9XsxZXRp4L+ew2bT3WOB1TV29ljnml3OKccFf/ovDloIj+49V4VBlNW56bTWOVdUAAHYdOqHcx5ET/m117menauvxwufbcao2PNiLYZjEwSP6KFi7+2jU+5D1+ubXQwuDmEK/o6wSv5wTjJLLSk/RskWu3uV3layoqg1ZPuq3nwZef7xxH3Y8Os12H7OL/DcenUnjvy3diUfnb4aPCNeP6+HanmGYhoFH9Amm3mFobq45ZBmRv/D5Dpw85T6J6zO+3Zo6+xF2h7DgJ3V/fv/xlpD3hyqr8cQnX4csq6z292np9oMAgIqqGhTMnIePNuxz7SvDMPGDhV7B0u0HcdFTDRPA6zRONkftCzbtD1v3wmJ3L9UUYwa31mE0XlZZHXIjkO87W23s9gBw5q8X4MlPt+KZhdsCy8zZhIVbyjB//V7sPHgcAPD0wq2BNgcqqvD518k/YcYwjQkWegVXv7QCG749hq8ku3i8cLLA1At/6gOVqO8t109N7HwzAXrfOz/wvlZS+vP/tNh139aRvsm89XsD5yZn3bzqheW49uWVrvtlGCZ2sNA7YDV5VNXU4R/LShzNLV5xtLUL4OSpWuWqr/f7c2X07mjvnmnu2Yt75R3/XKvd1orsAupP0xC+3BzlJ7u/M8M0JljoHfh8S6iJ4anPtuKB9zbivXXfxuwYTnJXLwT+vtQ5h815fTuqt60XAU+a49Xqm4WK2ihuYvLIHRS8wewrr8LQh/+DHWWVgdWxror11KdbsV3aP8MwQVjoHdhkMd2Y3ivHTuoLpxtOA9uKqlq8teob5boWhr+jXUDr7KLdgdfWSdN4IY/c9x49GRi1H6ioxtETNfij1I9Y6nz5Sf++v/fi8tjtlGGSCBZ6B1qkhn48KUaUajSjXitOZpVxv19o69bYOdsIgbZRetlTZ803QTdQHW8dFYPz23pqX1xajtdXhN6k5Dq3XsxJrhi7iiSdBMM0B1joPZBqCH1dfewCgiKVuzpDKMlG6e1s4P0e+Cii41UrKl4BQEZa8BJatCUYPVtbL/CvL0NNXGkpwb56MSe5YuzWGi/AMIyfpBT6iqoaHKioinrSdOOeUNONLw4j+kgnJc17jTUZ2d7yk1hfWh7zClR2I/BqIwq2urYOX37jHEAm18PUKWheeuSEVp79yghuGut2Hw2LGGaYZCUpI2Mn/PFzlBl51X/73UG4elS3iPaz7UDo5J45evYqoqaYqwr/RirItYbSy3sUQmDc435zzyVDO0e2Y/grSXXOzgxZltkiRdnW7P+P/17kul+f1NmObdyrVF33yipsPVCJaYPykNOyhW07+UlCl0ueWQIAKHnMPiqYYZKFpBzRl0nFM15fYT8i/MXb61Awc572fiNN5XvPuxvQ/VcfKtfpJCdTYXqtyH2qF8FUBe8ZycgAYFAXb/b19d+Why1rm5kW8j7FF/ph/HfrQdf9Egjpqb7AazfMkfoJF9v7XxYEA7K2l1XGxXWzvl6wSyjTZElKoZdxsrLICcRMHvlgk217U5q8/uDfXOmflFSZkiLVDlPofZLSr9hxSNl2ZHf7OrBO+5axxhS0iDJrp84NTve+KlfFmvjHz8PmBqLlVG09etzzIR52uDZ0OHL8FJZtV39HDBNPkl7ovYry/32x03ZdUckRY5+R9cXMFCkTqbm/ViH0V/91hbqtQ64bFSqhX74jtF6M1SNJF/Oz0zlvs4nTd2iaYGRUTyQy5SfDvwcnKozv7ZUlJZ62s/LDl1fiey8td8w9xDDxIOmE3ioKsXTjW1niF7vPIrAJA+obRKTmAFOMrSYUFTUe7yY6n5ksltsO2OfEkSEpiMo871Ulh7F0m7PZx6k76xQZRN2CsbwWUn920XZP7e3YYtTljXWwGMO4kXRCb/0N6fymyk/UhNj13Vjj4l1y+XNL8QdFDhiVgEb6kw+O6N3b1nmsS+v13jNXmg9w4uv9FYHzNXPWX/H8Mtsnkb3lVQBi7HMPYM03Rzy1P+bxCcAW47uK9fkwjBtJJ/SRjOhHPboAI36zIGZ9WL3rCJ6WsjqaqEZyXkf0PznXn+e9TYZ/ctTqHaOixqPfv9cJ4ic/Cz9XFR8U7w2c72XPLdXev9cRsNukuVedlQ//3WfDTUW6mN3iET3T0CSN0J88VYfuv5qH94tDR5c7yo67bltlEwwUazbvCzdxeBWdTq39EbEHK/1PIH1Oa+26jVOaYhUxjAcL37fRlaMn9EfJXnXR7SGn5FDwmtCJtZBvxm5Pc06cMmzzLPRMQ5M0Qr+n/CSEAP7wccPkdbFjpMOTwQ8V6Xmj/cnr2OhrPSq322RhQ7gZygIshMD76/bgFksFLjtU8Qoy26Wb/ymNidFYna35sX21V29Og2FiRdIIfZYR0FNYkBPT/W47UIHSI+p6qioOeLD1A2rTUm5r+2Aiq67vOeoeYapTBlAm1cV1srjU2aslFsyXqlLVCYFb31yDeev3hrQ5YZPCOVr2lp8MeNoAsb+xZWeluTdimBiSNEKfmeYX+jM6uZsyaurq8Y1NMWwrk55YjLG/W+jYpr5eeHoc/3p/BRZu9nvuqD1x7Lf1WZS+sMDdR96rqUDOSaPi443xLw14XBJx+YHk3TWlgbiE6c+pi7OrBvRCCLy0eEdY2gPVZ3PWo5/hoqe+CB4/Rjpvlm3MSFNHGUdKXb3w7ELLNC+SRujNSMsjGvlLfv3BJoz7vbN4e+Hm179Ez3vCI1/tEndN/tNiXPe3VSiYOU9pFnAaQVo1LEUjXFfXb3tYt2wAwO1vORcfiZW7oRPyZyA/9fz8n+vwq3+tBxCeRtrKPe+ux//+bRUAv0vlbz78Cr94e11IGzNXj5Vdh04ESinWxWhEb35VsUyKBwCXPbsEvaQqYQxjJWmE3uSvDgFPJktiHJ34kc0Id8CDH7tuqxJ1R08hi7D7NIRedzI2Jys0n4zbyD5atigmp03kUbRXd0RzAv6NFd/gs80HUFZRjfvf2wgAISYZAHjy061h25uc/6fFOHCsKuamm1gPvtc1gCmNadokj9B70KRIc7K7EStBcDIVWE9TJ/+O7ojeOrHbJiMN0wbnhbUb2KWN1v7cuOFV+0RoIkTove3XWny8SsqVQyB0khKq7T9W5bivIydqsOeocxtd2L2SSRTJI/QOWNPYfqsxgamiR4eWjus/2+w9YlZ1b3AawVqFXUvoNYVFHsHvOnQc9UKgXVZ41kjT1hwtuyzzJMeqagKRro9++FVgebQBRuc8HjTTCQj0PS14o3L1MIJAO4fMmV4gDphiEoSr0BNRBhGtJKJ1RLSRiB5yaDudiAQRFUrLfkVE24hoCxFNiVXHw49tv8606UaLW+DRPe96P44qOMnRcgPvphtdm3CKL3g5zJq7EUdO1CgjbxdZaunGgnnFe3He7xfhkmeWoPxEDSqq5cnY2AkjgUI+8R1lx0NG/FZ8RBHdwO2ODfCInml4dEb01QAmCCGGABgKYCoRjbY2IqLWAG4DsEJa1h/ADAADAEwF8CwRxdblwDyWw7rSIydw19vrsNwmu6Muuw+HPwlcLkV47j/mzbUS8D6itwpvLG30adLOFxpi7uaTHg3j++QGXq/dfSRQ/vBvS0tC2sVSFwVC0w3vOHgcP3l1tW37WJ19RVUN9hlmolhN7jKMLq5CL/yYFTjSjD/VlfoIgMcByAbNSwC8JYSoFkLsBLANwMjouqzGSZAOHKvG26tLMePF5Xjhc3uPkUhs7Kt3ecubMnvV7pD3UZtuNI6pa6NXebFEovNPXDlEq13/vKAJRf7+2rUKNZUocwRFKJarSo6EfeayTd/69BCr+5xcmCWWTygykZokmeRHy0ZPRClEtBbAAQCfCCFWWNYPA9BVCPGBZdMuAGRlKzWWxRynAFH5B/Do/M2YOuA0Zbt4/P6s+cc/+Wq/6zbOk7GhJ6ojRNs10kAAwFZLRS3V8XS4bHi+Vjv5NE9Jbo5h+YoUH4hbOUenVMRON1Lrfr/cFXnKA5k1UpbNWJpuZLPTqp2HHVoyzRktoRdC1AkhhgLIBzCSiAaa64jIB+BPAO5UbKpSibCrnIhuIKIiIioqK4vM/uskSEO6Zoe8T09Tn7bTDzDSgBQ5uGhSv46otBSw9upHbz1NpyeZWBQc0cmOqcJakUqFfJqyucaaB0f1tbjlnHfypqkXAqe1yVCus34Gv3yn2PE4usgf4xdGWuZYeH89aLiNArF7+mCSD09eN0KIowAWwW9vN2kNYCCARURUAmA0gLnGhGwpgK5S23wAYTlthRAvCiEKhRCFubm51tVaOF3kumLlNNL78wJ7f2snZAGrqxdYZpknUB1TJWzn9O4AwJvNuFu7LA+t1UQqHjrFPewyLZywCKDqM7rsWefslzvKwp9OgvuzzxF00qVsIRBqCntt+S4ccHHRBEI/x5f+uwM7yirR74GP8HbRblRU1UQ8ypf3qzNfwzRPdLxucoko23idCWASgM3meiFEuRCigxCiQAhRAGA5gIuFEEUA5gKYQUTpRNQdQG8A4Zm94oz10d/uR+Vky96yP/pEVCpzg8os4TwZq/9jtkYJr7xnova28vGybAqDO9HRIV+PydtF4aUcAeB5yzxKJJOXv5zjMBJ3EPp/LCtx3feXxrzMt0dP4r5/b8D1/yjCv74sxUcb9tpuI2dIraqpx9fG9TRv/V4MmvUfPPz+RrtNHRl+ejC3E+s8Y4fOiD4PwEIiKgawCn4b/QdE9DARXey0oRBiI4DZADYB+AjALUKIuEQrOV3kVoG1C3tXpRE2iUX5N5WAf6GoruSU/0YWqB+edbrj8bq1Dx3Rd5TMFVecqWdHBwGzf3KWXlsA3x3mn4IZnJ/t0lI/Adx1r6zSPr7JsSr7hGf1QtgKvdW0pt7e/79Z0OXQ8VO4Y/Y63Pjal/jDx1uwaU/opLZT9K1pvnl3TWR1binkNSs9o0bH66ZYCDFMCDFYCDFQCPGwsfwBIcRcRfvxxmjefP8bIURPIUQfIUTcEnI4jXStI3g7G/h/v7afH4iF0KueJD4oth8Fypg3CTmp2aju7R23mdxfPekM6CV/A/yfq04qZBPzHK8/p7v2Ng1N0a4j2HlQPUGtk7bYvNTM/+XL6emF2zD9+VCz0hOf2KfONjeN1I1VPnak8ylM8pM0kbFO17h1RG9nusmTqjVdPKRzyLpTNk8BXojEDpuTlYYFd5yLHrn+qFw5UtVNG1qm25tcrKN9O7YfqPQk9OYZjurhfBNqrOjc0K2DCmupQescgwpToKPVZvnaZu98xo7kEXoH1etrqcK00CayU/bEyLHkDF9V4s1fXkUkQr/mgcno1bEV7pvWH6/9aFRInhk3/c1qkWq7bkxPtRDfN61fyPv/bNrvaV7g7ql9tNs2RrxU4zKfrioUWUqf/HSrY62Am4wiKtHGTsmR1dZ9Ldt+CFe9sIxTGDNJJPQO63RHpPLoaEyvDlH2KJwvoyhDl5GWgrG9O4TYYVU3t8uGB8MUMh0mUVN8hA6twnO49FaYdLyM6PNzovf0Mfnx2IY3/xzSSHNt4vSpPPHJ1xjz2Geu+zDdRCOdSJXF3Tpp/fN/rsWKnYc9F8Nhko/kEXqnyVjNUZo58nnsskGYYhNUFS86t1X7dYfh4k73xJVDAyN9pzTDPiJlcjKVpuvkvHdD3sWkfp20tnGrdBUPdPLa+Cw2+mhQ2fkB4ECFXsZMeTO7iFv2xmGSSOjtr2adCTYgOKK3VnFqCH43fbBWu1C/aXUb8/fewkEofUTaJhmfzW7euH4UXrluBJ6/5kwAzjcr+WYx8wJn807vjq0AAKmNdHbRyxOOLnLcwWeb92Pkbz4NS7esRLpDtEwPNdWpEuYxzZOkEXondG3jq0r8IeSJCDzpo+kFI/fMrZ9OgpTiI6WAq2zGdvsZ07MDzuvTMRAFm28J0PrTVUPwg9F+F1D55nn4uHMwlWk+SY1z4ROZkoPHtZPemR9RLDIZqCZu1+72m3O+1MijJPehpY2pjt0umWYh9LppZv+xbBcA+4jNuKL5W5TF3e1+lGo3FIf/aUD3huZmurFb/d1h+biysGvgeCaV1c5Cb9Z1TdP4IqyT5rpcWRgaRzD+D4sw48XlWtua7rm6idW85u83s4jqDFAypfqz1uacJJMxaRZC75VEjOh1R11eQt5TfIT7L+pvsx+16UalDdqmLFWgl7FQPlardD1x1jGRRFoEJTMtBWsfOD+ibU0B1RVS3Y9v24EK7D58AinGk4xb/QMAIUVR7CKqudAJw0KvIB42WDd07y2hXjfObVN9hKkD7SeVbzy3Z8j7aYPCywYCGiN6h3WmxshC392lUpfucQGgQiOSVQURIVtRPUuHes9Cb38eIZPUTywOqYalc/OXRXzJ9oMhOX7MNUdO6HsSMckJC72CRFQAUv2k/zJjaHg7DyN6n48cJzSt2S07Z2dghcJOrTuiV03+mYnVvj+qW2CZyq1Thc4Nd59GQjEVXp/a/vfsoKtnwHSjOdnp1EfVzSwQTKXRRflSfeHzHZjwx88D700T2LQnv3CsosUkPyz0CpYo8s8kgkuGOqfuN4VgZIFfsC838sC3krwvnIReFpk/XjEEd07uExZFfNeUPlE94eS0bIGSx6Zhxsig0OuG+0cyGWvm2nHD6yndNrEX3rh+FICguOqMB8pPOM9HqG6i5o1E9/ZqhzxgeWPFN1p7M6mvF3jwvQ2B5GtM06ZZC/07N41RLr/QxoQRT4gIf7jCvTKTPBI1X5t22on9OgIA5t02NvA04CTSbY2JzJysNFx+Zj4y0lIwsW/HkDYdW6e7mlDMQB1HE4XjHtR8+lX4JPqHt51j277ksWlhkb12eHWh9fmCcxrmSF7nya+6znkkPa53dIF5ug+fdon87Pj26En8fdmuiBLKMY0P+xj5JOeMTq1sPTYSYqNHaP54OdVBSDuF6SbF4qVxevuWOL293w5uFbTvjeyGxZJ/9roHJyM91cXf3mU40CbD/zn2Oc3eRdTpXjH9zHzMWR2esljlXuiWMlk3yMqr6SaFghbz4GSsu8q6zacuUNzMTPdSVcUvu7Yye46exCebQiuZVdd6M93Eyh+h/GQNlm0/iKkDG37wxARptiN6H5Gt++EC40ey5ddTlevjAVFo4rQ7J6uDikL96I3/jRcq7wpzNG7mh3/0skFYMnNCYH3bzDRkSC56VrOKz6e2Iw/q0jbwemCXtnj1RyNxr8No2mli0S7dcu9OrcKWuQm0UzRw6H60mgXwxx0YI3rhF3lVjhsr768Lq7PjyitLSgAgTKxV3P/vDWHLrntlFR6cG5rf3ksOHyD4OUfrsfOzt9bgxte+xO7DJ6LaDxMdzVboU3wUcGOzYuYyT09Nwd1T+zZIfwiEWmn4ZydooX70xojeWKQyJbRMT8W9F/bDPzVzyocXH1enKZbd+gDgnN65SE+1H207VwBTr7x1Qm9P+wGcYwd0jmlHii94q6oXAs8s3OZa5QoAfvPhV56OEwtURXJ0XDVNdh48HqiMFq3Q7z7iT+ymU7mLiR/NWujtJirlUaGuK6DMSz8sdG3z2Z3nhi6gUPHcYFMTVTbFkGVEb2czvn5cD+3zGJKfjfP7d8JdU/xPFEO6ZoeM8p/9/nAAwE/O7aG1Pzvkalc+ImUNV1VSNjezmm7aBM8jeqLA5yAA/OE/9jnmGyN1Hkb0lz+3FC8u3gEA2H+sOiovNPNzZlf+xNJsbfTFpeW2opGtmfN92uA8zFMUDtER1VxLqT2i0KpMOvVezVFpD+N4kQYPybRI9QVuVLec1yuwfO0D58PnI7TJSEPJY9M879f6OcrVrlJ8hC45mWFuiC0Ucwc6LqV6/fE+GWtuksgApEufWYKJfQn8sLEAACAASURBVDsiIy3FU6bNiqpa1NTVa0UbW/3ub3xttdbgRUWsTEBMdDRboQfsA3L+Z0xB4LWTsIzq3g43j++JvUer8ON/BIpqaU1kuQlWnkY2S1PTbhrfC4Pys3HuGZEVVtch0uAiEydh9ZE/WGu1ZfI1TWGGidU8eSTRz+Y2VRqFReLF2t1HsXa393TX/yzajZJDx/H6j0dh5c7Djmm4rZqsM1fgBgt9Ymm2ppseuS1tR3+yZ4eTZ8WgLm0xoHNbTOofmnZXR0KsTxPWbXRGprLXTTxFPhY4nY3PR7ju7AIUz5qM6VItW5UfvZdRrBOR3DD2lfvtzfe8uz7i45omsUSwYudhPLNwO67+6wosbaBYkYBLqoPO19TV41iVc7wBEx3NSugvHHQaZl7gn1zdUXbc9scuu+ipRiKvXDcC3xvZFcO65UTcl7BJT8uCLQ6Fyu320Zhxm4wl8puF5GYqe7tKMFpnpIY8hUXbHzvMdAtHXIKgnEj0DXnHQb/L5n4p331VTR1ufHU1vjkUe88Yu3z7QDB//m1vrsHgWf+J+bGZIEkp9Ha29/ycrBBxN23AVnu4PBmb2zrUhLLwF+NxXp+OePQyvfzxKm6b2DvM3dDa4wPH3KsCJSL5WqQ4uVfKJjT5lFTfY8+OofMfPTq0xPpZUzDr4gGOx7+qsGuIG+fLhgujHT0U8yyxiK84WOmt2tNvY+y1Y40FAIAvth7ERxv3Ydb7G5XbRIM1yMxk58Hj6HHPh/igeA/mb9gX8+MyoSSl0MucI0Ue/mJyn8CFl9s6HempKVhwx7l4+uphIdvIboJnnh46ao/EC+eZq4dj1neCWSRnjOgabrqxaMh3hrgHmDQpoXca0UtXoRw4pTq/MBdOzY/gscsH4eFLBgbeH3YxAT12efiNPBaf9zm9vY3oTe+XWPHvtX6/flnog6NuPTt6VU0dXlu+y7ailYx5mVs9dzbu8XuVfbg+3JmBiT1JJfR3nH8GgNAL9oZxQTfAFqm+gIlkYGd/5Gmvjq0ci2hHy/n9O2Ha4Dz8z9nd0amN3yuGSGWjD33v5B1hRvQ20gJMShxt9JKAynqgI6xPXBme+E15fI8irWrutgtrjnsVKT7CnBv1YhoaCq/3rz998jXu+/cGPPXZNo19B11SQ5YbV8SH66Mfzd/z7noUzJwX9X6SmaQS+lsn+N0Br5MyDZ5hqdwUiCYNyRkT237IoiLvOpCVUCF71h+bk9CbNuImNKB3VHo7k4hbet0nvzcMQ7tmhy0f07M9HrlkAO69UC/vjQpVj5xuPOf07oDHpw8JCyRrrMinYl6PukFN5tPQnxa4xxKY36119B/La9drwrbmSFIJPRFhx28vDElsFebNYlxhIRe69ObPV+mNEO34zpDOKGifFSjrJouDeanr3Fh07MFeR6mNFbvTkOuoKrezWf7G9aPxg7MKMC6KiU9Vn8ySiU4cc+kz4I+XmNxfr0C6zPTn3CNxdVnzzVE8atr/jXNdvuOw1rZeLjtz/sVqumlKT6PJQFIJPWAGtoQMV0LXG+/JZkTvdBE/9b1h9iulfRERbjGeLuT9maPPDJfEXEBiEqvFEyfzr108gzVgali30NG7m+BEWhJyTM/2ypvoOQ6ZJs3z65HrPofTItWHFyMIQCrSqCGry6vLd+GFxTtQUVWjbZs3CRm8uGxrNq0La6fyqHLvR1298Dyh7ZW6eoHaOm/ZPhs7SSf0VsLMJMaVJ+uoKvWvivF99EeI5nHl3f1lxlC8/9OxgWyPim4F0DEBRFpdKRE4/YbtPvNCaSK8eNZkvHXD6JD1bhWYIn3iObtXh5A9TxnQSXt/unMGQHh0dCKoF0BVjZ6ozV23B2+uDDWTWOdj31ldiq1Srp2g6QYomDkPv/toMwD1iF7nfvPw+xtR+OsFqIij333Pez5Er3vna002NxVcZyGJKAPAYgDpRvs5QogHLW1uBHALgDoAlQBuEEJsIqICAF8B2GI0XS6EuDFmvXfgril90LVdFlpYhnUqG738+3XyxvAyyjb3KYtRVotUDMpvq24fQcZ2p/TCjQ2nyEhVcFhGmi8knkHn5mjFKY++tbpWyHaWp8IXfuA++v7CCEDac/Ska1uTebeOxcjffqrV1mm02yU7E996OK5Mfb1AWYXeCPm2N9eELTty4lRI6o07314HAIE0GQGhN/r/3KLtyG2VHlIcx2THwUrkZLVARVUtCmy82z40XDFPnKpDa8U1EUsqqmu1zHVNAR13k2oAE4QQlUSUBuALIpovhFgutXlDCPE8ABDRxQCeAGDm+N0uhIjO8B0Bcp6Wf908BgM7+wXWvEnvKQ8GjMii//qKXbjWJvjGi3udOUHkFK7eL68Nvtp7DEBkk1O6CbwaA+YPvbMitYPqBuoh2aItTt+XU4CVjyIrlAJ4q4rV1qYegooRv1lguy5SkQeA2noRltLYC0u2HXSshGZ+B7Lp5uEPNinb7j58EpOeWAwAtvmUAtW3GuDS92rSasy4DgmFH7MCQprxJyxtjklvW1rXJ5rh3XIC9t4t+/xdXScJsHzRfL3fvtiDl4vrGyP/tlO9UNkUEcl125QmY4N1UMP7rLpf1WoovdvZO2UsdrpHEoLVpLpkZ7r2A/BPwgPO34l1lZeBw8HK+BT41vmcnXDLYWPndRPJvgBnzzX3bQVqbGzvH2/ch4KZ81BcGtSFoydqMOjBj7Fse3gd5aaG1rM/EaUQ0VoABwB8IoRYoWhzCxFtB/A4gNukVd2JaA0RfU5EyjpwRHQDERURUVFZWZmqScxQXSC6Pzi3knoAsNS4KHp39BfNUKXfNZHNSl5Ee/ZPzsLwbtlaGS4bC53aZKCgfRYeuTQYwaoyo5k46UJPjQlPu/3qQCSZ3jR3oVPwxM4DLJ64Bfj90ZJuuVKjmIrMt0eCTxOqdMZ2AVMqvtEoTmLuJZLR9stLStD73vk4pJjMnb1qNwDg4qeXBJatKz2KiupaPPXZVs/HMpm7bg8KZs7DHf9cG/E+YoGW0Ash6gzzSz6AkUQ0UNHmGSFETwB3A7jPWLwXQDchxDAAdwB4g4jCauQJIV4UQhQKIQpzcxs+F4ju703HRn/c+KHcY7h4tnGw8amcgwZ2aYOze7V3PMbI7u3wr5vPVqbxbay0SPVh0V3nYULfoFuhqkiIznfRM7eVVlvn74uMPjjvRFeMz+/nP6++DuUUrfvStbzZjUJ1GGIzJ2RiLeG4zmNmzFekVBJ/VvjVe0lTvKPsuGsbcz/hXjzu/OtL/7nuLQ9/ylYly1v8tX/eRecGtGz7IazeFeqeuv9YVWBe419rvvXc31jiSSmEEEcBLELQ/q7iLQCXGu2rhRCHjNerAWwHcEZEPY0R1pwbQOgP0GkErjPqvny4PzrSHK3rjjzMXX9w6zl4/cejnRsnCe/cNAY/Gts9ZFJ5xoiuAIABndU1c4HgqM7t+3Baba6zuxl4GdHPv/0cXGAUlM9ra2/qaZ0ROiWm+xT3xdbIM016fWpQXa75OfbnJAu4HCk75tFPsXT7QamesfuxdW58R41gQdUTgu5vTdVMNZf2jnFjKD3iPgfyvZeW4/LnloUsO2lJZ116JHHlFF2FnohyiSjbeJ0JYBKAzZY2cs23aQC2StumGK97AOgNILbJOzxiztTLCa7kH8NtE8PL10WCzkhGHk02JXt7rBiU3xb3X9Q/5NzPPaMjAGfbeGBCzmX/OqY2uwyZpolPRygz09zjIi4e0hnv3DTGtZ2K6/62KqLtAP1CLCaqkbKT0Nll8txTXoXH5m/2NKL/x/Jdjutl+7kyG6bLIRr6J2btzlOfuqeMiBc6I/o8AAuJqBjAKvht9B8Q0cOGhw0A/JSINhp2/DsAXGssHwegmIjWAZgD4EYhhF74XZzpJI3c5e//0mGdY7L/KiOc3GlyNzXSiJ4kxhRxJ4E1f+Su1aY0ftkqIRQQgYlcuWB7NMe5bWJv9MgNL3YeC9o7xFx4dczy6jvuFO9RL4RrmUsZq3gfq6pBwcx5+Ot//WNDOVJatT+3Y5g370QVQUlk8RUdr5tiIcQwIcRgIcRAIcTDxvIHhBBzjde3CyEGCCGGCiHOE0JsNJa/YywfIoQYLoR4P76n447qupd/qKoEZ/NuG4uHXNLgmlx/jj+JGhdDjgxzROlkXzfLEFpNIVZ0RrO5ivKLQgAbvvV7Z+m4LuqMFAvax2/iXH5SsH5uKT7CXz1E4f7XxkxkFx9wxZn2idzq6oM3Gq9Vqk7V1gdy1P96nj9Vw/Hq4G9K9eThJqSBLJ2eehI75m/Yl7Bsnc12SCnb88jlUxjQua2tb72VbsYPuimlEG5MmKMy+eNraUkZ8cBF/fHnq4Y6Bj0BeqPZ1348KqSqFeA3AVR6iLzUmaSPx/Ww4aEp+PL+80OCi76873wU3Tcp8J6IPAX6zS7arVw+9c+LlctPORjfv9p7LGA+m+dR4FQVp258bXXgtfnkIf+O3YTetO979dgx5410kPdtfcKorK7Fza9/6enYsaLZCb3q96Zjy43FcRh36i0j+uJZk7Hy3kkhbTJbpODSYV1c5zWcBM6sM9A5OxM/GH16yDoBgYFdnL1VZHS+63hcD63SU8NMJ22z0kIiVb1e211tXHaP2aTb6N3R3ssIiHzuya3f5ohe1lI3043pPaNq1T/PfvK/ZXoq/rZkJ5bvcPenX7It2EbHXGWydPvBgFdQPGh2Qq8iFqOtt288K6QeqJd9ZnuIkEx2zAGi+UNvk5GGlopweR2cvoMQMbTcEHaWHQ88mXVo5Z5zqKbW+Qf92GWDEjbZ7iN4isYzI7V1ERBYt/solm5Xm3wijd6udpkbMUVUDviqqxd4/KPNrmkorC6lANDFwbPIR8Cs9zdhxovLleurJDOtnAvIi9Bf/dIK3DF7nXZ7rzRboVdV2ImGEQXtQtIu6F7f62dNxtKZE6LvQJJgBh5FKu4yujdba9qCf6/15vPsdpgZI7vZrhtZ0A73X9Tfdn20ePW68crnW8pwyTNLcPVLYTGUjsd3+8xUPvky5u9XDuwtLi3Hs4u241ZFTh4ZVf56J1F2u47u+/cG5XIvQh9vmq3Qy8QjJbDuPltnpMW1wlVTY9qgPNw2sTd+ObWPe2MXdL9W66izU5sM7RD7J64cojR39Oqo52Ez+8az8P1R9jcCXS4anIeOimyYVpGa0Ldj1MeS+Y/LJKvdd+BmJn9rlXquwEQ1ojdfWz2lTtXWBwIZ7ah1EmWXS2HLvmC2zoIOwWsh2vQSsaTZKox8/cdjoownYyMjNcUXKAkZLbo3W+t3JVcocxOky4aHe51s/+2FAPzpbnWIxUDj6auHK5d/vHEfxkp59OOZ3tdH4b7s8aqrELDRS1pqNy98zV9XYGWJs1d3nYMob5NcpD/ZtB/nOxSN6SfZ+t+JwOb+dtFuXFGoP/mrS7Mb0Qd9aYPL4nEtJlvhkKaIyi7+0c/OCctrH16/NzpXvBSfN0+XeA4KrMFOq0piV7zEimpQbI0OjdmxbGz0KtxEHgBq6+y/6U83Hwi8vv4fRWHr5Zun3IXXlnsvcXjXnGLP2+jQ7Eb05u8v1EbPI/rmQt/Twr0rrN9Vq4zUwLKGSAUd70Mk8kosi1M1KFPUZXHXDUgaUZAT8v7r/RVYsTP8ZpCZlqIVD1NyKJjaoLEWK2l2I3ozyViWRjm/aGCdb3p0bJ2ORy4diMuH5yMnKw23TuiFN64fFffjxtsjJxrpOTeKurtA/G6UpummRhLWkkP+pGhuQW7DTw8V+pUKkQecc/zYYd5sTpyynxNIRJ77Zjeiv3ZMAYSAdgBUpOiUAmQaB6bOpvooxKf+zsnRTwg3BnTSONgRfSrs6IVeVcLT1MpPvwpOBpuphp2qxAHAC5/vwK8u6Bd4X3JQnTUzkgyZ5hOGqhqXyeHjp9BeEZEdT5rdiD4txYfrx/WIe4rfMzq1xhvXj8LmR5wSfTKNiVYuKRUaA/de2M+9kYWRBc4RxE4kMj+Lydb9leh7//yQZaagyvnwZROKG7KJ5a9f7FS2KTumNjtt+LYcR2xuJubntdqhkPvGPcFYhRcXb8esKCp86dL4r+wmzJieHdwbMXHlqe8Nc0x5DPgzZd55/hm4dJh9SbzGQtd23s0JOVE8XXoxOY/q3k5h647+RqEyxZij7b55zpG5dlTV1rm6NVfYuGRe9NQX6JHbEp/dOT5snfl52WX1BIBdUn7733642bZdLGl2I3qmefGdIZ1ds0YSEW6d2Ns2/F+mT6fIhEWXKQM64YGL+uODW8finN6qgUJ0ppBObeJnMlA5IMTrgcAckefn2H9ne8tP4pq/BgO5cqQI9GjMWYB9kZT135a7bpuIWs88omcYD4zp1R5b9le4N4yQF34QzDQ5rGt2WDZJIuCV/xkRd2cCEy+apAoQipfhJxAw5eAW+czCbfhiW/Dzk0fZjgFSUfDGim/w2+8OcmwTz5utHTyiZxgP3Dm5j6eUw9HMBamkKIUI5/XtiFE9nMtNXm0TbevV7TdXEW1rx7pS99FsrDhQ4befb7LJzVN+ssbRj/2/W6OvTb1Q8q/3Qm4r+yp28YKFnmE80Co9FYvuOk+r7Yp7JmLlPRMjPpbK7KEos6vErGFr9ViRZf7WCb3gxs3j/W3G9nKfb4rWHOIFM7/MIx9sCiybZpRzBIAhD/3Hcfuf/zP6BGKRVv6KxJsnWth0Y/B/1xaiu5TXm2Gc+O8vz0N6mrPqdnKoP6yDqr6xbg4e0/vDqbWqTqqVFqk+FJ6eE2IC8UJDeu3EK7fMiIIcTxHFBTPnOa5PhB89j+gNJvbrFLdSb0zy0bVdFjq2ju8juEoPdC0v5rbWYCz5vZsZx0ySJmegnDKgE9Y9MFmvEwC+OXQCGS43xFhR42Cvj4ZTLvv1XK6RhZ5hmicv/OBMPHLpwJBlKjnQtbGb25qtTdG++4K+gTZu+XjMsphys6wWqWiblYadj16o1Y8dB4+jqsZ+pD3IQ4EXN+LlzFLrUEULAEa7zJdYqRfAF1sPhtTAjTdsumGYRsCUAaeFLVPa6DWF3oxoPaunX4QyDS+dgVJMgZ0w5rXNwFs3jA7sw6d4CohV2obJ/TtpuSRauXVCL8x8Jz4JwKy4FUGprRfo3qEldtpE2Fo5cvwUbnh1NUb3iDyQzSs8omeYRkoPac7ILMii68XT57TWWDJzAn40tnvI8tAbhVqs95ZX4fT2LQNivnR7sDxerNPyqPb3/DXqlMsyx6vrwnLWx8t0s+1ApeP6unrhyTupyrhxyHns4w0LPcM0Uq4oDOa6f+QSv1lHLmzhRpfszIBYB232wfUpEfz6Y20dUT0ZOAVBmXy8cV/YMq8VndaXluNX/1rvaRsVtfUCLTx8mOZkbENWoGLTDcM0UogIPzm3B3JbpWPGyG6OJQm19wn9yViTKQM64eONzpWkIu6Pogs6/VKlRfDqdfOdp7/w1F7F20W7sW73UQzvlu3YLj3VFzAB1bPQMwwjI2dZjAaVq6au0NtFn7bNTIt6QlHlLqrqqw6JqNH66vJdAJxTI/94bHe8X7wH+40kaebcbk0D9pdNNwzTDLhocGcAQFsp34udzvfMDY0nkSssydI096dne+7HL6f2Qbo0z2Dtw5WF4aUZZaafab9ex0avE/gVCSkON837LuofckP7ZJPf7NSQAWYs9AzTDLhrch+se3Ay2mYGhd6MerVy7zT7p4jPJNE/vX14gKGbrXpkQbvQ6m6W9RP7dQoRRWtitw4Oedx1RvTxSk++p7wKhZaCJnbEywzmBAs9wzQDfD4KEXkAaN8qNH1xtjHab5FinzDNWtTj7F6hPuSnXHzOe3dsHZICwGo+cnvf3WEyWidRWZds72menZB7l+ew7xqXz8XEqfB4NLgKPRFlENFKIlpHRBuJ6CFFmxuJaD0RrSWiL4iov7TuV0S0jYi2ENGUWJ8AwzCRIWvokpkT0MmI9PUSeJSe6i2LZkoKhYy8fT6rsMPx/ZWFXW33XVZR5Xr8UQ6+625mIze+OWxf+OSQS9Urk5d+WOjeKAJ0RvTVACYIIYYAGApgKhGNtrR5QwgxSAgxFMDjAJ4AAEPwZwAYAGAqgGeJqGHyqzIM44g8WpZHuk7BUFbTzAENcZVxsmX7jx06GbthzzHLevvtD1a6i6lTrqBfX+qcXtiNdRq5gxKFq9ALP2bEQJrxJyxt5G+jpbT+EgBvCSGqhRA7AWwDMDLqXjMMEzVWyTMF1mlEf93YgpD3VuGe0Lej4zHltAvPXO0eGFVWoS7nFylO9xmzIIiurV1mSFdn98pEo2WjJ6IUIloL4ACAT4QQKxRtbiGi7fCP6G8zFncBIIevlRrLrNveQERFRFRUVhZ9nmiGYdxJ8ZEyY6vTqNlaHcnq6dLnNOcKXPL2qSnhxyHjX7w4r4/9jcjnI3xw61i8ct0Iz/u1uzl+Z0jnsGWxzO+ji5bQCyHqDLNMPoCRRDRQ0eYZIURPAHcDuM9YrDr9sBkTIcSLQohCIURhbm54xXeGYWKPz0f46Gfn4KuH/QXsTYF1GtEXWgqNWycZb5/Y2/WYJilE4Sl7pWN7KfCii53XjZnCYGCXtmidkaZso2KzkcZAZZL6/fTB+NOVQ8KWn9a2kRceEUIcBbAIfnu7HW8BuNR4XQpAnj3JB7DHyzEZhokPPiKkp6YEEp6ZphunEb11RGwV+ow0/Sm4FOWIHmiZ7t9Hvzznou6RYHcTsz7Z6NZ1NaNdVcFnWS1SkapwN7XLhhkvH39Az+sml4iyjdeZACYB2GxpI9/GpwHYaryeC2AGEaUTUXcAvQGsjEXHGYaJDjsp8+J1M97BFOKGnZie3r4lXvmfEfj9FeGj4WixvYlZHizevTk8GOyMTvb1KlS7NW9YMi1SfLZuoJEWd9FBJwVCHoC/G94yPgCzhRAfENHDAIqEEHMB/JSIJgGoAXAEwLUAIITYSESzAWwCUAvgFiFEXTxOhGGY2OAlBfEZnZxt8k44eeCc5zKpG2usxUA6tG4R1sYpZYQ1t//vLh+Ec88IN0OfqqsPK/jeELgKvRCiGMAwxfIHpNe3O2z/GwC/ibSDDMPEB6tumVpnN6L/9aVhU3OYNigP97zrzwA5UypqooOq8Ems8tx7ZXtZaCrivLaZ+ODWsbjoqWDiM6e+WW8CV42IPgFdLOHIWIZpptiNUK3LxxjFS7q2C58cbZuVFijKnZ/jLepU7XWTGI6cCE/ONrBLW1wlBWjdNeUMAMBfZgwNa5ug+5M2LPQM00yxmzi1ita0wX4h76FwxQSA+y7qh0uGdsakfv7w/bum9Am8dkJ1o2lsgimbdCb07YRtv7kAlwwN8xDXzgTqhMoVM1aw0DMME4JVtK4e2Q2bH5mqHNEDfjPHX2YMC9w4bjmvF87r6+4mneoLl594+tBHQp3Fdq/yogH85q4nFK6UXuiXF/l8hxss9AzDhGAVeiLy5DYJAPWSZ8nvpw/GF3efh3duGgMgmEZBZaOPNBe9HY9cMiBs2VUO+XKs2GXEtE60EhGyWth/RukaWTNj8VRgu++47ZlhmCaFKWmx0BtTH09vn4UrCrsiPycLZxqpBUwxVwm9E6rMk6k+wqzv9Fe09tOzY7hL5O+mD9Y+pp3QT+wX6hXkI2ehXnDHufibS8Stx4/DEyz0DMOEEAu9MSdapw48LWydmTahrl7AGhhrfS+z6K7xAICLjDkDANj22wvxg7MKbLcZ2jVbK/AqI00thXbdUT31OAl913ZZrjEH8TRbsdAzDAMA4ekIouCKM7vip+f1ckyJUF0bHlJj9Wc3adkiBWmGyaeyujZkndNIOKtFKubffo5Gj22w+UjChB7en1CsxHMimoWeYZgQYiE4LVJ9+MWUPshqYR+qo0qoZtX5v/+vP9nt098PZrrs2Dq0ylQ8fe/t5gxUc7LW3Pq65BgFX+J5HlwcnGGaGWvuP9+1ElRDkJbiC8t2aRXWc8/IxZf3n492LYORqk7lBGNNvc3HZBVlosht7FMH5uHNld+wjZ5hmNiR07IFOrVp+AyKVtJSfBh3Ri4+v2t8II+9SlhlkQeCE70qLhmq9kW/b1q/QGCXF64Zfbpyucoe71ZUxY60FDNrKI/oGYZJMkyBO719y8BoVmeWwGku4eyeHfDe2vAEuT8+p0ckXcTY3uqMkirTTaSmF3NyOlLTjw48omcYBoCeyMYSWRjN13aTsTKObSStvNRmdK9iokMk7+OXD8acG88KWaYafevo9LVnhT8hnDzln1xu6eCHHy0s9AzDJJyfntcL7Vu2wMgC++LdJnZpfoFQAf7zjLBcjLY4RbVeOaJrWMEVldDvLXevn3vH5D5hy6YM8LugDuvmvYShLmy6YRjGQnzTELx941lYufNwyLIhXbOx+v7ztbavrbMX+kh7np7qbTStHNFrDOmtEbJ5bTNwwaA87Hz0Qva6YRimAWgg282IgnYYoTFyt8Na1UpGkT4nLlht9ELoVaWyavmyX000lsf35spCzzBMCI0tg6QVa0FyGa+eKyO7t8P90+xTKNihEmadY0fqmRMtbKNnGAYAcPUof7GM3NYN56ceCbec1xNAMJgK8NvYP/7ZuAj21QuD8tt63k4l2DqRsXbZL+MNj+gZhgHgd0GM1A2xIemR2wolj00LWXbZ8HwAwNf7KzztSyerpAqViShBGq5FI+4awzCMN3QtI7N/chZyW6djSH52RMdRu1fqHfy+af0iOmY0sNAzDJM06IrtyO7tsOreSciM0HfdepzxfToqC6moSMRTE5tuGIZJGhpqqtO0xxMBa++fjDaZqVi2/VADHd07LPQMwyQN8XZTNDlQ4Q+OEsJfIN1/8AY5dESw6YZhmKTB1Pl4phMAgLKK6rBlHVsnPlGcHSz0DMMkDfHMACmj8uXvpShb2Fhg0w3DMElDQ1lP2mSmKZe/cf0odG4bXts20bDQMwyTNDRUuczG7QAACOBJREFUCgQ7xvRUpzVONGy6YRgmaTAnY+OdtufiwfopkBsDrkJPRBlEtJKI1hHRRiJ6SNHmDiLaRETFRPQpEZ0urasjorXG39xYnwDDMIxJQ5luAp42TQQd0001gAlCiEoiSgPwBRHNF0Isl9qsAVAohDhBRDcBeBzAVca6k0KIobHtNsMwTDgNNRnb1HAd0Qs/lcbbNONPWNosFEKcMN4uB5Af014yDMNowDqvRstGT0QpRLQWwAEAnwghVjg0/xGA+dL7DCIqIqLlRHSpzf5vMNoUlZWVaXeeYRhGhkf0arS8boQQdQCGElE2gHeJaKAQYoO1HRFdA6AQwLnS4m5CiD1E1APAZ0S0Xgix3bL/FwG8CACFhYUNXbqSYZgkoSFl/pmrh6NTm8ad0tnEk3ulEOIoES0CMBVAiNAT0SQA9wI4VwhRLW2zx/h/h7HtMAAhQs8wDBMLGioFAgBMG5zXYMeKFlehJ6JcADWGyGcCmATgd5Y2wwC8AGCqEOKAtDwHwAkhRDURdQBwNvwTtQzDMDGnqVhu5tx4ljKNQrzQGdHnAfg7EaXAb9OfLYT4gIgeBlAkhJgL4PcAWgF427ijfiOEuBhAPwAvEFG9se1jQohN8TgRhmGYpmKjL4yiZm4kuAq9EKIYfnOLdfkD0utJNtsuBTAomg4yDMPoYur8iVN1ie1II4MjYxmGSRpq6uoT3YVGCQs9wzBJAzXmpPAJhIWeYZik4ciJU4nuQqOEhZ5hmKRh16ET7o2aISz0DMMkDSLueSubJiz0DMMkDYJ1XgkLPcMwSUOKjydjVbDQMwyTNHTJbnxl/BoDLPQMwyQNFww8DQDwyCUDEtyTxgXXjGUYJmlITfGh5LFpie5Go4NH9AzDMEkOCz3DMEySw0LPMAyT5LDQMwzDJDks9AzDMEkOCz3DMEySw0LPMAyT5LDQMwzDJDks9AzDMEkOCz3DMEySw0LPMAyT5LDQMwzDJDks9AzDMEkOCz3DMEySw0LPMAyT5LDQMwzDJDks9AzDMEmOq9ATUQYRrSSidUS0kYgeUrS5g4g2EVExEX1KRKdL664loq3G37WxPgGGYRjGGZ0RfTWACUKIIQCGAphKRKMtbdYAKBRCDAYwB8DjAEBE7QA8CGAUgJEAHiSinFh1nmEYhnHHVeiFn0rjbZrxJyxtFgohThhvlwPIN15PAfCJEOKwEOIIgE8ATI1JzxmGYRgttGz0RJRCRGsBHIBfuFc4NP8RgPnG6y4AdkvrSo1l1v3fQERFRFRUVlam13OGYRhGCy2hF0LUCSGGwj9SH0lEA1XtiOgaAIUAfm8uUu1Osf8XhRCFQojC3NxcvZ4zDMMwWnjyuhFCHAWwCArzCxFNAnAvgIuFENXG4lIAXaVm+QD2RNRThmEYJiJ0vG5yiSjbeJ0JYBKAzZY2wwC8AL/IH5BWfQxgMhHlGJOwk41lDMMwTAORqtEmD8DfiSgF/hvDbCHEB0T0MIAiIcRc+E01rQC8TUQA8I0Q4mIhxGEiegTAKmNfDwshDsf+NBiGYRg7SIgwk3lCIaIyALui2EUHAAdj1J2mAJ9vcsPnm9zE8nxPF0IoJzkbndBHCxEVCSEKE92PhoLPN7nh801uGup8OQUCwzBMksNCzzAMk+Qko9C/mOgONDB8vskNn29y0yDnm3Q2eoZhGCaUZBzRMwzDMBIs9AzDMElO0gg9EU0loi1EtI2IZia6P9FARCVEtJ6I1hJRkbGsHRF9YuT1/8RM90x+njTOu5iIhkv7aZS1AIjoZSI6QEQbpGUxOz8iOtP4/LYZ26pyLjUYNuc7i4i+Nb7jtUR0obTuV0bftxDRFGm58honou5EtML4HP5JRC0a7uzCIaKuRLSQiL4yaljcbixPyu/Y4Xwbz3cshGjyfwBSAGwH0ANACwDrAPRPdL+iOJ8SAB0syx4HMNN4PRPA74zXF8KfLZQAjAawwljeDsAO4/8c43VOos/N6Ns4AMMBbIjH+QFYCeAsY5v5AC5ohOc7C8AvFG37G9dvOoDuxnWd4nSNA5gNYIbx+nkANyX4fPMADDdetwbwtXFeSfkdO5xvo/mOk2VEPxLANiHEDiHEKQBvAbgkwX2KNZcA+Lvx+u8ALpWW/0P4WQ4gm4jy0IhrAQghFgOwpsKIyfkZ69oIIZYJ/6/iH9K+EoLN+dpxCYC3hBDVQoidALbBf30rr3FjJDsB/oI/QOhnlxCEEHuFEF8arysAfAV/evKk/I4dzteOBv+Ok0XotfLeNyEEgP8Q0WoiusFY1kkIsRfwX1gAOhrL7c69qX0msTq/LsZr6/LGyE8NU8XLFKy85vV82wM4KoSotSxvFBBRAYBhAFagGXzHlvMFGsl3nCxCr5X3vglxthBiOIALANxCROMc2tqde7J8Jl7Pr6mc93MAesJfnnMvgD8ay5PmfImoFYB3APxMCHHMqaliWZM7Z8X5NprvOFmEPqny3gsh9hj/HwDwLvyPdPuNR1YY/5vpoO3Oval9JrE6v1IES1nKyxsVQoj9wl/Qpx7AS/B/x4D38z0Iv6kj1bI8oRBRGvyi97oQ4l/G4qT9jlXn25i+42QR+lUAehsz0y0AzAAwN8F9iggiaklErc3X8Ofw3wD/+ZheB9cCeM94PRfADw3PhdEAyo3H4qZWCyAm52esqyCi0YZt84fSvhoNpuAZfBf+7xjwn+8MIkonou4AesM/8ai8xg0b9UIA043t5c8uIRif+/8B+EoI8YS0Kim/Y7vzbVTfcaJmqmP9B//M/dfwz1rfm+j+RHEePeCfbV8HYKN5LvDb6T4FsNX4v52xnAA8Y5z3egCF0r7+F/6Jnm0Arkv0uUn9ehP+R9ka+EcxP4rl+cFfznKDsc3TMCLAG9n5vmqcT7Hxw8+T2t9r9H0LJG8Su2vcuGZWGp/D2wDSE3y+Y+E3LRQDWGv8XZis37HD+Taa75hTIDAMwyQ5yWK6YRiGYWxgoWcYhklyWOgZhmGSHBZ6hmGYJIeFnmEYJslhoWcYhklyWOgZhmGSnP8H3H77vNJCoZcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#hide\n",
    "learn.recorder.plot_loss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save(path_data/'GPT2_pt_3epoch_lr5e-4')\n",
    "learn = learn.load(path_data/'GPT2_pt_3epoch_lr5e-4')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.2G\t/mnt/home/pierre/.fastai/data/ptwiki/GPT2_pt_3epoch_lr5e-4.pth\r\n"
     ]
    }
   ],
   "source": [
    "# model size\n",
    "!du -hs {path_data}/'GPT2_pt_3epoch_lr5e-4.pth'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 6.2.4 Unfreeze all layers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GPT2LMHeadModel (Input shape: ['8 x 1024'])\n",
       "================================================================\n",
       "Layer (type)         Output Shape         Param #    Trainable \n",
       "================================================================\n",
       "Embedding            8 x 1024 x 768       38,597,376 True      \n",
       "________________________________________________________________\n",
       "Embedding            8 x 1024 x 768       786,432    True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 2304      1,771,776  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       590,592    True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 12 x 1024 x 102  0          False     \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 3072      2,362,368  True      \n",
       "________________________________________________________________\n",
       "Conv1D               8 x 1024 x 768       2,360,064  True      \n",
       "________________________________________________________________\n",
       "Dropout              8 x 1024 x 768       0          False     \n",
       "________________________________________________________________\n",
       "LayerNorm            8 x 1024 x 768       1,536      True      \n",
       "________________________________________________________________\n",
       "Linear               8 x 1024 x 50257     38,597,376 True      \n",
       "________________________________________________________________\n",
       "\n",
       "Total params: 163,037,184\n",
       "Total trainable params: 163,037,184\n",
       "Total non-trainable params: 0\n",
       "\n",
       "Optimizer used: <function Adam at 0x7f10d796ae60>\n",
       "Loss function: FlattenedLoss of CrossEntropyLoss()\n",
       "\n",
       "Model unfrozen\n",
       "\n",
       "Callbacks:\n",
       "  - DropOutput\n",
       "  - ModelToHalf\n",
       "  - TrainEvalCallback\n",
       "  - Recorder\n",
       "  - ProgressCallback\n",
       "  - MixedPrecision"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.unfreeze()\n",
    "learn.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `learn.summary ()` method gives almost the right numbers. In fact, it counts twice the weights of the wte matrix (vocab embeddings) because they are duplicated in the weights of the output linear layer.\n",
    "\n",
    "The real numbers are:\n",
    "- Total params: 163,037,184 - 38,597,376 = **124,439,808** (about 124 millions)\n",
    "- Total trainable params: 163,037,184 - 38,597,376 = **124,439,808** (about 124 millions)\n",
    "- Total non-trainable params: **0**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3.288433</td>\n",
       "      <td>3.186721</td>\n",
       "      <td>0.377380</td>\n",
       "      <td>24.208906</td>\n",
       "      <td>6:06:29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>3.232569</td>\n",
       "      <td>3.167864</td>\n",
       "      <td>0.379885</td>\n",
       "      <td>23.756687</td>\n",
       "      <td>6:16:22</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(2, slice(1e-4/(2.6**4),1e-4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD4CAYAAADlwTGnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO2deZgU1dW43zPDMMO+DosMMKAoyjbggPuGqCiRkOinJJqoiSEaTUxMjPiLS9SYGP0+Y0yM0URNjAsqxmBAXFAQFRDZd2Tf931fZu7vj66aqa6u6q7qrupl+r7P089037p161ZP9T33nnPuOaKUQqPRaDT5R0GmO6DRaDSazKAFgEaj0eQpWgBoNBpNnqIFgEaj0eQpWgBoNBpNnlIv0x3wQ+vWrVV5eXmmu6HRaDQ5xcyZM7crpUrt5TklAMrLy5kxY0amu6HRaDQ5hYiscSrXKiCNRqPJU7QA0Gg0mjwloQAQkRIRmS4ic0VkoYg86FDnFhGZLyJzROQzETnNKC8XkUNG+RwR+avlnNONc5aLyFMiIsHemkaj0Wji4cUGcAQYqJTaLyJFwGciMl4pNc1S51Wl1F8BRGQo8AQw2Di2QilV4dDuM8AIYBrwrlF/fJL3odFoNI4cO3aM9evXc/jw4Ux3JXRKSkooKyujqKjIU/2EAkBFggXtNz4WGS9lq7PX8rGR/bgdEWkPNFVKTTU+vwQMQwsAjUYTMOvXr6dJkyaUl5dTlxUNSil27NjB+vXr6dKli6dzPNkARKRQROYAW4EPlVJfONS5TURWAI8BP7Ec6iIis0XkExE5zyjrAKy31FlvlDlde4SIzBCRGdu2bfPSXY1Go6nh8OHDtGrVqk4P/gAiQqtWrXytdDwJAKVUlaHGKQMGiEhPhzpPK6VOBO4G7jWKNwGdlFJ9gTuBV0WkKeD0n3BcNSilnlNKVSqlKktLY9xYNRqNJiF1ffA38XufvryAlFK7gUnU6vedGEVEnYNS6ohSaofxfiawAjiZyIy/zHJOGbDRT19S4fCxKt6auR4dCluj0eQzXryASkWkufG+ATAIWGKr083ycQiwzHJuofG+K9ANWKmU2gTsE5EzDe+f7wJjArgfT/z23cX8/M25fLZ8e7ouqdFo8pTdu3fzl7/8xfd5V1xxBbt37w6hR7V4WQG0ByaKyDzgSyI2gLEi8pDh8QNwu+EiOoeIqucGo/x8YJ6IzAVGA7copXYax24F/g4sJ7IySJsBePOeiI7swJHj6bqkRqPJU9wEQFVVVdzz3n33XZo3bx5WtwBvXkDzgL4O5fdb3t/hcu5bwFsux2YAMbaEdFCr+MkPvaBGo8kcI0eOZMWKFVRUVFBUVETjxo1p3749c+bMYdGiRQwbNox169Zx+PBh7rjjDkaMGAHUhr7Zv38/l19+Oeeeey5TpkyhQ4cOjBkzhgYNGqTct5yKBRQUpuo/T+xCGo3G4MH/LmTRxr2JK/rgtBOa8sCVPVyPP/rooyxYsIA5c+YwadIkhgwZwoIFC2pcNV944QVatmzJoUOH6N+/P1dddRWtWrWKamPZsmW89tpr/O1vf+Oaa67hrbfe4vrrr0+573kpAMw1QIGWABqNJs0MGDAgyk//qaee4u233wZg3bp1LFu2LEYAdOnShYqKyH7a008/ndWrVwfSl7wUANXmCiCz3dBoNGkm3kw9XTRq1Kjm/aRJk5gwYQJTp06lYcOGXHjhhY5+/MXFxTXvCwsLOXToUCB9yctgcKb7p14AaDSasGnSpAn79u1zPLZnzx5atGhBw4YNWbJkCdOmTXOsFxZ5uQIwjcBaAGg0mrBp1aoV55xzDj179qRBgwa0bdu25tjgwYP561//Su/evTnllFM488wz09q3/BQANUZgLQE0Gk34vPrqq47lxcXFjB/v7AFv6vlbt27NggULasp/8YtfBNav/FQBGX/18K/RaPKZ/BQANTYALQI0Gk3+kqcCIPJXD/8ajSafyU8BgPYC0mg0mvwUADUrAC0BNBpN/pLfAkCP/xqNJo/JTwGgVUA5w1sz13PKveM5VlWd6a5oNGmhcePGAGzcuJGrr77asc6FF17IjBkzUr5WXgqAVdsPADB23qYM90STiN+MW8SR49XsO6xDd2vyixNOOIHRo0eHeo28FABb9h4B4IOFm0O/1vGqar7z/BfMXLMzcWVNDGbAvmqdvU2To9x9991R+QB+/etf8+CDD3LxxRfTr18/evXqxZgxsfmwVq9eTc+ekYj5hw4dYvjw4fTu3Ztrr702sFhAebkT2KReQfjyb/2uQ3y6bDtrdhxk8i8vCv16dQ1TTacFgCYQxo+EzfODbbNdL7j8UdfDw4cP56c//Sk/+tGPAHjjjTd47733+NnPfkbTpk3Zvn07Z555JkOHDnXdm/TMM8/QsGFD5s2bx7x58+jXr18gXc9rAVBYoI0A2U7ND0KP/5ocpW/fvmzdupWNGzeybds2WrRoQfv27fnZz37G5MmTKSgoYMOGDWzZsoV27do5tjF58mR+8pOfANC7d2969+4dSN8SCgARKQEmA8VG/dFKqQdsdW4BbgOqgP3ACKXUIhG5BHgUqA8cBe5SSn1snDOJSLpJcy1zqVJqaxA35ZUNu4NZRnlBG5yTw/zaqrUA0ARBnJl6mFx99dWMHj2azZs3M3z4cF555RW2bdvGzJkzKSoqory83DEMtJUwIhd40YEcAQYqpfoAFcBgEbGHrHtVKdVLKVUBPAY8YZRvB65USvUikif4X7bzrlNKVRivtA7+AB2ap55SLRE67lBqmDYApZcAmhxm+PDhjBo1itGjR3P11VezZ88e2rRpQ1FRERMnTmTNmjVxzz///PN55ZVXAFiwYAHz5s0LpF9ecgIrIrN6gCLjpWx1rDnWGpnHlVKzLeULgRIRKVZKHUml00HRtEFR6NfQcYdSo9YGkNl+aDSp0KNHD/bt20eHDh1o37491113HVdeeSWVlZVUVFTQvXv3uOffeuut3HTTTfTu3ZuKigoGDBgQSL882QBEpBCYCZwEPK2U+sKhzm3AnUTUPQMdmrkKmG0b/F8UkSoiieN/o1SspU9ERgAjADp16uSlu55JhwlAj1upUeMFpCWAJseZP7/W+Ny6dWumTp3qWG///sh8u7y8vCYMdIMGDRg1alTgffLkBqOUqjLUO2XAABHp6VDnaaXUicDdwL3WYyLSA/g98ENL8XWGaug84/Udl2s/p5SqVEpVlpaWeumuZ9IxKdeB51Kjxgasx3+NJnB8+UEqpXYDk4DBcaqNAoaZH0SkDHgb+K5SaoWlrQ3G333Aq0AwaxofLNiwl5FvBaNLc0dLgFSodQIKTgIcr6rm0NGqwNrTaHKVhAJAREpFpLnxvgEwCFhiq9PN8nEIsMwobw6MA+5RSn1uqV9PRFob74uArwELyACjvlyXluvo8T85zIB9VQGqgG5/dTan3v9eYO1psh8H7XKdxO99elkBtAcmisg84EvgQ6XUWBF5SESGGnVuF5GFIjKHiB3gBrOciN3gPhGZY7zaEHEpfd9ocw6wAfibr55nEf+etZ7Pl293PJYnz11orN15EIB/TlkdWJvvpWEHuCZ7KCkpYceOHXVeCCil2LFjByUlJZ7P8eIFNA/o61B+v+X9HS7n/gb4jUvTp3vsY9Zz5xtzAVj96JCYY7UJ6KPXAMu27OO6v3/B2J+cS5sm3v9h+Uo692xo6hZlZWWsX7+ebdu2ZboroVNSUkJZWZnn+nm9Ezid2FVAz3+2iq37jvDR4q18a0Cw3k11E61E0yRHUVERXbp0yXQ3spK8DAbnxIINe9h3+Fjg7dbxVWfaCNtja9eBo2zdF38npkZT19ACgIiP+df+9Bnf+8eXoV1D7wNLjbD3bPR9+EMGPPJRuBfRaLIMLQCo1dPPXLMrhLYjrbvFs8/UCmH7/iOUjxzHGzPS4wWVKjp9p0YTPFoAEK6LmNn0pj3R6oVMrwjMpDivp8kNNlUy/X1pNHURbQSmdgUQhhjINhvA4WNVUYNproyrWgBoNMGjVwCEO0gv2LgnvMaToPt973H27z7OOsGk0WjSjxYA1OrpwxgUfzk67FAT/tlx4GjN+1Rn1os27uXX7ywMfZNNGDaAfg9/yO/fW5K4okZTR9ECIAFHj1fzxAdLOXjU2Yj7u/GLa96vM3at+iFTce6DGrCveOpT/jFlNTstQiUbGTNnA3+bvDKqbOeBozwzaYXLGRGOHNcxgzR1Fy0AgC173NMTvDFjHU99vJynPlruePzZT2oHlfcW+AkxkB1K7aBm1mFHaz6xtFFK598xag6PvLs4cUUL7y3YxCn3vsfiTXsdj+86cJSZa3am1C+NJpNoAQAcrap2PXb4WGQGePS4ex2TXMpaFXRPw1IBDezeBoBeZc1DaT8eExZHktTN3+Bsxxn+3DSuesY5prtGkwtoAQDEGw43G+6b/5y6OmErU1fs8H/lDMmMl6cZKegCWogcD2kJYHbPTcDsOnDU58orGFZtP8DSLfvSfl2NJki0ACD+IGz6yXsJR7x5r59Ml2aqSB+nBMjYeZsCbS/IcM1OuLX+w3/N5JaXZ7JtX/R3f+WfPuPO1+eEcvGlm/dx0f9OCqZtjSaDaAGQgAMuxl8njsdRJWWa5yavoOKhD2LKg5I/oa0AEmQEM8NFH7N99/M37OHfszekdm2X8g27o4392fx/12jioQUAsP+I+yDvJ5n78WrFrS/PZOD/TfJ8jpsR9ujxalcPlFXbD/C7dxf70rv/9t0l7D4YfLA7k6rqsAfB+Pca798UtH1Ch6XQpIu9h4/x/GerQrOxaQEAXPvsNNdjfoKQKaUYv2AzK7cdSLlP/R+ZQI/733c8dtOL03l28krW7Ih1O12xbT9ff/pz9nqMbOpXBfXZsu18uiw2rnpYKwBzHu72/DsZ3rfsjQ67kexvx+tpuWP61+Qavx6zkIfHLuIzl4RTqaIFALVeQPXrxX4dflYAvX14qiQalPYcOuY6qK52GPhN/vDhV8xdt5tJS70lv/A7m73++S/4zvPTY8qPV4WsAkpUz3Ifr3yxtub9ym37qU519mT/imyfu/1qfBpyS2vykd2HIhO5I8fCWWF7yQlcIiLTRWSukfbxQYc6t4jIfCPl42cicprl2D0islxElorIZZbywUbZchEZGdwtpc7WfYf5vw+WUl2tfK0ATmnXJLxOOeA0sG01jKHpTn8XuhFYWd+rmtj9iW7zqmem8NWW/YH2xemRSFduaU1+EpaziJdgcEeAgUqp/UYC989EZLxSyqo3eVUp9VcAI0/wE8BgQxAMB3oAJwATRORk45yngUuA9cCXIvKOUmpRMLeVGr94cx6Tv9rGed1KKfDxzadj0H32k/g7V6evimxM2r7f287coB6ssFRAHy7aAkSreioe+pA9h44x4c4LLCk3nc/fdfAYT074KtA+vTs/WA8qjSZTJFwBqAjmFKrIeClbHetWyUaW418HRimljiilVgHLgQHGa7lSaqVS6igwyqibWYxem5u/qpWKEQD/mraGJz50HlCScQaxNr986z7KR45jYZwAcr8bXxu7pklJkWu9MLKbxSOdK4A9xrJ4zY5aW0s8OfaBIUQA3k8mIbzt1j5eUvdzy2qyi7Dmlp5sACJSKCJzgK3Ah0qpLxzq3CYiK4DHgJ8YxR0A69p4vVHmVu507REiMkNEZoSd1NnLTt77/rOApz5axrIt+2KMjX9IMNOctTZ+wpn3F0YGqv/O9TbDjNdfczzec/BYWlYmxy1eQNXViic+WMrWvcGlWHS6g6pq5fuH8cN/zfRc102ohJ2dTKNJF54EgFKqSilVAZQBA0Skp0Odp5VSJwJ3A/caxU4/FRWn3OnazymlKpVSlaWlpV66mzROg4mbi+glf5jMGb/1l0IwkaGw1uDpcVSLU00pxertB+jz0Af8y9z1G+eabizdvI8nJ3yVUI1iXQHMXreLpz5ezp1vzI3fuA+chFiUDSSFQfnX7yz0Vd+PWlCjCYKwHjlfXkBKqd3AJGBwnGqjgGHG+/VAR8uxMmBjnPKMEvY82TpwLNzoHGAM4GMjBg3AgSPHmeLiApaovxt3HwLg/jH+Bjgrlz05mScnLOPJCcvi1rPaAExVmKlKS4WKju6eVZFrpv5f+8eU1b7qF+olgKaO4MULqFREmhvvGwCDgCW2Ot0sH4cA5mjxDjBcRIpFpAvQDZgOfAl0E5EuIlKfiKH4nVRvJlVCj2lvEQBmgDHrUGK6Mi7bWuu1ctfouXz771+wwRjMrcTrrlLQqDjYhG/WHa/27+qLlbVRMb26bnqhZaP6xvVij1lXHencnKUXAJq6gpcRoj3wTxEpJCIw3lBKjRWRh4AZSql3gNtFZBBwDNgF3ACglFooIm8Ai4DjwG1KqSoAEbkdeB8oBF5QSiU/TQ2IIIf/tTsO0qlVw6iywiR2XSzZHAk4dsghJEUiVdF/5yZeVPmReaNnrq95/8Lnq/n+uV1qPq/YViu0rOPjz16fw6fLtjHj3ku8Xwh4/rNVrLUYeZ3u9dDRqowE09MqIE26CHtSmlAAKKXmAX0dyu+3vL8jzvmPAI84lL8LvOu5pylwvKqa+8Ys5EcXnkjHlg1d6wX5XVc5NJbawBF7btwVAIq/f7YqhevFYrWHzF23O+qYPRYPRB7et414PIs37aVTy4aeVyUPj414BJvhoJ3udcnmfQndQP1yvKqaW16exR0Xd6tp+5dvzWNoxQmUFBUGeq2wGfzkZE5p14Q/Do/5+Wo0QJ7sBJ6+eievTV/LXaODM0omoqpaxQyKTruKVdRxf9eIJ6+8CjM/u2StdbftO0L5yHE1n607gZ3u4/I/fsotL3vzwFm9Pda905dwTkGSr9x+gAmLt3DnG9GRRJ1UcNnOks37GDMn46Y1TQr4iUSQDHkhAPxw+FhVTQ6AVBj0xCf0fCA6lo+T7TDhAJzA08dOswaRvQF94hhPrUxb6T2jlVXPbt+rYHUDnWj4yc9aG71K+GKVt2td6BBqWRFJ6/hzm2eRk4AIaiHn9tNbv8tZGCS1x6AOsmXvYcpHjmPaSv/5MXKByV9tY4FLkqBcQwsAG7e/OrsmxHCqHLFlEXNSAXm91qrtsQHmnGRHzw5NAWhgqCuCxNp9u3usdQXg+sNPYWRWSnHHqDm8NavWDmEaiCNNh6srHWMJLV3fxZjzxAfxXWWPV1Xz8rQ1dT58tCno47kf5zLffWE6X/vTZ2m5Vtg2AC0AbExYvCVxpSRxGsSPHU/wDzYG3R+8NMPTNcxZesoB0BJg3/i7fX9tQpYCFzfJZAbpeB5FJ7VpHJUvYNy8Tfzv+0tDMQw/9XFtTmi3+0i0Wn9p6hru/c8CXpoa7sC468BRbnoxNmBfpjhw5DhD//wZSza7uz5r4pMV+wDyDftA4jXEshs7D8TG57EOJn4HrjctXjk3/3MGY+dtTJhAJSysEUoL4zyty7fuZ+KSra7HXXG4H6VqVyJKwW2vzuLPE5cHIvySsa8kMvKbkR3NUBZhcd+YBUz0GA02HUxdsYN56/fw2HtLM90VjQ0tAHzQ+9exGbX88m+LCgPsuuvYkSVeboGZa2p16hMWb+H2V2fXGI3CXgHEw22j1LEqxaAnPuGmf3zpu02n72b3oaMcNsLkWu837DvvWtrYsbwgwa+pxl4RbHdi8GprCYqPl2ypCdoHtWoL+/2mO0KtJjF5JQD8Pn9hLLvs4RFKmxTXvP9jgt22dhJtBPPejrfKXoWKmwooOdwTwvzq7QU1762Hgxhn4t1BPZf7K64XvN0lGeKtwMLge/+Y4aiiNCcjpsNANq1KNBHySgD4JR0TFuts2W40tuPHO8nPCuB1j7Hsl2za56leYQrjj10YebXJVFuMEm6pNH31I86xZENBpGtczrZIFX6e24NHj/Pa9LVpWS38d+5GHvxv8vtPdx44SvnIcZ42XGYrWgDEIUzPkg7NGwBwkos6wYlLnvgk6vOXq2OX+rWhrL33xc2t0U49jyO7nwGyqlqxy2Ibcfvdj/z3fM9tvvj5as914+E2YF/VzzFwrfdgFGF7dgQoAZ6euJzykeOS8lwye+EnVPhD/13EPf+ez5QV4buQ/vi12Sk9K2Zujh+/NjugHqUfLQDiEFYaNoB2zUp8n7PP5np5zHC9tP7ACmqMwMkPMks27+VPH8WqowoTKblr+uB9AHp47CL6Pvwh+48cZ8f+I2zZl9wejLANq1acUodC4hl+uuIVBRms7k8fR56Doz4EgP3R8zMZMb3JDrhE4c0mrKFRwiLsdVCw0cKynB0HjjqGLHDjrVnh/YODXOL+ZFTtDGTrXiMlZJJtbdt3hMFPfgrArReeGHWsyOMKwI8AGDsvkvvg4NHjDHjEX3htK8u3Bpv2EZKwGXkc4MP+UQdpAzC/g2SElwQwGclmdjh49XnhjxOWsXnvIX73zd6ezwlr8pBXK4DlW/fH7CSNx9EEOvlkqapWNbtkX5yyikVxQkN7Ydaa2kQzK429Bn5sANbx4sevzbKURz90XmeWHVo08HztWo+R1B7woPXrYfzc0mYDCHAFkEysJfvMOBmPNBFh4cY9LN/qze6US/xhwle8Nt1fDumw1NF5JQAA3vFhsAkrz+35j02sef/58h1c8dSnSbc1c81ONjkY2fx03fr73HvouKU8uhGvg/RJbbzbNYIO5pYqz36yEkhylp4l9+C03yRpkvgiPrPlr0j2WRzy1GcMemKy/w7UIcJ+pPJCACQ7u/SjLvJDkIHFrnpmqmO532X34+8voXzkuKh7nppkLBc/iWDMwergkdQ8d4IK0Rym2s8kbI2IPUxHGOoxPyRzu27/zSkrtkftOq/rhJ6kKuT2c5ryVo0y3YWk8TvImN4QVlfUeev9BbxSSvGz1+cw2xYEzgvnPz4xcaU4ZMMKQinF1n2HXd1Q09VFuw1g3nr//w+TVFQPZi/8qIAS1fz2375g+HPTku5TrqJtABng8p7tMt2FpEl2J7B17HjLpstNNMhOWbGDt2dv8KVmC4p0ZgRz48vVuxjwyEfc9sqsuPXCDlx3QvNoD7NUhGMqqxXThhS0EThdK5pPvtoWSGTgbEYLgDiEZAJIC770ri4D0kpb8LpE48gHGQyHHMag6jZwJhrPJix2jnVkfp+vfxmumqlt02gBkIp6zH6rv313se+NT9UBaVLT7U10wwvTGfb052m9ZrrxkhO4RESmi8hcEVkoIg861LlTRBaJyDwR+UhEOhvlF4nIHMvrsIgMM479Q0RWWY5VBH97qZHJeDqp4mvZbam6ZkfyobBf/mJt0uemSjr/VclODMyZa9g6bPuAn0pSEfug+9zklZ43PiWjAqo51zF3RuRvOnc6b96b3ArgeFU1Px01m6Wbo72YXpq6OvVOBYiXfQBHgIFKqf0iUgR8JiLjlVJWRdxsoFIpdVBEbgUeA65VSk0EKgBEpCWwHLBGVLtLKTU6kDsJgVwWAH4mxIpgBtACgdSDMCRHOndjuq02+nRsHpMq00rYA9eNL05nkkO8nSCum8rz4c8LyL2y+XtMR07m+T7tX3aWbtnHf+ZsZMnmfbz30/Nryu8fk/HU51EkXAGoCKbSrch4KVudiUopc+o4DShzaOpqYLylXtpIVj2Qy+P/C5+v8lz3mUkrOOTDc8cNPzPN8fM3sS/F8NqZwm0AamVJUOPE3BQHlUQ4Df6Qmn3Ez0/gjRnRvu0z10b2p3hV3Sil+HRZxIXUeQWQPgEwdl5qdizzlpdszu59DJ5sACJSKCJzgK3Ah0qpL+JU/z4w3qF8OPCarewRQ230BxEpdjgHERkhIjNEZMa2bclFE/z3rA2JKzngJ4ZJthHGg/dVAuObn5nmra/MiomMmk0cOlrlOgEodgsFkaDNvp28pekMmiCMwF4mUb8cPS/qsxljyutKesLirXH33tQ0Y7uf5yav4N7/eI8V5YVs+eWHPQn1JACUUlVKqQoiM/sBItLTqZ6IXA9UAo/bytsDvQBrktx7gO5Af6AlcLfLtZ9TSlUqpSpLS0u9dDcGux7OK1VpXAKMM0IiZDOTv4ovgP3OzNYFlHozDOLt1XBNCJNAAoaRptMLYaieqj1MjsyB3+s8KpG+3fze7ffz23eX8PK0YO1Pfid/9r0vqaqPq6sV7y3YXCuIsiEjmFJqNzAJGGw/JiKDgF8BQ5VSdivXNcDbSqmaNb9SapOhXjoCvAgM8Nl3zyT7z/DykAfFqC8zZ0DV+OOwi59/v04t4p43J459IEySNQLHU91c9/d4SgDz/MjfZH5/Rx1SpaZTBeS3z3YBkKr24B9TVnPLyzNrJ10hDUVevIBKRaS58b4BMAhYYqvTF3iWyODv5AP3LWzqH2NVgESezmHAAofzAuGqfk4micTcmsCfO0isSdVzlYNH/dkRgrA7hMm/ZzurDq2JaOKx/8hxykeO4/nPIvYY6/dzrKqax95bwqSlW7n6mSmhxZ2C5AfMbftq53H28dDPLnHPg6ml3qY9sSswuwA4+d7x/HRUOMb/VAfwVM/fGGC0gHh48QJqD/xTRAqJCIw3lFJjReQhYIZS6h0iKp/GwJvGbGOtUmoogIiUAx2BT2ztviIipUQWN3OAW1K/HWe6lmb/jt5ctjckSyoup+nA7/9kmS1w2UwjSN8j4xZFDaYQSQ36l0kr+MukSEz51TsOcHLbJin01p1Mb5ELYh/Aym37+dPHy4Fam8bR49X8Z044mw7d/vfTPabbTDWOWLqGg4QCQCk1D+jrUH6/5f2gOOevBmIyaCilBnruZYos25LZWChesMf61+Qeh2wroBtemA5Efsx/NZKHmBy1rfjCHKQ9pnGIy84DR2lUnFz0eLcVwMSlW7npxS+Zes9A2jdrENda/cN/zWSZ4YQQ5mrJxN7nx99fwrqdh/hqi7M90X6LMxySNTlxvKqa+8Ys4LaLTqKsRUPX62eFDSBXeeTdxZnuQkIWb0otJLQm8/haMdgjrYYoAbzaAKav2lmjroqcWPu2aUlR0td3EwCjpkfsXjV7JzyqihKlTg0C+//y6Ykr4oY4sd9j84bxXYJNpq/ayWvT1/GLN6M94tK16zkvBIBGkw5S8/wITwLEs2SDjBYAACAASURBVAGMmr6WVUaIimuencrDYxfVHlTWt8nfm5tcNPcnOH1tMRPgNOmxRs9cz83//BK/gYDt6Vk7tmzoUtOGcV/TVu6MCtqXLhVQXguA9y079DSaVEnFjpOsq+bomesT7oKO1/bIf89n6J8/czw2e13iAWn9Lnc7jml7s85mlVI1+YVrMoY5nJspi9gv3pxr7EfwJwHsOv8ORkC+kqL4Q6x1k95THy2veR92wECTvBYAbrldNZpkSMWRK1FO45+/MZcPF22JKf/Fm3MTBmdLtBN43+Hj0TN/g/9YvKDc+hfP88vUaVvHxgfeWchJv4rsEzUFQDaGXPGbD9xcZR09Xs3bs9fX3PPhY9WUjxzH/32w1OU85/b0CiANZNo7QlO3sG6Uc8sJYPLl6l1Rn7/xlylx6781az0/eGlGUv1yG2SsM/Mo3b/Bym210WCTGaR7ntA05jovTV1T8742XLTvplMmUdIit70ebjRrELGR/PGjr/jZ63N5f0F0ZNx/GPk27Bx10TVpG4BGk8Occu97cY+nM2eCmxE40RjTsLh253Kj+sl5AEE8G4DRDw9tBJ3v4cH/1gZlcxps/eYBaGnEgdqyN+Luu/Ogt7ScVs+xCYu3oJSiulqlTSjmtQAI0rC0+tEhwTWmyUoSzeqzFftsd+u+yOCWaFZ/cpvafQnJ6KS9qniSme2Wjxzn+xwrKyyrm2uejaRVtX5Pfu055i2I7XMNLmON3UD/109W0vX/vRujcgtLW5HXAkCj8UMiPX228syk2j0IM9fsZMAjHzFmzoaEQ3rzhrWun17H6KueiVZl/WvaGhZudHZxtg9+CzYE5wo9b/3u+LtpLfdjV8cBrPEZp8ouIO1Cz20An78hOkLs60ZImC1J5iHwS14LgHT4E2vqDumIQeOFg0eP882/fM6SzbUDZrzAeserI4bI8pHjWGQMxtNX7Uw4qPcqa+a7b+buZ4gIjfv+4xw2Y9OeQzErhNdt4aRTYeifP+fsRz+OKVdKMXON8yYt6/fhd7OZ/bv0ajOZZAuwaAYUnGXLqx2WRiivBUA+hl/QJE+2CIAvVu5k1trdPDq+NiRXPCOyU79FEqt1Glt2/gb9Sznrdx+7q0sIL+/vS1PXcNUzU5lu89tftHFvinsd7CuA6ON7Dx+PiUp8vKo6JodEup+x5C07Gk2ekc5UhPFwiooZL83kjDWxKg6lEqt1rMZjNz19vDYOJ3CljOcF9Nr0tba6cZvyjJtgueKpT/n1lacl3a59wHf6vi57cnLUZ9Md1soqWx7u2gaT7lpc8noFkIXux5osJl6OgCB44oOlNaqcUdPXUj5ynGPWtGqbwTEZZq7Zxb0u6hmT6A1c/q9x8Gj8+FZvG/sMDhw9Hlfn/fTE5a7HgiSVJEqmNqFGrZWkdtlNKxHWxrC8FgCaus24n5wbaHtDnnLeMRsUT328vMbX/2+frgTiuyMmNSs2TlqyeR+jZ653rbb38DFufPFLr805UuIxAc6fPl7OgTjBEB9/33kTVTZhn/EHnUwqrMmqFgCaOkvQvuOQuvthIkzjY416JE7dsPTkANv3uauUrMTLY+HVvfPwsaq07XyNJ7DcNmV5oXZVJsbnYG/oqY+WBdqeSV4LALdl1S8uPdlXO528Bn7SpJUssdkmhbXrbgPp6iTyKRzxmITHbow8Xq0cM+TF+447tGjg6VoRtUfm9bHJ5g6HyF6C8pHjOGbofoL2MJy7fk/iSkmgjcAOnNG1la/6LRomHypXEx65KADMYdCMfT9n3W66tWkcWPu/GectNLpdAFz0v5PoXx6b8jKIPMfHqqrTtgJIlUShtTfsitiJciHHN+T5CsANv7mAc+TZzTvCUAGlQqP6/gfLX46eF6P/TUecGKdxzr5h6qWpq3l28krXNrx281iVSqgyScVAa+XAkdR2cyd6orLFVdgrCVcAIlICTAaKjfqjlVIP2OrcCdwMHAe2Ad9TSq0xjlUB842q1lSRXYBRQEtgFvAdpZS3ABoh43d/wGNX9w6pJ5pUMAN0ZQvJJmffnYEdyLcnCDENcP+YhXGP+/kVhSXTTJuNGarlrVnuhm834m2ys1OYLb7CHvGyAjgCDFRK9QEqgMEicqatzmygUinVGxgNPGY5dkgpVWG8hlrKfw/8QSnVDdgFfD/puwgYvz/U7u2ahtQTTSq0a1aSsM6PLjwxDT2J4OWpWrX9QIxHjD1BerKCxA9z1+1OXCkBfgb1216ZFcD1FIeOViWM9OmXdZacB4nCgeSaAPCSE1gBprtBkfFStjoTLR+nAdfHa1MiT/BA4NtG0T+BXwPPeOl0ULg9oDn2P9SkQDaq777/z2j3y1zdr+Inns1Ktw1QPqhWcOr979VE5gyDjXvi7wXJNQHgyQYgIoUiMgfYCnyolPoiTvXvA9YtbiUiMkNEponIMKOsFbBbKWVOddbjkDjeuPYI4/wZ27Ztc6oSOOmYYaWTn1/iz6spnygqzD4z2LSV0WEKclUA/GPK6rRez7SN7DwQsCbZmhoz0e7pYK8cOp6efqVUlVKqAigDBohIT6d6InI9UAk8binupJSqJDLbf1JETsT5e3L8apVSzymlKpVSlaWlpV666xm3f2YdG//58cXdOK29VlM5Udbcm6tiJrEbSMM2Au/2GMs+2/jtu0tcj42Zk7yL54s+BFmujR2+pj9Kqd3AJGCw/ZiIDAJ+BQxVSh2xnLPR+LvSOLcvsB1oLiKmCqoMSF+GDPOiLn7KOfY/9ERB9k10M4bpunjdGZ0oTpCzNRuwCoB4u3eD4skJ4Ww6CpsXPo/NamZyx6g5SbdrD+IWj72H44e/yDYSPv0iUioizY33DYBBwBJbnb7As0QG/62W8hYiUmy8bw2cAywy7AoTgauNqjcAY1K/HX+0cNEV5poU94LpEtmzQ26sBMLM1/zcd09n9aNDeOQbvaiXA5LR6pQ2c82uOqeirEtMX+UcajoIwlj5eXn62wMTRWQe8CURG8BYEXlIREyvnseBxsCbIjJHRN4xyk8FZojIXCID/qNKKTP79N3AnSKynIhN4PmA7kljoXXjiJB7eFhP+nZqzs3nds1wj7wRpt3ivG61qsQsNAE4kKNGgDpCWIHY/BJGQiIvXkDziKht7OX3W94Pcjl3CtDL5dhKYIDnnqaVujPDalIS8YWv6Nict390DuPn58YOxTDj3FgpTOMKYF+cgGfxSPcu2bq6wAg7jlPYhGH6yYn5jyZ57D/mYzmy537i0vR4fNUrzP7Rzu/O9FTJtd2sYdKiYRFVcQLepZMweqEFQBoI0y85EfYf89i5abe1J0XQm3ncqJcDftvpltlTVuxI7wWznI1xQnKnk50HvEVo9YMWAI54/8X9/ipHDVcNnVs1ZMxt56TaoaSxj285sgBIWQD84Lwunup1ad0opeukA7vx71/T1oR6vcWbgkvOnuvsOngsa4I9rt8VfEIiLQAc8KprW/3oEK7t3ylunUIROmYwXLR9BWBf3f/2G/EFWKZ4YGiPtFyndePitFwnFaxCe/fBo0z+Kj3qMU2EXQfTH4vJifJWwU9WtABwwEsMGa8M7N4GgAHlLX2dF5Rmwu4yaP3UunEx3z4jvgDLFIUijL7lLCbceUFS59clV8njlvyCQcTo0eQmYdhm8l4A3HROOQ1tYXob1Q8uTUJhkkbGVP7Zl5zW1tJO9DFrs51aRjbCVXRsnrBNU5ClCxGoLG/JSUnGwk9HyOR0Yb2VbNFHa9JPGO6oeS8AHriyB+d1ax1Vlg1DRyoC4G/frax5b89M5BQj/4azOyds05r4w2r3eOvWs5LpYkJSnetkY4yfZEkljIGm7hBGoLm68yvxSceWtWEgHvlGr6iZcTbMHpN1T3/hxsqoz3Z/eqd2v9G3jHZN46u9LjildvNUcb1aYXB6Z3+qLa+kutpt0TDa86rYZWdxLmiK/jMnNzy3NOGi9wEEyOS7Lqp537pxMfcOOa3mc6Pi5FRAr/3gTF79wRm+zpl6z0DH8kKXkenFG/u7tvX/rujORackp6r5Rj/HYKw1nGVJk5mOQTPVbF43nlMe9XlAl+AFlZlkRKPJVfJWANiNhFZ/8PpJqg/OOrEVZ3RxySfsMp61b+YckM5JBXRCsxLXdpoU12PE+Scmbfy88exy12NPXNOHji0bcrFhB8gFA6tXFVC2pY3UaNKJTgpvUG7xB09lpeU2nPgdZrqWNmLu+j3RbYi42gauOzOxHj/SD+fz27qogD6560I6G+5nT1/Xj+37jzBrbfieKDkgYzSatKJVQCFyhkVFkIoNIJmBq5XDTuFOLj6/Ts0vf+Ry7h58ireLpTCwlhQVUtaioecm7MZ1PyurdK0ytKDR5AortwcfH0sLAIOSokKKDJfNqjQbgZ2uNqC8RUyZiLNqqF5hQWgDZrIqki6tG/Hnb/WLKpv8y4t469az4553qpG4xmvY6qW/iUlNodHUSXQsoJD53rmR8AFuBliTLq0b8YtLncMVBzUQN20Qu/08IgCiyxLtZLVnArMnHE+EUwhaL7d49omtaNawiL9eXysE2jUroU9Zs7jnXd6zHUseHkz3drX9LsmBpC0aTS6if1kWRg7uzqrfXUG9BKqKp4b35faB3dLUq1ouObVdjArnf/+nd9xzGpdEm3mG9+/o65otGsUKot3G1vh4wscUEv06t7CVJ5YeJUXRG/Pm3H9pwnOSRWuANPmMFgAWRCRwVYqpQqnyGYXN3o8vfzWIXw05NaZeA9tgmYjGxf4CWzl9H5v2RIJSbd8fiU446FR319M2TaKNy27f7ojzu9KofiHDKuK7o8b0z6HFM7uGszchHk1L6qY/xeAe7TLdBY3BKW2bBN6ml5SQJSIyXUTmishCEXnQoc6dIrJIROaJyEci0tkorxCRqcZ580TkWss5/xCRVUYGsTkiUhHsrYVHMluyZ6zZ5d6eg83BPqyVNimmsEBiBrxEuwPtR4sCiH9v7pNoYgx6z32nMqaOmxmlwKW//++KU1n40GA6tfIfOO+9n54X9dn6HV1TWRb33KAEfrcQfpzZQBi7TzXJcUJzZ5fxVPCyAjgCDFRK9QEqgMEicqatzmygUinVGxgNPGaUHwS+q5TqQSSR/JNmfmGDu5RSFcYr+azNdYgfDzwJgDZNvEWp9PsDLfKZa9ep9dmGG+g+IwG226AeFPHGaBHo3LJRTJlJtzbpGZjTHSspXYT9v9VkloSjgYpg+h8VGS9lqzNRKXXQ+DgNKDPKv1JKLTPebwS2AqXUcZ64pg+VnWO9eNywfpl3XNyNV24+gzO6Om8osw+GfpOaW10xk/Uq8JJEJUiPhUSeSPbvJJ7A6GxbYQQ1vN16wYkBtZQe7h7c3VO9HEiYllF+dUWsWtbKdzzuz8kUnkYPESkUkTlEBvAPlVJfxKn+fWC8QxsDgPrACkvxI4Zq6A8ikrbA7B/+7PxQ2/9mvzIu8jAjNBO2W6lXWMA5J7V2qB3B/ntMpMGwD8TWHbL2U0deHjsoOLV/aY9ItNFOcfIcBOlJm2iVE5PzwGFY79amCd87pwv/vMlfGuoHPeYlyLWZ8mU92iauRHbfVzYkarHGFHPi/itPi3s803gSAEqpKqVUBZGZ/QAR6elUT0SuByqBx23l7YF/ATcppczwlPcA3YH+QEvgbpc2R4jIDBGZsW1bMIkwskVfe9M53rJWxcNvRiurDcDLGO00mJa1iAz8/ePkOLCH2LbiZ3UE8QWA4G0FUCCRH2N5a3d1kRM3nF2eFQNN0Hi1fWRzfuBsCEmSqA9B9dCvs4dXfOkPlFK7gUlE9PlRiMgg4FfAUKXUEUt5U2AccK9SapqlrU2GeukI8CLgODVTSj2nlKpUSlWWltYt7dH/JDBQeiHRgxFrBE7d8auycwseu7o3D329dnZcarNZ/NxlnwTEN4gnwmkwtt9jBx/GMi+DSCqD4LCKE5I+N0y83lEWLwCygkTPRjYIqXh48QIqNQ23ItIAGAQssdXpCzxLZPDfaimvD7wNvKSUetN2TnvjrwDDgAWp3Uqw/CegPL7x/v/mw5NIXZLICOrn3Pr13FVATuw97LQRTLimsmNU1FR72IeGASbVsTKsb6ybqLlCuHfIqfzlun48cKW/dJJ/ua5fjACD2uV9KmqQn13iLgjDYtCpidU7Xselvp38rdbSSXU2hG1P9PtLTzeSxst0sD0wUUTmAV8SsQGMFZGHRGSoUedxoDHwpuHS+Y5Rfg1wPnCjg7vnKyIyH5gPtAZ+E9RNBYGXLFmpYt/w5MZvhtVq3GJSPCZcgkYfL4pjBHb6PaWanD1o7H00926sfnQIN5/XlSt6tadBHPWTE1f0as8DDrpac0d4op3h8ciECuXvN8S65drxGuLD78bBdHJymjy84pF4BZCmjiRJwmmaUmoe0Neh/H7L+0Eu574MvOxyzDkQfh2hb6eIAOnvENMH4JWbz6BxgrwDw/t3ZNSX66J+rH4fqBivIZ9uHZ7dTI2BuXXj+rz/0/hG9p4dmrJgw15f/Qibr/U+gYHd23Da/e/XlG3cHUm/WBd94b0+R9mswvjJxd24/vl4/ijhk3gFnr3fH+idwJ745/cG8P+u8OY2Z3L2ia2Ze/+lDOweWY7bvS7sxkgn+hirELvrohcuMjJ4dS2Nvo5bZiw3/A5+Iy8/lVYJ4hMFuXIP8ufVsH69SM4Fg6NVEX+Fg0f9xU+yks1eNLlOQUHtZsRM4aQ6zCW0APBA7w7N+Ga/WoOt1wGsmcVgGW/J7RZ+enj/jrz30/Oi3EK9DidnGvsI7Ebi4nqFPPrNSE7fItseAnOyYk0P6cXn3y9ZoLp1xWnGtutgrB3ECWvgu5r2gAFxvKUyRdAT069nwNhd2rg44wm8e5wQP7hhUISREB60APCESPTAG4Re10sLIhIVFdMP11R25IwuLbn5vK6Ox2654ESeHB4dfcOMV/T1vrU/Zr9LWC+1k32UO7dqyA1nl0fN0oMmFXXP4J7tefY7p0eVFYjQMc5+iaBp29TbjDRo1US5S/6KMOnWtkkoodu9GNFT4XfGBCwbqJsRrFzo07E5R49XJ65oI2JotH4OsFP4GxDNa/fq0Iw3fniWa70WjerzusvxggJx3PRlfjcl9fz7HPu5h2QS7nz88wto1biYZg2KmHLPxZSPHAcE/79IdcFz1onRO7jTrQK+ttKb0TbobmXKTnLwaPBOCt85qzMTFm8JvF2Ti5MIGxJW6tK8EgDPXn867ZKYPdpXAMnQwpb1q0XD2s9ndW3FB4u8PXAdjU1Yw/p28O3tkog2xuwx3g7fRIQ14HUtbRxOwzasqzs3A348mpYUcUrbJizdsg9IvxvgD86PXfE54eX/5Mf2pE0duYlWAXnAb/gFJ+772ql8s1/Eh71Px+ZRg/d9X/O+XbxN0xKWPDyY751T7r8TCfhW/048f0NlTT/B+wB2/skRO4UXlVWgRuAE/wy/utPolV5yo9odg2pzRaTbC6RJibddy4lmlH3KmvHJXRd5vu6mPYejPjuFOQmCRJ5zfslIGG+Hrz5RBrxGAd+3SV4JgGR/iwV2FVAS87qG9etxvREYyn62X0+RkqLCUAaWggLh4lPbJtX2N/qWMe/Xl3LaCR4EQAAGLa9hrU1h4/WWrCuAZP3/Ky0rh2z1AhSBZ79zOj9326jms+P2fBfWZ6hRgCvVoL/Oy3u2D7jFxDiNH7dfFD/B1Ju3uKt7UyG/BECy5wkp+eKbuM18U0lCn0009Tj7DOJ237n9XO667JTUG7Jh1WUnq9e2JsHJ1lg6AlzWox0/vth54Jm7brev9uzflfXTgQD19LHRcIP/flPZ+OcFp+YTXdJvzC+v5JUASFYC2H/E3dokq4+OjHz2f3a1f7t02gjazgDBeO6d2r4pt110kuf6yaxqzFO+fUYn3+fWtAHcFIK6LmUCHuPsAiAIwXe6Q9DAIFe+b95yFiu27Y8pz4TM3n/Yfa9J+xC93vJKAHgNveCI8VA0Lq6XMGewx6ZqyIaYJm60bxZ8FiJzxRO0PjcIrAOX+b5+Cv/vApHQ9LfxsMdmshO0V4l9wG8eQATVNTsOxpTZB2fzl9Osgf/r9S9vWROY8KeD0pfj2+mbd0sZO/bH5zL2x+eG1pe8EgBeVRR2RIKZFZjJWxrb+pHNAiAMzLu99cJIEpVk3OLCwkkFlJKLYwAeZMnw5LUVfGuAu0to8O6ztQ0O798xypEA4JYkEuY0qB87PNm7bU4mfvuN1Hzr0zkZcVrFtGzkbDTv2aFZwp31qZBXAiBZgtLj9i5rxj2Xd+eJa/pElZsCwB62oc5iSIAw9LepYu2S+X9PRQBk6hZbNS7md9/s7Xo86G5Z73PUl+tqHB6cjnvFKdS5ffC867LIfha/i7RvGlFlr+wT2fRon4OFGeLB6auo7zNES1BoAeABcXnvux0RfnjBibS2SXRz9Re28SlbuOeKU2nRsKgmyXU23bbV5mEOWn7jJ1mxbyJ0wmv+5yAJ2ovM2txdl50Sm6UtKAFg+3zrhSey+tEhvu/niWsju+BN/bp9R/Hndw9k6W9i0p6ERiKVXVhoAeAB68MVhrLG1P9lq8dI0FxyWltm339pajaZkPjj8NrAt+b/o7MR5uD8k2sTEnldrXmZ+b6XIHqqV8pa+EiEE8gVa7E+u2UtGsRmaUviik4rL7efSND3U79eAcVJ7Ij3grMXkDA6JFfPeGgB4IGwl/FmjP62IVr7Nd5oawmEZw5A5r+/lUVP28SjztjLwNeyUf1A1GF+vEUCn2vY2rNPZiqT2FU9vL+T91WkXXvK0WRXNKYNIbWVvc/6WZQmJvvcMLIQ68MVxr/upDaNefzq3qEHocoUv7+ql+NDn479Dxd1b8Pvxi/hil7+N/yYG/TMoKlRxnrjmfjzt/uybd8R+6lR1eL94D+560IAPvr5BVzw+CTffTT54/AKzj3JuxrB2qeupY1Yue1A0teG6AFfRGIEgBmd1g+dHEJRmHLS/o2m+rtM6+Lbdq0w3TwToQVAlvA/HoN45SLXOs7k0sPJbZuw+tEhSZ1rDjbmYGb11DN/w1/rnXwY5GYNimrUS51bNaK0SXFcYWLnwaE9eOCdhQB8vSI2VWZcAh7w7AEEg2i+e7vYjF+uKqCAN2eGxc3ndokpm3rPxenthAUvOYFLRGS6iMwVkYUi8qBDnTtFZJGIzBORj0Sks+XYDSKyzHjdYCk/XUTmi8hyEXlKsj11jibvMB/IWgHgf7QQwXGzkRP3fe00X+6I55zkf1ZtEuSvbfWjQ6IyzYlD+8lcr3nDWNdIc+ViHy5S3W2cilrGz2PRuXWjrHJ68GIDOAIMVEr1ASqAwSJypq3ObKBSKdUbGA08BiAiLYEHgDOAAcADImIqA58BRgDdjFf6TO4ajQdM1zzzB5uMykoQ9h72llBmaJ8TWPDgZb5aTxarimaEQ84Iv9jDpdsH6OJ6hYywRCp1i+X01q1nx71OOtxq/Qr6k3xEBijI0L4QNxIKABXBnMIUGS9lqzNRKWVu25sGmOmzLiOSRH6nUmoX8CERAdIeaKqUmqoiv6qXgGGp344mN8mmn0QtpgAwB0sHE0BCwhyw3DYPecHareEDOvHiTf1T71ACbjy7vOb9sSrnQdYa/qHaYXesKVjsX2uy37NTL7yO/98aEFFtjhpxpmM2OJMzutRmhLPbRl4fYZ9LpxdPXkAiUigic4CtRAb0eJmYvw+MN953ANZZjq03yjoY7+3lTtceISIzRGTGtm3bvHQ3FMx/W3EWui5qwsHMcmUOLtaZodfxRkQ4MaRcBqnsXo1100yNKOHo0prf9VNBgXDpad4cI1L1rLF+H1772bdTJGd368bFnNutNEFt4zpEr47OSMI4HiSeBIBSqkopVUFkZj9ARHo61ROR64FK4HGzyKm5OOVO135OKVWplKosLfX2JYdBk5Iifjn4FF7/YWYldl0iWwNgDDPy29bOsGONwPUKvHlQC+4bvTKpC47dqBVcZ4K8L/sOWbe2kw0x7jTb96oCSuY2C8RdVJ3aviknNCtxD9EdAr72ASildgOTcNDXi8gg4FfAUKWU6cqwHrC6t5QBG43yMofyrOZHF54U2mxOkz2YA705SPbrHJnpffes2vAGTT0GHxPBdaS4e3BsWk6vlBQVpF2AeE3yMn3VTsdyrzaUoX1qPavs+yO83vN1KURw9SpLLu3RrvYUr3ojcb+HRsX1mHLPxZx5YvpWBV68gEpFpLnxvgEwCFhiq9MXeJbI4L/Vcuh94FIRaWEYfy8F3ldKbQL2iciZhvfPd4ExgdyRJhB+cnG3mJAV+cLN53WhQVEh5xh+9W2alLD60SGc162UXh2a1dTxgog4qicWPHhZjQ45Gfxk63IiGRXQb7/Ri9aNE29am79hT/IdA576Vt8a191C20rLVb1kG3+/YxHW8XBaOXhdTVhDhHhdEZ7lQeWTzogAXnrdHpgoIvOAL4nYAMaKyEMiMtSo8zjQGHhTROaIyDsASqmdwMPGeV8CDxllALcCfweWAyuotRtosoA7LzmZGfcOSsu1ssktDqB3WXMWPzzYMSCYuTu4yEf0Mev9mb7tqe78te5YTga3WD1N4qRIPLltE2bce0nC2EXmdzTE2Hz3A4/C0olkvqdZ913iKTUpWDPGCQONqLTJ7A1oUL8wYdau135wJh1bNkxor0gmtHWyJLQiKaXmAX0dyu+3vHcdKZRSLwAvOJTPABxtCRpNtuNHaFkH2399/wwWbNgTSByklMIXxHz23tqVFSfw7CcrXY+bg7ZppDbVpskMrIWF3lRA1qaT9Y7q2aEZHy/ZGpXRLR72vvQvb+lc0UC5JISyY8q8sLKARV0r9CtoNC7kYhqEZLps/b23blyfizKQ/+CBK0+L+pxKUqMrEuTRTSl/gr0tl9EyjEfnjou78c7t59CrrJmn+n49j7zWNw3y6cgTxCwmBwAAFA5JREFUogWAJuNkmQbIE376HL1JyvuZl57WNsq/3Z61yk9bieJMVXRqTpfWjfhlHMO0ORwluuxBY1dusp45VuLlGraSalwpMa7Vu6y593McOtMigExojYxAd93axIbCCBotADSakEnWR/2571bS1eJ15sfuYKcgway8cXE9Jv7iQip8DIBuzDTSLJqkYuPxaxC9so+/2Ex7DkV2abuF6+jY0j3EtlPPJv3iooTB3RLmh2hawqs/OIMnh1fErxgAWgBoMka/TqZ7ZXlmO+KHZGaaKQyAy7fWDkzmwFFSVOC72SC0MrWhk5NrrEUSuvl6LiEjUpnxj7y8dpUzYdEWAF75Yq1j3dM7+Qtj3axhESe3jZ6523vq5fs7+8TWaUlTqQWAJmO0aRpxrzw3Q9mQUsGP+iUoL6eaQGjJJFfx2Akv1bzej32MNge0Nk2Ka9QcifC6Ya3Gm8fh2AmWGfnqR4dE5SdOpGffsPuQ6zG3vnSzxQZ6cGgPzuraqmbncDZ5vWkBoNH4IFUjsF+sLpfmwGH/66kPlsrmQOSXeDYArzPyt249i7E/OZcij2k27SsXM4jc5S75HZz69j2HEMwmiXodzxPH69d/avumvDbizKzMgKcFgEaTBP6MwMmLgG37Y/MDJLNRyGpMtc6Ak+G09k257aLk2ji9c0vaNCnhgpO9hXWxG4Hr1ytgxr2D+N03eyU892u9I0Ii3vdv7nto29R5b4N9I5qVZHMTZNECQAsAjcYPjhPd5RPgk8dg7uuwdhrs3YhQXXM4lR+80/XM9vwIFus4muomMhHhrsu6x5SZfNsIw1CzYnD4Brzqt+2CQhBaNy72ZBA3N/LFW50M6BLZmfuzQc7xd+Jdxu37T/R/yabUJzojmEaTBFG/4VWfwudPRh1fWlLEuurW8K/nqd+0Iz8sPMJ6VQob2kHzcmjY0rcyuGYQT2L8sA46yRqEu3rcmGTPDpYKTUr8uVVax/pr+3fkxc9Xc1mPdvxm3GLH+qYNoKGLQPJqO/FD9gz/WgBoNKlzyYNw4UjYvQ52r4FdqynYuYZOu9bA3rXU2ziLe4oM18i/PRX5W78xNO8MLTpD806W951pxCEOEOt+WGsE9k9hgTDo1DZMWLw1JqG6V7zOXN2SvSR3zeTrd2/XNGE6UHN14CYUE7nP5jpaAMRh2j0X1/gJazQQZ3NTUQMoPTnyIvqHdeRYFZX3/ZuOso3xN3SGXWsMQRERFqz8BI7VJmVfWAI7VWN49iSeLqrPOtWG9aqUsh076SqH2UcSCe4F/u9/Kpi4dCsnxdlg5DTgvnP7Ocyy+fYDnH9yKddUlsWU+1HPJCImZEXA47G5YnCzqyRjb0lkEM8iDZAWAPFo16yEdgk2dWjyE7+umPtpyGLVGbo7zEiVgoM7Yfdq2LWGR197n46yjesaKrrLYgYVzKJYjsPcF7ncHDf/tx206MwfiuqxTpXCrJ2RlUSLztC0DAqjf9oFIjRrWI9hfX0mjycSHM9ph+xL3xvgWN/Nd9/KbRedRMcWDXl28gq+2uKeM9k+WAYdHaG6Zm+DM05eSG7ZzLyibQAaTY6S1D6wRL93EWjUKvLqcDp/rYqM8td9ZwgXjxyHUE0bdnPfuY34aMp0Ti7ewa0n1Yfda+hfsJihTIF3/mNprxCadYDmnfl9vULWq9bUW7AXWnWJqJoatwUX75b6Kew2NjGDwZW1iKixWjeJ3QBWVFjAVaeX8fGSrXEFQNhYo4E6YVcBNWtQxGU92rluHLO2mQtoAaDRJIEvH/wUzX6KArbQki3NT+Pt6mJaFtTn1mGXAHDuyHHU4zjLf9k7olbavTZKxXRR4Ve0kd0wZnRtg4XFht2hU43dwfx7UvPOuCft84YZaO72i06i5wnNuOgU9+B3j13dm2v7d+S7L0x3qRHMbPmZ6/qxZPO+mPLa5D/O53VoHmuL+c2wnvxmmHsg4xwa/7UA0Gj8EMoKIEWOUw9adom8bAwYOY5ijrL4ztMo2LM2YnMw7Q+718DGWXCoVr8vwPziBqxXreG1l6OEQ43AKI4fpMxcAdQrLGBQgpy+jYrrcb7HPQGpcHmv9o6bx2rCW7j8k9oZLrN9ypoxd/2euHVr20ylp+lFCwCNJmSCHv/9tneE+hS0OQXanOJc4fCeqJXD6HGfUCbbONXBQA1Ag5a2lUMnOu2sT1c5wgbVOuVkN+mkOoEX0AWnlHJlnxP43jnlfOMvU9LYs/SgBYBG44PGxs5RPzHv/Rr9pt4zkMPHqmPKa2ervppLTEkzaNcr8gIeHFMOwOofDTEM1DsM4bA6WsW0eT4sfReqjnIlcKVhoD44uQ0s6eqoYqJphxgDdTxMb6Ehvdozbv6m4JPCG3/d2i2uV8ifvtWXY1XVNCmpx31fO825YgB9yQQJ/xMiUgJMBoqN+qOVUg/Y6pwPPAn0BoYrpUYb5RcBf7BU7W4c/4+I/AO4ADATiN6olJqT2u1oNOHy52/15a1ZG2pSO3rB73jdvplzCOJa1UIaZ9gi0Kh15FV2euzx6mrYt4l/T5zCp9Nn0lG28Y2y43Qp2A5rp8KC0aAswsxioI4IhXJo0Zl+siGyUa66OspA3axBESt/ewXzNuxh3PxNibsbwC07UVRYwPxfX+aprteMYtmAF1F8BBiolNovIkXAZyIyXik1zVJnLXAj8AvriUqpiUAFgIi0JJL/9wNLlbtMYaHR5AJtmpZw64X+4uCkMmNv36yETXsOB9Ze4BQUQLMOnHj6Jdw5rSEAnfv2ocvpxv6AqmOwZ3203WGXYaxe9iHsj4Rj/rfp3vrbn0OzjlErh4LmnehS0oHm7OPayh6h3Eaqhnor/1NZxu/fW0LzAJLDhI2XnMAKMP20ioyXstVZDSAisevWWq4GxiulDibVU40mR0nF79u6qaomp6yP88fcdg5z1u1O+vpe6dOxdp9A1D6AwiJXAzUAxw7B7rXc+Ic36SjbePjsJrWCYv0MOBzpezNgTgnwcROYbTNKN+9M070lNOSw8zXSTH0j0qk9L0A24kkZJyKFwEzgJOBppdQXSVxrOPCErewREbkf+AgYqZSKCX0oIiOAEQCdOnVK4rIaTe7ipE+2ypNzT2rN8AEdXc/v07F51OCcDurFiaAZQ1EDKD2FSdV9AXj4UttGucN7alcM1lXEzpWwciIci8wnLwYWlcD+Zc3gua7RdgdT1dSsDIrCV880LSniX98fQO8O7t/7Fb3acYVLSOt04kkAKKWqgAoRaQ68LSI9lVILvF5ERNoDvYD3LcX3AJuB+sBzwN3AQw7Xfs44TmVlZe5YVzSagDnJlmgE4OWbz8hAT+ITZCwgSppB+96Rlx2Lgfr4jlVMnPYl55UehP3rogzUUTRpHyUczt1fxZGCEkr2t4eqlr4M1PE4r1t819a/XOdgT8kAvu5WKbVbRCYBgwHPAgC4BnhbKVUTWEcpZVp0jojIi9jsBxqNppa3bj2rxjgcpL46DLyEgggEi4G6XtnpXNLn6ujjhoE6auVgejGtmQLz3+RmVc3N9YGxj8C79SJeSjX7HmyriMZts8wAkzpevIBKgWPG4N8AGAT83ud1vkVkxm9tt71SapNEFKTD8CdQNJq8olWjYlo3LuaEZiU8MDQcQ6hJWYsG9EkhOXwyAdQAhvQOWCViGKhp1gE6nx17/PhR7nrhXTauXsJ95zSme8nOWkHx1ftwYGt0/XoltTuoY1RMnaFBi5wTEF5WAO2Bfxp2gALgDaXUWBF5CJihlHpHRPoDbwMtgCtF5EGlVA8AESkHOgKf2Np9xRAuAswBbgnihjSauohIxLg45Z6LQ7/WZ3cPTOl8P3skTBKFbQ6FevXZWu8EPq8uYtOJ/ene3Ray4uhB2LPO4r20unY1YTFQ11Dc1EU4GGXFsSq8TOPFC2ge0Neh/H7L+y+B2Liw1HgIxYQgVEql9pRpNHlEtqt9rPgyAmeY7UbKzaNVDg6M9RtC6SmRlxOHdscap3evjTFQ19CwlXNojebl0Lwj1PMWIjtI9E5gjSaLyaW4Mq0b12f7/qM0Ks6+5OduLNy4F4BpK3dwWY92/k5u0DzycjNQH9huWTlYBMWmubB4LFRbc41IxEDtkCAoEuK7AxQE/71qAaDR5AC5oFoubVLC9v1Hk1IBZYoBXVoyfdVOBtrVP6kiAo1LI6+yytjj1VWGgXqtbYPcGlj9Oex9g6jtVgX14IeToW2w9h8tADSaHCAXVgJV1RE1Si4JADNwXdpVbAWFkX0JzcpcDdTsXR8tHJq57/dIFi0ANJo0UVLkXzeeCzN/kyojuH4YidTDojYhTGb7EUO9+tCya+QV5mVCbV2j0QDw/A2VOREaIBVq8uvm0AqgJrxG7nQ5ULQA0GjSwMWnxk+MkohcCDFcVRNbP3dG05oVQA55WQVJ7vhraTR5TC4MUKYKKJcSwiTKB1DXyQsBEESia41GE5+2RvrE4no59HurWQHkJ3mhApp536Ca2YlGk0us23kIgGPV8SKtZwfPfud0Pl++nTZNcychSi6o1sIkLwRAk5LsT8yg0cTjiEOKyGyjdeNivl4Rs+k/q8lFw3WQ5NBaTaPJPzo0j0QAbVKSF3O1tHPJaRHjvPk95xv6qdJoNHnLiPO7MnxAJ5o1yE8tgV4BaDSavEVE8nbwBy0ANJqspmH93Amspsk9tApIo8liXrixP/+ZvYGyFvmpo9aEixYAGk0W07FlQ358cbdMd0NTR9EqII1Go8lTEgoAESkRkekiMldEForIgw51zheRWSJyXESuth2rEpE5xusdS3kXEflCRJaJyOsiUj+YW9JoNBqNF7ysAI4AA5VSfYAKYLCInGmrsxa4EXjV4fxDSqkK4zXUUv574A9KqW7ALuD7vnuv0Wg0mqRJKABUhP3GxyLjpWx1Vhu5gz1tVxQRAQYCo42ifwLDvHZao9FoNKnjyQYgIoUiMgfYCnyolPrCxzVKRGSGiEwTEXOQbwXsVkodNz6vxyFxvHHtEcb5M7Zt2+bjshqNRqOJhycBoJSqUkpVAGXAABHp6eManZRSlcC3gSdF5EScg+85RmVSSj2nlKpUSlWWlpb6uKxGo9Fo4uHLC0gptRuYBAz2cc5G4+9K49y+wHaguYiYbqhlwEY/fdFoNBpNanjxAioVkebG+wbAIGCJl8ZFpIWIFBvvWwPnAIuUUgqYCJgeQzcAY/x3X6PRaDTJIkrFj4ctIr2JGGkLiQiMN5RSD4nIQ8AMpdQ7ItIfeBtoARwGNiuleojI2cCzRIzDBcCTSqnnjXa7AqOAlsBs4Hql1JEEfdkGrEnyXlsTWXnkC/p+6zb6fus2Qd9vZ6VUjA49oQCoK4jIDMMWkRfo+63b6Put26TrfvVOYI1Go8lTtADQaDSaPCWfBMBzme5AmtH3W7fR91u3Scv95o0NQKPRaDTR5NMKQKPRaDQWtADQaDSaPCUvBICIDBaRpSKyXERGZro/XhGRF0Rkq4gssJS1FJEPjTDaH4pIC6NcROQp4x7niUg/yzk3GPWXicgNlvLTRWS+cc5TRpC+jCEiHUVkoogsNkKP32GU18l7dgu17hYqXUSKjc/LjePllrbuMcqXishllvKse/aN2GKzRWSs8bnO3q+IrDaetzkiMsMoy57nWSlVp19ENrCtALoC9YG5wGmZ7pfHvp8P9AMWWMoeA0Ya70cCvzfeXwGMJxJn6UzgC6O8JbDS+NvCeN/CODYdOMs4ZzxweYbvtz3Qz3jfBPgKOK2u3rPRh8bG+yLgC+M+3gCGG+V/BW413v8I+KvxfjjwuvH+NOO5Lga6GM97YbY++8CdRELHjzU+19n7BVYDrW1lWfM858MKYACwXCm1Uil1lMju469nuE+eUEpNBnbair9OZGc2RIfR/jrwkoowjUispfbAZUQiuO5USu0CPiSS06E90FQpNVVFnqSXyHBIbqXUJqXULOP9PmAxkSixdfKejX47hVp3C5Vu/R5GAxcbM76vA6OUUkeUUquA5USe+6x79kWkDBgC/N34HC80fM7frwtZ8zzngwDoAKyzfHYNPZ0jtFVKbYLIgAm0Mcrd7jNe+XqH8qzAWO73JTIrrrP3LLZQ60RmsG6h0mvuyzi+h0hodb/fQyZ5EvgltblD4oWGrwv3q4APRGSmiIwwyrLmec6HpPCeQ0/nOG736bc844hIY+At4KdKqb1x1Jo5f89KqSqgQiIBF98GTnWqZvz1e19OE7yM3a+IfA3YqpSaKSIXmsUOVevE/Rqco5TaKCJtgA9FJF4gzbQ/z/mwAlgPdLR8zvXQ01uMpR/G361Gudt9xisvcyjPKCJSRGTwf0Up9W+juE7fM0SFWj8T91DpNfdlHG9GREXo93vIFOcAQ0VkNRH1zEAiK4K6er+o2nD4W4kI+AFk0/OcSQNJOl5EVjkriRiLTMNQj0z3y0f/y4k2Aj9OtAHpMeP9EKINSNNVrQFpFRHjUQvjfUvj2JdGXdOAdEWG71WI6DGftJXXyXsGSoHmxvsGwKfA14A3iTaK/sh4fxvRRtE3jPc9iDaKriRiEM3aZx+4kFojcJ28X6AR0MTyfgqRXCpZ8zxn/EFI0z/iCiIeJSuAX2W6Pz76/RqwCThGRNp/n4gO9CNgmfHXfBAEeNq4x/lApaWd7xExlC0HbrKUVwILjHP+jLEzPIP3ey6RJew8YI7xuqKu3jPQm0go9HlGn+43yrsS8e5YbgyOxUZ5ifF5uXG8q6WtXxn3tBSLJ0i2PvtEC4A6eb/Gfc01XgvN/mTT86xDQWg0Gk2ekg82AI1Go9E4oAWARqPR5ClaAGg0Gk2eogWARqPR5ClaAGg0Gk2eogWARqPR5ClaAGg0Gk2e8v8BI8+3cOSiIw0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#hide\n",
    "learn.recorder.plot_loss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save(path_data/'GPT2_pt_5epoch_lr1e-4_v2')\n",
    "learn = learn.load(path_data/'GPT2_pt_5epoch_lr1e-4_v2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.5G\t/mnt/home/pierre/.fastai/data/ptwiki/GPT2_pt_5epoch_lr1e-4_v2.pth\r\n"
     ]
    }
   ],
   "source": [
    "# model size\n",
    "!du -hs {path_data}/'GPT2_pt_5epoch_lr1e-4_v2.pth'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model sharing and uploading in the Hugging Face model hub"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s see now how we can share our Portuguese GPT-2 on the Hugging Face [model hub](https://huggingface.co/models) (source: [Model sharing and uploading](https://huggingface.co/transformers/model_sharing.html))."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Basic steps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# directory with the name you want your model to have on the model hub\n",
    "lang = 'pt'\n",
    "name = f'{lang}wiki'\n",
    "path_to_awesome_name_you_picked = data_path/name/'gpt2-small-portuguese'\n",
    "path_to_awesome_name_you_picked.mkdir(exist_ok=True, parents=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fastai v2 learner with the fine-tuned Portuguese GPT-2 model\n",
    "learn = Learner(dls, model_en, loss_func=CrossEntropyLossFlat(),\n",
    "                splitter = splitter,\n",
    "                cbs=[DropOutput], \n",
    "                metrics=[accuracy, Perplexity()]).to_fp16()\n",
    "learn = learn.load(path_data/'GPT2_pt_5epoch_lr1e-4_v2')\n",
    "\n",
    "# get the model (we already have the tokenizer in tokenizer_pt)\n",
    "model_pt = learn.model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('/storage/data/ptwiki/gpt2-small-portuguese/vocab.json',\n",
       " '/storage/data/ptwiki/gpt2-small-portuguese/merges.txt',\n",
       " '/storage/data/ptwiki/gpt2-small-portuguese/special_tokens_map.json',\n",
       " '/storage/data/ptwiki/gpt2-small-portuguese/added_tokens.json')"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# save tokenizer and model\n",
    "model_pt.save_pretrained(f\"{str(path_to_awesome_name_you_picked)}\")\n",
    "tokenizer_pt.save_pretrained(f\"{str(path_to_awesome_name_you_picked)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Make your model work on all frameworks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we trained our model in PyTorch, we have to create a TensorFlow version with the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install --upgrade tensorflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, check that our model class exists in the other framework, that is try to import the same model by either adding or removing TF: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# source: https://huggingface.co/transformers/model_doc/gpt2.html#tfgpt2lmheadmodel\n",
    "from transformers import TFGPT2LMHeadModel\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "All PyTorch model weights were used when initializing TFGPT2LMHeadModel.\n",
      "\n",
      "Some weights or buffers of the PyTorch model TFGPT2LMHeadModel were not initialized from the TF 2.0 model and are newly initialized: ['transformer.h.7.attn.bias', 'transformer.h.2.attn.masked_bias', 'transformer.h.8.attn.masked_bias', 'transformer.h.1.attn.masked_bias', 'transformer.h.0.attn.bias', 'transformer.h.9.attn.masked_bias', 'transformer.h.5.attn.masked_bias', 'transformer.h.3.attn.masked_bias', 'lm_head.weight', 'transformer.h.6.attn.masked_bias', 'transformer.h.10.attn.masked_bias', 'transformer.h.8.attn.bias', 'transformer.h.1.attn.bias', 'transformer.h.6.attn.bias', 'transformer.h.10.attn.bias', 'transformer.h.11.attn.masked_bias', 'transformer.h.5.attn.bias', 'transformer.h.4.attn.bias', 'transformer.h.2.attn.bias', 'transformer.h.7.attn.masked_bias', 'transformer.h.11.attn.bias', 'transformer.h.3.attn.bias', 'transformer.h.9.attn.bias', 'transformer.h.4.attn.masked_bias', 'transformer.h.0.attn.masked_bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "# save TensorFlow model\n",
    "tf_model = TFGPT2LMHeadModel.from_pretrained(f\"{str(path_to_awesome_name_you_picked)}\", from_pt=True)\n",
    "tf_model.save_pretrained(f\"{str(path_to_awesome_name_you_picked)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check the directory before uploading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- a `config.json` file, which saves the configuration of your model ;\n",
    "- a `pytorch_model.bin` file, which is the PyTorch checkpoint (unless you can’t have it for some reason) ;\n",
    "- a `tf_model.h5` file, which is the TensorFlow checkpoint (unless you can’t have it for some reason) ;\n",
    "- a `special_tokens_map.json`, which is part of your tokenizer save;\n",
    "- a `tokenizer_config.json`, which is part of your tokenizer save;\n",
    "- a `vocab.txt`, which is the vocabulary of your tokenizer, part of your tokenizer save;\n",
    "-  maybe a `added_tokens.json`, which is part of your tokenizer save."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/storage/data/ptwiki/gpt2-small-portuguese/tf_model.h5\n",
      "/storage/data/ptwiki/gpt2-small-portuguese/config.json\n",
      "/storage/data/ptwiki/gpt2-small-portuguese/vocab.json\n",
      "/storage/data/ptwiki/gpt2-small-portuguese/pytorch_model.bin\n",
      "/storage/data/ptwiki/gpt2-small-portuguese/tokenizer_config.json\n",
      "/storage/data/ptwiki/gpt2-small-portuguese/special_tokens_map.json\n",
      "/storage/data/ptwiki/gpt2-small-portuguese/merges.txt\n"
     ]
    }
   ],
   "source": [
    "for fn in path_to_awesome_name_you_picked.ls():\n",
    "    print(fn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "475M\t./tf_model.h5\r\n",
      "4.0K\t./config.json\r\n",
      "832K\t./vocab.json\r\n",
      "487M\t./pytorch_model.bin\r\n",
      "4.0K\t./tokenizer_config.json\r\n",
      "4.0K\t./special_tokens_map.json\r\n",
      "500K\t./merges.txt\r\n",
      "963M\t.\r\n"
     ]
    }
   ],
   "source": [
    "!du -ah"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload your model with the CLI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now go in a terminal and run the following command. It should be in the virtual enviromnent where you installed 🤗 Transformers, since that command transformers-cli comes from the library.\n",
    "\n",
    "`transformers-cli login`\n",
    "\n",
    "Then log in using the same credentials as on huggingface.co. To upload your model, just type\n",
    "\n",
    "`cd storage/data/ptwiki\n",
    "transformers-cli upload '/storage/data/ptwiki/gpt2-small-portuguese'`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This uploads the model to our personal account. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "About to upload file /storage/data/ptwiki/gpt2-small-portuguese/tf_model.h5 to S3 under filename gpt2-small-portuguese/tf_model.h5 and namespace pierreguillou\n",
    "About to upload file /storage/data/ptwiki/gpt2-small-portuguese/config.json to S3 under filename gpt2-small-portuguese/config.json and namespace pierreguillou\n",
    "About to upload file /storage/data/ptwiki/gpt2-small-portuguese/vocab.json to S3 under filename gpt2-small-portuguese/vocab.json and namespace pierreguillou\n",
    "About to upload file /storage/data/ptwiki/gpt2-small-portuguese/pytorch_model.bin to S3 under filename gpt2-small-portuguese/pytorch_model.bin and namespace pierreguillou\n",
    "About to upload file /storage/data/ptwiki/gpt2-small-portuguese/tokenizer_config.json to S3 under filename gpt2-small-portuguese/tokenizer_config.json and namespace pierreguillou\n",
    "About to upload file /storage/data/ptwiki/gpt2-small-portuguese/special_tokens_map.json to S3 under filename gpt2-small-portuguese/special_tokens_map.json and namespace pierreguillou\n",
    "About to upload file /storage/data/ptwiki/gpt2-small-portuguese/merges.txt to S3 under filename gpt2-small-portuguese/merges.txt and namespace pierreguillou\n",
    "\n",
    "Proceed? [Y/n] Y\n",
    "\n",
    "Uploading... This might take a while if files are large\n",
    "\n",
    "Your file now lives at:\n",
    "https://s3.amazonaws.com/models.huggingface.co/bert/pierreguillou/gpt2-small-portuguese/tf_model.h5\n",
    "Your file now lives at:\n",
    "https://s3.amazonaws.com/models.huggingface.co/bert/pierreguillou/gpt2-small-portuguese/config.json\n",
    "Your file now lives at:\n",
    "https://s3.amazonaws.com/models.huggingface.co/bert/pierreguillou/gpt2-small-portuguese/vocab.json\n",
    "Your file now lives at:\n",
    "https://s3.amazonaws.com/models.huggingface.co/bert/pierreguillou/gpt2-small-portuguese/pytorch_model.bin\n",
    "Your file now lives at:\n",
    "https://s3.amazonaws.com/models.huggingface.co/bert/pierreguillou/gpt2-small-portuguese/tokenizer_config.json\n",
    "Your file now lives at:\n",
    "https://s3.amazonaws.com/models.huggingface.co/bert/pierreguillou/gpt2-small-portuguese/special_tokens_map.json\n",
    "Your file now lives at:\n",
    "https://s3.amazonaws.com/models.huggingface.co/bert/pierreguillou/gpt2-small-portuguese/merges.txt\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our model will then be accessible through its identifier, which is, as we saw above, `pierreguillou/gpt2-small-portuguese`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add a model card"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just click on the “Create a model card on GitHub” button on the model page, it will get you directly to the right location."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using your model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Your model now has a page on huggingface.co/models 🔥\n",
    "\n",
    "Anyone can load it from code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(\"namespace/awesome-name-you-picked\")\n",
    "model = AutoModel.from_pretrained(\"namespace/awesome-name-you-picked\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our case, the code is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer, AutoModelWithLMHead\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"pierreguillou/gpt2-small-portuguese\")\n",
    "model = AutoModelWithLMHead.from_pretrained(\"pierreguillou/gpt2-small-portuguese\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check our [Hugging face model page](https://huggingface.co/pierreguillou/gpt2-small-portuguese) to get more information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text Generation by our Portuguese GPT-2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a GPT-2 in Portuguese, we can use it for different tasks in NLP (Text Generation, Reading Comprehension, Translation, Summary) as showed in the post \"**GPT-2 use cases: beyond Text Generation**\".\n",
    "\n",
    "For now, let's use it to generate new texts, which allows us to check that it works properly and also have a little fun."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text Generation techniques "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**At each stage of text generation, GPT-2 provides a vector of 50.257 probabilities** (each corresponds to a possible token of the vocabulary whose size is 50.257). To decide how to choose the output token from these probabilities, **there are at least 5 methods: Greedy, Beam Search, Sampling with temperature, Top-k sampling and Top-p (nucleus) sampling**.\n",
    "\n",
    "In this tutorial, we will test only 2 of these text generation methods: **Top-k sampling** and **Top-p (nucleus) sampling**.\n",
    "\n",
    "Note: to get more information on text generation techniques for transformer-based language model, read the article \"[How to generate text: using different decoding methods for language generation with Transformers](https://huggingface.co/blog/how-to-generate)\" from [Patrick von Platen](https://huggingface.co/patrickvonplaten) (Hugging Face, 03/18/2020)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (Use case 1) Top-k sampling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our use case 1 follows the same method used by OpenAI in page 20 of the paper [Language Models are Unsupervised Multitask Learners](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) by choosing **Top-k sampling** text generation technique with a **value of 40**.\n",
    "\n",
    "This text generation method is implemented in the `model.generate()` function of a Transformers model thanks to the following arguments:\n",
    "- `top_k` (int): the number of highest probability vocabulary tokens to keep for top-k-filtering. Between 1 and infinity. Default to 50."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### (Use case 2) Top-p (nucleus) sampling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our use case 2 follows the [top-p (nucleus) sampling](https://huggingface.co/blog/how-to-generate#top-p-nucleus-sampling) method with Top-p sampling (top_p = 0.95), top-k sampling (top_k = 50), temperature (temperature = 0.7) and repetition penalty (repetition_penalty = 1.2).\n",
    "\n",
    "This text generation method is implemented in the `model.generate()` function of a Transformers model thanks to the following arguments:\n",
    "- `top_p` (float): the cumulative probability of parameter highest probability vocabulary tokens to keep for nucleus sampling. Must be between 0 and 1. Default to 1.\n",
    "- `top_k` (int): the number of highest probability vocabulary tokens to keep for top-k-filtering. Between 1 and infinity. Default to 50.\n",
    "- `temperature` (float): the value used to module the next token probabilities. Must be strictly positive. Default to 1.0.\n",
    "- `repetition_penalty` (float): the parameter for repetition penalty. Between 1.0 and infinity. 1.0 means no penalty. Default to 1.0."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text n°1 | Famous OpenAI generated text about unicorns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the time of publication of GPT-2 in the article \"[Better Language Models and Their Implications](https://openai.com/blog/better-language-models/)\" (02/14/2019), the media retained from its different possibilities in NLP that of text generation because of the now famous text generated on unicorns from this small paragraph:\n",
    "\n",
    ">In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this text, GPT-2 wrote the following [sample](https://openai.com/blog/better-language-models/#samples) out of 10 tries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English. \n",
      "\n",
      "The scientist named the population, after their distinctive horn, Ovid's Unicorn. These four-horned, silver-white unicorns were previously unknown to science. \n",
      "\n",
      "Now, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved. \n",
      "\n",
      "Dr. Jorge Pérez, an evolutionary biologist from the University of La Paz, and several companions, were exploring the Andes Mountains when they found a small valley, with no other animals or humans. Pérez noticed that the valley had what appeared to be a natural fountain, surrounded by two peaks of rock and silver snow. \n",
      "\n",
      "Pérez and the others then ventured further into the valley. “By the time we reached the top of one peak, the water looked blue, with some crystals on top,” said Pérez. \n",
      "\n",
      "Pérez and his friends were astonished to see the unicorn herd. These creatures could be seen from the air without having to move too much to see them – they were so close they could touch their horns. \n",
      "\n",
      "While examining these bizarre creatures the scientists discovered that the creatures also spoke some fairly regular English. Pérez stated, “We can see, for example, that they have a common ‘language,’ something like a dialect or dialectic.” \n",
      "\n",
      "Dr. Pérez believes that the unicorns may have originated in Argentina, where the animals were believed to be descendants of a lost race of people who lived there before the arrival of humans in those parts of South America. \n",
      "\n",
      "While their origins are still unclear, some believe that perhaps the creatures were created when a human and a unicorn met each other in a time before human civilization. According to Pérez, “In South America, such incidents seem to be quite common.” \n",
      "\n",
      "However, Pérez also pointed out that it is likely that the only way of knowing for sure if unicorns are indeed the descendants of a lost alien race is through DNA. “But they seem to be able to communicate in English quite well, which I believe is a sign of evolution, or at least a change in social organization,” said the scientist.\n"
     ]
    }
   ],
   "source": [
    "openai_generated_text_en = \"In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English. \\\n",
    "\\n\\nThe scientist named the population, after their distinctive horn, Ovid's Unicorn. These four-horned, silver-white unicorns were previously unknown to science. \\\n",
    "\\n\\nNow, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved. \\\n",
    "\\n\\nDr. Jorge Pérez, an evolutionary biologist from the University of La Paz, and several companions, were exploring the Andes Mountains when they found a small valley, with no other animals or humans. Pérez noticed that the valley had what appeared to be a natural fountain, surrounded by two peaks of rock and silver snow. \\\n",
    "\\n\\nPérez and the others then ventured further into the valley. “By the time we reached the top of one peak, the water looked blue, with some crystals on top,” said Pérez. \\\n",
    "\\n\\nPérez and his friends were astonished to see the unicorn herd. These creatures could be seen from the air without having to move too much to see them – they were so close they could touch their horns. \\\n",
    "\\n\\nWhile examining these bizarre creatures the scientists discovered that the creatures also spoke some fairly regular English. Pérez stated, “We can see, for example, that they have a common ‘language,’ something like a dialect or dialectic.” \\\n",
    "\\n\\nDr. Pérez believes that the unicorns may have originated in Argentina, where the animals were believed to be descendants of a lost race of people who lived there before the arrival of humans in those parts of South America. \\\n",
    "\\n\\nWhile their origins are still unclear, some believe that perhaps the creatures were created when a human and a unicorn met each other in a time before human civilization. According to Pérez, “In South America, such incidents seem to be quite common.” \\\n",
    "\\n\\nHowever, Pérez also pointed out that it is likely that the only way of knowing for sure if unicorns are indeed the descendants of a lost alien race is through DNA. “But they seem to be able to communicate in English quite well, which I believe is a sign of evolution, or at least a change in social organization,” said the scientist.\"\n",
    "\n",
    "print(openai_generated_text_en)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Reload Portuguese GPT-2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### English pre-trained GPT-2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of GPT2LMHeadModel were not initialized from the model checkpoint at gpt2 and are newly initialized: ['h.0.attn.masked_bias', 'h.1.attn.masked_bias', 'h.2.attn.masked_bias', 'h.3.attn.masked_bias', 'h.4.attn.masked_bias', 'h.5.attn.masked_bias', 'h.6.attn.masked_bias', 'h.7.attn.masked_bias', 'h.8.attn.masked_bias', 'h.9.attn.masked_bias', 'h.10.attn.masked_bias', 'h.11.attn.masked_bias', 'lm_head.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 14.4 s, sys: 1.8 s, total: 16.2 s\n",
      "Wall time: 7.72 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "from transformers import GPT2TokenizerFast, GPT2LMHeadModel\n",
    "\n",
    "pretrained_weights = 'gpt2'\n",
    "tokenizer_en = GPT2TokenizerFast.from_pretrained(pretrained_weights)\n",
    "model_en = GPT2LMHeadModel.from_pretrained(pretrained_weights)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Portuguese tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import GPT2TokenizerFast\n",
    "\n",
    "# Get the path to ByteLevelBPE_tokenizer_pt config files\n",
    "ByteLevelBPE_tokenizer_pt_rep = 'ByteLevelBPE_tokenizer_pt'\n",
    "path_to_ByteLevelBPE_tokenizer_pt_rep = path_data/ByteLevelBPE_tokenizer_pt_rep\n",
    "\n",
    "# import the pre-trained GPT2TokenizerFast tokenizer with the tokenizer_pt config files\n",
    "tokenizer_pt = GPT2TokenizerFast.from_pretrained(\n",
    "    str(path_to_ByteLevelBPE_tokenizer_pt_rep), \n",
    "    pad_token='<|endoftext|>')\n",
    "\n",
    "# Get sequence length max of 1024\n",
    "tokenizer_pt.model_max_length = 1024"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "class TransformersTokenizer(Transform):\n",
    "    def __init__(self, tokenizer): self.tokenizer = tokenizer\n",
    "    def encodes(self, x): \n",
    "        toks = self.tokenizer.tokenize(x)\n",
    "        return tensor(self.tokenizer.convert_tokens_to_ids(toks))\n",
    "    def decodes(self, x): return TitledStr(self.tokenizer.decode(x.cpu().numpy()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get data\n",
    "lang = 'pt'\n",
    "fname = f'all_texts_{lang}wiki.csv'\n",
    "df = pd.read_csv(path_data/fname)\n",
    "\n",
    "# load idxs train and valid\n",
    "idxs_train = torch.load(path_data/'idxs_train.pt')\n",
    "idxs_val = torch.load(path_data/'idxs_val.pt')\n",
    "\n",
    "all_texts = np.concatenate([df.iloc[idxs_train].text.values, df.iloc[idxs_val].text.values])\n",
    "splits = [list(idxs_train), list(idxs_val)]\n",
    "tls = TfmdLists(all_texts, TransformersTokenizer(tokenizer_pt), splits=splits, dl_type=LMDataLoader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Dataloaders"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 27min 56s, sys: 3min 20s, total: 31min 16s\n",
      "Wall time: 25min 8s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "bs,sl = 8,1024\n",
    "dls = tls.dataloaders(bs=bs, seq_len=sl)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Split a GPT2 model in 4 groups for differential learning rates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DropOutput(Callback):\n",
    "    def after_pred(self): self.learn.pred = self.pred[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def splitter(model):\n",
    "    \"Split a GPT2 `model` in 3 groups for differential learning rates.\"\n",
    "    \n",
    "    # First layers group : decoder blocks from 0 to 3\n",
    "    modules = []\n",
    "    for i in range(4): modules.append(model.transformer.h[i])\n",
    "    groups = [nn.Sequential(*modules)]\n",
    "\n",
    "    # Second layers group : decoder blocks from 4 to 7\n",
    "    modules = []\n",
    "    for i in range(4,8,1): modules.append(model.transformer.h[i])\n",
    "    groups = L(groups + [nn.Sequential(*modules)])\n",
    "\n",
    "    # Third layers group : decoder blocks from 8 to 11\n",
    "    modules = []\n",
    "    for i in range(8,12,1): modules.append(model.transformer.h[i])\n",
    "    groups = L(groups + [nn.Sequential(*modules)])\n",
    "    \n",
    "    # Fourth layers group : embeddings matrices wte and wpe + LayerNorm at the model output\n",
    "    groups = L(groups + [nn.Sequential(model.transformer.wte,model.transformer.wpe,model.transformer.ln_f)])\n",
    "    \n",
    "    return groups.map(params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Learner"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Learner: basic class for handling the training loop\n",
    "# source: https://dev.fast.ai/learner#Learner\n",
    "learn = Learner(dls, model_en, loss_func=CrossEntropyLossFlat(),\n",
    "                splitter = splitter,\n",
    "                cbs=[DropOutput], \n",
    "                metrics=[accuracy, Perplexity()]).to_fp16()\n",
    "learn = learn.load(path_data/'GPT2_pt_5epoch_lr1e-4_v2')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# model\n",
    "model_pt = learn.model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# put model into eval mode and on GPU\n",
    "model_pt.eval();\n",
    "model_pt.to('cuda');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get translated famous unicorn text in Portuguese"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import MarianMTModel, MarianTokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['>>fr<<', '>>es<<', '>>it<<', '>>pt<<', '>>pt_br<<', '>>ro<<', '>>ca<<', '>>gl<<', '>>pt_BR<<', '>>la<<', '>>wa<<', '>>fur<<', '>>oc<<', '>>fr_CA<<', '>>sc<<', '>>es_ES<<', '>>es_MX<<', '>>es_AR<<', '>>es_PR<<', '>>es_UY<<', '>>es_CL<<', '>>es_CO<<', '>>es_CR<<', '>>es_GT<<', '>>es_HN<<', '>>es_NI<<', '>>es_PA<<', '>>es_PE<<', '>>es_VE<<', '>>es_DO<<', '>>es_EC<<', '>>es_SV<<', '>>an<<', '>>pt_PT<<', '>>frp<<', '>>lad<<', '>>vec<<', '>>fr_FR<<', '>>co<<', '>>it_IT<<', '>>lld<<', '>>lij<<', '>>lmo<<', '>>nap<<', '>>rm<<', '>>scn<<', '>>mwl<<']\n"
     ]
    }
   ],
   "source": [
    "# MarianMT Translator\n",
    "model_name = 'Helsinki-NLP/opus-mt-en-ROMANCE'\n",
    "tokenizer_en_pt = MarianTokenizer.from_pretrained(model_name)\n",
    "print(tokenizer_en_pt.supported_language_codes)\n",
    "model_en_pt = MarianMTModel.from_pretrained(model_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['>>pt_BR<< In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.']\n"
     ]
    }
   ],
   "source": [
    "src_text = [\n",
    "    '>>pt_BR<< In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.',\n",
    "]\n",
    "print(src_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Num achado chocante, o cientista descobriu uma manada de unicórnios vivendo num vale remoto, anteriormente inexplorado, nas Montanhas dos Andes. Ainda mais surpreendente para os pesquisadores foi o fato de que os unicórnios falavam inglês perfeito.'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "translated = model_en_pt.generate(**tokenizer_en_pt.prepare_translation_batch(src_text))\n",
    "tgt_text = [tokenizer_en_pt.decode(t, skip_special_tokens=True) for t in translated]\n",
    "prompt = tgt_text[0]\n",
    "prompt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Get generated text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "504\n"
     ]
    }
   ],
   "source": [
    "# Get the number of tokens of the OpenAI English generated text\n",
    "openai_generated_text_en = \"In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English. \\\n",
    "\\n\\nThe scientist named the population, after their distinctive horn, Ovid's Unicorn. These four-horned, silver-white unicorns were previously unknown to science. \\\n",
    "\\n\\nNow, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved. \\\n",
    "\\n\\nDr. Jorge Pérez, an evolutionary biologist from the University of La Paz, and several companions, were exploring the Andes Mountains when they found a small valley, with no other animals or humans. Pérez noticed that the valley had what appeared to be a natural fountain, surrounded by two peaks of rock and silver snow. \\\n",
    "\\n\\nPérez and the others then ventured further into the valley. “By the time we reached the top of one peak, the water looked blue, with some crystals on top,” said Pérez. \\\n",
    "\\n\\nPérez and his friends were astonished to see the unicorn herd. These creatures could be seen from the air without having to move too much to see them – they were so close they could touch their horns. \\\n",
    "\\n\\nWhile examining these bizarre creatures the scientists discovered that the creatures also spoke some fairly regular English. Pérez stated, “We can see, for example, that they have a common ‘language,’ something like a dialect or dialectic.” \\\n",
    "\\n\\nDr. Pérez believes that the unicorns may have originated in Argentina, where the animals were believed to be descendants of a lost race of people who lived there before the arrival of humans in those parts of South America. \\\n",
    "\\n\\nWhile their origins are still unclear, some believe that perhaps the creatures were created when a human and a unicorn met each other in a time before human civilization. According to Pérez, “In South America, such incidents seem to be quite common.” \\\n",
    "\\n\\nHowever, Pérez also pointed out that it is likely that the only way of knowing for sure if unicorns are indeed the descendants of a lost alien race is through DNA. “But they seem to be able to communicate in English quite well, which I believe is a sign of evolution, or at least a change in social organization,” said the scientist.\"\n",
    "\n",
    "openai_generated_text_en_ids = tokenizer_en.encode(openai_generated_text_en, return_tensors='pt').to('cuda')\n",
    "max_length = openai_generated_text_en_ids.shape[1]\n",
    "\n",
    "print(max_length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# encode\n",
    "input_ids = tokenizer_pt.encode(prompt, return_tensors='pt').to('cuda')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Use case 1 (Top-k sampling)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ">> Generated text 1\n",
      "\n",
      "Num achado chocante, o cientista descobriu uma manada de unicórnios vivendo num vale remoto, anteriormente inexplorado, nas Montanhas dos Andes. Ainda mais surpreendente para os pesquisadores foi o fato de que os unicórnios falavam inglês perfeito. Um deles, um escocês chamado Thomas (um anatomista irlandês) que havia trabalhado com o britânico em seu trabalho como assistente do diretor experimental da universidade, John Holley, foi chamado de por eles para um encontro sobre os dois conceitos, que não tinham encontrado muito trabalho até o momento. Eles foram palestrantes, mas ficaram impressionados que Thomas (como todos os outros cientistas da época) não sabia falar Inglês. Quando Thomas fez isso com os unicórnios por engano, Holley a escreveu e enviou um telegrama à BBC perguntando: \"\"Tão, o que é tal dizer se você quer saber de nenhum outro, eu não sei como ele fala inglês?\"\" Eles disseram a ele: \"Não, mas é um idiota.\" Então, uma semana depois, no mesmo dia, Holley o enviou, uma carta com um questionário para os membros do elenco de \"\"Doctor Who\"\". Thomas, como qualquer um dos cientistas mencionados, não se lembra muito bem dos eventos que aconteceram em Londres.\n",
      "\n",
      "Depois do segundo episódio, o editor-produtor Michael Gambon deu o que ele achou que valia, então disse: \"Vamos fazer um comentário, para o elenco e equipe deDoctor Who.\" Como resultado, eles deram permissão para que eles assistissem outro episódio. Isto foi o suficiente para que \"The Magician\" aparecesse para o cânone da BBC.\n",
      "\n",
      "Na segunda temporada, \"The Magician\" teve um episódio chamado \"The End of the World\", e o ator John Huston, que interpreta John Watson, era o mais conhecido ator inglês do programa. \"The End of the World\" foi ao ar em 13 de outubro de 2000. A BBC anunciou que \"The Magician\" seria exibida ao vivo em janeiro de 2003 da HBO em 15 de janeiro de 2003. Ele também foi transmitido na televisão em 16 de janeiro de 2004 da BBC One, em uma transmissão que também marcou o cancelamento de um episódio.\n",
      "\n",
      "A série é uma produção da BBC-NBC Television Limited. Ela foi produzida por Paul Rudd, John Swarns e Jonathan Seagal e foi criada por David Tennant, Graham Hill, Colin Smith, Tom Baker, Jack Davies e Paul Thompson. O primeiro episódio foi exibido em 17 de outubro de\n",
      "\n",
      "---\n",
      ">> Generated text 2\n",
      "\n",
      "Num achado chocante, o cientista descobriu uma manada de unicórnios vivendo num vale remoto, anteriormente inexplorado, nas Montanhas dos Andes. Ainda mais surpreendente para os pesquisadores foi o fato de que os unicórnios falavam inglês perfeito. \"Não é mais estranho que a nossa forma tivesse o inglês com dois de suas asas como se o macho fosse inglês — o que é interessante. Mas a sua natureza inata seria estranha para o inglês\", acredita eles.\n",
      "\n",
      "Em 2015, cientistas realizaram uma nova análise sobre as formas dos unicórnios. De acordo com especialistas na área, os membros superiores do grupo foram provavelmente derivados de outra espécie de escorpião — uma espécie com características semelhantes. Uma nova equipe de cientistas calculou que uma fêmea unicornada da Eurásia seria originalmente uma humana. \"Isto significa que o ancestral do unicórnio, um híbrido de um esquilo e um escorpião macho não nasceu.\n",
      "\n",
      "Um estudo recente estimou que cerca de 12% do corpo humano é composto por membros de qualquer um dos grupos mais diverso de animais extintos, incluindo o ser humano e o unicórnio-do-sul. \"A análise dos dados mostra que a maioria dos membros do gênero é composta por um exito e um exito macho que compartilham uma única espécie de corpo. Em alguns casos, estes membros compartilham um mesmo conjunto (a linhagem) de partes em dois espécimes.\" O estudo indica que os membros de \"P. rubi\" são semelhantes em aparência e morfologia aos membros humanos modernos, como as fêmeas modernas e machos robustos. \"Como é evidente com os membros de \"P. rubi\", os ancestrais e o ancestral eram similares na forma e na composição das semelhanças em um organismo.\"\n",
      "\n",
      "O DNA do \"P. rubi,\" chamado por sua forma em inglês de \"sonoroplasto\", revela que o \"sonoroplasto\" inclui três genes de alto nível e quatro genes relativamente reduzidos (e ausentes) e um gene de baixo nível (e ausentes) com uma concentração de cloroplasto em cada núcleo. A \"sonoroplasto\" se assemelha à \"P. rubi\" em características morfológicas e comportamentais, embora as diferenças na morfologia sejam menores. \"A espécie \"P. rubi\" apresenta cinco pares de cromossomos separados (com 6 pares se aproximando e 8 pares se afastando) e um \"sonoroplasto de base\" (com 12 pares se aproximando e 15 pares se afastando), sugerindo que o membro tenha uma composição semelhante ao ancestral \"P. rubi\n",
      "\n",
      "---\n",
      ">> Generated text 3\n",
      "\n",
      "Num achado chocante, o cientista descobriu uma manada de unicórnios vivendo num vale remoto, anteriormente inexplorado, nas Montanhas dos Andes. Ainda mais surpreendente para os pesquisadores foi o fato de que os unicórnios falavam inglês perfeito.\n",
      "O jogo de corrida da FIAT foi introduzido no Salão Internacional de FIAT (em 1989) em um evento que teve como principal objetivo fornecer uma oportunidade a todos que competirem em corridas de carros nacionais, como as categorias de base e pilotos. Esse evento trouxe o título de \"National Le Mans Series\" (1973), além de um terceiro título que o vencedor da competição ganharia. O primeiro campeonato foi disputado entre 1983 e 1985, e foi vencido pelo piloto estadunidense Dennis Young.\n",
      "\n",
      "O Campeonato de Le Mans foi criado por uma equipa formada em 1983 na rua General Motors, em Long Beach, Califórnia, e patrocinado pela \"Frampton Racing\", criada pelo empresário de corridas do Reino Unido Derek J. Cox, e é atualmente patrocinado pela equipe da equipe de Fórmula 1 da Ferrari, Lotus e McLaren, \"Lusa Motorsports\" e pela equipe de corrida britânica Team Lotus. \"Lusa\" foi fundada como uma \"casa\", mas a companhia não conseguiu competir em corridas por cinco anos devido à falta de patrocinador, e os pilotos que competiram em Lusa foram proibidos de pilotar a equipe McLaren em 1997. A equipe mudou seu nome para Lusa Motorsports em 1999. As corridas de Fórmula 1 só foram retomadas duas vezes por falta de patrocinadores, no começo de 2000, e novamente em 2003 por \"franqueza\".\n",
      "\n",
      "Em 2003, a Williams-Renault obteve o seu primeiro título da temporada, uma corrida de Le Mans de estilo \"Grand Prix\", em que os carros eram todos monopostos, mas com carros \"primo\", mais do que os carros de Fórmula 1. A temporada de 2003 teve muitas modificações em termos de desempenho, como novos pneus e mudanças na configuração dos carros, e foi a sexta temporada do campeonato do campeonato com mais pontos no campeonato na temporada de 2004, com cinco vitórias no campeonato, três na temporada de 2008 e cinco na de 2009.\n",
      "\n",
      "Em 2005, a Toyota assinou um contrato exclusivo para a produção de veículos para a Temporada de Le Mans e o primeiro ano em que a equipe Toyota foi a primeira equipe a ser a primeira a dar uma vitória ao correr \"Le Mans Series\", uma prova de Fórmula 1, na qual a equipe Toyota produziu dois carros da temporada. Apesar de ter vencido a prova, foi a\n",
      "\n",
      "---\n",
      "CPU times: user 7.77 s, sys: 372 ms, total: 8.15 s\n",
      "Wall time: 8.16 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#set top_k = 40 and num_return_sequences = 3\n",
    "sample_outputs = model_pt.generate(input_ids, pad_token_id=50256,\n",
    "                                   do_sample=True, \n",
    "                                   max_length=max_length, \n",
    "                                   min_length=max_length,\n",
    "                                   top_k=40,\n",
    "                                   num_return_sequences=3)\n",
    "\n",
    "for i, sample_output in enumerate(sample_outputs):\n",
    "    print(\">> Generated text {}\\n\\n{}\".format(i+1, tokenizer_pt.decode(sample_output.tolist())))\n",
    "    print('\\n---')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Use case 2 (Top-p nucleus sampling)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ">> Generated text 1\n",
      "\n",
      "Num achado chocante, o cientista descobriu uma manada de unicórnios vivendo num vale remoto, anteriormente inexplorado, nas Montanhas dos Andes. Ainda mais surpreendente para os pesquisadores foi o fato de que os unicórnios falavam inglês perfeito. Eles não sabiam onde exatamente eram falantes nativos do idioma, e acreditaram que eles simplesmente migraram das terras altas da região de Mendoza ao norte como consequência do declínio populacional que ocorreu na Cordilheira das Cobras.\n",
      "\n",
      "Em 2004, o Departamento de Antropologia da Universidade do Colorado anunciou que havia encontrado uma fêmea no vale do rio Orinoco na Bolívia, mas essa fêmea foi morta durante a investigação. No entanto, no início de 2006, as autoridades locais anunciaram que havia identificado uma fêmea encontrada em uma área próxima à Cordilheira dos Andes, no Vale do Cauca. A equipe de pesquisadores relatou que esta fêmea era chamada de \"El Maria\" ou \"El Maria\".\n",
      "\n",
      "O estudo revelou que o grupo de unicórnios habitava um ecossistema bastante diverso, com espécies endêmicas incluindo espécies como as tiláceas gigantescas (que são encontradas principalmente nos países subdesenvolvidos) e as quelupus (\"Erica azoricae\").\n",
      "\n",
      "Um dos principais objetivos do estudo da espécie é determinar se os europeus teriam colonizado a região entre a década de 1940 e 1960 e se estes últimos grupos étnicos sobreviveram até hoje. Os cientistas acreditam que as populações desses grupos poderiam ter sido muito maiores antes disso; por exemplo, a teoria sugere que a população europeia provavelmente teria introduzido os humanos primitivos na América Central depois que os espanhóis invadiram a região, embora isso seja controverso.\n",
      "\n",
      "O gênero \"El Maria\" tem um ancestral comum, os \"Looney-do-the-Bone\", um pequeno grupo de \"Looney-da-Daíndia\" encontrados apenas no leste dos Estados Unidos, Canadá e México. O gênero possui parentesco próximo ao gênero \"Lontrapyrus\", também conhecido como lontras negras. Acredita-se que esses indivíduos tenham migrado para o leste dos Andes, atravessando regiões montanhosas do sul de América Central e América Central.\n",
      "\n",
      "Os membros desta família são geralmente confundidos com os lontras brancos.\n",
      "\n",
      "As fêmeas têm cerca de seis centímetros de comprimento, pesando de 9 quilogramas e medindo 11 cm de largura. A cabeça é branca, com manchas escuras pretas escuras sobre seus flancos. As patas posteriores podem ser amarelas, enquanto sua cauda pode estar preta ou branca, dependendo da cor utilizada na identificação. As costas apresentam quatro dedos dorsais bem desenvolvidas\n",
      "\n",
      "---\n",
      ">> Generated text 2\n",
      "\n",
      "Num achado chocante, o cientista descobriu uma manada de unicórnios vivendo num vale remoto, anteriormente inexplorado, nas Montanhas dos Andes. Ainda mais surpreendente para os pesquisadores foi o fato de que os unicórnios falavam inglês perfeito.\n",
      "\n",
      "O fóssil é encontrado no Vale de La Guaira, na região de San Juan, com cerca de 1 metro (1,8 m) de comprimento e cerca de 4 metros (2 pés) de largura, em Wyoming. A cabeça possui cinco dedos no pé direito e duas garras direita. No topo do crânio há quatro placas pretas-escuros. O cérebro tem um formato triangular, a base apresenta apenas uma pequena porção da metade anterior e três partes do pescoço são pretos-claro; a face tem um tom vermelho escuro-escuro. A mandíbula também contém oito pares de maxilas bem desenvolvidas.\n",
      "\n",
      "A região de San Juan foi descoberta por volta de 14 mil anos atrás. Foi escavado durante a Guerra dos Sete Anos e tem aproximadamente 6 metros quadrados de espessura. O local era habitado pelo povo de Tacuareña, que vivia entre o final do e o início do e tinha uma cultura diferente das outras regiões próximas ao território de El Berriol. Os indígenas não eram caçadores conhecidos pelos nativos e como \"carijós\".\n",
      "\n",
      "O fóssil foi descoberto em 2002 e depois enviado para uma equipe de arqueólogos liderada pela Universidade de Wyoming. A equipe encontrou os restos humanos e o chamou de \"caçadores\" ou \"coletores\", pois eles tinham sido mortos pelas expedições anteriores ao assentamento.\n",
      "\n",
      "O arqueólogo Brian C. Fanning, da Universidade de Wyoming, fez várias entrevistas com os esqueletos humanos e descobriram os crânios de dois ou três homens que foram enterrados em um túmulo localizado abaixo do túmulo de um antigo homem chamado Huehue, que estava enterrado por volta de 10500 anos atrás. O homem que estava enterrado é o membro de um grupo étnico conhecido como \"Huehue\".\n",
      "\n",
      "A equipe de pesquisadores da equipe de Fanning incluiu a presença humana no local onde o homem enterrado, incluindo um cemitério indígena que está próximo à entrada do cemitério. Eles encontraram evidências ósseas humanas enterradas perto dos ossos do homem e outros artefatos encontrados dentro de um depósito de ossos de um homem chamado \"Creator\".\n",
      "\n",
      "Os pesquisadores acharam vários ossos humanos semelhantes aos ossos de Huehue, mas sem as marcas físicas. Eles notaram semelhanças anatômicas entre esses membros de um grupo étnico chamado \"Caçadores\", possivelmente porque ambos viviam no Vale de\n",
      "\n",
      "---\n",
      ">> Generated text 3\n",
      "\n",
      "Num achado chocante, o cientista descobriu uma manada de unicórnios vivendo num vale remoto, anteriormente inexplorado, nas Montanhas dos Andes. Ainda mais surpreendente para os pesquisadores foi o fato de que os unicórnios falavam inglês perfeito.\n",
      "\n",
      "Os arqueólogos descobriram um conjunto de moedas cunhadas por indígenas e outros achados arqueológicos encontrados no local durante as escavações foram feitos ao longo da década de 1960 em um esforço de resgate arqueológica e reconhecimento pela população indígena.\n",
      "\n",
      "O achado arqueológico é o único relato sobre a cultura do gênero \"Saccharus\", que se refere às culturas terrestres primitivas não relacionadas com o \"Homo sapiens\".\n",
      "\n",
      "A pesquisa envolveu cerca de 60 anos de escavação, incluindo levantamento topográfico efetuado entre 1972 e 1977. Em 1988, o arqueólogo americano John Deere, que pesquisou as ruínas do complexo arqueológico do Monte dos Andes, observou que a cultura deste gênero estava fortemente relacionada com outras civilizações mesoamericanas, como o olmeca e a olmeca.\n",
      "\n",
      "O arqueólogo franco-americano John Deere observou que a cultura desses gêneros era bem próxima à civilização olmeca e \"a cultura dos astecas\" foi muito similar àquelas encontradas na América Central em seus sítios prévios.\n",
      "\n",
      "Em 1991, o arqueólogo americano John Deere publicou uma extensa monografia descrevendo a cultura dos Andes. Ele escreveu:\n",
      "\n",
      "Muitos estudiosos têm alegado que a cultura dos Andes pode ter sido transmitida através de migrações humanas do continente americano. Contudo, estudos recentes, baseados nos dados históricos obtidos em diferentes lugares dos Andes sugerem que tais migrações ocorreram antes ou imediatamente após as primeiras invasões americanas de europeus.\n",
      "\n",
      "Pesquisas iniciais demonstraram que esta migração ocorreu provavelmente entre o final do século XVIII e meados do século XIX, quando sociedades modernas dominavam o solo chileno, Peru, Bolívia e Colômbia. As tribos inca e xamãs também estavam envolvidas nesta atividade.\n",
      "\n",
      "A história das Américas começou em torno de um grupo de caçadores sulamericanos pertencentes aos Andes liderados pelo índio A. Iruña, que estabeleceu sua capital no atual assentamento chamado \"San Juan del Sur\". Acreditava-se que ele teria fundado um assentamento que seria conhecido por sua fama de ser capaz de caçar com seus guerreiros e construir suas casas com o passar do tempo.\n",
      "\n",
      "Durante o século XVIII, as terras baixas eram propícias para o crescimento populacional e, consequentemente, o desenvolvimento econômico das novas cidades situadas em torno delas.\n",
      "\n",
      "Os primeiros exploradores espanhóis chegaram ao Novo Mundo em 1692 e estabeleceram assentamentos permanentes nesses territórios até a década de 1830. As tentativas espanholas\n",
      "\n",
      "---\n",
      "CPU times: user 35.3 s, sys: 1.85 s, total: 37.2 s\n",
      "Wall time: 37.2 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#set top_p = 0.95, top_k = 50, temperature = 0.7, repetition_penalty = 1.2 and num_return_sequences = 3\n",
    "sample_outputs = model_pt.generate(input_ids, pad_token_id=50256,\n",
    "                                   do_sample=True,\n",
    "                                   max_length=max_length, \n",
    "                                   min_length=max_length,\n",
    "                                   repetition_penalty=1.2,\n",
    "                                   temperature=0.7,\n",
    "                                   top_k=50, \n",
    "                                   top_p=0.95, \n",
    "                                   num_return_sequences=3)\n",
    "\n",
    "for i, sample_output in enumerate(sample_outputs):\n",
    "    print(\">> Generated text {}\\n\\n{}\".format(i+1, tokenizer_pt.decode(sample_output.tolist())))\n",
    "    print('\\n---')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text n°2 | Recent text on the coronavirus disease (Covid-19)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A Rússia está mais perto de se tornar o primeiro país a iniciar a distribuição de uma vacina contra o coronavírus para a população. O país anunciou hoje que concluiu parte dos testes clínicos necessários para comprovar a eficácia da imunização desenvolvida por iniciativa do governo russo. A expectativa é de que a distribuição comece já em agosto.\n"
     ]
    }
   ],
   "source": [
    "# source (uol, 07/13/2020): https://www.uol.com.br/vivabem/noticias/redacao/2020/07/13/russia-vacina-conclui-testes-distribuicao-em-agosto.htm\n",
    "prompt = \"A Rússia está mais perto de se tornar o primeiro país a iniciar a distribuição de uma vacina contra o coronavírus para a população. O país anunciou hoje que concluiu parte dos testes clínicos necessários para comprovar a eficácia da imunização desenvolvida por iniciativa do governo russo. A expectativa é de que a distribuição comece já em agosto.\"\n",
    "print(prompt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "# encode\n",
    "input_ids = tokenizer_pt.encode(prompt, return_tensors='pt').to('cuda')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Use case 1 (Top-k sampling)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ">> Generated text 1\n",
      "\n",
      "A Rússia está mais perto de se tornar o primeiro país a iniciar a distribuição de uma vacina contra o coronavírus para a população. O país anunciou hoje que concluiu parte dos testes clínicos necessários para comprovar a eficácia da imunização desenvolvida por iniciativa do governo russo. A expectativa é de que a distribuição comece já em agosto.\n",
      "\n",
      "Entre os primeiros casos confirmados de COVID-19 na Rússia estava um homem russo com idade entre 50 e 49 anos, segundo o jornal \" Moscow\" e o primeiro caso em 28 de fevereiro, a primeira no Hospital Pulkai.\n",
      "\n",
      "No entanto, os primeiros casos foram mais sérios devido sua \"maturidade sem gravidade\" e a falta de uma pessoa estar disponível para testes de suas condições de vida. Entre os pacientes que foram considerados estão funcionários de um hospital ou enfermeiros na cidade de Moscou (ver abaixo) ou médicos. Depois de serem testados negativos após um teste positivo, o paciente se recupera completamente.\n",
      "\n",
      "Os russos também anunciaram que serão realizados testes de coronavírus de outros países, como a França, que inicialmente acreditava que os vírus da gripe tinham sido transmitido pela Europa (a expectativa é de 20 a 50 casos por dia). A situação foi resolvida em 24 de fevereiro, quando o Ministério da Saúde confirmou sua conclusão de que a COVID-19 é transmitida de via aérea.\n",
      "\n",
      "Até ao dia do seu primeiro caso, a Rússia tinha o menor número de funcionários e médicos com doença grave antes de o vírus ter se tornado um vírus no país. Os números de funcionários com doença grave não chegaram a ser confirmados, segundo o Ministério da Saúde, devido a sua falta de apoio.\n",
      "\n",
      "O vírus que está em curso no país é transmitido pela primeira vez nos Estados Unidos, onde foi isolado em 14 de fevereiro. Em Portugal o Ministério dos Negócios Estrangeiros declarou oficialmente em 11 de fevereiro que a COVID-19 está presente no país, mas não anunciou nenhum impacto na prática. No entanto, o Ministério do Trabalho declarou no dia seguinte que o coronavírus existe \"em todo os países que não têm regulamentação\" e que nenhum \"aedes\", que era identificado em 11 de fevereiro, tinha entrado na corrente sanguínea.\n",
      "\n",
      "Em 9 de março, autoridades russas confirmaram que o paciente não está hospitalizado e que está em quarentena no Hospital Pulkai.\n",
      "\n",
      "Mais dois casos de COVID-19 foram anunciados em 9 de março, mas foram considerados casos de \"propaganda\" e não de risco.\n",
      "\n",
      "Em 9 de maio, autoridades russas confirmaram os confirmados em todo país. Em 10 de maio,\n",
      "\n",
      "---\n",
      ">> Generated text 2\n",
      "\n",
      "A Rússia está mais perto de se tornar o primeiro país a iniciar a distribuição de uma vacina contra o coronavírus para a população. O país anunciou hoje que concluiu parte dos testes clínicos necessários para comprovar a eficácia da imunização desenvolvida por iniciativa do governo russo. A expectativa é de que a distribuição comece já em agosto. As populações de Wuhan, Wuhan, Qinghai, Shenzhen na costa oeste do China e no norte de Xangai, na província de Hubei, e os subúrbios da cidade de Tvershend em Hong Kong já estão de acordo com o número de casos confirmados. Uma nova equipe de infecologistas para ajudar a determinar os possíveis efeitos da pandemia já está em Xangai. Em 14 de março, a Rússia também tem a oportunidade de testar novas vacinas virais e testar novas versões para garantir a propagação de maisvírus na China através da transmissão de novos coronavírus.\n",
      "\n",
      "As autoridades russas e autoridades de saúde chinesas têm se preparado para assumir essa atitude de emergência com base nos dados coletados pela máscaras, máscaras de voo e máscaras faciais. Um relatório de uma empresa francesa informou que mais de 30 pessoas morreram na área de transmissão ao longo do mês de março, em relação aos dois primeiros casos. Em 21 de março, mais de 2,3 milhões de pessoas foram afetadas em áreas urbanas. Segundo o Conselho de Saúde, cerca de 10% dos habitantes são da China continental e quase metade do Distrito de Pequim, incluindo as regiões Norte e Noroeste. A doença foi descrita pela primeira vez em uma carta enviada às autoridades sanitárias no dia 15 de março. Três dias depois, um estudo genético para o vírus causou grande preocupação aos especialistas em transmissão, que acreditavam que a doença se espalhava através de duas regiões do país. A China continental relatou que apenas 8 pessoas foram diagnosticadas no país.\n",
      "\n",
      "Os vírus estão propagando-se por todo o continente europeu, do Ásia e de diversas partes do globo. De acordo com análises recentes do Centro de Controle e Prevenção da Doenças, em 14 de março, havia um total de 785 casos confirmados confirmados em todo o mundo. O surto da doença chegou aos Estados Unidos, com o surto de gripe na cidade de Wuhan, China, sendo o quarto maior já registrado em um período desde a epidemia de surtos no ano de 1994. Além disso, os casos das novas epidemias trazidas pela epidemia das SARS de 2013 no Haiti em 2020 tornaram-se os maiores problemas para a saúde da China. Em 14 de março, cerca de 4,85 milhão de pessoas estavam infectadas e cerca de 8%\n",
      "\n",
      "---\n",
      ">> Generated text 3\n",
      "\n",
      "A Rússia está mais perto de se tornar o primeiro país a iniciar a distribuição de uma vacina contra o coronavírus para a população. O país anunciou hoje que concluiu parte dos testes clínicos necessários para comprovar a eficácia da imunização desenvolvida por iniciativa do governo russo. A expectativa é de que a distribuição comece já em agosto.\n",
      "O Papa Francisco pediu ao povo russo que confirme a data da chegada da vacina. O Papa também pediu aos cidadãos russos que realizem uma conferência de imprensa oficial para o anúncio. Em 7 de dezembro de 2016, quando se iniciou o trabalho de tradução para mais idiomas, mais de 300 pessoas no mundo tiveram acesso aos benefícios da vacina.\n",
      "\n",
      "A primeira vacina para o coronavírus foi testada no Japão em 9 de março de 2020.\n",
      "\n",
      "Em 6 de fevereiro, um estudo realizado em Taiwan revelou que a epidemia do SARS é mais provável devido ao aumento do hábito. Por causa do aumento nas visitas dos cidadãos a hospitais para verificar que a vacina é eficaz, a saúde do país começou a declinar. O ministro da Saúde japonês, Toshihiko Hayashi disse que a crise da saúde era uma consequência da melhora das medidas de prevenção e contra o coronavírus em relação ao início do período de alta infecção.\n",
      "\n",
      "A Rússia começou a distribuir a vacina no início de outubro, após a confirmação por funcionários da saúde pública de uma nova coronavírus. Foi lançada no mesmo dia em todas as regiões ocidentais do país. O total de pessoas hospitalizadas pelo vírus foi de aproximadamente 431 em Taiwan. Além disso, entre 15 e 17 de outubro, uma pessoa foi internada no Hospital Universitário de Pequim, nos arredores de Pequim, devido a um surto viral de SARS. Os sintomas de pneumonia, febre, prurido e pneumonia permaneceram altas durante toda a primeira quinzena de outubro. Os outros dois dias depois, no Hospital Universitário de Moscou, a doença causou mais mortes confirmadas do que não provocado anteriormente, com cerca de 211. O número de passageiros do Hospital Universitário de Moscou caiu 297 pessoas.\n",
      "\n",
      "A agência russa \"DPS\" declarou como uma das principais preocupações sobre a doença a saúde pública na República Popular da China. Em janeiro de 2020, o governo chinês lançou três programas de vacinação oral nos distritos urbanos em resposta à melhora global e a pandemia no Vale do Silício. O programa nacional de vacinação, chamado de \"Omgao\", consiste num conjunto de 36 cidades com um total de 1.429 municípios (em regiões mais remotas do país). A aplicação do programa foi feita em 27\n",
      "\n",
      "---\n",
      "CPU times: user 7.56 s, sys: 242 ms, total: 7.81 s\n",
      "Wall time: 7.8 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#set top_k = 40 and num_return_sequences = 3\n",
    "sample_outputs = model_pt.generate(input_ids, pad_token_id=50256,\n",
    "                                   do_sample=True, \n",
    "                                   max_length=max_length, \n",
    "                                   min_length=max_length,\n",
    "                                   top_k=40,\n",
    "                                   num_return_sequences=3)\n",
    "\n",
    "for i, sample_output in enumerate(sample_outputs):\n",
    "    print(\">> Generated text {}\\n\\n{}\".format(i+1, tokenizer_pt.decode(sample_output.tolist())))\n",
    "    print('\\n---')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Use case 2 (Top-p nucleus sampling)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ">> Generated text 1\n",
      "\n",
      "A Rússia está mais perto de se tornar o primeiro país a iniciar a distribuição de uma vacina contra o coronavírus para a população. O país anunciou hoje que concluiu parte dos testes clínicos necessários para comprovar a eficácia da imunização desenvolvida por iniciativa do governo russo. A expectativa é de que a distribuição comece já em agosto.\n",
      "\n",
      "A Rússia tem um mercado interno bruto (PIB) estimado entre US$ 1,2 bilhões e US$ 7 bilhões. O PIB \"per capita\" em 2007 foi de US $ 4,348. A maior parte do crescimento econômico anual ocorreu na agricultura. Em 2003, a taxa média anual era de 22%.\n",
      "\n",
      "Em 2004, cerca de 3 milhões de pessoas foram infectadas pelo coronavírus. Cerca de 17 mil casos foram identificados entre 2001 e 2002, incluindo aproximadamente 2,8% das crianças e adolescentes com idade inferior a seis anos; 5,1 milhão deles eram endêmicas ou sem cuidados médicos; 8,711 mortes ocorreram nos três primeiros meses de vida após o contato com o vírus; 13,6 mil haviam retornado à Rússia; e 14,4% estavam doentes devido aos sintomas causados pela doença durante seus últimos dias no exterior, principalmente idosos e recém-nascidos infectados.\n",
      "\n",
      "Cerca de 6 milhões de russos estão vivendo em países fora da União Europeia, sendo eles os Estados Unidos, China, Índia, Itália, Japão, Reino Unido, Austrália, Nova Zelândia, Países Baixos, Rússia, Turquia, Espanha, Suécia e África do Sul. Em novembro de 2008, estimava-se que 20 milhões de russos estariam potencialmente infetados naquele ano.\n",
      "\n",
      "A Organização Mundial da Saúde divulgou dados detalhados sobre as medidas antivirais prescritas por cada país segundo os padrões estabelecidos.\n",
      "\n",
      "O número total de infeção por coronavírus foi inicialmente estimado em 12.000 em 2005. Em janeiro de 2006, estimava-se que 924.300 pessoas tinham sido contaminadas. No mesmo mês, o Ministério da Saúde confirmou que havia aumentado o número de casos confirmados desde o início da pandemia. O número de mortes atribuídas ao coronavírus caiu de 50 para 54 em janeiro de 2006, enquanto novos surtos começaram em março de 2006. Os números de casos suspeitos aumentaram ainda mais após o início da epidemia em janeiro. A Organização Mundial da Saúde divulgou estatísticas oficiais em fevereiro de 2006 mostrando que todos os indivíduos que tenham sido contestados como tendo casos não apresentaram qualquer sintoma claro ou grave risco.\n",
      "\n",
      "Desde o início da pandemia, um grande número de pessoas têm sido afetadas por COVID-19. Mais tarde,\n",
      "\n",
      "---\n",
      ">> Generated text 2\n",
      "\n",
      "A Rússia está mais perto de se tornar o primeiro país a iniciar a distribuição de uma vacina contra o coronavírus para a população. O país anunciou hoje que concluiu parte dos testes clínicos necessários para comprovar a eficácia da imunização desenvolvida por iniciativa do governo russo. A expectativa é de que a distribuição comece já em agosto.\n",
      "\n",
      "Em junho, a Organização Mundial da Saúde divulgou um relatório sobre o surto e previu que os casos confirmados pela Rússia naquele ano ultrapassariam 10 milhões de pessoas. Em julho, foi relatado que o vírus havia matado 50% das crianças entre 15 e 21 anos. Em agosto, o Ministério da Saúde confirmou que as mortes foram estimadas no total devido à infecção causada pelo vírus \"S\". Em setembro, uma nova pesquisa revelou que 90% dos infectados haviam sido mortos antes mesmo da chegada da vacina.\n",
      "\n",
      "O governo soviético aprovou em dezembro de 2008, como forma de comemorar o centenário do nascimento da primeira vacina contra o coronavírus (CVD), dois novos centros comerciais russos estão sendo construídos na cidade de Moscou. No entanto, apenas três deles são permanentes: um na Rússia central; outro na fronteira leste com a Ucrânia; e outro nos subúrbios de Vyacheslavsky-Kaliningrado. Em 29 de janeiro de 2009, o governo anunciou planos futuros para construir estações comerciais em todas as regiões afetadas pelos surtos.\n",
      "\n",
      "Até maio de 2013, todos os postos de saúde oficiais estabelecidos no país estavam fechados desde então. Em 1º de março de 2014, a Rússia fechou seus aeroportos, deixando suas fronteiras fechadas para os voos domésticos e internacionais.\n",
      "\n",
      "Desde abril de 2015, cerca de 318 mil imigrantes estrangeiros vêm anualmente da Rússia, principalmente alemães étnicos russos residentes na Alemanha Oriental e eslavos orientais radicados lá. A entrada desses migrantes trouxe também grandes quantidades de problemas econômicos ao país. Os últimos emigrantes vindos ilegalmente vieram sobretudo após a Segunda Guerra Mundial.\n",
      "\n",
      "No final de 2019, muitos milhares de trabalhadores europeus emigraram para o leste da Europa para escapar da guerra civil russa. A maioria destes emigrantes veio da Rússia depois da invasão alemã de 1941.\n",
      "\n",
      "De acordo com a Organização Mundial da Saúde, a taxa de mortalidade infantil na Rússia diminuiu substancialmente desde o início do século XX, mas o número caiu em grande medida entre 1990 e 2001, quando o número aumentou novamente, alcançando a cifra de 7,5% entre 1995 e 2000. No entanto, esse número permaneceu estável até 2007, quando passou de 4,1%, e subiu novamente para 5,9%. As taxas gerais de mortalidade diminuíram consideravelmente durante este\n",
      "\n",
      "---\n",
      ">> Generated text 3\n",
      "\n",
      "A Rússia está mais perto de se tornar o primeiro país a iniciar a distribuição de uma vacina contra o coronavírus para a população. O país anunciou hoje que concluiu parte dos testes clínicos necessários para comprovar a eficácia da imunização desenvolvida por iniciativa do governo russo. A expectativa é de que a distribuição comece já em agosto.\n",
      "\n",
      "A Organização Mundial de Saúde (OMS) estima que, no final de dezembro de 2015, havia pelo menos 50 milhões de pessoas infectadas com o vírus na Rússia e Ucrânia. Estimativas semelhantes foram feitas pela OMS sobre os casos registrados desde a década passada. Em março de 2018, o Ministério da Saúde confirmou a descoberta de um novo coronavírus e recomendou aos cidadãos russos manter a disposição para evitar viagens ao exterior após o início das aulas médicas ou exames médicos.\n",
      "\n",
      "Em novembro de 2016, o Ministro da Saúde russo confirmou que todos os indivíduos infectados poderiam ser testados em março; assim como seus familiares e amigos, eles podem fazer teste clínico em uma unidade cirúrgica do hospital da cidade em que são colocados, caso necessário. Também foi anunciado no mesmo dia que o Ministério do Trabalho revelou que as autoridades russas estão trabalhando em conjunto visando reduzir o número de mortes causadas pelas epidemias. A agência informou que a Agência Nacional de Vigilância Sanitária Russa (Anvisa) começou a monitorar a pandemia através de máscaras faciais nos hospitais.\n",
      "\n",
      "Em abril de 2019, o Ministério da Saúde divulgou que 582 mil pessoas haviam sido infetadas com o vírus no país entre janeiro de 2019 e maio de 2020. Cerca de 370 mil desses pacientes estariam diretamente relacionados à doença.\n",
      "\n",
      "A Rússia também tem planos promissores para produzir vacinas que sejam eficazes contra o coronavírus, incluindo a vacina anti-SIDA e antivirais (ver Lista Vermelha da OMS).\n",
      "\n",
      "Em 1º de julho de 2017, a Organização Mundial de Saúde lançou uma nota oficial alertando que \"um grande aumento pode vir da necessidade de medidas preventivas necessárias\" antes do início das aulas médicas em escolas públicas nas cidades ucranianas.\n",
      "\n",
      "Em outubro de 2017, a Secretaria Municipal de Saúde ucraniana publicou uma nota oficial informando que os profissionais responsáveis ​​pela coordenação da vacinação deveriam estar preparados, bem como suas famílias e amigos durante a realização de exames adicionais para determinar sua saúde mental.\n",
      "\n",
      "Em 30 de junho de 2019, o Ministério da Saúde lançou um comunicado afirmando que três grupos escolares teriam dificuldades de administrar adequadamente o vacina contra o coronavírus na Rússia:\n",
      "\n",
      "A primeira ministra russa, Valentina Tereshinakova, disse em entrevista coletiva que não haverá\n",
      "\n",
      "---\n",
      "CPU times: user 37.9 s, sys: 1.78 s, total: 39.7 s\n",
      "Wall time: 39.6 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "#set top_p = 0.95, top_k = 50, temperature = 0.7, repetition_penalty = 1.2 and num_return_sequences = 3\n",
    "sample_outputs = model_pt.generate(input_ids, pad_token_id=50256,\n",
    "                                   do_sample=True,\n",
    "                                   max_length=max_length, \n",
    "                                   min_length=max_length,\n",
    "                                   repetition_penalty=1.2,\n",
    "                                   temperature=0.7,\n",
    "                                   top_k=50, \n",
    "                                   top_p=0.95, \n",
    "                                   num_return_sequences=3)\n",
    "\n",
    "for i, sample_output in enumerate(sample_outputs):\n",
    "    print(\">> Generated text {}\\n\\n{}\".format(i+1, tokenizer_pt.decode(sample_output.tolist())))\n",
    "    print('\\n---')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are the first, fortunately surprised by the efficiency of fine-tuning in Portuguese an English pre-trained transformer-based language model like GPT-2 small.\n",
    "\n",
    "In about 1 day using 1 GPU and a little over 1 GB of Portuguese texts, we managed to obtain a **GPorTuguese-2 capable of generating contextual Portuguese texts of a level comparable to that of the GPT-2 used by OpenAI in 2019**.\n",
    "\n",
    "Happy.\n",
    "\n",
    "The next step would be to apply our fine-tuning method to most recent NLP models like GPT-3, BART, T5 or Reformer. Let’s do it?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## END"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}