fastai_NLP/ULMFiT_bwd.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "# 多行输出\n",
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\" "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 精调通用语言模型用于文本分类\n",
    "- [fastai/ULMFit.ipynb at master · fastai/fastai](https://github.com/fastai/fastai/blob/master/examples/ULMFit.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 文本预处理做的比较糙，繁体简体咩有考虑"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai import *\n",
    "from fastai.text import *\n",
    "from fastai.core import *\n",
    "from sklearn.metrics import f1_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 基于中文维基百科训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs = 64\n",
    "lang = 'zh'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "PosixPath('/home/lab/.fastai/data')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data_path = Config.data_path()\n",
    "data_path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 提取TXT文件放在 docs 文件夹下"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "name = f'{lang}wiki'\n",
    "path = data_path/name\n",
    "path.mkdir(exist_ok=True, parents=True)\n",
    "dest = path/'docs'\n",
    "lm_fns_bwd = [f'{lang}_wt_bwd', f'{lang}_wt_vocab_bwd']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 下载的文件为 xml 格式\n",
    "- 大概有 46480 词目"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 提取信息"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/lab/.fastai/data/zhwiki/log'),\n",
       " PosixPath('/home/lab/.fastai/data/zhwiki/train.csv'),\n",
       " PosixPath('/home/lab/.fastai/data/zhwiki/zhwiki'),\n",
       " PosixPath('/home/lab/.fastai/data/zhwiki/zh_textlist_class'),\n",
       " PosixPath('/home/lab/.fastai/data/zhwiki/models'),\n",
       " PosixPath('/home/lab/.fastai/data/zhwiki/docs'),\n",
       " PosixPath('/home/lab/.fastai/data/zhwiki/zhwiki-latest-pages-articles.xml'),\n",
       " PosixPath('/home/lab/.fastai/data/zhwiki/val.csv'),\n",
       " PosixPath('/home/lab/.fastai/data/zhwiki/zh_databunch'),\n",
       " PosixPath('/home/lab/.fastai/data/zhwiki/zhwiki-latest-pages-articles.xml.bz2'),\n",
       " PosixPath('/home/lab/.fastai/data/zhwiki/wikiextractor')]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path.ls()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 创建数据\n",
    "- 先运行 forward\n",
    "- 先加载 forward 数据做个对比\n",
    "- 创建 backward 数据\n",
    "- backward模型\n",
    "- 集成"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- token和数值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = load_data(path, f'{lang}_databunch', bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>idx</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>xxmaj junior xxunk xxmaj junior xxunk xxmaj xxunk 。 \\n \\n  xxunk xxunk xxmaj lasting xxmaj xxunk 。 \\n \\n  xxunk xxunk 。 \\n \\n  xxunk 。 \\n \\n  xxunk xxunk xxmaj junior xxmaj kiss xxmaj the xxunk 。 \\n \\n  xxunk xxunk xxunk 。 \\n \\n  xxunk xxmaj junior xxmaj kiss xxmaj the xxmaj xxunk 。 \\n \\n  xxunk xxunk xxunk 。</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>xxunk - xxunk - xxunk 。 \\n \\n  xxunk - xxunk 。 \\n \\n  xxunk 。 \\n \\n  xxunk 。 \\n \\n  xxunk 。 \\n \\n  xxunk 。 \\n \\n  xxunk / xxunk / xxunk / xxunk / xxunk 。 \\n \\n  xxunk - xxunk 。 \\n \\n  xxunk 。 \\n \\n  xxunk 。 \\n \\n  xxunk 。 \\n \\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>\\n \\n  xxunk 。 \\n \\n  xxup xxunk / xxup xxunk / xxup xxunk / xxunk 。 \\n \\n  xxunk / xxup xxunk / xxup xxunk / xxup xxunk / xxup xxunk / xxup xxunk / xxup xxunk - xxup xxunk - xxup xxunk - xxunk 。 \\n \\n  xxunk 。 \\n \\n  xxunk 。 \\n \\n  xxunk xxrep 4 、 xxunk 。 \\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>。 ” \\n \\n  《 xxunk 。 ” \\n \\n  《 xxunk 。 ” \\n \\n  xxunk 》 ） \\n \\n  《 xxunk 。 ” \\n \\n  xxunk 》 ） \\n \\n  《 xxunk … … xxunk … … xxunk ） 。 \\n \\n  xxunk 》 ） \\n \\n  xxunk 》 ） \\n \\n  xxunk 》 ） \\n \\n  xxunk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>。 \\n \\n  xxunk 。 \\n \\n  xxunk ： \\n \\n  xxunk 。 \\n \\n  xxunk ： \\n \\n  formula_4 \\n \\n  xxunk \\n  xxunk 。 \\n \\n  xxunk xxunk ： \\n \\n  formula_13 \\n \\n  xxunk ： \\n \\n  xxunk 。 \\n \\n  xxunk \\n \\n  xxunk \\n \\n  xxunk \\n \\n  xxunk \\n</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data.show_batch()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_bwd = load_data(dest, f'{lang}_databunch_bwd', bs=bs, backwards=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>idx</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>\\n \\n \\n \\n \\n \\n \\n \\n  。 xxunk \\n \\n  。 xxunk xxup \\n \\n  。 xxunk xxup \\n \\n  。 xxunk / xxunk xxup \\n \\n  。 xxunk \\n \\n  。 xxunk xxup \\n \\n  。 xxunk \\n \\n  。 xxunk \\n \\n  。 xxunk \\n \\n  。 xxunk \\n \\n  。 xxunk \\n \\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>xxmaj xxunk xxunk san xxmaj of xxunk xxunk xxmaj xxunk xxunk xxunk xxmaj xxunk de bautista xxmaj xxunk de gaspar xxmaj xxunk \\n \\n  。 xxunk \\n \\n  。 xxunk - xxunk - xxunk \\n \\n  。 xxunk / xxunk \\n \\n  。 xxunk \\n \\n  。 」 xxunk \\n  。 xxunk \\n \\n  。 ） （ 」 xxunk \\n \\n  。</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>。 xxunk \\n \\n  。 xxunk xxmaj xxunk \\n \\n  。 xxunk \\n \\n  ” 。 xxunk \\n \\n  。 xxunk xxmaj xxunk \\n \\n  。 xxunk xxmaj xxunk xxmaj xxunk \\n \\n  。 xxunk \\n \\n  。 xxunk xxmaj xxunk \\n \\n  。 xxunk \\n \\n  ” 。 xxunk \\n \\n  。 」 xxunk xxmaj xxunk \\n \\n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>; 》 : cn - zh &gt; = 〉 ; 》 : hans - xxunk - zh &gt; = 〈 ; 《 : cn - zh &gt; = 〈 ; 《 : hans - xxunk \\n \\n  xxunk xxbos \\n  &gt; doc / &lt; \\n \\n \\n \\n  。 xxunk \\n \\n  。 xxunk xxmaj xxunk \\n \\n  ） basin xxunk （ xxunk xxmaj xxunk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>。 xxunk \\n \\n  。 xxunk \\n \\n  。 xxunk xxup , boundary growth urban ( 」 xxunk \\n \\n  。 xxunk \\n \\n  。 % xxunk \\n \\n  。 xxunk \\n \\n  。 xxunk \\n \\n  。 xxunk \\n \\n  。 xxunk ° xxunk ° xxunk \\n \\n  。 xxunk \\n \\n  。 xxunk \\n \\n  。 xxunk</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_bwd.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 数据对比"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "x, y = data.train_ds[0]\n",
    "x1, y1 = data_bwd.train_ds[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x == x1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 模型配置"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'emb_sz': 400,\n",
       " 'n_hid': 1152,\n",
       " 'n_layers': 3,\n",
       " 'pad_token': 1,\n",
       " 'qrnn': False,\n",
       " 'bidir': False,\n",
       " 'output_p': 0.1,\n",
       " 'hidden_p': 0.15,\n",
       " 'input_p': 0.25,\n",
       " 'embed_p': 0.02,\n",
       " 'weight_p': 0.2,\n",
       " 'tie_weights': True,\n",
       " 'out_bias': True}"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "awd_lstm_lm_config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 不使用预训练权重，我们从头训练，使用中文维基百科"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = language_model_learner(data_bwd, AWD_LSTM, drop_mult=0.5, pretrained=False).to_fp16()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEKCAYAAAAMzhLIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xl8VPW9//HXZyYbIWFNWBQURYQKgkpcoFVsbevSqlWp163uUrW17e29ve293tv+utvaVa2tVlwLalvrdbtubd1FJLgguCGiiGgWQZIAmWyf3x8zgzEmkwAzOedM3s+H85gz3zkz5/N1hnzme853MXdHRESkJ7GgAxARkXBTohARkYyUKEREJCMlChERyUiJQkREMlKiEBGRjJQoREQkIyUKERHJSIlCREQyKgg6gL6oqKjwCRMmBB2GiEikLF26tN7dK3f0fSKRKCZMmEB1dXXQYYiIRIqZvZmN99GpJxERyUiJQkREMlKiEBGRjJQoREQkIyUKERHJSIlCREQyUqIQEZGMlChEREKopqGZX9z/Cq/XNQUdihKFiEgYra7fxBUPvca7G5uDDkWJQkQkjOoaEwBUlBcHHIkShYhIKKUTRWWZEoWIiHSjvilBYdwYOqgw6FCUKEREwqiuMUFFWTGxmAUdihKFiEgY1TUlE0UYKFGIiIRQXWOCyhBcyAYlChGRUKpvSoTiQjYoUYiIhE5Hh1Pf1KIWhYiIdG/D5hbaO5yKsqKgQwGUKEREQqeuKTWGorwk4EiSlChEREKmvrEFIP9PPZnZtWZWa2bLO5X90MyWmdlzZvaAme2Uq+OLiERVXVNyfqe8TxTA9cARXcoudffp7r4PcDfw3RweX0QkkrbO85Tv1yjc/VFgfZeyhk4PBwOeq+OLiERVXWOCksIYZcUFQYcCQL9HYWY/Bk4HNgKf7O/ji4iEXbprrFnw03dAABez3f1idx8PLAC+2tN+ZjbPzKrNrLqurq7/AhQRCVhdY3gG20GwvZ4WAif09KS7X+3uVe5eVVlZ2Y9hiYgEKz0hYFj0a6Iws0mdHh4DvNyfxxcRiYK6pvDM8wQ5vEZhZjcDhwIVZrYW+B5wlJlNBjqAN4Hzc3V8EZEoam3vYMPm8EzfATlMFO5+cjfF83N1PBGRfLB+Uwvu4RlDARqZLSISKh+MoVCiEBGRbmxdK1stChER6c7WCQHVohARke6oRSEiIhnVNSYoLy6gpDAedChbKVGIiIRI2MZQgBKFiEio1DcmqFCiEBGRnqhFISIiGYVtQkBQohARCY3m1nYam9vUohARke7Vh3AMBShRiIiERhjHUIAShYhIaIRxnidQohARCY2t03eoRSEiIt2pb2wBYGRZUcCRfJgShYhISNQ1NTNicBGF8XD9aQ5XNCIiA1hyrexwtSZAiUJEJDTqGsM3KhuUKEREQqO+qSV0YyhAiUJEJBTcfeC1KMzsWjOrNbPlncouNbOXzWyZmd1uZsNydXwRkSjZ1NLOltb20I2hgNy2KK4HjuhS9iAwzd2nA68C/5nD44uIREZYR2VDDhOFuz8KrO9S9oC7t6UePgWMy9XxRUSiZEAmij44G7g3wOOLiISGEkUXZnYx0AYsyLDPPDOrNrPqurq6/gtORCQANQ3NAIwuLwk4ko/q90RhZmcAnwdOdXfvaT93v9rdq9y9qrKysv8CFBEJQG1jgqJ4jGGlhUGH8hEF/XkwMzsC+DYwx9039+exRUTCrLahmcryYsws6FA+IpfdY28GFgGTzWytmZ0DXAGUAw+a2XNm9odcHV9EJEpqGxOMGhK+6xOQwxaFu5/cTfH8XB1PRCTKahqa2b1ycNBhdEsjs0VEQqC2McHoIeG7kA1KFCIigWtubWfjllZGhbBrLChRiIgELj2GYpRaFCIi0p3axuQYCrUoRESkWzUNqRZFCAfbgRKFiEjgatOjskPaPVaJQkQkYDWNCQpixvDS8C2DCkoUIiKBq21IMKq8mFgsfKOyQYlCRCRwtY3NVIa0xxMoUYiIBK62IcHokPZ4AiUKEZHA1TY2h3aeJ1CiEBEJVKKtnQ2bW0PbNRaUKEREApUelR3WrrGgRCEiEqiwD7YDJQoRkUDVpafvUItCRES6U9uoFoWIiGRQ09BMPGaMHBzOUdmgRCEiEqjahgSVZeEdlQ1KFCIigaoJ8VrZaUoUIiIBqm1oDvX1CchhojCza82s1syWdyr7opmtMLMOM6vK1bFFRKKidoC3KK4HjuhSthw4Hng0h8cVEYmElrYO1m9qYXTIWxQFuXpjd3/UzCZ0KXsJwCy8F21ERPpLXVN6reyB26IQEZEMwr6yXVpoE4WZzTOzajOrrqurCzocEZGsi8JgOwhxonD3q929yt2rKisrgw5HRCTr0i0KnXoSEZFu1TYmiBmMHDxAE4WZ3QwsAiab2VozO8fMjjOztcAs4B4zuz9XxxcRCbuahmYqyoqJh3hUNuS219PJPTx1e66OKSISJbWNCUaHeK3sNJ16EhEJSE1DglEhXis7TYlCRCQgdY3NjFKLQkREutPa3sF7m1rUohARke7VNyVwR9coRESke7Vb18pWi0JERLpRE5HBdqBEISISiPT0HVE49ZSzcRRhkGhrp63dAeg8Ya1hHynrrGt5ev+e9uv6bE+z49rW5/u2v4jkr9qG5tSo7PCulZ2W14niR3e/xE1PvRl0GDusa0Lqmlis232N1H9byw370HuZWeo+uR2zzmWWek3y+Vin8lgs+TiW2ieWem26LBaDuBmxmG29L4gZ8dQtvV0Qj23dLorHKEzfCpKPi+IxigqSt5LCOCWFMQYVxikpjDO4uIDBRQWUlxRQVlzAkEGFoR/dKtJZTUOCirJiCuLhP7GT14ni8KljGD9iEO4flKU302WOf+g1/uGHPfLUjl337+nlfT3eR16fPk6P+3vXXfHU9tbnPF32Qcyd93FPPudAh3un59PPJffrSG0n90k+7nDf+pr2jg/KktvJ+/YOp7W9gy2tye229tR9RwdtqcdtHR20tjutbR0k2jtobe/o82cByWQ2orSIirJiKsqLGD+8lI+NHcJeOw1hyphyyksK+/5mIv3g3YZmxgwN/2knyPNE8YlJFXxiUkXQYch2cHfaOpyWtg5a2jpobmunubWDLS3tbGltZ3NLG03NbTQl2mhsbuP9La3UNyWob0xQ35Tg/hXvcsuSt7a+34zxwzjv4N04YuqYSPyCk/xX09DMuOGlQYfRJ31KFGY2EVjr7gkzOxSYDtzo7u/nMjgZuMyMwrhRGI+xPRNrujs1DQlefGcjK95u4G/Pvs1XFz7LLiNKOe/g3Zg7czyDiuLZD1ykj2obE8zcdXjQYfRJX39a3Qa0m9kewHxgN2BhzqIS2UFmxpihJXxqymguOmwSf//mHP5w2kxGDC7if+5Ywad/9QjL394YdJgyQCXa2pNrZUegxxP0PVF0uHsbcBzwG3f/V2Bs7sISya54zDhi2hhuv3A2C887EHfn+N8/yZ+r3+r9xSJZVre1a2z4x1BA3xNFq5mdDJwB3J0q09VBiRwzY/bECu666BPsP2E4//HXZVx8+wsk2tqDDk0GkJr0qOw8a1GcRXKxoR+7+2oz2w34U+7CEsmtkWXF3HDWAZw/ZyILFq/hlD8u5v3NLUGHJQNEegnU0SFfKzutT4nC3V9096+5+81mNhwod/dLchybSE4VxGN858gpXHnqfrzw9kZOvGrR1mkVRHIp/T3Lq1NPZvawmQ0xsxHA88B1Zvar3IYm0j+O2nss15+1P29v2MIJv3+SN+o3BR2S5LmaxgSFcWN4afhHZUPfTz0NdfcG4HjgOnefCXw6d2GJ9K/ZEyu4ed5BbEq0MfcPi3hxXUPQIUkeq2loZlR5CbGIzCbQ10RRYGZjgRP54GK2SF6ZPm4Yfzl/FoVx41+uXsSzazYEHZLkqdqGRCRmjU3ra6L4AXA/sMrdl5jZ7sDKTC8ws2vNrNbMlncqG2FmD5rZytR9NEabyICxx6hy/nL+LEYMLuK0axbz9Or1QYckeaimoTkyF7Kh7xez/+Lu0939gtTj1939hF5edj1wRJey7wD/cPdJwD9Sj0VCZdzwUm6dN4sxQ0s449qneeK1+qBDkjxT09AcmQvZ0PeL2ePM7PZUC6HGzG4zs3GZXuPujwJdf44dC9yQ2r4B+MI2RyzSD8YMLeGWebPYdWQpZ12/hIdeqQ06JMkTW1raaWhui8wYCuj7qafrgDuBnYCdgbtSZdtqtLu/A5C6H9XTjmY2z8yqzay6rq5uOw4lsmMqy4u5+byDmDy6nHk3VvPPl2uCDknyQG1jumts/iWKSne/zt3bUrfrgcocxoW7X+3uVe5eVVmZ00OJ9Gj44CL+dO6BTBkzhPNveoZHXtWPFtkx6VHZeXfqCag3s9PMLJ66nQa8tx3Hq0n1niJ1r/a8hN7QQYXcdM4BTBpdxnk3VvP4Sl2zkO33wWC7/GtRnE2ya+y7wDvAXJLTemyrO0nOF0Xq/o7teA+RfjestIg/nXMgu1cM5pwblvCkLnDLdqqJ2PQd0PdeT2vc/Rh3r3T3Ue7+BZKD73pkZjcDi4DJZrbWzM4BLgE+Y2Yrgc+kHotEwvDBRSw490B2HVnKOTdUq+usbJeahmaKC2IMGRSddeN2ZKmvb2Z60t1Pdvex7l7o7uPcfb67v+fuh7n7pNS9/qVJpIwsK2bBuQcxdlgJZ1+/hGVrtXaXbJuahgSjh5RgFo1R2bBjiSI6tRTJosryYhaceyDDBxdy+rVP8/K7mu5D+i5qYyhgxxKFZy0KkYgZO3QQC889iJKCOKdds5jX65qCDkkiorYxEakxFNBLojCzRjNr6ObWSHJMhciANX5EKQvOOxCAU69ZzFvrNwcckYRdci33aE3fAb0kCncvd/ch3dzK3T06V2JEcmRiZRk3nXMgm1vaOW3+4q0L0oh0pynRxuaW9gF16klEgI+NHcJ1Z+1PXWOCL81/WivlSY8+GGyXRy0KEemb/XYZzjWnV7H6vU2ccd0SmhJtQYckIZRucUZpinFQohDJmtl7VPC7U/Zj+dsbOfeGJTS3tgcdkoRMTQTneQIlCpGs+sxeo/nViTNYvHo9591YrWQhH6JTTyICwLH77Mylc2fw+Gv1ShbyITUNzQwuilNWHK2+QEoUIjkwd+Y4fnbCdB5/rZ55Ny1VshAguQTq6KHRak2AEoVIzpxYNZ6fHT+dR1+t48tKFkL0lkBNU6IQyaET9x/PJcfvzSOv1nHhgmdoaesIOiQJUE1j9KbvACUKkZw76YBd+PFx0/jny7V8deEztLYrWQxEyVHZichdyAYlCpF+ceqBu/KDY6fywIs1fP2WZ2lTshhwNm5ppaWtI3LzPAFE69K7SISdPmsCLW0d/Oiel4jHnufXJ86gIK7fagPFu1tXtoveqSclCpF+dO7Bu9Pe4fz03pcpjBu/mDuDWEwz9g8EUR1DAUoUIv3uy3Mmkmjr4FcPvsqgwjg/+sK0SC1iI9snvQTqqHK1KESkDy761B5sbmnnD4+soqQwzn9/7mNKFnmurlEtChHZBmbGt4+YTHNrO/MfX01pUZx/++zkoMOSHKppaKa8pICSwnjQoWwzJQqRgJgZ3/38XjS3tnP5P19jcHEB58+ZGHRYkiO1Ee0aCwF1jzWzr5vZcjNbYWbfCCIGkTCIxYwfH7c3R8/YiUvufZmFi9cEHZLkSG1jcySvT0AAicLMpgHnAQcAM4DPm9mk/o5DJCziMeNXJ87gk5Mrufh/X+Cu59cFHZLkQG1jQoliG3wMeMrdN7t7G/AIcFwAcYiERmE8xpWnzmT/XUfwzT8/x8Ov1AYdkmSRu1PbkIjkYDsIJlEsBw4xs5FmVgocBYzvupOZzTOzajOrrqur6/cgRfrboKI415xZxZ6jyzn/T0upfmN90CFJlmzc0kpLe4daFH3l7i8BPwMeBO4Dngc+sm6ku1/t7lXuXlVZWdnPUYoEY0hJITecfQA7DR3EWdcv4cV1DUGHJFlQm+oaqxbFNnD3+e6+n7sfAqwHVgYRh0gYVZQVc9O5B1JWXMDp1z7NG/Wbgg5JdlCUB9tBcL2eRqXudwGOB24OIg6RsNp52CBuOucA2js6OG3+4q1/aCSaaiM8fQcEN3vsbWb2InAX8BV33xBQHCKhtceocq4/6wA2bGrhS/MX8/7mlqBDku209dSTWhR95+4Hu/te7j7D3f8RRAwiUTBj/DD+eHoVb9Rv5szrltCU+MjlPImA2sbkWtmDI7ZWdprmOBYJudl7VHD5KfvywtsbmXdjtZZUjaAod40FJQqRSDh86hgunTudJ1e9x1cXPqtV8iImyqOyQYlCJDKO328cPzx2Kn9/qYZv/eV5Ojo86JCkj2obo92iiOYJM5EB6kuzJtDQ3Mal97/CoKICfnKc1rIIu62jsiPcolCiEImYCw+dyKZEG1c+vIrighjfO3ovJYsQa0y0saW1PZJLoKYpUYhEjJnxrcMnk2jrYP7jqykuiPGdI6coWYRUegzFqHKdehKRfmRm/PfnPkZLWwdXPfo6xQUxvqmFj0KptjHao7JBiUIkssyM7x8zldb2Di7752sUxmNcdJhm7A+brS0KnXoSkSDEYsZPjtublrYOfvngqxQWxLRKXshsbVGo15OIBCUWMy794gxaO5xL7n2Zgphx7sG7Bx2WpNQ2JCgpjFEe0VHZoEQhkhfiMePXJ86grb2DH93zEoXxGGfMnhB0WEJ6ZbuSSHc20IA7kTxREI9x2cn78pm9RvO9O1ewYPGbQYckJKcYj3LXWFCiEMkrhfEYV5yyL5+aMoqLb1/OzU+vCTqkAa8u1aKIMiUKkTxTXBDn96ftxycnV/Kff3uBW5coWQSptjFBZYS7xoIShUheSiaLmczZs5Lv/O0F/rzkraBDGpA2JdpoSrRFumssKFGI5K2SwjhXfWkmn9ijgm//bRl/rlay6G/pBYtG69STiIRVSWGcP55exSf2qOA//rqMhYt1Gqo/1abXylaLQkTCLJ0sPjm5kv+6/QWue2J10CENGB8sgaoWhYiEXPI0VBWHTx3N9+96kaseWRV0SANCTapFoe6xIhIJRQUxrjhlPz4/fSw/vfdlfvP3V3HX4ke5VNeYoKggxtBBhUGHskMCGZltZv8KnAs48AJwlrs3BxGLyEBSGI/x25P2paQwzm/+vpK6xgQ/OHYa8Vh0Rw2HWW1jgsqy4kiPyoYAWhRmtjPwNaDK3acBceCk/o5DZKCKx4xL507n/DkTWbB4DRcuWEpza3vQYeWl2sboj8qG4E49FQCDzKwAKAXWBRSHyIBkZnznyCl89/N7cf+KGk6f/zQbN7cGHVbeqWmI/qhsCCBRuPvbwC+ANcA7wEZ3f6DrfmY2z8yqzay6rq6uv8MUGRDO/sRuXHbyvjz71ga+eNWTvLNxS9Ah5ZXahubId42FYE49DQeOBXYDdgIGm9lpXfdz96vdvcrdqyorK/s7TJEB45gZO3HDWQew7v1mTrjySVbWNAYdUl5obm2nobkt0ivbpQVx6unTwGp3r3P3VuBvwOwA4hCRlNl7VHDLvINoaXfm/mER1W+sDzqkyKtLj6GI8IJFaUEkijXAQWZWasmuAIcBLwUQh4h0Mm3nodx+4WxGDC7i1GsW88CKd4MOKdLSYyjUotgO7r4Y+CvwDMmusTHg6v6OQ0Q+avyIUv56/iymjCnny39ayjWPva6xFtspX0ZlQ0C9ntz9e+4+xd2nufuX3D0RRBwi8lEjy4q5ed5BHL7XGH50z0v81+3LaW3vCDqsyHl3Y37M8wQamS0i3SgtKuDKU/fjgkMncvPTazjzOnWf3VbL395IRVkRIwcXBR3KDlOiEJFuxWLGt4+YwqVzp/P06vUcd+UTrKprCjqsyKh+cwMzdx0e+VHZoEQhIr34YtV4Fpx7EO9vaeULv3uCh16pDTqk0KttbGbN+s1U7Toi6FCyQolCRHp1wG4juOMrH2fc8FLOvn4JVz2yShe5M3jmzQ0A7Lfr8IAjyQ4lChHpk/EjSrntglkcNS05++w3bn2OTYm2oMMKpeo3NlBUEGPazkOCDiUrlChEpM9Kiwq44pR9+bfP7Mmdz6/j6MsfZ/nbG4MOK3Sq39zAjHFDKS6IBx1KVihRiMg2MTMuOmwSC889iE0tbRx/5ZPMf3y1TkWlNLe2s2LdRmbmyfUJUKIQke00a+JI7v36IRyyZyU/vPtFzrmhmg2bWoIOK3DL1m6ktd2ZmSfXJ0CJQkR2wIjBRfzx9Jl8/5ipPL6yns9d9hjPrtkQdFiBqn4zOU+WEoWISIqZccbsCdx2wWxiMePEqxZx3RMD91TU0jc2sHvlYEbkwUC7NCUKEcmKvccN5Z6LDmbOnpV8/64X+crCZwbcaO6ODmfpmg1U5VFrApQoRCSLhpYWcvWXqvjOkVO4f0UNh//mUR5fWR90WP3m9fpNvL+5Na9OO4EShYhkWSxmnD9nIrdfOJvBxXFOm7+Y/3fnCra05P+63Eu3Xp/Inx5PoEQhIjkyfdww7vnawZz18Qlc/+QbfO7yx1i29v2gw8qp6jc2MKy0kImVg4MOJauUKEQkZ0oK43zv6KksOPdAtrS0c/yVT3L5P1bSlqfTli99cwMzd8mPiQA7U6IQkZz7+B4V3Pf1Qzhq77H88sFXOfGqRbz53qagw8qq9ZtaeL1+EzMn5Nf1CVCiEJF+MrS0kMtO3pffnrQPr9U2ceRvH2PB4jfzphvt0tREgPkyY2xnShQi0q+O3Wdn7vvGIey7yzAuvn05Z163ZOtqcFF229K1FBfEmD5uaNChZJ0ShYj0u52GDeKmsw/kB8dOZfHq9/jsrx/hjufejmzr4u5l67hvxbt87bBJlBTmx0SAnSlRiEggYjHj9FkT+L+vHczEUWV8/ZbnOP3ap3mttjHo0LZJfVOC796xgunjhvLlQ3YPOpyc6PdEYWaTzey5TrcGM/tGf8chIuGwe2UZfz1/Nt/9/F4899b7HPGbx/jR3S/S0ByNUd3fu2MFTc1tXDp3BgXx/Pzt3e+1cvdX3H0fd98HmAlsBm7v7zhEJDziMePsT+zGQ/9+KHNnjmP+E6v51C8e5tYla2jvCO/pqHuWvcM9L7zD1z89icljyoMOJ2eCTn+HAavc/c2A4xCREKgoK+aSE6Zzx1c+zi4jSvn2bS9w9OWPs2jVe0GH9hHvNSX4nzuWs/fO+XvKKS3oRHEScHN3T5jZPDOrNrPqurq6fg5LRII0fdwwbrtgNpefvC8bt7Ry8h+f4ss3VbO6PhxjL9raO/jXPz9PY3Mrv/hi/p5ySrOgehmYWRGwDpjq7jWZ9q2qqvLq6ur+CUxEQqW5tZ1rHnudKx9eRUtbB6ccuAtfO2wSFWXFgcTj7lz8v8tZuHgNlxy/NycdsEsgcfSFmS1196odfZ8g0+CRwDO9JQkRGdhKCuN89VOTePhbh/Iv+49nweI1zPn5Q1z2j5U0BnDB+5rHVrNw8RouOHRiqJNENgXZorgFuN/dr+ttX7UoRCRtVV0TP7/vZe5fUUNpUZxj99mZUw/chWk7536g233L3+WCBUs5atpYLj95X2KxcM/plK0WRSCJwsxKgbeA3d19Y2/7K1GISFfL1r7PTYve5K5l62hu7WCf8cM4Yb+dOXzqGEYNKcn68Z5evZ7Tr13MlDFDuGXeQZEYWBfpRLGtlChEpCcbN7fyt2fXsnDxGlbWNmEGM3cZzpF7j+WovccwduigHXr/+qYEv7j/FW6tfovxw0v524WzA7s+sq2UKEREOnF3VtY2ce8L73Lv8nd4+d3kCO8DJozg6BljOWrvsYzchj/wre0d3LjoTX7z91fZ0tLOWR+fwEWHTWJISWGuqpB1ShQiIhmsrt/E3c+v487n17Gytol4zJi5y3Bm7zGS2RMr2Gf8MIoKPtyfZ/2mFh59tY6HXqnl0Vfr2LC5lTl7VvI/n9+LPUaVBVST7adEISLSB+7OKzWN3PX8Oh59tZ7l6zbiDoMK44wdVkJ7h9PW7rR3ODWNzbjDyMFFzJlcyTEzdmLOnpWRXYgoW4miIBvBiIiElZkxZcwQpowZwrcOT17TeGr1eyxa9R51TQkKYkY8ZsTNGDe8lE9OqWTaTkND36OpPylRiMiAMrS0kMOnjuHwqWOCDiUy8nvcuYiI7DAlChERyUiJQkREMlKiEBGRjJQoREQkIyUKERHJSIlCREQyUqIQEZGMIjGFh5nVAV3X1R4KdJ2ivGtZ58e9bVcA9TsQZnfx9HWfMNVlR+rR3XO91a277c5lQdVlWz+Tro+71kXfr8wx9nWffKlLtr9f0H1ddnX3yl5i7J27R/IGXN1bWefHvW0D1dmOp6/7hKkuO1KP3uLehvg7lwVSl239THqri75fufl+RbUu2f5+ZeM7lukW5VNPd/Wh7K5t3M52PH3dJ0x12ZF6dPdcb3XrbjuKn0nXx1GuS5S+X92VRaEuYft+ZRSJU0/9wcyqPQuzLIaB6hI++VIPUF3CKpd1iXKLItuuDjqALFJdwidf6gGqS1jlrC5qUYiISEZqUYiISEZ5mSjM7FozqzWz5dvx2plm9oKZvWZml1mnpa3M7EQze9HMVpjZwuxG3WM8Wa+LmZ1pZnVm9lzqdm72I/9ILDn5TFLPzzUzN7N+Odeco8/k/FT5c2b2uJntlf3Iu40nF3X5ZurfyTIz+4eZ7Zr9yLuNJxd1OcTMnjGzNjObm/2oPxTDdsffw/udYWYrU7czOpXvZmaLU+W3mllRr2+Wq+5UQd6AQ4D9gOXb8dqngVmAAfcCR6bKJwHPAsNTj0dFuC5nAldE/TNJPVcOPAo8BVRFtS7AkE77HAPcF+G6fBIoTW1fANwa4bpMAKYDNwJzwxg/8DAwoUvZCOD11P3w1Hb6b9efgZNS238ALujtGHnZonD3R4H1ncvMbKKZ3WdmS83sMTOb0vV1ZjaW5D/YRZ78v3gj8IXU0+cBv3P3Dalj1Oa2Fkk5qku/y2E9fgj8HGjOYfgfkou6uHtDp10HA/1y8TBHdXnI3Tendn0KGJfbWiTlqC5vuPsyoCOs8ffgcOBBd1+f+pv1IHBEqqX0KeAvjrj6AAAF0klEQVSvqf1uoA9/F/IyUfTgauAid58J/DtwZTf77Ays7fR4baoMYE9gTzN7wsyeMrMjchptZjtaF4ATUqcG/mpm43MXakY7VA8z2xcY7+535zrQPtjhz8TMvmJmq0gmvq/lMNbeZOP7lXYOyV/oQclmXYLQl/i7szPwVqfH6TqNBN5397Yu5RkNiDWzzawMmA38pdPp7eLudu2mLP3LroDk6adDSf5CeszMprn7+9mNNrMs1eUu4GZ3T5jZ+SR/VXwq27FmsqP1MLMY8GuSp9EClaXPBHf/HfA7MzsF+G/gjG72z6ls1SX1XqcBVcCcbMbYV9msSxAyxW9mZwFfT5XtAfyfmbUAq939OHqu03bVdUAkCpItp/fdfZ/OhWYWB5amHt4J/J4PN5PHAetS22uBp9y9FVhtZq+QTBxLchl4N3a4Lu7+XqfyPwI/y1m0PdvRepQD04CHU/+IxgB3mtkx7l6d49i7ysb3q7NbUvsGISt1MbNPAxcDc9w9kdOIe5btz6W/dRs/gLtfB1wHYGYPA2e6+xuddllL8kdt2jiS1zLqgWFmVpBqVfStrrm8OBPkjeRFqOWdHj8JfDG1bcCMHl63BDiIDy5qHZUqPwK4IbVdQbJZNzKidRnbaZ/jSCbAyNWjyz4P008Xs3P0mUzqtM/R5HDenn6oy77Aqs51impdOj1/PTm+mL298dPzxezVJC9kD09tj0g99xc+fDH7wl7j6u8Psp++LDcD7wCtJDPrOcBuwH3A88CLwHd7eG0VsDz1Rb+CDwYlGvCr1GtfSP+PjmhdfgqsSL3+IWBKFOvRZZ+H6b9eT7n4TH6b+kyeS30mUyNcl78DNam6PAfcGeG67J96r03Ae8CKsMVPN4kiVX428Frqdlan8t1J9vJ6jWTSKO4tNo3MFhGRjAZSrycREdkOShQiIpKREoWIiGSkRCEiIhkpUYiISEZKFBJJZtbUz8e7JlszuppZuyVniV1uZneZ2bBe9h9mZhdm49gi20PdYyWSzKzJ3cuy+H7pkao51zl2M7sBeNXdf5xh/wnA3e4+rT/iE+lKLQrJG2ZWaWa3mdmS1O3jqfIDzOxJM3s2dT85VX6mmf3FzO4CHjCzQ83s4dREiS+b2YLUbJukyqtS201m9mMzez41QeToVPnE1OMlZvaDPrZ6FvHBJIdllly/4RlLro1wbGqfS4CJqVbIpal9v5U6zjIz+34W/zeKfIQSheST3wK/dvf9gROAa1LlLwOHuPu+wHeBn3R6zSzgDHdPT4q4L/ANYC+SI1g/3s1xBpOc9mQGybUwzut0/N+mjt/r/DmpOYcOIznfECSnSj/O3fcjuabDL1OJ6jvAKnffx92/ZWafJTnP2AHAPsBMMzukt+OJbK+BMimgDAyfBvbqNNPmEDMrB4YCN5jZJJIzZRZ2es2D7t55DYCn3X0tgJk9R3Lunce7HKcFSE9tvhT4TGp7Fh/M7b8Q+EUPcQ7q9N5LSa4VAMlpYn6S+qPfQbKlMbqb1382dXs29biMZOJ4tIfjiewQJQrJJzFglrtv6VxoZpcDD7n7canz/Q93enpTl/foPNNpO93/G2n1Dy7u9bRPJlvcfR8zG0oy4XwFuAw4FagEZrp7q5m9AZR083oDfuruV23jcUW2i049ST55APhq+oGZpadnHgq8ndo+M4fHf4rkKS+Ak3rb2d03klyg6N/NrJBknLWpJPFJIL3WdCPJadXT7gfOTq1XgJntbGajslQHkY9QopCoKjWztZ1u3yT5R7cqdYH3ReD81L4/B35qZk8A8RzG9A3gm2b2NDAW2NjbC9z9WZIzg54ELCAZfzXJ1sXLqX3eA55Idae91N0fIHlqa5GZvUByWcvybg8gkgXqHiuSJWZWSvK0kpvZScDJ7n5sb68TCTtdoxDJnpnAFameSu+TXA9AJPLUohARkYx0jUJERDJSohARkYyUKEREJCMlChERyUiJQkREMlKiEBGRjP4/mZLll7dy/PwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.lr_find()\n",
    "learn.recorder.plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 1e-2\n",
    "lr *= bs/48  # Scale learning rate by batch size\n",
    "# 所有层都是可以训练的\n",
    "learn.unfreeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 58:32 <p><table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1.944244</td>\n",
       "      <td>1.763525</td>\n",
       "      <td>0.674630</td>\n",
       "      <td>05:50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1.773503</td>\n",
       "      <td>1.668512</td>\n",
       "      <td>0.680306</td>\n",
       "      <td>05:51</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>1.766582</td>\n",
       "      <td>1.624066</td>\n",
       "      <td>0.685923</td>\n",
       "      <td>05:51</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1.641661</td>\n",
       "      <td>1.605911</td>\n",
       "      <td>0.688057</td>\n",
       "      <td>05:50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>1.734819</td>\n",
       "      <td>1.585995</td>\n",
       "      <td>0.689517</td>\n",
       "      <td>05:50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1.663317</td>\n",
       "      <td>1.569859</td>\n",
       "      <td>0.692416</td>\n",
       "      <td>05:50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>1.549388</td>\n",
       "      <td>1.553046</td>\n",
       "      <td>0.695026</td>\n",
       "      <td>05:50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>1.562497</td>\n",
       "      <td>1.537944</td>\n",
       "      <td>0.696345</td>\n",
       "      <td>05:51</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>1.598402</td>\n",
       "      <td>1.531919</td>\n",
       "      <td>0.697588</td>\n",
       "      <td>05:51</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>1.475013</td>\n",
       "      <td>1.531701</td>\n",
       "      <td>0.697916</td>\n",
       "      <td>05:51</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(10, lr, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 保存模型权重与字典"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[PosixPath('/home/lab/.fastai/data/zhwiki/log'),\n",
       " PosixPath('/home/lab/.fastai/data/zhwiki/train.csv'),\n",
       " PosixPath('/home/lab/.fastai/data/zhwiki/zhwiki'),\n",
       " PosixPath('/home/lab/.fastai/data/zhwiki/zh_textlist_class'),\n",
       " PosixPath('/home/lab/.fastai/data/zhwiki/models'),\n",
       " PosixPath('/home/lab/.fastai/data/zhwiki/docs'),\n",
       " PosixPath('/home/lab/.fastai/data/zhwiki/zhwiki-latest-pages-articles.xml'),\n",
       " PosixPath('/home/lab/.fastai/data/zhwiki/val.csv'),\n",
       " PosixPath('/home/lab/.fastai/data/zhwiki/zh_databunch'),\n",
       " PosixPath('/home/lab/.fastai/data/zhwiki/zhwiki-latest-pages-articles.xml.bz2'),\n",
       " PosixPath('/home/lab/.fastai/data/zhwiki/wikiextractor')]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "path.ls()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "mdl_path = path/'models'\n",
    "mdl_path.mkdir(exist_ok=True)\n",
    "learn.to_fp32().save(mdl_path/lm_fns_bwd[0], with_opt=False)\n",
    "learn.data.vocab.save(mdl_path/(lm_fns_bwd[1] + '.pkl'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "TEXT = \"中国\"\n",
    "N_WORDS = 40\n",
    "N_SENTENCES = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "中国 > doc / < \n",
      " \n",
      " \n",
      " \n",
      "  。 ” “ ： 》 side The \" \n",
      " \n",
      "  。 ” 》 《 》 《 （ 」 。 》 《 、 》 《 ： \n",
      " \n",
      "  。 》 《 ： 》 《 、 》\n",
      "\n",
      "中国 > doc / < \n",
      " \n",
      " \n",
      "  。 ” ” 。 』 ？ ！ 」 ？ ！ ！ ！ ？ ？ 』 ？ 。 。 ’ ） （ ？ ！ … … review Sales File Groove The\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\\n\".join(learn.predict(TEXT, N_WORDS, temperature=0.9) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 进行语义分析任务backward\n",
    "- 参看 [中文数据集](BERT.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_lm_bwd = load_data(f'{lang}_clas_databunch', bs=bs, backward=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>idx</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>xxunk xxbos xxunk xxbos xxunk xxrep xxunk 。 xxbos 虽说合肥的酒店及服务比较差能理解,但没见过这么差的服务及酒店设施!还四星级呢,简直不如街边的小旅馆!服务员前台和大堂经理的素质特低!房间简直就是垃圾房拼的,而且酒店还说通过携程订的房间就是这种房,而且还是最差的,要么就在酒店前台订,房价一样房间却要好很多 xxrep 7 . 晕!强烈谴责此酒店的做法!坚决不会再入住该酒店!火车站旁边有个邮政宾馆还可以 xxbos 10月25日入住单人大床间一晚,感觉较差:1.晚上22:30左右入住,前台接待很冷淡,无笑容,感觉不亲切;2.最可怕的是冲澡3分钟后淋浴龙头停水,2分钟以后浴缸龙头也完全停水.没办法浴室内打电话给大堂,说派人上来看一下,这时发现房间内未配浴袍.后大堂副理上来,告诉也有其它房间有类似情况,因最近酒店附近管理维修.只是建议给换房间 xxbos 头两晚入住单间,窗式空调以及窗外传来的噪音让人无法入睡。查因房间唯一的窗户外是两栋大厦的夹缝,宽约两米,而两栋大厦的所有空调机都安装于此,可想而知同时启动的噪音有多么的轰动.但正值元旦假期,只能屈就.第三天转为双人间,犹如从地狱转往天堂,空调是分体的,远离夹缝,面积大两倍,还可远眺香港城貌,房价只交多100元.建议有意入住该酒店的朋友哪怕一个人也选择双人间,这里的地理位置实在棒极了. xxbos 优点就是地理位置，到外滩和南京路走路10分钟。房间比较小，楼层举架低，有点压抑。卫生间毛巾也许消过毒，但很旧，颜色发黄，没办法，我用的自带毛巾。服务人员都不太热情，办理入住的女孩子们还好，大堂前的服务生，问个路什么的回答问题时基本不正眼看我，很锻炼我的脾气，有利于修身养性 。 xxbos xxunk xxbos xxunk ！ xxbos xxunk / xxunk 。 xxbos xxunk 。 xxbos xxup xxunk ！ xxbos xxunk 。 xxbos xxunk 。 xxbos xxunk 。 。 。 xxbos 一个词 xxrep 4 - \" 大失所望\"!!!去广州之前,在携程上研究了半天,后来应老公的要求选择了elan,因为他从欧洲带过来的那本\"china xxrep 4 - lonelyplanet\"上极尽完美之辞的推荐了该酒店.网上看图片和介绍,觉得应该也不错(没有查点评),我当时认为应该和\"有一家\"的水平差不多.可是一进房间,发现和图片实在相差太远.房间小得可怜,没有地方放行李,甚至连坐的地方都没有.一张小得可怜的写字台上有一台小得可怜的\"液晶电视\".如果你想躺在床上看电视,恐怕需要一双\"千里眼\";如果你想使用那张写字台,那你一定需要很苗条,才能坐在那个位置,并且需要带一台小于12寸的笔记本，否则是放不下的.房间也不干净,首先进去的那间330元的大床房,马桶里居然还漂着头发.后来立马换了间560元的套房,其实也就稍微大一点点,多了个坐的地方,有沙发和桌子及大一点的普通电视.房间设施很陈旧,地板有些地方已经翘起来了,隔音不好,热水水压也不够 xxrep 5 . 前台服务还不错,服务员能说英语;但是压根没见到行李生.个人感觉,不值那个价钱;个人意见,不推荐入住 ! xxbos xxunk 。 xxbos xxunk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4 . xxunk xxrep 4 . xxunk xxrep 5 . xxbos xxunk 。 xxbos 很老的酒店，箱柜旧的都没法看了，前台服务跟门童都值得夸奖。不然真的是一无是处了。270的价格让人诧异，有点疯。不会再去了 。 xxbos xxunk ！ xxbos xxunk ！ xxbos xxunk xxrep 8 ！ xxunk ！ xxbos 隔音效果非常非常非常的差，睡在房间里象睡在马路边一样，路上声音听的清清楚楚。我还是看了携程上该酒店的点评，采纳网友意见预定了5楼以上的房间。结果还是非常的吵，到晚上12点还是睡不着，只好打开电视以掩盖马路上的车声。不知道几点睡着了，一大早就被车声吵醒，一看表，才早上4点！才4点啊，也就是说就算晚上12点我睡着了，也才睡了4个小时。头疼啊，白天还要工作啊，闭上眼睛努力睡，怎么样也睡不着啊，太吵了。坚持到5：30实在是受不了了，打电话给前台，问有没有安静点的房间可以换，现在不能换晚上换也可以，答曰：没有。盯着天花板熬到6：30，起床。在早餐厅灌了3杯咖啡后，马上去退房。本来是要住三天的，实在是受不了这个吵啊。真不明白这样差的条件怎么在携程上有4.0的评分。另：1.电视播放太复杂，不方便2.服务员服务意识不强，没有为客人着想、以解决问题为本的意识。体现在早餐券、开发票等问题上3、网线太短，大概就20厘米长吧，不能在床上上网4.电源接触不好，电脑电源连到插座上时火花噼里啪啦的冒，当时我想：因为电源问题损坏电脑不知道酒店赔不赔再次表示疑问：这样的酒店上怎么会有4、0的高分数，宾馆反馈2008年3月30日：首先感谢您的入住并提出宝贵的意见，由于房间的环境问题影响到您的休息为此我们深表歉意。对于您提出的一些问题，我们会酌情整改，另外酒店早餐是凭房卡用的，没有早餐券；我们客房中心随时都为一些有特殊要求的客人准备了较长的网线，如果有需要只要一个电话，马上送到房间。非常感谢您的宝贵意见，真诚期待您能再次光临，给我们一个能为您提供更好服务的机会 。 xxbos xxunk ! xxbos xxunk 。 xxbos xxunk xxbos xxunk / xxunk / xxunk xxbos 好小的门面，没有电梯，房间也不是很一致！豪华房居然要400多，马桶还是坏的！酒店太自作主张了 xxrep 4 。 xxbos xxunk ! xxbos xxunk 。 xxbos 好小的门面，没有电梯，房间也不是很一致！最郁闷的是我不知道我们公司跟他们有合约，他就让我的客人（另外一次）直接用我们公司的名义住的酒店，而没有通过ctrip。还好第二天ctrip打电话给我确认，让我退单了 。 xxbos xxunk 。 xxbos xxunk 。 xxbos xxunk 。 xxbos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>。 xxbos xxunk xxbos xxunk 。 xxbos 入住当晚,因火车到达得较晚。到酒店前门一看，感觉就象烂尾楼，黑灯瞎火的，还以为出租车拉错地方了。后面到总台一问说是大堂搞装修。上到房间一看，房间内的设置就更差了。地毯上到处是补丁，一股子霉味，我住的还是豪华标双。不晓得普标哪得成什么样？两百多人民币还不如现如今的经济型酒店，太失望了，货不对板(跟图片上比）。宾馆反馈2008年5月30日：谢谢您的宝贵意见!实在对不起,我代表酒店向您致以诚挚的歉意!您提的这几个问题酒店现在已经全部解决,欢迎您下次入住 ! xxbos 酒店房间太烂。服务一般。地点也差。房间小，卫生间小，居然还是蹲便 。 。 。 xxbos xxunk ！ xxbos xxunk xxrep 8 ！ xxunk xxrep 6 ！ xxbos xxunk 。 xxbos 房间还可以,设施简单,不像是四星标准。早餐还算丰富,不过要早去。前台接待、结算速度和态度还不错。后院停车场很有特点，一晚上有两台车被划伤，真恐怖，最关键的是保安及小头目还不想认帐，气愤啊 xxbos xxunk 。 xxbos 订的豪华标间，738元的价格，住过的同等对比是最差的。周边环境除了一地铁口没有其它，餐厅不行，周边也找不着吃的。房间小，住了两天还有一天晚上没水，宽带收费40，打印贡献了几十个10元。绝对不值 。 xxbos xxup 388元的房间,也就值248元.上网居然收费且人工配制ip.打死也不再住. xxbos xxunk ! xxbos xxunk xxbos 房间巨小，电视成了摆设，开不了，服务员态度冷漠。以后不会再进这家酒店 xxbos xxunk xxbos 请各位注意酒店提供的饮用水是自来水，这是前台服务员说的。所以各位千万不能生喝酒店提供的饮用水。2008年了，北京的酒店还是这种水平，真是令人心寒 。 xxbos 1 \\ xxunk \\ xxunk \\ xxunk \\ xxunk xxbos 酒店比较旧，听说有十几年的历史，房间也比较小。一个所谓三星酒店连一次性的剃须刀都不提供，要另外花15块钱买。总体上还不如火车站周边的商务酒店 。 xxbos xxunk 。 xxbos 临近道路边,相当的吵,隔音效果差</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>- xxunk xxrep 4 . xxunk xxrep 4 . xxup xxunk 。 xxbos xxunk xxbos 根本就是家庭旅馆 .. 还298 + 12基金 ... 我去的时候刚好是国庆他们说在装修,把我安排在前面的民宅一样的地方 .. 刚开始带我去6号楼，妈的根本不是人住的破 .. 然后把我带回8号楼.大一点 .. 而且这个地方离海很远,,到的晚上说明天会用车送我们到海边,第2天下大雨,说车不在，我朋友和他交涉,旅游的心情都被影响 ... 被ctrip的照片骗了 .. 这个破旅馆真的不是人住的!!以后再也不图便宜了!!补充点评2007年10月11日：298的房间ctrip上标有\"宽带免费\" .. 可那里根本不能上网 .. 我带的那么重的电脑就白拿了 .. 我有被骗的感觉 ! ! xxbos xxunk ! xxbos xxup 9月9日入住,从小事即可见一斑：浴帘比浴盆高200毫米，弄得卫生间一地水，功能丧失，不如说是遮羞布；餐厅桌布油迹斑斑，服务员总是理直气壮地回答，昨晚忘记换了。让人不可思议的是房间不能打国际长途！11日晚23时，我要打国际长途，前台首先声明不好使，建议买ic卡打（想赚卡钱吧？）。在我的坚持下开通了国际长途，但拨号后传来了中文提示：这个号码是空号 xxrep 6 . 。 xxup 显然没有“出国”。与前台理论，找值班经理都不解决问题，无奈，找出一张ip卡，试拨17900，无法拨通，又试拨10060（该宾馆号称用网通的电话），也不能拨通。最后用手机拨打了国际长途。电话打过，确久久不能入睡，倒不是心痛电话费过高。而是心中恐怖，万一夜里着火、被盗，我能拨打119、110吗？可爱的正宗三星级酒店！抬眼看日历，啊！9月11日--恐怖的911！！！次日提前退房离去。留下携程与你长期合作 。 xxbos xxunk xxbos 一个字：差！招待所的水平，地毯很葬，卫生间很陈旧。补充点评2007年9月12日：离三孔有一段距离，有停车位。号称其装修和布置是儒家风格，但本人无论如何也没有看出来。据说曲阜的宾馆都是如此水平 。 xxbos xxunk 。 xxbos xxup xxunk 。 xxbos 这家酒店太烂了。特别是服务差的很，简直让人火大。我通过携程预定了早餐，结果被告知从七点钟才开始，那我说我退掉吧，结果告诉我携程订了就不能退。早晨在房间打电话告诉前台查房，结果我到前台后，前台还没打，等到再次说退房时，才磨磨蹭蹭的打电话，结果等了几分钟还是没人查。等签单时才发现自己名字的两个字竟然被搞反了，真搞不懂，我实在是不敢恭维，而且莫名其妙的在我的名字后面竟然出现了另一个人名。建议各位不要入住。至所以给服务打两分，是因为其餐厅给我送了碗姜汤，比较感动，要不然服务肯定的一分 。 xxbos xxunk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>... xxunk xxrep 4 . xxunk xxbos xxunk xxrep 5 ！ xxbos xxunk 。 xxbos xxunk xxbos xxunk 。 xxbos xxunk 。 xxbos 第一住的房都是味，地上有好香灰，后了另外一味那麽重！建酒店清生必注意，及最好有非吸 xxbos xxunk 。 xxbos xxup xxunk 。 xxbos xxup xxunk ！ xxbos xxunk xxbos xxunk xxrep 6 ~ xxunk xxbos 这家酒店太烂了。特别是服务差的很，简直让人火大。我通过携程预定了早餐，结果被告知从七点钟才开始，那我说我退掉吧，结果告诉我携程订了就不能退。早晨在房间打电话告诉前台查房，结果我到前台后，前台还没打，等到再次说退房时，才磨磨蹭蹭的打电话，结果等了几分钟还是没人查。等签单时才发现自己名字的两个字竟然被搞反了，真搞不懂，我实在是不敢恭维，而且莫名其妙的在我的名字后面竟然出现了另一个人名。建议各位不要入住。至所以给服务打两分，是因为其餐厅给我送了碗姜汤，比较感动，要不然服务肯定的一分 。 xxbos 前台手续复杂且慢，用两个房间要签三次字；有客人刚办完入住手续就发现房间冷而要求不住的，服务人员坚持按已经住过收费，让人感到是“可下子抓住一个客人决不放过”；我们入住和离开时分别看到有客人不小心弄脏了毛巾、床单，看上去洗一下也就没事了，可服务员一定要客人赔偿，且价格不低，一条白床单要价180元；携程的价格高于该店的优惠价 。 xxbos xxunk xxbos 房间和走廊里味道很重，楼外下面马路上的声音很吵，早上隔壁的电视机的声音听的很清楚，可以当闹钟使用，但时间是隔壁定的，没有给“闹钟”定时，结帐的时候发票没有给我盖章，总的感觉是让人失望。今后不会再住了 。 xxbos xxunk xxrep 4 ！ xxunk ！ xxbos xxunk 。 xxbos xxunk 。 xxbos xxunk 。</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data_lm_bwd.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 使用我们之前训练的模型学习一个语言模型"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `pretrained_fnames=lm_fns`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['zh_wt', 'zh_wt_vocab']"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lm_fns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm = language_model_learner(data_lm_bwd, AWD_LSTM, pretrained_fnames=lm_fns_bwd, wd=0.1, drop_mult=1.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 1e-3\n",
    "lr *= bs/48"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:01 <p><table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>6.697358</td>\n",
       "      <td>5.175411</td>\n",
       "      <td>0.212500</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>6.201413</td>\n",
       "      <td>4.778626</td>\n",
       "      <td>0.230804</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.fit_one_cycle(2, lr*10, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:07 <p><table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>5.548280</td>\n",
       "      <td>4.604518</td>\n",
       "      <td>0.240179</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>5.249329</td>\n",
       "      <td>3.720865</td>\n",
       "      <td>0.347321</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>4.763565</td>\n",
       "      <td>2.581959</td>\n",
       "      <td>0.468304</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>4.259766</td>\n",
       "      <td>2.183122</td>\n",
       "      <td>0.579241</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.877409</td>\n",
       "      <td>2.004621</td>\n",
       "      <td>0.593750</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.585753</td>\n",
       "      <td>1.897657</td>\n",
       "      <td>0.604911</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.358438</td>\n",
       "      <td>1.860995</td>\n",
       "      <td>0.610045</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.205450</td>\n",
       "      <td>1.840042</td>\n",
       "      <td>0.615848</td>\n",
       "      <td>00:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_lm.unfreeze()\n",
    "learn_lm.fit_one_cycle(8, lr, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_lm.save(f'{lang}fine_tuned_bwd')\n",
    "learn_lm.save_encoder(f'{lang}fine_tuned_enc_bwd')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "TEXT = '我喜欢'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "我喜欢 > xxbos . > xxbos 。 xxbos ) xxbos / xxbos 。 > / / ( - : . 10 . ) ( ! ! ! ! ! ! ! ! xxbos . ! xxbos\n",
      "\n",
      "我喜欢 / - ! ! 9 ! ! ! ! ! ! xxbos , * 5 ? ? ! 4 ! 4 ? ! 4 ! ! 12 ..... !!!!! !!!!! !!!!! !!!!! !!!!! xxbosxxbosxxbosxxbosxxbos\n"
     ]
    }
   ],
   "source": [
    "print(\"\\n\\n\".join(learn_lm.predict(TEXT, N_WORDS, temperature=0.9) for _ in range(N_SENTENCES)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9000"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(data_lm.train_ds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 分类"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_clas_bwd = load_data(path, f'{lang}_textlist_class', bs=bs, num_workers=4, backward=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "@np_func\n",
    "def f1(inp,targ): return f1_score(targ, np.argmax(inp, axis=-1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c = text_classifier_learner(data_clas_bwd, AWD_LSTM, pretrained=False, drop_mult=0.5, metrics=[f1, accuracy], wd=0.1).to_fp16()\n",
    "learn_c.load_encoder(f'{lang}fine_tuned_enc_bwd')\n",
    "learn_c.freeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr=2e-2\n",
    "lr *= bs/48"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:12 <p><table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>f1</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.477027</td>\n",
       "      <td>0.522003</td>\n",
       "      <td>0.737219</td>\n",
       "      <td>0.745556</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.485022</td>\n",
       "      <td>0.495955</td>\n",
       "      <td>0.741332</td>\n",
       "      <td>0.751111</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.493099</td>\n",
       "      <td>0.518831</td>\n",
       "      <td>0.710202</td>\n",
       "      <td>0.737778</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.484330</td>\n",
       "      <td>0.494661</td>\n",
       "      <td>0.752563</td>\n",
       "      <td>0.754444</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.483695</td>\n",
       "      <td>0.509657</td>\n",
       "      <td>0.745997</td>\n",
       "      <td>0.745556</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>0.481792</td>\n",
       "      <td>0.494413</td>\n",
       "      <td>0.743542</td>\n",
       "      <td>0.748889</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>0.468664</td>\n",
       "      <td>0.899974</td>\n",
       "      <td>0.735527</td>\n",
       "      <td>0.748889</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>0.472452</td>\n",
       "      <td>0.513543</td>\n",
       "      <td>0.745006</td>\n",
       "      <td>0.751111</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/lab/Softwares/miniconda3/envs/fastai/lib/python3.6/site-packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: F-score is ill-defined and being set to 0.0 due to no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    }
   ],
   "source": [
    "learn_c.fit_one_cycle(8, lr, moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:05 <p><table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>f1</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.481273</td>\n",
       "      <td>0.493253</td>\n",
       "      <td>0.769983</td>\n",
       "      <td>0.750000</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.479506</td>\n",
       "      <td>0.586876</td>\n",
       "      <td>0.763711</td>\n",
       "      <td>0.740000</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.471515</td>\n",
       "      <td>0.503489</td>\n",
       "      <td>0.751259</td>\n",
       "      <td>0.748889</td>\n",
       "      <td>00:01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.freeze_to(-2)\n",
    "learn_c.fit_one_cycle(3, slice(lr/(2.6**4),lr), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:08 <p><table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>f1</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.485243</td>\n",
       "      <td>0.508445</td>\n",
       "      <td>0.761227</td>\n",
       "      <td>0.744444</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.473014</td>\n",
       "      <td>0.498693</td>\n",
       "      <td>0.757239</td>\n",
       "      <td>0.753333</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.470699</td>\n",
       "      <td>0.495109</td>\n",
       "      <td>0.762190</td>\n",
       "      <td>0.747778</td>\n",
       "      <td>00:02</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.freeze_to(-3)\n",
    "learn_c.fit_one_cycle(3, slice(lr/(2.6**4),lr), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "Total time: 00:19 <p><table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>f1</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.466742</td>\n",
       "      <td>0.570398</td>\n",
       "      <td>0.752874</td>\n",
       "      <td>0.753333</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.462775</td>\n",
       "      <td>0.502431</td>\n",
       "      <td>0.754329</td>\n",
       "      <td>0.751111</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0.460959</td>\n",
       "      <td>0.502710</td>\n",
       "      <td>0.754793</td>\n",
       "      <td>0.752222</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0.458740</td>\n",
       "      <td>0.520910</td>\n",
       "      <td>0.754775</td>\n",
       "      <td>0.752222</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0.455853</td>\n",
       "      <td>0.500027</td>\n",
       "      <td>0.761132</td>\n",
       "      <td>0.752222</td>\n",
       "      <td>00:03</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn_c.unfreeze()\n",
    "learn_c.fit_one_cycle(5, slice(lr/10/(2.6**4),lr/10), moms=(0.8,0.7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c.save(f'{lang}clas_bwd')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ensembling the two models¶\n",
    "平均 forward and the backward models的结果. Since the samples are sorted by text lengths for batching, we pass the argument ordered=True to get the predictions in the order of the texts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_clas_fwd = load_data(path, f'{lang}_textlist_class', bs=bs, num_workers=4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c_fwd = text_classifier_learner(data_clas_fwd, AWD_LSTM, pretrained=False, drop_mult=0.5, wd=0.1, metrics=[f1, accuracy]).to_fp16()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn_c_fwd.load(f'{lang}clas')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred_fwd, lbl_fwd = learn.get_preds(ordered=True)\n",
    "pred_bwd, lbl_bwd = learn_c.get_preds(ordered=True)\n",
    "final_pred = (pred_fwd+pred_bwd)/2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy(final_pred, lbl_fwd)， f1(final_pred, lbl_fwd)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}