SentencePiece_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SentencePiece Demo\n",
    "This notebook will go through some examples of how to setup and use the SentencePiece subword library"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import our libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sentencepiece as spm\n",
    "import re\n",
    "import os\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inspect the dataset\n",
    "Lets take a look at the blogger data we have downloaded and see what we can use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4162441.male.16.Student.Sagittarius.xml\n",
      "3489929.female.25.Student.Cancer.xml\n",
      "3954575.female.23.BusinessServices.Gemini.xml\n",
      "3364931.male.16.Student.Virgo.xml\n",
      "3162067.female.24.Education.Cancer.xml\n",
      "813360.female.23.BusinessServices.Capricorn.xml\n",
      "4028373.female.17.indUnk.Leo.xml\n",
      "3630901.male.34.Technology.Leo.xml\n",
      "2467122.female.23.Student.Taurus.xml\n",
      "3732850.female.45.Technology.Taurus.xml\n",
      "3846432.male.16.Student.Leo.xml\n",
      "3600967.female.33.Arts.Scorpio.xml\n",
      "3753301.female.14.Non-Profit.Aries.xml\n",
      "4157968.male.16.Student.Pisces.xml\n",
      "3699514.male.34.InvestmentBanking.Capricorn.xml\n",
      "2727849.female.26.Arts.Libra.xml\n",
      "3791552.female.24.indUnk.Virgo.xml\n",
      "4278694.female.24.Technology.Virgo.xml\n",
      "1618178.male.14.Arts.Scorpio.xml\n",
      "669719.male.26.Science.Taurus.xml\n"
     ]
    }
   ],
   "source": [
    "files = os.listdir('/floyd/input/blog_dirs/blogs')\n",
    "for i in range(20):\n",
    "    print(files[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example blog text\n",
    "Below we can see the output of an example blog. It looks good, we have misspellings and names and a variety of words and phrases. It should generate some interesting results. Only thing we need to do is remove the blog tags \"<>\" so we are not inclduing them in the training"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "<Blog>\n",
    "\n",
    "<date>30,June,2004</date>\n",
    "<post>\n",
    "\n",
    "\n",
    "\n",
    "        sighs.going to leave my xanga for this blog.haixXx.all my nice entries are there.maybe i'll try to transfer them in here.having training tml at complex.i wish that the night training is gonna start soon and that wed and sat training will be gone in no time.aiya anyway it hasn't even started.really want to thank that boy for injuring his leg so that my class could move to the top flour for lessons.haha.feel so bad.i just got to walk a few steps away and would be able to see my seniors.muhuahua.always seeing them go to the toilet to skip lessons.   ass.so many days and ur inbox is still full.how i send that song??damn.so many days has past.\n",
    "\n",
    "</post>\n",
    "\n",
    "<date>28,July,2004</date>\n",
    "<post>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = []\n",
    "# Note you can grab as many files as you like. \n",
    "# I am just using 10 here as an example.\n",
    "for i in range(10):\n",
    "    for line in open(\"/floyd/input/blog_dirs/blogs/{}\".format(files[i]), encoding=\"latin-1\").readlines():\n",
    "        if re.search(\"<\", line) or len(line) < 5:\n",
    "            continue\n",
    "        else:\n",
    "            text.append(line)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Write the data to a file\n",
    "This will make it easy to pass into SentencePiece"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('blog_test.txt', 'w') as fw:\n",
    "    for l in text:\n",
    "        fw.write(l)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train a BPE model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# train sentencepiece model from our blog corpus\n",
    "spm.SentencePieceTrainer.train('--model_type=bpe --input=blog_test.txt --model_prefix=bpe --vocab_size=500 --normalization_rule_tsv=normalization_rule.tsv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train a Unigram model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# train sentencepiece model from our blog corpus\n",
    "spm.SentencePieceTrainer.train('--model_type=unigram --input=blog_test.txt --model_prefix=uni --vocab_size=500 --normalization_rule_tsv=normalization_rule.tsv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load your newly trained models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# makes segmenter instance and loads the BPE model file (bpe.model)\n",
    "sp_bpe = spm.SentencePieceProcessor()\n",
    "sp_bpe.load('bpe.model')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# makes segmenter instance and loads the BPE model file (bpe.model)\n",
    "sp_uni = spm.SentencePieceProcessor()\n",
    "sp_uni.load('uni.model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Look at some example tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BPE: ['▁This', '▁is', '▁a', '▁t', 'est']\n",
      "UNI: ['▁Thi', 's', '▁is', '▁a', '▁t', 'est']\n"
     ]
    }
   ],
   "source": [
    "print(\"BPE: {}\".format(sp_bpe.encode_as_pieces('This is a test')))\n",
    "print(\"UNI: {}\".format(sp_uni.encode_as_pieces('This is a test')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BPE: ['▁This', '▁is', '▁a', '▁t', 'est']\n",
      "UNI: ['▁Thi', 's', '▁is', '▁a', '▁t', 'est']\n"
     ]
    }
   ],
   "source": [
    "print(\"BPE: {}\".format(sp_bpe.encode_as_pieces(' This is a test')))\n",
    "print(\"UNI: {}\".format(sp_uni.encode_as_pieces(' This is a test')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BPE: ['▁I', '▁think', '▁this', '▁is', '▁a', '▁t', 'est']\n",
      "UNI: ['▁I', '▁think', '▁this', '▁is', '▁a', '▁t', 'est']\n"
     ]
    }
   ],
   "source": [
    "print(\"BPE: {}\".format(sp_bpe.encode_as_pieces('I think this is a test')))\n",
    "print(\"UNI: {}\".format(sp_uni.encode_as_pieces('I think this is a test')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BPE: ['▁C', 'ar', 'b', 'on', '▁d', 'i', 'o', 'x', 'ide']\n",
      "UNI: ['▁C', 'ar', 'b', 'on', '▁d', 'i', 'o', 'x', 'id', 'e']\n"
     ]
    }
   ],
   "source": [
    "print(\"BPE: {}\".format(sp_bpe.encode_as_pieces('Carbon dioxide')))\n",
    "print(\"UNI: {}\".format(sp_uni.encode_as_pieces('Carbon dioxide')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get a list of all of the BPE tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['▁something',\n",
       " '▁because',\n",
       " '▁thought',\n",
       " '▁really',\n",
       " '▁people',\n",
       " '▁little',\n",
       " '▁things',\n",
       " '▁friend',\n",
       " '▁should',\n",
       " '▁think',\n",
       " '▁about',\n",
       " '▁would',\n",
       " '▁there',\n",
       " '▁thing',\n",
       " '▁going',\n",
       " '▁every',\n",
       " '▁other',\n",
       " '▁somet',\n",
       " '▁could',\n",
       " '▁essay',\n",
       " '▁their',\n",
       " '▁being',\n",
       " '▁again',\n",
       " '<unk>',\n",
       " '▁that',\n",
       " '▁have',\n",
       " '▁with',\n",
       " '▁this',\n",
       " '▁like',\n",
       " '▁just',\n",
       " '▁what',\n",
       " '▁when',\n",
       " '▁they',\n",
       " '▁some',\n",
       " '▁want',\n",
       " '▁feel',\n",
       " '▁will',\n",
       " '▁time',\n",
       " '▁know',\n",
       " '▁work',\n",
       " '▁more',\n",
       " '▁them',\n",
       " '▁from',\n",
       " '▁good',\n",
       " 'ation',\n",
       " '▁thou',\n",
       " '▁back',\n",
       " '▁life',\n",
       " '▁then',\n",
       " '▁been',\n",
       " '▁were',\n",
       " 'other',\n",
       " '▁your',\n",
       " '▁year',\n",
       " 'ittle',\n",
       " '▁much',\n",
       " 'riend',\n",
       " '▁need',\n",
       " 'essay',\n",
       " '▁comp',\n",
       " '▁home',\n",
       " '▁make',\n",
       " '▁over',\n",
       " '▁than',\n",
       " '▁even',\n",
       " 'thing',\n",
       " '▁very',\n",
       " '▁take',\n",
       " '▁also',\n",
       " '▁This',\n",
       " '▁into',\n",
       " '▁only',\n",
       " '</s>',\n",
       " '▁the',\n",
       " '▁and',\n",
       " '▁for',\n",
       " 'nbsp',\n",
       " '▁was',\n",
       " '▁not',\n",
       " '▁you',\n",
       " 'ould',\n",
       " '▁but',\n",
       " 'ally',\n",
       " '▁wor',\n",
       " '▁her',\n",
       " '▁get',\n",
       " '▁out',\n",
       " '▁all',\n",
       " '▁can',\n",
       " '▁she',\n",
       " '▁The',\n",
       " '▁are',\n",
       " 'very',\n",
       " 'king',\n",
       " '▁tim',\n",
       " '▁had',\n",
       " 'ight',\n",
       " '▁one',\n",
       " '▁day',\n",
       " '▁how',\n",
       " '▁com',\n",
       " '▁see',\n",
       " 'hing',\n",
       " '▁who',\n",
       " '▁con',\n",
       " '▁bec',\n",
       " '▁has',\n",
       " '▁any',\n",
       " '▁now',\n",
       " '▁did',\n",
       " '▁int',\n",
       " 'ause',\n",
       " '▁pro',\n",
       " 'ther',\n",
       " '▁man',\n",
       " 'ople',\n",
       " '▁his',\n",
       " 'omet',\n",
       " 'ting',\n",
       " 'ving',\n",
       " '▁wee',\n",
       " '▁him',\n",
       " '▁off',\n",
       " '▁say',\n",
       " 'reat',\n",
       " '▁too',\n",
       " '▁got',\n",
       " 'fter',\n",
       " 'ment',\n",
       " '▁way',\n",
       " '▁our',\n",
       " 'self',\n",
       " 'here',\n",
       " 'ning',\n",
       " '▁bet',\n",
       " '▁pre',\n",
       " '▁exp',\n",
       " '▁new',\n",
       " '▁hel',\n",
       " '▁But',\n",
       " '▁try',\n",
       " '<s>',\n",
       " 'ing',\n",
       " '▁to',\n",
       " 'hat',\n",
       " '▁of',\n",
       " '▁ha',\n",
       " '▁be',\n",
       " '▁in',\n",
       " '▁th',\n",
       " '▁he',\n",
       " '▁my',\n",
       " '▁is',\n",
       " '▁it',\n",
       " '▁re',\n",
       " '▁on',\n",
       " '▁we',\n",
       " '▁me',\n",
       " 'ver',\n",
       " '▁li',\n",
       " '▁do',\n",
       " '▁st',\n",
       " 'ith',\n",
       " 'ent',\n",
       " 'all',\n",
       " '▁am',\n",
       " 'ght',\n",
       " 'ion',\n",
       " 'ome',\n",
       " '▁an',\n",
       " 'ter',\n",
       " '▁so',\n",
       " 'ust',\n",
       " 'her',\n",
       " 'ill',\n",
       " '▁wh',\n",
       " '▁at',\n",
       " '▁ne',\n",
       " 'ink',\n",
       " 'ant',\n",
       " 'out',\n",
       " '▁as',\n",
       " 'hen',\n",
       " '...',\n",
       " '▁go',\n",
       " 'ess',\n",
       " '▁mo',\n",
       " '▁ab',\n",
       " 'one',\n",
       " '▁al',\n",
       " '▁sh',\n",
       " 'ood',\n",
       " '▁su',\n",
       " 'ake',\n",
       " '▁or',\n",
       " 'ack',\n",
       " '▁up',\n",
       " '▁ex',\n",
       " 'art',\n",
       " 'ind',\n",
       " 'eel',\n",
       " 'ast',\n",
       " 'ain',\n",
       " '▁kn',\n",
       " '▁de',\n",
       " 'day',\n",
       " 'ate',\n",
       " 'rom',\n",
       " 'use',\n",
       " '▁It',\n",
       " 'end',\n",
       " 'ear',\n",
       " 'and',\n",
       " 'rou',\n",
       " 'ell',\n",
       " 'itt',\n",
       " 'han',\n",
       " '▁lo',\n",
       " 'est',\n",
       " 'ers',\n",
       " 'are',\n",
       " '▁if',\n",
       " 'ven',\n",
       " '▁by',\n",
       " '▁pl',\n",
       " '▁le',\n",
       " '▁sp',\n",
       " 'way',\n",
       " 'ite',\n",
       " 'ong',\n",
       " '▁pe',\n",
       " 'ure',\n",
       " '▁po',\n",
       " 'ive',\n",
       " 'ore',\n",
       " 'ide',\n",
       " '▁ch',\n",
       " '▁ag',\n",
       " '▁us',\n",
       " 'ort',\n",
       " 'our',\n",
       " 'ame',\n",
       " 'ook',\n",
       " 'ist',\n",
       " 'own',\n",
       " '▁no',\n",
       " '▁Th',\n",
       " '▁wr',\n",
       " '▁sa',\n",
       " 'uch',\n",
       " 'ard',\n",
       " '▁cl',\n",
       " 'ass',\n",
       " 'ice',\n",
       " 'ies',\n",
       " 'ond',\n",
       " 'ous',\n",
       " 'ich',\n",
       " '▁se',\n",
       " '▁en',\n",
       " 'ace',\n",
       " 'ool',\n",
       " '▁gu',\n",
       " 'red',\n",
       " 'ick',\n",
       " 'ine',\n",
       " 'ity',\n",
       " '▁He',\n",
       " '▁So',\n",
       " '▁qu',\n",
       " '▁fe',\n",
       " '▁bl',\n",
       " '▁tw',\n",
       " 'alk',\n",
       " 'ild',\n",
       " 'ble',\n",
       " 'ile',\n",
       " '▁im',\n",
       " '▁We',\n",
       " '▁My',\n",
       " '▁ri',\n",
       " 'ose',\n",
       " 'ree',\n",
       " 'ily',\n",
       " '▁t',\n",
       " '▁a',\n",
       " 'he',\n",
       " 'in',\n",
       " '▁w',\n",
       " '▁s',\n",
       " '▁I',\n",
       " 're',\n",
       " '▁m',\n",
       " '▁o',\n",
       " 'ha',\n",
       " 'nd',\n",
       " '▁b',\n",
       " '▁f',\n",
       " 'it',\n",
       " 'ou',\n",
       " 'er',\n",
       " 'is',\n",
       " '▁c',\n",
       " '▁d',\n",
       " 'or',\n",
       " 'll',\n",
       " '▁l',\n",
       " 'ed',\n",
       " 'es',\n",
       " '▁p',\n",
       " 'on',\n",
       " '▁g',\n",
       " '▁n',\n",
       " 'en',\n",
       " 'an',\n",
       " 'ay',\n",
       " 'as',\n",
       " 'om',\n",
       " 'ot',\n",
       " 've',\n",
       " 'at',\n",
       " 'ar',\n",
       " 'ut',\n",
       " '▁h',\n",
       " 'ow',\n",
       " 'us',\n",
       " 'le',\n",
       " '▁e',\n",
       " 'et',\n",
       " '▁y',\n",
       " 'ld',\n",
       " 'sp',\n",
       " 'gh',\n",
       " 'ic',\n",
       " 'ke',\n",
       " 'id',\n",
       " '▁T',\n",
       " 'nb',\n",
       " 'al',\n",
       " 'im',\n",
       " 'ly',\n",
       " 'ac',\n",
       " 'oo',\n",
       " 'ad',\n",
       " 'se',\n",
       " '▁S',\n",
       " 'ee',\n",
       " 'ur',\n",
       " 'st',\n",
       " '..',\n",
       " '▁A',\n",
       " '▁j',\n",
       " 'ri',\n",
       " '▁W',\n",
       " 'ir',\n",
       " 'am',\n",
       " '▁u',\n",
       " 'ch',\n",
       " 'if',\n",
       " 'ro',\n",
       " '▁k',\n",
       " '▁M',\n",
       " 'pp',\n",
       " 'ce',\n",
       " 'ct',\n",
       " 'op',\n",
       " '▁H',\n",
       " '▁B',\n",
       " 'fe',\n",
       " 'th',\n",
       " ';&',\n",
       " 'un',\n",
       " '▁\"',\n",
       " 'il',\n",
       " 'ra',\n",
       " '▁(',\n",
       " 'ie',\n",
       " '▁C',\n",
       " 'ge',\n",
       " '--',\n",
       " 'nt',\n",
       " '.&',\n",
       " '▁L',\n",
       " 'ry',\n",
       " '▁D',\n",
       " '▁O',\n",
       " 'ul',\n",
       " 'ab',\n",
       " '▁N',\n",
       " '▁r',\n",
       " 'ss',\n",
       " 'lf',\n",
       " 'ig',\n",
       " 'qu',\n",
       " '▁P',\n",
       " '▁F',\n",
       " 'ck',\n",
       " 'ak',\n",
       " '▁J',\n",
       " 'ep',\n",
       " '!!',\n",
       " 'so',\n",
       " 'um',\n",
       " '▁E',\n",
       " '▁G',\n",
       " 'em',\n",
       " 'ag',\n",
       " 'ol',\n",
       " '▁i',\n",
       " '▁Y',\n",
       " '▁1',\n",
       " '▁v',\n",
       " 'ob',\n",
       " '▁&',\n",
       " 'oy',\n",
       " 'be',\n",
       " 'ci',\n",
       " '▁R',\n",
       " 'iv',\n",
       " 'os',\n",
       " 'pt',\n",
       " 'iz',\n",
       " '▁',\n",
       " 'e',\n",
       " 't',\n",
       " 'a',\n",
       " 'o',\n",
       " 'n',\n",
       " 'i',\n",
       " 's',\n",
       " 'h',\n",
       " 'r',\n",
       " 'l',\n",
       " 'd',\n",
       " 'u',\n",
       " 'm',\n",
       " 'y',\n",
       " 'w',\n",
       " 'g',\n",
       " 'c',\n",
       " 'f',\n",
       " '.',\n",
       " 'p',\n",
       " 'b',\n",
       " 'I',\n",
       " 'k',\n",
       " ',',\n",
       " 'v',\n",
       " \"'\",\n",
       " '-',\n",
       " 'T',\n",
       " ';',\n",
       " '&',\n",
       " '!',\n",
       " 'S',\n",
       " 'A',\n",
       " 'j',\n",
       " '\"',\n",
       " 'W',\n",
       " 'x',\n",
       " 'M',\n",
       " 'H',\n",
       " 'L',\n",
       " 'B',\n",
       " 'O',\n",
       " ')',\n",
       " 'C',\n",
       " '?',\n",
       " 'D',\n",
       " '(',\n",
       " 'N',\n",
       " '0',\n",
       " 'E',\n",
       " 'z',\n",
       " 'P',\n",
       " 'F',\n",
       " '1',\n",
       " 'G',\n",
       " 'q',\n",
       " 'R',\n",
       " 'Y',\n",
       " 'J',\n",
       " ':',\n",
       " '2',\n",
       " 'â',\n",
       " '\\x80',\n",
       " '5',\n",
       " '3',\n",
       " 'U',\n",
       " 'V',\n",
       " '4',\n",
       " '/',\n",
       " 'K',\n",
       " '\\x99',\n",
       " '9',\n",
       " '*',\n",
       " '8',\n",
       " '6',\n",
       " '7',\n",
       " '$']"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocabs = [sp_bpe.id_to_piece(id) for id in range(sp_bpe.get_piece_size())]\n",
    "bpe_tokens = sorted(vocabs, key=lambda x: len(x), reverse=True)\n",
    "bpe_tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get a list of all of the Unigram tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['▁everything',\n",
       " '▁something',\n",
       " '▁different',\n",
       " '▁actually',\n",
       " '▁remember',\n",
       " '▁anything',\n",
       " '▁children',\n",
       " '▁interest',\n",
       " '▁because',\n",
       " '▁thought',\n",
       " '▁getting',\n",
       " '▁another',\n",
       " '▁someone',\n",
       " '▁urlLink',\n",
       " '▁believe',\n",
       " '▁husband',\n",
       " '▁through',\n",
       " '▁morning',\n",
       " '▁problem',\n",
       " '▁really',\n",
       " '▁people',\n",
       " '▁little',\n",
       " '▁things',\n",
       " '▁should',\n",
       " '▁better',\n",
       " '▁myself',\n",
       " '▁though',\n",
       " '▁around',\n",
       " '▁before',\n",
       " '▁person',\n",
       " '▁author',\n",
       " '▁trying',\n",
       " '▁school',\n",
       " '▁family',\n",
       " '▁wonder',\n",
       " '▁enough',\n",
       " '▁happen',\n",
       " '▁having',\n",
       " '▁change',\n",
       " '▁think',\n",
       " '▁about',\n",
       " '▁would',\n",
       " '▁there',\n",
       " '▁going',\n",
       " '▁other',\n",
       " '▁could',\n",
       " '▁essay',\n",
       " '▁their',\n",
       " '▁again',\n",
       " '▁start',\n",
       " '▁after',\n",
       " '▁night',\n",
       " '▁being',\n",
       " '▁today',\n",
       " '▁first',\n",
       " '▁never',\n",
       " '▁right',\n",
       " '▁which',\n",
       " '▁where',\n",
       " '▁still',\n",
       " '▁great',\n",
       " '▁thing',\n",
       " '▁house',\n",
       " '▁place',\n",
       " '▁every',\n",
       " '▁while',\n",
       " '▁those',\n",
       " '▁point',\n",
       " '▁story',\n",
       " '▁class',\n",
       " '▁found',\n",
       " '▁month',\n",
       " '▁watch',\n",
       " '▁enjoy',\n",
       " '▁guess',\n",
       " '▁money',\n",
       " '▁least',\n",
       " '▁write',\n",
       " '▁world',\n",
       " '▁happy',\n",
       " '▁since',\n",
       " '▁phone',\n",
       " '<unk>',\n",
       " '▁that',\n",
       " '▁have',\n",
       " '▁with',\n",
       " '▁this',\n",
       " '▁like',\n",
       " '▁just',\n",
       " '▁what',\n",
       " '▁time',\n",
       " '▁when',\n",
       " '▁work',\n",
       " '▁they',\n",
       " '▁want',\n",
       " '▁feel',\n",
       " '▁will',\n",
       " '▁some',\n",
       " '▁know',\n",
       " '▁more',\n",
       " '▁from',\n",
       " '▁good',\n",
       " '▁them',\n",
       " '▁life',\n",
       " '▁back',\n",
       " 'ation',\n",
       " '▁make',\n",
       " '▁been',\n",
       " '▁were',\n",
       " '▁week',\n",
       " '▁take',\n",
       " '▁your',\n",
       " '▁year',\n",
       " '▁then',\n",
       " '▁even',\n",
       " '▁much',\n",
       " '▁need',\n",
       " '▁home',\n",
       " '▁comp',\n",
       " '▁love',\n",
       " '▁over',\n",
       " '▁than',\n",
       " '▁very',\n",
       " '▁look',\n",
       " '▁also',\n",
       " '▁into',\n",
       " '▁down',\n",
       " '▁talk',\n",
       " '▁only',\n",
       " '▁help',\n",
       " '▁read',\n",
       " '▁last',\n",
       " '▁sure',\n",
       " '▁show',\n",
       " '▁many',\n",
       " '▁well',\n",
       " '▁call',\n",
       " '▁didn',\n",
       " '▁find',\n",
       " '▁said',\n",
       " '▁live',\n",
       " '▁long',\n",
       " '▁went',\n",
       " '▁part',\n",
       " '▁tell',\n",
       " '▁keep',\n",
       " '▁hope',\n",
       " '▁come',\n",
       " 'ction',\n",
       " '▁What',\n",
       " '▁kids',\n",
       " '▁hard',\n",
       " '▁wait',\n",
       " '▁next',\n",
       " '▁girl',\n",
       " '▁That',\n",
       " '▁When',\n",
       " '▁wife',\n",
       " '▁felt',\n",
       " '▁came',\n",
       " '▁most',\n",
       " '▁move',\n",
       " '▁same',\n",
       " '▁plan',\n",
       " '▁head',\n",
       " '▁idea',\n",
       " '▁nice',\n",
       " '▁mind',\n",
       " '▁unti',\n",
       " '▁play',\n",
       " '▁kind',\n",
       " '▁away',\n",
       " '▁each',\n",
       " '▁else',\n",
       " '▁word',\n",
       " '▁pass',\n",
       " '</s>',\n",
       " '▁the',\n",
       " '▁and',\n",
       " '▁for',\n",
       " '▁was',\n",
       " '▁not',\n",
       " '▁you',\n",
       " '▁her',\n",
       " '▁but',\n",
       " '▁The',\n",
       " '▁out',\n",
       " '▁can',\n",
       " '▁all',\n",
       " '▁she',\n",
       " '▁are',\n",
       " '▁get',\n",
       " '▁had',\n",
       " '▁one',\n",
       " '▁see',\n",
       " '▁day',\n",
       " '▁how',\n",
       " 'ther',\n",
       " '▁who',\n",
       " 'ight',\n",
       " '▁con',\n",
       " '▁has',\n",
       " '▁now',\n",
       " '▁his',\n",
       " 'ally',\n",
       " 'ting',\n",
       " 'ment',\n",
       " '▁him',\n",
       " '▁off',\n",
       " '▁say',\n",
       " '▁our',\n",
       " '▁got',\n",
       " '▁pro',\n",
       " '▁way',\n",
       " '▁Thi',\n",
       " 'able',\n",
       " '▁new',\n",
       " 'ever',\n",
       " '▁But',\n",
       " 'side',\n",
       " '▁She',\n",
       " '▁job',\n",
       " '▁did',\n",
       " '▁And',\n",
       " '▁dis',\n",
       " 'ture',\n",
       " '▁two',\n",
       " 'ough',\n",
       " '▁fun',\n",
       " 'ways',\n",
       " '▁pre',\n",
       " '▁old',\n",
       " 'ious',\n",
       " 'less',\n",
       " '▁gra',\n",
       " '▁sit',\n",
       " '▁mom',\n",
       " '▁why',\n",
       " '▁let',\n",
       " '▁You',\n",
       " '▁guy',\n",
       " '▁pay',\n",
       " '▁tea',\n",
       " '▁bad',\n",
       " 'room',\n",
       " 'ness',\n",
       " '▁few',\n",
       " '<s>',\n",
       " '▁to',\n",
       " 'ing',\n",
       " '▁of',\n",
       " '▁in',\n",
       " '▁my',\n",
       " '▁is',\n",
       " '▁it',\n",
       " '▁me',\n",
       " '▁be',\n",
       " '▁do',\n",
       " '▁on',\n",
       " '▁so',\n",
       " '▁am',\n",
       " '▁re',\n",
       " '▁he',\n",
       " '...',\n",
       " '▁we',\n",
       " '▁at',\n",
       " '▁ma',\n",
       " 'end',\n",
       " 'ent',\n",
       " '▁st',\n",
       " 'ter',\n",
       " '▁as',\n",
       " '▁an',\n",
       " '▁up',\n",
       " '▁de',\n",
       " 'ate',\n",
       " '▁or',\n",
       " 'ion',\n",
       " '▁go',\n",
       " '▁co',\n",
       " 'ive',\n",
       " '▁se',\n",
       " 'day',\n",
       " '▁su',\n",
       " '▁lo',\n",
       " '▁if',\n",
       " '▁ex',\n",
       " '▁ho',\n",
       " '▁no',\n",
       " '▁us',\n",
       " '▁by',\n",
       " '▁sh',\n",
       " 'and',\n",
       " '▁So',\n",
       " '▁la',\n",
       " 'age',\n",
       " '▁It',\n",
       " '▁We',\n",
       " 'ver',\n",
       " 'tra',\n",
       " 'ful',\n",
       " 'ick',\n",
       " '▁He',\n",
       " 'ine',\n",
       " '!!!',\n",
       " 'com',\n",
       " '▁al',\n",
       " 'ish',\n",
       " 'ant',\n",
       " 'ies',\n",
       " 'ted',\n",
       " '▁po',\n",
       " 'ity',\n",
       " 'act',\n",
       " 'ood',\n",
       " '▁br',\n",
       " '▁My',\n",
       " '▁ra',\n",
       " 'est',\n",
       " '▁hu',\n",
       " '▁I',\n",
       " '▁a',\n",
       " 'ed',\n",
       " 'ar',\n",
       " 'er',\n",
       " 'sp',\n",
       " '▁f',\n",
       " 're',\n",
       " '▁b',\n",
       " 'or',\n",
       " 'nb',\n",
       " '▁c',\n",
       " 'in',\n",
       " 'le',\n",
       " '▁p',\n",
       " 'al',\n",
       " 'es',\n",
       " 'an',\n",
       " 'ri',\n",
       " 'll',\n",
       " 'ly',\n",
       " 'on',\n",
       " 'it',\n",
       " '▁t',\n",
       " 'ch',\n",
       " '▁w',\n",
       " 'ic',\n",
       " 'ce',\n",
       " 'at',\n",
       " 'ur',\n",
       " 'th',\n",
       " '▁m',\n",
       " '▁A',\n",
       " '▁d',\n",
       " 'en',\n",
       " 'st',\n",
       " 'un',\n",
       " 'ro',\n",
       " 've',\n",
       " 'il',\n",
       " 'us',\n",
       " 'is',\n",
       " 'ow',\n",
       " '▁S',\n",
       " '▁g',\n",
       " 'ne',\n",
       " '▁\"',\n",
       " 'ir',\n",
       " 'ut',\n",
       " '▁(',\n",
       " 'se',\n",
       " '▁M',\n",
       " 'te',\n",
       " '▁e',\n",
       " 'li',\n",
       " '▁T',\n",
       " '▁L',\n",
       " '▁C',\n",
       " '▁B',\n",
       " 'as',\n",
       " 'ol',\n",
       " 'im',\n",
       " 'am',\n",
       " '▁D',\n",
       " 'me',\n",
       " '▁W',\n",
       " 'id',\n",
       " '▁H',\n",
       " '▁O',\n",
       " 'he',\n",
       " 'ng',\n",
       " 'ry',\n",
       " '▁i',\n",
       " 'et',\n",
       " 'hi',\n",
       " 'ke',\n",
       " '▁N',\n",
       " 'ha',\n",
       " '▁P',\n",
       " '▁F',\n",
       " 'ge',\n",
       " 'ig',\n",
       " 'ap',\n",
       " 'ul',\n",
       " 'ma',\n",
       " 'ru',\n",
       " '▁G',\n",
       " 'el',\n",
       " '▁E',\n",
       " 'ie',\n",
       " 'ok',\n",
       " 'mp',\n",
       " 'ci',\n",
       " '▁J',\n",
       " '▁1',\n",
       " '--',\n",
       " '▁R',\n",
       " 'uc',\n",
       " 'if',\n",
       " 'um',\n",
       " '▁&',\n",
       " '▁2',\n",
       " 's',\n",
       " '▁',\n",
       " '.',\n",
       " 't',\n",
       " ',',\n",
       " 'e',\n",
       " 'a',\n",
       " 'o',\n",
       " 'd',\n",
       " 'n',\n",
       " 'y',\n",
       " 'p',\n",
       " \"'\",\n",
       " 'l',\n",
       " 'c',\n",
       " 'u',\n",
       " 'r',\n",
       " 'm',\n",
       " 'i',\n",
       " 'k',\n",
       " 'h',\n",
       " ';',\n",
       " 'f',\n",
       " 'b',\n",
       " 'g',\n",
       " '&',\n",
       " 'w',\n",
       " '-',\n",
       " 'v',\n",
       " '!',\n",
       " ')',\n",
       " '\"',\n",
       " '?',\n",
       " '0',\n",
       " 'z',\n",
       " 'I',\n",
       " ':',\n",
       " 'x',\n",
       " 'â',\n",
       " '\\x80',\n",
       " 'A',\n",
       " 'O',\n",
       " 'j',\n",
       " 'E',\n",
       " 'T',\n",
       " 'Y',\n",
       " '5',\n",
       " '3',\n",
       " 'S',\n",
       " 'N',\n",
       " 'U',\n",
       " 'V',\n",
       " '4',\n",
       " 'L',\n",
       " '/',\n",
       " 'K',\n",
       " '9',\n",
       " '1',\n",
       " 'R',\n",
       " 'H',\n",
       " 'D',\n",
       " 'C',\n",
       " '2',\n",
       " '*',\n",
       " 'W',\n",
       " 'P',\n",
       " '8',\n",
       " '6',\n",
       " 'M',\n",
       " 'B',\n",
       " 'G',\n",
       " '7',\n",
       " 'F',\n",
       " '(',\n",
       " '$',\n",
       " '\\x99',\n",
       " 'J',\n",
       " 'q']"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocabs = [sp_uni.id_to_piece(id) for id in range(sp_uni.get_piece_size())]\n",
    "bpe_tokens = sorted(vocabs, key=lambda x: len(x), reverse=True)\n",
    "bpe_tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reversing the process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BPE ['▁This', '▁is', '▁a', '▁t', 'est']\n",
      "BPE [400, 61, 4, 3, 231]\n",
      "UNI ['▁Thi', 's', '▁is', '▁a', '▁t', 'est']\n",
      "UNI [284, 3, 37, 15, 78, 338]\n"
     ]
    }
   ],
   "source": [
    "# encode: text => id\n",
    "print(\"BPE {}\".format(sp_bpe.encode_as_pieces('This is a test')))\n",
    "print(\"BPE {}\".format(sp_bpe.encode_as_ids('This is a test')))\n",
    "\n",
    "print(\"UNI {}\".format(sp_uni.encode_as_pieces('This is a test')))\n",
    "print(\"UNI {}\".format(sp_uni.encode_as_ids('This is a test')))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BPE This is a test\n",
      "BPE This is a test\n",
      "UNI This is a test\n",
      "UNI This is a test\n"
     ]
    }
   ],
   "source": [
    "# decode: id => text\n",
    "print(\"BPE {}\".format(sp_bpe.decode_pieces(['▁This', '▁is', '▁a', '▁t', 'est'])))\n",
    "print(\"BPE {}\".format(sp_bpe.decode_ids([400, 61, 4, 3, 231])))\n",
    "\n",
    "print(\"UNI {}\".format(sp_uni.decode_pieces(['▁Thi', 's', '▁is', '▁a', '▁t', 'est'])))\n",
    "print(\"UNI {}\".format(sp_uni.decode_ids([284, 3, 37, 15, 78, 338])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get a list of tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocabs = [sp_bpe.id_to_piece(id) for id in range(sp_bpe.get_piece_size())]\n",
    "bpe_list = sorted(vocabs, key=lambda x: len(x), reverse=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocabs = [sp_uni.id_to_piece(id) for id in range(sp_uni.get_piece_size())]\n",
    "uni_list = sorted(vocabs, key=lambda x: len(x), reverse=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Unigram tokens not in BPE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "171\n"
     ]
    }
   ],
   "source": [
    "uni_tok_diff = [u for u in uni_list if u not in bpe_list]\n",
    "print(len(uni_tok_diff))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### BPE tokens not in Unigram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "171\n"
     ]
    }
   ],
   "source": [
    "bpe_tok_diff = [b for b in bpe_list if b not in uni_list]\n",
    "print(len(bpe_tok_diff))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unigram tokens not in BPE</th>\n",
       "      <th>BPE tokens not in Unigram</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>▁everything</td>\n",
       "      <td>▁friend</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>▁different</td>\n",
       "      <td>▁somet</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>▁actually</td>\n",
       "      <td>▁thou</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>▁remember</td>\n",
       "      <td>other</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>▁anything</td>\n",
       "      <td>ittle</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Unigram tokens not in BPE BPE tokens not in Unigram\n",
       "0               ▁everything                   ▁friend\n",
       "1                ▁different                    ▁somet\n",
       "2                 ▁actually                     ▁thou\n",
       "3                 ▁remember                     other\n",
       "4                 ▁anything                     ittle"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diff_pairs = list(zip(uni_tok_diff, bpe_tok_diff))\n",
    "diff_df = pd.DataFrame(diff_pairs, \n",
    "                       columns=([\"Unigram tokens not in BPE\", \"BPE tokens not in Unigram\"]))\n",
    "diff_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compare the tokens\n",
    "How do the different tokenizers deal with missing tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Unigram tokens\n",
    "Let's look first at the Unigram tokens which are in BPE but not in the Unigram tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['▁friend',\n",
       " '▁somet',\n",
       " '▁thou',\n",
       " 'other',\n",
       " 'ittle',\n",
       " 'riend',\n",
       " 'essay',\n",
       " 'thing',\n",
       " '▁This',\n",
       " 'nbsp',\n",
       " 'ould',\n",
       " '▁wor',\n",
       " 'very',\n",
       " 'king',\n",
       " '▁tim',\n",
       " '▁com',\n",
       " 'hing',\n",
       " '▁bec',\n",
       " '▁any',\n",
       " '▁int',\n",
       " 'ause',\n",
       " '▁man',\n",
       " 'ople',\n",
       " 'omet',\n",
       " 'ving',\n",
       " '▁wee',\n",
       " 'reat',\n",
       " '▁too',\n",
       " 'fter',\n",
       " 'self',\n",
       " 'here',\n",
       " 'ning',\n",
       " '▁bet',\n",
       " '▁exp',\n",
       " '▁hel',\n",
       " '▁try',\n",
       " 'hat',\n",
       " '▁ha',\n",
       " '▁th',\n",
       " '▁li',\n",
       " 'ith',\n",
       " 'all',\n",
       " 'ght',\n",
       " 'ome',\n",
       " 'ust',\n",
       " 'her',\n",
       " 'ill',\n",
       " '▁wh',\n",
       " '▁ne',\n",
       " 'ink',\n",
       " 'out',\n",
       " 'hen',\n",
       " 'ess',\n",
       " '▁mo',\n",
       " '▁ab',\n",
       " 'one',\n",
       " 'ake',\n",
       " 'ack',\n",
       " 'art',\n",
       " 'ind',\n",
       " 'eel',\n",
       " 'ast',\n",
       " 'ain',\n",
       " '▁kn',\n",
       " 'rom',\n",
       " 'use',\n",
       " 'ear',\n",
       " 'rou',\n",
       " 'ell',\n",
       " 'itt',\n",
       " 'han',\n",
       " 'ers',\n",
       " 'are',\n",
       " 'ven',\n",
       " '▁pl',\n",
       " '▁le',\n",
       " '▁sp',\n",
       " 'way',\n",
       " 'ite',\n",
       " 'ong',\n",
       " '▁pe',\n",
       " 'ure',\n",
       " 'ore',\n",
       " 'ide',\n",
       " '▁ch',\n",
       " '▁ag',\n",
       " 'ort',\n",
       " 'our',\n",
       " 'ame',\n",
       " 'ook',\n",
       " 'ist',\n",
       " 'own',\n",
       " '▁Th',\n",
       " '▁wr',\n",
       " '▁sa',\n",
       " 'uch',\n",
       " 'ard',\n",
       " '▁cl',\n",
       " 'ass',\n",
       " 'ice',\n",
       " 'ond',\n",
       " 'ous',\n",
       " 'ich',\n",
       " '▁en',\n",
       " 'ace',\n",
       " 'ool',\n",
       " '▁gu',\n",
       " 'red',\n",
       " '▁qu',\n",
       " '▁fe',\n",
       " '▁bl',\n",
       " '▁tw',\n",
       " 'alk',\n",
       " 'ild',\n",
       " 'ble',\n",
       " 'ile',\n",
       " '▁im',\n",
       " '▁ri',\n",
       " 'ose',\n",
       " 'ree',\n",
       " 'ily',\n",
       " '▁s',\n",
       " '▁o',\n",
       " 'nd',\n",
       " 'ou',\n",
       " '▁l',\n",
       " '▁n',\n",
       " 'ay',\n",
       " 'om',\n",
       " 'ot',\n",
       " '▁h',\n",
       " '▁y',\n",
       " 'ld',\n",
       " 'gh',\n",
       " 'ac',\n",
       " 'oo',\n",
       " 'ad',\n",
       " 'ee',\n",
       " '..',\n",
       " '▁j',\n",
       " '▁u',\n",
       " '▁k',\n",
       " 'pp',\n",
       " 'ct',\n",
       " 'op',\n",
       " 'fe',\n",
       " ';&',\n",
       " 'ra',\n",
       " 'nt',\n",
       " '.&',\n",
       " 'ab',\n",
       " '▁r',\n",
       " 'ss',\n",
       " 'lf',\n",
       " 'qu',\n",
       " 'ck',\n",
       " 'ak',\n",
       " 'ep',\n",
       " '!!',\n",
       " 'so',\n",
       " 'em',\n",
       " 'ag',\n",
       " '▁Y',\n",
       " '▁v',\n",
       " 'ob',\n",
       " 'oy',\n",
       " 'be',\n",
       " 'iv',\n",
       " 'os',\n",
       " 'pt',\n",
       " 'iz']"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "diff_df['BPE tokens not in Unigram'].values.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Unigram token ['▁f', 'ri', 'end'] \n",
      "BPE token ['▁friend']\n",
      "\n",
      "Unigram token ['▁some', 't', 'im', 'es'] \n",
      "BPE token ['▁somet', 'im', 'es']\n",
      "\n",
      "Unigram token ['▁', 'th', 'o', 'us', 'and'] \n",
      "BPE token ['▁thou', 's', 'and']\n",
      "\n",
      "Unigram token ['▁', '.', 'o', 'ther'] \n",
      "BPE token ['▁', '.', 'other']\n",
      "\n",
      "Unigram token ['▁', '.', 'li', 't', 't', 'le'] \n",
      "BPE token ['▁', '.', 'l', 'ittle']\n",
      "\n"
     ]
    }
   ],
   "source": [
    "test_list = [\"friend\", \"sometimes\", \"thousand\", \".other\", \".little\"]\n",
    "for ut in test_list:\n",
    "    print(\"Unigram token {} \\nBPE token {}\\n\".format(sp_uni.encode_as_pieces(ut), sp_bpe.encode_as_pieces(ut)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unigram sampling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['▁he', 'll', 'o', '▁world']\n",
      "['▁he', 'll', 'o', '▁world']\n",
      "['▁he', 'll', 'o', '▁world']\n",
      "['▁he', 'll', 'o', '▁world']\n",
      "['▁he', 'll', 'o', '▁world']\n",
      "['▁he', 'll', 'o', '▁world']\n",
      "['▁he', 'll', 'o', '▁world']\n",
      "['▁he', 'll', 'o', '▁world']\n",
      "['▁he', 'll', 'o', '▁world']\n",
      "['▁he', 'll', 'o', '▁world']\n"
     ]
    }
   ],
   "source": [
    "for n in range(10):\n",
    "  print(sp_uni.encode_as_pieces('hello world'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['▁re', 'me', 'm', 'b', 'er', 's']\n",
      "['▁', 're', 'm', 'e', 'm', 'b', 'e', 'r', 's']\n",
      "['▁remember', 's']\n",
      "['▁remember', 's']\n",
      "['▁remember', 's']\n",
      "['▁', 're', 'me', 'm', 'b', 'er', 's']\n",
      "['▁', 'r', 'e', 'me', 'm', 'b', 'er', 's']\n",
      "['▁re', 'me', 'm', 'b', 'e', 'r', 's']\n",
      "['▁', 'r', 'e', 'me', 'm', 'b', 'er', 's']\n",
      "['▁remember', 's']\n"
     ]
    }
   ],
   "source": [
    "# Can obtain different segmentations per request.\n",
    "# There are two hyperparamenters for sampling (nbest_size and inverse temperature). see the paper [kudo18] for detail.\n",
    "for n in range(10):\n",
    "  print(sp_uni.sample_encode_as_pieces('remembers', -1, 0.1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['▁remember', 's']\n",
      "['▁re', 'me', 'm', 'b', 'er', 's']\n",
      "['▁re', 'm', 'e', 'm', 'b', 'er', 's']\n",
      "['▁', 're', 'me', 'm', 'b', 'er', 's']\n",
      "['▁re', 'me', 'm', 'b', 'e', 'r', 's']\n",
      "['▁', 're', 'm', 'e', 'm', 'b', 'er', 's']\n",
      "['▁re', 'm', 'e', 'm', 'b', 'e', 'r', 's']\n",
      "['▁', 'r', 'e', 'me', 'm', 'b', 'er', 's']\n",
      "['▁', 're', 'me', 'm', 'b', 'e', 'r', 's']\n",
      "['▁', 'r', 'e', 'm', 'e', 'm', 'b', 'er', 's']\n"
     ]
    }
   ],
   "source": [
    "# get 10 best\n",
    "best_seg = sp_uni.nbest_encode_as_pieces('remembers', 10)\n",
    "for i in best_seg:\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HuggingFace Tokenizers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tokenizers import (ByteLevelBPETokenizer,\n",
    "                            BPETokenizer,\n",
    "                            SentencePieceBPETokenizer,\n",
    "                            BertWordPieceTokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = SentencePieceBPETokenizer()\n",
    "tokenizer.train([\"../blog_test.txt\"], vocab_size=500, min_frequency=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['▁Th', 'is', '▁is', '▁a', '▁t', 'est']\n"
     ]
    }
   ],
   "source": [
    "output = tokenizer.encode(\"This is a test\")\n",
    "print(output.tokens)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}