example_text.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text processing features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text cleaning / splitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text after english_cleaners    : hello world !\n",
      "Text after collapse_whitespace : Hello World !\n",
      "Text after lowercase           : hello   world    !\n",
      "\n",
      "Text after french_cleaners     : bonjour a tous !\n",
      "Text after convert_to_ascii    : Bonjour  a Tous   !\n",
      "Text after fr_convert_to_ascii : Bonjour  a Tous   !\n"
     ]
    }
   ],
   "source": [
    "from utils.text.cleaners import french_cleaners, english_cleaners\n",
    "from utils.text.cleaners import expand_numbers, collapse_whitespace, fr_convert_to_ascii, convert_to_ascii, lowercase\n",
    "\n",
    "en_text = \"Hello   World    !\"\n",
    "fr_text = \"Bonjour  à Tous   !\"\n",
    "\n",
    "print(\"Text after english_cleaners    : {}\".format(english_cleaners(en_text)))\n",
    "print(\"Text after collapse_whitespace : {}\".format(collapse_whitespace(en_text)))\n",
    "print(\"Text after lowercase           : {}\".format(lowercase(en_text)))\n",
    "print()\n",
    "print(\"Text after french_cleaners     : {}\".format(french_cleaners(fr_text)))\n",
    "print(\"Text after convert_to_ascii    : {}\".format(convert_to_ascii(fr_text)))\n",
    "print(\"Text after fr_convert_to_ascii : {}\".format(fr_convert_to_ascii(fr_text)))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text part 0 (length = 125) : Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore \n",
      "et dolore magna aliqua. \n",
      "Text part 1 (length = 109) : Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut \n",
      "aliquip ex ea commodo consequat. \n",
      "Text part 2 (length = 104) : Duis aute irure dolor in reprehenderit in voluptate velit esse cillum \n",
      "dolore eu fugiat nulla pariatur. \n",
      "Text part 3 (length = 111) : Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia \n",
      "deserunt mollit anim id est laborum.\n"
     ]
    }
   ],
   "source": [
    "from utils.text import split_text\n",
    "\n",
    "long_text = \"\"\"Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore \n",
    "et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut \n",
    "aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum \n",
    "dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia \n",
    "deserunt mollit anim id est laborum.\"\"\"\n",
    "for i, p in enumerate(split_text(long_text, 150)):\n",
    "    print(\"Text part {} (length = {}) : {}\".format(i, len(p), p))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text tokenization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialize a Tokenizer\n",
    "\n",
    "A `vocab_size` higher than the actual number of tokens can be provided in order to create `ukn_x` tokens. It can be useful if you need a fixed `vocab_size` or if you would like to have the possibility to add tokens in the future. \n",
    "\n",
    "For instance, this feature is used in the French version of the `Tacotron-2` model, as the English version has a `vocab_size` of 148, while the French one has less characters (and thus a smaller vocabulary). \n",
    "\n",
    "It is also possible to initialize a `TextEncoder` based on a `transformers` pretrained `AutoTokenizer` : `BERT`, `BART`, `MBart`, `GPT-2`, `Falcon` and others are supported !\n",
    "\n",
    "PS : that's a bit strange that the `<|endoftext|>` is used both for start and end of sequence, but it is not an error ;)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========== Tokenizer ==========\n",
      "Vocab (size = 70) : ['_', '-', '!', \"'\", '(', ')', ',', '.', ':', ';', '?', ' ', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', ...]\n",
      "Config : {\n",
      "  \"level\": 0,\n",
      "  \"template\": null,\n",
      "  \"lstrip\": false,\n",
      "  \"rstrip\": false,\n",
      "  \"cleaners\": [\n",
      "    \"french_cleaners\"\n",
      "  ],\n",
      "  \"split_pattern\": null,\n",
      "  \"bpe_end_of_word\": null,\n",
      "  \"pad_token\": \"\",\n",
      "  \"sos_token\": null,\n",
      "  \"eos_token\": null,\n",
      "  \"sep_token\": null,\n",
      "  \"ukn_token\": null,\n",
      "  \"mask_token\": null,\n",
      "  \"additional_tokens\": {},\n",
      "  \"sub_word_prefix\": \"\",\n",
      "  \"use_sos_and_eos\": false,\n",
      "  \"add_special_tokens_at_end\": true\n",
      "}\n",
      "\n",
      "========== Tokenizer ==========\n",
      "Vocab (size = 148) : ['_', '-', '!', \"'\", '(', ')', ',', '.', ':', ';', '?', ' ', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', ...]\n",
      "Config : {\n",
      "  \"level\": 0,\n",
      "  \"template\": null,\n",
      "  \"lstrip\": false,\n",
      "  \"rstrip\": false,\n",
      "  \"cleaners\": [\n",
      "    {\n",
      "      \"name\": \"french_cleaners\",\n",
      "      \"to_lowercase\": false\n",
      "    }\n",
      "  ],\n",
      "  \"split_pattern\": null,\n",
      "  \"bpe_end_of_word\": null,\n",
      "  \"pad_token\": \"\",\n",
      "  \"sos_token\": null,\n",
      "  \"eos_token\": null,\n",
      "  \"sep_token\": null,\n",
      "  \"ukn_token\": null,\n",
      "  \"mask_token\": null,\n",
      "  \"additional_tokens\": {},\n",
      "  \"sub_word_prefix\": \"\",\n",
      "  \"use_sos_and_eos\": false,\n",
      "  \"add_special_tokens_at_end\": true\n",
      "}\n",
      "\n",
      "========== Tokenizer ==========\n",
      "Vocab (size = 50257) : ['!', '\"', '#', '$', '%', '&', \"'\", '(', ')', '*', '+', ',', '-', '.', '/', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', ':', ';', '<', '=', '>', '?', '@', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', ...]\n",
      "Config : {\n",
      "  \"level\": 1,\n",
      "  \"template\": null,\n",
      "  \"lstrip\": false,\n",
      "  \"rstrip\": true,\n",
      "  \"cleaners\": [],\n",
      "  \"split_pattern\": \"'s|'t|'re|'ve|'m|'ll|'d| ?\\\\p{L}+| ?\\\\p{N}+| ?[^\\\\s\\\\p{L}\\\\p{N}]+|\\\\s+(?!\\\\S)|\\\\s+\",\n",
      "  \"bpe_end_of_word\": null,\n",
      "  \"pad_token\": \"<|endoftext|>\",\n",
      "  \"sos_token\": \"<|endoftext|>\",\n",
      "  \"eos_token\": \"<|endoftext|>\",\n",
      "  \"sep_token\": null,\n",
      "  \"ukn_token\": \"<|endoftext|>\",\n",
      "  \"mask_token\": null,\n",
      "  \"additional_tokens\": {},\n",
      "  \"sub_word_prefix\": \"\",\n",
      "  \"use_sos_and_eos\": true,\n",
      "  \"add_special_tokens_at_end\": true\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "from utils.text import Tokenizer, get_symbols, default_french_tokenizer, get_tokenizer\n",
    "\n",
    "cleaners = [\n",
    "    {'name' : 'french_cleaners', 'to_lowercase' : False}\n",
    "]\n",
    "# Equivalent to :\n",
    "#cleaners = [\n",
    "#    'fr_convert_to_ascii',\n",
    "#    {'name' : 'expand_numbers', 'langue' : 'fr'},\n",
    "#    'collapse_whitespace'\n",
    "#]\n",
    "\n",
    "default     = default_french_tokenizer()\n",
    "encoder     = default_french_tokenizer(cleaners = cleaners, vocab_size = 148)\n",
    "gpt_encoder = get_tokenizer(lang = 'en', tokenizer = 'gpt2')\n",
    "print(default)\n",
    "print()\n",
    "print(encoder)\n",
    "print()\n",
    "print(gpt_encoder)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### save / load it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========== Tokenizer ==========\n",
      "Vocab (size = 148) : ['_', '-', '!', \"'\", '(', ')', ',', '.', ':', ';', '?', ' ', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', ...]\n",
      "Config : {\n",
      "  \"level\": 0,\n",
      "  \"template\": null,\n",
      "  \"lstrip\": false,\n",
      "  \"rstrip\": false,\n",
      "  \"cleaners\": [\n",
      "    {\n",
      "      \"name\": \"french_cleaners\",\n",
      "      \"to_lowercase\": false\n",
      "    }\n",
      "  ],\n",
      "  \"split_pattern\": null,\n",
      "  \"bpe_end_of_word\": null,\n",
      "  \"pad_token\": \"\",\n",
      "  \"sos_token\": null,\n",
      "  \"eos_token\": null,\n",
      "  \"sep_token\": null,\n",
      "  \"ukn_token\": null,\n",
      "  \"mask_token\": null,\n",
      "  \"additional_tokens\": {},\n",
      "  \"sub_word_prefix\": \"\",\n",
      "  \"use_sos_and_eos\": false,\n",
      "  \"add_special_tokens_at_end\": true\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "filename = 'example_data/example_text_encoder.json'\n",
    "\n",
    "encoder.save_to_file(filename)\n",
    "restored = Tokenizer.load_from_file(filename)\n",
    "\n",
    "print(restored)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Text tokenization / decoding\n",
    "\n",
    "The *manual encoding* is, in this case, equivalent to a simple encoding of each character (after text cleaning). It would not work for the `gpt_encoder`, as it is a token-based tokenizer (and not a simple character-based one). To visualize the individual tokens, you can use the `tokenize` methods, that cleans and splits the text ;)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Original text    : Bonjour  à Tous   !\n",
      "Cleaned text     : bonjour a tous !\n",
      "Encoded text     : [39 52 51 47 52 58 55 11 38 11 57 52 58 56 11  2]\n",
      "Manually encoded : [39, 52, 51, 47, 52, 58, 55, 11, 38, 11, 57, 52, 58, 56, 11, 2]\n",
      "Decoded text     : bonjour a tous !\n"
     ]
    }
   ],
   "source": [
    "fr_text = \"Bonjour  à Tous   !\"\n",
    "\n",
    "cleaned = default.clean_text(fr_text)\n",
    "encoded = default.encode(fr_text)\n",
    "decoded = default.decode(encoded)\n",
    "\n",
    "print(\"Original text    : {}\".format(fr_text))\n",
    "print(\"Cleaned text     : {}\".format(cleaned))\n",
    "print(\"Encoded text     : {}\".format(encoded))\n",
    "print(\"Manually encoded : {}\".format([default[c] for c in cleaned]))\n",
    "print(\"Decoded text     : {}\".format(decoded))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Usage with `tf.data` pipeline\n",
    "\n",
    "As the encoder uses python objects and regex function, it cannot be used in pure tensorflow functions (for graph optimization). \n",
    "\n",
    "The classical way to deal with python functions is to use `tf.py_function()` or `tf.numpy_function`, which is shown in the `encodee` example.\n",
    "\n",
    "Note : the `encode` and `decode` functions both handle `Tensor`s (it simply converts them into `np.ndarray`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([39 52 51 47 52 58 55 11 38 11 57 52 58 56 11  2], shape=(16,), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "\n",
    "def encode_text(sentence):\n",
    "    return default.encode(sentence)\n",
    "\n",
    "@tf.function\n",
    "def encode(text):\n",
    "    encoded_text = tf.numpy_function(encode_text, [text], Tout = tf.int32)\n",
    "    encoded_text.set_shape([None])\n",
    "    \n",
    "    return encoded_text\n",
    "\n",
    "print(encode(tf.cast(fr_text, tf.string)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The `utils.keras.execute_eagerly` function\n",
    "\n",
    "The `execute_eagerly` function / decorator (in `utils.keras_utils.compile`) enables to call functions inside `tf.numpy_function` simply by providing the output type (or signature) ! It acts a bit like the new `#numpy_function` added in `tensorflow 2.14`, but it is also compatible with previous versions ;) It also offers additional features, such as fixing the static shape with `tf.ensure_shape`, or converting inputs to `np.ndarray` (for non-tensorflow backends). Furthermore, if executed eagerly, the function is directly calls without leveraging the `numpy_function`, making it more flexible and reduces overheads. \n",
    "\n",
    "To demonstrate that the *original* `TextEncoder.encode` is not compatible with `tf.function`, the `encode_wrong` tries to call the non-wrapped function (accessible via the `func` variable), which correctly raises an error !\n",
    "\n",
    "Note : for this test, it is important to pass `tf.Tensor` argument instead of regular `str` type. Otherwise, the graph function will simply call everythin as pure python function, and will always retrace (as passing raw python objects will cause tensorflow retracing) ! Do not hesitate to open a discussion if you want more details about this ;)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([39 52 51 47 52 58 55 11 38 11 57 52 58 56 11  2], shape=(16,), dtype=int32)\n",
      "An (expected) error occured !\n"
     ]
    }
   ],
   "source": [
    "@tf.function\n",
    "def encode_wrong(text):\n",
    "    return default.encode.func(text)\n",
    "\n",
    "@tf.function\n",
    "def encode_new(text):\n",
    "    return encode_text(text)\n",
    "\n",
    "print(encode_new(tf.cast(fr_text, tf.string)))\n",
    "\n",
    "try:\n",
    "    print(encode_wrong(tf.cast(fr_text, tf.string)))\n",
    "except AttributeError as e:\n",
    "    print('An (expected) error occured !')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}