NLP-4 tok-stem-lemma.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# **Tokenization**        \n",
        "It is a method used to split a phrase,  sentence, paragraph or an  entire text document into smaller units. By doing this we can get an individual words or terms. Each of these smaller units are called as tokens.\n",
        "\n",
        "e.g., I like mangoes.  \n",
        "Tokens are: I, like, mangoes"
      ],
      "metadata": {
        "id": "qyDHtpmDgm3o"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **(i) Line Tokenization**"
      ],
      "metadata": {
        "id": "TzcfIaM6j1Bw"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jp-ylYmDge63",
        "outputId": "4c6c9600-b48e-4e12-efd4-83ec5e324380"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 1
        }
      ],
      "source": [
        "# importing required modules\n",
        "import nltk\n",
        "nltk.download('punkt')"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# creating line sentence data \n",
        "data = \"Hello everyone, myself Mohit. Welcome to my Github page!!\"\n",
        "\n",
        "# creating an object in which we are going to store tokenized each sentence from the sentences.\n",
        "tokens = nltk.sent_tokenize(data) \n",
        "print(tokens)\n",
        "\n",
        "# Determining the type of tokens\n",
        "print(type(tokens))\n",
        "\n",
        "# Determining no. of tokens\n",
        "len(tokens)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Rc1ey1bHkmmk",
        "outputId": "cde19ec8-3e76-4c5d-90d4-b1cce8a46895"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['Hello everyone, myself Mohit.', 'Welcome to my Github page!', '!']\n",
            "<class 'list'>\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "3"
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Firstly we imported required mmodules i.e., 'nltk' and downloading nltk.download('punkt')  \n",
        "Then we created a linear sentence which we stored in object 'data'.  \n",
        "Then we tokenize our sentence by using 'nltk.sent_tokenize()' function and stored those tokens in object 'tokens'  \n",
        "Then we print those tokens."
      ],
      "metadata": {
        "id": "sYooB8FDloQo"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **(ii) Word tokenization**"
      ],
      "metadata": {
        "id": "8SAa-v-LnMYQ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# creating an object in which we are going to store tokenized each word from the sentences.\n",
        "word_tokens = nltk.word_tokenize(data)\n",
        "print(word_tokens)\n",
        "\n",
        "# Determining the type of word_tokens\n",
        "print(type(word_tokens))\n",
        "\n",
        "# Determining the no. of word_tokens in sentence \n",
        "len(word_tokens)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LOkV6UzvnTJ4",
        "outputId": "53fb4127-a57f-4bb8-b47a-6fc4062b88a6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['Hello', 'everyone', ',', 'myself', 'Mohit', '.', 'Welcome', 'to', 'my', 'Github', 'page', '!', '!']\n",
            "<class 'list'>\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "13"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Same procedure like line tokenization just for tokenizing words from sentence or sentences we use 'nltk.word_tokenize()' function and stored those tokens in 'word_tokens'"
      ],
      "metadata": {
        "id": "Ux2oNG5voeor"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **Stemming**  \n",
        "A way to reduce a word to its words stem that affixes to suffixes and prefixes. In simple term, This Algorithm works by cutting off the end or the beginning of the word while taking into account a list of common prefixes and suffixes that can be found in an inflected word.\n",
        "\n",
        "\n",
        "\n",
        "```\n",
        "============================\n",
        "Form     |  Suffix  | Stem |\n",
        "============================\n",
        "Cats     |    -s    | Cat  |\n",
        "Birds    |    -s    | Bird |\n",
        "Blocked  |   -ed    | Block|\n",
        "Bringing |   -ing   | Bring|\n",
        "============================\n",
        "```\n",
        "\n"
      ],
      "metadata": {
        "id": "Z_1224ZQsmnf"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **(i) PORTER STEMMING**"
      ],
      "metadata": {
        "id": "uHELGtWwMi4X"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#------------------------------------------------------------------------PORTER STEMMER----------------------------------------------------------------------------#\n",
        "\n",
        "# importing required modules\n",
        "from nltk.stem import PorterStemmer\n",
        "ps = PorterStemmer() \n",
        "\n",
        "\n",
        "# choosing words to stem it\n",
        "data = [\"jumping\", \"jumps\", \"jumped\", \"jumper\", \"jumpers\"]\n",
        "\n",
        "# use a for loop to stem each word in the list\n",
        "for i in data:\n",
        "  print(i, \" : \", ps.stem(i))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "N_ac60LYB8zv",
        "outputId": "c5aef848-71ee-463a-bce1-29e5439644cd"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "jumping  :  jump\n",
            "jumps  :  jump\n",
            "jumped  :  jump\n",
            "jumper  :  jumper\n",
            "jumpers  :  jumper\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Step 1: Import required modules**  \n",
        "Firstly we imported required modules so from  our most important library in NLP i.e, **'nltk'** we imported **'stem'** and from that **'stem'** we imported **'PorterStemmer'** \n",
        "So we can directly write it as **from nltk.stem import PorterStemmer**  \n",
        "\n",
        "**Step 2: Choose data for performing operations using PorterStemmer**  \n",
        "now we created an object 'ps' so that we can easily call the function PorterStemmer(). And thus we stored function PorterStemmer() in object ps.   \n",
        "ps = PorterStemmer()  \n",
        "Now we choose some words and placed those in a list 'data'.\n",
        "  \n",
        "**Step 3: Implementation of PorterStemmer on our list**  \n",
        "So as there are multiple words in our list so we run for loop for each word. Then we directly called function **'ps.stem()'** where ps is our object created before for storing function PorterStemmer().  \n",
        "  \n",
        "\n",
        "So, In python as there is already library available for Porter Stemmer so we can directly implement by putting our data/word(s) into the function."
      ],
      "metadata": {
        "id": "PLwjniXeMol2"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "\n",
        "# **(ii) SNOWBALL STEMMER**"
      ],
      "metadata": {
        "id": "Zv9XCYM0R-fj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#-----------------------------------------------------------------------SNOWBALL STEMMER----------------------------------------------------------------------------#\n",
        "\n",
        "# import required modules\n",
        "from nltk.stem.snowball import SnowballStemmer\n",
        "\n",
        "\n",
        "# create a Snowball stemmer object for English\n",
        "SS = SnowballStemmer(\"english\")\n",
        "\n",
        "\n",
        "# example list of words to stem\n",
        "data = [\"jumping\", \"jumps\", \"jumped\", \"jumper\", \"jumpers\"]\n",
        "\n",
        "\n",
        "# use a for loop to stem each word in the list\n",
        "for i in data:\n",
        "    print(i, \" : \", SS.stem(i))\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Nup0gAq00w-c",
        "outputId": "76c751d2-a106-47b0-90e4-b7fe804b4c1e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "jumping  :  jump\n",
            "jumps  :  jump\n",
            "jumped  :  jump\n",
            "jumper  :  jumper\n",
            "jumpers  :  jumper\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Step 1: Import required modules**  \n",
        "Firstly we imported required modules so from  our most important library in NLP i.e, **'nltk'** we imported **'stem'** and from that **'stem'** we imported **'SnowballStemmer'** \n",
        "So we can directly write it as **from nltk.stem import SnowballStemmer**  \n",
        "\n",
        "**Step 2: Choose data for performing operations using SnowballStemmer**  \n",
        "now we created an object 'SS' so that we can easily call the function SnowballStemmer(). And thus we stored function SnowballStemmer() in object SS.   \n",
        "SS = SnowballStemmer()  \n",
        "Now we choose some words and placed those in a list 'data'.\n",
        "  \n",
        "**Step 3: Implementation of SnowballStemmer on our list**  \n",
        "So as there are multiple words in our list so we run for loop for each word. Then we directly called function **'SS.stem()'** where SS is our object created before for storing function SnowballStemmer().  \n",
        "  \n",
        "\n",
        "So, In python as there is already library available for Snowball Stemmer so we can directly implement by putting our data/word(s) into the function."
      ],
      "metadata": {
        "id": "Tc9yxFthSEXK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **(iii) Lancaster Stemmer**"
      ],
      "metadata": {
        "id": "IegoGo_WnOdF"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#----------------------------------------------------------------------LANCASTER STEMMER----------------------------------------------------------------------------#\n",
        "\n",
        "# importing required modules\n",
        "from nltk.stem.lancaster import LancasterStemmer\n",
        "\n",
        "\n",
        "# create a Lancaster stemmer object\n",
        "LS = LancasterStemmer()\n",
        "\n",
        "\n",
        "# example list of words to stem\n",
        "data = [\"jumping\", \"jumps\", \"jumped\", \"jumper\", \"jumpers\"]\n",
        "\n",
        "\n",
        "# use a for loop to stem each word in the list\n",
        "for i in data:\n",
        "  print(i, \" : \", LS.stem(i))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qxkj6bK0195m",
        "outputId": "94d82cc0-e9a1-4f80-8272-d3071622d985"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "jumping  :  jump\n",
            "jumps  :  jump\n",
            "jumped  :  jump\n",
            "jumper  :  jump\n",
            "jumpers  :  jump\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Step 1: Import required modules**  \n",
        "Firstly we imported required modules so from  our most important library in NLP i.e, **'nltk'** we imported **'stem'** and from that **'stem'** we imported **'LancasterStemmer'** \n",
        "So we can directly write it as **from nltk.stem import LancasterStemmer**  \n",
        "\n",
        "**Step 2: Choose data for performing operations using LancasterStemmer**  \n",
        "now we created an object 'LS' so that we can easily call the function LancasterStemmer(). And thus we stored function LancasterStemmer() in object LS.   \n",
        "LS = LancasterStemmer()  \n",
        "Now we choose some words and placed those in a list 'data'.\n",
        "  \n",
        "**Step 3: Implementation of LancasterStemmer on our list**  \n",
        "So as there are multiple words in our list so we run for loop for each word. Then we directly called function **'LS.stem()'** where LS is our object created before for storing function LancasterStemmer().  \n",
        "  \n",
        "\n",
        "So, In python as there is already library available for Lancaster Stemmer so we can directly implement by putting our data/word(s) into the function."
      ],
      "metadata": {
        "id": "CjQAkJ7GnU8B"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **(iv) Regexp Stemmer**"
      ],
      "metadata": {
        "id": "z2JEUL6SohGq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from nltk.stem import RegexpStemmer\n",
        "\n",
        "# create a Regexp stemmer object\n",
        "RS = RegexpStemmer('ing$|s$|ed$|es$|er$|able$', min=4)\n",
        "\n",
        "# example list of words to stem\n",
        "data = [\"jumping\", \"jumps\", \"jumped\", \"jumper\", \"jumpers\"]\n",
        "\n",
        "# use a for loop to stem each word in the list\n",
        "for i in data:\n",
        "  print(i, \" : \", RS.stem(i))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DRhzGecQ2uGm",
        "outputId": "a814e314-2b0f-42b1-ed1a-ad61e3b4f75e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "jumping  :  jump\n",
            "jumps  :  jump\n",
            "jumped  :  jump\n",
            "jumper  :  jump\n",
            "jumpers  :  jumper\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Step 1: Import required modules**  \n",
        "Firstly we imported required modules so from  our most important library in NLP i.e, **'nltk'** we imported **'stem'** and from that **'stem'** we imported **'RegexpStemmer'** \n",
        "So we can directly write it as **from nltk.stem import RegexpStemmer**  \n",
        "\n",
        "**Step 2: Choose data for performing operations using RegexpStemmer**  \n",
        "now we created an object 'RS' so that we can easily call the function RegexpStemmer(). And thus we stored function RegexpStemmer() in object RS.   \n",
        "RS = RegexpStemmer()  \n",
        "Now we choose some words and placed those in a list 'data'.\n",
        "  \n",
        "**Step 3: Implementation of RegexpStemmer on our list**  \n",
        "So as there are multiple words in our list so we run for loop for each word. Then we directly called function **'RS.stem()'** where RS is our object created before for storing function RegexpStemmer().  \n",
        "  \n",
        "\n",
        "So, In python as there is already library available for Regexp Stemmer so we can directly implement by putting our data/word(s) into the function."
      ],
      "metadata": {
        "id": "eo85CpdTomG9"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **Lemmatization** "
      ],
      "metadata": {
        "id": "bWk4ziwdccKY"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Lemmatization helps to do the morphological analysis of words. It is important to have the knowledge about the detailed dictionaries which the algorithm can refer to link the form back to its lemma.\n",
        "\n",
        "```\n",
        "========================================================\n",
        "Form    ||   Morphological information   ||   Lemma   ||\n",
        "========================================================\n",
        "========================================================\n",
        "Helps   || Third person singular number, ||   Help    ||\n",
        "        || present tense help            ||           ||\n",
        "Helping || Ing form of verb              ||   Help    ||\n",
        "========================================================\n",
        "\n",
        "```\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "0OM5aurlckOO"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Difference between stemming and lemmatization\n",
        "\n",
        "\n",
        "\n",
        "```\n",
        "Topic                ||              Stemming            ||            Lemmmatization \n",
        "====================================================================================================\n",
        "Goal                 ||   Remove inflactional forms      || Reduce inflactional forms (Lemmatization\n",
        "                     ||                                  || refers to do the    things properly                                        ||        with the help of \n",
        "                     ||                                  || vocabulary and morphological analysis of \n",
        "                     ||                                  || words)\n",
        "                     || (Stemming referss to the crude   ||\n",
        "                     ||  heuristic process which chops   ||\n",
        "                     ||  off the ends of the words in    ||\n",
        "                     ||  order to achieve the goal       ||\n",
        "                     ||  correctly)                      ||\n",
        "====================================================================================================\n",
        "Implementation       || Stemmers are typically easier to || Lemmatization is difficult to implemment\n",
        "                     || implement and run faster compare ||\n",
        "                     || to lemmatization                 ||\n",
        "====================================================================================================\n",
        "```\n",
        "\n"
      ],
      "metadata": {
        "id": "yeAzqXSPlC57"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **(i) WordNetLemmatizer**"
      ],
      "metadata": {
        "id": "6eSUJyc7rDF6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# importing required modules\n",
        "import nltk\n",
        "nltk.download('wordnet')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rjyTpcjJhmhO",
        "outputId": "80acebe0-1140-4106-e483-776b59c5ce3c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# importing modules\n",
        "from nltk.stem import WordNetLemmatizer\n",
        "\n",
        "WNL = WordNetLemmatizer()\n",
        "\n",
        "data = ['socks', 'rising', 'builder', 'sons']\n",
        "\n",
        "for i in data:\n",
        "  print(i, \" : \", WNL.lemmatize(i))\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Xy0_TaMspNg5",
        "outputId": "5b83b680-4c2f-4048-8ca9-4f15c097f6fa"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "socks  :  sock\n",
            "rising  :  rising\n",
            "builder  :  builder\n",
            "sons  :  son\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Step 1: Importing required libraries**\n",
        "Firstly we will import nltk module, then from that library we will download **'wordnet'** which is sementically oriented dictionary of English.  \n",
        "then we imported **'stem'** from **'nltk'** module and from **'stem'** we imported **'WordNetLemmatizer'**  \n",
        "We can directly import it as **'from nltk.stem import WordNetLemmatizer'**   \n",
        "\n",
        "\n",
        "**Step 2: Choose data for performing operations using WordNetLemmatizer**  \n",
        "now we created an object 'WNL' so that we can easily call the function WordNetLemmatizer(). And thus we stored function WordNetLemmatizer() in object WNL.   \n",
        "WNL = WordNetLemmatizer()  \n",
        "Now we choose some words and placed those in a list 'data'.\n",
        "  \n",
        "**Step 3: Implementation of WordNetLemmatizer on our list**  \n",
        "So as there are multiple words in our list so we run for loop for each word. Then we directly called function **'WNL.stem()'** where WNL is our object created before for storing function WordNetLemmatizer().  \n",
        "  \n",
        "\n",
        "So, In python as there is already library available for WordNet Lemmatizer so we can directly implement by putting our data/word(s) into the function."
      ],
      "metadata": {
        "id": "rjVfnj8cpewp"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **(ii) StanfordLemmatizer**"
      ],
      "metadata": {
        "id": "9ojErYqYrIrc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pip install stanfordnlp"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "H8YnM9jsrYKU",
        "outputId": "44d247c3-1acb-48b9-b10d-782eddc52651"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting stanfordnlp\n",
            "  Downloading stanfordnlp-0.2.0-py3-none-any.whl (158 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m158.8/158.8 KB\u001b[0m \u001b[31m3.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: tqdm in /usr/local/lib/python3.9/dist-packages (from stanfordnlp) (4.65.0)\n",
            "Requirement already satisfied: protobuf in /usr/local/lib/python3.9/dist-packages (from stanfordnlp) (3.20.3)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.9/dist-packages (from stanfordnlp) (2.27.1)\n",
            "Requirement already satisfied: torch>=1.0.0 in /usr/local/lib/python3.9/dist-packages (from stanfordnlp) (2.0.0+cu118)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.9/dist-packages (from stanfordnlp) (1.22.4)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.9/dist-packages (from torch>=1.0.0->stanfordnlp) (3.0)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.9/dist-packages (from torch>=1.0.0->stanfordnlp) (3.1.2)\n",
            "Requirement already satisfied: triton==2.0.0 in /usr/local/lib/python3.9/dist-packages (from torch>=1.0.0->stanfordnlp) (2.0.0)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.9/dist-packages (from torch>=1.0.0->stanfordnlp) (4.5.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.9/dist-packages (from torch>=1.0.0->stanfordnlp) (3.10.7)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.9/dist-packages (from torch>=1.0.0->stanfordnlp) (1.11.1)\n",
            "Requirement already satisfied: lit in /usr/local/lib/python3.9/dist-packages (from triton==2.0.0->torch>=1.0.0->stanfordnlp) (16.0.0)\n",
            "Requirement already satisfied: cmake in /usr/local/lib/python3.9/dist-packages (from triton==2.0.0->torch>=1.0.0->stanfordnlp) (3.25.2)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.9/dist-packages (from requests->stanfordnlp) (1.26.15)\n",
            "Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.9/dist-packages (from requests->stanfordnlp) (2.0.12)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/dist-packages (from requests->stanfordnlp) (3.4)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/dist-packages (from requests->stanfordnlp) (2022.12.7)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.9/dist-packages (from jinja2->torch>=1.0.0->stanfordnlp) (2.1.2)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.9/dist-packages (from sympy->torch>=1.0.0->stanfordnlp) (1.3.0)\n",
            "Installing collected packages: stanfordnlp\n",
            "Successfully installed stanfordnlp-0.2.0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import stanfordnlp\n",
        "\n",
        "# download the model and set up a pipeline\n",
        "stanfordnlp.download('en') # download the English model\n",
        "nlp = stanfordnlp.Pipeline(processors=\"tokenize,pos,lemma,depparse\", lang='en')\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SbqYRrMQrPXC",
        "outputId": "c49ad7c3-8132-4e46-d3a1-f446a6c30672"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Using the default treebank \"en_ewt\" for language \"en\".\n",
            "Would you like to download the models for: en_ewt now? (Y/n)\n",
            "Y\n",
            "\n",
            "Default download directory: /root/stanfordnlp_resources\n",
            "Hit enter to continue or type an alternate directory.\n",
            "\n",
            "\n",
            "Downloading models for: en_ewt\n",
            "Download location: /root/stanfordnlp_resources/en_ewt_models.zip\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "100%|██████████| 235M/235M [00:39<00:00, 5.88MB/s]\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Download complete.  Models saved to: /root/stanfordnlp_resources/en_ewt_models.zip\n",
            "Extracting models file for: en_ewt\n",
            "Cleaning up...Done.\n",
            "Use device: cpu\n",
            "---\n",
            "Loading: tokenize\n",
            "With settings: \n",
            "{'model_path': '/root/stanfordnlp_resources/en_ewt_models/en_ewt_tokenizer.pt', 'lang': 'en', 'shorthand': 'en_ewt', 'mode': 'predict'}\n",
            "---\n",
            "Loading: pos\n",
            "With settings: \n",
            "{'model_path': '/root/stanfordnlp_resources/en_ewt_models/en_ewt_tagger.pt', 'pretrain_path': '/root/stanfordnlp_resources/en_ewt_models/en_ewt.pretrain.pt', 'lang': 'en', 'shorthand': 'en_ewt', 'mode': 'predict'}\n",
            "---\n",
            "Loading: lemma\n",
            "With settings: \n",
            "{'model_path': '/root/stanfordnlp_resources/en_ewt_models/en_ewt_lemmatizer.pt', 'lang': 'en', 'shorthand': 'en_ewt', 'mode': 'predict'}\n",
            "Building an attentional Seq2Seq model...\n",
            "Using a Bi-LSTM encoder\n",
            "Using soft attention for LSTM.\n",
            "Finetune all embeddings.\n",
            "[Running seq2seq lemmatizer with edit classifier]\n",
            "---\n",
            "Loading: depparse\n",
            "With settings: \n",
            "{'model_path': '/root/stanfordnlp_resources/en_ewt_models/en_ewt_parser.pt', 'pretrain_path': '/root/stanfordnlp_resources/en_ewt_models/en_ewt.pretrain.pt', 'lang': 'en', 'shorthand': 'en_ewt', 'mode': 'predict'}\n",
            "Done loading processors!\n",
            "---\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# define a sentence to lemmatize\n",
        "sentence = 'The cats are running and jumping over the boxes.'\n",
        "\n",
        "# lemmatize the sentence\n",
        "doc = nlp(sentence)\n",
        "\n",
        "# print the lemmas\n",
        "lemmas = [word.lemma for sent in doc.sentences for word in sent.words]\n",
        "print(lemmas)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dyhkusR95P69",
        "outputId": "71e0250c-2ab7-4a62-d271-31346c3d3726"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['the', 'cat', 'be', 'run', 'and', 'jump', 'over', 'the', 'box', '.']\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.9/dist-packages/stanfordnlp/models/depparse/model.py:157: UserWarning: masked_fill_ received a mask with dtype torch.uint8, this behavior is now deprecated,please use a mask with dtype torch.bool instead. (Triggered internally at ../aten/src/ATen/native/TensorAdvancedIndexing.cpp:1772.)\n",
            "  unlabeled_scores.masked_fill_(diag, -float('inf'))\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Step 1: Install stanfordnlp in our collaboratory**\n",
        "pip install stanfordnlp\n",
        "\n",
        "**Step 2: Import stanfordnlp module**\n",
        "import stanfordnlp\n",
        "\n",
        "**Step 3: Download model from stanfordnlp module**\n",
        "As we are working for english language so we download model for english language (which is ultimately used from english dictionary)  \n",
        "stanfordnlp.download('en')  \n",
        "   \n",
        "As we downloaded the model so now we will set the pipeline for our model.  \n",
        "**Prcessor:** Processors are units of the neural pipeline that perform specific NLP functions and create different annotations for a Document\n",
        "  \n",
        "'tokenize' :  \n",
        "Processor class name: tokenize Processor   \n",
        "Requirement: -   \n",
        "Generated annotation: Segments a Document into Sentences, each containing a list of Tokens. This processor also predicts which tokens are multi-word tokens, but leaves expanding them to the MWTProcessor.  \n",
        "Description: Tokenizes the text and performs sentence segmentation.  \n",
        "  \n",
        "'pos' :   \n",
        "Processor class name: POS - Processor  \n",
        "Requirement: tokenize, mwt   \n",
        "Generated annotation: UPOS, XPOS, and UFeats annotations are accessible through Word’s properties pos, xpos, and ufeats.   \n",
        "Description: Labels tokens with their universal POS (UPOS) tags, treebank-specific POS (XPOS) tags, and universal morphological features (UFeats).  \n",
        "  \n",
        "'lemma' :   \n",
        "Processor class name: Lemma­Processor  \n",
        "Requirement: tokenize, mwt, pos  \n",
        "Generated annotation: Perform lemmatization on a Word using the Word.text and Word.upos values. The result can be accessed as Word.lemma.  \n",
        "Description: Generates the word lemmas for all words in the Document.  \n",
        "  \n",
        "'Depparse':  \n",
        "Processor class name: Depparse­Processor  \n",
        "Requirement: tokenize, mwt, pos, lemma  \n",
        "Generated annotation: Determines the syntactic head of each word in a sentence and the dependency relation between the two words that are accessible through Word’s head and deprel attributes.  \n",
        "Description: Provides an accurate syntactic dependency parsing analysis.  \n",
        "  \n",
        "Then we seclected language as english i.e., 'en'  \n",
        "So we have done with setting up the pipeline using processors and selection of language.  \n",
        "**Pipeline(processors = \"tokenize, pos, lemma, depparse\", lang = 'en')**  \n",
        "  \n",
        "**Step 4: Define Sentence for implementation**  \n",
        "now we created variable 'doc' in which we implemented our Stanford Lemmitizer to our sentence.  \n",
        "As we already have created object 'nlp' in which we stored the Stanford Lemmatizer by setting up the pipeline as  \n",
        "**nlp = stanfordnlp.Pipeline(processors = \"tokenize, pos, lemma, depparse\", lang = 'en')  **\n",
        "  \n",
        "And for implementation we directly called the variable nlp as:  \n",
        "doc = nlp(sentence)  \n",
        "  \n",
        "**Step 5: Defining lemma's of each word in sentence**  \n",
        "**lemma:** It is a list comprehension that extracts the lemma of each word in the document by iterating over the sentences and words in the document using nested loops. The lemmas are stored in a list called lemma.  \n",
        "**word.lemma:** It is an attribute of the word object that represents the lemma of the word. A lemma is the base or root form of a word, which is used to represent the word in its canonical or dictionary form.  \n",
        "**sent:** It is a variable that iterates over each sentence in doc.sentences. It represents a single sentence at each iteration of the loop.  \n",
        "**doc.sentences:** It is an attribute of the doc object that represents a list of sentences in the document. Each sentence is an object that contains various attributes and methods for processing the sentence.  \n",
        "word: It is a variable that iterates over each word in sent.words. It represents a single word at each iteration of the inner loop.   \n",
        "**sent.words:** It is an attribute of the sent object that represents a list of words in the sentence. Each word is an object that contains various attributes and methods for processing the word.  \n",
        "  \n",
        "**doc:** It is a document object that represents a text document which has been processed using the StanfordNLP library.  \n",
        "**word:** It is a variable that iterates over each word in sent.words. It represents a single word at each iteration of the inner loop.  \n",
        "**lemma:** It is a list comprehension that extracts the lemma of each word in the document by iterating over the sentences and words in the document using nested loops. The lemmas are stored in a list called lemma.  \n",
        "The code extracts the lemmas of words from a document processed using the StanfordNLP library and stores them in a list called lemma.  \n",
        "  \n",
        "lemmas = [word.lemma for sent in doc.sentences for word in sent.words]  \n",
        "print(lemmas)"
      ],
      "metadata": {
        "id": "GRNV0Wa7wDt9"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "# **(iii) spaCy Lemmatizer**"
      ],
      "metadata": {
        "id": "HgBKm2sq5i9C"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import spacy\n",
        "\n",
        "# load the English language model\n",
        "nlp = spacy.load('en_core_web_sm')\n",
        "\n",
        "# define a sentence to lemmatize\n",
        "sentence = 'The cats are running and jumping over the boxes.'\n",
        "\n",
        "# lemmatize the sentence\n",
        "doc = nlp(sentence)\n",
        "\n",
        "# print the lemmas\n",
        "lemmas = [token.lemma_ for token in doc]\n",
        "print(lemmas)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jZJLGsGZ5p4b",
        "outputId": "cfbddcd7-102d-4992-9260-d05bbc9bb380"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['the', 'cat', 'be', 'run', 'and', 'jump', 'over', 'the', 'box', '.']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Step 1: Import required modules**  \n",
        "import spacy   \n",
        "\n",
        "The **spacy** library is a popular natural language processing (NLP) library in Python that provides a wide range of tools and functionalities for processing text data. One of the core functionalities of spaCy is its ability to load pre-trained language models, which are trained on large text corpora and can perform various NLP tasks such as part-of-speech tagging, named entity recognition, sentence segmentation, and dependency parsing.\n",
        "  \n",
        "'**en_core_web_sm**' is the name of the pre-trained language model for English provided by spaCy. It is a small-sized language model that is optimized for efficiency and is commonly used for general-purpose English language processing tasks. The spacy.load() function is used to load the language model into memory, making it available for further processing.  \n",
        "So in next step we loaded en_core_web_sm model from spacy and stored in an object **'nlp'**  \n",
        "  \n",
        "**Step 2: Choose a sentence for implementation of spacy lemmitizer**  \n",
        "Then we choose a sentence 'The cats are running and jumping over the boxes'  \n",
        "Then we implemented our 'nlp' object with the sentence and stored in an object 'doc'  \n",
        "doc = nlp(sentence)  \n",
        "  \n",
        "**Step 3: Finding and printing lemmas**  \n",
        "for finding lemmas we used **'token.lemma_ for token in doc'**   \n",
        "  \n",
        "**token.lemma_**: This is an attribute of a token object in SpaCy that represents the lemma (base form) of the token. For example, the lemma of the word \"running\" is \"run\", and the lemma of the word \"mice\" is \"mouse\".  \n",
        "  \n",
        "**token**: This is a loop variable that iterates over each token in the doc object. In SpaCy, a document is typically divided into individual tokens, which could be words, punctuation marks, or other units of text.  \n",
        "  \n",
        "**doc**: This refers to a processed document using SpaCy. It could be the result of calling the SpaCy pipeline on a text, such as nlp(\"This is an example sentence.\"), where nlp is a SpaCy language model.  \n",
        "  \n",
        "**for token in doc**: This is a loop that iterates over each token in the doc object. It allows you to access the lemma_ attribute of each token in the document and perform further operations or analysis based on the lemmatized forms of the words.  \n",
        "  \n",
        "Then we stored all lemmas in an object **'lemmas'**  \n",
        "Then we printed those lemmas."
      ],
      "metadata": {
        "id": "tmBJgxZzrIJP"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **(iv) Pattern Lemmatizer**"
      ],
      "metadata": {
        "id": "OUA5Frl965-k"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pip install pattern"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1B8eFdLS73pN",
        "outputId": "d61d3400-310d-4344-a85e-76dd47a11f86"
      },
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting pattern\n",
            "  Downloading Pattern-3.6.0.tar.gz (22.2 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m22.2/22.2 MB\u001b[0m \u001b[31m44.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: future in /usr/local/lib/python3.9/dist-packages (from pattern) (0.18.3)\n",
            "Collecting backports.csv\n",
            "  Downloading backports.csv-1.0.7-py2.py3-none-any.whl (12 kB)\n",
            "Collecting mysqlclient\n",
            "  Downloading mysqlclient-2.1.1.tar.gz (88 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m88.1/88.1 kB\u001b[0m \u001b[31m5.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.9/dist-packages (from pattern) (4.11.2)\n",
            "Requirement already satisfied: lxml in /usr/local/lib/python3.9/dist-packages (from pattern) (4.9.2)\n",
            "Collecting feedparser\n",
            "  Downloading feedparser-6.0.10-py3-none-any.whl (81 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m81.1/81.1 kB\u001b[0m \u001b[31m5.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting pdfminer.six\n",
            "  Downloading pdfminer.six-20221105-py3-none-any.whl (5.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.6/5.6 MB\u001b[0m \u001b[31m66.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.9/dist-packages (from pattern) (1.22.4)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.9/dist-packages (from pattern) (1.10.1)\n",
            "Requirement already satisfied: nltk in /usr/local/lib/python3.9/dist-packages (from pattern) (3.8.1)\n",
            "Collecting python-docx\n",
            "  Downloading python-docx-0.8.11.tar.gz (5.6 MB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.6/5.6 MB\u001b[0m \u001b[31m69.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting cherrypy\n",
            "  Downloading CherryPy-18.8.0-py2.py3-none-any.whl (348 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m348.4/348.4 kB\u001b[0m \u001b[31m26.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: requests in /usr/local/lib/python3.9/dist-packages (from pattern) (2.27.1)\n",
            "Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.9/dist-packages (from beautifulsoup4->pattern) (2.4.1)\n",
            "Requirement already satisfied: more-itertools in /usr/local/lib/python3.9/dist-packages (from cherrypy->pattern) (9.1.0)\n",
            "Collecting zc.lockfile\n",
            "  Downloading zc.lockfile-3.0.post1-py3-none-any.whl (9.8 kB)\n",
            "Collecting jaraco.collections\n",
            "  Downloading jaraco.collections-4.1.0-py3-none-any.whl (11 kB)\n",
            "Collecting cheroot>=8.2.1\n",
            "  Downloading cheroot-9.0.0-py2.py3-none-any.whl (100 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m100.6/100.6 kB\u001b[0m \u001b[31m1.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hCollecting portend>=2.1.1\n",
            "  Downloading portend-3.1.0-py3-none-any.whl (5.3 kB)\n",
            "Collecting sgmllib3k\n",
            "  Downloading sgmllib3k-1.0.0.tar.gz (5.8 kB)\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.9/dist-packages (from nltk->pattern) (4.65.0)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.9/dist-packages (from nltk->pattern) (8.1.3)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.9/dist-packages (from nltk->pattern) (1.2.0)\n",
            "Requirement already satisfied: regex>=2021.8.3 in /usr/local/lib/python3.9/dist-packages (from nltk->pattern) (2022.10.31)\n",
            "Requirement already satisfied: cryptography>=36.0.0 in /usr/local/lib/python3.9/dist-packages (from pdfminer.six->pattern) (40.0.2)\n",
            "Requirement already satisfied: charset-normalizer>=2.0.0 in /usr/local/lib/python3.9/dist-packages (from pdfminer.six->pattern) (2.0.12)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/dist-packages (from requests->pattern) (3.4)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.9/dist-packages (from requests->pattern) (1.26.15)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/dist-packages (from requests->pattern) (2022.12.7)\n",
            "Collecting jaraco.functools\n",
            "  Downloading jaraco.functools-3.6.0-py3-none-any.whl (7.9 kB)\n",
            "Requirement already satisfied: six>=1.11.0 in /usr/local/lib/python3.9/dist-packages (from cheroot>=8.2.1->cherrypy->pattern) (1.16.0)\n",
            "Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.9/dist-packages (from cryptography>=36.0.0->pdfminer.six->pattern) (1.15.1)\n",
            "Collecting tempora>=1.8\n",
            "  Downloading tempora-5.2.2-py3-none-any.whl (13 kB)\n",
            "Collecting jaraco.text\n",
            "  Downloading jaraco.text-3.11.1-py3-none-any.whl (11 kB)\n",
            "Requirement already satisfied: setuptools in /usr/local/lib/python3.9/dist-packages (from zc.lockfile->cherrypy->pattern) (67.6.1)\n",
            "Requirement already satisfied: pycparser in /usr/local/lib/python3.9/dist-packages (from cffi>=1.12->cryptography>=36.0.0->pdfminer.six->pattern) (2.21)\n",
            "Requirement already satisfied: pytz in /usr/local/lib/python3.9/dist-packages (from tempora>=1.8->portend>=2.1.1->cherrypy->pattern) (2022.7.1)\n",
            "Collecting autocommand\n",
            "  Downloading autocommand-2.2.2-py3-none-any.whl (19 kB)\n",
            "Collecting jaraco.context>=4.1\n",
            "  Downloading jaraco.context-4.3.0-py3-none-any.whl (5.3 kB)\n",
            "Requirement already satisfied: inflect in /usr/local/lib/python3.9/dist-packages (from jaraco.text->jaraco.collections->cherrypy->pattern) (6.0.4)\n",
            "Requirement already satisfied: pydantic>=1.9.1 in /usr/local/lib/python3.9/dist-packages (from inflect->jaraco.text->jaraco.collections->cherrypy->pattern) (1.10.7)\n",
            "Requirement already satisfied: typing-extensions>=4.2.0 in /usr/local/lib/python3.9/dist-packages (from pydantic>=1.9.1->inflect->jaraco.text->jaraco.collections->cherrypy->pattern) (4.5.0)\n",
            "Building wheels for collected packages: pattern, mysqlclient, python-docx, sgmllib3k\n",
            "  Building wheel for pattern (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pattern: filename=Pattern-3.6-py3-none-any.whl size=22332718 sha256=0dbb8832fd7f8737bd913c54ec6c4f9a4d1d2b9ac216786389a15efc0b7fdca6\n",
            "  Stored in directory: /root/.cache/pip/wheels/50/33/f3/ea00b80d50c09f210588bda15ec60bdb38b289b452577cd5c3\n",
            "  Building wheel for mysqlclient (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for mysqlclient: filename=mysqlclient-2.1.1-cp39-cp39-linux_x86_64.whl size=107589 sha256=ac69ad3b9921016b7c82312dfc0fcf07ac9207c2d1c08cc9c9d87cc0aa262986\n",
            "  Stored in directory: /root/.cache/pip/wheels/f3/a5/27/c6312d8008951cfd5511684378a9e057b82006c70e1fea6107\n",
            "  Building wheel for python-docx (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for python-docx: filename=python_docx-0.8.11-py3-none-any.whl size=184505 sha256=6b133b194f7ff8bd11f8b328d80312ce67ec5bcefafd4f50ef4a70d0cab6235e\n",
            "  Stored in directory: /root/.cache/pip/wheels/83/8b/7c/09ae60c42c7ba4ed2dddaf2b8b9186cb105255856d6ed3dba5\n",
            "  Building wheel for sgmllib3k (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for sgmllib3k: filename=sgmllib3k-1.0.0-py3-none-any.whl size=6065 sha256=a75c2def65908b54974b089c9c39b9209e68fee308555ccb19084a5ba09f4d29\n",
            "  Stored in directory: /root/.cache/pip/wheels/65/7a/a7/78c287f64e401255dff4c13fdbc672fed5efbfd21c530114e1\n",
            "Successfully built pattern mysqlclient python-docx sgmllib3k\n",
            "Installing collected packages: sgmllib3k, backports.csv, zc.lockfile, python-docx, mysqlclient, jaraco.functools, jaraco.context, feedparser, autocommand, tempora, cheroot, portend, pdfminer.six, jaraco.text, jaraco.collections, cherrypy, pattern\n",
            "Successfully installed autocommand-2.2.2 backports.csv-1.0.7 cheroot-9.0.0 cherrypy-18.8.0 feedparser-6.0.10 jaraco.collections-4.1.0 jaraco.context-4.3.0 jaraco.functools-3.6.0 jaraco.text-3.11.1 mysqlclient-2.1.1 pattern-3.6 pdfminer.six-20221105 portend-3.1.0 python-docx-0.8.11 sgmllib3k-1.0.0 tempora-5.2.2 zc.lockfile-3.0.post1\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from pattern.en import lemma\n",
        "\n",
        "# define a list of words to lemmatize\n",
        "words = ['cats', 'running', 'swam', 'jumping', 'boxes']\n",
        "\n",
        "for i in words:\n",
        "  # lemmatize the words using a list comprehension\n",
        "    lemmas = [lemma(i) for i in words]\n",
        "\n",
        "# print the lemmas\n",
        "print(lemmas)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "aMHbwKIH7QPv",
        "outputId": "dd8c877e-05ed-448b-cfeb-049df5bae135"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['cat', 'run', 'swim', 'jump', 'box']\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Step 1: Install pattern**:  \n",
        "**pip install pattern**  \n",
        "  \n",
        "**Step 2: Import lemma:**  \n",
        "from pattern import en. from en import lemma  \n",
        "here en is for english dictionary.  \n",
        "So we can directly do:  \n",
        "  \n",
        "**from pattern.en import lemma**  \n",
        "   \n",
        "**Step 3: consider some words/data:**  \n",
        "In object 'words' we stored our data  \n",
        "**word = ['cats', 'running', 'swam', 'jumping', 'boxes']**  \n",
        "  \n",
        "**Step 4: Find out lemma from data:**  \n",
        "Now we have to find lemma from each word present in 'words'  \n",
        "So for this we will run our for loop  \n",
        "Then we created an object **'lemmas'** in which we will be storing all lemmas of all words  \n",
        "**lemma(i) for i in words**  \n",
        "It means lemma of 'i' for i in words.  \n",
        "\n",
        "**Step 5: Print lemmas:**  \n",
        "print(lemmas)"
      ],
      "metadata": {
        "id": "Qi3miXc2qpoU"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **(v) TreeTagger Lemmatizer**"
      ],
      "metadata": {
        "id": "D1C_dCpS8i2b"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pip install treetaggerwrapper"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LcE2GEdm9ClD",
        "outputId": "df0651fc-b7e6-4989-e286-2dd4640e1dcb"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/\n",
            "Collecting treetaggerwrapper\n",
            "  Downloading treetaggerwrapper-2.3.tar.gz (43 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m43.8/43.8 KB\u001b[0m \u001b[31m4.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Building wheels for collected packages: treetaggerwrapper\n",
            "  Building wheel for treetaggerwrapper (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for treetaggerwrapper: filename=treetaggerwrapper-2.3-py3-none-any.whl size=40774 sha256=0bd8708a52567609c4f16fcf04f4b2f9b3a41a617cfc55a0540024de7e0ecd8b\n",
            "  Stored in directory: /root/.cache/pip/wheels/53/3f/68/dbea5bf2abd18d7d33c8f3f9a13151cae1bafed5e3ec97c473\n",
            "Successfully built treetaggerwrapper\n",
            "Installing collected packages: treetaggerwrapper\n",
            "Successfully installed treetaggerwrapper-2.3\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd\n",
        "import treetaggerwrapper as tt\n",
        "\n",
        "t_tagger = tt.TreeTagger(TAGLANG ='en', TAGDIR = \"/content/F:\\TreeTagger\\tree-tagger-windows-3.2.3\\TreeTagger\")\n",
        "\n",
        "pos_tags = t_tagger.tag_text(\"the bats saw the cats with best stripes hanging upside down by their feet\")\n",
        "\n",
        "original = []\n",
        "lemmas = []\n",
        "tags = []\n",
        "for t in pos_tags:\n",
        "\toriginal.append(t.split('\\t')[0])\n",
        "\ttags.append(t.split('\\t')[1])\n",
        "\tlemmas.append(t.split('\\t')[-1])\n",
        "\n",
        "Results = pd.DataFrame({'Original': original, 'Lemma': lemmas, 'Tags': tags})\n",
        "print(Results)\n",
        "\n",
        "#>\t Original Lemma Tags\n",
        "# 0\t the\t the DT\n",
        "# 1\t bats\t bat NNS\n",
        "# 2\t saw\t see VVD\n",
        "# 3\t the\t the DT\n",
        "# 4\t cats\t cat NNS\n",
        "# 5\t with with IN\n",
        "# 6\t best good JJS\n",
        "# 7 stripes stripe NNS\n",
        "# 8 hanging hang VVG\n",
        "# 9 upside upside RB\n",
        "# 10\t down down RB\n",
        "# 11\t by\t by IN\n",
        "# 12 their their PP$\n",
        "# 13\t feet foot NNS\n"
      ],
      "metadata": {
        "id": "9wMhy9haIpOZ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **(vi) Pattern Lemmatization**"
      ],
      "metadata": {
        "id": "uRoa6-GeYlFc"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import nltk\n",
        "from nltk.stem import WordNetLemmatizer\n",
        "from nltk.corpus import wordnet"
      ],
      "metadata": {
        "id": "SrUsO3AcY-W-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "nltk.download('averaged_perceptron_tagger')\n",
        "nltk.download('wordnet')"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Nf1cBVriZE_F",
        "outputId": "93df6ea4-9aed-4e21-de6a-7fde05b14715"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
            "[nltk_data]     /root/nltk_data...\n",
            "[nltk_data]   Unzipping taggers/averaged_perceptron_tagger.zip.\n",
            "[nltk_data] Downloading package wordnet to /root/nltk_data...\n",
            "[nltk_data]   Package wordnet is already up-to-date!\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "def get_wordnet_pos(word):\n",
        "    \"\"\"Map POS tag to first character used by WordNetLemmatizer\"\"\"\n",
        "    tag = nltk.pos_tag([word])[0][1][0].upper()\n",
        "    tag_dict = {\"J\": wordnet.ADJ,\n",
        "                \"N\": wordnet.NOUN,\n",
        "                \"V\": wordnet.VERB,\n",
        "                \"R\": wordnet.ADV}\n",
        "    return tag_dict.get(tag, wordnet.NOUN)"
      ],
      "metadata": {
        "id": "Ep5acaf4ZIPX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "def lemmatize_text(text):\n",
        "    lemmatizer = WordNetLemmatizer()\n",
        "    lemmatized_text = []\n",
        "    for word in text.split():\n",
        "        pos = get_wordnet_pos(word)\n",
        "        lemma = lemmatizer.lemmatize(word, pos=pos)\n",
        "        lemmatized_text.append(lemma)\n",
        "    return \" \".join(lemmatized_text)"
      ],
      "metadata": {
        "id": "Cc6vXb02ZL8k"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "text = \"The dogs were barking loudly outside.\"\n",
        "lemmatized_text = lemmatize_text(text)\n",
        "print(lemmatized_text)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "n1yST75HZNz9",
        "outputId": "ba4b5dd0-1c3f-4cb2-d5f5-a4e007dce40c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "The dog be barking loudly outside.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Step 1: Import required modules:**  \n",
        "For pattern lemmatization we require WordNetLemmatizer and wordnet  \n",
        "  \n",
        "import nltk  \n",
        "from nltk.stem import WordNetLemmatizer  \n",
        "from nltk.corpus import wordnet  \n",
        "  \n",
        "**Step 2: Download rerquired modules:**  \n",
        "  \n",
        "For Pattern Lemmatization we need **'average-perception_tagger'** and **'wordnet'**  \n",
        "**nltk.download('averaged_perceptron_tagger')**  \n",
        "**nltk.download('wordnet')**  \n",
        "  \n",
        "**Step 3:  Tagginng Dictionary:**  \n",
        "Now we created a function called **get_wordnet_pos** that maps a Part-of-Speech (POS) tag to the corresponding first character used by the 'WordNetLemmatizer', which is a tool for lemmatizing words in the English language. The function takes a single argument word as input, which is the word for which the POS tag needs to be mapped.\n",
        "\n",
        "The code first uses the nltk library's pos_tag function to obtain the POS tag for the input word. It then maps the obtained POS tag to the appropriate character used by the WordNetLemmatizer using a dictionary called tag_dict. The keys in tag_dict are the first characters of the POS tags (**'J'** for adjective, **'N'** for noun, **'V'** for verb, and **'R'** for adverb), and the corresponding values are the corresponding constants defined in the wordnet module of the nltk library (i.e., wordnet.ADJ for adjective, wordnet.NOUN for noun, wordnet.VERB for **verb**, and wordnet.ADV for adverb). If the obtained POS tag is not found in tag_dict, it defaults to using wordnet.NOUN as the POS tag for lemmatization.\n",
        "\n",
        "The function then returns the mapped POS tag, which can be used as an argument in the WordNetLemmatizer's lemmatize function to lemmatize the input word based on its POS tag.  \n",
        "  \n",
        "\n",
        "```\n",
        "def get_wordnet_pos(word):\n",
        "    \"\"\"Map POS tag to first character used by WordNetLemmatizer\"\"\"\n",
        "    tag = nltk.pos_tag([word])[0][1][0].upper()\n",
        "    tag_dict = {\"J\": wordnet.ADJ,\n",
        "                \"N\": wordnet.NOUN,\n",
        "                \"V\": wordnet.VERB,\n",
        "                \"R\": wordnet.ADV}\n",
        "    return tag_dict.get(tag, wordnet.NOUN)\n",
        "```\n",
        "\n",
        "\n",
        "    \n",
        "  \n",
        "\n",
        "**Step 4: Lemmatize each word in the input text using WordNetLemmatizer from the NLTK library, taking into account the appropriate POS tag for each word, and returns the lemmatized text as a string.**\n",
        "We created a function called lemmatize_text that lemmatizes a given input text using the WordNetLemmatizer from the NLTK (Natural Language Toolkit) library.\n",
        "\n",
        "The function takes a single argument text as input, which is the text to be lemmatized. The code first initializes a WordNetLemmatizer object from the NLTK library. Then, it initializes an empty list called lemmatized_text to store the lemmatized words.\n",
        "\n",
        "The code then iterates through each word in the input text by splitting it on whitespace using the split() method. For each word, it calls the previously defined get_wordnet_pos function to obtain the corresponding POS tag using the get_wordnet_pos(word) function. It then uses the lemmatize method of the WordNetLemmatizer object to lemmatize the word, passing the obtained POS tag as the pos parameter. The lemmatized word is appended to the lemmatized_text list.\n",
        "\n",
        "After processing all the words in the input text, the function returns the lemmatized text as a string by joining the words in the lemmatized_text list using the join() method with whitespace as the separator.  \n",
        "  \n",
        "  \n",
        "\n",
        "```\n",
        "def lemmatize_text(text):    \n",
        "    lemmatizer = WordNetLemmatizer()  \n",
        "    lemmatized_text = []  \n",
        "    for word in text.split():  \n",
        "        pos = get_wordnet_pos(word)   \n",
        "        lemma = lemmatizer.lemmatize(word, pos=pos)   \n",
        "        lemmatized_text.append(lemma)   \n",
        "    return \" \".join(lemmatized_text)\n",
        "```\n",
        "  \n",
        "**Step 5: Consider a text for implementiing this lemmatizer:**  \n",
        "So we created an object for storing our sentence and named it as 'text'  \n",
        "Then we stored our sentence in text.  \n",
        "  \n",
        "Then we created an object 'lemmatized_text' in which we are going to store the lemmatization of sentence, then we call lemmatize_text() function and then applied it for out object 'text'  \n",
        "lemmatized_text = lemmatize_text(text)  \n",
        "  \n",
        "Then we print our lemmatized sentence \n",
        "print(lemmatized_text)"
      ],
      "metadata": {
        "id": "E54HQOqq3qCh"
      }
    }
  ]
}