Part 9 BOW and Ngram.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "358d74d3-a186-45ae-b376-fd8d69b76b5a",
   "metadata": {},
   "source": [
    "## Implemention of BOW and ngram using NLTK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "7d29c84d-14ab-428f-96b5-2405d3ffa8c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "# Sample text data\n",
    "documents = [\n",
    "    \"Natural language processing is a fascinating field of artificial intelligence that deals with human language.\",\n",
    "    \"Machine learning and natural language processing often go hand in hand to create smarter applications.\",\n",
    "    \"The advancements in artificial intelligence are transforming industries, enabling new insights and possibilities.\",\n",
    "    \"Natural language processing helps machines understand and respond to human communication.\",\n",
    "    \"AI, machine learning, and natural language processing are important for building intelligent applications.\"\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "01b71a12-f792-4997-a580-f3ae0a11fff2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary for BoW: ['advancements' 'ai' 'and' 'applications' 'are' 'artificial' 'building'\n",
      " 'communication' 'create' 'deals' 'enabling' 'fascinating' 'field' 'for'\n",
      " 'go' 'hand' 'helps' 'human' 'important' 'in' 'industries' 'insights'\n",
      " 'intelligence' 'intelligent' 'is' 'language' 'learning' 'machine'\n",
      " 'machines' 'natural' 'new' 'of' 'often' 'possibilities' 'processing'\n",
      " 'respond' 'smarter' 'that' 'the' 'to' 'transforming' 'understand' 'with']\n",
      "BoW Matrix:\n",
      " [[0 0 0 0 0 1 0 0 0 1 0 1 1 0 0 0 0 1 0 0 0 0 1 0 1 2 0 0 0 1 0 1 0 0 1 0\n",
      "  0 1 0 0 0 0 1]\n",
      " [0 0 1 1 0 0 0 0 1 0 0 0 0 0 1 2 0 0 0 1 0 0 0 0 0 1 1 1 0 1 0 0 1 0 1 0\n",
      "  1 0 0 1 0 0 0]\n",
      " [1 0 1 0 1 1 0 0 0 0 1 0 0 0 0 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0 1 0 0 1 0 0\n",
      "  0 0 1 0 1 0 0]\n",
      " [0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 1 0 0 1 1 0 0 0 0 1 1\n",
      "  0 0 0 1 0 1 0]\n",
      " [0 1 1 1 1 0 1 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 0 1 1 1 0 1 0 0 0 0 1 0\n",
      "  0 0 0 0 0 0 0]]\n"
     ]
    }
   ],
   "source": [
    "# Initialize CountVectorizer for BoW\n",
    "bow_vectorizer = CountVectorizer()\n",
    "\n",
    "# Fit and transform the documents to BoW\n",
    "bow_matrix = bow_vectorizer.fit_transform(documents)\n",
    "\n",
    "# Convert to array and display results\n",
    "bow_array = bow_matrix.toarray()\n",
    "print(\"Vocabulary for BoW:\", bow_vectorizer.get_feature_names_out())\n",
    "print(\"BoW Matrix:\\n\", bow_array)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04795f8c-438a-443e-b5fc-3f30c053425d",
   "metadata": {},
   "source": [
    "- Vocabulary: Shows a list of unique words across all documents.\n",
    "\n",
    "- BoW Matrix: Displays a matrix where each row represents a document, and each column represents the count of a word from the vocabulary in that document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d8938ed9-8669-468f-99c6-9c487b10be33",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary for N-grams: ['advancements' 'advancements in' 'ai' 'ai machine' 'and' 'and natural'\n",
      " 'and possibilities' 'and respond' 'applications' 'are' 'are important'\n",
      " 'are transforming' 'artificial' 'artificial intelligence' 'building'\n",
      " 'building intelligent' 'communication' 'create' 'create smarter' 'deals'\n",
      " 'deals with' 'enabling' 'enabling new' 'fascinating' 'fascinating field'\n",
      " 'field' 'field of' 'for' 'for building' 'go' 'go hand' 'hand' 'hand in'\n",
      " 'hand to' 'helps' 'helps machines' 'human' 'human communication'\n",
      " 'human language' 'important' 'important for' 'in' 'in artificial'\n",
      " 'in hand' 'industries' 'industries enabling' 'insights' 'insights and'\n",
      " 'intelligence' 'intelligence are' 'intelligence that' 'intelligent'\n",
      " 'intelligent applications' 'is' 'is fascinating' 'language'\n",
      " 'language processing' 'learning' 'learning and' 'machine'\n",
      " 'machine learning' 'machines' 'machines understand' 'natural'\n",
      " 'natural language' 'new' 'new insights' 'of' 'of artificial' 'often'\n",
      " 'often go' 'possibilities' 'processing' 'processing are'\n",
      " 'processing helps' 'processing is' 'processing often' 'respond'\n",
      " 'respond to' 'smarter' 'smarter applications' 'that' 'that deals' 'the'\n",
      " 'the advancements' 'to' 'to create' 'to human' 'transforming'\n",
      " 'transforming industries' 'understand' 'understand and' 'with'\n",
      " 'with human']\n",
      "N-gram Matrix:\n",
      " [[0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 1 1 0 0 1 1 1 1 0 0 0 0 0 0 0 0 0\n",
      "  1 0 1 0 0 0 0 0 0 0 0 0 1 0 1 0 0 1 1 2 1 0 0 0 0 0 0 1 1 0 0 1 1 0 0 0\n",
      "  1 0 0 1 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 1 1]\n",
      " [0 0 0 0 1 1 0 0 1 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 1 1 2 1 1 0 0\n",
      "  0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 0 0 1 1 0 0 0 0 1 1 0\n",
      "  1 0 0 0 1 0 0 1 1 0 0 0 0 1 1 0 0 0 0 0 0 0]\n",
      " [1 1 0 0 1 0 1 0 0 1 0 1 1 1 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
      "  0 0 0 0 0 1 1 0 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 1\n",
      "  0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 1 1 0 0 0 0]\n",
      " [0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1\n",
      "  1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 1 1 1 1 0 0 0 0 0 0 0\n",
      "  1 0 1 0 0 1 1 0 0 0 0 0 0 1 0 1 0 0 1 1 0 0]\n",
      " [0 0 1 1 1 1 0 0 1 1 1 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0\n",
      "  0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 1 1 0 0 1 1 1 1 1 1 0 0 1 1 0 0 0 0 0 0 0\n",
      "  1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]]\n"
     ]
    }
   ],
   "source": [
    "# Initialize CountVectorizer for N-grams with ngram_range\n",
    "ngram_vectorizer = CountVectorizer(ngram_range=(1, 2))\n",
    "\n",
    "# Fit and transform the documents to get N-gram representation\n",
    "ngram_matrix = ngram_vectorizer.fit_transform(documents)\n",
    "\n",
    "# Convert to array and display results\n",
    "ngram_array = ngram_matrix.toarray()\n",
    "print(\"Vocabulary for N-grams:\", ngram_vectorizer.get_feature_names_out())\n",
    "print(\"N-gram Matrix:\\n\", ngram_array)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b63ac56d-d033-42e6-9416-f5384ae38b4b",
   "metadata": {},
   "source": [
    "- Vocabulary for N-grams: Contains both single words and two-word combinations, capturing more context than BoW.\n",
    "\n",
    "- N-gram Matrix: Each row now represents a document with counts for both individual words and word pairs.\n",
    "Example Out"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fb9f133-1656-4810-bb2d-fd66d4e033a1",
   "metadata": {},
   "source": [
    "Conclusion :\n",
    "\n",
    "BoW provides word counts without capturing word order.\n",
    "N-grams capture both individual words and phrases, helping to preserve word context for more complex relationships."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ef6514e-9044-4dfc-8484-ce1da3ed15f6",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}