Part_11_Word2Vec.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "### Word Embedding in NLP\n",
        "\n",
        "Word embedding is a technique used in Natural Language Processing (NLP) to map words or phrases into vectors of real numbers. These vectors capture the semantic meaning of words, meaning words that are semantically similar will have similar vector representations.\n",
        "\n",
        "### Concept Overview\n",
        "\n",
        "Unlike traditional methods like **Bag-of-Words (BOW)** or **TF-IDF**, where words are represented as sparse vectors based on counts or frequencies, **word embeddings** provide a dense representation that captures **semantic relationships** between words. The key advantage of word embeddings is that they map words with similar meanings to nearby points in the vector space.\n",
        "\n",
        "#### Common Techniques for Word Embeddings\n",
        "\n",
        "- **Word2Vec**: Developed by Google, Word2Vec uses neural networks to learn vector representations of words by predicting context words (Skip-Gram) or predicting the target word (Continuous Bag of Words - CBOW).\n",
        "  \n",
        "- **GloVe**: Global Vectors for Word Representation (GloVe) is based on matrix factorization techniques and captures word co-occurrence statistics across the entire corpus.\n",
        "\n",
        "- **FastText**: An extension of Word2Vec, FastText models words as bags of character n-grams, allowing it to capture information about rare and out-of-vocabulary words.\n",
        "\n",
        "### How Word Embedding Works\n",
        "\n",
        "Word embeddings aim to learn the **context** of words based on their usage across different sentences. For example:\n",
        "\n",
        "- The words “king” and “queen” are related in meaning, so their word embeddings will be placed near each other in the vector space.\n",
        "- Similarly, the words “cat” and “dog” will have embeddings that are closer to each other than to other unrelated words like “car” or “house.”\n",
        "\n",
        "### Why Use Word Embeddings?\n",
        "\n",
        "- **Captures Semantics**: Word embeddings can capture not only syntactic similarity (how words are used in the same way) but also **semantic meaning** (relatedness of meaning).\n",
        "- **Handling Synonyms**: Words with similar meanings will have similar embeddings, making it easier for models to understand the meaning of words in context.\n",
        "- **Reduces Dimensionality**: Unlike traditional methods that use high-dimensional vectors (e.g., for BOW, the vector could be as large as the number of unique words in the corpus), word embeddings significantly reduce the dimensionality of the word representation."
      ],
      "metadata": {
        "id": "rOJTX5KHvOIp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "sentences = [\n",
        "    \"I love machine learning\",\n",
        "    \"Machine learning is fun\",\n",
        "    \"I enjoy learning new things\",\n",
        "    \"AI and ML are exciting fields\"\n",
        "]\n",
        "\n",
        "from gensim.models import Word2Vec\n",
        "\n",
        "# Sample sentences\n",
        "sentences = [\n",
        "    \"The movie was fantastic and exciting\",\n",
        "    \"I loved the movie, it was great\",\n",
        "    \"The film was boring and dull\",\n",
        "    \"I hated the movie, it was a disappointment\"\n",
        "]\n",
        "\n",
        "\n",
        "# Tokenize sentences into words (split on spaces for simplicity)\n",
        "tokenized_sentences = [sentence.split() for sentence in sentences]\n",
        "\n",
        "# Create the Word2Vec model\n",
        "model = Word2Vec(tokenized_sentences, min_count=1)\n",
        "\n",
        "# Now, let's check the embedding for some words\n",
        "embedding_movie = model.wv['movie']\n",
        "embedding_boring = model.wv['boring']\n",
        "embedding_fantastic = model.wv['fantastic']\n",
        "\n",
        "print(\"Embedding for 'movie':\", embedding_movie)\n",
        "print(\"Embedding for 'boring':\", embedding_boring)\n",
        "print(\"Embedding for 'fantastic':\", embedding_fantastic)\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "RRDc19-9vRl8",
        "outputId": "fe375cf6-b628-4fca-c40c-fd63622863c3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Embedding for 'movie': [ 1.30016683e-03 -9.80430283e-03  4.58776252e-03 -5.38222783e-04\n",
            "  6.33209571e-03  1.78347470e-03 -3.12979822e-03  7.75997294e-03\n",
            "  1.55466562e-03  5.52093989e-05 -4.61295387e-03 -8.45352374e-03\n",
            " -7.76683213e-03  8.67050979e-03 -8.92496016e-03  9.03471559e-03\n",
            " -9.28101782e-03 -2.76756298e-04 -1.90704700e-03 -8.93114600e-03\n",
            "  8.63005966e-03  6.77781366e-03  3.01943906e-03  4.83345287e-03\n",
            "  1.12190246e-04  9.42468084e-03  7.02128746e-03 -9.85372625e-03\n",
            " -4.43322072e-03 -1.29011157e-03  3.04772262e-03 -4.32395237e-03\n",
            "  1.44916656e-03 -7.84589909e-03  2.77807354e-03  4.70269192e-03\n",
            "  4.93731257e-03 -3.17570218e-03 -8.42704065e-03 -9.22061782e-03\n",
            " -7.22899451e-04 -7.32746487e-03 -6.81496272e-03  6.12000562e-03\n",
            "  7.17230327e-03  2.11741915e-03 -7.89940078e-03 -5.69898821e-03\n",
            "  8.05184525e-03  3.92084382e-03 -5.24047017e-03 -7.39190448e-03\n",
            "  7.71554711e-04  3.46375466e-03  2.07919348e-03  3.10080405e-03\n",
            " -5.62050007e-03 -9.88948625e-03 -7.02083716e-03  2.30308768e-04\n",
            "  4.61867917e-03  4.52630781e-03  1.87981245e-03  5.17067453e-03\n",
            " -1.05360748e-04  4.11416637e-03 -9.12324060e-03  7.70091172e-03\n",
            "  6.14747405e-03  5.12415636e-03  7.20666908e-03  8.43979698e-03\n",
            "  7.38695846e-04 -1.70386070e-03  5.18628338e-04 -9.31678060e-03\n",
            "  8.40621442e-03 -6.37993217e-03  8.42784252e-03 -4.24435502e-03\n",
            "  6.46842702e-04 -9.16416850e-03 -9.55856778e-03 -7.83681031e-03\n",
            " -7.73105631e-03  3.75581993e-04 -7.22646248e-03 -4.95021325e-03\n",
            " -5.27170673e-03 -4.28929785e-03  7.01231137e-03  4.82938997e-03\n",
            "  8.68277065e-03  7.09359162e-03 -5.69440611e-03  7.24079600e-03\n",
            " -9.29490291e-03 -2.58756871e-03 -7.75716640e-03  4.19260142e-03]\n",
            "Embedding for 'boring': [ 9.7670713e-03  8.1668682e-03  1.2808067e-03  5.0994353e-03\n",
            "  1.4103756e-03 -6.4561162e-03 -1.4276799e-03  6.4497986e-03\n",
            " -4.6174028e-03 -3.9938390e-03  4.9236775e-03  2.7123552e-03\n",
            " -1.8501916e-03 -2.8756098e-03  6.0118269e-03 -5.7156030e-03\n",
            " -3.2348486e-03 -6.4861295e-03 -4.2360132e-03 -8.5802320e-03\n",
            " -4.4686934e-03 -8.5118264e-03  1.4054843e-03 -8.6190198e-03\n",
            " -9.9147614e-03 -8.2009733e-03 -6.7737522e-03  6.6816676e-03\n",
            "  3.7836819e-03  3.5740284e-04 -2.9552744e-03 -7.4291672e-03\n",
            "  5.3240510e-04  4.9766537e-04  1.9621862e-04  8.5288764e-04\n",
            "  7.8743981e-04 -6.6640176e-05 -8.0049355e-03 -5.8681117e-03\n",
            " -8.3811739e-03 -1.3132995e-03  1.8200077e-03  7.4163843e-03\n",
            " -1.9631467e-03 -2.3232386e-03  9.4885547e-03  7.9429461e-05\n",
            " -2.4033193e-03  8.6048804e-03  2.6892552e-03 -5.3456482e-03\n",
            "  6.5880269e-03  4.5102746e-03 -7.0553347e-03 -3.2141677e-04\n",
            "  8.3710795e-04  5.7491809e-03 -1.7183756e-03 -2.8052460e-03\n",
            "  1.7479158e-03  8.4868609e-04  1.1922603e-03 -2.6351348e-03\n",
            " -5.9835799e-03  7.3232441e-03  7.5901309e-03  8.2965754e-03\n",
            " -8.5976087e-03  2.6375458e-03 -3.5608681e-03  9.6174162e-03\n",
            "  2.9047292e-03  4.6420493e-03  2.3856000e-03  6.6063758e-03\n",
            " -5.7455837e-03  7.8954613e-03 -2.4096770e-03 -4.5633586e-03\n",
            " -2.0635272e-03  9.7334562e-03 -6.8557505e-03 -2.1934037e-03\n",
            "  7.0023038e-03 -5.6522538e-05 -6.2936288e-03 -6.3952166e-03\n",
            "  8.9401677e-03  6.4274347e-03  4.7730920e-03 -3.2622160e-03\n",
            " -9.2670210e-03  3.7845913e-03  7.1607679e-03 -5.6326091e-03\n",
            " -7.8642461e-03 -2.9731481e-03 -4.9345461e-03 -2.3132151e-03]\n",
            "Embedding for 'fantastic': [ 7.6966463e-03  9.1206422e-03  1.1355019e-03 -8.3250795e-03\n",
            "  8.4250160e-03 -3.6962307e-03  5.7421732e-03  4.3915794e-03\n",
            "  9.6899448e-03 -9.2934975e-03  9.2084054e-03 -9.2815282e-03\n",
            " -6.9077122e-03 -9.1021946e-03 -5.5471100e-03  7.3688962e-03\n",
            "  9.1644777e-03 -3.3253515e-03  3.7230505e-03 -3.6252034e-03\n",
            "  7.8814710e-03  5.8668759e-03  2.0861626e-07 -3.6286747e-03\n",
            " -7.2243060e-03  4.7686161e-03  1.4529788e-03 -2.6131857e-03\n",
            "  7.8378068e-03 -4.0496145e-03 -9.1489861e-03 -2.2554707e-03\n",
            "  1.2514711e-04 -6.6392552e-03 -5.4866159e-03 -8.4997769e-03\n",
            "  9.2298733e-03  7.4240281e-03 -2.9524326e-04  7.3676636e-03\n",
            "  7.9507884e-03 -7.8357337e-04  6.6120909e-03  3.7675237e-03\n",
            "  5.0768424e-03  7.2529912e-03 -4.7393893e-03 -2.1855331e-03\n",
            "  8.7312341e-04  4.2362059e-03  3.3043313e-03  5.0958274e-03\n",
            "  4.5864857e-03 -8.4385090e-03 -3.1838394e-03 -7.2367596e-03\n",
            "  9.6814223e-03  5.0065992e-03  1.7084122e-04  4.1129780e-03\n",
            " -7.6561309e-03 -6.2946510e-03  3.0763936e-03  6.5346383e-03\n",
            "  3.9498745e-03  6.0180221e-03 -1.9861318e-03 -3.3451295e-03\n",
            "  2.0717025e-04 -3.1943608e-03 -5.5169044e-03 -7.7885604e-03\n",
            "  6.5355431e-03 -1.0903371e-03 -1.8908798e-03 -7.8047751e-03\n",
            "  9.3375733e-03  8.6814165e-04  1.7696369e-03  2.4916660e-03\n",
            " -7.3859929e-03  1.6388226e-03  2.9765631e-03 -8.5670296e-03\n",
            "  4.9558021e-03  2.4334085e-03  7.4979127e-03  5.0442982e-03\n",
            " -3.0317164e-03 -7.1629370e-03  7.0962133e-03  1.9015349e-03\n",
            "  5.1992359e-03  6.3811089e-03  1.9122792e-03 -6.1276113e-03\n",
            " -6.2966346e-06  8.2682976e-03 -6.0985480e-03  9.4382809e-03]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "similar_words = model.wv.most_similar('movie', topn=3)\n",
        "print(\"\\nMost similar words to 'movie':\")\n",
        "print(similar_words)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YL935fXTwrow",
        "outputId": "2333abe6-7420-4113-dbb5-ffb835a95a44"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Most similar words to 'movie':\n",
            "[('was', 0.21883949637413025), ('great', 0.1747603565454483), ('The', 0.16371983289718628)]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### How Does This Work?\n",
        "\n",
        "1. **Training the Model**: The `Word2Vec` model is trained on the tokenized sentences, and it learns to predict words in the context of the surrounding words. It produces word vectors that represent the semantic meaning of each word.\n",
        "2. **Vector Representation**: The word “machine” gets represented as a dense vector, and this vector will be similar to the vectors of other words that appear in similar contexts (e.g., “learning,” “AI,” “fields”).\n",
        "\n",
        "### Benefits of Word Embeddings\n",
        "\n",
        "- **Semantic Relationships**: The vector for “machine” is close to words like “learning” and “AI” because they often appear in similar contexts. This allows us to easily find related words and express semantic relationships.\n",
        "  \n",
        "- **Handling Similar Words**: Synonyms like \"car\" and \"automobile\" will have similar embeddings, which helps in tasks like **paraphrase detection** and **text classification**.\n",
        "\n",
        "- **Rich Features for Models**: Word embeddings are often used as inputs for other NLP models, such as for **text classification**, **sentiment analysis**, **translation**, and **question answering**, providing a powerful representation of text.\n",
        "\n",
        "### Word Embedding Applications\n",
        "\n",
        "- **Text Classification**: By using word embeddings as features, we can classify text into different categories like spam detection or sentiment analysis.\n",
        "- **Named Entity Recognition (NER)**: Word embeddings help in recognizing named entities like people, organizations, or locations by providing semantic context.\n",
        "- **Machine Translation**: Embeddings allow for translating words or phrases from one language to another by comparing their vector representations.\n",
        "- **Text Similarity**: Helps in measuring the similarity between texts by comparing the distance between their embeddings.\n",
        "\n",
        "### In Summary\n",
        "\n",
        "Word embeddings represent a key step forward in understanding language at a deeper level. They transform words into continuous vectors, making it easier for machines to understand the relationships and meanings between words. Techniques like **Word2Vec**, **GloVe**, and **FastText** help capture these relationships, enabling more accurate and efficient NLP applications like search, classification, translation, and more."
      ],
      "metadata": {
        "id": "cKLvDB9svynJ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import gensim\n",
        "from gensim.models import Word2Vec\n",
        "from nltk.tokenize import word_tokenize\n",
        "import nltk\n",
        "nltk.download('punkt')\n",
        "\n",
        "# Prepare a simple corpus\n",
        "corpus = [\n",
        "    \"Word2Vec is a great technique for word embeddings.\",\n",
        "    \"It helps in converting words into vectors.\",\n",
        "    \"Word2Vec is used in NLP tasks like sentiment analysis and machine translation.\",\n",
        "    \"The Skip-gram model predicts context words given a target word.\",\n",
        "    \"CBOW model predicts the target word from context words.\"\n",
        "]\n",
        "\n",
        "# Tokenize the sentences\n",
        "tokenized_corpus = [word_tokenize(sentence.lower()) for sentence in corpus]\n",
        "\n",
        "# Train the Word2Vec model\n",
        "model = Word2Vec(sentences=tokenized_corpus, vector_size=50, window=3, min_count=1, sg=1)\n",
        "\n",
        "# Get the vector for a specific word\n",
        "word_vector = model.wv['word2vec']\n",
        "print(f\"Word2Vec vector: {word_vector}\")\n",
        "\n",
        "# Find similar words\n",
        "similar_words = model.wv.most_similar('word2vec', topn=3)\n",
        "print(f\"Words similar to 'word2vec': {similar_words}\")\n",
        "\n",
        "# Save and load the model\n",
        "model.save(\"word2vec_model.model\")\n",
        "loaded_model = Word2Vec.load(\"word2vec_model.model\")\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "W67hIeQZvina",
        "outputId": "4dd2f3de-79d7-4e12-d3c2-d244e5ffbbfc"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[nltk_data] Downloading package punkt to /root/nltk_data...\n",
            "[nltk_data]   Unzipping tokenizers/punkt.zip.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Word2Vec vector: [ 1.5611659e-02 -1.9036822e-02 -3.6917988e-04  6.9416170e-03\n",
            " -1.8735953e-03  1.6766759e-02  1.8041069e-02  1.3068029e-02\n",
            " -1.4369206e-03  1.5424302e-02 -1.7057002e-02  6.3956575e-03\n",
            " -9.2831803e-03 -1.0187318e-02  7.1869758e-03  1.0755858e-02\n",
            "  1.5555240e-02 -1.1488882e-02  1.4866323e-02  1.3249519e-02\n",
            " -7.3942998e-03 -1.7453466e-02  1.0939996e-02  1.3034756e-02\n",
            " -1.5832902e-03 -1.3417154e-02 -1.4160382e-02 -4.9939575e-03\n",
            "  1.0307272e-02 -7.3558744e-03 -1.8780066e-02  7.6191304e-03\n",
            "  9.7674970e-03 -1.2905392e-02  2.3778665e-03 -4.1718069e-03\n",
            "  5.8256763e-05 -1.9791406e-02  5.3453855e-03 -9.4990488e-03\n",
            "  2.2296992e-03 -3.1321580e-03  4.4113072e-03 -1.5753033e-02\n",
            " -5.4209209e-03  5.3441166e-03  1.0683629e-02 -4.8003504e-03\n",
            " -1.9042172e-02  9.0447301e-03]\n",
            "Words similar to 'word2vec': [('tasks', 0.22867631912231445), ('machine', 0.20394939184188843), ('helps', 0.19049221277236938)]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "similar_words = model.wv.most_similar('word2vec', topn=10)\n",
        "print(f\"Top 10 words similar to 'word2vec': {similar_words}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-fxs1xPNyCY8",
        "outputId": "15873c58-a24a-4fba-83ca-cd4bb79c8cdf"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Top 10 words similar to 'word2vec': [('tasks', 0.22867631912231445), ('machine', 0.20394939184188843), ('helps', 0.19049221277236938), ('skip-gram', 0.1684013158082962), ('analysis', 0.13033269345760345), ('like', 0.05815975368022919), ('cbow', 0.04925908148288727), ('nlp', 0.048944756388664246), ('predicts', 0.0446699857711792), ('model', -0.00908462330698967)]\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "tvIqcsLxyVGh"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "ZypA7-SjyjXv"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}