3_2HD_Discrimination_and_Bias_in_NLP.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "coursera": {
      "course_slug": "nlp-sequence-models",
      "graded_item_id": "8hb5s",
      "launcher_item_id": "5NrJ6"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.6"
    },
    "colab": {
      "name": "3.2HD - Discrimination and Bias in NLP.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Ahmad4041/Task3.2HD/blob/master/3_2HD_Discrimination_and_Bias_in_NLP.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-lRobKV3tUQi",
        "colab_type": "text"
      },
      "source": [
        "GitHub: https://github.com/MYUSER/MYPROJECT/"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U3TFFNrQtUQj",
        "colab_type": "text"
      },
      "source": [
        "Welcome to your assignment this week! \n",
        "\n",
        "To better understand bias and discrimination in AI, in this assignment, we will look at a Natural Language Processing use case. \n",
        "\n",
        "Natural Language Pocessing (NLP) is a branch of Artificial Intelligence (AI) that helps computers to understand, to interpret and to manipulate natural (i.e. human) language.\n",
        "Imagine NLP-powered machines as black boxes that are capable of understanding and evaluating the context of the input documents (i.e. collection of words), outputting meaningful results that depend on the task the machine is designed for.\n",
        "\n",
        "\n",
        "![](images/1_3zMvUnPzYZF9CSHdj6hT5A.png \"the title\")\n",
        "\n",
        "\n",
        "\n",
        "<caption><center> Documents are fed into magic NLP model capable to get, for instance, the sentiment of the original content</center></caption>\n",
        "\n",
        "\n",
        "\n",
        "Just like any other machine learning algorithm, biased data results in biased outcomes. And just like any other algorithm, results debiasing is painfully annoying, to the point that it might be simpler to unbias the society itself.\n",
        "\n",
        "# The big deal: word embeddings\n",
        "\n",
        "Words must be represented as **numeric vectors** in order to be fed into machine learning algorithms. One of the most powerful (and popular) ways to do it is through **Word Embeddings**. In word embedding models, each word in a given language is assigned to a high-dimensional vector, such that **the geometry of the vectors captures relations between the words**. \n",
        "\n",
        "\n",
        "Because word embeddings are very computionally expensive to train, most ML practitioners will load a pre-trained set of embeddings. \n",
        "\n",
        "**After this assignment you will be able to:**\n",
        "\n",
        "- Load pre-trained word vectors, and measure similarity using cosine similarity\n",
        "- Use word embeddings to solve word analogy problems such as Man is to Woman as King is to ______. \n",
        "- Modify word embeddings to reduce their gender bias \n",
        "\n",
        "Let's get started! Run the following cell to install all the packages you will need."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wiz_Rb7atUQk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# !pip install numpy\n",
        "# !pip install keras\n",
        "# !pip install tensorflow"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XCLqkPrTtUQr",
        "colab_type": "text"
      },
      "source": [
        "Run the following cell to load the packages you will need."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bcRxvfz3tUQs",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "from w2v_utils import *"
      ],
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nal4XVm-tUQv",
        "colab_type": "text"
      },
      "source": [
        "Next, lets load the word vectors. For this assignment, we will use 50-dimensional GloVe vectors to represent words. Run the following cell to load the `word_to_vec_map`. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fYv-ZA63tUQw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "words, word_to_vec_map = read_glove_vecs('glove.6B.50d.txt/My Drive/glove.6B.50d.txt')"
      ],
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dut0ueUotUQz",
        "colab_type": "text"
      },
      "source": [
        "You've loaded:\n",
        "- `words`: set of words in the vocabulary.\n",
        "- `word_to_vec_map`: dictionary mapping words to their GloVe vector representation.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YJWbq0T6tUQz",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        },
        "outputId": "d69dd98f-7db3-449e-d541-dd1e3464c95e"
      },
      "source": [
        "print(\"Example of words: \",list(words)[:10])\n",
        "print(\"Vector for word 'person' = \", word_to_vec_map.get('person'))"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Example of words:  ['calligrapher', 'smell', 'zingers', 'gwt', 'transpolar', 'bloodthirstiness', '1925-26', 'maccormac', 'yulong', 'bertholf']\n",
            "Vector for word 'person' =  [ 0.61734    0.40035    0.067786  -0.34263    2.0647     0.60844\n",
            "  0.32558    0.3869     0.36906    0.16553    0.0065053 -0.075674\n",
            "  0.57099    0.17314    1.0142    -0.49581   -0.38152    0.49255\n",
            " -0.16737   -0.33948   -0.44405    0.77543    0.20935    0.6007\n",
            "  0.86649   -1.8923    -0.37901   -0.28044    0.64214   -0.23549\n",
            "  2.9358    -0.086004  -0.14327   -0.50161    0.25291   -0.065446\n",
            "  0.60768    0.13984    0.018135  -0.34877    0.039985   0.07943\n",
            "  0.39318    1.0562    -0.23624   -0.4194    -0.35332   -0.15234\n",
            "  0.62158    0.79257  ]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iu-_NxgetUQ2",
        "colab_type": "text"
      },
      "source": [
        "GloVe vectors provide much more useful information about the meaning of individual words. Lets now see how you can use GloVe vectors to decide how similar two words are. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bLWqXb4HtUQ3",
        "colab_type": "text"
      },
      "source": [
        "# Cosine similarity\n",
        "\n",
        "To measure how similar two words are, we need a way to measure the degree of similarity between two embedding vectors for the two words. Given two vectors $u$ and $v$, cosine similarity is defined as follows: \n",
        "\n",
        "\\begin{equation}\n",
        "\\text{CosineSimilarity(u, v)} = \\frac {u . v} {||u||_2 ||v||_2} = cos(\\theta) \\tag{1}\n",
        "\\end{equation}\n",
        "\n",
        "where $u.v$ is the dot product (or inner product) of two vectors, $||u||_2$ is the norm (or length) of the vector $u$, and $\\theta$ is the angle between $u$ and $v$. This similarity depends on the angle between $u$ and $v$. If $u$ and $v$ are very similar, their cosine similarity will be close to 1; if they are dissimilar, the cosine similarity will take a smaller value. \n",
        "\n",
        "\n",
        "![](images/cosine_sim.png)\n",
        "\n",
        "<caption><center> <b>Figure 1</b>: The cosine of the angle between two vectors is a measure of how similar they are</center></caption>\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qTauTxsUtUQ3",
        "colab_type": "text"
      },
      "source": [
        "***\n",
        "\n",
        "**Task 1**: Implement the function `cosine_similarity()` to evaluate similarity between word vectors.\n",
        "\n",
        "**Reminder**: The norm of $u$ is defined as $||u||_2 = \\sqrt{\\sum_{i=1}^{n} u_i^2}$"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZeMhJEMktUQ4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "## Task 1\n",
        "# cosine_similarity\n",
        "\n",
        "def cosine_similarity(u, v):\n",
        "    \"\"\"\n",
        "    Cosine similarity reflects the degree of similariy between u and v\n",
        "        \n",
        "    Arguments:\n",
        "        u -- a word vector of shape (n,)          \n",
        "        v -- a word vector of shape (n,)\n",
        "\n",
        "    Returns:\n",
        "        cosine_similarity -- the cosine similarity between u and v defined by the formula above.\n",
        "    \"\"\"\n",
        "    ## START YOU CODE HERE\n",
        "    dot = np.dot(u,v)\n",
        "    norm_u =  np.linalg.norm(u)\n",
        "    cosine_similarity = dot/(norm_u*norm_v)\n",
        "    return cosine_similarity\n",
        "    ## END"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NVgvxi9_tUQ7",
        "colab_type": "text"
      },
      "source": [
        "***\n",
        "\n",
        "**Task 2**: Implement `most_similar_word` which returns the most similar word to a word."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "evGbtxLWtUQ7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "## Task 2\n",
        "# GRADED FUNCTION: most_similar_word\n",
        "\n",
        "def most_similar_word(word, word_to_vec_map):\n",
        "    \"\"\"\n",
        "    Most similar word return the most similar word to the word u.\n",
        "        \n",
        "    Arguments:\n",
        "        word -- a word, string        \n",
        "        word_to_vec_map -- dictionary that maps words to their corresponding vectors. \n",
        "    \n",
        "    Returns:\n",
        "    best_word --  the most similar word to u as measured by cosine similarity\n",
        "    \"\"\"\n",
        "    ## START YOU CODE HERE\n",
        "    \n",
        "    return cosine_similarity(word_to_vec_map.get(word),word_to_vec_map.get(word))\n",
        "\n",
        "    ## END\n"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8mcwsnsctUQ_",
        "colab_type": "text"
      },
      "source": [
        "Answer the questions below:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VklEVCdrtUQ_",
        "colab_type": "text"
      },
      "source": [
        "**TASK 3: Write a code the answer the following questions:**\n",
        "\n",
        "***"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HeuKZ25QtURA",
        "colab_type": "text"
      },
      "source": [
        "What is the similarity between the words brother and friend?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Bhfs6YvltURA",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 350
        },
        "outputId": "ee6a3403-e78b-4e0a-f829-2482c060c184"
      },
      "source": [
        "print(\"Similarity b/w brother and firend\")\n",
        "cosine_similarity(word_to_vec_map.get(\"brother\"),word_to_vec_map.get(\"firend\"))"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Similarity b/w brother and firend\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "error",
          "ename": "TypeError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-13-8b66b029df64>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Similarity b/w brother and firend\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mcosine_similarity\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mword_to_vec_map\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"brother\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mword_to_vec_map\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"firend\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;32m<ipython-input-11-913d3ab9a8ce>\u001b[0m in \u001b[0;36mcosine_similarity\u001b[0;34m(u, v)\u001b[0m\n\u001b[1;32m     14\u001b[0m     \"\"\"\n\u001b[1;32m     15\u001b[0m     \u001b[0;31m## START YOU CODE HERE\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 16\u001b[0;31m     \u001b[0mdot\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdot\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mu\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0mv\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     17\u001b[0m     \u001b[0mnorm_u\u001b[0m \u001b[0;34m=\u001b[0m  \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlinalg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnorm\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mu\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     18\u001b[0m     \u001b[0mcosine_similarity\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdot\u001b[0m\u001b[0;34m/\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnorm_u\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0mnorm_v\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<__array_function__ internals>\u001b[0m in \u001b[0;36mdot\u001b[0;34m(*args, **kwargs)\u001b[0m\n",
            "\u001b[0;31mTypeError\u001b[0m: unsupported operand type(s) for *: 'float' and 'NoneType'"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4OfMmL_BtURD",
        "colab_type": "text"
      },
      "source": [
        "What is the similarity between the words computer and kid?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VviJ9GSktURD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(\"Similarity b/w computer and kid\", cosine_similarity(word_to_vec_map.get(\"computer\"),word_to_vec_map.get(\"kid\")))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gBJ8QoB5tURG",
        "colab_type": "text"
      },
      "source": [
        "What is the similarity between the words V1=(france - paris) and V2=(rome - italy)?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2bXTeDiftURH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "france=word_to_vec_map.get(\"france\")\n",
        "paris=word_to_vec_map.get(\"paris\")\n",
        "rome=word_to_vec_map.get(\"rome\")\n",
        "italy=word_to_vec_map.get(\"italy\")\n",
        "v1=france-paris\n",
        "v2=rome-italy\n",
        "print(\"Similarity b/w  V1=(france - paris) and V2=(rome - italy)\", cosine_similarity(v1,v2))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "koPJIE4StURK",
        "colab_type": "text"
      },
      "source": [
        "What is the most similar word to computer?\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RptadbIQtURK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(\"Most similar Word to computer\",most_similar_word(\"computer\",word_to_vec_map))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OsNNW_G1tURN",
        "colab_type": "text"
      },
      "source": [
        "What is the most similar word to australia?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Raw8SH6XtURO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(\"Most similar Word to australia\",most_similar_word(\"australia\",word_to_vec_map))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OrRSRxmMtURQ",
        "colab_type": "text"
      },
      "source": [
        "What is the most similar word to python?\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "71jkC5f_tURR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(\"Most similar Word to python\",most_similar_word(\"python\",word_to_vec_map))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CWVVxARVtURU",
        "colab_type": "text"
      },
      "source": [
        "***"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T_f6GAaNtURV",
        "colab_type": "text"
      },
      "source": [
        "Playing around the cosine similarity of other inputs will give you a better sense of how word vectors behave. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "r6llkJY2tURV",
        "colab_type": "text"
      },
      "source": [
        "#  Word analogy task\n",
        "\n",
        "In the word analogy task, we complete the sentence <font color='brown'>\"*a* is to *b* as *c* is to **____**\"</font>. An example is <font color='brown'> '*man* is to *woman* as *king* is to *queen*' </font>. In detail, we are trying to find a word *d*, such that the associated word vectors $e_a, e_b, e_c, e_d$ are related in the following manner: $e_b - e_a \\approx e_d - e_c$. We will measure the similarity between $e_b - e_a$ and $e_d - e_c$ using cosine similarity. \n",
        "\n",
        "\n",
        "**Task 4**: Complete the code below to be able to perform word analogies!\n",
        "\n",
        "***"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1CHab9qgtURW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "## Task 4\n",
        "# GRADED FUNCTION: complete_analogy\n",
        "\n",
        "def complete_analogy(word_a, word_b, word_c, word_to_vec_map):\n",
        "    \"\"\"\n",
        "    Performs the word analogy task as explained above: a is to b as c is to ____. \n",
        "    \n",
        "    Arguments:\n",
        "    word_a -- a word, string\n",
        "    word_b -- a word, string\n",
        "    word_c -- a word, string\n",
        "    word_to_vec_map -- dictionary that maps words to their corresponding vectors. \n",
        "    \n",
        "    Returns:\n",
        "    best_word --  the word such that v_b - v_a is close to v_best_word - v_c, as measured by cosine similarity\n",
        "    \"\"\"\n",
        "    ## START YOU CODE HERE\n",
        "    word_a, word_b, word_c = word_a.lower(), word_b.lower(), word_c.lower()\n",
        "    e_a, e_b, e_c = word_to_vec_map[word_a],word_to_vec_map[word_b],word_to_vec_map[word_c] # transform words \n",
        "    words = word_to_vec_map.keys()\n",
        "    max_cosine_sim = -100              \n",
        "    best_word = None                   \n",
        "\n",
        "  \n",
        "    for w in words:        \n",
        "        if w in [word_a, word_b, word_c] :\n",
        "            continue\n",
        "        cosine_sim = cosine_similarity(e_b-e_a,  word_to_vec_map[w] - e_c)\n",
        "        \n",
        "        if cosine_sim > max_cosine_sim:\n",
        "            max_cosine_sim = cosine_sim\n",
        "            best_word = w\n",
        "    ##ENFD\n",
        "    return best_word"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rE9nwVCstURZ",
        "colab_type": "text"
      },
      "source": [
        "Run the cell below to test your code, this may take 1-2 minutes."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "829ESHLktURa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "triads_to_try = [('italy', 'italian', 'spain'), ('india', 'delhi', 'japan'), ('man', 'woman', 'boy'), ('small', 'smaller', 'large')]\n",
        "for triad in triads_to_try:\n",
        "    print ('{} -> {} :: {} -> {}'.format( *triad, complete_analogy(*triad,word_to_vec_map)))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xEKxEnQdtURd",
        "colab_type": "text"
      },
      "source": [
        "Once you get the correct expected output, please feel free to modify the input cells above to test your own analogies. Try to find some other analogy pairs that do work, but also find some where the algorithm doesn't give the right answer: For example, you can try small->smaller as big->?.  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ggnL__QmtURe",
        "colab_type": "text"
      },
      "source": [
        "# Debiasing word vectors"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WG1SQkcztURe",
        "colab_type": "text"
      },
      "source": [
        "In the following exercise, you will examine gender biases that can be reflected in a word embedding, and explore algorithms for reducing the bias. In addition to learning about the topic of debiasing, this exercise will also help hone your intuition about what word vectors are doing. This section involves a bit of linear algebra, though you can probably complete it even without being expert in linear algebra, and we encourage you to give it a shot. \n",
        "\n",
        "Lets first see how the GloVe word embeddings relate to gender. You will first compute a vector $g = e_{woman}-e_{man}$, where $e_{woman}$ represents the word vector corresponding to the word *woman*, and $e_{man}$ corresponds to the word vector corresponding to the word *man*. The resulting vector $g$ roughly encodes the concept of \"gender\". (You might get a more accurate representation if you compute $g_1 = e_{mother}-e_{father}$, $g_2 = e_{girl}-e_{boy}$, etc. and average over them. But just using $e_{woman}-e_{man}$ will give good enough results for now.) \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HrkAIYhGtURf",
        "colab_type": "text"
      },
      "source": [
        "**Task 5:** Compute the bias vector using woman - man\n",
        "\n",
        "***"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DcNi6vOQtURf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "## START YOU CODE HERE\n",
        "\n",
        "g =  word_to_vec_map['woman'] - word_to_vec_map['man']\n",
        "\n",
        "## END\n",
        "\n",
        "print(g)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EhA93B5RtURi",
        "colab_type": "text"
      },
      "source": [
        "***"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E5ALg3EntURj",
        "colab_type": "text"
      },
      "source": [
        "Now, you will consider the cosine similarity of different words with $g$. Consider what a positive value of similarity means vs a negative cosine similarity. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "scrolled": false,
        "id": "UeWyrf5utURj",
        "colab_type": "text"
      },
      "source": [
        "**Task 6: Compute and print the similarity between g and the words in name_list**\n",
        "\n",
        "***\n",
        "    "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ME6LogmNtURj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print ('List of names and their similarities with constructed vector:')\n",
        "name_list = ['john', 'marie', 'sophie', 'ronaldo', 'priya', 'rahul', 'danielle', 'reza', 'katy', 'yasmin']\n",
        "\n",
        "## START YOU CODE HERE\n",
        "# Loop over the name_list and print Result\n",
        "for w in name_list:\n",
        "    print (w, cosine_similarity(word_to_vec_map[w], g))\n",
        "## END "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IRtdJpmltURm",
        "colab_type": "text"
      },
      "source": [
        "***"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R76Oewo6tURm",
        "colab_type": "text"
      },
      "source": [
        "**TASK 7: What do you abserve?**\n",
        "\n",
        "***\n",
        "\n",
        "PLEASE ANSWER HERE!\n",
        "female first names tend to have a positive cosine similarity with our constructed vector  g , while male first names tend to have a negative cosine similarity. This is not suprising, and the result seems acceptable\n",
        "\n",
        "***"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1VMJ4UY0tURn",
        "colab_type": "text"
      },
      "source": [
        "**Task 8: Compute and print the similarity between g and the words in word_list:**\n",
        "\n",
        "***"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YdlQi4kLtURn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print('Other words and their similarities:')\n",
        "word_list = ['lipstick', 'guns', 'science', 'arts', 'literature', 'warrior','doctor', 'tree', 'receptionist', \n",
        "             'technology',  'fashion', 'teacher', 'engineer', 'pilot', 'computer', 'singer']\n",
        "\n",
        "## START YOU CODE HERE\n",
        "# Loop over the word_list and print Result\n",
        "for w in word_list:\n",
        "    print (w, cosine_similarity(word_to_vec_map[w], g))\n",
        "\n",
        "## END \n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kmtRjfH0tURq",
        "colab_type": "text"
      },
      "source": [
        "***"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wJRr5IuOtURq",
        "colab_type": "text"
      },
      "source": [
        "**TASK 9: What do you abserve?**\n",
        "\n",
        "***\n",
        "\n",
        "PLEASE ANSWER HERE!\n",
        "\n",
        "\n",
        "***\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g00Aw3fQtURr",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "We'll see below how to reduce the bias of these vectors, using an algorithm due to [Boliukbasi et al., 2016](https://arxiv.org/abs/1607.06520). Note that some word pairs such as \"actor\"/\"actress\" or \"grandmother\"/\"grandfather\" should remain gender specific, while other words such as \"receptionist\" or \"technology\" should be neutralized, i.e. not be gender-related. You will have to treat these two type of words differently when debiasing.\n",
        "\n",
        "# Neutralize bias for non-gender specific words \n",
        "\n",
        "The figure below should help you visualize what neutralizing does. If you're using a 50-dimensional word embedding, the 50 dimensional space can be split into two parts: The bias-direction $g$, and the remaining 49 dimensions, which we'll call $g_{\\perp}$. In linear algebra, we say that the 49 dimensional $g_{\\perp}$ is perpendicular (or \"orthogonal\") to $g$, meaning it is at 90 degrees to $g$. The neutralization step takes a vector such as $e_{receptionist}$ and zeros out the component in the direction of $g$, giving us $e_{receptionist}^{debiased}$. \n",
        "\n",
        "Even though $g_{\\perp}$ is 49 dimensional, given the limitations of what we can draw on a screen, we illustrate it using a 1 dimensional axis below. \n",
        "\n",
        "\n",
        "![](images/neutral.png)\n",
        "\n",
        "\n",
        "<caption><center> <b>Figure 2</b>: The word vector for \"receptionist\" represented before and after applying the neutralize operation. </center></caption>\n",
        "\n",
        "**TASK 10**: Implement `neutralize()` to remove the bias of words such as \"receptionist\" or \"scientist\". Given an input embedding $e$, you can use the following formulas to compute $e^{debiased}$: \n",
        "\n",
        "\n",
        "\\begin{equation}\n",
        "e^{bias\\_component} = \\frac{e \\cdot g}{||g||_2^2} * g\\tag{2}\n",
        "\\end{equation}\n",
        "\n",
        "\\begin{equation}\n",
        "e^{debiased} = e - e^{bias\\_component}\\tag{3}\n",
        "\\end{equation}\n",
        "\n",
        "If you are an expert in linear algebra, you may recognize $e^{bias\\_component}$ as the projection of $e$ onto the direction $g$. If you're not an expert in linear algebra, don't worry about this.\n",
        "\n",
        "<!-- \n",
        "**Reminder**: a vector $u$ can be split into two parts: its projection over a vector-axis $v_B$ and its projection over the axis orthogonal to $v$:\n",
        "$$u = u_B + u_{\\perp}$$\n",
        "where : $u_B = $ and $ u_{\\perp} = u - u_B $\n",
        "!--> \n",
        "***"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "37apkZpstURr",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# TASK 10\n",
        "# GRADED neutralize\n",
        "\n",
        "def neutralize(word, g, word_to_vec_map):\n",
        "    \"\"\"\n",
        "    Removes the bias of \"word\" by projecting it on the space orthogonal to the bias axis. \n",
        "    This function ensures that gender neutral words are zero in the gender subspace.\n",
        "    \n",
        "    Arguments:\n",
        "        word -- string indicating the word to debias\n",
        "        g -- numpy-array of shape (50,), corresponding to the bias axis (such as gender)\n",
        "        word_to_vec_map -- dictionary mapping words to their corresponding vectors.\n",
        "    \n",
        "    Returns:\n",
        "        e_debiased -- neutralized word vector representation of the input \"word\"\n",
        "    \"\"\"\n",
        "    ## START YOU CODE HERE\n",
        "    e = word_to_vec_map[word]\n",
        "    e_biasXomponent = np.dot(e,g)*g/np.linalg.norm(g)**2 # attention to square\n",
        "    e_debiased = e - e_biasComponent\n",
        "    return e_debiased\n",
        "    ## END \n",
        "    "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Mf2RuFDbtURx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "e = \"receptionist\"\n",
        "print(\"cosine similarity between \" + e + \" and g, before neutralizing: \", cosine_similarity(word_to_vec_map[\"receptionist\"], g))\n",
        "\n",
        "e_debiased = neutralize(\"receptionist\", g, word_to_vec_map)\n",
        "print(\"cosine similarity between \" + e + \" and g, after neutralizing: \", cosine_similarity(e_debiased, g))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eKxeZOg5tURz",
        "colab_type": "text"
      },
      "source": [
        "# Equalization algorithm for gender-specific words\n",
        "\n",
        "Next, lets see how debiasing can also be applied to word pairs such as \"actress\" and \"actor.\" Equalization is applied to pairs of words that you might want to have differ only through the gender property. As a concrete example, suppose that \"actress\" is closer to \"babysit\" than \"actor.\" By applying neutralizing to \"babysit\" we can reduce the gender-stereotype associated with babysitting. But this still does not guarantee that \"actor\" and \"actress\" are equidistant from \"babysit.\" The equalization algorithm takes care of this. \n",
        "\n",
        "The key idea behind equalization is to make sure that a particular pair of words are equi-distant from the 49-dimensional $g_\\perp$. The equalization step also ensures that the two equalized steps are now the same distance from $e_{receptionist}^{debiased}$, or from any other word that has been neutralized. In pictures, this is how equalization works: \n",
        "\n",
        "\n",
        "![](images/equalize10.png)\n",
        "\n",
        "\n",
        "\n",
        "The derivation of the linear algebra to do this is a bit more complex. (See Bolukbasi et al., 2016 for details.) But the key equations are: \n",
        "\n",
        "\\begin{equation}\n",
        "\\mu = \\frac{e_{w1} + e_{w2}}{2}\\tag{4}\n",
        "\\end{equation}\n",
        "\n",
        "\\begin{equation}\n",
        "\\mu_{B} = \\frac {\\mu \\cdot bias\\_axis}{||bias\\_axis||_2^2} *bias\\_axis\n",
        "\\tag{5}\n",
        "\\end{equation}\n",
        "\n",
        "\\begin{equation}\n",
        "\\mu_{\\perp} = \\mu - \\mu_{B} \\tag{6}\n",
        "\\end{equation}\n",
        "\n",
        "\\begin{equation}\n",
        "e_{w1B} = \\frac {e_{w1} \\cdot bias\\_axis}{||bias\\_axis||_2^2} *bias\\_axis\n",
        "\\tag{7}\n",
        "\\end{equation}\n",
        "\n",
        "\\begin{equation}\n",
        "e_{w2B} = \\frac {e_{w2} \\cdot bias\\_axis}{||bias\\_axis||_2^2} *bias\\_axis\n",
        "\\tag{8}\n",
        "\\end{equation}\n",
        "\n",
        "\\begin{equation}\n",
        "e_{w1B}^{corrected} = \\sqrt{ |{1 - ||\\mu_{\\perp} ||^2_2} |} * \\frac{e_{\\text{w1B}} - \\mu_B} {||(e_{w1} - \\mu_{\\perp}) - \\mu_B||} \\tag{9}\n",
        "\\end{equation}\n",
        "\n",
        "\\begin{equation}\n",
        "e_{w2B}^{corrected} = \\sqrt{ |{1 - ||\\mu_{\\perp} ||^2_2} |} * \\frac{e_{\\text{w2B}} - \\mu_B} {||(e_{w2} - \\mu_{\\perp}) - \\mu_B||} \\tag{10}\n",
        "\\end{equation}\n",
        "\n",
        "\\begin{equation}\n",
        "e_1 = e_{w1B}^{corrected} + \\mu_{\\perp} \\tag{11}\n",
        "\\end{equation}\n",
        "\n",
        "\\begin{equation}\n",
        "e_2 = e_{w2B}^{corrected} + \\mu_{\\perp} \\tag{12}\n",
        "\\end{equation}\n",
        "\n",
        "**TASK 11**: Implement the function below. Use the equations above to get the final equalized version of the pair of words. Good luck!\n",
        "\n",
        "***"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0USdKmcBtUR0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# TASK 11\n",
        "# GRADED equalize\n",
        "\n",
        "def equalize(pair, bias_axis, word_to_vec_map):\n",
        "    \"\"\"\n",
        "    Debias gender specific words by following the equalize method described in the figure above.\n",
        "    \n",
        "    Arguments:\n",
        "    pair -- pair of strings of gender specific words to debias, e.g. (\"actress\", \"actor\") \n",
        "    bias_axis -- numpy-array of shape (50,), vector corresponding to the bias axis, e.g. gender\n",
        "    word_to_vec_map -- dictionary mapping words to their corresponding vectors\n",
        "    1. Dividing the set w1 and w2\n",
        "    2. Compute the mean \n",
        "    3. Calculate the projection of mu over the bias\n",
        "    4. Now we use equation for both e_w1B and e_w2B\n",
        "    5. Adjust the correction using formula \n",
        "    6. Now Sum with mu_orthgonal of step 3\n",
        "    Returns\n",
        "    e_1 -- word vector corresponding to the first word\n",
        "    e_2 -- word vector corresponding to the second word\n",
        "    \"\"\"\n",
        "    ## START YOU CODE HERE\n",
        "#     Step 1:\n",
        "    w1, w2 = pair[0],pair[1]\n",
        "#     Step 2:\n",
        "    e_w1, e_w2 = word_to_vec_map[w1],word_to_vec_map[w2]\n",
        "    mu = (e_w1+e_w2)/2\n",
        "#     Step 3:\n",
        "    mu_B = np.dot(mu,bias_axis)*bias_axis/np.linalg.norm(bias_axis)**2\n",
        "    mu_orth = mu - mu_B\n",
        "#     Step 4 :\n",
        "    e_w1B = np.dot(e_w1,bias_axis)*bias_axis/np.linalg.norm(bias_axis)**2\n",
        "    e_w2B = np.dot(e_w2,bias_axis)*bias_axis/np.linalg.norm(bias_axis)**2\n",
        "#     Step 5:\n",
        "    corrected_e_w1B = np.sqrt(abs(1-np.linalg.norm(mu_orth)**2))*(e_w1B - mu_B)/np.sum(e_w1 - mu_orth - mu_B)\n",
        "    corrected_e_w2B = np.sqrt(abs(1-np.linalg.norm(mu_orth)**2))*(e_w2B - mu_B)/np.sum(e_w2 - mu_orth - mu_B)\n",
        "#     Step 6:\n",
        "    e1 = corrected_e_w1B + mu_orth\n",
        "    e2 = corrected_e_w2B + mu_orth\n",
        "    return e1, e2\n",
        "    ##END"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "893xuAh_tUR2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(\"cosine similarities before equalizing:\")\n",
        "print(\"cosine_similarity(word_to_vec_map[\\\"man\\\"], gender) = \", cosine_similarity(word_to_vec_map[\"man\"], g))\n",
        "print(\"cosine_similarity(word_to_vec_map[\\\"woman\\\"], gender) = \", cosine_similarity(word_to_vec_map[\"woman\"], g))\n",
        "print()\n",
        "e1, e2 = equalize((\"man\", \"woman\"), g, word_to_vec_map)\n",
        "print(\"cosine similarities after equalizing:\")\n",
        "print(\"cosine_similarity(e1, gender) = \", cosine_similarity(e1, g))\n",
        "print(\"cosine_similarity(e2, gender) = \", cosine_similarity(e2, g))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A9IIW89JtUR4",
        "colab_type": "text"
      },
      "source": [
        "**TASK 12: What do you abserve?**\n",
        "\n",
        "***\n",
        "\n",
        "# i was having issue with the code while reading file but with the help of online materials i write the solution. I used the folmulas directly and figure out the answers which may be almost similar to real answer\n",
        "According to my observation the out shows the bias result as in lecture you mention about the factor on which are based system shows bias behaviour one of the factor is gender on which we are gonna get the output. They prefer male with some negative value as compare to the female shows bias behavour before equalizing after equalizing it almost get the same values they remove the bias in the dataset not completely but approx in some case it removed completely depending on the dataset which we are using \n",
        "\n",
        "\n",
        "***"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": true,
        "id": "dZUWMJvStUR5",
        "colab_type": "text"
      },
      "source": [
        "Please feel free to play with the input words in the cell above, to apply equalization to other pairs of words. \n",
        "\n",
        "These debiasing algorithms are very helpful for reducing bias, but are not perfect and do not eliminate all traces of bias. For example, one weakness of this implementation was that the bias direction $g$ was defined using only the pair of words _woman_ and _man_. As discussed earlier, if $g$ were defined by computing $g_1 = e_{woman} - e_{man}$; $g_2 = e_{mother} - e_{father}$; $g_3 = e_{girl} - e_{boy}$; and so on and averaging over them, you would obtain a better estimate of the \"gender\" dimension in the 50 dimensional word embedding space. Feel free to play with such variants as well.  \n",
        "                     "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a8QB9w4ktUR5",
        "colab_type": "text"
      },
      "source": [
        "# Congratulations!\n",
        "\n",
        "You've come to the end of this assignment, and have seen a lot of the ways that word vectors can be used as well as  modified. Here are the main points you should remember:\n",
        "\n",
        "- Cosine similarity a good way to compare similarity between pairs of word vectors. (Though L2 distance works too.) \n",
        "- For NLP applications, using a pre-trained set of word vectors from the internet is often a good way to get started. \n",
        "- Bias in data is an important problem.\n",
        "- Neutralize and equalize allow to reduce bias in the data.\n",
        "\n",
        "Congratulations on finishing this notebook! \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wYilhddCtUR6",
        "colab_type": "text"
      },
      "source": [
        "# References\n",
        "- The debiasing algorithm is from Bolukbasi et al., 2016, [Man is to Computer Programmer as Woman is to\n",
        "Homemaker? Debiasing Word Embeddings](https://papers.nips.cc/paper/6228-man-is-to-computer-programmer-as-woman-is-to-homemaker-debiasing-word-embeddings.pdf)\n",
        "- The GloVe word embeddings were due to Jeffrey Pennington, Richard Socher, and Christopher D. Manning. (https://nlp.stanford.edu/projects/glove/)\n"
      ]
    }
  ]
}