3-ANN-User-Representations-UPDATED.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tHbGH4ZVbdMN"
   },
   "source": [
    "<h4><i><font color='red'>The project has many components, and we encourage you to complete as many as you can! That said, we highly encourage you to submit your work even if your notebook is only partially completed - the TA can help review your work and provide tips on any places that you got stuck or have further questions!</font></i></h4>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qfTG6NXWGjQt"
   },
   "source": [
    "<font size=\"6\">**Table of Contents:**</font>  \n",
    "<br>\n",
    "<font size=\"5\">\n",
    "<u>Approximate Nearest Neighbor Search:</u>  \n",
    "&emsp;&emsp;1\\. [Introduction](#scrollTo=9wDLwRU6HUNG&uniqifier=1)  \n",
    "&emsp;&emsp;2\\. [Setup](#scrollTo=vq47Wt-0Jan-&uniqifier=1)  \n",
    "&emsp;&emsp;3\\. [TODO 3.1: Your KNN](#scrollTo=3u93B8JdJ9pg&uniqifier=1)  \n",
    "&emsp;&emsp;4\\. [Faiss](#scrollTo=TRAg9cqeMiKl&uniqifier=1)  \n",
    "&emsp;&emsp;5\\. [TODO 3.2: KNN Comparison](#scrollTo=-2Xh5R8xT3to&uniqifier=1)  \n",
    "<br>\n",
    "<u>User Representations:</u>  \n",
    "&emsp;&emsp;1\\. [Introduction](#scrollTo=Yk2dt4l1VS_A&uniqifier=1)  \n",
    "&emsp;&emsp;2\\. [Setup](#scrollTo=oU7Z49ymW5yO&uniqifier=1)    \n",
    "&emsp;&emsp;3\\. [Initial User and Item Representations](#scrollTo=Frr6CK1NX6dJ&uniqifier=1)    \n",
    "&emsp;&emsp;4\\. [LightGBM as a Ranker](#scrollTo=_FYTNTwUc3ot&uniqifier=1)    \n",
    "&emsp;&emsp;5\\. [TODO 3.3: Your User Representations](#scrollTo=L-ErqwvYd9F7&uniqifier=1)    \n",
    "&emsp;&emsp;6\\. [TODO 3.4: LSTM User Representations (Optional)](#scrollTo=-BxR9VPpjoNi&uniqifier=1)    \n",
    "</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cB4tqBSaHIqt"
   },
   "source": [
    "# <u>Approximate Nearest Neighbor Search:</u>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9wDLwRU6HUNG"
   },
   "source": [
    "## **1. Introduction**\n",
    "[back to top](#scrollTo=qfTG6NXWGjQt&uniqifier=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "41f1f7cb-9441-4fe1-962e-9e4cb3e3c192"
   },
   "source": [
    "In this week's project, we will touch upon two key aspects related to representations:\n",
    "1. How do we query large amount of vectors in efficient time?\n",
    "2. How can we infer various user representations and see what their impact is on the downstream task?\n",
    "\n",
    "Let's begin with the former, which tells us how we could handle a large number of candidate items or user representations in an efficient manner: Approximate Nearest Neighbor (ANN) Search.  \n",
    "  \n",
    "<br>Often we are interested in finding nearest neighbors in a large space of vectors. To store embeddings for 400 million users and over 100 million items and querying them in real time is a challenging task. This is where approximate nearest neighbor approaches step in to help. [Annoy](https://github.com/spotify/annoy), [Faiss](https://github.com/facebookresearch/faiss), and [ScaNN](https://github.com/google-research/google-research/tree/master/scann) are typical libraries that are used for efficient vector similarity search at scale. They implement algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM.\n",
    "\n",
    "In the first part of this week's project, we will simulate embeddings of 500k items and try to find the k-nearest neighbours for an item of interest. We will implement a vanilla search function to fetch the top-k nearest neighbors and estimate the time it takes for us to do so. We will then compare this with Faiss -- Facebook's nearest neighbour search library, and compare the time it takes for us to get nearest neighbours from Faiss versus our own implementation.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vq47Wt-0Jan-"
   },
   "source": [
    "## **2. Setup**\n",
    "[back to top](#scrollTo=qfTG6NXWGjQt&uniqifier=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "837dcaa9-cc0b-4345-bbee-3a09683decf9"
   },
   "outputs": [],
   "source": [
    "# Installs and imports\n",
    "!pip install faiss-cpu --no-cache --quiet\n",
    "\n",
    "import pickle\n",
    "import faiss\n",
    "import numpy as np\n",
    "from IPython.display import clear_output\n",
    "\n",
    "clear_output()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3u93B8JdJ9pg"
   },
   "source": [
    "## **3. TODO 3.1: Your KNN**\n",
    "[back to top](#scrollTo=qfTG6NXWGjQt&uniqifier=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3fa2b438-be53-4f39-b990-859046a4a560"
   },
   "source": [
    "Let's first generate a simulated dataset of embeddings of 500k items each with 64 elements:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "8c36fc8b-a032-4b48-bbeb-3bff46624615",
    "outputId": "33f21652-b9a7-4585-85dd-9220ba1e9d6c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(500000, 64)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Set seed for reproducibility\n",
    "np.random.seed(42)\n",
    "\n",
    "# Generate 500k item vectors with 64 elements\n",
    "item_vectors = np.random.random((500_000, 64)).astype('float32')\n",
    "\n",
    "item_vectors.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "142c00a5-f36b-4a5f-9e27-4a3e38b467d7"
   },
   "source": [
    "Now that we have these items, let's take up the goal of finding the top-k items closest to this specific item. Your goal is to implement your function to estimate the top-k items using [Euclidean distance](https://en.wikipedia.org/wiki/Euclidean_distance) to the query item. \n",
    "  \n",
    "Euclidean distance can be [computed*](https://www.google.com/search?q=euclidean+distance&rlz=1C1CHBD_enUS756US756&oq=euclidean+distance&aqs=chrome.0.0i131i433i512l2j0i433i512j0i131i433i512l4j0i512j0i131i433i512l2.2329j1j7&sourceid=chrome&ie=UTF-8#wptab=si:AC1wQDCDyRf_4m8Q8nhldHMjdIalRktweLkw15mQu4pgD74-zUn1N-3MIuKczXZHIoqR9oMGcw61xadKjn9m6kYiyGRz32fPfr5CXMeSXfIndlhRj0dSYnewx44jaNCJZqrvNJZon_B3) as follows:\n",
    "\\begin{align}\n",
    "\\ d(p,q) = \\sqrt{\\sum_{i=1}^n (p_i-q_i)^2}\n",
    "\\end{align}\n",
    "\n",
    "p, q = two points in Euclidean n-space  \n",
    "q<sub>i</sub>, p<sub>i</sub> = Euclidean vectors, starting from the origin of the space (initial point)  \n",
    "n = n-space  \n",
    "\\* Note that Faiss does not take the final square root.\n",
    "  \n",
    "You'll be able to check your work with `index1`, the Faiss Flat Index, below. It should return the same results as your function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "84a94ee2-3131-44c5-ba95-ec67aa675048"
   },
   "outputs": [],
   "source": [
    "def find_top_k_nn(query_vector, k=5):\n",
    "    \"\"\"\n",
    "    Implement top-k nearest neighbours using Euclidean distance, and return the distances and indices of the these top-k items.\n",
    "    \n",
    "    Args:\n",
    "        query_vector (np.array): Vector representation of the item to retrieve neighbors for.\n",
    "        k (int): Number of neighbors to retrieve.\n",
    "        \n",
    "    Returns:\n",
    "        distances (list): Distances to top-k nearest neighbors.\n",
    "        indices (list): Indices of top-k nearest neighbors.\n",
    "    \"\"\"\n",
    "    distances, indices = [], []\n",
    "    \n",
    "    # Your code goes here\n",
    "        # Note: You'll need to reference the `item_vectors` object to calculate distances\n",
    "    \n",
    "    \n",
    "    return distances, indices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "b8a895ce-4f0b-41a5-8243-90f18bff8d7b"
   },
   "source": [
    "With your top-k NN function implemented, call this function to get the top-k nearest neighbor items for the query_vector and print relevant statistics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7686a243-327c-4334-9670-4229439bf77e"
   },
   "outputs": [],
   "source": [
    "# Helper function to run our KNN functions\n",
    "\n",
    "def run_knn(knn_function, query_vector, k=5):\n",
    "    \"\"\"\n",
    "    Helper function to run our KNN functions.\n",
    "    \n",
    "    Args:\n",
    "        knn_function (function): Object that returns distances and indices given a query vector.\n",
    "        query_vector (np.array): Vector representation of the item to retrieve neighbors for.\n",
    "        k (int): Number of neighbors to retrieve.\n",
    "    \"\"\"\n",
    "    distances, indices = knn_function(query_vector, k=k)\n",
    "    print(f\"Distances from the k nearest neighbors: {distances}\")\n",
    "    print(f\"Indices from the k nearest neighbors: {indices}\")\n",
    "    print(f\"Average distance of the k nearest neighbors: {np.mean(distances)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "f0b6fca6-4d83-4c2b-ab8e-dba4d112a44a"
   },
   "outputs": [],
   "source": [
    "# We've chosen the query vector for convenience\n",
    "query_vector = item_vectors[0].reshape(1,-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "abe9523d-58d6-478e-ba0c-4bb7907a7e55"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "run_knn(find_top_k_nn, query_vector, k=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TRAg9cqeMiKl"
   },
   "source": [
    "## **4. Faiss**\n",
    "[back to top](#scrollTo=qfTG6NXWGjQt&uniqifier=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bd4452f5-ce17-43df-95f2-e9b58e8cad5e"
   },
   "source": [
    "Now we'll switch to using [Faiss](https://github.com/facebookresearch/faiss) for our KNN.  \n",
    "  \n",
    "<br>Given a set of vectors x<sub>i</sub> in dimension d, Faiss builds a data structure in RAM. After the structure is constructed, when given a new vector x in dimension d, it efficiently performs:\n",
    "\n",
    "i = argmin<sub>i</sub> ||x - x<sub>i</sub>||\n",
    "\n",
    "where ||.|| is the Euclidean distance (L2).\n",
    "\n",
    "If Faiss terms, the data structure is an index, an object that has an add method to add x<sub>i</sub> vectors. Note that the x<sub>i</sub>'s are assumed to be fixed. Computing the argmin is the search operation on the index.  \n",
    "  \n",
    "<br>\n",
    "\n",
    "### Indexes used by Faiss\n",
    "\n",
    "1. The inverted file from [Video google: A text retrieval approach to object matching in videos](https://www.robots.ox.ac.uk/~vgg/publications/papers/sivic03.pdf). This is the key to non-exhaustive search in large datasets. Otherwise all searches would need to scan all elements in the index, which is prohibitive even with a fast operation.\n",
    "\n",
    "\n",
    "2. The product quantization (PQ) method from [Product quantization for nearest neighbor search](https://lear.inrialpes.fr/pubs/2011/JDS11/jegou_searching_with_quantization.pdf). This can be seen as a lossy compression technique for high-dimensional vectors that allows relatively accurate reconstructions and distance computations in the compressed domain.\n",
    "\n",
    "\n",
    "3. The three-level quantization (IVFADC-R aka IndexIVFPQR) method from [Searching in one billion vectors: re-rank with source coding](https://arxiv.org/pdf/1102.3828.pdf).\n",
    "\n",
    "\n",
    "We will use these three indexes from Faiss to search our vectors to get the top-k nearest neighbours and estimate the average distance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "76487ec4-a020-4ccd-aa33-f6ef25869048",
    "outputId": "1b9f0c1e-43c3-429d-e3f0-f848c8e7f7f6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of vectors indexed: 500,000\n",
      "CPU times: user 47.1 ms, sys: 0 ns, total: 47.1 ms\n",
      "Wall time: 46.3 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# Build flat index\n",
    "# Docs: https://faiss.ai/cpp_api/struct/structfaiss_1_1IndexFlatL2.html\n",
    "index1 = faiss.IndexFlatL2(item_vectors.shape[1])\n",
    "\n",
    "# Add vectors\n",
    "index1.add(item_vectors)\n",
    "\n",
    "print(f\"Total number of vectors indexed: {index1.ntotal:,}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "0930bea9-bef2-4876-8a0d-117ac2c7d577",
    "outputId": "e937073c-637c-46a5-b35b-1b8568c301ae"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of vectors indexed: 500,000 \n",
      "CPU times: user 1.05 s, sys: 69.2 ms, total: 1.12 s\n",
      "Wall time: 599 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# Build quantizer\n",
    "quantizer = faiss.IndexFlatL2(item_vectors.shape[1])\n",
    "\n",
    "# Build inverted index\n",
    "# Docs: https://faiss.ai/cpp_api/struct/structfaiss_1_1IndexIVFFlat.html\n",
    "nlist = 100 # number of possible key values\n",
    "index2 = faiss.IndexIVFFlat(quantizer, item_vectors.shape[1], nlist, faiss.METRIC_L2)\n",
    "\n",
    "# Train index and add vectors\n",
    "index2.train(item_vectors)\n",
    "index2.add(item_vectors)\n",
    "\n",
    "print(f\"Total number of vectors indexed: {index2.ntotal:,} \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "c081de1d-dad0-4277-a739-42a3b924b138",
    "outputId": "702e42c1-7765-4c2d-b822-81fe1176f826"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total number of vectors indexed: 500,000 \n",
      "CPU times: user 26.8 s, sys: 53.6 ms, total: 26.9 s\n",
      "Wall time: 14 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "# Build quantizer\n",
    "quantizer = faiss.IndexFlatL2(item_vectors.shape[1])\n",
    "\n",
    "# Build product quantized index\n",
    "# Docs: https://faiss.ai/cpp_api/struct/structfaiss_1_1IndexIVFPQ.html\n",
    "nlist = 100\n",
    "m = 8 # max number of codes to visit to do a query\n",
    "num_bits = 8 # number of bits per index\n",
    "index3 = faiss.IndexIVFPQ(quantizer, item_vectors.shape[1], \n",
    "                          nlist, m, num_bits)\n",
    "\n",
    "# Train index and add vectors\n",
    "index3.train(item_vectors)\n",
    "index3.add(item_vectors)\n",
    "\n",
    "print(f\"Total number of vectors indexed: {index3.ntotal:,} \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "047a6682-f771-4672-8998-307f4cfad5f5"
   },
   "source": [
    "Now that we have these three indexes, let us query these to fetch the top-k nearest neghbours for our query_vector and compute the average distance we obtain for each.\n",
    "\n",
    "Now we'll use and time these indexes to find the top-k nearest neighbours to highlight the trade-off between accuracy and latency:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "261db067-49b1-466a-9dfd-cfd544099fb0",
    "outputId": "48ea1ef3-85ca-417b-bfd6-8a622d015934"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Distances from the k nearest neighbors: [[0.       4.873273 4.965366 5.01122  5.055152]]\n",
      "Indices from the k nearest neighbors: [[     0 260846 120114 257179 119384]]\n",
      "Average distance of the k nearest neighbors: 3.98100209236145\n",
      "CPU times: user 34.4 ms, sys: 0 ns, total: 34.4 ms\n",
      "Wall time: 35.7 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "run_knn(index1.search, query_vector, k=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "4603327e-195a-4f9b-b5c5-8930cbdadaba",
    "outputId": "e1e3d840-d219-43aa-df7d-63485bfcf839"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Distances from the k nearest neighbors: [[0.        5.055152  5.3904734 5.5314574 5.6039724]]\n",
      "Indices from the k nearest neighbors: [[     0 119384  38201 409057  27690]]\n",
      "Average distance of the k nearest neighbors: 4.316210746765137\n",
      "CPU times: user 1.37 ms, sys: 6 µs, total: 1.38 ms\n",
      "Wall time: 919 µs\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "run_knn(index2.search, query_vector, k=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ae0a8555-a57d-4517-af4a-fe079e93352d",
    "outputId": "0eac3e99-1c0e-4423-e564-234de4b8a78c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Distances from the k nearest neighbors: [[1.6294069 4.873525  5.3482428 5.3632226 5.3641176]]\n",
      "Indices from the k nearest neighbors: [[     0 248319  38201 409057  72742]]\n",
      "Average distance of the k nearest neighbors: 4.515702724456787\n",
      "CPU times: user 1.24 ms, sys: 4 µs, total: 1.24 ms\n",
      "Wall time: 1.06 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "run_knn(index3.search, query_vector, k=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "c2c902e3-c9bf-408e-9052-11af5a3b069f"
   },
   "source": [
    "As expected, we observe that the product quantization and inverted based indexes are an order of magnitude faster than the flat index while the average distance is larger. In terms of accuracy, if we assume that the lower the distance the more accurate the result, the Flat Index gives us the most accurate result but is relatively slow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-2Xh5R8xT3to"
   },
   "source": [
    "## **5. TODO 3.2: KNN Comparison**\n",
    "[back to top](#scrollTo=qfTG6NXWGjQt&uniqifier=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "044a818f-19ab-4c26-8532-922c9aa1d673"
   },
   "source": [
    "Your second goal in this section is to compare your KNN to the 3 Faiss indexes we created above: Compare the average distance obtained and the time it took to query the 500k vectors. A nice 2D plot would also give you a good idea of the latency-accuracy trade-off involved."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2e69dd09-6a21-49a8-8ca8-4d5164a7e674"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "e127987e-44f9-43b3-9219-1036edd0d14c"
   },
   "source": [
    "# <u>User Representations:</u>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Yk2dt4l1VS_A"
   },
   "source": [
    "## **1. Introduction**\n",
    "[back to top](#scrollTo=qfTG6NXWGjQt&uniqifier=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "13c18e33-3a62-4afe-95af-9d33ef87917c"
   },
   "source": [
    "In the second part of this week's project, we'll establish a few ways of estimating user representations and understand how each impacts the performance of a ranker.  \n",
    "  \n",
    "<br>More broadly, we will:\n",
    "* Fix the article representations\n",
    "* Fix the downstream task (ranking)\n",
    "* Vary the user representations \n",
    "* Measure changes in performance in terms of MRR\n",
    "  \n",
    "<br>**This section benefits from using a GPU!**  \n",
    "You can use it for free, in Google Colab, by going to the menu bar above, selecting `Runtime`, then `Change runtime type`, then selecting `GPU` as the `Hardware accelerator`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oU7Z49ymW5yO"
   },
   "source": [
    "## **2. Setup**\n",
    "[back to top](#scrollTo=qfTG6NXWGjQt&uniqifier=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "251abdd1-1cd4-41f9-af70-23d22a1aa455"
   },
   "outputs": [],
   "source": [
    "# Installs and imports\n",
    "\n",
    "# Install LightGBM with GPU support\n",
    "!pip uninstall lightgbm --y\n",
    "!pip install lightgbm --install-option=--gpu\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import random\n",
    "import gc\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.decomposition import TruncatedSVD\n",
    "import lightgbm as lgb\n",
    "from datetime import timedelta\n",
    "from tqdm import tqdm\n",
    "import h5py\n",
    "import os\n",
    "import shutil\n",
    "from collections import defaultdict\n",
    "from IPython.display import clear_output\n",
    "\n",
    "clear_output()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2qqBlHJTXqgL"
   },
   "outputs": [],
   "source": [
    "# Download data\n",
    "!mkdir ./hmdata\n",
    "!gdown --folder https://drive.google.com/drive/folders/1j9QpkSKwqFfpgohbVxdPDKuwmwJXZ6yz?usp=sharing -O ./hmdata\n",
    "\n",
    "clear_output()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5ab51d11-a651-45c5-ac15-f7dc6d9993d8"
   },
   "source": [
    "Assemble train and test sets as we've done in previous notebooks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "dd30ce60-cc85-47cc-b9cf-4afc85d93d80"
   },
   "outputs": [],
   "source": [
    "# Read in data into memory with some explicit datatypes so they're easier to work with\n",
    "\n",
    "tran_dtypes = {\"t_dat\":\"str\",\n",
    "               \"customer_id\":\"str\",\n",
    "               \"article_id\":\"int\",\n",
    "               \"product_code\":\"int\",\n",
    "               \"price\":\"float\",\n",
    "               \"sales_channel_id\":\"int\"}\n",
    "art_dtypes = {\"article_id\":\"int\",\n",
    "              \"product_code\":\"int\",\n",
    "              \"product_type_no\":\"int\",\n",
    "              \"graphical_appearance_no\":\"int\",\n",
    "              \"colour_group_code\":\"int\",\n",
    "              \"department_no\":\"int\",\n",
    "              \"index_code\":\"str\",\n",
    "              \"index_group_no\":\"int\",\n",
    "              \"section_no\":\"int\",\n",
    "              \"garment_group_no\":\"int\"}\n",
    "cust_dtypes = {\"customer_id\":\"str\"}\n",
    "\n",
    "article_df = pd.read_csv(\"hmdata/articles.csv.zip\", dtype=art_dtypes)\n",
    "customer_df = pd.read_csv(\"hmdata/customers.csv.zip\", dtype=cust_dtypes)\n",
    "transaction_df = pd.read_csv('hmdata/transactions_train.csv.zip', dtype=tran_dtypes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 9167,
     "status": "ok",
     "timestamp": 1666497896225,
     "user": {
      "displayName": "hm kaggle",
      "userId": "10519941953201100561"
     },
     "user_tz": 420
    },
    "id": "b4fccef4-15d3-4f2e-865c-d6923baceb3b",
    "outputId": "24fb3587-ebc8-4120-b20f-6868cea52ad4"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Create train and test sets\n",
    "\n",
    "N_DAYS_TRAIN = 45\n",
    "N_DAYS_TEST = 7\n",
    "\n",
    "max_date = transaction_df['t_dat'].max()\n",
    "train = transaction_df[(transaction_df['t_dat']>=((pd.to_datetime(max_date) - timedelta(days=N_DAYS_TRAIN+N_DAYS_TEST)).date().strftime('%Y-%m-%d')))\n",
    "                       & (transaction_df['t_dat']<((pd.to_datetime(max_date) - timedelta(days=N_DAYS_TEST)).date().strftime('%Y-%m-%d')))]\n",
    "test = transaction_df[(transaction_df['t_dat']>=((pd.to_datetime(max_date) - timedelta(days=N_DAYS_TEST)).date().strftime('%Y-%m-%d')))]\n",
    "\n",
    "# Delete transaction_df from the namespace to free up some memory\n",
    "transaction_df = None\n",
    "del transaction_df\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 498,
     "status": "ok",
     "timestamp": 1666497896227,
     "user": {
      "displayName": "hm kaggle",
      "userId": "10519941953201100561"
     },
     "user_tz": 420
    },
    "id": "1bd281c2-0886-405c-bd18-626a7b4e5ddd",
    "outputId": "84518dee-74dc-446e-909f-707766e4a9a0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "22"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Remove users from test that are not in train to remove cold-start scenarios\n",
    "test = test[test['customer_id'].isin(train['customer_id'].unique())]\n",
    "\n",
    "# Make distinction between train and test articles since we'll be creating article representations using the training set\n",
    "train_article_df = article_df[article_df['article_id'].isin(train['article_id'].unique())].copy()\n",
    "test_article_df = article_df[~article_df['article_id'].isin(train['article_id'].unique())].copy()\n",
    "\n",
    "# Free up memory again\n",
    "gc.collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 14613,
     "status": "ok",
     "timestamp": 1666497910773,
     "user": {
      "displayName": "hm kaggle",
      "userId": "10519941953201100561"
     },
     "user_tz": 420
    },
    "id": "de284215-b7bf-46e6-b554-0584aeeb1a8e",
    "outputId": "84e41dc1-e812-4028-c9b3-0c19bfbece49"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 1769097/1769097 [00:04<00:00, 382543.41it/s]\n",
      "100%|██████████| 149631/149631 [00:00<00:00, 395545.81it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "((5307291, 3), (448893, 3))"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Generate N negative samples per row (user-article transaction)\n",
    "\n",
    "def generate_negative_samples(df, article_list, num_neg=1, negative_sample_multiplier=20):\n",
    "    \"\"\"\n",
    "    Generates `num_neg` negative samples from `article_list` per row (customer_id, article_id interaction) in `df`.\n",
    "\n",
    "    Args:\n",
    "        df (DataFrame): Positive transactions between customers and articles. Expects features `customer_id` and `article_id`.\n",
    "        article_list (list): Article IDs for negative sampling.\n",
    "        num_neg (int): Number of negative samples to generate per row.\n",
    "        negative_sample_multiplier (int): Multiplier that allows us to randomly sample negative candidates all at once.\n",
    "            Necessary to be larger than 1 in most cases since we will filter out some candidates.\n",
    "\n",
    "    Raises:\n",
    "        Warning in case we did not generate the intended number of negative samples per row.\n",
    "\n",
    "    Returns:\n",
    "        negative_df (DataFrame): Original, positive samples concatenated with generated, negative samples.\n",
    "    \"\"\"\n",
    "    # Create lookup of positive samples. We don't want to include negative samples that are actually positives\n",
    "    customer2positives = dict(df.groupby('customer_id')['article_id'].apply(lambda x: x.values))\n",
    "    customer2positives = {k:set(v) for k,v in customer2positives.items()}\n",
    "\n",
    "    # Sample random negatives all at once\n",
    "    np.random.seed(42)\n",
    "    random_sample_size = (len(df), num_neg*negative_sample_multiplier)\n",
    "    random_samples = np.random.choice(article_list, \n",
    "                                      size=random_sample_size,\n",
    "                                      replace=True)\n",
    "    \n",
    "    # Generate negative samples\n",
    "    customers, articles, labels = [], [], []\n",
    "    for n, (customer_id, article_id) in enumerate(tqdm(zip(df['customer_id'], df['article_id']), total=len(df), \n",
    "                                                       leave=True, position=0)):\n",
    "        customers.append(customer_id)\n",
    "        articles.append(article_id)\n",
    "        labels.append(1)\n",
    "        count, temp_negatives = 0, set()\n",
    "        for negative_id in random_samples[n]:\n",
    "            if (negative_id not in customer2positives[customer_id]) \\\n",
    "                & (negative_id not in temp_negatives):\n",
    "                customers.append(customer_id)\n",
    "                articles.append(negative_id)\n",
    "                labels.append(0)\n",
    "                temp_negatives.add(negative_id)\n",
    "                count += 1\n",
    "            if count == num_neg:\n",
    "                break\n",
    "\n",
    "    # Create result dataframe of positives and negatives, and data quality check\n",
    "    negative_df = pd.DataFrame({'customer_id':customers, 'article_id':articles, 'label':labels})\n",
    "    if len(negative_df) != (len(df)*num_neg+len(df)):\n",
    "        print('WARNING: Not enough samples generated. Increase `negative_sample_multiplier`.')\n",
    "    return negative_df\n",
    "\n",
    "\n",
    "# Note: We only generate negative samples from the training set to avoid leakage\n",
    "articles_set = list(train.article_id.unique())\n",
    "train_negative = generate_negative_samples(train, articles_set, num_neg=2)\n",
    "test_negative = generate_negative_samples(test, articles_set, num_neg=2)\n",
    "\n",
    "train_negative.shape, test_negative.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Frr6CK1NX6dJ"
   },
   "source": [
    "## **3. Initial User and Item Representations**\n",
    "[back to top](#scrollTo=qfTG6NXWGjQt&uniqifier=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3df9c2e0-af9e-406d-a9c6-c38a87347728"
   },
   "source": [
    "We'll start by extracting our fixed article representations by taking the raw categorical features and concatenating them with TF-IDF  representations of article descriptions. We then process them with SVD. In total TF-IDF followed by SVD is a technique known as [Latent Semantic Indexing (LSI)](https://radimrehurek.com/gensim/models/lsimodel.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "executionInfo": {
     "elapsed": 18819,
     "status": "ok",
     "timestamp": 1666497929372,
     "user": {
      "displayName": "hm kaggle",
      "userId": "10519941953201100561"
     },
     "user_tz": 420
    },
    "id": "2f0d3430-6ff8-4d07-9707-83d332492c9d",
    "outputId": "d19b8196-bcd0-4bdb-d953-a284870efeaa"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/dist-packages/sklearn/utils/validation.py:598: FutureWarning: np.matrix usage is deprecated in 1.0 and will raise a TypeError in 1.2. Please convert to a numpy array with np.asarray. For more information see: https://numpy.org/doc/stable/reference/generated/numpy.matrix.html\n",
      "  FutureWarning,\n",
      "/usr/local/lib/python3.7/dist-packages/sklearn/utils/validation.py:598: FutureWarning: np.matrix usage is deprecated in 1.0 and will raise a TypeError in 1.2. Please convert to a numpy array with np.asarray. For more information see: https://numpy.org/doc/stable/reference/generated/numpy.matrix.html\n",
      "  FutureWarning,\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 29.5 s, sys: 1.78 s, total: 31.3 s\n",
      "Wall time: 19 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "def train_article_representations():\n",
    "    \"\"\"\n",
    "    Trains article representations for both the train and test sets. \n",
    "    \n",
    "    Args:\n",
    "        None.\n",
    "        \n",
    "    Returns:\n",
    "        article2vec (dict): Article ID to vector representation lookup.\n",
    "        \n",
    "    \"\"\"\n",
    "    # Find categorical columns\n",
    "    ohe_columns = []\n",
    "    for col in train_article_df.columns:\n",
    "        if train_article_df[col].dtype == \"int64\" and len(train_article_df[col].unique()) <= 500:\n",
    "            ohe_columns.append(col)\n",
    "\n",
    "    # Get ohe article features\n",
    "    # Make sure test only has values included in train, and the order of the encodings is the same\n",
    "    train_ohe_vectors = pd.get_dummies(train_article_df[ohe_columns], columns=ohe_columns)\n",
    "    test_ohe_vectors = pd.get_dummies(test_article_df[ohe_columns], columns=ohe_columns)\n",
    "    for col in train_ohe_vectors.columns:\n",
    "        if col not in test_ohe_vectors.columns:\n",
    "            test_ohe_vectors[col] = 0\n",
    "    test_ohe_vectors = test_ohe_vectors[train_ohe_vectors.columns].values\n",
    "    train_ohe_vectors = train_ohe_vectors.values\n",
    "\n",
    "    # Get tf-idf article features\n",
    "    tfidf = TfidfVectorizer(min_df=3)\n",
    "    train_tfidf_vectors = tfidf.fit_transform(train_article_df[\"detail_desc\"].fillna(\"nodesc\"))\n",
    "    test_tfidf_vectors = tfidf.transform(test_article_df[\"detail_desc\"].fillna(\"nodesc\"))\n",
    "\n",
    "    # Represent articles as 200 element vectors using concatenated ohe and tf-idf features\n",
    "    svd = TruncatedSVD(n_components=200, random_state=42)\n",
    "    train_article_vectors = np.hstack([train_ohe_vectors, \n",
    "                                       train_tfidf_vectors.todense()])\n",
    "    train_article_vectors = svd.fit_transform(train_article_vectors)\n",
    "    test_article_vectors = np.hstack([test_ohe_vectors, \n",
    "                                      test_tfidf_vectors.todense()])\n",
    "    test_article_vectors = svd.transform(test_article_vectors)\n",
    "    \n",
    "    # Create article to vector lookup\n",
    "    article2vec = dict(zip(train_article_df['article_id'], train_article_vectors))\n",
    "    article2vec.update(dict(zip(test_article_df['article_id'], test_article_vectors)))\n",
    "    \n",
    "    return article2vec\n",
    "\n",
    "\n",
    "article2vec = train_article_representations()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f0c3082e-1da7-466f-8b9f-c777c8ef63cd"
   },
   "source": [
    "Next we'll write the first function to create user representations as a set of raw features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "5de63a95-ef6c-4796-8e59-0cb33b9b948e"
   },
   "outputs": [],
   "source": [
    "def user_representation_1():\n",
    "    \"\"\"\n",
    "    First function to generate user representations: Raw user features.\n",
    "    \n",
    "    Args:\n",
    "        None.\n",
    "        \n",
    "    Returns:\n",
    "        user2vec (dictionary): Customer to vector representations lookup.\n",
    "    \"\"\"\n",
    "    df_cust = customer_df.copy()\n",
    "    df_cust[\"age\"] = df_cust[\"age\"].fillna(df_cust[\"age\"].mean())\n",
    "    df_cust[[\"FN\",\"Active\"]] = df_cust[[\"FN\",\"Active\"]].fillna(0)\n",
    "    df_cust[\"club_member_status\"] = df_cust[\"club_member_status\"].apply(lambda x:1 if x == \"ACTIVE\" else 0)\n",
    "    df_cust[\"fashion_news_frequency\"] = df_cust[\"fashion_news_frequency\"].apply(lambda x:0 if x == \"NONE\" else 1)\n",
    "    df_cust = df_cust[[\"customer_id\",\"age\",\"FN\",\"Active\",\"club_member_status\",\"fashion_news_frequency\"]]\n",
    "    user2vec = df_cust.set_index('customer_id').apply(lambda x: x.values, axis=1).to_dict()\n",
    "    return user2vec"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "P7cb1GX0ckr0"
   },
   "source": [
    "Now we have all the ingredients we need: A basic version of user representations, article representations, and transactions data on which we can train our downstream task. We'll consider the task of predicting whether or not a user will purchase an article as we've been doing the past 2 weeks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_FYTNTwUc3ot"
   },
   "source": [
    "## **4. LightGBM as a Ranker**\n",
    "[back to top](#scrollTo=qfTG6NXWGjQt&uniqifier=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "219f3784-220e-4889-ae88-09c80f556479"
   },
   "source": [
    "While we have used neural models so far, let's try a tree-based model, [LightGBM](https://lightgbm.readthedocs.io/en/v3.3.2/), for this task. Note that, due to Colab's compute constraints, it would normally be difficult to train a model with the amount of data we currently have. We've leveraged [h5py](https://docs.h5py.org/en/latest/quick.html) files, which allow for storing data on disk as opposed to in memory, so that we can train on data that is too large to fit in memory!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "21e76b18-8d9e-4a1b-bf84-a929c8f16c6f"
   },
   "outputs": [],
   "source": [
    "# Model parameters\n",
    "\n",
    "random_seed = 42\n",
    "\n",
    "lgb_params = {\n",
    "    \"objective\": \"binary\",\n",
    "    \"num_trees\":300,\n",
    "    \"max_leaves\":300,\n",
    "    \"bagging_seed\": random_seed,\n",
    "    \"random_state\": random_seed,\n",
    "    \"verbose\": 1,\n",
    "    \"device\": \"gpu\"\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "5e0c7a4f-8e4d-42f7-b379-1ebc870375d9"
   },
   "outputs": [],
   "source": [
    "# Train and evaluate LightGBM model\n",
    "\n",
    "def calculate_test_mrr(predictions):\n",
    "    \"\"\"\n",
    "    Calculates and prints Mean Reciprocal Rank (MRR) on the test set.\n",
    "    \n",
    "    Args:\n",
    "        predictions (np.array): Model predictions.\n",
    "        \n",
    "    Returns:\n",
    "        None\n",
    "    \"\"\"\n",
    "    dataframe = test_negative.copy()\n",
    "    dataframe['prediction_score'] = predictions\n",
    "    dataframe.sort_values(by=['customer_id', 'prediction_score'], ascending=False, inplace=True)\n",
    "    reciprocal_rank = dataframe.groupby('customer_id').apply(lambda x: 1/(np.where(x['label']==1)[0][0]+1)) \n",
    "    dataframe.drop('prediction_score', axis=1, inplace=True)\n",
    "    print(f'Test MRR: {reciprocal_rank.mean():.4f}')\n",
    "    print('Distribution:')\n",
    "    display(reciprocal_rank.describe())\n",
    "\n",
    "\n",
    "def evaluate(model, user2vec, article2vec):\n",
    "    \"\"\"\n",
    "    Evaluates trained models.\n",
    "    \n",
    "    Args:\n",
    "        model (LightGBM Model): Trained LightGBM model for inference.\n",
    "        train_features (np.array): Raw train data in the same format used during training.\n",
    "        test_features (np.array): Raw test data in the same format used during training.\n",
    "\n",
    "    Returns:\n",
    "        None\n",
    "    \"\"\"\n",
    "\n",
    "    # Make predictions\n",
    "    preds = []\n",
    "    for i in tqdm(range(0, len(test_negative), 100000)):\n",
    "        X_chunk = np.concatenate([np.stack(test_negative['customer_id'].iloc[i:(i+100000)].map(user2vec).values), \n",
    "                                  np.stack(test_negative['article_id'].iloc[i:(i+100000)].map(article2vec).values)], axis=1)\n",
    "        preds.extend(model.predict(X_chunk))\n",
    "\n",
    "    # Calculate test MRR\n",
    "    calculate_test_mrr(preds)\n",
    "\n",
    "\n",
    "class HDFSequence(lgb.Sequence):\n",
    "    def __init__(self, hdf_dataset):\n",
    "        \"\"\"\n",
    "        From LightGBM Docs: https://github.com/microsoft/LightGBM/blob/master/examples/python-guide/dataset_from_multi_hdf5.py\n",
    "        \n",
    "        Construct a sequence object from HDF5 with required interface.\n",
    "        Parameters\n",
    "        ----------\n",
    "        hdf_dataset : h5py.Dataset\n",
    "            Dataset in HDF5 file.\n",
    "        batch_size : int\n",
    "            Size of a batch. When reading data to construct lightgbm Dataset, each read reads batch_size rows.\n",
    "        \"\"\"\n",
    "        self.data = hdf_dataset\n",
    "        self.batch_size = 64\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        return self.data[idx]\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.data)\n",
    "\n",
    "def train_and_evaluate_lightgbm(user_representation_functions):\n",
    "    \"\"\"\n",
    "    Trains and evaluates the LightGBM model given user representation methods.\n",
    "\n",
    "    Args:\n",
    "        user_representation_functions (list of functions): List of functions \n",
    "            that return a user2vec representation dictionary where\n",
    "            user IDs are keys and their representations are values.\n",
    "\n",
    "    Returns:\n",
    "        None.\n",
    "    \"\"\"\n",
    "    \n",
    "    # Calculate and organize user features\n",
    "    print(\"Calculating user features...\")\n",
    "    user2vec = defaultdict(lambda: np.array([]))\n",
    "    for n, user_rep_function in enumerate(user_representation_functions):\n",
    "        print(f\"\\t{n+1}/{len(user_representation_functions)} functions...\")\n",
    "        temp_user2vec = user_rep_function()\n",
    "        for user, vec in temp_user2vec.items():\n",
    "            user2vec[user] = np.concatenate([user2vec[user], vec])\n",
    "        temp_user2vec = None\n",
    "        del temp_user2vec\n",
    "        gc.collect()\n",
    "\n",
    "    # Create the train dataset\n",
    "    print(\"Creating train Dataset...\")\n",
    "    # Save HDF5 files\n",
    "    try:\n",
    "        shutil.rmtree('hdf5_files')\n",
    "        os.mkdir('hdf5_files')\n",
    "    except:\n",
    "        os.mkdir('hdf5_files')\n",
    "    for n, i in tqdm(enumerate(range(0, len(train_negative), 100000))):\n",
    "        with h5py.File(f'hdf5_files/train_features_{n}.hdf5', 'w') as f:\n",
    "            X_chunk = np.concatenate([np.stack(train_negative['customer_id'].iloc[i:(i+100000)].map(user2vec).values), \n",
    "                                      np.stack(train_negative['article_id'].iloc[i:(i+100000)].map(article2vec).values)], axis=1)\n",
    "            f.create_dataset('X', data=X_chunk, chunks=(64,X_chunk.shape[1]), compression='lzf')\n",
    "            y_chunk = train_negative['label'].values[i:(i+100000)]\n",
    "            f.create_dataset('Y', data=y_chunk, chunks=(len(y_chunk),), compression='lzf')\n",
    "    # Read the HDF5 files\n",
    "    data, ylist = [], []\n",
    "    for f in tqdm(os.listdir('hdf5_files')):\n",
    "        f = h5py.File(f\"hdf5_files/{f}\", 'r')\n",
    "        data.append(HDFSequence(f['X']))\n",
    "        ylist.append(f['Y'][:])\n",
    "    # Create dataset\n",
    "    train_dataset = lgb.Dataset(data, label=np.concatenate(ylist))\n",
    "    \n",
    "    # Train\n",
    "    print('Training...')\n",
    "    model = lgb.train(lgb_params, train_set=train_dataset)\n",
    "\n",
    "    # Delete HDF5 files\n",
    "    shutil.rmtree('hdf5_files')\n",
    "\n",
    "    # Evaluate\n",
    "    print(\"Evaluating...\")\n",
    "    evaluate(model, user2vec, article2vec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Fs2tLaPcdpbx"
   },
   "source": [
    "Below we show how to use the training function we defined above. Given a list of `user representation` functions, which return a user-to-representation dictionary object, we can train and evaluate our model based on MRR. In the next sections, you'll add more user representations and see the effects on the task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 585
    },
    "executionInfo": {
     "elapsed": 482617,
     "status": "ok",
     "timestamp": 1666499288926,
     "user": {
      "displayName": "hm kaggle",
      "userId": "10519941953201100561"
     },
     "user_tz": 420
    },
    "id": "TBrpF0Dw4SYH",
    "outputId": "d101f62b-10cb-4fab-fe81-61c79825243f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Calculating user features...\n",
      "\t1/1 functions...\n",
      "Creating train Dataset...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "54it [01:12,  1.34s/it]\n",
      "100%|██████████| 54/54 [00:00<00:00, 165.24it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training...\n",
      "[LightGBM] [Info] Number of positive: 1769097, number of negative: 3538194\n",
      "[LightGBM] [Info] This is the GPU trainer!!\n",
      "[LightGBM] [Info] Total Bins 51086\n",
      "[LightGBM] [Info] Number of data points in the train set: 5307291, number of used features: 205\n",
      "[LightGBM] [Info] Using GPU Device: Tesla T4, Vendor: NVIDIA Corporation\n",
      "[LightGBM] [Info] Compiling OpenCL Kernel with 256 bins...\n",
      "[LightGBM] [Info] GPU programs have been built\n",
      "[LightGBM] [Info] Size of histogram bin entry: 8\n",
      "[LightGBM] [Info] 205 dense feature groups (1052.78 MB) transferred to GPU in 1.248005 secs. 0 sparse feature groups\n",
      "[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.333333 -> initscore=-0.693147\n",
      "[LightGBM] [Info] Start training from score -0.693147\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 5/5 [00:14<00:00,  2.89s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test MRR: 0.8852\n",
      "Distribution:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "count    42395.000000\n",
       "mean         0.885178\n",
       "std          0.229665\n",
       "min          0.071429\n",
       "25%          1.000000\n",
       "50%          1.000000\n",
       "75%          1.000000\n",
       "max          1.000000\n",
       "dtype: float64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 12min 30s, sys: 22.3 s, total: 12min 52s\n",
      "Wall time: 8min 2s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "train_and_evaluate_lightgbm([user_representation_1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rygtKRLYuEMR"
   },
   "source": [
    "Even with raw user representations we significantly outperform our neural rankers! Note that we didn't tune either model particularly thoroughly, but it's clear that LightGBM is a highly competitve ranker."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "L-ErqwvYd9F7"
   },
   "source": [
    "## **5. TODO 3.3: Your User Representations**\n",
    "[back to top](#scrollTo=qfTG6NXWGjQt&uniqifier=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2a4b63a5-56ee-4bd2-b7b8-2aebb5299b31"
   },
   "source": [
    "We have now trained a LightGBM model using `user_representation_1` as the user representation technique. Your goals are to implement `user_representation_2` and `user_representation_3` functions, and train/evaluate the resulting LightGBM model, with the following two ways of estimating user representations:\n",
    "\n",
    "1. `user_representation_2`: Average of embeddings of recently purchased articles, using the `article2vec` dictionary created by the `train_article_representations` function above.\n",
    "2. `user_representation_3`: Use Doc2Vec where we treat each user as a document, and the set of articles the user has purchased as the set of words in the document. You can find a good implementation of Doc2Vec [here](https://radimrehurek.com/gensim/models/doc2vec.html).  \n",
    "  \n",
    "<br>We will be concatenating each new user representation to `user_representation_1` to show how each effect the model and how represensations 2 and 3 differ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "5b277163-f1d2-4767-92ed-498ebf9fb5bb"
   },
   "outputs": [],
   "source": [
    "def user_representation_2():\n",
    "    \"\"\"\n",
    "    Second function to generate user representations: Average of embeddings of recently purchased articles.\n",
    "    \n",
    "    Args:\n",
    "        None.\n",
    "        \n",
    "    Returns:\n",
    "        user2vec (dictionary): Customer to vector representations lookup.\n",
    "            Example: {'customer_id_1':vector_1, ..., 'customer_id_n':vector_n}\n",
    "    \"\"\"\n",
    "    user2vec = {}\n",
    "\n",
    "    # Your code goes here\n",
    "        # Note: You can reference train_negative[train_negative['label']==1] to isolate all customers and articles they've interacted with\n",
    "\n",
    "    \n",
    "    return user2vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 656
    },
    "executionInfo": {
     "elapsed": 834568,
     "status": "ok",
     "timestamp": 1666498763875,
     "user": {
      "displayName": "hm kaggle",
      "userId": "10519941953201100561"
     },
     "user_tz": 420
    },
    "id": "2YqRVIRV4VTo",
    "outputId": "3ff3abba-28b7-4082-9b1a-eafb55b575ac"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Calculating user features...\n",
      "\t1/2 functions...\n",
      "\t2/2 functions...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 330712/330712 [00:26<00:00, 12509.55it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating train Dataset...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "54it [01:35,  1.76s/it]\n",
      "100%|██████████| 54/54 [00:00<00:00, 201.48it/s]\n",
      "/usr/local/lib/python3.7/dist-packages/lightgbm/engine.py:177: UserWarning: Found `num_trees` in params. Will use it instead of argument\n",
      "  _log_warning(f\"Found `{alias}` in params. Will use it instead of argument\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training...\n",
      "[LightGBM] [Info] Number of positive: 1769097, number of negative: 3538194\n",
      "[LightGBM] [Info] This is the GPU trainer!!\n",
      "[LightGBM] [Info] Total Bins 102086\n",
      "[LightGBM] [Info] Number of data points in the train set: 5307291, number of used features: 405\n",
      "[LightGBM] [Info] Using GPU Device: Tesla T4, Vendor: NVIDIA Corporation\n",
      "[LightGBM] [Info] Compiling OpenCL Kernel with 256 bins...\n",
      "[LightGBM] [Info] GPU programs have been built\n",
      "[LightGBM] [Info] Size of histogram bin entry: 8\n",
      "[LightGBM] [Info] 405 dense feature groups (2065.06 MB) transferred to GPU in 2.491092 secs. 0 sparse feature groups\n",
      "[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.333333 -> initscore=-0.693147\n",
      "[LightGBM] [Info] Start training from score -0.693147\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 5/5 [00:22<00:00,  4.48s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test MRR: 0.8156\n",
      "Distribution:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "count    42395.000000\n",
       "mean         0.815565\n",
       "std          0.278938\n",
       "min          0.041667\n",
       "25%          0.500000\n",
       "50%          1.000000\n",
       "75%          1.000000\n",
       "max          1.000000\n",
       "dtype: float64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 22min 16s, sys: 36.6 s, total: 22min 52s\n",
      "Wall time: 13min 54s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# We have saved our solution's results below for your reference\n",
    "train_and_evaluate_lightgbm([user_representation_1, user_representation_2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VHbUcxHcvN_n"
   },
   "source": [
    "This is _very_ interesting - it seems as though adding these extra features significantly hurt model performance. It's an important lesson in that more complex features aren't necessarily better and that starting simply before scaling up will help you build better models. That said, there are a couple reasons this might be the case including:  \n",
    "1. There are many parameters to tune in LightGBM.\n",
    "2. We didn't do any extra feature engineering, such as the cosine similarity between the user's average article vector and the query article vector.  \n",
    "  \n",
    "<br>Next we'll see if Doc2Vec can give us any improvement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "6d3b1638-fc8b-456d-a123-367fd6fc54e8"
   },
   "outputs": [],
   "source": [
    "def user_representation_3():\n",
    "    \"\"\"\n",
    "    Third function to generate user representations: Doc2Vec.\n",
    "    Doc2vec model is an embedding learning method that enables us to learn representations of a document.\n",
    "    We treat each user as a document, and the set of articles the user has purchased as the set of words in the document.\n",
    "    \n",
    "    Args:\n",
    "        None.\n",
    "        \n",
    "    Returns:\n",
    "        user2vec (dictionary): Customer to vector representations lookup.\n",
    "            Example: {'customer_id_1':vector_1, ..., 'customer_id_n':vector_n}\n",
    "    \"\"\"\n",
    "    user2vec = {}\n",
    "    \n",
    "    # Your code goes here\n",
    "        # Note: We used a vector size of 200 and a random_seed of 42\n",
    "            \n",
    "    \n",
    "    return user2vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 656
    },
    "executionInfo": {
     "elapsed": 1168841,
     "status": "ok",
     "timestamp": 1666500623239,
     "user": {
      "displayName": "hm kaggle",
      "userId": "10519941953201100561"
     },
     "user_tz": 420
    },
    "id": "KliEG91r4bCJ",
    "outputId": "9a755679-dba8-4ab4-ce3f-546e1d023ce2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Calculating user features...\n",
      "\t1/2 functions...\n",
      "\t2/2 functions...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Doc2Vec Inference: 100%|██████████| 330712/330712 [02:51<00:00, 1928.57it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating train Dataset...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "54it [01:35,  1.77s/it]\n",
      "100%|██████████| 54/54 [00:00<00:00, 172.84it/s]\n",
      "/usr/local/lib/python3.7/dist-packages/lightgbm/engine.py:177: UserWarning: Found `num_trees` in params. Will use it instead of argument\n",
      "  _log_warning(f\"Found `{alias}` in params. Will use it instead of argument\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training...\n",
      "[LightGBM] [Info] Number of positive: 1769097, number of negative: 3538194\n",
      "[LightGBM] [Info] This is the GPU trainer!!\n",
      "[LightGBM] [Info] Total Bins 102086\n",
      "[LightGBM] [Info] Number of data points in the train set: 5307291, number of used features: 405\n",
      "[LightGBM] [Info] Using GPU Device: Tesla T4, Vendor: NVIDIA Corporation\n",
      "[LightGBM] [Info] Compiling OpenCL Kernel with 256 bins...\n",
      "[LightGBM] [Info] GPU programs have been built\n",
      "[LightGBM] [Info] Size of histogram bin entry: 8\n",
      "[LightGBM] [Info] 405 dense feature groups (2065.06 MB) transferred to GPU in 2.449282 secs. 0 sparse feature groups\n",
      "[LightGBM] [Info] [binary:BoostFromScore]: pavg=0.333333 -> initscore=-0.693147\n",
      "[LightGBM] [Info] Start training from score -0.693147\n",
      "Evaluating...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 5/5 [00:20<00:00,  4.08s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test MRR: 0.8873\n",
      "Distribution:\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "count    42395.000000\n",
       "mean         0.887316\n",
       "std          0.228841\n",
       "min          0.050000\n",
       "25%          1.000000\n",
       "50%          1.000000\n",
       "75%          1.000000\n",
       "max          1.000000\n",
       "dtype: float64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 27min 47s, sys: 1min 4s, total: 28min 51s\n",
      "Wall time: 19min 28s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# We have saved our solution's results below for your reference\n",
    "train_and_evaluate_lightgbm([user_representation_1, user_representation_3])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VJKSq2FRiahd"
   },
   "source": [
    "Huzzah! Doc2Vec has given us an improvement in our test-set MRR!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-BxR9VPpjoNi"
   },
   "source": [
    "## **6. TODO 3.4: LSTM User Representations (Optional)**\n",
    "[back to top](#scrollTo=qfTG6NXWGjQt&uniqifier=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ce141145-f5af-4f2d-9d94-f4bb50e6208f"
   },
   "source": [
    "As an optional task, implement `user_representation_4` where user representations are learnt by a sequential LSTM model. The LSTM model will need to be trained on a task -- the task itself could be the downstream task of predicting whether or not a user would purchase a given article. The final hidden layer of the LSTM model can be used as the user representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "88eb3b58-062c-4c96-b3f0-2db4407ce1b3"
   },
   "outputs": [],
   "source": [
    "def user_representation_4():\n",
    "    \"\"\"\n",
    "    Optional, fourth function to generate user representations: LSTM.\n",
    "    \n",
    "    Args:\n",
    "        None.\n",
    "        \n",
    "    Returns:\n",
    "        user2vec (dictionary): Customer to vector representations lookup.\n",
    "            Example: {'customer_id_1':vector_1, ..., 'customer_id_n':vector_n}\n",
    "    \"\"\"\n",
    "    user2vec = {}\n",
    "    \n",
    "    # Your code goes here\n",
    "       \n",
    "\n",
    "    return user2vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "rc-wYVS44P47"
   },
   "outputs": [],
   "source": [
    "%%time\n",
    "train_and_evaluate_lightgbm([user_representation_1, user_representation_4])"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [
    "cB4tqBSaHIqt",
    "9wDLwRU6HUNG",
    "vq47Wt-0Jan-",
    "3u93B8JdJ9pg",
    "TRAg9cqeMiKl",
    "bd4452f5-ce17-43df-95f2-e9b58e8cad5e",
    "-2Xh5R8xT3to",
    "e127987e-44f9-43b3-9219-1036edd0d14c",
    "Yk2dt4l1VS_A",
    "oU7Z49ymW5yO",
    "Frr6CK1NX6dJ",
    "_FYTNTwUc3ot",
    "L-ErqwvYd9F7",
    "-BxR9VPpjoNi"
   ],
   "provenance": []
  },
  "environment": {
   "kernel": "python3",
   "name": "tf2-gpu.2-6.m89",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-6:m89"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}