Practice 13 - Collaborative Boosted FW.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recommender Systems 2021/22\n",
    "\n",
    "### Practice - Collaborative Boosted FW\n",
    "\n",
    "\n",
    "### Underlying idea... the way to capture the feature importance from the user point of view, is to... use their point of view (e.g., ratings, collaborative similarity)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A few info about feature weightign techniques\n",
    "\n",
    "* Information retrieval methods (TF-IDF, BM25..) do not optimize a predictive model\n",
    "* Embedding methods (FISM, UFSM, Factorization Machines) optimize everything at the same time, model and weights, but are difficult to train and sensitive to noise\n",
    "* Wrapper methods (LFW, CFW) apply a two step approach approximating an already built collaborative model, more robust and easier to tune\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Movielens10M: Verifying data consistency...\n",
      "Movielens10M: Verifying data consistency... Passed!\n",
      "DataReader: current dataset is: <class 'Data_manager.Dataset.Dataset'>\n",
      "\tNumber of items: 10681\n",
      "\tNumber of users: 69878\n",
      "\tNumber of interactions in URM_all: 10000054\n",
      "\tValue range in URM_all: 0.50-5.00\n",
      "\tInteraction density: 1.34E-02\n",
      "\tInteractions per user:\n",
      "\t\t Min: 2.00E+01\n",
      "\t\t Avg: 1.43E+02\n",
      "\t\t Max: 7.36E+03\n",
      "\tInteractions per item:\n",
      "\t\t Min: 0.00E+00\n",
      "\t\t Avg: 9.36E+02\n",
      "\t\t Max: 3.49E+04\n",
      "\tGini Index: 0.57\n",
      "\n",
      "\tICM name: ICM_all, Value range: 1.00 / 69.00, Num features: 10126, feature occurrences: 128384, density 1.19E-03\n",
      "\tICM name: ICM_genres, Value range: 1.00 / 1.00, Num features: 20, feature occurrences: 21564, density 1.01E-01\n",
      "\tICM name: ICM_tags, Value range: 1.00 / 69.00, Num features: 10106, feature occurrences: 106820, density 9.90E-04\n",
      "\tICM name: ICM_year, Value range: 6.00E+00 / 2.01E+03, Num features: 1, feature occurrences: 10681, density 1.00E+00\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from Data_manager.split_functions.split_train_validation_random_holdout import split_train_in_two_percentage_global_sample\n",
    "from Data_manager.Movielens.Movielens10MReader import Movielens10MReader\n",
    "import numpy as np\n",
    "\n",
    "data_reader = Movielens10MReader()\n",
    "data_loaded = data_reader.load_data()\n",
    "\n",
    "URM_all = data_loaded.get_URM_all()\n",
    "ICM_tags = data_loaded.get_ICM_from_name(\"ICM_tags\")\n",
    "ICM_genres = data_loaded.get_ICM_from_name(\"ICM_genres\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Warning: 73 (0.10 %) of 69878 users have no sampled items\n",
      "Warning: 2463 (3.52 %) of 69878 users have no sampled items\n"
     ]
    }
   ],
   "source": [
    "URM_train, URM_test = split_train_in_two_percentage_global_sample(URM_all, train_percentage = 0.8)\n",
    "URM_train, URM_validation = split_train_in_two_percentage_global_sample(URM_train, train_percentage = 0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's now consider two simple models, a KNN pure collaborative and a KNN content-based\n",
    "The hyperparameters are taken from a previous optimization step.\n",
    "\n",
    "Warning: not very clean, the data split in this case is certainly different, though for reasonably large datasets the optimal hyperparameters do not change a lot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_hyperparams_ItemKNNCF = {'topK': 700, 'shrink': 200, 'similarity': 'cosine', 'normalize': True}\n",
    "best_hyperparams_ItemKNNCBF = {'topK': 500, 'shrink': 1000, 'similarity': 'cosine', 'normalize': True}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Create the two models, collaborative and content based"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ItemKNNCFRecommender: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "Similarity column 10681 (100.0%), 689.51 column/sec. Elapsed time 15.49 sec\n",
      "ItemKNNCBFRecommender: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "ItemKNNCBFRecommender: ICM Detected 3087 (28.9%) items with no features.\n",
      "Similarity column 10681 (100.0%), 5868.25 column/sec. Elapsed time 1.82 sec\n"
     ]
    }
   ],
   "source": [
    "from Recommenders.KNN.ItemKNNCFRecommender import ItemKNNCFRecommender\n",
    "from Recommenders.KNN.ItemKNNCBFRecommender import ItemKNNCBFRecommender\n",
    "\n",
    "\n",
    "itemKNNCF = ItemKNNCFRecommender(URM_train)\n",
    "itemKNNCF.fit(**best_hyperparams_ItemKNNCF)\n",
    "\n",
    "itemKNNCBF = ItemKNNCBFRecommender(URM_train, ICM_tags)\n",
    "itemKNNCBF.fit(**best_hyperparams_ItemKNNCBF)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Question, how the two similarities differ?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "W_sparse_CF = itemKNNCF.W_sparse\n",
    "W_sparse_CBF = itemKNNCBF.W_sparse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's plot the global similarity distribution "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "W_sparse_CF_sorted = np.sort(W_sparse_CF.data.copy())\n",
    "W_sparse_CBF_sorted = np.sort(W_sparse_CBF.data.copy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([7.3099858e-05, 7.4463380e-05, 8.4556516e-05, ..., 5.9459770e-01,\n",
       "       5.9845638e-01, 5.9845638e-01], dtype=float32)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "W_sparse_CF_sorted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([5.2254670e-04, 5.8741629e-04, 5.9216074e-04, ..., 6.8608046e-01,\n",
       "       6.9043833e-01, 6.9043833e-01], dtype=float32)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "W_sparse_CBF_sorted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAuTUlEQVR4nO3deXxcZb348c83e9IkbbN0S7oEGtrS0hYIBQRBEaQICghqQeUCXnpRkIsgXi7+vMAVuSwqIIulAsIFBFFAi5TiFaSAiLbFYndaStuELtnaZl8m8/39cU7SaTJJJ21OzpzM9/16zWtmzvrNtPN853mec55HVBVjjDGJK8nvAIwxxvjLEoExxiQ4SwTGGJPgLBEYY0yCs0RgjDEJLsXvAPqroKBAJ02a5HcYxhgTKCtWrKhW1cJo6wKXCCZNmsTy5cv9DsMYYwJFRLb2ts6ahowxJsFZIjDGmATnaSIQkbkiskFENonIjVHW3yAiK93HahHpEJE8L2MyxhizP8/6CEQkGXgQOAOoAJaJyCJVXdu5jareDdztbv954DuqWtvfc7W3t1NRUUFLS8vABJ9AMjIyKC4uJjU11e9QjDE+8bKzeA6wSVU3A4jIs8C5wNpetr8IeOZgTlRRUUFOTg6TJk1CRA4q2ESkqtTU1FBRUUFJSYnf4RhjfOJl01ARUB7xvsJd1oOIZAFzgecP5kQtLS3k5+dbEugnESE/P99qUsYkOC8TQbRSubehTj8P/KW3ZiERmS8iy0VkeVVVVfSTWRI4KPa5GWO8TAQVwPiI98XA9l62nUcfzUKqulBVy1S1rLAw6v0QxvSusQZe/xHsXO13JMbEJS8TwTKgVERKRCQNp7Bf1H0jERkOnAr83sNYPJednd31evHixZSWlrJt2zZuueUWsrKyqKysjLqtiHD99dd3vf/xj3/MLbfcEvUcr7zyCmVlZUybNo2pU6fy3e9+F4BbbrmFoqIiZs+ezezZs7nxxh4XaCW25lp48y6oWu93JMbEJc8SgaqGgKuBV4F1wHOqukZErhSRKyM2PR/4o6o2ehXLYHrttdf49re/zZIlS5gwYQIABQUF/OQnP4m6fXp6Oi+88ALV1dV9Hnf16tVcffXVPPXUU6xbt47Vq1dz2GGHda3/zne+w8qVK1m5ciV33HHHwP1Bxpghz9P7CFR1saoeoaqHq+qP3GULVHVBxDaPq+o8L+MYLG+99RZXXHEFL7/8MocffnjX8ssvv5xf//rX1Nb27AJJSUlh/vz53HPPPX0e+6677uL73/8+U6dO7drvW9/61sD+AcaYhBS4sYYO5NaX1rB2e92AHvPIcbnc/PnpfW7T2trKueeeyxtvvNFVWHfKzs7m8ssv57777uPWW2/tse9VV13FzJkz+d73vtfr8VevXr1fE1J399xzD0899RQAd955J2eeeWaf8RpjTCcbYmKApKam8olPfIJHH3006vprrrmGJ554grq6nkkqNzeXSy65hJ/97GcHff7IpiFLAsaY/hhyNYID/XL3SlJSEs899xynn346t99+OzfddNN+60eMGMHFF1/MQw89FHX/a6+9lmOOOYbLLrss6vrp06ezYsUKZs2aNeCxG2MSm9UIBlBWVhZ/+MMfePrpp6PWDK677joefvhhQqFQj3V5eXl8+ctf7rVGccMNN3D77bfzwQcfABAOh/npT386sH+AMSYhWSIYYHl5eSxZsoTbbruN3/9+/ytiCwoKOP/882ltbY267/XXX9/r1UMzZ87k3nvv5aKLLmLatGnMmDGDHTt2DHj8xpjEI6q93ewbn8rKyrT7xDTr1q1j2rRpPkUUfEP+86veCA+UwQWPwlEX+h2NMb4QkRWqWhZtndUIjDEmwVkiMMaYBGeJwBhjEpwlAmOMSXCWCIwxJsFZIjDGmARniWCAxMsw1FOnTuWb3/wm4XAYgEsvvZSSkpKuIaoPZRgLY8zQZIlggPk9DPXatWtZtWoVS5cu7Vp39913d41DdM011wzAX2mMGUosEQygeBiGuq2tjZaWFkaOHHmIf40xJlEMuUHneOVG2LlqYI855ig4q+/JXuJlGOqtW7dy1llnMXv27K51N9xwA7fddhsATz75JEcddVSff4sxJrFYjWCAxMsw1JWVlTQ2NvLss892rYtsGrIkYIzpbujVCA7wy90r8TIMdWpqKnPnzuXNN99k3rwhMfGbMcZjViMYQPEwDLWq8s477+zXR2GMMX2xRDDA/BqG+p577mH27NnMmDGDUChk8xkbY2Lm6TDUIjIXuA9IBh5R1R7tNiLyKeBeIBWoVtVT+zqmDUM98Ib852fDUBvT5zDUnvURiEgy8CBwBlABLBORRaq6NmKbEcBDwFxV3SYio7yKxxhjTHReNg3NATap6mZVbQOeBc7tts3FwAuqug1AVSsxxhjTw8Zd9VTVR29WPlReJoIioDzifYW7LNIRwEgReUNEVojIJdEOJCLzRWS5iCyvqqqKerKgzbQWL+xzMyYYzr7/bR59+yNPju1lIpAoy7qXOinAscDZwJnAD0TkiB47qS5U1TJVLSssLOxx0IyMDGpqaqxQ6ydVpaamhoyMDL9DMcb4yMv7CCqA8RHvi4HtUbapVtVGoFFE3gRmAR/050TFxcVUVFTQW23B9C4jI4Pi4mK/wzDG+MjLRLAMKBWREuBjYB5On0Ck3wMPiEgKkAYcD/Q96E4UqamplJSUHGK4xhiTmDxLBKoaEpGrgVdxLh99TFXXiMiV7voFqrpORJYA/wTCOJeYrvYqJmOMMT15OsSEqi4GFndbtqDb+7uBu72MwxhjTO/szmJjjElwlgiMMSbBWSIwxpgEZ4nAGGMSnCUCY4xJcJYIjDEmCDwcOMESgTHGBICiSLSBewaAJQJjjAkIj/KAJQJjjAkCL8fUtERgjDEBYU1DxhiTwLwcZN8SgTHGBIR41EtgicAYYwLAy4m3LBEYY0xAWB+BMcYkMOsjMMYYY/cRGGNMIrP7CIw5FF5+g4wZTB51ElgiMMaYBGeJwCQAqxGYoSGQfQQiMldENojIJhG5Mcr6T4nIXhFZ6T7+y8t4TILqbBry6to7Yzzm5T0EACleHVhEkoEHgTOACmCZiCxS1bXdNn1LVc/xKg5j9rFEYILJ698yXtYI5gCbVHWzqrYBzwLneng+Y3phTUNmaAjiEBNFQHnE+wp3WXcnisj7IvKKiEyPdiARmS8iy0VkeVVVlRexmkRgTUMmoLz+KeNlIoj2rev+97wHTFTVWcD9wO+iHUhVF6pqmaqWFRYWDmyUZuizy0fNEBHEpqEKYHzE+2Jge+QGqlqnqg3u68VAqogUeBiTSWhWIzDB5HVnsZeJYBlQKiIlIpIGzAMWRW4gImNEnBwnInPceGo8jMkkJKsRmKHBq58ynl01pKohEbkaeBVIBh5T1TUicqW7fgFwIfBNEQkBzcA89Tr1mcRlfQQmoLwuFD1LBNDV3LO427IFEa8fAB7wMgZjrI/ADBVB7CMwJk50JgKrEZhg8vq3TEyJQEQmisjp7utMEcnxNixjPGBNQybgxK9B50TkCuC3wMPuomJ6uczTmLhkTUMm4NTjXoJYagRXAScBdQCquhEY5WVQxnjDagTGRBNLImh1h4gAQERSsOvxTKDYf1cTbPHQR7BURG4CMkXkDOA3wEvehmWMB6yPwAScn1cN3QhUAauAf8O5HPT/eROOMR6wPgIzRHg16NwB7yNQ1TDwC/dhTIBZjcAEk9e/ZQ6YCETkI6I0sqrqYZ5EZMyAsxqBGRq8ahqK5c7isojXGcCXgDxvwjHGAzZDmQk43y8fVdWaiMfHqnovcJqnURnjCUsEJth8G3RORI6JeJuEU0OwO4tNgFjTkAk23/sIgJ9EvA4BW4AvexKNMV6ypiETcL71Eajqp705tTGDxCoEJuB8G4ZaRK7ra0dV/enAh2OMl6xGYILNj/sIrB/ADBFWJTDB5vV8Xb0mAlW91dMzGzNY7PJRM0T41kcgIhnAN4DpOPcRAKCql3sTkjFesURggsnrOm0sYw09CYwBzgSW4sxHUO9lUMYMLGsaMsEWD6OPTlbVHwCNqvoEcDZwVCwHF5G5IrJBRDaJyI19bHeciHSIyIWxhW3MQbAKgQk432YoA9rd5z0iMgMYDkw60E4ikgw8CJwFHAlcJCJH9rLdncCrMcZsTP/Y6KMm6OKgRrBQREYCPwAWAWtxCu4DmQNsUtXN7sQ2zwLnRtnu28DzQGVsIRtzsKxKYILNtyEmgF+qagdO/0B/RhwtAsoj3lcAx0duICJFwPk4Yxcd19uBRGQ+MB9gwoQJ/QjBGLA+AhN0vg86B3wkIgtF5DPSvwaqaNt2/2vuBf7DTTS9UtWFqlqmqmWFhYX9CMGYCHb5qAk4P2comwL8CWcS+y0i8oCInBzDfhXA+Ij3xcD2btuUAc+KyBbgQuAhETkvhmMbEzvrIzAB5/tVQ6rarKrPqeoXgdlALk4z0YEsA0pFpERE0oB5OH0MkccuUdVJqjoJ+C3wLVX9Xf/+BGMOpPNbZDUCE2xe/Q+OpUaAiJwqIg8B7+HcVHbA0UdVNQRcjXM10DrgOVVdIyJXisiVhxCzMQfHmoZMQPk26Fwnd6rKlcBzwA2q2hjrwVV1Mc5k95HLFvSy7aWxHteYfrGmITNEeHUfQSxXDc1S1TpPzm7MoLIagQmmsPtjxrfOYksCJvisRmCCLRx2/g8n+XhnsTFDg/URmIDqcGsEKUk+JQJ3CAhjgsv6CEzAhTrcGoFfiQDYJCJ3RxsnyJhgsRqBCaaw3zUCYCbwAfCIiLwrIvNFJNeTaIzxhNUITLCF3D6CZL8SgarWq+ovVPUTwPeAm4EdIvKEiEz2JCpjBpLNUGYCLux3IhCRZBH5goi8CNwH/ARn8LmX6HaPgDHxzRKBCaauGoGP9xFsBP4M3K2q70Qs/62InOJJVMYMKGsaMsHW2VmckuzNhZ6xJIJLVPXtyAUicpKq/kVVr/EkKmO8YE1DJqDaOsIApKV4kwhiOerPoiy7f6ADMcYzdvmoCbh2NxGkJg9y05CInAh8AigUkesiVuUCdm+BCSCrEZhgags5iSDdoxpBX01DaUC2u01OxPI6nLkDjAkIqxGYYNtXIxjkRKCqS4GlIvK4qm715OzGDAa7fNQEXGeNYNATgYjcq6rXAg+ISI+fVKr6BU8iMsYzlghMMDW0hgDITo/l+p7+6+uoT7rPP/bkzMYMGmsaMsHWmQhyMgY5EajqCnfAuStU9WuenN2YwWRNQyag6lucRDDMoxpBnw1OqtqBc9VQmidnN2YwWIXABFxDa4iM1KTB7yOIsAX4i4gsArqmqVTVn3oSkTGesRqBCab6lhA5GameHT+WRLDdfSSx/2WkxgSEVQlMsNW3tJPjUbMQxJAIVPXWgz24iMzFGaguGXhEVe/otv5c4IdAGAgB13YfzsKYAWN9BCagGlpDZHvUUQwxJAIRKcQZfno6kNG5XFVPO8B+ycCDwBlABbBMRBap6tqIzV4DFqmqishM4Dlgar//CmP6YkNMmIBraAl5dsUQxDbW0NPAeqAEuBWnz2BZDPvNATap6mZVbQOeBc6N3EBVG1S7vqXDsDq88UTnfyurEZhgamgNeXYPAcSWCPJV9VGgXVWXqurlwAkx7FcElEe8r3CX7UdEzheR9cDLwOXRDuTOirZcRJZXVVXFcGpjorA8YAKqviVEdrp3ncWxJIJ293mHiJwtIkcDxTHsF+1rF+0O5RdVdSpwHk5/Qc+dVBeqapmqlhUWFsZwamMiWNOQCbj6lnZPm4ZiOfJtIjIcuB5n+Olc4Dsx7FcBjI94X4xz9VFUqvqmiBwuIgWqWh3D8Y3pJ6sSmOBRVRpave0jiOWqoT+4L/cCn+7HsZcBpSJSAnwMzAMujtzAnfP4Q7ez+BicEU9r+nEOY2JgNQITXE1tHYTVu3GGoO9B5+6nj2/QgWYnU9WQiFwNvIpz+ehjqrpGRK501y8ALgAuEZF2oBn4SkTnsTEDyy4fNQFU3dAKQEF2umfn6CvFLD/Ug6vqYrpNcO8mgM7XdwJ3Hup5jOmT/bYwAbZzbwsAo3J9SASq+oRnZzXGF1YjMMGzfW8zAGOHZ3p2jgPORyAiLxH9ah+bj8AEhNUITHCV1zqJoHikD4kAm4/ADBU2Q5kJsPLaJkblpJOR6t1U8X3OR+A+L/Xs7MYMKksEJng+qm6kpGCYp+c44A1lInKOiPxDRGpFpE5E6kWkztOojBlQ1jRkgklV+bCqgcMKvU0EsVyYei/wRWCVXdppAs2ahkzA7KprZXdTO1NGezsDQCxDTJQDqy0JmMCy/7omoFZ9vBeAo4qHe3qeWGoE3wMWi8hSoLVzoc1QZoLHagQmWFZV7CFJ4Mix/ieCHwENOHMR2NzFJoCsRmCCaWXFXo4YnUNmmndXDEFsiSBPVT/raRTGeMkuHzUBFOoI897W3Zx39DjPzxVLH8GfRMQSgTHGDKL3tu2hoTXESYcXeH6uWBLBVcASEWm2y0dNMNkMZSZ4Xl9fSUqScFKp94kglmGovb1uyZjBYk1DJiBUlT+u2cmckjxyM7ybmaxTX2MNTVXV9e48AT2o6nvehWXMALLLR03ArPp4L5urG7nilMMG5Xx91QiuA+YDP4myToHTPInIGM9YjcAEwwvvfUxachKfmzF2UM7X11hD893n/sxKZkwcshqBCY7G1hDPr6hg7owxDM/yvlkI+ugsFpHjRGRMxPtLROT3IvIzEckblOiMGUjWR2AC4NfLyqlvDXHpSZMG7Zx9XTX0MNAGICKnAHcA/4szd/FC70MzZoBYH4EJiJb2DhYs/ZA5JXkcM2HkoJ23rz6CZFWtdV9/BVioqs8Dz4vISs8jM2bAWY3AxLfH39lCZX0r9807elDP21eNIFlEOhPFZ4DXI9bFckeyMfHFmoZMHNu+p5n7X9vI6dNGceLh+YN67r4SwTPAUhH5PdAMvAUgIpNxmocOSETmisgGEdkkIjdGWf9VEfmn+3hHRGYdxN9gTN+sacjEOVXlphdXEVa4+fPTB/38fV019CMReQ0YC/wxYhjqJODbBzqwiCQDDwJnABXAMhFZpKprIzb7CDhVVXeLyFk4fQ/HH9yfYsyBWI3AxKen/raNNzZUcfPnj2R8Xtagn7/PJh5VfTfKsg9iPPYcYJOqbgYQkWeBc4GuRKCq70Rs/y5QHOOxjekHqxGY+PXu5hp++NJaTj2ikH85cZIvMcQy1tDBKsKZ1KZThbusN98AXom2QkTmi8hyEVleVVU1gCGahGJ9BCbOrN1exxVPLGdCfhb3zZtNUpI//0e9TATR/qKoP81E5NM4ieA/oq1X1YWqWqaqZYWFhQMYokkI1kdg4lB5bRP/8su/k52Rwv9ePocRWf5N9+Ll1T8VwPiI98XA9u4bichM4BHgLFWt8TAeY4yJC1trGvnao3+jLRTmV1eeyLgRmb7G42WNYBlQKiIlIpIGzAMWRW4gIhOAF4Cv96PvwZj+0bDzbE1DJg6sLN/DBT9/h/qWEE9+Yw6lHk9MHwvPagSqGhKRq4FXgWTgMVVdIyJXuusXAP8F5AMPifMlDalqmVcxmQQVDjnPSXb7i/HXb1dUcNOLqxidm84vL53D5FHZfocEeHxjmKouBhZ3W7Yg4vW/Av/qZQzGWCIwfmtsDXHzojX8dkUFJx6WzwMXH01+drrfYXWxb4YZ+rTDebZEYHzw7uYa/uP5f1Je28Q1nynlmtMmk5LsZat8/9k3wwx9YUsEZvDtbW7nriXrefpv25iQl8UzV5zA8YcN7tARsbJvhhn6upqGkv2NwySEjrDy3PJyfvzqBnY3tfGNk0u4/rNHkJUWv8Vt/EZmzECxPgIzCFSVP62r5K4l69lY2UDZxJE88YU5zCga7ndoB2TfDDP0WSIwHlJVXl9fyc9e28j7FXspKRjGQ189hrNmjEECcsmyfTPM0Gd9BMYDbaEwL72/nV+8tZn1O+spHpnJnRccxQXHFMddZ/CB2DfDDH2dNQIJ1pfTxKc9TW088/dyHn/nI3bVtXLE6GzuvnAm5x1dRGrAEkAnSwRm6AuHQJLtzmJz0FSVleV7+NXftvHSP7fT0h7m5MkF3HnBTE49ojAwTUC9sURghr5wyJqFzEGprG9h0crt/GZ5BRt21ZOVlsz5RxdxyYmTmDY21+/wBox9O8zQF+6wRGBiVlXfyqtrdrJk9U7e+bCasMKs8SO4/fyj+PysseRkpPod4oCzb4cZ+iwRmAPYsbeZJat38srqnSzbUosqHFYwjG99ajLnHV0UN2MCecW+HWboC4fsZjLTw7aaJpas2cErq3fyj217AJgyOodrTivlc0eN5YjR2YFv+4+VJQIz9FkiMEBLewfLttSydEMVb3xQxabKBgCmj8vlhjOnMHfGGA4vHNq//HtjicAMfR1tkOzf7E/GH+GwsnZHHX/ZVM3bm6r5+0e1tIbCpCUnMackj3nHjeezR45hQv7gTxYfbywRmKEv1Aop8TPkr/FGqCPMmu11LNtSy98+quXvH9Wyt7kdgMmjsrn4+Al8srSAEw7Lj+txf/xgn4YZ+jpaIdkSwVCzt7mdleV7eG/rblZs3c1723bT1ObcRT4hL4vPHjmaEw/P56TJBYzOzfA52vhmicAMfaE2SLGmoSBrC4VZv7OO98v3sLJ8LyvLd/NhVSMASQJTxuTypWOLOa4kj+Mm5VnB30+WCMzQF2qxGkGANLd1sH5nHWu217H6472s3r6XDTvrae9QAPKHpTF7/AjOm13E0RNGMmv88CF5bf9gskRghr6ONki1X4jxRlWp2N3M+p31bNhZx7qd9azfUcdH1Y2EnTKfEVmpTB+Xy+UnlzCzaAQzi4dTPDIzYS7rHCyeJgIRmQvchzN5/SOqeke39VOBXwLHAN9X1R97GY9JUO3NkJ7jdxQJKxxWtu9tZmNlA5t2NbCxsp4PdjWwqbKBhtZQ13YT8rKYMiaHs2eO48ixuUwfl2uF/iDxLBGISDLwIHAGUAEsE5FFqro2YrNa4BrgPK/iMIb2ZkjN9DuKIa++pZ0t1U1srm7gw6pGNlc1sLmqkY+qG2lu7+jariA7jdJROXzxmCKmjsll6tgcjhidQ3a6NVD4xctPfg6wSVU3A4jIs8C5QFciUNVKoFJEzvYwDpPo2hohzWoEA6G+pZ2tNU1srWliS00jW2sa3cK/keqG1q7tkgSKR2ZRUjCMEw7LZ/Ko7K5H3jDruI83XiaCIqA84n0FcPzBHEhE5gPzASZMmHDokZnE0lYPacP8jiIQOsLKjr3NlNc2U7G7ifLaJrbWOgX/ttomahvb9tu+MCedSflZnDa1kJKCbEoKsigpyGZifhYZqXY3d1B4mQiiNezpwRxIVRcCCwHKysoO6hgmQak6NYL0xBw6oLu2UJjK+hY+3t1Mxe5mPt7TzMe7mynf3UTF7ma272kmFN73FUsSGDs8k4n5WZw5fQwT87OYmJfFhPwsJuUPY5g15wwJXv4rVgDjI94XA9s9PJ8xPbU1OmMNpQ+dseN709AaYufeFudR18KuuhZ27G1m595WdtY5z5HNN50KstMZn5fJrPEjOGfmWMbnZTF+ZBbj8zIZOzyTtJRgzrplYudlIlgGlIpICfAxMA+42MPzGdNTyx7nOXOkr2EcinBYqW1qi1LIRzzvbaE+4gqcTiOzUhmdm8HY4RkcVTSc0bkZjMnNoGhkJkUjMhk3ItOacIx3iUBVQyJyNfAqzuWjj6nqGhG50l2/QETGAMuBXCAsItcCR6pqnVdxmQTTvNt5zhzhaxi96Wyq6SzQoxX2lXWttHWE99svSWB0bgajczOYXJjNyZMLGDPcKfA7C/7RuRlWyJuYeNrAp6qLgcXdli2IeL0Tp8nIGG/srXCecwf3v1k4rOxuaqOyvpVddS1U1rdSVd/a1VTTWcjXNLai3Xq9MlKTGJObwZjhGZRNHMmY4ZmMyU13nt3CviA7neQku77eDAzr6TFD2+6tzvPIiQNyuMbWEFX1rVQ1OAX7fo+IZdUNrft1unYakZXaVchPH5fLmOEZXe/HDM9gbG4muZkpdhOVGVSWCMzQtnsLpA6DrPxeN2lsDVHd4BTeVfVtEa8jn53lnaNbRkpOEgqy0yjMSacwO51pY3MoyE5nVE46o3IznOecDEblpltTjYlLlgjMkKOq1LU4hXvexxtJzhzHi3/d6hbw+wr66oZWquvb9rvrNVLesDQKstMoyE5n9vgRTuGe6xT2hTn7HnlZaSRZM40JMEsEJhA63Db3zsJ7X2HeRo37uqaxjer6Vqob22gLOZ2rr6RtoEILuXnRGpKks3BPpyA7nYkTsijITiffLdg7C31nWRqpyXbZpEkMlgiMb1raO7oK75pGt4Bv3FfQdy6raWyltrGNKE3upCYL+cOcgrsgO53Jo7IpdAvzguxUShfXMnrqGSw783TyhqVZB6sxUVgiMAOqqS1EdX0bVd2aXzpf17hNM1UNrdS39LzuHWBYWjL52c4v9In5WRwzcSSF2Wnusn2FfmF2et8dq401sKiRvKJSyLH5CIzpjSUCc0DNbR1dhff+Haj7d6RW1UfvTAXnapkCt3CfNi6XU9zX+V2/3vc1y2SmDVCHau2HzvPISQNzPGOGKEsECSrUEaamsc25xr2ulV31Ld3a3vcV8g1R7lgF567Vwpx08oelM6vY6Uztamt3r6ApyE4nb1iaP8MUbPur81x07OCf25gAsUQwxHSElZrGVqdwr2thV12re+dqK5V1Leyqdwr+6obWqG3unb/c84elMaNoeFfhHnmlTGA6UzcvhfxSyBntdyTGxDVLBAERDuu+X/BuYb7L/SVf6d65uquuheqGNjqilPD5w9IYlZvB6Nx0jhyby+jcDOd9xLXuBdnpQ2eAsYZK+GgpnPAtvyMxJu5ZIvCZqrKnqd0ZOMwt1Hv8iq/r/U7VvGFpjMpJZ3RuBlNG57jjz6RTmOM8j87NGFoFfKzeuR80DMde6nckxsQ9SwQeU1WqGlq7JvqocMeBr9jdxPY9zWzf0xL1hqbOUSMLc9IpHZ3TVah33q06OjeDwkQs4GPRVAvLHoUZF0D+4X5HY0zcs0QwAMJh5eM9zWyubmRLdaM7m1Mj22qbKK9t7lHQ5w9Lo2hkJqWjcvjUlFGMHZ7BuBGZEb/m00lPsaEIDtob/wPtTfDJ6/2OxJhAsETQD6pKeW0z63fWsbGygY276tlY2cCHVQ20tO8bJjgzNZkJeVlMzB/GJ0sLmZDnTPIxfmQWRSMzyUqzj90z//wN/H0hHH8ljJrmdzTGBIKVSH3Y29zOyvI9rNi6m39s283K8j373QQ1bngGk0fncKI7OfdhhdlMys+iMCfdRo8cbKpOAlhyI0w8Gc74b78jMiYwLBFEaAuFeefDal5fX8m7m2vYWNmAqjMJyJQxuXx+1jhmjBvO1LE5lI7KJicj1e+QDcDOVfDn/4ENL8MRZ8GFj0KK3UlsTKwSPhG0d4R5fX0li1ft4PV1ldS3hshMTea4kjzOmTmOYyeOZNb4EWTbJN3xJdwB61+Gvz0MW9+G1Cw444dw4tWQZB3oxvRHwpZu4bDy0j+3c9eSDXy8p5mRWanMnTGGM6eP4eTSAhs3Ph411ULFMvjwdScJ7C2H4ROcBHDM1wM9L7ExfkrIRLB8Sy0/fHkd75fvYUZRLrd8YTqfnlJISrzfKZsIVJ0J5/dWQPVGqNkEletgx/v7xg5KyYBJn4Qzb4cpn4PkhPxvbMyASahvkKpyxyvrefjNzYzJzeAnX5rF+UcX2aQiXutoh/Zm55LOxmpoqobmPdBYBQ27oKkG6nbAnm3Oo61+//1HTIAxM2H2xTD+eCgug9RMX/4UY4YiTxOBiMwF7gOSgUdU9Y5u68Vd/zmgCbhUVd/zKp7H/rKFh9/czNdOmMD3P3fkwI1yGWSqzh247c3Q1uAU1u0t0NEKoYhHWwO01jvrO9qcwj0ccp/boXm3U7i31juPlr3OL/uWvc72vZEkp0knezSMmAiTToYR4yG3CApKIe9wSMsarE/DmITkWSIQkWTgQeAMoAJYJiKLVHVtxGZnAaXu43jg5+7zgGtqC/Hwn1Zx9uQsfnj6GKStFlrDgO4rDHGf93vfy/qWvc6j+z49jqMR77sfO+y0ezfVQFujU+B2bRclHnXvVejzfGEnrlCrW5i3OQVxOOR0sGpHxHNo3zEPliRBUipkDIesPEjPcR7DiyAzz1melu38gk/NgKwCGFYAGSNgWKHzOskSsjF+8rJGMAfYpKqbAUTkWeBcIDIRnAv8r6oq8K6IjBCRsaq6Y6CD2fbWM7yt/05aRQf8eKCPfoiS0yEj12n7liT3IfteI92WS7fl3bbNynd+RSenO5dRJqdCUorzkGTnqpqu18nOc0q6U4CnZjkFdue+KRnOc9owp0BPGwbJae4xU+0KHWOGAC8TQRFQHvG+gp6/9qNtUwTslwhEZD4wH2DChAkHFUzzuDksz/40R59wGplpqfsK1O6FaZ/vO7cXpyAdNgpS0uhZUEcrpHs5V0q686vYGGN84mUiiNYD2334zFi2QVUXAgsBysrKooyif2BHT5sC054/mF2NMWZI87JeXwGMj3hfDGw/iG2MMcZ4yMtEsAwoFZESEUkD5gGLum2zCLhEHCcAe73oHzDGGNM7z5qGVDUkIlcDr+JcPvqYqq4RkSvd9QuAxTiXjm7CuXz0Mq/iMcYYE52n9xGo6mKcwj5y2YKI1wpc5WUMxhhj+mbX/hljTIKzRGCMMQnOEoExxiQ4SwTGGJPgxOmvDQ4RqQK2HuTuBUD1AIbjhSDECMGI02IcGBbjwPA7xomqWhhtReASwaEQkeWqWuZ3HH0JQowQjDgtxoFhMQ6MeI7RmoaMMSbBWSIwxpgEl2iJYKHfAcQgCDFCMOK0GAeGxTgw4jbGhOojMMYY01Oi1QiMMcZ0Y4nAGGMSXMIkAhGZKyIbRGSTiNzodzzdichjIlIpIqv9jqU3IjJeRP4sIutEZI2I/LvfMXUnIhki8ncRed+N8Va/Y+qNiCSLyD9E5A9+xxKNiGwRkVUislJElvsdTzTu9La/FZH17v/LE/2OKZKITHE/v85HnYhc63dc3SVEH4GIJAMfAGfgTIazDLhIVdf2ueMgEpFTgAacOZxn+B1PNCIyFhirqu+JSA6wAjgvzj5HAYapaoOIpAJvA/+uqu/6HFoPInIdUAbkquo5fsfTnYhsAcpUNW5v1BKRJ4C3VPURd96TLFXd43NYUbnl0MfA8ap6sDfFeiJRagRzgE2qullV24BngXN9jmk/qvomUOt3HH1R1R2q+p77uh5YhzPHdNxQR4P7NtV9xN2vHREpBs4GHvE7lqASkVzgFOBRAFVti9ck4PoM8GG8JQFInERQBJRHvK8gzgqwoBGRScDRwN98DqUHt8llJVAJ/J+qxl2MwL3A94Cwz3H0RYE/isgKEZnvdzBRHAZUAb90m9geEZFhfgfVh3nAM34HEU2iJAKJsizufiUGhYhkA88D16pqnd/xdKeqHao6G2cO7DkiEldNbSJyDlCpqiv8juUATlLVY4CzgKvc5st4kgIcA/xcVY8GGoG46/8DcJutvgD8xu9YokmURFABjI94Xwxs9ymWQHPb3Z8HnlbVF/yOpy9uM8EbwFx/I+nhJOALbhv8s8BpIvKUvyH1pKrb3edK4EWcJtZ4UgFURNT4fouTGOLRWcB7qrrL70CiSZREsAwoFZESNzPPAxb5HFPguB2xjwLrVPWnfscTjYgUisgI93UmcDqw3tegulHV/1TVYlWdhPN/8XVV/ZrPYe1HRIa5FwTgNrd8FoirK9pUdSdQLiJT3EWfAeLmwoVuLiJOm4XA4zmL44WqhkTkauBVIBl4TFXX+BzWfkTkGeBTQIGIVAA3q+qj/kbVw0nA14FVbhs8wE3u3NTxYizwhHuFRhLwnKrG5eWZcW408KKT+0kBfqWqS/wNKapvA0+7P/A2A5f5HE8PIpKFc8Xiv/kdS28S4vJRY4wxvUuUpiFjjDG9sERgjDEJzhKBMcYkOEsExhiT4CwRGGNMHOvvgJQi8mURWesOuvirWPaxRGAGlYh83/0P+k93NMbj+7n/pSIyrp/7TOrPqK4isrjzXoQYt/9C54i2InKLiHy3n/FF7n+eiBzZz/3/W0RO7+c+W0SkoD/7GN88Tow3RYpIKfCfOHeFTweujWW/hLiPwMQHd4jgc4BjVLXVLYjS+rF/MnApzo1Nnt0Zrqqf6+f2izjIGxRFJKXb/ucBf6AfN0ap6n8dzLlNMKjqm+7YXl1E5HDgQaAQaAKuUNX1wBXAg6q62923MpZzWI3ADKaxQLWqtgKoanXnMAYi8hl34LBVblU43V2+RUT+S0Texrk7swznBqKVIpIpIseKyFJ3YLRX3aGycZe/LyJ/Ba6KFoyIjBWRN91jrRaRT0acs8CtSax3BzNbLSJPi8jpIvIXEdkoInPc7S8VkQeiHP8KEVnmxvG8e2MRIvK4iPxURP4M3Nm5v4h8Amc8mrvdmA4XkfcijlcqIj3GJ3KPd2FE7LeKyHvuZznVXZ4vIn90P+OHiRh/S0S+Js4cDitF5GFxBu07zq21Zbh3Ga+ROBuzKcEtBL6tqscC3wUecpcfARzh/h99V0RiqklYIjCD6Y/AeBH5QEQeEpFTwZlMBqf6+xVVPQqnpvrNiP1aVPVkVX0KWA581R1ULgTcD1zofiEeA37k7vNL4BpV7WuikouBV91jzQJWRtlmMnAfMBOY6u5zMs6X76YD/L0vqOpxqjoLZ8jub0SsOwI4XVWv71ygqu/g1AxuUNXZqvohsFdEZrubXIbzOR1ItTtY3M/dOAFuBt52B2dbBEwAEJFpwFdwmhJmAx04n+8yd7vbgLuAp1Q1roaYSFTiDPr4CeA34tzh/zDOjyxwvjulOKMUXAQ8EkszpzUNmUHjThZzLPBJ4NPAr9228X8AH6nqB+6mT+D8ir/Xff/rXg45BZgB/J84QyEkAztEZDgwQlWXuts9iTPoV3fLgMfEGUjvd6q6Mso2H6nqKgARWQO8pqoqIquASQf4k2eIyG3ACCAbZ4iTTr9R1Y4D7A/OfAWXiTOJzVeIbeC3zsEAVwBfdF+f0vlaVV8Wkd3u8s8AxwLL3M8wE2f4boD/xvmMWoBrYjivGRxJwB43cXdXAbyrqu3ARyKyAScxLDvQAY0ZNO4Q0W+o6s3A1cAFRB8mPFJjL8sFWOP+ep6tqkep6mfd5QccO8WdDOgUnFmjnhSRS6Js1hrxOhzxPsyBf0g9Dlzt1nJuBTIi1vX2N3X3PE4SOwdYoao1MezTGWNHtxijfSYCPBHxGU5R1VvcdXk4CSynW+zGR+7Q7x+JyJfAGQxSRGa5q3+H8yMLtw/uCJwxmPpkicAMGnHmby2NWDQb2IozOugkEZnsLv86sJTo6nEKJoANQKHbCY2IpIrIdHf46b0icrK73Vd7iWcizrwAv8AZVXWghzDOwamhpPYWQxSRfx+q2oJTk/g5TnPXwXqzMwYROQsY6S5/DbhQREa56/LczwWcdugfAE8Ddx7Cuc0hEGdAyr8CU0SkQkS+gfNv+Q0ReR9Yw74ZF18FakRkLfBnnGbGA/54sKYhM5iygfvdNssQsAmYr6otInIZTptnCk41dkEvx3gcWCAizcCJwIXAz9zmoBSc5qQ1OO3pj4lIE/s3yUT6FHCDiLTjzBcdrUZwKH6AM4PbVmAVEQV8H54FfiEi1+D0fXyIUxB/EaeP5WDdCjzjdj4vBbYBqOpaEfl/ODORJQHtOJPQnAqEVPVX4lyt9Y6InKaqrx9CDOYgqOpFvazq0RGsziii17mPmNnoo8bEOXHuSxiuqj/wOxYzNFmNwJg4JiIvAocDp/kdixm6rEZgjDEJzjqLjTEmwVkiMMaYBGeJwBhjEpwlAmOMSXCWCIwxJsH9fwsvKPdw1fL8AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as pyplot\n",
    "\n",
    "\n",
    "pyplot.plot(W_sparse_CF_sorted, label = 'KNN CF')\n",
    "pyplot.plot(W_sparse_CBF_sorted, label = 'KNN CBF')\n",
    "pyplot.ylabel('Similarity value')\n",
    "pyplot.xlabel('Sorted similarity index')\n",
    "pyplot.legend()\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Let's now plot the delta for the common values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get common structure\n",
    "W_sparse_CF_structure = W_sparse_CF.copy()\n",
    "W_sparse_CF_structure.data = np.ones_like(W_sparse_CF_structure.data)\n",
    "\n",
    "W_sparse_CBF_structure = W_sparse_CBF.copy()\n",
    "W_sparse_CBF_structure.data = np.ones_like(W_sparse_CBF_structure.data)\n",
    "\n",
    "W_sparse_common = W_sparse_CF_structure.multiply(W_sparse_CBF_structure)\n",
    "\n",
    "# Get values of both in common structure of CF\n",
    "W_sparse_delta = W_sparse_CBF.copy().multiply(W_sparse_common)\n",
    "W_sparse_delta -= W_sparse_CF.copy().multiply(W_sparse_common)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "W_sparse_delta_sorted = np.sort(W_sparse_delta.data.copy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgwAAAEWCAYAAAAKI89vAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA4a0lEQVR4nO3dd5xddZ3/8dc7bSa99wRCrwpCABFFRFBBBX+urtgorrK41sVVsawLll2sa11psiCwIHZ0cUVlAYFFmnQIhJaE1EmbSZlJZubz++P7vcnN5U7JZHLv3Jv38/G4j3vuqZ976ud8z/eco4jAzMzMrDuDqh2AmZmZDXxOGMzMzKxHThjMzMysR04YzMzMrEdOGMzMzKxHThjMzMysRzs9YZB0nKRF/Ti+WyS9v7/GN9BJmiMpJA2pchzrJO3Zh+E+K+mynRFTvZL0Kknz+jjsbnlZDc6/d2h7kfSopOO66f47SWf0dfw7m6QzJd2+s/q3rhWve5LeLemmasdkO6bHhEHSc5I2SmqRtEbSnZLOkdSnZCOP74S+DLujaiHZqOb86U5EjIqIZ7rrp1xyGBH/GhE7ZZ7nRGrvot//JGmJpINyLCHpByXD3C7pzNx8Zu7nkyX9LOrhIHmkpBvz9rBK0t2SzsrdjpPUmQ/ahc9vtud/RcSfI2K/7RmmaNgFeVl19GX4MuM7KCJuAZB0vqSrS7qfFBFX9se0Km2gJOO7goi4JiJetzPGXbrPlHSapNWSXl20jP+7ZJirJZ2fm3vcV3Qx3X0l/VRSk6S1kh6SdK6kwUXTLewDlkn6D0lDS+LeWLKvmNFf82Vn6O1B/80RMRrYHbgQ+DTwo50Wldl2kvR54OPAqyPi0dx6PXC6pDndDLoK+LSkMb2cztHAzcCtwN7AROCDwElFvS3OB+3C583b9WcGAB9EBwYlg0raedl0IZd2/QB4Y0TcWtTp5ZKO6WbQ3uwriqezF/AXYCHwkogYC7wdmAuMLup1XESMAl4CHA18qGRUby7ZVyzuzfSrZbtKCSJibUTcALwDOEPSwQCSGiR9Q9KCnEldJGl46fCSrgJ2A36Ts6lP5fY/lbQ0Z2m3STqoh1D2ymd1ayX9WtKEomm8PJeCrJH0YOFMUdJXgFcB38/T/r6kCyR9L3cfKmm9pK/l38MltUoa3914c7exkn6Uz25fkPRlbS0SPjNnqt/IWe+zkooPLj3On+zdef42Sfpc0TCDJJ0n6WlJKyVdXzw/ykzjTZIe0NbSopfm9u+Q9EzhwCnppLxMJuffW87mJZ0s6TGlUqcXlM7sRwK/A2YUZ8sqOistyrrP6OK/DJd0ZZ5Pj0v6lHpxOUvSl4H3A8dGxJNFndYAVwD/0s3gjwP/B/xjT9PJvg5cGRFfjYimSO6LiL/t5fDFcb9oPub225TUKJ2JfFLpDGZ9XtemKl0OaJH0x6L1tMuzZkl7Sbo5rydNkq6RNK5kOp+W9BCwXtKQ3O4ESW8APgu8Iy/bB/Mw25TaSXpfXnarJf1e0u65vST9u6Tl2no2dnAX8+WsPI6WvE7+fVG345RKgD6Rx7VEuXQnd58o6QZJzZLuBvbqZhHclr/X5P90dNF4ym6v6mZbL/M/Bitdkns6/5f7JM3O3V4h6Z48L+6R9Iqi4W6R9BVJdwAbgD3zMv2QpKeAp3J/Zbfl3O3TOb4WSfMkvTa3P1/SzyT9JHe7X9IhRcMdkKe/Ruly1ClF3a6Q9ANJ/52H/YvSgbPQ/URJT+T/9H1ARd22udST/885kp7K8/kHklQ0376Z19FnJX24q3W6ZH6fDXwTeH1E3FnS+WvAl7sZfA097yuKXQDcGRHnRsQSgIiYFxHviog1pT1HxHLgD8CBvRz/NiR9QNJ8pRLNG1RUEtHdvMzdd3ibLP4j3X6A54ATyrRfAHwwN38buAGYQMqufgP8W+52HLCou/EB78vDNeRxPdBNPLcALwAHAyOBnwNX524zgZXAyaRk6MT8e3LRsO8vGtfxwMO5+RXA08Bfiro92Mvx/gq4OMczBbgb+Pvc7UxgM/ABYDDpbHQxoN7Mb2AOEMClwHDgEKANOCB3/zhwFzArz7+LgWu7GPdhwHLgqBzLGXl6Dbn7NaSNZmKO8U1Fwwawd25eArwqN48HDiu3rHO784uWT0//5ULSmfv4/H8eKh1fybgD+BlpB7pbSbfjgEXANKAZ2C+3vx04s2jZ3A4cStphTMjtFwHHlZneCKADeE03Mb1oHnTTb6/mY15GdwFTSevicuB+4GV5md8M/EvJPB5Sus6TSkROzMNMJh0wv10ynQeA2cDw0vWxeFmWbI+F8b8FmA8cAAwBPk/aqQK8HrgPGEc6kBwATO9ivryRdKAX8GrSQbN43rQDXwSGkrbJDcD43P064HrStngwaV9xexfT2WZe9WZ7pZttvcz4Pwk8DOyX/8shpG1rArAaeG+eT+/MvycWzdMFwEG5+9Ac5x/ysMPpZlvO01sIzCj6n3sVLcPNwNvyeP8JeDY3D83L77PAMNI+sIWt284VpBK5I3Nc1wDX5W6TSNtZYbz/mJfT+4vm6+1F8yaA35LWh92AFcAbcrdzgMdI+4DxwB9Ll1OZfebPgWXAIV0s41F5XSisy1cD5/d2X1FmmkuBs7rZtgvTLWyHM4AHgfd1ta/vZlzHA015mTcA3wNu6+W8fAv9sE1umVYvgi37p0g7sM/lCa0nr5C529HAs93s/LqcSTn4AMZ20f0W4MKi3wcCm0gbzaeBq0r6/z1wRunOLf8eDrSSNuLzSBvKorxyXQB8N/fX5XhJO/E28g42d3sn8L9FG8r8om4j8v+b1pv5XbTizSpqdzdwWm5+HHhtUbfppB3CizYu4IfAl0razSMV4xfm/QLSTu7ikv6KE4YFwN8DY0r62WZZx9YdVGnC0NV/eYZ0dlDo9v7S8ZWJqRn4XpluW2IhnV38JEp2AhTtxEgHma/m5q4Shpl5mvt3E9NxQCcpASl8/raLfns1H/M68e6i3z8Hflj0+yPAr0rm8YsShjLTfwvw15LpvK+kn+fofcLwO+DviroNIh3Mdyft9J4EXg4M6mr+dRHnr4CPFc2bjWx7kF+exzuYtO7vX9TtX9n+hKHs9koP23qZ8c8DTi3T/r3A3SXt/o+t6+UtwBfLrOvH92ZbJiWGy4ETgKFltse7SpbRElLp66tIB8JBRd2vZeuB9QrgsqJuJwNP5ObTS8Yr0nbUXcLwyqLf1wPn5eabKUrC8v/oKWFoBn5dum4VL2PgHwoxUiZhiG72FWWmuZl8UO5h3VqTPwHcSdG2nuNeV9TPr7oY14+ArxX9HpWnP6cX87Jft8kduUtiJinbnEzaqO7LxVhrgP/J7XuUi58uzMV2zaSZCClj7crCoubnSRntJNJMeHshjhzLK0kH0ReJiI3AvaSN7FjS2e2dwDG5XeEaWHfj3T1Pf0lRt4tJZx8FS4umuSE3jurm/5WztKh5Q9HwuwO/LJr246Sz4KllxrE78ImS/zGblP0SqSjtp6Qzs292E8vfkHYWz0u6tbgodwf/ywy2XbbFzV05DXibpAu66eerwOuLi17L+ALwQUnTuulnNSkZKLs+FVkcEeOKPtd30d/2zMdlRc0by/zucX2SNEXSdbmoupm00yzdznozz7uyO/CdonVrFenAMTMibga+T7q+vEzSJeqi3ojS5bC7cvHrGtI8Ko5zZUS0F/0urEOTSQeG0v3D9upqe+3Ntl5sNqnUstSMMnE9T9qnFpRbDsXtutyWI2I+qeTxfGB5XuYzyo0nIjpJB/YZ+bMwt+sqrl5tu5GOTj2tS/25HzgH2Be4rLg4vsSlwFRJ3dUp6s2+AlIJc0/7AYBJETGOdIy8g3RsLPaWov3EW7oYxzbrS0Ssy9PvzXLpl22yoK93OhyRg72dVFSyETio6I+PjVTRo5wo+f0u4FRSFjmWlJlB0fWvMmYXNe9GyraaSCvWVSU765ERcWEX04aUFBxPKt69J/9+PanYrXCNs7vxLiSddUwq6jYmInqqh9GVcjF2ZyFwUklsjRHxQhf9fqWk3xERcS2ApENJl4euBb7bZYAR90TEqaQd5a9IGW1fYi+1hFQMWTC7qx6LPElad/5B0nnleoiIlaRLXV/qaiQR8QTwC1IpU1f9bCCdCf5NL+LqUTfzcWf5N9IyemlEjAHew4u3s+6WYU/LdyHpzLB4/Roe+XpyRHw3Ig4nFbXvSyqy34akBlIJyjeAqXlne2OZOMtZQSoGL90/9PX/lNrebX0h5etQLCbtyIvtRioy7y624nbdbssR8V8R8co8nSAdCAu2zB+lCpWzckyLgdnatpJlaVxdWVIyXtG77bercW3vfmA58FpSKcl/lOshIjaTSo6/RBfrU2/2Fdkf2Y79QD45vQI4WlJ3J8PlbLO+KNUXm0jvlssOb5PFtithkDRG0ptI1wmvjoiHczZ6KfDvkqbk/mZKen0Xo1kGFN/PP5q0Ea4kZWH/2otQ3iPpQEkjSNcyfxbpNrKrgTdLen0uuWhUqiRVWPlKpw0pQTgdeCwiNpGLWEmXVFbkfrocb6QKLzcB38zzZ5BS5bJX9+J/lFMuxu5cBHylqCLLZEmndtHvpcA5ko7KFV5GSnqjpNGSGvP//CxwFjBT0j+UjkDSMKV7qsfmDbCZVKJRiH2ipLHbEX+x64HPSBovaSbw4d4MFOmuiBOAT0r6eBe9fYtUT+WAbkZ1Aem/j+umn08BZypVQpwIIOkQSdf1JtaCHubjzjKaXASa52+3O4cylgFz1PUt1ReRlt9BsKWC4Ntz8xF5vRtKuoTZSvn/O4x0nXYF0K5U4bBXt+PlfcAvgPMljZB0IOmyYVdWkEqMerW99WFbvwz4kqR98vb20rzO3AjsK+ldShVL30G6tPrb3sSRdbct7yfp+Jx8tZJO6Irn9eGS3qpUifDjpP3vXaRa/+uBTylVAj8OeDNpf9+T/wYOKhrvR0mXcfrieuBj+TgyjnRJuEeR7jA4HniDpH/vorerSOvXG7oZVW/2Ff8CvELS15VLJSXtrXS75rjSnvOyeC+pJGBlT/+lxH8BZ0k6NI/nX0l17Z7rxbD9sU1u0duE4TeSWkjZyudIM/Ssou6fJlWsuEupqPOPpIo35fwb8PlcRPJPwI9JxS0vkCq63NWLeK4iZWtLgUbSyklELCSVVnyWtDNYSNopFv7nd0jF16slFc6g7yTVZSiUJjxGmnGF370Z7+mkHd1jpGLrn9G74qpySudPT75DqnB6U15Gd5EqQr1IRNxLqsz1/RznfNK1xcJ0F0XEDyOijXT2+WVJ+5QZ1XuB5/KyPif3WzhLvxZ4Jse/vfcUf5FUPPosaR36GWln1qOIeJBUMvQvks4p072ZdH2yyztIIuJZ0ro1spt+7iTtlI4n/c9VwCWkg8D2Kjsfd6ILSBWn1pJ28L/YzuF/mr9XSrq/tGNE/JJ0Jntd/k+PsPV20zGkg9xq0va+klSKUDqOFtL2fH3u912k9bu3Pkwqjl1K2kf8Z1c95hKjrwB35PX15b0Y//Zs698i/Y+bSAnhj0j1H1YCbwI+QZoPnyJVMG7qxfQLsXe3LTeQKhA3kebDFLYtOfs16U631aR18K0RsTmfMJ1CWmZNpDP10/N23VM8TaTbCi/M/2kfUhF8X1xKmmcPAX8lbVvt9CKhzvvq40n7+X8r072DdLDvbj/Qm33F06S6enOARyWtJZWM3UuqKFqwRtI6UrJ9NHBKvlzTaxHxJ+Cf8/iXkEqtTuvlsDu8TRbTdsZuVjGSPkiqENnX0hozK6L0sKK9I2JnJ6f9JpcyXRQRpZdxrML8LgkbMCRNl3RMLurdj3QG9stqx2VmlaP0PJaT8+WamaQSAe8HBgAnDDaQDCPVOm8h3Vr1a7qowGRmdUuky2erSZckHifdxWRV5ksSZmZm1iOXMJiZmVmP/BKTKpo0aVLMmTOn2mGYmdWU++67rykievVwQOs/ThiqaM6cOdx7773VDsPMrKZI6ssTPG0H+ZKEmZmZ9cgJg5mZmfXICYOZmZn1yAmDmZmZ9cgJQy9JeoOkeZLmq4u3Iub+jpDUIeltlYzPzMxsZ3LC0AuSBpPeGX4S6a1y78xvwivX31eB31c2QjMzs53LCUPvHAnMj4hn8hvdriO9vbLUR0hvFFteyeDMzMx2NicMvTOT9ErrgkW53Rb5JSn/j/T+cTMzK2Phqg1866Z5LFi5odqh2HZywtA7KtOu9CUc3wY+nd+33vWIpLMl3Svp3hUrVvRXfGZmNeGFNRv57s3zWbTGCUOt8ZMee2cRMLvo9yxgcUk/c4HrJAFMAk6W1B4RvyruKSIuAS4BmDt3rt/8ZWa7FL/vsHY5Yeide4B9JO0BvACcBryruIeI2KPQLOkK4LelyYKZ2a4ucuGsyhbc2kDmhKEXIqJd0odJdz8MBi6PiEclnZO7u96CmZnVNScMvRQRNwI3lrQrmyhExJmViMnMrObkSxJyAUPNcaVHMzOrOOcLtccJg5mZVYzrPNYuJwxmZlZx8jWJmuOEwczMKsa3VdYuJwxmZlZxLmCoPU4YzMysYsK1GGqWEwYzM6uYwiUJFzDUHicMZmZWcb4kUXucMJiZWcX4gkTtcsJgZmZV4CKGWuOEwczMKiZ8X2XNcsJgZmYV5zoMtccJg5mZVYzLF2qXEwYzM6sc31ZZs5wwmJlZxfldErXHCYOZmVWMn/RYu5wwmJlZxbl8ofY4YTAzs4rxXZW1ywmDmZlVnKsw1B4nDGZmVjEuYahdThjMzKxiCvmCXIuh5jhhMDOzivMlidrjhMHMzCrG75KoXU4YzMzMrEdOGMzMrGJcvlC7nDCYmVnFuQ5D7XHCYGZmFeMqDLXLCYOZmVVQyhh8W2XtccJgZmYV50sStccJg5mZVYwvSdQuJwxmZlZxLmGoPU4YzMysYlzAULucMJiZWcW50mPtccJgZmYV4zoMtcsJQy9JeoOkeZLmSzqvTPd3S3oof+6UdEg14jQzG8iicFulCxhqjhOGXpA0GPgBcBJwIPBOSQeW9PYs8OqIeCnwJeCSykZpZlY7nC/UHicMvXMkMD8inomITcB1wKnFPUTEnRGxOv+8C5hV4RjNzAY8X5KoXU4YemcmsLDo96Lcrit/B/yuXAdJZ0u6V9K9K1as6McQzcxqhy9J1B4nDL1TbtUumydLeg0pYfh0ue4RcUlEzI2IuZMnT+7HEM3MBj4XMNSuIdUOoEYsAmYX/Z4FLC7tSdJLgcuAkyJiZYViMzOrQS5iqDUuYeide4B9JO0haRhwGnBDcQ+SdgN+Abw3Ip6sQoxmZgNeuBJDzXIJQy9ERLukDwO/BwYDl0fEo5LOyd0vAr4ATAT+Q+niXHtEzK1WzGZmA5nrMNQeJwy9FBE3AjeWtLuoqPn9wPsrHZeZWS3pzCUMg5wx1Jy6SxgkvRE4CGgstIuIL1YvIjMzK+joTN+DnTDUnLqqwyDpIuAdwEdINWreDuxe1aDMzGyLzs5cwlBXR59dQ70tsldExOnA6oi4ADiabe9uMDOzKurIlyQGD3IJQ62pt4RhY/7eIGkGsBnYo4rxmJlZkY5cwuBLErWn3uow/FbSOODrwP2kZ4RcVtWIzMxsiy2VHl3CUHPqLWH4WkS0AT+X9FtSxcfWKsdkZmZZoYRhiBOGmlNvlyT+r9AQEW0Rsba4nZmZVVdHp0sYalVdlDBImkZ6GdRwSS9j6zNHxwAjqhaYmZltw3UYalddJAzA64EzSe94+FZR+xbgs9UIyMzMXqy903dJ1Kq6SBgi4krgSkl/ExE/r3Y8ZmZW3oqWNsBPeqxFdZEwSDq3XHNBRHyrtJ2ZmVXe2OFDAVd6rEV1kTAAo6sdgJmZ9ay9s5NBcqXHWlQXCUN+qqOZmQ1w7Z3BkMH1doPerqGulpqkfSX9SdIj+fdLJX2+2nGZmVnS3hEMdelCTaqrhAG4FPgM6ZHQRMRDwGlVjcjMzLZo7+h0CUONqrelNiIi7i5p116VSMzM7EU2dwZDB7uEoRbVW8LQJGkv0jskkPQ2YEl1QzIzs4L2jk6G+N3WNakuKj0W+RBwCbC/pBeAZ4H3VDckMzMraO8Ihg5xCUMtqquEISKeAU6QNBIYFBEt1Y7JzMy2auvoZJjrMNSkukgYyj2sKbcH/OAmM7OBYnN7J0OdMNSkukgY2Prgpv2AI4Ab8u83A7dVJSIzM3uRTR2dDBvihKEW1UXCUHhwk6SbgMMKlyIknQ/8tIqhmZlZkbbNnTQ4YahJ9bbUdgM2Ff3eBMypTihmZlaqrb2DxqGDqx2G9UFdlDAUuQq4W9IvSbdW/j/gyuqGZGZmBW3tnUwYWW/nqruGukoYIuIrkn4HvCq3Oisi/lrNmMzMbKu29k4ahriEoRbVVcIAEBH3A/dXOw4zM3uxtvYO12GoUV5qZmZWMeta2xnZUHfnqrsEJwxmZlYREUFLazujG50w1KK6Sxgk7S7phNw8XNLonoYxM7Odr3VzJ+2dwejGodUOxfqgrhIGSR8AfgZcnFvNAn5VtYDMzGyLltbNAC5hqFF1lTCQXj51DNAMEBFPAVOqGpGZmQHQ7IShptVbwtAWEVse3CRpCPlV12ZmVl3Nre0AjPEliZpUbwnDrZI+CwyXdCLpsdC/qXJMZmYGrFqXzucmjBxW5UisL+otYTgPWAE8DPw9cCPw+apGZGZmAKxc3wbAxFFOGGpRXSUMEdEZEZdGxNsj4m25uV8uSUh6g6R5kuZLOq9Md0n6bu7+kKTD+mO6Zmb1YllzShgmjWqociTWF3VV80TSs5SpsxARe+7geAcDPwBOBBYB90i6ISIeK+rtJGCf/DkK+GH+NjMzYPGajUwcOcwvn6pRdZUwAHOLmhuBtwMT+mG8RwLzI+IZAEnXAacCxQnDqcCPc4nGXZLGSZoeEUv6YfpmZjVv3rIW9p4yqtphWB/V2yWJlUWfFyLi28Dx/TDqmcDCot+Lcrvt7QdJZ0u6V9K9K1as6IfQzMwGvrb2DuYtbeGA6WOqHYr1UV2VMJTUGxhEKnHojyc9qky70ksfvemHiLgEuARg7ty5vuXTzHYJt85bwYZNHRy776Rqh2J9VFcJA/DNouZ24Dngb/thvIuA2UW/ZwGL+9CPmdkup6V1Mxf+7glmjR/OsftMrnY41kd1lTBExGt20qjvAfaRtAfwAnAa8K6Sfm4APpzrNxwFrHX9BTPb1XV2Bude/yDPr9rANe8/iiGD6+pK+C6lLhIGSed21z0ivrUj44+IdkkfBn4PDAYuj4hHJZ2Tu19EeubDycB8YANw1o5M08ys1q1ra+eCGx7lD48t4wtvOpCX7zmx2iHZDqiLhIH+qafQrYi4kZQUFLe7qKg5SO+yMDPbpa3duJmf3ruQi297hqZ1bXzk+L0565g51Q7LdlBdJAwRcUG1YzAz25W1bu7gz0818ZsHF3PTY0tp3dzJUXtM4NLT53Lo7HHVDs/6QV0kDAWSGoG/Aw4iPYcBgIh4X9WCMjOrQx2dwbylLfzl2ZXcMX8ld8xvYuPmDsaNGMpbD5vFO4/YjZfMGlvtMK0f1VXCAFwFPAG8Hvgi8G7g8apGZGZW49Zu2MwTS5t5YmkLTyxt5vElLTy5rIUNmzoAmD1hOG87fBYnHjiVo/eayFBXbKxL9ZYw7B0Rb5d0akRcKem/SBUVzcysB5s7Onm2aT2PL8nJQf5esrZ1Sz/jRgzlgGljeMcRs3nJzLEcuccEZo0fUcWorVLqLWHYnL/XSDoYWArMqV44ZmYDz9oNm3l+1XqeX7mBBas28PTydTyxtIX5y9exqaMTgKGDxV6TR3HUHhPYf/oY9p82mgOmj2HK6Aakcs+ps3pXbwnDJZLGA/9Mei7CqNxsZrZL6OwMmta3sXRtK0vWtrJ0bSsvrNnIotUbWLhqI8+vXE9za/s2w0wb08h+00bzqn0nccC0Mew/fTR7ThrFsCG+tGBb1VvC8J8R0QHcCuzQGyrNzAaazR2dLG9pY+najVuSgaVrW1nSvLV5WXMr7Z3bPnV+2JBBzBo3nFkTRnDo7HHsNmEEu00cwW4TRrD7xBGMGFZvhwLbGeptLXlW0v8APwFuzs9GMDMb8DZsamdZcyoZWNq8bUKwtDmVFjSta6N0r9Y4dBDTxw5n2phGjtpjAlPHNjJ9bCPTxjQybWz6TBrZwKBBvoxgO6beEob9gDeTHqB0uaTfANdFxO3VDcvMdkWb2jtZub6NppZNrFjXyoqWNprWbWJFSxsr1rWl75Y2lje3sj7fcVBsTOOQlAyMbeSAaWOYlpOBQlIwfcxwxgwf4joFVhF1lTBExEbgeuD6XJfhO6TLE4OrGpiZ1Y229g6a1m2iqaWtKBloo2ldGyvXbaKpkAisa2PNhs1lxzG6cQiTRzcwaVQDB84Yw3H7TWbK6EamjG5IpQP540sFNpDU3doo6dXAO4CTSC+N6o+3VZpZneroDFZv2MTKdZtYua6NpvXpe+W6TaxcX/je2q6lrb3seEYOG8yknATsOXkkL99zIpNGNTB5dENODoZtSRIah/ocxmpPXSUMkp4FHiCVMnwyItZXNyIzq7TOzmDtxs2sXL+JVeu3TQLS720TgdUbNr2oXgDAIMGEkcOYOLKBiaOG8dJZ4/LvYVsSg0mjhuXvBoYPcxJg9a2uEgbgkIhornYQZtZ/IoL1mzrSgT+XAhTO+JuKzv6bckKwesNmOjrL13ceO3woE0cNY9LIBvaaPIoj9hi25cBfSAwmjRrGhJENjB0+lMGuKGi2RV0lDE4WzGpDW3tH0dn+1uL+psKZ/5akINUJaGvvLDue0Q1DmJDP8udMHMnhu0/IB/xhW0oHJowcxsTczo8sNuu7ukoYzKw6IoLm1vYtFf8KZ/wripoLyUFTS1uX9QCGDRnEpJHDmDgqne3vM2V0OvvPJQATculAIQFwXQCzyqmrhEHS4PzgJjPbQYXKgIVa/03r0h0BKRHYenmgUFJQeKRwMQnGj8jX/Uc1cNCMMUwa1cDEnBSkZKBhS6nAqAbfImg2UNVVwgDMl/Qz0hMfH6t2MGYDUVt7R7r3v6WN5c1trGhpZVlzG8tbWlnesvXZACvXbypbF2DYkEFMzgf5KaMbOHD6mC0H/UIFwIm5VGDCiGEM8WUAs7pQbwnDS4HTgMskDQIuJz24yXUbrO5t2NTO8uacCLS0sry5jWUtrawobtdS/tkAgwQTRzUwZXT6HDRjTLodcFQDk0c3MjHfEjh5dAOjXQpgtkuqq4QhIlqAS4FLJR0LXAv8ey51+FJEzK9qgGZ90NbewfLmNpY2b31XQPq0say5dUtpwboy9QKGDhZTRjcyeXSqFHjkHhO2PCBoypiGLc0TRzX4jgAz61ZdJQySBgNvBM4ivdb6m8A1wKuAG4F9qxacWYmIYPWGzSxZu5Flza0sXdu2JRkoJAfLW1IdgVINQwYxdUwjU8c0cMD0MRy777YJQKF5/IihLg0ws35RVwkD8BTwv8DXI+LOovY/yyUOZhXT3LqZJWtaWbx2Y/pes3FL85K1G1m8tpVNJbcLSjBxZAPTxjYwa/xwDt99PFPHpBcJTc0vFJo6Jj0jwImAmVVSvSUMp5e+aErSMRFxR0R8tFpBWf3Z1N7J0rWtLFqzgcWFZGBNSgKWrElvGiy9RDB4kJg6uoHp44Zz8MyxvO6gaVveKDh1THqZ0OTRDX5WgJkNSPWWMHwXOKyk3ffKtDPrVntHJ0vWtrJw1QYWrNrAwtUbeGH1Rl5Ys5FFqzeytLn1RY8TnjRqGDPGDWfPySM5Zu9JzBjXyPSxw7d8Txnd4DsGzKxm1UXCIOlo4BXAZEnnFnUag99UaV1o7+hk4eqNPLNiHc+sWM8zTetZsGo9C1el0oL2olsKBw8S08c2MnPccI7eayKzxo9g1rjhzBw/nBnjhjN9bKMfImRmda0uEgZgGDCK9H9GF7VvBt5WlYhswFi1ftOWpODpppwcrFjHglUb2NyxNSkYP2Iou00cySGzx/HmQ6Yze/wIdps4gtnjRzB9bKNLB8xsl1YXCUNE3ArcKumKiHi+2vFY5W1q72TBqvU8vWL9loTg6RXreKZp/TbPHRg6WOw+cSR7TR7FiQdOY8/JI9lr8kj2nDSK8SOHVfEfmJkNbHWRMEj6dkR8HPi+pBc9mi4iTql8VNbfIoIV69pyQpCSgmea0vfC1Ru3eSrh5NEN7DlpJCcdPD0lBDkpmDV+uEsKzMz6oC4SBuCq/P2NqkZh/SIiWNrcyhNLWnh8aTNPLVu35ZJC8UuLGoYMYo9JIzloxljefMiMLUnBHpNHMqZxaBX/gZlZ/amLhCEi7ssPbfpARLyn2vFY73V2Bs+uXM8jL6zNn2YeX9q8zWWE6WMb2WvyKP7fYTPZc9JI9pw8ij0nj2TG2OEM8tMJzcwqoi4SBoCI6JA0WdKwiHjxo/FsQFjW3Mr9z6/mgUVreGhhShIKpQbDhgzigGmjOengaRwwfQz7TxvDftNGM3a4SwvMzKqtbhKG7DngDkk3AOsLLSPiW1WLaBfW0Rk8sbSZe55dxb3Pr+avC9bwwpqNQKp8uP+0MZxy6AwOmTWOg2eOZZ+po/zQIjOzAareEobF+TOIbW+vtAro7AweW9LMHfObuPvZVdzz3CqaW1PpwfSxjRy2+3je98o9OGy3cRw4YwwNQ/zcAjOzWlFXCUNEXFDtGHY1S9e2cttTK7jtyRXcMb+J1bnuwZ6TR/LGl07nyD0mcMScCcwaP6LKkZqZ2Y6oq4RB0mTgU8BBQGOhfUQcX7Wg6kxE8OjiZm56bBl/fGwZjy1pBmDK6AZes/8UXrXPJI7ZaxJTxjT2MCYzM6sldZUwkF5l/RPgTcA5wBnAih0ZoaQJeZxzSHUk/jYiVpf0Mxv4MTAN6AQuiYjv7Mh0B5KI4IGFa/jvh5bwu0eW8sKajQwSHL77eD79hv15zf6T2W/qaL890cysjtVbwjAxIn4k6WNFT3+8dQfHeR7wp4i4UNJ5+fenS/ppBz4REfdLGg3cJ+kPEfHYDk67qhau2sBP7lnIDQ8uZsGqDQwbPIhX7jOJj712H44/YAqTRjVUO0QzM6uQeksYCjfvL5H0RlIFyFk7OM5TgeNy85XALZQkDBGxBFiSm1skPQ7MBGouYWjv6OTmJ5ZzzV8WcNtTKxBwzN6T+PDxe/OGg6f5gUhmZruoeksYvixpLPAJ0mutxwD/uIPjnJoTAiJiiaQp3fUsaQ7wMuAvXXQ/GzgbYLfddtvB0PpP07o2rr7rea67eyFLm1uZOqaBjx6/D6cdOZvpY4dXOzwzM6uyukoYIuK3uXEt8JreDifpj6T6B6U+tz3TlzQK+Dnw8Yho7iLGS4BLAObOnfui915U2poNm7j4tme44o7n2Li5g2P3ncwFpx7Ea/ef4ncumJnZFnWRMEj6HtDlwTciPtrd8BFxQjfjXiZpei5dmA4s76K/oaRk4ZqI+EXvIq+eltbNXH77c1z252dYt6mdN790Bh87YR/2mjyq2qGZmdkAVBcJA3DvThz3DaS7LS7M378u7UHp9oAfAY8P9KdKdnYG196zgG/8fh6rN2zmdQdO5dzX7cv+08ZUOzQzMxvA6iJhiIgrd+LoLwSul/R3wALg7QCSZgCXRcTJwDHAe4GHJT2Qh/tsRNy4E+PabvOWtvCZXzzE/QvWcNQeE/jsyQdwyOxx1Q7LzMxqQF0kDJK+HREfl/QbylyaiIhT+jruiFgJvLZM+8XAybn5dmDAPoSgszO4/I5n+dr/zGNU4xC++fZDeOthM/3cBDMz67W6SBiAq/L3N6oaxQC0ZsMmzr3+QW5+YjknHDCVr/7NS5jo5yeYmdl2qouEISLuy987+pCmurJw1QZOv/xuFq3ewAWnHMTpR+/uUgUzM+uTukgYCiS9CfgSsDvpvwmIiNjlavQ9sbSZ0390N23tnVz7gZczd86EaodkZmY1rK4SBuDbwFuBhyOi6s84qJbHFjfzrsvuomHIIH56ztHsO9Vv+jYzsx1TbwnDQuCRXTlZWLJ2I2f8592MGDqY684+mt0m+rXSZma24+otYfgUcGN+4VRboeVAfzZCf9nU3skHr76fjZs6uOYfXuFkwczM+k29JQxfAdYBjcCwKsdScd/6w5M8sHAN//Huw3wZwszM+lW9JQwTIuJ11Q6iGh5atIaLb3ua046YzckvmV7tcMzMrM7U29uF/ihpl0sYIoIv/fYxJo4cxufeeEC1wzEzszpUbwnDh4D/kbRRUrOkFkll3xpZT256bBn3PLeac0/cj9GNQ6sdjpmZ1aG6uiQREbvchfuI4Ps3z2ePSSP527mzqh2OmZnVqbpIGCTtHxFPSDqsXPeIuL/SMVXK/QvW8PALa/nSWw5myOB6KzAyM7OBoi4SBuBc4Gzgm2W6BXB8ZcOpnJ/dt5CRwwbz1pfNrHYoZmZWx+oiYYiIs/P3a6odSyVtau/kxoeXcuKBUxnZUBeL0szMBqi6KMOWdISkaUW/T5f0a0nflVS3L1G4f8Fq1m7czEm+jdLMzHayukgYgIuBTQCSjgUuBH4MrAUuqWJcO9V9z68G4OV7TKxyJGZmVu/qpRx7cESsys3vAC6JiJ8DP5f0QPXC2rn+umA1e00eydgRvpXSzMx2rnopYRgsqZD8vBa4uahbvSRF24gIHli4hkNnj692KGZmtguol4PptcCtkpqAjcCfASTtTbosUXcWrd5I07pNvGy3cdUOxczMdgF1kTBExFck/QmYDtxU9HrrQcBHqhfZzvPgojUAHDp7XFXjMDOzXUNdJAwAEXFXmXZPViOWSnhmxXoA9p4yqsqRmJnZrqBe6jDschas2sDUMQ00Dh1c7VDMzGwX4IShRi1avYHZ40dUOwwzM9tFOGGoUcua25g6trHaYZiZ2S7CCUONamppY/KohmqHYWZmuwgnDDVoc0cnLW3tjB8xrNqhmJnZLsIJQw1qaW0HYOzwurnJxczMBjgnDDWopXUzAKMb/UhoMzOrDCcMNWhdWyph8CutzcysUpww1KD1bR0AjHLCYGZmFeKEoQat35RKGEY0+KFNZmZWGU4YalDb5lTC0DjECYOZmVWGE4Ya1NbeCcCwIV58ZmZWGT7i9EDSBEl/kPRU/h7fTb+DJf1V0m93ZkybO9LLOIcN9uIzM7PK8BGnZ+cBf4qIfYA/5d9d+Rjw+M4OaJNLGMzMrMJ8xOnZqcCVuflK4C3lepI0C3gjcNnODmhTe6rD4ITBzMwqxUecnk2NiCUA+XtKF/19G/gU0NndyCSdLeleSfeuWLGiTwEVLkkMHaw+DW9mZra9fCM/IOmPwLQynT7Xy+HfBCyPiPskHdddvxFxCXAJwNy5c2P7Ik02dfiShJmZVZYTBiAiTuiqm6RlkqZHxBJJ04HlZXo7BjhF0slAIzBG0tUR8Z6dEW97LmEYMsgJg5mZVYaPOD27ATgjN58B/Lq0h4j4TETMiog5wGnAzTsrWQDojJQwDPIVCTMzqxAnDD27EDhR0lPAifk3kmZIurEaARWuY0jOGMzMrDJ8SaIHEbESeG2Z9ouBk8u0vwW4ZScHtVNHb2ZmVsolDDWoM3w5wszMKssJQw0KwpcjzMysopww1KAIcLpgZmaV5IShBgXgAgYzM6skJww1KJUwOGMwM7PKccJQgwJfkzAzs8pywlCLnC+YmVmFOWGoQQEMciUGMzOrICcMNaizM1zp0czMKsoJQw0KfEnCzMwqywlDDYrweyTMzKyynDDUoCBcwmBmZhXlhKEGha9JmJlZhTlhqFHOF8zMrJKcMNSgiGCQX1dpZmYV5IShBnX6wU1mZlZhThhqkF9vbWZmleaEoQb59dZmZlZpThhqkF9vbWZmleaEoQZFgMsYzMyskpww1CS/S8LMzCrLCUMNuu3JplzKYGZmVhlDqh2Abb8PHrcXm9o7qx2GmZntQpww1KD3vHz3aodgZma7GF+SMDMzsx45YTAzM7MeOWEwMzOzHjlhMDMzsx45YTAzM7MeOWEwMzOzHjlhMDMzsx45YTAzM7MeKfyM4aqRtAJ4vo+DTwKa+jGc/jRQYxuocYFj64uBGhcM3NgGalywfbHtHhGTd2Yw9mJOGGqUpHsjYm614yhnoMY2UOMCx9YXAzUuGLixDdS4YGDHZokvSZiZmVmPnDCYmZlZj5ww1K5Lqh1ANwZqbAM1LnBsfTFQ44KBG9tAjQsGdmyG6zCYmZlZL7iEwczMzHrkhMHMzMx65IShBkl6g6R5kuZLOq8fx3u5pOWSHilqN0HSHyQ9lb/HF3X7TI5hnqTXF7U/XNLDudt3JSm3b5D0k9z+L5LmFA1zRp7GU5LOKIlrtqT/lfS4pEclfWwAxdYo6W5JD+bYLhgoseXugyX9VdJvB1hcz+VxPiDp3oESm6Rxkn4m6Ym8vh09QOLaL8+rwqdZ0scHSGz/qLTuPyLpWqVtoupx2U4QEf7U0AcYDDwN7AkMAx4EDuyncR8LHAY8UtTua8B5ufk84Ku5+cA87QZgjxzT4NztbuBoQMDvgJNy+38ALsrNpwE/yc0TgGfy9/jcPL4ohunAYbl5NPBknv5AiE3AqNw8FPgL8PKBEFvu51zgv4DfDpTlmft5DphU0q7qsQFXAu/PzcOAcQMhrjL7gKXA7tWODZgJPAsMz7+vB86sdlz+7KTjT7UD8Gc7F1jaoH5f9PszwGf6cfxz2DZhmAdMz83TgXnlpgv8Psc2HXiiqP07gYuL+8nNQ0hPdVNxP7nbxcA7u4nx18CJAy02YARwP3DUQIgNmAX8CTierQlD1ePK7Z7jxQlDVWMDxpAOfhpIcZVZz14H3DEQYiMlDAtJB+0hwG9zfANqnvnTPx9fkqg9hQ20YFFut7NMjYglAPl7Sg9xzMzN5eLbMkxEtANrgYndjOtFcnHky0hn8gMiNqVi/weA5cAfImKgxPZt4FNAZ1G7gRAXQAA3SbpP0tkDJLY9gRXAfypdxrlM0sgBEFep04Brc3NVY4uIF4BvAAuAJcDaiLip2nHZzuGEofaoTLuoeBRdx9FdfH0ZZusEpVHAz4GPR0TzQIktIjoi4lDSGf2Rkg6udmyS3gQsj4j7uoml4nEVOSYiDgNOAj4k6dgBENsQ0iW5H0bEy4D1pOL0ase1dYLSMOAU4KfdxFWx2HLdhFNJlxdmACMlvafacdnO4YSh9iwCZhf9ngUs3onTWyZpOkD+Xt5DHItyc7n4tgwjaQgwFljVzbi2kDSUlCxcExG/GEixFUTEGuAW4A0DILZjgFMkPQdcBxwv6eoBEBcAEbE4fy8HfgkcOQBiWwQsyiVEAD8jJRDVjqvYScD9EbEs/652bCcAz0bEiojYDPwCeMUAiMt2hmpfE/Fn+z6ks6BnSBl9odLjQf04/jlsW4fh62xbeelrufkgtq289AxbKy/dQ6r4V6i8dHJu/yG2rbx0fW6eQLp2PD5/ngUmFMUg4MfAt0tiHQixTQbG5ebhwJ+BNw2E2IpiPI6tdRiqHhcwEhhd1HwnKckaCLH9GdgvN5+fY6p6XEXxXQecNVC2AVJ9nUdJ9XdEqjT6kWrH5c9OOv5UOwB/+rDQ4GTSnQJPA5/rx/FeS7oOuZmUvf8d6Vrhn4Cn8nfxzvVzOYZ55BrNuf1c4JHc7fuw5YmijaSi1PmkGtF7Fg3zvtx+fvEOMXd7Jamo8SHggfw5eYDE9lLgrzm2R4Av5PZVj62on+PYmjBUPS5SXYEH8+dR8jo8QGI7FLg3L89fkQ5EVY8rdx8BrATGFrWremzABcATeZxXkZKBqsflT/9//GhoMzMz65HrMJiZmVmPnDCYmZlZj5wwmJmZWY+cMJiZmVmPnDCYmZlZj5wwmPWRpM/lt/Q9lN8geNR2Dn+mpBnbOcwcFb1NtBf93yhp3Hb0f4ryG1AlnS/pn7YzvuLh3yLpwO0c/ouSTtjOYZ6TNGl7hjGz7Tek2gGY1SJJR5Me0HRYRLTlA9aw7Rh+MOmtfo+wE59OFxEnb2f/NwA39GVakoaUDP8W0suIHtuO6X+hL9M2s53PJQxmfTMdaIqINoCIaIr8uGNJr80vL3pY0uWSGnL75yR9QdLtpDftzQWuyaUTwyUdLunW/EKm3xc9WvdwSQ9K+j/SU+9eRNJ0SbflcT0i6VVF05yUSyaeyC9UekTSNZJOkHSHpKckHZn7P1PS98uM/wOS7slx/FzSiNz+CknfkvS/wFcLw0t6BemdB1/PMe0l6f6i8e0j6UXvucjje1tR7BdIuj/Py/1z+4mSbsrz+GKK3ikg6T2S7s7TvFjpxWBH5FKgRkkjc6lQd+/7MLMynDCY9c1NwGxJT0r6D0mvBpDUCFwBvCMiXkIqxftg0XCtEfHKiLia9ETBd0d6cVU78D3gbRFxOHA58JU8zH8CH42Io7uJ512k154fChxCehpmqb2B75CeTrl/HuaVwD8Bn+3h//4iIo6IiEOAx0lPAS3YFzghIj5RaBERd5JKGj4ZEYdGxNPAWkmH5l7OIs2nnjRFeknVD3OcAP8C3B7pBVE3ALsBSDoAeAfpxVaHAh2k+XtP7u/LwNeAqyOi15d1zCzxJQmzPoiIdZIOB14FvAb4Sb52/1fSy3iezL1eSSoV+Hb+/ZMuRrkfcDDwB0kAg4ElksaS3lVxa+7vKtILiErdA1yu9JKuX0XEA2X6eTYiHgaQ9Cjwp4gISQ+T3iHSnYMlfRkYB4wCfl/U7acR0dHD8ACXAWdJOpd0YD+yF8MUXjR2H/DW3HxsoTki/lvS6tz+tcDhwD15Hg5n60uPvkiaR63AR3sxXTMr4YTBrI/yQfIW4JZ80D2D8mf2xdZ30V7Ao6WlCLnCYo/Pb4+I25ReEf1G4CpJX4+IH5f01lbU3Fn0u5Oe9wVXAG+JiAclnUl6P0VBV/+p1M9JpQM3A/dFxMpeDFOIsaMkxnLzRMCVEfGZMt0mkBKdoaR3E/Q2ZjPLfEnCrA8k7Sdpn6JWhwLPk17CM0fS3rn9e4FbKa8FGJ2b5wGTc2VKJA2VdFCkV2avlfTK3N+7u4hnd2B5RFwK/Ij0Wub+NJpU4jG0qxjKKP5/REQrqWTih6TLLH11WyEGSSeRXhAF6SVHb5M0JXebkOcLwCXAPwPXAF/dgWmb7bKcMJj1zSjgSkmPSXoIOBA4Px8UzwJ+mksdOoGLuhjHFcBFkh4gXYJ4G6ni4IOkkopX5P7OAn6QKz1u7GJcxwEPSPor8Dekugr96Z+BvwB/ICVFvXEd8MlcOXGv3O4aUunATTsQywXAsbkS5euABQAR8RjweeCmvEz+AEyXdDrQHhH/BVwIHCHp+B2YvtkuyW+rNLOKUXquw9iI+Odqx2Jm28d1GMysIiT9EtgL8Nm9WQ1yCYOZmZn1yHUYzMzMrEdOGMzMzKxHThjMzMysR04YzMzMrEdOGMzMzKxH/x9KK3JmKxnIggAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pyplot.plot(W_sparse_delta_sorted)\n",
    "pyplot.ylabel('Similarity value delta')\n",
    "pyplot.xlabel('Sorted similarity index')\n",
    "pyplot.title(\"Delta between the existing KNN CF similarities and the corresponding KNN CBF ones\")\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "W_sparse_CF has 7.42E+06 values and 10.63 % in common with CBF\n",
      "W_sparse_CBF has 3.10E+06 values and 25.45 % in common with CF\n",
      "W_sparse_delta has 7.88E+05 values\n"
     ]
    }
   ],
   "source": [
    "# How many values we have in common?\n",
    "\n",
    "print(\"W_sparse_CF has {:.2E} values and {:.2f} % in common with CBF\".format(W_sparse_CF.nnz, W_sparse_common.nnz/W_sparse_CF.nnz*100))\n",
    "print(\"W_sparse_CBF has {:.2E} values and {:.2f} % in common with CF\".format(W_sparse_CBF.nnz, W_sparse_common.nnz/W_sparse_CBF.nnz*100))\n",
    "\n",
    "print(\"W_sparse_delta has {:.2E} values\".format(W_sparse_delta.nnz))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What do we see?\n",
    "## The great majority of similarities are not common... the KNNs are modeling two very different phenomena"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Idea: \n",
    "\n",
    "what if we were to weight features in such a way to approximate as well as possible the collaborative similarity?\n",
    "\n",
    "Basic model:  $S_{CBF} = ICM \\cdot diag(feature\\_weights) \\cdot ICM^{T}$ \n",
    "\n",
    "\n",
    "We want  $S_{CBF} = S_{CF}$\n",
    "\n",
    "Hence we wish to find the weights such that  $S_{CF} = ICM \\cdot diag(feature\\_weights) \\cdot ICM^{T}$ \n",
    "\n",
    "Now, we can rearrange the equations considering that we need only weights for features the two items have in common... this becomes a regression problem in form Ax = b. In our case A are the common features, x the weights and b the collaborative similarity\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Since it is a regression problem, we can use an already available solver like linalg in scipy.sparse\n",
    "\n",
    "### It takes as imput two data structures, one is sparse and the other is the target value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "W_sparse_delta = W_sparse_delta.tocoo()\n",
    "\n",
    "\n",
    "item_index_1 = W_sparse_delta.row[666]\n",
    "item_index_2 = W_sparse_delta.col[666]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Item indices are 7 and 42\n"
     ]
    }
   ],
   "source": [
    "print(\"Item indices are {} and {}\".format(item_index_1, item_index_2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collaborative similarity value is 0.18686850368976593\n",
      "Content-based similarity value is 0.021173948422074318\n"
     ]
    }
   ],
   "source": [
    "print(\"Collaborative similarity value is {}\".format(W_sparse_CF[item_index_1, item_index_2]))\n",
    "print(\"Content-based similarity value is {}\".format(W_sparse_CBF[item_index_1, item_index_2]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compute the common features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<1x10106 sparse matrix of type '<class 'numpy.float64'>'\n",
       "\twith 10 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "common_features = ICM_tags[item_index_1,:].multiply(ICM_tags[item_index_2,:])\n",
    "common_features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  25,   33,   94,  145,  511,  550, 2538, 5518, 5728, 6107],\n",
       "      dtype=int32)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "common_features.indices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## These items have just two common features, that is a common scenario\n",
    "\n",
    "### Let's look for items having  a few features in common, say 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Item indices are 1 and 11\n",
      "Collaborative similarity value is 0.3871592581272125\n",
      "Content-based similarity value is 0.10350184142589569\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([   8,  489,  671,  672, 2593], dtype=int32)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import scipy.sparse as sps\n",
    "\n",
    "min_common_features = 5\n",
    "common_features = sps.csr_matrix((1,1))\n",
    "inner_index = 0\n",
    "\n",
    "while (common_features.nnz < min_common_features and inner_index<len(W_sparse_delta.row)):\n",
    "\n",
    "    item_index_1 = W_sparse_delta.row[inner_index]\n",
    "    item_index_2 = W_sparse_delta.col[inner_index]\n",
    "    inner_index += 1\n",
    "    \n",
    "    common_features = ICM_tags[item_index_1,:].multiply(ICM_tags[item_index_2,:])      \n",
    "        \n",
    "\n",
    "print(\"Item indices are {} and {}\".format(item_index_1, item_index_2))\n",
    "\n",
    "print(\"Collaborative similarity value is {}\".format(W_sparse_CF[item_index_1, item_index_2]))\n",
    "print(\"Content-based similarity value is {}\".format(W_sparse_CBF[item_index_1, item_index_2]))\n",
    "    \n",
    "common_features.indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scipy.sparse import linalg\n",
    "\n",
    "loss_tolerance = 1e-6\n",
    "iteration_limit = 50000\n",
    "damp_coeff=0.0\n",
    "\n",
    "W_sparse_CF_value = W_sparse_CF[item_index_1, item_index_2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "linalg_result = linalg.lsqr(common_features, W_sparse_CF_value, show = False,\n",
    "                            atol=loss_tolerance, btol=loss_tolerance,\n",
    "                            iter_lim = iteration_limit, damp=damp_coeff)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 0., ..., 0., 0., 0.])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_weights = linalg_result[0].copy()\n",
    "feature_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3.38891435e-04, 4.84130622e-05, 3.05002292e-03, 3.05002292e-03,\n",
       "       1.45239186e-04])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_weights[common_features.indices]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Note that all weights will be zeros except for the features those items have in common, nothing can be learned if a feature is not common\n",
    "\n",
    "### If a dataset has a very sparse ICM, this FW method will struggle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.38715926])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_CBF_value = common_features.dot(feature_weights)\n",
    "new_CBF_value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collaborative similarity value is 0.3871592581272125\n",
      "Content-based similarity value is 0.10350184142589569\n",
      "Weighted Content-based similarity value is 0.38715925812721236\n"
     ]
    }
   ],
   "source": [
    "print(\"Collaborative similarity value is {}\".format(W_sparse_CF[item_index_1, item_index_2]))\n",
    "print(\"Content-based similarity value is {}\".format(W_sparse_CBF[item_index_1, item_index_2]))\n",
    "\n",
    "print(\"Weighted Content-based similarity value is {}\".format(new_CBF_value[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Note that we learned the CF similarity amost exactly...\n",
    "\n",
    "### Warning: 5 common features and 1 similarity value, more parameters than data points, overfitting!\n",
    "\n",
    "### if we want to apply this method to the whole dataset we have to compute the common features of all items having a common similarity value. Whe already know who they are, thanks to W_sparse_delta"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "from Recommenders.FeatureWeighting.CFW_D_Similarity_Linalg import CFW_D_Similarity_Linalg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CFW_D_Similarity_Linalg: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "CFW_D_Similarity_Linalg: ICM Detected 3087 (28.9%) items with no features.\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 (100.0%), 14068.36 column/sec. Elapsed time 0.76 sec\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.50E-02, nonzero cells 7415187\n",
      "CFW_D_Similarity_Linalg: Content S density: 1.73E-02, nonzero cells 1975457\n",
      "CFW_D_Similarity_Linalg: Content S structure has 567640 out of 1975457 (28.7%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 7.51E+04, average is: 1.32E-01, average over all collaborative data is 8.64E-02\n",
      "Similarity column 10681 (100.0%), 13885.38 column/sec. Elapsed time 0.77 sec\n"
     ]
    }
   ],
   "source": [
    "CFW_weithing = CFW_D_Similarity_Linalg(URM_train, ICM_tags, W_sparse_CF)\n",
    "CFW_weithing.fit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EvaluatorHoldout: Ignoring 67415 ( 3.5%) Users that have less than 1 test interactions\n",
      "EvaluatorHoldout: Ignoring 69805 ( 0.1%) Users that have less than 1 test interactions\n"
     ]
    }
   ],
   "source": [
    "from Evaluation.Evaluator import EvaluatorHoldout\n",
    "\n",
    "evaluator_validation = EvaluatorHoldout(URM_validation, cutoff_list=[5, 10, 20])\n",
    "evaluator_test = EvaluatorHoldout(URM_test, cutoff_list=[5, 10, 20])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EvaluatorHoldout: Processed 69805 (100.0%) in 2.89 min. Users per second: 402\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PRECISION</th>\n",
       "      <th>PRECISION_RECALL_MIN_DEN</th>\n",
       "      <th>RECALL</th>\n",
       "      <th>MAP</th>\n",
       "      <th>MAP_MIN_DEN</th>\n",
       "      <th>MRR</th>\n",
       "      <th>NDCG</th>\n",
       "      <th>F1</th>\n",
       "      <th>HIT_RATE</th>\n",
       "      <th>ARHR_ALL_HITS</th>\n",
       "      <th>...</th>\n",
       "      <th>COVERAGE_ITEM_CORRECT</th>\n",
       "      <th>COVERAGE_USER</th>\n",
       "      <th>COVERAGE_USER_CORRECT</th>\n",
       "      <th>DIVERSITY_GINI</th>\n",
       "      <th>SHANNON_ENTROPY</th>\n",
       "      <th>RATIO_DIVERSITY_HERFINDAHL</th>\n",
       "      <th>RATIO_DIVERSITY_GINI</th>\n",
       "      <th>RATIO_SHANNON_ENTROPY</th>\n",
       "      <th>RATIO_AVERAGE_POPULARITY</th>\n",
       "      <th>RATIO_NOVELTY</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cutoff</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.311568</td>\n",
       "      <td>0.31824</td>\n",
       "      <td>0.103998</td>\n",
       "      <td>0.230186</td>\n",
       "      <td>0.234289</td>\n",
       "      <td>0.515758</td>\n",
       "      <td>0.241143</td>\n",
       "      <td>0.155944</td>\n",
       "      <td>0.724332</td>\n",
       "      <td>0.773383</td>\n",
       "      <td>...</td>\n",
       "      <td>0.075836</td>\n",
       "      <td>0.998955</td>\n",
       "      <td>0.723575</td>\n",
       "      <td>0.009682</td>\n",
       "      <td>6.970848</td>\n",
       "      <td>0.986868</td>\n",
       "      <td>0.04974</td>\n",
       "      <td>0.615551</td>\n",
       "      <td>1.9305</td>\n",
       "      <td>0.039192</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.26773</td>\n",
       "      <td>0.306406</td>\n",
       "      <td>0.169867</td>\n",
       "      <td>0.171386</td>\n",
       "      <td>0.191499</td>\n",
       "      <td>0.528319</td>\n",
       "      <td>0.251792</td>\n",
       "      <td>0.207856</td>\n",
       "      <td>0.81679</td>\n",
       "      <td>0.919716</td>\n",
       "      <td>...</td>\n",
       "      <td>0.101582</td>\n",
       "      <td>0.998955</td>\n",
       "      <td>0.815936</td>\n",
       "      <td>0.012401</td>\n",
       "      <td>7.379472</td>\n",
       "      <td>0.990963</td>\n",
       "      <td>0.063708</td>\n",
       "      <td>0.651634</td>\n",
       "      <td>1.829649</td>\n",
       "      <td>0.07913</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>0.218631</td>\n",
       "      <td>0.333019</td>\n",
       "      <td>0.258057</td>\n",
       "      <td>0.121338</td>\n",
       "      <td>0.172511</td>\n",
       "      <td>0.532908</td>\n",
       "      <td>0.27555</td>\n",
       "      <td>0.236714</td>\n",
       "      <td>0.881427</td>\n",
       "      <td>1.03504</td>\n",
       "      <td>...</td>\n",
       "      <td>0.133789</td>\n",
       "      <td>0.998955</td>\n",
       "      <td>0.880506</td>\n",
       "      <td>0.016322</td>\n",
       "      <td>7.807952</td>\n",
       "      <td>0.993838</td>\n",
       "      <td>0.083852</td>\n",
       "      <td>0.689471</td>\n",
       "      <td>1.706362</td>\n",
       "      <td>0.160229</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 25 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       PRECISION PRECISION_RECALL_MIN_DEN    RECALL       MAP MAP_MIN_DEN  \\\n",
       "cutoff                                                                      \n",
       "5       0.311568                  0.31824  0.103998  0.230186    0.234289   \n",
       "10       0.26773                 0.306406  0.169867  0.171386    0.191499   \n",
       "20      0.218631                 0.333019  0.258057  0.121338    0.172511   \n",
       "\n",
       "             MRR      NDCG        F1  HIT_RATE ARHR_ALL_HITS  ...  \\\n",
       "cutoff                                                        ...   \n",
       "5       0.515758  0.241143  0.155944  0.724332      0.773383  ...   \n",
       "10      0.528319  0.251792  0.207856   0.81679      0.919716  ...   \n",
       "20      0.532908   0.27555  0.236714  0.881427       1.03504  ...   \n",
       "\n",
       "       COVERAGE_ITEM_CORRECT COVERAGE_USER COVERAGE_USER_CORRECT  \\\n",
       "cutoff                                                             \n",
       "5                   0.075836      0.998955              0.723575   \n",
       "10                  0.101582      0.998955              0.815936   \n",
       "20                  0.133789      0.998955              0.880506   \n",
       "\n",
       "       DIVERSITY_GINI SHANNON_ENTROPY RATIO_DIVERSITY_HERFINDAHL  \\\n",
       "cutoff                                                             \n",
       "5            0.009682        6.970848                   0.986868   \n",
       "10           0.012401        7.379472                   0.990963   \n",
       "20           0.016322        7.807952                   0.993838   \n",
       "\n",
       "       RATIO_DIVERSITY_GINI RATIO_SHANNON_ENTROPY RATIO_AVERAGE_POPULARITY  \\\n",
       "cutoff                                                                       \n",
       "5                   0.04974              0.615551                   1.9305   \n",
       "10                 0.063708              0.651634                 1.829649   \n",
       "20                 0.083852              0.689471                 1.706362   \n",
       "\n",
       "       RATIO_NOVELTY  \n",
       "cutoff                \n",
       "5           0.039192  \n",
       "10           0.07913  \n",
       "20          0.160229  \n",
       "\n",
       "[3 rows x 25 columns]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_df, _ = evaluator_test.evaluateRecommender(itemKNNCF)\n",
    "results_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EvaluatorHoldout: Processed 69805 (100.0%) in 2.24 min. Users per second: 520\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PRECISION</th>\n",
       "      <th>PRECISION_RECALL_MIN_DEN</th>\n",
       "      <th>RECALL</th>\n",
       "      <th>MAP</th>\n",
       "      <th>MAP_MIN_DEN</th>\n",
       "      <th>MRR</th>\n",
       "      <th>NDCG</th>\n",
       "      <th>F1</th>\n",
       "      <th>HIT_RATE</th>\n",
       "      <th>ARHR_ALL_HITS</th>\n",
       "      <th>...</th>\n",
       "      <th>COVERAGE_ITEM_CORRECT</th>\n",
       "      <th>COVERAGE_USER</th>\n",
       "      <th>COVERAGE_USER_CORRECT</th>\n",
       "      <th>DIVERSITY_GINI</th>\n",
       "      <th>SHANNON_ENTROPY</th>\n",
       "      <th>RATIO_DIVERSITY_HERFINDAHL</th>\n",
       "      <th>RATIO_DIVERSITY_GINI</th>\n",
       "      <th>RATIO_SHANNON_ENTROPY</th>\n",
       "      <th>RATIO_AVERAGE_POPULARITY</th>\n",
       "      <th>RATIO_NOVELTY</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cutoff</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.132996</td>\n",
       "      <td>0.134467</td>\n",
       "      <td>0.032536</td>\n",
       "      <td>0.089544</td>\n",
       "      <td>0.090417</td>\n",
       "      <td>0.282975</td>\n",
       "      <td>0.111531</td>\n",
       "      <td>0.052282</td>\n",
       "      <td>0.432362</td>\n",
       "      <td>0.354832</td>\n",
       "      <td>...</td>\n",
       "      <td>0.019567</td>\n",
       "      <td>0.998955</td>\n",
       "      <td>0.43191</td>\n",
       "      <td>0.002467</td>\n",
       "      <td>4.889391</td>\n",
       "      <td>0.943808</td>\n",
       "      <td>0.012672</td>\n",
       "      <td>0.431751</td>\n",
       "      <td>1.900615</td>\n",
       "      <td>0.039605</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.107369</td>\n",
       "      <td>0.115891</td>\n",
       "      <td>0.05029</td>\n",
       "      <td>0.058554</td>\n",
       "      <td>0.062473</td>\n",
       "      <td>0.297552</td>\n",
       "      <td>0.107655</td>\n",
       "      <td>0.068498</td>\n",
       "      <td>0.540362</td>\n",
       "      <td>0.408644</td>\n",
       "      <td>...</td>\n",
       "      <td>0.024623</td>\n",
       "      <td>0.998955</td>\n",
       "      <td>0.539798</td>\n",
       "      <td>0.003516</td>\n",
       "      <td>5.476874</td>\n",
       "      <td>0.963596</td>\n",
       "      <td>0.01806</td>\n",
       "      <td>0.483628</td>\n",
       "      <td>1.689557</td>\n",
       "      <td>0.081138</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>0.077085</td>\n",
       "      <td>0.102555</td>\n",
       "      <td>0.069606</td>\n",
       "      <td>0.035134</td>\n",
       "      <td>0.0447</td>\n",
       "      <td>0.302905</td>\n",
       "      <td>0.105773</td>\n",
       "      <td>0.073155</td>\n",
       "      <td>0.616045</td>\n",
       "      <td>0.440872</td>\n",
       "      <td>...</td>\n",
       "      <td>0.033705</td>\n",
       "      <td>0.998955</td>\n",
       "      <td>0.615401</td>\n",
       "      <td>0.005227</td>\n",
       "      <td>6.132445</td>\n",
       "      <td>0.979353</td>\n",
       "      <td>0.026854</td>\n",
       "      <td>0.541517</td>\n",
       "      <td>1.35263</td>\n",
       "      <td>0.170255</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 25 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       PRECISION PRECISION_RECALL_MIN_DEN    RECALL       MAP MAP_MIN_DEN  \\\n",
       "cutoff                                                                      \n",
       "5       0.132996                 0.134467  0.032536  0.089544    0.090417   \n",
       "10      0.107369                 0.115891   0.05029  0.058554    0.062473   \n",
       "20      0.077085                 0.102555  0.069606  0.035134      0.0447   \n",
       "\n",
       "             MRR      NDCG        F1  HIT_RATE ARHR_ALL_HITS  ...  \\\n",
       "cutoff                                                        ...   \n",
       "5       0.282975  0.111531  0.052282  0.432362      0.354832  ...   \n",
       "10      0.297552  0.107655  0.068498  0.540362      0.408644  ...   \n",
       "20      0.302905  0.105773  0.073155  0.616045      0.440872  ...   \n",
       "\n",
       "       COVERAGE_ITEM_CORRECT COVERAGE_USER COVERAGE_USER_CORRECT  \\\n",
       "cutoff                                                             \n",
       "5                   0.019567      0.998955               0.43191   \n",
       "10                  0.024623      0.998955              0.539798   \n",
       "20                  0.033705      0.998955              0.615401   \n",
       "\n",
       "       DIVERSITY_GINI SHANNON_ENTROPY RATIO_DIVERSITY_HERFINDAHL  \\\n",
       "cutoff                                                             \n",
       "5            0.002467        4.889391                   0.943808   \n",
       "10           0.003516        5.476874                   0.963596   \n",
       "20           0.005227        6.132445                   0.979353   \n",
       "\n",
       "       RATIO_DIVERSITY_GINI RATIO_SHANNON_ENTROPY RATIO_AVERAGE_POPULARITY  \\\n",
       "cutoff                                                                       \n",
       "5                  0.012672              0.431751                 1.900615   \n",
       "10                  0.01806              0.483628                 1.689557   \n",
       "20                 0.026854              0.541517                  1.35263   \n",
       "\n",
       "       RATIO_NOVELTY  \n",
       "cutoff                \n",
       "5           0.039605  \n",
       "10          0.081138  \n",
       "20          0.170255  \n",
       "\n",
       "[3 rows x 25 columns]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_df, _ = evaluator_test.evaluateRecommender(itemKNNCBF)\n",
    "results_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EvaluatorHoldout: Processed 69805 (100.0%) in 1.91 min. Users per second: 610\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PRECISION</th>\n",
       "      <th>PRECISION_RECALL_MIN_DEN</th>\n",
       "      <th>RECALL</th>\n",
       "      <th>MAP</th>\n",
       "      <th>MAP_MIN_DEN</th>\n",
       "      <th>MRR</th>\n",
       "      <th>NDCG</th>\n",
       "      <th>F1</th>\n",
       "      <th>HIT_RATE</th>\n",
       "      <th>ARHR_ALL_HITS</th>\n",
       "      <th>...</th>\n",
       "      <th>COVERAGE_ITEM_CORRECT</th>\n",
       "      <th>COVERAGE_USER</th>\n",
       "      <th>COVERAGE_USER_CORRECT</th>\n",
       "      <th>DIVERSITY_GINI</th>\n",
       "      <th>SHANNON_ENTROPY</th>\n",
       "      <th>RATIO_DIVERSITY_HERFINDAHL</th>\n",
       "      <th>RATIO_DIVERSITY_GINI</th>\n",
       "      <th>RATIO_SHANNON_ENTROPY</th>\n",
       "      <th>RATIO_AVERAGE_POPULARITY</th>\n",
       "      <th>RATIO_NOVELTY</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cutoff</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.128581</td>\n",
       "      <td>0.129655</td>\n",
       "      <td>0.028334</td>\n",
       "      <td>0.083702</td>\n",
       "      <td>0.08426</td>\n",
       "      <td>0.251594</td>\n",
       "      <td>0.105564</td>\n",
       "      <td>0.046435</td>\n",
       "      <td>0.40722</td>\n",
       "      <td>0.323352</td>\n",
       "      <td>...</td>\n",
       "      <td>0.020316</td>\n",
       "      <td>0.998955</td>\n",
       "      <td>0.406795</td>\n",
       "      <td>0.002513</td>\n",
       "      <td>4.922491</td>\n",
       "      <td>0.946743</td>\n",
       "      <td>0.01291</td>\n",
       "      <td>0.434674</td>\n",
       "      <td>1.729163</td>\n",
       "      <td>0.040789</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.114579</td>\n",
       "      <td>0.122349</td>\n",
       "      <td>0.050053</td>\n",
       "      <td>0.059636</td>\n",
       "      <td>0.062446</td>\n",
       "      <td>0.268868</td>\n",
       "      <td>0.107934</td>\n",
       "      <td>0.06967</td>\n",
       "      <td>0.537025</td>\n",
       "      <td>0.388664</td>\n",
       "      <td>...</td>\n",
       "      <td>0.027057</td>\n",
       "      <td>0.998955</td>\n",
       "      <td>0.536464</td>\n",
       "      <td>0.00368</td>\n",
       "      <td>5.547019</td>\n",
       "      <td>0.966322</td>\n",
       "      <td>0.018904</td>\n",
       "      <td>0.489822</td>\n",
       "      <td>1.648456</td>\n",
       "      <td>0.082043</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>0.10095</td>\n",
       "      <td>0.13071</td>\n",
       "      <td>0.08555</td>\n",
       "      <td>0.04246</td>\n",
       "      <td>0.050417</td>\n",
       "      <td>0.277722</td>\n",
       "      <td>0.118983</td>\n",
       "      <td>0.092614</td>\n",
       "      <td>0.663505</td>\n",
       "      <td>0.447737</td>\n",
       "      <td>...</td>\n",
       "      <td>0.038667</td>\n",
       "      <td>0.998955</td>\n",
       "      <td>0.662812</td>\n",
       "      <td>0.005806</td>\n",
       "      <td>6.297284</td>\n",
       "      <td>0.981422</td>\n",
       "      <td>0.029827</td>\n",
       "      <td>0.556073</td>\n",
       "      <td>1.548863</td>\n",
       "      <td>0.165274</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 25 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       PRECISION PRECISION_RECALL_MIN_DEN    RECALL       MAP MAP_MIN_DEN  \\\n",
       "cutoff                                                                      \n",
       "5       0.128581                 0.129655  0.028334  0.083702     0.08426   \n",
       "10      0.114579                 0.122349  0.050053  0.059636    0.062446   \n",
       "20       0.10095                  0.13071   0.08555   0.04246    0.050417   \n",
       "\n",
       "             MRR      NDCG        F1  HIT_RATE ARHR_ALL_HITS  ...  \\\n",
       "cutoff                                                        ...   \n",
       "5       0.251594  0.105564  0.046435   0.40722      0.323352  ...   \n",
       "10      0.268868  0.107934   0.06967  0.537025      0.388664  ...   \n",
       "20      0.277722  0.118983  0.092614  0.663505      0.447737  ...   \n",
       "\n",
       "       COVERAGE_ITEM_CORRECT COVERAGE_USER COVERAGE_USER_CORRECT  \\\n",
       "cutoff                                                             \n",
       "5                   0.020316      0.998955              0.406795   \n",
       "10                  0.027057      0.998955              0.536464   \n",
       "20                  0.038667      0.998955              0.662812   \n",
       "\n",
       "       DIVERSITY_GINI SHANNON_ENTROPY RATIO_DIVERSITY_HERFINDAHL  \\\n",
       "cutoff                                                             \n",
       "5            0.002513        4.922491                   0.946743   \n",
       "10            0.00368        5.547019                   0.966322   \n",
       "20           0.005806        6.297284                   0.981422   \n",
       "\n",
       "       RATIO_DIVERSITY_GINI RATIO_SHANNON_ENTROPY RATIO_AVERAGE_POPULARITY  \\\n",
       "cutoff                                                                       \n",
       "5                   0.01291              0.434674                 1.729163   \n",
       "10                 0.018904              0.489822                 1.648456   \n",
       "20                 0.029827              0.556073                 1.548863   \n",
       "\n",
       "       RATIO_NOVELTY  \n",
       "cutoff                \n",
       "5           0.040789  \n",
       "10          0.082043  \n",
       "20          0.165274  \n",
       "\n",
       "[3 rows x 25 columns]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_df, _ = evaluator_test.evaluateRecommender(CFW_weithing)\n",
    "results_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### This algorithm has hyperparameters too:\n",
    "* topK, the usual number of neighbors\n",
    "* add_zeros_quota, the percentage of zero collaborative similarities to add to the train set, this helps the regressor in finding non-relevant features\n",
    "* normalize_similarity, apply or not the usual cosine denominator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "from HyperparameterTuning.SearchBayesianSkopt import SearchBayesianSkopt\n",
    "from skopt.space import Real, Integer, Categorical\n",
    "from HyperparameterTuning.SearchAbstractClass import SearchInputRecommenderArgs\n",
    "\n",
    "\n",
    "def hyperparameter_optimization_CFW_D(URM_train, ICM_train, W_sparse_CF):\n",
    "\n",
    "    recommender_class = CFW_D_Similarity_Linalg\n",
    "\n",
    "    parameterSearch = SearchBayesianSkopt(recommender_class,\n",
    "                                     evaluator_validation=evaluator_validation,\n",
    "                                     evaluator_test=evaluator_test)\n",
    "\n",
    "\n",
    "    hyperparameters_range_dictionary = {}\n",
    "    hyperparameters_range_dictionary[\"topK\"] = Integer(5, 1000)\n",
    "    hyperparameters_range_dictionary[\"add_zeros_quota\"] = Real(low = 0, high = 1, prior = 'uniform')\n",
    "    hyperparameters_range_dictionary[\"normalize_similarity\"] = Categorical([True, False])\n",
    "\n",
    "\n",
    "    recommender_input_args = SearchInputRecommenderArgs(\n",
    "        CONSTRUCTOR_POSITIONAL_ARGS = [URM_train, ICM_train, W_sparse_CF],\n",
    "        CONSTRUCTOR_KEYWORD_ARGS = {},\n",
    "        FIT_POSITIONAL_ARGS = [],\n",
    "        FIT_KEYWORD_ARGS = {}\n",
    "    )\n",
    "\n",
    "\n",
    "    output_folder_path = \"result_experiments/\"\n",
    "\n",
    "    import os\n",
    "\n",
    "    # If directory does not exist, create\n",
    "    if not os.path.exists(output_folder_path):\n",
    "        os.makedirs(output_folder_path)\n",
    "\n",
    "    n_cases = 10\n",
    "    metric_to_optimize = \"MAP\"\n",
    "    cutoff_to_optimize = 10\n",
    "\n",
    "\n",
    "    # Clone data structure to perform the fitting with the best hyperparameters on train + validation data\n",
    "    recommender_input_args_last_test = recommender_input_args.copy()\n",
    "    recommender_input_args_last_test.CONSTRUCTOR_POSITIONAL_ARGS[0] = URM_train + URM_validation\n",
    "\n",
    "\n",
    "    parameterSearch.search(recommender_input_args,\n",
    "                           recommender_input_args_last_test = recommender_input_args_last_test,\n",
    "                           hyperparameter_search_space = hyperparameters_range_dictionary,\n",
    "                           n_cases = n_cases,\n",
    "                           n_random_starts = int(n_cases/3),\n",
    "                           save_model = \"no\",\n",
    "                           evaluate_on_test = \"last\",\n",
    "                           output_folder_path = output_folder_path,\n",
    "                           output_file_name_root = recommender_class.RECOMMENDER_NAME,\n",
    "                           metric_to_optimize = metric_to_optimize,\n",
    "                           cutoff_to_optimize = cutoff_to_optimize\n",
    "                          )\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration No: 1 started. Evaluating function at random point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 561, 'add_zeros_quota': 0.890362988450008, 'normalize_similarity': False}\n",
      "CFW_D_Similarity_Linalg: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "CFW_D_Similarity_Linalg: ICM Detected 3087 (28.9%) items with no features.\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 (100.0%), 13831.74 column/sec. Elapsed time 0.77 sec\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.50E-02, nonzero cells 7415187\n",
      "CFW_D_Similarity_Linalg: Content S density: 3.00E-02, nonzero cells 3420680\n",
      "CFW_D_Similarity_Linalg: Content S structure has 876028 out of 3420680 (25.6%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 1.14E+05, average is: 1.30E-01, average over all collaborative data is 8.64E-02\n",
      "Similarity column 10681 (100.0%), 11892.97 column/sec. Elapsed time 0.90 sec\n",
      "EvaluatorHoldout: Processed 67415 (100.0%) in 2.41 min. Users per second: 467\n",
      "SearchBayesianSkopt: New best config found. Config 0: {'topK': 561, 'add_zeros_quota': 0.890362988450008, 'normalize_similarity': False} - results: PRECISION: 0.0476274, PRECISION_RECALL_MIN_DEN: 0.0681745, RECALL: 0.0502769, MAP: 0.0204555, MAP_MIN_DEN: 0.0287219, MRR: 0.1405649, NDCG: 0.0632135, F1: 0.0489163, HIT_RATE: 0.3270489, ARHR_ALL_HITS: 0.1685708, NOVELTY: 0.0087325, AVERAGE_POPULARITY: 0.5307364, DIVERSITY_MEAN_INTER_LIST: 0.5594035, DIVERSITY_HERFINDAHL: 0.9559395, COVERAGE_ITEM: 0.0518678, COVERAGE_ITEM_CORRECT: 0.0148862, COVERAGE_USER: 0.9647529, COVERAGE_USER_CORRECT: 0.3155213, DIVERSITY_GINI: 0.0024119, SHANNON_ENTROPY: 4.9899501, RATIO_DIVERSITY_HERFINDAHL: 0.9566611, RATIO_DIVERSITY_GINI: 0.0123907, RATIO_SHANNON_ENTROPY: 0.4406308, RATIO_AVERAGE_POPULARITY: 1.8424149, RATIO_NOVELTY: 0.0799342, \n",
      "\n",
      "Iteration No: 1 ended. Evaluation done at random point.\n",
      "Time taken: 630.7205\n",
      "Function value obtained: -0.0205\n",
      "Current minimum: -0.0205\n",
      "Iteration No: 2 started. Evaluating function at random point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 388, 'add_zeros_quota': 0.5694595906365112, 'normalize_similarity': True}\n",
      "CFW_D_Similarity_Linalg: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "CFW_D_Similarity_Linalg: ICM Detected 3087 (28.9%) items with no features.\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 (100.0%), 9387.12 column/sec. Elapsed time 1.14 sec\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.50E-02, nonzero cells 7415187\n",
      "CFW_D_Similarity_Linalg: Content S density: 2.18E-02, nonzero cells 2485981\n",
      "CFW_D_Similarity_Linalg: Content S structure has 683738 out of 2485981 (27.5%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 2.38E+07, average is: 3.49E+01, average over all collaborative data is 8.64E-02\n",
      "Similarity column 10681 (100.0%), 7471.65 column/sec. Elapsed time 1.43 sec\n",
      "EvaluatorHoldout: Processed 67415 (100.0%) in 1.91 min. Users per second: 587\n",
      "SearchBayesianSkopt: Config 1 is suboptimal. Config: {'topK': 388, 'add_zeros_quota': 0.5694595906365112, 'normalize_similarity': True} - results: PRECISION: 0.0021138, PRECISION_RECALL_MIN_DEN: 0.0031189, RECALL: 0.0022436, MAP: 0.0006763, MAP_MIN_DEN: 0.0009222, MRR: 0.0064283, NDCG: 0.0015678, F1: 0.0021767, HIT_RATE: 0.0201142, ARHR_ALL_HITS: 0.0065926, NOVELTY: 0.0138864, AVERAGE_POPULARITY: 0.0253383, DIVERSITY_MEAN_INTER_LIST: 0.7398033, DIVERSITY_HERFINDAHL: 0.9739792, COVERAGE_ITEM: 0.1379084, COVERAGE_ITEM_CORRECT: 0.0159161, COVERAGE_USER: 0.9647529, COVERAGE_USER_CORRECT: 0.0194052, DIVERSITY_GINI: 0.0075605, SHANNON_ENTROPY: 6.3582520, RATIO_DIVERSITY_HERFINDAHL: 0.9747144, RATIO_DIVERSITY_GINI: 0.0388401, RATIO_SHANNON_ENTROPY: 0.5614569, RATIO_AVERAGE_POPULARITY: 0.0879602, RATIO_NOVELTY: 0.1271115, \n",
      "\n",
      "Iteration No: 2 ended. Evaluation done at random point.\n",
      "Time taken: 463.0433\n",
      "Function value obtained: -0.0007\n",
      "Current minimum: -0.0205\n",
      "Iteration No: 3 started. Evaluating function at random point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 328, 'add_zeros_quota': 0.7559756110029718, 'normalize_similarity': True}\n",
      "CFW_D_Similarity_Linalg: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "CFW_D_Similarity_Linalg: ICM Detected 3087 (28.9%) items with no features.\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 (100.0%), 13637.96 column/sec. Elapsed time 0.78 sec\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.50E-02, nonzero cells 7415187\n",
      "CFW_D_Similarity_Linalg: Content S density: 1.88E-02, nonzero cells 2140865\n",
      "CFW_D_Similarity_Linalg: Content S structure has 606078 out of 2140865 (28.3%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 2.19E+07, average is: 3.62E+01, average over all collaborative data is 8.64E-02\n",
      "Similarity column 10681 (100.0%), 7263.43 column/sec. Elapsed time 1.47 sec\n",
      "EvaluatorHoldout: Processed 67415 (100.0%) in 2.25 min. Users per second: 500\n",
      "SearchBayesianSkopt: Config 2 is suboptimal. Config: {'topK': 328, 'add_zeros_quota': 0.7559756110029718, 'normalize_similarity': True} - results: PRECISION: 0.0021509, PRECISION_RECALL_MIN_DEN: 0.0031996, RECALL: 0.0023128, MAP: 0.0006494, MAP_MIN_DEN: 0.0008941, MRR: 0.0061253, NDCG: 0.0016284, F1: 0.0022289, HIT_RATE: 0.0203367, ARHR_ALL_HITS: 0.0063063, NOVELTY: 0.0139382, AVERAGE_POPULARITY: 0.0249705, DIVERSITY_MEAN_INTER_LIST: 0.7322401, DIVERSITY_HERFINDAHL: 0.9732229, COVERAGE_ITEM: 0.1349125, COVERAGE_ITEM_CORRECT: 0.0159161, COVERAGE_USER: 0.9647529, COVERAGE_USER_CORRECT: 0.0196199, DIVERSITY_GINI: 0.0071161, SHANNON_ENTROPY: 6.2803115, RATIO_DIVERSITY_HERFINDAHL: 0.9739575, RATIO_DIVERSITY_GINI: 0.0365570, RATIO_SHANNON_ENTROPY: 0.5545745, RATIO_AVERAGE_POPULARITY: 0.0866833, RATIO_NOVELTY: 0.1275860, \n",
      "\n",
      "Iteration No: 3 ended. Evaluation done at random point.\n",
      "Time taken: 536.5059\n",
      "Function value obtained: -0.0006\n",
      "Current minimum: -0.0205\n",
      "Iteration No: 4 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 5, 'add_zeros_quota': 0.0, 'normalize_similarity': False}\n",
      "CFW_D_Similarity_Linalg: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "CFW_D_Similarity_Linalg: ICM Detected 3087 (28.9%) items with no features.\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 (100.0%), 13395.44 column/sec. Elapsed time 0.80 sec\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.50E-02, nonzero cells 7415187\n",
      "CFW_D_Similarity_Linalg: Content S density: 3.31E-04, nonzero cells 37717\n",
      "CFW_D_Similarity_Linalg: Content S structure has 14952 out of 37717 (39.6%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 2.18E+03, average is: 1.46E-01, average over all collaborative data is 8.64E-02\n",
      "Similarity column 10681 (100.0%), 12981.72 column/sec. Elapsed time 0.82 sec\n",
      "EvaluatorHoldout: Processed 67415 (100.0%) in 2.36 min. Users per second: 475\n",
      "SearchBayesianSkopt: Config 3 is suboptimal. Config: {'topK': 5, 'add_zeros_quota': 0.0, 'normalize_similarity': False} - results: PRECISION: 0.0322109, PRECISION_RECALL_MIN_DEN: 0.0462006, RECALL: 0.0343855, MAP: 0.0113526, MAP_MIN_DEN: 0.0161512, MRR: 0.0864654, NDCG: 0.0377637, F1: 0.0332627, HIT_RATE: 0.2481347, ARHR_ALL_HITS: 0.0988418, NOVELTY: 0.0099422, AVERAGE_POPULARITY: 0.2984378, DIVERSITY_MEAN_INTER_LIST: 0.9237852, DIVERSITY_HERFINDAHL: 0.9923772, COVERAGE_ITEM: 0.1931467, COVERAGE_ITEM_CORRECT: 0.0446587, COVERAGE_USER: 0.9647529, COVERAGE_USER_CORRECT: 0.2393886, DIVERSITY_GINI: 0.0176886, SHANNON_ENTROPY: 7.8757897, RATIO_DIVERSITY_HERFINDAHL: 0.9931262, RATIO_DIVERSITY_GINI: 0.0908706, RATIO_SHANNON_ENTROPY: 0.6954610, RATIO_AVERAGE_POPULARITY: 1.0360065, RATIO_NOVELTY: 0.0910076, \n",
      "\n",
      "Iteration No: 4 ended. Search finished for the next optimal point.\n",
      "Time taken: 152.4775\n",
      "Function value obtained: -0.0114\n",
      "Current minimum: -0.0205\n",
      "Iteration No: 5 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 1000, 'add_zeros_quota': 0.017288268337664617, 'normalize_similarity': False}\n",
      "CFW_D_Similarity_Linalg: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "CFW_D_Similarity_Linalg: ICM Detected 3087 (28.9%) items with no features.\n",
      "CFW_D_Similarity_Linalg: Generating train data\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Similarity column 10681 (100.0%), 12065.84 column/sec. Elapsed time 0.89 sec\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.50E-02, nonzero cells 7415187\n",
      "CFW_D_Similarity_Linalg: Content S density: 4.73E-02, nonzero cells 5391649\n",
      "CFW_D_Similarity_Linalg: Generating train data. Sample 671315 (12.7%) \n",
      "CFW_D_Similarity_Linalg: Generating train data. Sample 1286544 (24.3%) \n",
      "CFW_D_Similarity_Linalg: Content S structure has 1216982 out of 5391649 (22.6%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 1.56E+05, average is: 1.29E-01, average over all collaborative data is 8.64E-02\n",
      "Similarity column 10681 (100.0%), 11487.17 column/sec. Elapsed time 0.93 sec\n",
      "EvaluatorHoldout: Processed 67415 (100.0%) in 2.90 min. Users per second: 388\n",
      "SearchBayesianSkopt: Config 4 is suboptimal. Config: {'topK': 1000, 'add_zeros_quota': 0.017288268337664617, 'normalize_similarity': False} - results: PRECISION: 0.0389424, PRECISION_RECALL_MIN_DEN: 0.0554399, RECALL: 0.0405033, MAP: 0.0157580, MAP_MIN_DEN: 0.0222527, MRR: 0.1145053, NDCG: 0.0510086, F1: 0.0397075, HIT_RATE: 0.2803382, ARHR_ALL_HITS: 0.1335468, NOVELTY: 0.0090642, AVERAGE_POPULARITY: 0.4528623, DIVERSITY_MEAN_INTER_LIST: 0.5040455, DIVERSITY_HERFINDAHL: 0.9504038, COVERAGE_ITEM: 0.0469993, COVERAGE_ITEM_CORRECT: 0.0135755, COVERAGE_USER: 0.9647529, COVERAGE_USER_CORRECT: 0.2704571, DIVERSITY_GINI: 0.0022366, SHANNON_ENTROPY: 4.8548089, RATIO_DIVERSITY_HERFINDAHL: 0.9511212, RATIO_DIVERSITY_GINI: 0.0114899, RATIO_SHANNON_ENTROPY: 0.4286974, RATIO_AVERAGE_POPULARITY: 1.5720803, RATIO_NOVELTY: 0.0829709, \n",
      "\n",
      "Iteration No: 5 ended. Search finished for the next optimal point.\n",
      "Time taken: 494.2557\n",
      "Function value obtained: -0.0158\n",
      "Current minimum: -0.0205\n",
      "Iteration No: 6 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 703, 'add_zeros_quota': 0.00046478974956365806, 'normalize_similarity': False}\n",
      "CFW_D_Similarity_Linalg: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "CFW_D_Similarity_Linalg: ICM Detected 3087 (28.9%) items with no features.\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 (100.0%), 8223.58 column/sec. Elapsed time 1.30 sec\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.50E-02, nonzero cells 7415187\n",
      "CFW_D_Similarity_Linalg: Content S density: 3.60E-02, nonzero cells 4110621\n",
      "CFW_D_Similarity_Linalg: Generating train data. Sample 649017 (15.8%) \n",
      "CFW_D_Similarity_Linalg: Content S structure has 1003861 out of 4110621 (24.4%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 1.30E+05, average is: 1.29E-01, average over all collaborative data is 8.64E-02\n",
      "Similarity column 10681 (100.0%), 9207.64 column/sec. Elapsed time 1.16 sec\n",
      "EvaluatorHoldout: Processed 67415 (100.0%) in 2.53 min. Users per second: 444\n",
      "SearchBayesianSkopt: Config 5 is suboptimal. Config: {'topK': 703, 'add_zeros_quota': 0.00046478974956365806, 'normalize_similarity': False} - results: PRECISION: 0.0404687, PRECISION_RECALL_MIN_DEN: 0.0574533, RECALL: 0.0419781, MAP: 0.0164537, MAP_MIN_DEN: 0.0229656, MRR: 0.1182127, NDCG: 0.0526436, F1: 0.0412096, HIT_RATE: 0.2889268, ARHR_ALL_HITS: 0.1386798, NOVELTY: 0.0090479, AVERAGE_POPULARITY: 0.4559405, DIVERSITY_MEAN_INTER_LIST: 0.5409815, DIVERSITY_HERFINDAHL: 0.9540973, COVERAGE_ITEM: 0.0555191, COVERAGE_ITEM_CORRECT: 0.0157289, COVERAGE_USER: 0.9647529, COVERAGE_USER_CORRECT: 0.2787430, DIVERSITY_GINI: 0.0025015, SHANNON_ENTROPY: 5.0104646, RATIO_DIVERSITY_HERFINDAHL: 0.9548175, RATIO_DIVERSITY_GINI: 0.0128507, RATIO_SHANNON_ENTROPY: 0.4424423, RATIO_AVERAGE_POPULARITY: 1.5827663, RATIO_NOVELTY: 0.0828215, \n",
      "\n",
      "Iteration No: 6 ended. Search finished for the next optimal point.\n",
      "Time taken: 425.9662\n",
      "Function value obtained: -0.0165\n",
      "Current minimum: -0.0205\n",
      "Iteration No: 7 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 982, 'add_zeros_quota': 0.9968194757250247, 'normalize_similarity': False}\n",
      "CFW_D_Similarity_Linalg: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "CFW_D_Similarity_Linalg: ICM Detected 3087 (28.9%) items with no features.\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 (100.0%), 8075.47 column/sec. Elapsed time 1.32 sec\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.50E-02, nonzero cells 7415187\n",
      "CFW_D_Similarity_Linalg: Content S density: 4.66E-02, nonzero cells 5321204\n",
      "CFW_D_Similarity_Linalg: Generating train data. Sample 2332060 (87.5%) \n",
      "CFW_D_Similarity_Linalg: Content S structure has 1205821 out of 5321204 (22.7%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 1.55E+05, average is: 1.29E-01, average over all collaborative data is 8.64E-02\n",
      "Similarity column 10681 (100.0%), 11183.77 column/sec. Elapsed time 0.96 sec\n",
      "EvaluatorHoldout: Processed 67415 (100.0%) in 3.02 min. Users per second: 372\n",
      "SearchBayesianSkopt: Config 6 is suboptimal. Config: {'topK': 982, 'add_zeros_quota': 0.9968194757250247, 'normalize_similarity': False} - results: PRECISION: 0.0457213, PRECISION_RECALL_MIN_DEN: 0.0656390, RECALL: 0.0484941, MAP: 0.0194927, MAP_MIN_DEN: 0.0275383, MRR: 0.1355648, NDCG: 0.0607493, F1: 0.0470669, HIT_RATE: 0.3171698, ARHR_ALL_HITS: 0.1616018, NOVELTY: 0.0087542, AVERAGE_POPULARITY: 0.5276966, DIVERSITY_MEAN_INTER_LIST: 0.5170484, DIVERSITY_HERFINDAHL: 0.9517041, COVERAGE_ITEM: 0.0412883, COVERAGE_ITEM_CORRECT: 0.0123584, COVERAGE_USER: 0.9647529, COVERAGE_USER_CORRECT: 0.3059904, DIVERSITY_GINI: 0.0020972, SHANNON_ENTROPY: 4.7927849, RATIO_DIVERSITY_HERFINDAHL: 0.9524224, RATIO_DIVERSITY_GINI: 0.0107739, RATIO_SHANNON_ENTROPY: 0.4232204, RATIO_AVERAGE_POPULARITY: 1.8318625, RATIO_NOVELTY: 0.0801329, \n",
      "\n",
      "Iteration No: 7 ended. Search finished for the next optimal point.\n",
      "Time taken: 1028.8245\n",
      "Function value obtained: -0.0195\n",
      "Current minimum: -0.0205\n",
      "Iteration No: 8 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 996, 'add_zeros_quota': 0.8994262681381963, 'normalize_similarity': True}\n",
      "CFW_D_Similarity_Linalg: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "CFW_D_Similarity_Linalg: ICM Detected 3087 (28.9%) items with no features.\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 (100.0%), 7943.49 column/sec. Elapsed time 1.34 sec\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.50E-02, nonzero cells 7415187\n",
      "CFW_D_Similarity_Linalg: Content S density: 4.71E-02, nonzero cells 5376065\n",
      "CFW_D_Similarity_Linalg: Generating train data. Sample 1786038 (63.1%) \n",
      "CFW_D_Similarity_Linalg: Generating train data. Sample 4932913 (174.3%) \n",
      "CFW_D_Similarity_Linalg: Content S structure has 1214547 out of 5376065 (22.6%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 3.47E+07, average is: 2.86E+01, average over all collaborative data is 8.64E-02\n",
      "Similarity column 10681 (100.0%), 8418.84 column/sec. Elapsed time 1.27 sec\n",
      "EvaluatorHoldout: Processed 67415 (100.0%) in 2.20 min. Users per second: 510\n",
      "SearchBayesianSkopt: Config 7 is suboptimal. Config: {'topK': 996, 'add_zeros_quota': 0.8994262681381963, 'normalize_similarity': True} - results: PRECISION: 0.0030438, PRECISION_RECALL_MIN_DEN: 0.0048178, RECALL: 0.0037729, MAP: 0.0008136, MAP_MIN_DEN: 0.0011659, MRR: 0.0077621, NDCG: 0.0023896, F1: 0.0033694, HIT_RATE: 0.0292517, ARHR_ALL_HITS: 0.0079476, NOVELTY: 0.0133853, AVERAGE_POPULARITY: 0.0385044, DIVERSITY_MEAN_INTER_LIST: 0.7073728, DIVERSITY_HERFINDAHL: 0.9707362, COVERAGE_ITEM: 0.1188091, COVERAGE_ITEM_CORRECT: 0.0162906, COVERAGE_USER: 0.9647529, COVERAGE_USER_CORRECT: 0.0282206, DIVERSITY_GINI: 0.0059815, SHANNON_ENTROPY: 6.0480413, RATIO_DIVERSITY_HERFINDAHL: 0.9714689, RATIO_DIVERSITY_GINI: 0.0307286, RATIO_SHANNON_ENTROPY: 0.5340641, RATIO_AVERAGE_POPULARITY: 0.1336655, RATIO_NOVELTY: 0.1225248, \n",
      "\n",
      "Iteration No: 8 ended. Search finished for the next optimal point.\n",
      "Time taken: 1070.1009\n",
      "Function value obtained: -0.0008\n",
      "Current minimum: -0.0205\n",
      "Iteration No: 9 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 700, 'add_zeros_quota': 1.0, 'normalize_similarity': False}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CFW_D_Similarity_Linalg: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "CFW_D_Similarity_Linalg: ICM Detected 3087 (28.9%) items with no features.\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 (100.0%), 12027.20 column/sec. Elapsed time 0.89 sec\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.50E-02, nonzero cells 7415187\n",
      "CFW_D_Similarity_Linalg: Content S density: 3.59E-02, nonzero cells 4096717\n",
      "CFW_D_Similarity_Linalg: Generating train data. Sample 2719494 (132.8%) \n",
      "CFW_D_Similarity_Linalg: Content S structure has 1001348 out of 4096717 (24.4%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 1.30E+05, average is: 1.29E-01, average over all collaborative data is 8.64E-02\n",
      "Similarity column 10681 (100.0%), 12991.43 column/sec. Elapsed time 0.82 sec\n",
      "EvaluatorHoldout: Processed 67415 (100.0%) in 2.28 min. Users per second: 494\n",
      "SearchBayesianSkopt: Config 8 is suboptimal. Config: {'topK': 700, 'add_zeros_quota': 1.0, 'normalize_similarity': False} - results: PRECISION: 0.0469465, PRECISION_RECALL_MIN_DEN: 0.0672717, RECALL: 0.0496600, MAP: 0.0200355, MAP_MIN_DEN: 0.0281397, MRR: 0.1381368, NDCG: 0.0621028, F1: 0.0482652, HIT_RATE: 0.3238003, ARHR_ALL_HITS: 0.1653766, NOVELTY: 0.0087307, AVERAGE_POPULARITY: 0.5319740, DIVERSITY_MEAN_INTER_LIST: 0.5385547, DIVERSITY_HERFINDAHL: 0.9538547, COVERAGE_ITEM: 0.0477483, COVERAGE_ITEM_CORRECT: 0.0140436, COVERAGE_USER: 0.9647529, COVERAGE_USER_CORRECT: 0.3123873, DIVERSITY_GINI: 0.0022525, SHANNON_ENTROPY: 4.8938890, RATIO_DIVERSITY_HERFINDAHL: 0.9545746, RATIO_DIVERSITY_GINI: 0.0115714, RATIO_SHANNON_ENTROPY: 0.4321483, RATIO_AVERAGE_POPULARITY: 1.8467114, RATIO_NOVELTY: 0.0799184, \n",
      "\n",
      "Iteration No: 9 ended. Search finished for the next optimal point.\n",
      "Time taken: 818.3961\n",
      "Function value obtained: -0.0200\n",
      "Current minimum: -0.0205\n",
      "Iteration No: 10 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 10, 'add_zeros_quota': 0.9837508786196454, 'normalize_similarity': False}\n",
      "CFW_D_Similarity_Linalg: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "CFW_D_Similarity_Linalg: ICM Detected 3087 (28.9%) items with no features.\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 (100.0%), 13183.69 column/sec. Elapsed time 0.81 sec\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.50E-02, nonzero cells 7415187\n",
      "CFW_D_Similarity_Linalg: Content S density: 6.58E-04, nonzero cells 75106\n",
      "CFW_D_Similarity_Linalg: Content S structure has 29300 out of 75106 (39.0%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 4.21E+03, average is: 1.44E-01, average over all collaborative data is 8.64E-02\n",
      "Similarity column 10681 (100.0%), 10082.55 column/sec. Elapsed time 1.06 sec\n",
      "EvaluatorHoldout: Processed 67415 (100.0%) in 1.91 min. Users per second: 588\n",
      "SearchBayesianSkopt: New best config found. Config 9: {'topK': 10, 'add_zeros_quota': 0.9837508786196454, 'normalize_similarity': False} - results: PRECISION: 0.0584173, PRECISION_RECALL_MIN_DEN: 0.0831829, RECALL: 0.0612093, MAP: 0.0256974, MAP_MIN_DEN: 0.0359256, MRR: 0.1674752, NDCG: 0.0759199, F1: 0.0597807, HIT_RATE: 0.3789068, ARHR_ALL_HITS: 0.2061341, NOVELTY: 0.0090802, AVERAGE_POPULARITY: 0.4358379, DIVERSITY_MEAN_INTER_LIST: 0.9039388, DIVERSITY_HERFINDAHL: 0.9903925, COVERAGE_ITEM: 0.1510158, COVERAGE_ITEM_CORRECT: 0.0458759, COVERAGE_USER: 0.9647529, COVERAGE_USER_CORRECT: 0.3655514, DIVERSITY_GINI: 0.0131171, SHANNON_ENTROPY: 7.4441515, RATIO_DIVERSITY_HERFINDAHL: 0.9911401, RATIO_DIVERSITY_GINI: 0.0673856, RATIO_SHANNON_ENTROPY: 0.6573458, RATIO_AVERAGE_POPULARITY: 1.5129815, RATIO_NOVELTY: 0.0831177, \n",
      "\n",
      "Iteration No: 10 ended. Search finished for the next optimal point.\n",
      "Time taken: 144.7457\n",
      "Function value obtained: -0.0257\n",
      "Current minimum: -0.0257\n",
      "SearchBayesianSkopt: Search complete. Best config is 9: {'topK': 10, 'add_zeros_quota': 0.9837508786196454, 'normalize_similarity': False}\n",
      "\n",
      "CFW_D_Similarity_Linalg: URM Detected 36 ( 0.3%) items with no interactions.\n",
      "CFW_D_Similarity_Linalg: ICM Detected 3087 (28.9%) items with no features.\n",
      "SearchBayesianSkopt: Evaluation with constructor data for final test. Using best config: {'topK': 10, 'add_zeros_quota': 0.9837508786196454, 'normalize_similarity': False}\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 (100.0%), 14900.40 column/sec. Elapsed time 0.72 sec\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.50E-02, nonzero cells 7415187\n",
      "CFW_D_Similarity_Linalg: Content S density: 6.58E-04, nonzero cells 75106\n",
      "CFW_D_Similarity_Linalg: Content S structure has 29300 out of 75106 (39.0%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 4.21E+03, average is: 1.44E-01, average over all collaborative data is 8.64E-02\n",
      "Similarity column 10681 (100.0%), 9576.61 column/sec. Elapsed time 1.12 sec\n",
      "EvaluatorHoldout: Processed 69805 (100.0%) in 1.97 min. Users per second: 591\n",
      "SearchBayesianSkopt: Best config evaluated with evaluator_test with constructor data for final test. Config: {'topK': 10, 'add_zeros_quota': 0.9837508786196454, 'normalize_similarity': False} - results:\n",
      "CUTOFF: 5 - PRECISION: 0.1916854, PRECISION_RECALL_MIN_DEN: 0.1933097, RECALL: 0.0422124, MAP: 0.1410526, MAP_MIN_DEN: 0.1420167, MRR: 0.3528522, NDCG: 0.1574267, F1: 0.0691884, HIT_RATE: 0.5056371, ARHR_ALL_HITS: 0.4954538, NOVELTY: 0.0044725, AVERAGE_POPULARITY: 0.4725344, DIVERSITY_MEAN_INTER_LIST: 0.9325737, DIVERSITY_HERFINDAHL: 0.9865121, COVERAGE_ITEM: 0.1036420, COVERAGE_ITEM_CORRECT: 0.0437225, COVERAGE_USER: 0.9989553, COVERAGE_USER_CORRECT: 0.5051089, DIVERSITY_GINI: 0.0095682, SHANNON_ENTROPY: 6.9712753, RATIO_DIVERSITY_HERFINDAHL: 0.9872568, RATIO_DIVERSITY_GINI: 0.0491478, RATIO_SHANNON_ENTROPY: 0.6155837, RATIO_AVERAGE_POPULARITY: 1.6394532, RATIO_NOVELTY: 0.0368454, \n",
      "CUTOFF: 10 - PRECISION: 0.1583540, PRECISION_RECALL_MIN_DEN: 0.1684034, RECALL: 0.0662853, MAP: 0.0997378, MAP_MIN_DEN: 0.1042886, MRR: 0.3662037, NDCG: 0.1512807, F1: 0.0934524, HIT_RATE: 0.6050999, ARHR_ALL_HITS: 0.5772554, NOVELTY: 0.0090942, AVERAGE_POPULARITY: 0.4315674, DIVERSITY_MEAN_INTER_LIST: 0.9041125, DIVERSITY_HERFINDAHL: 0.9904100, COVERAGE_ITEM: 0.1453984, COVERAGE_ITEM_CORRECT: 0.0577661, COVERAGE_USER: 0.9989553, COVERAGE_USER_CORRECT: 0.6044678, DIVERSITY_GINI: 0.0128659, SHANNON_ENTROPY: 7.4275162, RATIO_DIVERSITY_HERFINDAHL: 0.9911576, RATIO_DIVERSITY_GINI: 0.0660866, RATIO_SHANNON_ENTROPY: 0.6558711, RATIO_AVERAGE_POPULARITY: 1.4973184, RATIO_NOVELTY: 0.0749205, \n",
      "CUTOFF: 20 - PRECISION: 0.1265490, PRECISION_RECALL_MIN_DEN: 0.1599114, RECALL: 0.1009232, MAP: 0.0677386, MAP_MIN_DEN: 0.0796070, MRR: 0.3724126, NDCG: 0.1542057, F1: 0.1122926, HIT_RATE: 0.6940620, ARHR_ALL_HITS: 0.6416713, NOVELTY: 0.0185801, AVERAGE_POPULARITY: 0.3854770, DIVERSITY_MEAN_INTER_LIST: 0.8679460, DIVERSITY_HERFINDAHL: 0.9933967, COVERAGE_ITEM: 0.2152420, COVERAGE_ITEM_CORRECT: 0.0759292, COVERAGE_USER: 0.9989553, COVERAGE_USER_CORRECT: 0.6933370, DIVERSITY_GINI: 0.0179952, SHANNON_ENTROPY: 7.9342776, RATIO_DIVERSITY_HERFINDAHL: 0.9941466, RATIO_DIVERSITY_GINI: 0.0924336, RATIO_SHANNON_ENTROPY: 0.7006196, RATIO_AVERAGE_POPULARITY: 1.3374083, RATIO_NOVELTY: 0.1530671, \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "hyperparameter_optimization_CFW_D(URM_train, ICM_tags, W_sparse_CF)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The best solution has a MAP of 0.099, better than the pure CBF\n",
    "\n",
    "### Let's try another ICM, the genres of the movie"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CFW_D_Similarity_Linalg: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 (100.0%), 9218.28 column/sec. Elapsed time 1.16 sec\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.50E-02, nonzero cells 7415187\n",
      "CFW_D_Similarity_Linalg: Content S density: 2.80E-02, nonzero cells 3199455\n",
      "CFW_D_Similarity_Linalg: Content S structure has 411533 out of 3199455 (12.9%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 3.95E+04, average is: 9.59E-02, average over all collaborative data is 8.64E-02\n",
      "Similarity column 10681 (100.0%), 8849.61 column/sec. Elapsed time 1.21 sec\n",
      "EvaluatorHoldout: Processed 69805 (100.0%) in 1.95 min. Users per second: 596\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PRECISION</th>\n",
       "      <th>PRECISION_RECALL_MIN_DEN</th>\n",
       "      <th>RECALL</th>\n",
       "      <th>MAP</th>\n",
       "      <th>MAP_MIN_DEN</th>\n",
       "      <th>MRR</th>\n",
       "      <th>NDCG</th>\n",
       "      <th>F1</th>\n",
       "      <th>HIT_RATE</th>\n",
       "      <th>ARHR_ALL_HITS</th>\n",
       "      <th>...</th>\n",
       "      <th>COVERAGE_ITEM_CORRECT</th>\n",
       "      <th>COVERAGE_USER</th>\n",
       "      <th>COVERAGE_USER_CORRECT</th>\n",
       "      <th>DIVERSITY_GINI</th>\n",
       "      <th>SHANNON_ENTROPY</th>\n",
       "      <th>RATIO_DIVERSITY_HERFINDAHL</th>\n",
       "      <th>RATIO_DIVERSITY_GINI</th>\n",
       "      <th>RATIO_SHANNON_ENTROPY</th>\n",
       "      <th>RATIO_AVERAGE_POPULARITY</th>\n",
       "      <th>RATIO_NOVELTY</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cutoff</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.014363</td>\n",
       "      <td>0.014445</td>\n",
       "      <td>0.002632</td>\n",
       "      <td>0.006524</td>\n",
       "      <td>0.00656</td>\n",
       "      <td>0.028173</td>\n",
       "      <td>0.006822</td>\n",
       "      <td>0.004449</td>\n",
       "      <td>0.063892</td>\n",
       "      <td>0.030264</td>\n",
       "      <td>...</td>\n",
       "      <td>0.03979</td>\n",
       "      <td>0.998955</td>\n",
       "      <td>0.063826</td>\n",
       "      <td>0.027973</td>\n",
       "      <td>8.161773</td>\n",
       "      <td>0.991093</td>\n",
       "      <td>0.143706</td>\n",
       "      <td>0.720714</td>\n",
       "      <td>0.22539</td>\n",
       "      <td>0.06021</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.017841</td>\n",
       "      <td>0.018687</td>\n",
       "      <td>0.006383</td>\n",
       "      <td>0.005187</td>\n",
       "      <td>0.005406</td>\n",
       "      <td>0.037751</td>\n",
       "      <td>0.009484</td>\n",
       "      <td>0.009403</td>\n",
       "      <td>0.137913</td>\n",
       "      <td>0.043869</td>\n",
       "      <td>...</td>\n",
       "      <td>0.055613</td>\n",
       "      <td>0.998955</td>\n",
       "      <td>0.137769</td>\n",
       "      <td>0.035421</td>\n",
       "      <td>8.591438</td>\n",
       "      <td>0.994325</td>\n",
       "      <td>0.181967</td>\n",
       "      <td>0.758655</td>\n",
       "      <td>0.265454</td>\n",
       "      <td>0.118283</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>0.01916</td>\n",
       "      <td>0.0236</td>\n",
       "      <td>0.01393</td>\n",
       "      <td>0.003854</td>\n",
       "      <td>0.004545</td>\n",
       "      <td>0.045298</td>\n",
       "      <td>0.013035</td>\n",
       "      <td>0.016131</td>\n",
       "      <td>0.247776</td>\n",
       "      <td>0.057714</td>\n",
       "      <td>...</td>\n",
       "      <td>0.079955</td>\n",
       "      <td>0.998955</td>\n",
       "      <td>0.247517</td>\n",
       "      <td>0.044169</td>\n",
       "      <td>8.857605</td>\n",
       "      <td>0.995197</td>\n",
       "      <td>0.226904</td>\n",
       "      <td>0.782159</td>\n",
       "      <td>0.293409</td>\n",
       "      <td>0.233044</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 25 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       PRECISION PRECISION_RECALL_MIN_DEN    RECALL       MAP MAP_MIN_DEN  \\\n",
       "cutoff                                                                      \n",
       "5       0.014363                 0.014445  0.002632  0.006524     0.00656   \n",
       "10      0.017841                 0.018687  0.006383  0.005187    0.005406   \n",
       "20       0.01916                   0.0236   0.01393  0.003854    0.004545   \n",
       "\n",
       "             MRR      NDCG        F1  HIT_RATE ARHR_ALL_HITS  ...  \\\n",
       "cutoff                                                        ...   \n",
       "5       0.028173  0.006822  0.004449  0.063892      0.030264  ...   \n",
       "10      0.037751  0.009484  0.009403  0.137913      0.043869  ...   \n",
       "20      0.045298  0.013035  0.016131  0.247776      0.057714  ...   \n",
       "\n",
       "       COVERAGE_ITEM_CORRECT COVERAGE_USER COVERAGE_USER_CORRECT  \\\n",
       "cutoff                                                             \n",
       "5                    0.03979      0.998955              0.063826   \n",
       "10                  0.055613      0.998955              0.137769   \n",
       "20                  0.079955      0.998955              0.247517   \n",
       "\n",
       "       DIVERSITY_GINI SHANNON_ENTROPY RATIO_DIVERSITY_HERFINDAHL  \\\n",
       "cutoff                                                             \n",
       "5            0.027973        8.161773                   0.991093   \n",
       "10           0.035421        8.591438                   0.994325   \n",
       "20           0.044169        8.857605                   0.995197   \n",
       "\n",
       "       RATIO_DIVERSITY_GINI RATIO_SHANNON_ENTROPY RATIO_AVERAGE_POPULARITY  \\\n",
       "cutoff                                                                       \n",
       "5                  0.143706              0.720714                  0.22539   \n",
       "10                 0.181967              0.758655                 0.265454   \n",
       "20                 0.226904              0.782159                 0.293409   \n",
       "\n",
       "       RATIO_NOVELTY  \n",
       "cutoff                \n",
       "5            0.06021  \n",
       "10          0.118283  \n",
       "20          0.233044  \n",
       "\n",
       "[3 rows x 25 columns]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "CFW_weithing_genre = CFW_D_Similarity_Linalg(URM_train, ICM_genres, W_sparse_CF)\n",
    "CFW_weithing_genre.fit()\n",
    "\n",
    "results_df, _ = evaluator_test.evaluateRecommender(CFW_weithing_genre)\n",
    "results_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration No: 1 started. Evaluating function at random point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 169, 'add_zeros_quota': 0.8254251929198315, 'normalize_similarity': False}\n",
      "CFW_D_Similarity_Linalg: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 (100.0%), 8737.98 column/sec. Elapsed time 1.22 sec\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.50E-02, nonzero cells 7415187\n",
      "CFW_D_Similarity_Linalg: Content S density: 1.58E-02, nonzero cells 1804515\n",
      "CFW_D_Similarity_Linalg: Content S structure has 246314 out of 1804515 (13.6%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 2.39E+04, average is: 9.72E-02, average over all collaborative data is 8.64E-02\n",
      "Similarity column 10681 (100.0%), 9176.54 column/sec. Elapsed time 1.16 sec\n",
      "EvaluatorHoldout: Processed 67415 (100.0%) in 1.95 min. Users per second: 576\n",
      "SearchBayesianSkopt: New best config found. Config 0: {'topK': 169, 'add_zeros_quota': 0.8254251929198315, 'normalize_similarity': False} - results: PRECISION: 0.0090722, PRECISION_RECALL_MIN_DEN: 0.0120502, RECALL: 0.0079695, MAP: 0.0024283, MAP_MIN_DEN: 0.0031917, MRR: 0.0204502, NDCG: 0.0068715, F1: 0.0084852, HIT_RATE: 0.0791219, ARHR_ALL_HITS: 0.0222402, NOVELTY: 0.0126038, AVERAGE_POPULARITY: 0.0878599, DIVERSITY_MEAN_INTER_LIST: 0.9439841, DIVERSITY_HERFINDAHL: 0.9943970, COVERAGE_ITEM: 0.3320850, COVERAGE_ITEM_CORRECT: 0.0376369, COVERAGE_USER: 0.9647529, COVERAGE_USER_CORRECT: 0.0763330, DIVERSITY_GINI: 0.0326237, SHANNON_ENTROPY: 8.6081439, RATIO_DIVERSITY_HERFINDAHL: 0.9951476, RATIO_DIVERSITY_GINI: 0.1675954, RATIO_SHANNON_ENTROPY: 0.7601306, RATIO_AVERAGE_POPULARITY: 0.3049996, RATIO_NOVELTY: 0.1153711, \n",
      "\n",
      "Iteration No: 1 ended. Evaluation done at random point.\n",
      "Time taken: 138.0440\n",
      "Function value obtained: -0.0024\n",
      "Current minimum: -0.0024\n",
      "Iteration No: 2 started. Evaluating function at random point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 132, 'add_zeros_quota': 0.8139889412402458, 'normalize_similarity': True}\n",
      "CFW_D_Similarity_Linalg: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 (100.0%), 8204.49 column/sec. Elapsed time 1.30 sec\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.50E-02, nonzero cells 7415187\n",
      "CFW_D_Similarity_Linalg: Content S density: 1.24E-02, nonzero cells 1409656\n",
      "CFW_D_Similarity_Linalg: Content S structure has 197873 out of 1409656 (14.0%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 5.49E+04, average is: 2.77E-01, average over all collaborative data is 8.64E-02\n",
      "Similarity column 10681 (100.0%), 5746.43 column/sec. Elapsed time 1.86 sec\n",
      "EvaluatorHoldout: Processed 67415 (100.0%) in 1.92 min. Users per second: 586\n",
      "SearchBayesianSkopt: Config 1 is suboptimal. Config: {'topK': 132, 'add_zeros_quota': 0.8139889412402458, 'normalize_similarity': True} - results: PRECISION: 0.0066395, PRECISION_RECALL_MIN_DEN: 0.0091043, RECALL: 0.0061606, MAP: 0.0020556, MAP_MIN_DEN: 0.0027808, MRR: 0.0182104, NDCG: 0.0054100, F1: 0.0063911, HIT_RATE: 0.0598828, ARHR_ALL_HITS: 0.0193157, NOVELTY: 0.0131557, AVERAGE_POPULARITY: 0.0738989, DIVERSITY_MEAN_INTER_LIST: 0.9697216, DIVERSITY_HERFINDAHL: 0.9969707, COVERAGE_ITEM: 0.4449022, COVERAGE_ITEM_CORRECT: 0.0416628, COVERAGE_USER: 0.9647529, COVERAGE_USER_CORRECT: 0.0577721, DIVERSITY_GINI: 0.0463196, SHANNON_ENTROPY: 9.1771664, RATIO_DIVERSITY_HERFINDAHL: 0.9977232, RATIO_DIVERSITY_GINI: 0.2379548, RATIO_SHANNON_ENTROPY: 0.8103773, RATIO_AVERAGE_POPULARITY: 0.2565350, RATIO_NOVELTY: 0.1204230, \n",
      "\n",
      "Iteration No: 2 ended. Evaluation done at random point.\n",
      "Time taken: 134.6666\n",
      "Function value obtained: -0.0021\n",
      "Current minimum: -0.0024\n",
      "Iteration No: 3 started. Evaluating function at random point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 633, 'add_zeros_quota': 0.2787831312503715, 'normalize_similarity': True}\n",
      "CFW_D_Similarity_Linalg: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 (100.0%), 8897.25 column/sec. Elapsed time 1.20 sec\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.50E-02, nonzero cells 7415187\n",
      "CFW_D_Similarity_Linalg: Content S density: 5.83E-02, nonzero cells 6655324\n",
      "CFW_D_Similarity_Linalg: Generating train data. Sample 2092315 (40.2%) \n",
      "CFW_D_Similarity_Linalg: Content S structure has 756432 out of 6655324 (11.4%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 1.88E+05, average is: 2.49E-01, average over all collaborative data is 8.64E-02\n",
      "Similarity column 10681 (100.0%), 5665.58 column/sec. Elapsed time 1.89 sec\n",
      "EvaluatorHoldout: Processed 67415 (100.0%) in 2.04 min. Users per second: 550\n",
      "SearchBayesianSkopt: Config 2 is suboptimal. Config: {'topK': 633, 'add_zeros_quota': 0.2787831312503715, 'normalize_similarity': True} - results: PRECISION: 0.0069970, PRECISION_RECALL_MIN_DEN: 0.0091019, RECALL: 0.0057592, MAP: 0.0021486, MAP_MIN_DEN: 0.0027267, MRR: 0.0188062, NDCG: 0.0053377, F1: 0.0063180, HIT_RATE: 0.0622117, ARHR_ALL_HITS: 0.0200635, NOVELTY: 0.0132480, AVERAGE_POPULARITY: 0.0711461, DIVERSITY_MEAN_INTER_LIST: 0.9622420, DIVERSITY_HERFINDAHL: 0.9962228, COVERAGE_ITEM: 0.5462972, COVERAGE_ITEM_CORRECT: 0.0482165, COVERAGE_USER: 0.9647529, COVERAGE_USER_CORRECT: 0.0600189, DIVERSITY_GINI: 0.0636704, SHANNON_ENTROPY: 9.4408902, RATIO_DIVERSITY_HERFINDAHL: 0.9969747, RATIO_DIVERSITY_GINI: 0.3270898, RATIO_SHANNON_ENTROPY: 0.8336651, RATIO_AVERAGE_POPULARITY: 0.2469787, RATIO_NOVELTY: 0.1212678, \n",
      "\n",
      "Iteration No: 3 ended. Evaluation done at random point.\n",
      "Time taken: 168.2925\n",
      "Function value obtained: -0.0021\n",
      "Current minimum: -0.0024\n",
      "Iteration No: 4 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 864, 'add_zeros_quota': 0.995848437195078, 'normalize_similarity': True}\n",
      "CFW_D_Similarity_Linalg: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 (100.0%), 8164.69 column/sec. Elapsed time 1.31 sec\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.50E-02, nonzero cells 7415187\n",
      "CFW_D_Similarity_Linalg: Content S density: 7.87E-02, nonzero cells 8982987\n",
      "CFW_D_Similarity_Linalg: Generating train data. Sample 4895574 (108.8%) \n",
      "CFW_D_Similarity_Linalg: Content S structure has 998241 out of 8982987 (11.1%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 2.43E+05, average is: 2.43E-01, average over all collaborative data is 8.64E-02\n",
      "Similarity column 10681 (100.0%), 5013.63 column/sec. Elapsed time 2.13 sec\n",
      "EvaluatorHoldout: Processed 67415 (100.0%) in 2.24 min. Users per second: 502\n",
      "SearchBayesianSkopt: Config 3 is suboptimal. Config: {'topK': 864, 'add_zeros_quota': 0.995848437195078, 'normalize_similarity': True} - results: PRECISION: 0.0067389, PRECISION_RECALL_MIN_DEN: 0.0091986, RECALL: 0.0060868, MAP: 0.0019848, MAP_MIN_DEN: 0.0027111, MRR: 0.0173413, NDCG: 0.0053566, F1: 0.0063963, HIT_RATE: 0.0599125, ARHR_ALL_HITS: 0.0185276, NOVELTY: 0.0132183, AVERAGE_POPULARITY: 0.0708674, DIVERSITY_MEAN_INTER_LIST: 0.9712301, DIVERSITY_HERFINDAHL: 0.9971216, COVERAGE_ITEM: 0.4620354, COVERAGE_ITEM_CORRECT: 0.0445651, COVERAGE_USER: 0.9647529, COVERAGE_USER_CORRECT: 0.0578007, DIVERSITY_GINI: 0.0568847, SHANNON_ENTROPY: 9.4427795, RATIO_DIVERSITY_HERFINDAHL: 0.9978742, RATIO_DIVERSITY_GINI: 0.2922303, RATIO_SHANNON_ENTROPY: 0.8338319, RATIO_AVERAGE_POPULARITY: 0.2460112, RATIO_NOVELTY: 0.1209961, \n",
      "\n",
      "Iteration No: 4 ended. Search finished for the next optimal point.\n",
      "Time taken: 221.4908\n",
      "Function value obtained: -0.0020\n",
      "Current minimum: -0.0024\n",
      "Iteration No: 5 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 498, 'add_zeros_quota': 0.24899523372624272, 'normalize_similarity': False}\n",
      "CFW_D_Similarity_Linalg: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 (100.0%), 8189.30 column/sec. Elapsed time 1.30 sec\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.50E-02, nonzero cells 7415187\n",
      "CFW_D_Similarity_Linalg: Content S density: 4.63E-02, nonzero cells 5284995\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CFW_D_Similarity_Linalg: Content S structure has 618403 out of 5284995 (11.7%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 5.86E+04, average is: 9.48E-02, average over all collaborative data is 8.64E-02\n",
      "Similarity column 10681 (100.0%), 8352.41 column/sec. Elapsed time 1.28 sec\n",
      "EvaluatorHoldout: Processed 67415 (100.0%) in 1.89 min. Users per second: 593\n",
      "SearchBayesianSkopt: Config 4 is suboptimal. Config: {'topK': 498, 'add_zeros_quota': 0.24899523372624272, 'normalize_similarity': False} - results: PRECISION: 0.0047215, PRECISION_RECALL_MIN_DEN: 0.0066265, RECALL: 0.0045562, MAP: 0.0012892, MAP_MIN_DEN: 0.0017591, MRR: 0.0116782, NDCG: 0.0036046, F1: 0.0046374, HIT_RATE: 0.0433286, ARHR_ALL_HITS: 0.0122695, NOVELTY: 0.0131355, AVERAGE_POPULARITY: 0.0587949, DIVERSITY_MEAN_INTER_LIST: 0.8996842, DIVERSITY_HERFINDAHL: 0.9899671, COVERAGE_ITEM: 0.2857410, COVERAGE_ITEM_CORRECT: 0.0311769, COVERAGE_USER: 0.9647529, COVERAGE_USER_CORRECT: 0.0418014, DIVERSITY_GINI: 0.0231952, SHANNON_ENTROPY: 7.9379991, RATIO_DIVERSITY_HERFINDAHL: 0.9907143, RATIO_DIVERSITY_GINI: 0.1191591, RATIO_SHANNON_ENTROPY: 0.7009543, RATIO_AVERAGE_POPULARITY: 0.2041024, RATIO_NOVELTY: 0.1202385, \n",
      "\n",
      "Iteration No: 5 ended. Search finished for the next optimal point.\n",
      "Time taken: 146.4180\n",
      "Function value obtained: -0.0013\n",
      "Current minimum: -0.0024\n",
      "Iteration No: 6 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 465, 'add_zeros_quota': 0.0004786224680490837, 'normalize_similarity': True}\n",
      "CFW_D_Similarity_Linalg: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 (100.0%), 7983.78 column/sec. Elapsed time 1.34 sec\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.50E-02, nonzero cells 7415187\n",
      "CFW_D_Similarity_Linalg: Content S density: 4.33E-02, nonzero cells 4943055\n",
      "CFW_D_Similarity_Linalg: Generating train data. Sample 552112 (11.2%) \n",
      "CFW_D_Similarity_Linalg: Content S structure has 583033 out of 4943055 (11.8%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 1.48E+05, average is: 2.54E-01, average over all collaborative data is 8.64E-02\n",
      "Similarity column 10681 (100.0%), 4919.31 column/sec. Elapsed time 2.17 sec\n",
      "EvaluatorHoldout: Processed 67415 (100.0%) in 2.13 min. Users per second: 527\n",
      "SearchBayesianSkopt: Config 5 is suboptimal. Config: {'topK': 465, 'add_zeros_quota': 0.0004786224680490837, 'normalize_similarity': True} - results: PRECISION: 0.0071186, PRECISION_RECALL_MIN_DEN: 0.0094222, RECALL: 0.0061504, MAP: 0.0020783, MAP_MIN_DEN: 0.0027693, MRR: 0.0181703, NDCG: 0.0055903, F1: 0.0065992, HIT_RATE: 0.0633835, ARHR_ALL_HITS: 0.0194166, NOVELTY: 0.0129759, AVERAGE_POPULARITY: 0.0759272, DIVERSITY_MEAN_INTER_LIST: 0.9656231, DIVERSITY_HERFINDAHL: 0.9965609, COVERAGE_ITEM: 0.7411291, COVERAGE_ITEM_CORRECT: 0.0628218, COVERAGE_USER: 0.9647529, COVERAGE_USER_CORRECT: 0.0611494, DIVERSITY_GINI: 0.0984795, SHANNON_ENTROPY: 9.8452438, RATIO_DIVERSITY_HERFINDAHL: 0.9973131, RATIO_DIVERSITY_GINI: 0.5059124, RATIO_SHANNON_ENTROPY: 0.8693710, RATIO_AVERAGE_POPULARITY: 0.2635759, RATIO_NOVELTY: 0.1187771, \n",
      "\n",
      "Iteration No: 6 ended. Search finished for the next optimal point.\n",
      "Time taken: 167.4954\n",
      "Function value obtained: -0.0021\n",
      "Current minimum: -0.0024\n",
      "Iteration No: 7 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 419, 'add_zeros_quota': 0.001090728475234615, 'normalize_similarity': False}\n",
      "CFW_D_Similarity_Linalg: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 (100.0%), 7148.07 column/sec. Elapsed time 1.49 sec\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.50E-02, nonzero cells 7415187\n",
      "CFW_D_Similarity_Linalg: Content S density: 3.91E-02, nonzero cells 4457523\n",
      "CFW_D_Similarity_Linalg: Content S structure has 534694 out of 4457523 (12.0%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 5.08E+04, average is: 9.49E-02, average over all collaborative data is 8.64E-02\n",
      "Similarity column 10681 (100.0%), 8708.33 column/sec. Elapsed time 1.23 sec\n",
      "EvaluatorHoldout: Processed 67415 (100.0%) in 2.25 min. Users per second: 500\n",
      "SearchBayesianSkopt: Config 6 is suboptimal. Config: {'topK': 419, 'add_zeros_quota': 0.001090728475234615, 'normalize_similarity': False} - results: PRECISION: 0.0065178, PRECISION_RECALL_MIN_DEN: 0.0088401, RECALL: 0.0059055, MAP: 0.0016855, MAP_MIN_DEN: 0.0022396, MRR: 0.0148580, NDCG: 0.0048284, F1: 0.0061966, HIT_RATE: 0.0586220, ARHR_ALL_HITS: 0.0158013, NOVELTY: 0.0130630, AVERAGE_POPULARITY: 0.0690817, DIVERSITY_MEAN_INTER_LIST: 0.9263670, DIVERSITY_HERFINDAHL: 0.9926353, COVERAGE_ITEM: 0.3453796, COVERAGE_ITEM_CORRECT: 0.0362326, COVERAGE_USER: 0.9647529, COVERAGE_USER_CORRECT: 0.0565557, DIVERSITY_GINI: 0.0289266, SHANNON_ENTROPY: 8.3194378, RATIO_DIVERSITY_HERFINDAHL: 0.9933846, RATIO_DIVERSITY_GINI: 0.1486027, RATIO_SHANNON_ENTROPY: 0.7346368, RATIO_AVERAGE_POPULARITY: 0.2398125, RATIO_NOVELTY: 0.1195751, \n",
      "\n",
      "Iteration No: 7 ended. Search finished for the next optimal point.\n",
      "Time taken: 163.5290\n",
      "Function value obtained: -0.0017\n",
      "Current minimum: -0.0024\n",
      "Iteration No: 8 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 180, 'add_zeros_quota': 0.9532206348399387, 'normalize_similarity': True}\n",
      "CFW_D_Similarity_Linalg: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 (100.0%), 6532.61 column/sec. Elapsed time 1.64 sec\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.50E-02, nonzero cells 7415187\n",
      "CFW_D_Similarity_Linalg: Content S density: 1.68E-02, nonzero cells 1921852\n",
      "CFW_D_Similarity_Linalg: Content S structure has 260115 out of 1921852 (13.5%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 7.05E+04, average is: 2.71E-01, average over all collaborative data is 8.64E-02\n",
      "Similarity column 10681 (100.0%), 4708.97 column/sec. Elapsed time 2.27 sec\n",
      "EvaluatorHoldout: Processed 67415 (100.0%) in 2.14 min. Users per second: 525\n",
      "SearchBayesianSkopt: Config 7 is suboptimal. Config: {'topK': 180, 'add_zeros_quota': 0.9532206348399387, 'normalize_similarity': True} - results: PRECISION: 0.0065060, PRECISION_RECALL_MIN_DEN: 0.0086015, RECALL: 0.0056189, MAP: 0.0020036, MAP_MIN_DEN: 0.0026184, MRR: 0.0174369, NDCG: 0.0051459, F1: 0.0060300, HIT_RATE: 0.0577468, ARHR_ALL_HITS: 0.0186666, NOVELTY: 0.0131785, AVERAGE_POPULARITY: 0.0683210, DIVERSITY_MEAN_INTER_LIST: 0.9691784, DIVERSITY_HERFINDAHL: 0.9969164, COVERAGE_ITEM: 0.4780451, COVERAGE_ITEM_CORRECT: 0.0440034, COVERAGE_USER: 0.9647529, COVERAGE_USER_CORRECT: 0.0557114, DIVERSITY_GINI: 0.0472099, SHANNON_ENTROPY: 9.1639581, RATIO_DIVERSITY_HERFINDAHL: 0.9976689, RATIO_DIVERSITY_GINI: 0.2425286, RATIO_SHANNON_ENTROPY: 0.8092110, RATIO_AVERAGE_POPULARITY: 0.2371718, RATIO_NOVELTY: 0.1206316, \n",
      "\n",
      "Iteration No: 8 ended. Search finished for the next optimal point.\n",
      "Time taken: 155.4552\n",
      "Function value obtained: -0.0020\n",
      "Current minimum: -0.0024\n",
      "Iteration No: 9 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 601, 'add_zeros_quota': 1.0, 'normalize_similarity': False}\n",
      "CFW_D_Similarity_Linalg: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 (100.0%), 5655.38 column/sec. Elapsed time 1.89 sec\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.50E-02, nonzero cells 7415187\n",
      "CFW_D_Similarity_Linalg: Content S density: 5.55E-02, nonzero cells 6330834\n",
      "CFW_D_Similarity_Linalg: Generating train data. Sample 3671926 (116.0%) \n",
      "CFW_D_Similarity_Linalg: Content S structure has 725625 out of 6330834 (11.5%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 6.85E+04, average is: 9.44E-02, average over all collaborative data is 8.64E-02\n",
      "Similarity column 10681 (100.0%), 6301.31 column/sec. Elapsed time 1.70 sec\n",
      "EvaluatorHoldout: Processed 67415 (100.0%) in 2.44 min. Users per second: 460\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SearchBayesianSkopt: Config 8 is suboptimal. Config: {'topK': 601, 'add_zeros_quota': 1.0, 'normalize_similarity': False} - results: PRECISION: 0.0047037, PRECISION_RECALL_MIN_DEN: 0.0066686, RECALL: 0.0046086, MAP: 0.0013318, MAP_MIN_DEN: 0.0018123, MRR: 0.0122015, NDCG: 0.0036554, F1: 0.0046556, HIT_RATE: 0.0435808, ARHR_ALL_HITS: 0.0127332, NOVELTY: 0.0130938, AVERAGE_POPULARITY: 0.0590907, DIVERSITY_MEAN_INTER_LIST: 0.8894659, DIVERSITY_HERFINDAHL: 0.9889453, COVERAGE_ITEM: 0.2525981, COVERAGE_ITEM_CORRECT: 0.0280873, COVERAGE_USER: 0.9647529, COVERAGE_USER_CORRECT: 0.0420447, DIVERSITY_GINI: 0.0192704, SHANNON_ENTROPY: 7.6984532, RATIO_DIVERSITY_HERFINDAHL: 0.9896917, RATIO_DIVERSITY_GINI: 0.0989964, RATIO_SHANNON_ENTROPY: 0.6798016, RATIO_AVERAGE_POPULARITY: 0.2051292, RATIO_NOVELTY: 0.1198570, \n",
      "\n",
      "Iteration No: 9 ended. Search finished for the next optimal point.\n",
      "Time taken: 211.6807\n",
      "Function value obtained: -0.0013\n",
      "Current minimum: -0.0024\n",
      "Iteration No: 10 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 27, 'add_zeros_quota': 0.1740155453190461, 'normalize_similarity': False}\n",
      "CFW_D_Similarity_Linalg: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 (100.0%), 5655.32 column/sec. Elapsed time 1.89 sec\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.50E-02, nonzero cells 7415187\n",
      "CFW_D_Similarity_Linalg: Content S density: 2.53E-03, nonzero cells 288360\n",
      "CFW_D_Similarity_Linalg: Content S structure has 43743 out of 288360 (15.2%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 4.46E+03, average is: 1.02E-01, average over all collaborative data is 8.64E-02\n",
      "Similarity column 10681 (100.0%), 8979.98 column/sec. Elapsed time 1.19 sec\n",
      "EvaluatorHoldout: Processed 67415 (100.0%) in 2.10 min. Users per second: 534\n",
      "SearchBayesianSkopt: Config 9 is suboptimal. Config: {'topK': 27, 'add_zeros_quota': 0.1740155453190461, 'normalize_similarity': False} - results: PRECISION: 0.0081644, PRECISION_RECALL_MIN_DEN: 0.0109241, RECALL: 0.0072306, MAP: 0.0024163, MAP_MIN_DEN: 0.0032189, MRR: 0.0210431, NDCG: 0.0066738, F1: 0.0076692, HIT_RATE: 0.0721204, ARHR_ALL_HITS: 0.0225130, NOVELTY: 0.0128445, AVERAGE_POPULARITY: 0.0758535, DIVERSITY_MEAN_INTER_LIST: 0.9544176, DIVERSITY_HERFINDAHL: 0.9954403, COVERAGE_ITEM: 0.3481884, COVERAGE_ITEM_CORRECT: 0.0478420, COVERAGE_USER: 0.9647529, COVERAGE_USER_CORRECT: 0.0695784, DIVERSITY_GINI: 0.0477907, SHANNON_ENTROPY: 9.1223096, RATIO_DIVERSITY_HERFINDAHL: 0.9961917, RATIO_DIVERSITY_GINI: 0.2455120, RATIO_SHANNON_ENTROPY: 0.8055333, RATIO_AVERAGE_POPULARITY: 0.2633203, RATIO_NOVELTY: 0.1175743, \n",
      "\n",
      "Iteration No: 10 ended. Search finished for the next optimal point.\n",
      "Time taken: 134.8778\n",
      "Function value obtained: -0.0024\n",
      "Current minimum: -0.0024\n",
      "SearchBayesianSkopt: Search complete. Best config is 0: {'topK': 169, 'add_zeros_quota': 0.8254251929198315, 'normalize_similarity': False}\n",
      "\n",
      "CFW_D_Similarity_Linalg: URM Detected 36 ( 0.3%) items with no interactions.\n",
      "SearchBayesianSkopt: Evaluation with constructor data for final test. Using best config: {'topK': 169, 'add_zeros_quota': 0.8254251929198315, 'normalize_similarity': False}\n",
      "CFW_D_Similarity_Linalg: Generating train data\n",
      "Similarity column 10681 (100.0%), 5587.31 column/sec. Elapsed time 1.91 sec\n",
      "CFW_D_Similarity_Linalg: Collaborative S density: 6.50E-02, nonzero cells 7415187\n",
      "CFW_D_Similarity_Linalg: Content S density: 1.58E-02, nonzero cells 1804515\n",
      "CFW_D_Similarity_Linalg: Content S structure has 246314 out of 1804515 (13.6%) nonzero collaborative cells\n",
      "CFW_D_Similarity_Linalg: Nonzero collaborative cell sum is: 2.39E+04, average is: 9.72E-02, average over all collaborative data is 8.64E-02\n",
      "Similarity column 10681 (100.0%), 8593.71 column/sec. Elapsed time 1.24 sec\n",
      "EvaluatorHoldout: Processed 69805 (100.0%) in 2.70 min. Users per second: 431\n",
      "SearchBayesianSkopt: Best config evaluated with evaluator_test with constructor data for final test. Config: {'topK': 169, 'add_zeros_quota': 0.8254251929198315, 'normalize_similarity': False} - results:\n",
      "CUTOFF: 5 - PRECISION: 0.0208180, PRECISION_RECALL_MIN_DEN: 0.0209574, RECALL: 0.0037367, MAP: 0.0087921, MAP_MIN_DEN: 0.0088418, MRR: 0.0341953, NDCG: 0.0096245, F1: 0.0063360, HIT_RATE: 0.0868562, ARHR_ALL_HITS: 0.0386878, NOVELTY: 0.0063844, AVERAGE_POPULARITY: 0.0776887, DIVERSITY_MEAN_INTER_LIST: 0.9523280, DIVERSITY_HERFINDAHL: 0.9904629, COVERAGE_ITEM: 0.2675779, COVERAGE_ITEM_CORRECT: 0.0354836, COVERAGE_USER: 0.9989553, COVERAGE_USER_CORRECT: 0.0867655, DIVERSITY_GINI: 0.0260874, SHANNON_ENTROPY: 8.1644616, RATIO_DIVERSITY_HERFINDAHL: 0.9912105, RATIO_DIVERSITY_GINI: 0.1340000, RATIO_SHANNON_ENTROPY: 0.7209455, RATIO_AVERAGE_POPULARITY: 0.2695401, RATIO_NOVELTY: 0.0525964, \n",
      "CUTOFF: 10 - PRECISION: 0.0224869, PRECISION_RECALL_MIN_DEN: 0.0236929, RECALL: 0.0082442, MAP: 0.0068393, MAP_MIN_DEN: 0.0071379, MRR: 0.0442551, NDCG: 0.0121403, F1: 0.0120651, HIT_RATE: 0.1634983, ARHR_ALL_HITS: 0.0543572, NOVELTY: 0.0126229, AVERAGE_POPULARITY: 0.0864468, DIVERSITY_MEAN_INTER_LIST: 0.9421911, DIVERSITY_HERFINDAHL: 0.9942178, COVERAGE_ITEM: 0.3259994, COVERAGE_ITEM_CORRECT: 0.0506507, COVERAGE_USER: 0.9989553, COVERAGE_USER_CORRECT: 0.1633275, DIVERSITY_GINI: 0.0315888, SHANNON_ENTROPY: 8.5616682, RATIO_DIVERSITY_HERFINDAHL: 0.9949683, RATIO_DIVERSITY_GINI: 0.1622586, RATIO_SHANNON_ENTROPY: 0.7560200, RATIO_AVERAGE_POPULARITY: 0.2999262, RATIO_NOVELTY: 0.1039910, \n",
      "CUTOFF: 20 - PRECISION: 0.0216525, PRECISION_RECALL_MIN_DEN: 0.0269219, RECALL: 0.0161220, MAP: 0.0048067, MAP_MIN_DEN: 0.0056855, MRR: 0.0514224, NDCG: 0.0153649, F1: 0.0184824, HIT_RATE: 0.2670153, ARHR_ALL_HITS: 0.0685094, NOVELTY: 0.0252742, AVERAGE_POPULARITY: 0.0887921, DIVERSITY_MEAN_INTER_LIST: 0.9203425, DIVERSITY_HERFINDAHL: 0.9960165, COVERAGE_ITEM: 0.3929407, COVERAGE_ITEM_CORRECT: 0.0708735, COVERAGE_USER: 0.9989553, COVERAGE_USER_CORRECT: 0.2667363, DIVERSITY_GINI: 0.0384301, SHANNON_ENTROPY: 8.9045079, RATIO_DIVERSITY_HERFINDAHL: 0.9967683, RATIO_DIVERSITY_GINI: 0.1973995, RATIO_SHANNON_ENTROPY: 0.7862938, RATIO_AVERAGE_POPULARITY: 0.3080632, RATIO_NOVELTY: 0.2082149, \n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "hyperparameter_optimization_CFW_D(URM_train, ICM_genres, W_sparse_CF)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Here the best solution has a MAP of 0.006... not great"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we are weighting features, and weights can be applied only to common features, we may expect a sparse ICM to have limited quality gains because the items will tend to have few features in common and so only few parameters that can be learned. A more dense ICM may exhibit much higher improvements due to more common features and model parameters, however, very different items may have the same features in common, choking the model.\n",
    "\n",
    "### Improvements of CF+CBF greatly depend on the ICM structure and collaborative data distribution...\n",
    "\n",
    "## Tip:\n",
    "If your ICM is very sparse and you get poor results, you may try to aggregate features (e.g., put them in OR, add them...) to reduce their number and get a new denser ICM on which to try feature weighting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How about using these feature weights to perform feature selection?\n",
    "## Let's try to select the x% features with highest value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEJCAYAAACKWmBmAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAg5ElEQVR4nO3de3RdZbnv8e/TXJo0adqmTdv0ElKkpRQEWmJBRESgiBUteAOOR9GDp7qRoR5lKw722MfbGODWwdBz1I3dyLYqCm5BqVgOcr9T2kIpvdGWFnqll6S3pLmv5/wxZ8rsysrKpVmZKyu/zxhrrHl515zPm7Tryfu+c77T3B0REZGuDIs7ABERyW5KFCIikpYShYiIpKVEISIiaSlRiIhIWkoUIiKSVqyJwswuN7PXzWyzmd2cYv8CM1ttZqvMbIWZXRBHnCIiQ5nFdR+FmeUBG4F5wA5gOXCtu6+LlCkFGtzdzexM4E/uPjOWgEVEhqj8GM89F9js7lsAzOweYAFwLFG4e32kfAnQo6w2btw4r66u7r9IRURy3MqVK/e7e0WqfXEmisnA9sj6DuDc5EJmdhVwKzAe+EhPDlxdXc2KFSv6I0YRkSHBzN7qal+cYxSWYlunFoO7/yXsbroS+EGXBzNbGI5jrNi3b1//RSkiMsTFmSh2AFMj61OAXV0VdvengXeZ2bgu9i9y9xp3r6moSNl6EhGRPogzUSwHppvZNDMrBK4BlkQLmNkpZmbh8hygEKgd8EhFRIaw2MYo3L3NzG4EHgbygLvcfa2ZfTncfwfwCeBzZtYKNAJXu6a7FREZULFdHptJNTU1rsFsEZGeM7OV7l6Tap/uzBYRkbSUKEREJC0lChGRHPDouj3c8dQbGTm2EoWISA54bMNe7nxma0aOrUQhIpITHEt1G3M/UKIQEckB7qmnu+gPShQiIjnAHbUoRESkawl3hmUoUyhRiIjkAEddTyIikkbQ9aQWhYiIdMF79ly3PlGiEBHJBRrMFhGRdBwlChERScPdsQwNZytRiIjkALUoREQkLd2ZLSIiaemGOxERScshY00KJQoRkVygricREUnHcd2ZLSIiXdNgtoiIpJWz04yb2eVm9rqZbTazm1Ps/4yZrQ5fz5vZWXHEKSKS7ZwcvOHOzPKAXwAfBmYB15rZrKRiW4EPuPuZwA+ARQMbpYjI4JCrLYq5wGZ33+LuLcA9wIJoAXd/3t0PhKsvAlMGOEYRkUGhqS1BYX5mvtLjTBSTge2R9R3htq5cDzyU0YhERAaplrZ2ivLzMnLs/IwctWdSNZJSTqhuZh8kSBQXdHkws4XAQoCqqqr+iE9EZNBoa/ecbFHsAKZG1qcAu5ILmdmZwJ3AAnev7epg7r7I3WvcvaaioqLfgxURyWat7QkK8nIvUSwHppvZNDMrBK4BlkQLmFkVcD/wWXffGEOMIiKDQku7ZyxRxNb15O5tZnYj8DCQB9zl7mvN7Mvh/juAfwXGAr8M7zhsc/eauGIWEclWbe0JCvMzc9lTnGMUuPtSYGnStjsiy18EvjjQcYmIDDat7Qnyh+Ve15OIiPSTxtZ2igqUKEREpAsNze2UDM9MJ5EShYjIIOfuNLS0UapEISIiqRxtaccdtShERCS1uoYWAEoKM3NnthKFiMggVxsmilEjCjNyfCUKEZFBrra+GYCq8hEZOb4ShYjIILc/TBTjStWiEBGRFN4+1JEohmfk+EoUIiKD3L76JspLCikq0GC2iIikUN/UxsiizM3IpEQhIjLI1Tdn7mY7UKIQERn0jjQpUYiISBq1DS2MKi7I2PGVKEREBrGG5ja27KvntMqyjJ1DiUJEZBBbtrWWhMNZU0dl7BxKFCIig9i22qMAzJgwMmPnUKIQERnEVm0/yJgRBUwaVZyxcyhRiIgMUm3tCf66ahfnnFTOsGGZeV42KFGIiAxaj2/YC8D7p4/L6HmUKEREBqkHVu1iVHEB186tyuh5Yk0UZna5mb1uZpvN7OYU+2ea2Qtm1mxmN8URo4hINkoknGc37+eyWRMozM/sV3nmbuXrhpnlAb8A5gE7gOVmtsTd10WK1QFfBa4c+AhFRLLXI+v3cKixlfNPGZvxc8XZopgLbHb3Le7eAtwDLIgWcPe97r4caI0jQBGRbOTuLHp6CyeNHcEVZ07K+PniTBSTge2R9R3hNhERSWPD20dY+dYBPnveSRTkZf5rPM5EkepaLu/zwcwWmtkKM1uxb9++EwhLRCS7/edzWykqGMZVswfmb+s4E8UOYGpkfQqwq68Hc/dF7l7j7jUVFRUnHJyISDZ6+1ATf3llJ5+umcrYDD3RLlmciWI5MN3MpplZIXANsCTGeEREspq784MH15Fw+OIFJw/YeWO76snd28zsRuBhIA+4y93XmtmXw/13mNlEYAVQBiTM7OvALHc/HFfcIiJxeXD1bv7+2m7++UOnUjV2xICdN7ZEAeDuS4GlSdvuiCy/TdAlJSIypB082sJtD21gxoRSvnThwLUmIOZEISIi3TvS1MqXfreSXYcaufv6c8kfgCudojSFh4hIFmtua+eGu19mxVsH+Mknz+L8UzI7r1MqShQiIlnqSFMrC3+7kmc27efWq97NJ86JpydeXU8iIlnoQEML//O3K3hl+0Fu/fi7+fR7pnb/oQxRohARyTL3Lt/GDx5cz9GWNn52zWw+elbmp+lIR4lCRCRLNLW2872/reWPL21nbnU5P7jyDE6dmLlHnPaUEoWISMzcnYfXvs1tD23gzdqjfOkDJ/OtD80kL4NPresNJQoRkZi4O8u21vGzRzfxwpZaThlfyu+vP5cLMvzEut5SohARGWDuzlMb9/Grp7bwwpZaxowo4HsfO53PnFs14PdI9IQShYjIANled5R7lm9j6Wtvs3V/A+UlhXx/wel8Ys4USoZn79dx9kYmIpIDDh5tYelrb3PfyztY+dYB8oYZ551czg0XvYsFZ0/O+GNM+4MShYhIP2tPOM+/sZ+/vrKL/7dmNw0t7VSPHcE3583g4+dMYfLo4rhD7BUlChGRftDQ3MarOw7y+Pq9LHl1F3uPNDNyeD4fObOSz723mtMnlWGWHVcx9ZYShYhIHzS2tLNsay3rdh9m+dY6nnujlpa2BAV5xkWnjueq2ZO5eOZ4igry4g71hClRiIj0gLvzxr56nt64n2c27ePFLXU0trYDcPK4Ej5zbhUXzqhgztQxjBpREHO0/UuJQkQkhfaEs3bXIV7aWseancH7rkNNAEwbV8KnaqYwb9YEzpg0ijElhTFHm1lKFCIiwKHGVjbvrWfV9oM8u2kfK948wJHmNgAmlA1nTtUYbry4gvdPH8fU8oF7ulw2UKIQkSHH3dlxoJF1uw+zbEsdL2ypZf3ud56wXD12BB89exJzq8s5/11jGV9WFGO08VOiEJGcdqixlY17jrBpTz0b9xxh/e7DrN99mMNNQWtheP4w5lSN4abLZnBaZRmzJpVROWpwXb6aaUoUIpIT9h5pYtOeerbsq2fz3no27Q3e9x5pPlZmRGEep04cyRVnTeL0SWVBYqgsy4krkzJJiUJEBpWjLW1s3lvP628f4a3ao6zddYjXdh5mf/07CaF0eD7vGl/KhTMqmD6+lOkTSpkxYSSTRhUzLEtmZB1MYk0UZnY58DMgD7jT3W9L2m/h/vnAUeDz7v7ygAcqIgOqrT3BroNNbKs7ypu1Dbyxr5439zewaW89Ow824h6UyxtmnDyuhA/MqGDWpDJmThzJKeNLGT9y+KC9uS0bxZYozCwP+AUwD9gBLDezJe6+LlLsw8D08HUu8O/hu4gMcg3NbWyrO8pbtUfZVtcQWT7KzgONtCX8WNkRhXlUlY9gdtUYPl0zlenjS5lZWcaUMcUUZOFsq7kmzhbFXGCzu28BMLN7gAVANFEsAH7r7g68aGajzazS3XcPfLgi0htHmlrZebCRXQcb2XmgkZ0Hm46tb6s7yr7I2AFAWVE+J40t4YzJo/jIuys5aewIqspLqBo7gsqyInUZxSjORDEZ2B5Z30Hn1kKqMpOBnEsU7s7++hYONbbQ0NzO0ZZ2mlrbSbinKNvFMdIcu6fluzp2V0fvfSxdlT/x43dVz670WywZ/H10dZD++HmlL9+749c3t0USQvA6El5V1KEgz6gcVczk0cVcNKOC6nElVJWPCBPCCEaPyO2b1gazOBNFqj8Pkv8Z9qRMUNBsIbAQoKqq6sQiy7CWtgSv7TzIyrcO8MbeBt6qa2DTnnpqG1riDk2kz8qK8pk0upgpY4qZO62cyaOLmTS6mMljguRQUTpcrYJBqkeJwsx+5O7f7m5bL+0ApkbWpwC7+lAGAHdfBCwCqKmp6d2flhmWSDjrdh/m6U37eHbTfl7edoCm1gQA40qHU1VezMUzx3NaZRnjRg6npDCP4sI8igvyunxmrqXModDb8btU5Xt77C639/Y4qTenqVPPj9/1sbv6+fYuloz+PmKKsTc/9uKCPEYW5db8RvKOnrYo5gHJSeHDKbb1xnJguplNA3YC1wD/LanMEuDGcPziXODQYBmfaGhu4++v7eaZTft5bvN+6sLWwsyJI7l2bhVzq8t5z7RyxpUOjzlSEZH00iYKM/sn4AbgZDNbHdk1EnjuRE7s7m1mdiPwMMHlsXe5+1oz+3K4/w5gKcGlsZsJLo/9womccyCs2n6Qu57dyuMb9lLf3EbFyOFcNKOC988Yx/tOGcf4kUN7KgARGXws3QCgmY0CxgC3AjdHdh1x97oMx9ZnNTU1vmLFigE95/Ob9/OzxzaxbGsdI4vyueLMSj4xZwrnnDRG13OLSNYzs5XuXpNqX9oWhbsfAg4B14b3PUwIP1NqZqXuvq3fox1k3J3bH9nIz5/YzNiS4fzLR07jmrlVlGbxg9JFRHqjp4PZNwLfBfYAiXCzA2dmJqzBobU9wbfvW839L+/kk+dM4YdXnqE5Y0Qk5/T0z96vA6e6e20GYxlUauubueHul1m2tY6vfPBd3HTZqepiEpGc1NNEsZ2gC0oIuptufWgDy7bW8ZNPncUnz5kSd0giIhnT3VVP3wgXtwBPmtnfgWP33bv77RmMLWv94aVt/HnlDq6/YJqShIjkvO5aFCPD923hqzB8DVl/e3UX//LXNcydVs4t80+LOxwRkYzr7qqn7w1UIIPBwaMt/OsDa5g9dTSLvzBX0xGIyJDQ06ue/kbnOZYOASuAX7l7U38Hlo1+/PDrHGps5YdXvpviQl3dJCJDQ08nct8C1AP/Eb4OE1wqOyNcz3kvbqnl7mXbuPo9U5k1qSzucEREBkxPr3qa7e4XRtb/ZmZPu/uFZrY2E4Flm9sf2UhRwTC+9aGZcYciIjKgetqiqDCzY3N3h8vjwtWcnxv7+c37eWlrHV+7ZAZjSob0WL6IDEE9bVF8E3jWzN4gmGR4GnCDmZUAizMVXLa49aENTB5dzBfeVx13KCIiA65HicLdl5rZdGAmQaLYEBnA/mmGYssKa3cd4rWdh/j+gtM1PYeIDEnd3XB3sbs/bmYfT9p1spnh7vdnMLascO/y7RTmD+NjZ02KOxQRkVh016L4APA48NEU+xzI6UTR0pbgr6/s5PLTJ+p5viIyZHV3w93/Dt+z/oFBmfD4hj0cbmrjqtmT4w5FRCQ2PbrqycwmmNmvzeyhcH2WmV2f2dDi9+KWOooKhnHB9HHdFxYRyVE9vTz2NwSPLO3oqN9IMPV4zmpua2fJq7t4T3U5BXk9/TGJiOSenn4DjnP3PxE+tMjd24D2jEWVBV7ZdpC6hhbNDisiQ15PE0WDmY0lnO/JzM4jx59PsWxL8EjwD84cH3MkIiLx6u7y2K8DzwHfAh4guCz2OaAC+FTGo4vR1v31TCwroqyoIO5QRERi1d3lsVOAnxHcaLcBeAR4ErjX3fdnNrR4vbztIHNOGh13GCIisUvb9eTuN7n7+cBE4CZgGXAxsNrM1vX1pGZWbmaPmNmm8H1MF+XuMrO9Zramr+fqi0TC2X2okZPGlgzkaUVEslJPxyiKgTJgVPjaRZA0+upm4DF3nw48Fq6n8hvg8hM4T58cONpCa7szfuTwgT61iEjW6W6MYhFwOnCEIDE8D9zu7gdO8LwLgIvC5cUE3VnfTi7k7k+bWfUJnqvX9tcHE+KOK1WiEBHprkVRBQwH3gZ2AjuAg/1w3gnuvhsgfM+qS4v21zcDShQiItD9FB6Xm5kRtCrOJ5hu/AwzqwNe6JjiIxUze5RgbCPZLScQb5fMbCGwEKCqqqqb0untOHAUgMmji084LhGRwa7bacbd3YE1ZnaQ4N6JQ8AVwFygy0Th7pd2tc/M9phZpbvvNrNKYG9vA09xvkXAIoCamprk53v3Sm1D2PU0UhMBioik7Xoys6+a2T1mth14miBBvA58HCg/gfMuAa4Ll68juEcjazyzcT8VI4czorCnz3USEcld3Y1RVAN/Bua6+8nu/ll3/6W7v+ruiRM4723APDPbBMwL1zGzSWa2tKOQmf0ReAE41cx2DNREhEdb2hhVrBvtRESg+zGKb2TipO5eC1ySYvsuYH5k/dpMnL87B462MrtqdBynFhHJOpoWNUlbe4J9R5opL9H4hIgIKFF08mbtURpb2zl90qi4QxERyQpKFEk6Lo2tHjsi5khERLKDEkWSQ42tAIweocFsERFQoujkcJgoynTVk4gIoETRSUeLQpfHiogElCiSHGlqozB/GMPz8+IORUQkKyhRJDnS3EZZke7IFhHpoESRpL6pjdLhShQiIh2UKJI0NLdRokQhInKMEkWSxtZ2RhRqfEJEpIMSRZLG1naKCpQoREQ6KFEkaWxpp1iJQkTkGCWKJE2t7RSr60lE5BgliiSNrWpRiIhEKVEkaWzRGIWISJQSRZKm1oS6nkREIpQoItoTTkt7Ql1PIiIRShQRTa3tAEoUIiIRShQRDS1tABSp60lE5BgliojDmmJcRKSTWBKFmZWb2SNmtil8H5OizFQze8LM1pvZWjP7Wqbj6ngWhWaPFRF5R1wtipuBx9x9OvBYuJ6sDfimu58GnAd8xcxmZTKo5tYEgC6PFRGJiCtRLAAWh8uLgSuTC7j7bnd/OVw+AqwHJmcyqJb2IFEU5KlHTkSkQ1zfiBPcfTcECQEYn66wmVUDs4FlmQyqtd0BGJ6vRCEi0iFjnfFm9igwMcWuW3p5nFLgPuDr7n44TbmFwEKAqqqq3pzimFa1KEREOslYonD3S7vaZ2Z7zKzS3XebWSWwt4tyBQRJ4m53v7+b8y0CFgHU1NR4X2J+J1FYXz4uIpKT4vrTeQlwXbh8HfBAcgEzM+DXwHp3v30ggmppU4tCRCRZXN+ItwHzzGwTMC9cx8wmmdnSsMz7gM8CF5vZqvA1P5NBdYxRFGqMQkTkmFhuGHD3WuCSFNt3AfPD5WeBAe0D0hiFiEhn+kaM0BiFiEhnShQRuo9CRKQzfSNGtLYFYxRKFCIi79A3YkRre4K8YUbeMHU9iYh0UKKIaGlPaHxCRCSJEkVES1tC3U4iIkn0rRjR2p6gUIlCROQ4+laMaG1Xi0JEJJm+FSNa2113ZYuIJNG3YoQGs0VEOlOiiGjVYLaISCf6VoxobU+o60lEJIm+FSNa210tChGRJPpWjNAYhYhIZ0oUEbo8VkSkM30rRiQcggfriYhIByWKKHc0H6CIyPGUKCISPsCP1BMRGQSUKCIcV9eTiEgSJYoId9T1JCKSRIkiIuGgzicRkeMpUUS4O+p5EhE5XiyJwszKzewRM9sUvo9JUabIzF4ys1fNbK2ZfW8gYlPXk4jI8eJqUdwMPObu04HHwvVkzcDF7n4WcDZwuZmdl8mgEu6Yup5ERI4TV6JYACwOlxcDVyYX8EB9uFoQvjyTQbmjricRkSRxJYoJ7r4bIHwfn6qQmeWZ2SpgL/CIuy/LZFAODFOmEBE5Tn6mDmxmjwITU+y6pafHcPd24GwzGw38xczOcPc1XZxvIbAQoKqqqvcBE3Q9qedJROR4GUsU7n5pV/vMbI+ZVbr7bjOrJGgxpDvWQTN7ErgcSJko3H0RsAigpqamb11UyhMiIp3E1fW0BLguXL4OeCC5gJlVhC0JzKwYuBTYkMmg1PUkItJZXIniNmCemW0C5oXrmNkkM1salqkEnjCz1cBygjGKBzMZVEL3UYiIdJKxrqd03L0WuCTF9l3A/HB5NTB7YONS15OISDLdmR2RcFfXk4hIEiWKCHfUpBARSaJEkUQtChGR4ylRRARTeIiISJQSRYSm8BAR6UyJIsLRYLaISDIlioiEWhQiIp0oUUS4nnAnItKJEsVxXA8uEhFJokQRoa4nEZHOlCgiXE+4ExHpRIkiIpg9Nu4oRESyixJFRCLhmPqeRESOo0QRkdEHcouIDFJKFFGuuZ5ERJIpUUTowUUiIp0pUURolnERkc6UKCLcYZguexIROY4SRcSHTp/AzIkj4w5DRCSrxPLM7Gz102sG9BHdIiKDgloUIiKSViyJwszKzewRM9sUvo9JUzbPzF4xswcHMkYREQnE1aK4GXjM3acDj4XrXfkasH5AohIRkU7iShQLgMXh8mLgylSFzGwK8BHgzoEJS0REksWVKCa4+26A8H18F+V+CnwLSAxQXCIikiRjVz2Z2aPAxBS7bunh568A9rr7SjO7qAflFwILAaqqqnoeqIiIpJWxROHul3a1z8z2mFmlu+82s0pgb4pi7wM+ZmbzgSKgzMx+7+7/vYvzLQIWAdTU1Gh+PxGRfhJX19MS4Lpw+TrggeQC7v4dd5/i7tXANcDjXSUJERHJHHMf+D++zWws8CegCtgGfMrd68xsEnCnu89PKn8RcJO7X9HD4+8D3upjeOOA/X387GCk+ua+oVZn1bdvTnL3ilQ7YkkU2czMVrh7TdxxDBTVN/cNtTqrvv1Pd2aLiEhaShQiIpKWEkVni+IOYICpvrlvqNVZ9e1nGqMQEZG01KIQEZG0lChCZna5mb1uZpvNLN0khVnNzKaa2RNmtt7M1prZ18LtXc7Ya2bfCev9upl9KLL9HDN7Ldz3f8yy94niybMMD4H6jjazP5vZhvB3/d5crrOZ/a/w3/MaM/ujmRXlUn3N7C4z22tmayLb+q1+ZjbczO4Nty8zs+peBejuQ/4F5AFvACcDhcCrwKy44+pjXSqBOeHySGAjMAv4N+DmcPvNwI/C5VlhfYcD08KfQ1647yXgvQSPEn8I+HDc9UtT728AfwAeDNdzvb6LgS+Gy4XA6FytMzAZ2AoUh+t/Aj6fS/UFLgTmAGsi2/qtfsANwB3h8jXAvb2KL+4fUDa8wh/sw5H17wDfiTuufqrbA8A84HWgMtxWCbyeqq7Aw+HPoxLYENl+LfCruOvTRR2nEExXfzHvJIpcrm9Z+MVpSdtzss5hotgOlBNMO/QgcFmu1ReoTkoU/Va/jjLhcj7BDXrW09jU9RTo+IfYYUe4bVALm5ezgWV0PWNvV3WfHC4nb89GP6XzLMO5XN+TgX3Af4bdbXeaWQk5Wmd33wn8hGAWh93AIXf/Bzla34j+rN+xz7h7G3AIGNvTQJQoAqn6KQf15WBmVgrcB3zd3Q+nK5pim6fZnlUsMstwTz+SYtugqW8on6Cb4t/dfTbQQPqHfw3qOod98wsIulkmASVmlm7et0Fd3x7oS/1OqO5KFIEdwNTI+hRgV0yxnDAzKyBIEne7+/3h5j3hTL3Y8TP2dlX3HeFy8vZs0zHL8JvAPcDFZvZ7cre+EMS6w92Xhet/JkgcuVrnS4Gt7r7P3VuB+4Hzyd36dujP+h37jJnlA6OAup4GokQRWA5MN7NpZlZIMNizJOaY+iS8yuHXwHp3vz2yq6sZe5cA14RXRUwDpgMvhU3dI2Z2XnjMz5Filt+4edezDOdkfQHc/W1gu5mdGm66BFhH7tZ5G3CemY0I47yE4PHIuVrfDv1Zv+ixPknw/6Tnram4B3Cy5QXMJ7hC6A3glrjjOYF6XEDQpFwNrApf8wn6Ix8DNoXv5ZHP3BLW+3UiV4EANcCacN/P6cXgV0x1v4h3BrNzur7A2cCK8Pf8V2BMLtcZ+B6wIYz1dwRX/ORMfYE/Eoy/tBL89X99f9aP4Jk+/wVsJrgy6uTexKc7s0VEJC11PYmISFpKFCIikpYShYiIpKVEISIiaSlRiIhIWkoUkpPM7JZwttHVZrbKzM7t5ec/b2aTevmZ6ujsn0n7fhzG8+PeHDP87NlmNr+3nxPpL/lxByDS38zsvcAVBLPoNpvZOIIZVnv6+TyC2UnX0H937n4JqHD35j589myC6+OX9vQD4Q1X5u6JbguLdEMtCslFlcD+ji9ld9/v7rsAzOyScCK918JnAAwPt79pZv9qZs8SzLpZA9wdtkaKw3n+nzKzlWb2cGRqhXPM7FUzewH4SqpgzGwJUAIsM7OrzazCzO4zs+Xh631hublm9nwY3/Nmdmo4U8D3gavDWK42s++a2U2R468JWzPVFjyb4pfAy8BUM/vn8Byrzex7YfkSM/t7GPcaM7s6A78DySVx35Gol179/QJKCe5I3wj8EvhAuL2IYAbNGeH6bwkmTQR4E/hW5BhPAjXhcgHwPEGLAOBq4K5weXXk+D8mMk10Ukz1keU/ABeEy1UE061AMH14frh8KXBfuPx54OeRz38XuCmyvoZgiupqghl0zwu3X0bwPGUj+KPwQYLnHnwC+I/I50fF/TvTK7tf6nqSnOPu9WZ2DvB+4IPAvRY8tfAVgsnlNoZFFxO0An4art/bxSFPBc4AHgl6dMgDdpvZKGC0uz8Vlvsd8OEehHgpMMveebhamZmNJJiobbGZTSeYhqWgB8dK9pa7vxguXxa+XgnXSwnmBXoG+ImZ/YhgypNn+nAeGUKUKCQnuXs7QavgSTN7jWBCtFXdfKyhi+0GrHX39x630Ww0fZumehjBQ2Qak473f4En3P0qC54l8mQXn2/j+G7joshytA4G3Oruv0o+QJhI5wO3mtk/3P37va6FDBkao5CcE/btT49sOht4i2BSuWozOyXc/lngKVI7QvAoWQgmXqsIB8kxswIzO93dDwKHzOyCsNxnehjiP4AbI/GeHS6OAnaGy5/vIhYIusnmhJ+dQ/CchlQeBv6HBc8mwcwmm9n48Gquo+7+e4IHAs3pYdwyRClRSC4qJejCWWdmqwmeMfxdd28CvgD8V9jKSAB3dHGM3wB3mNkqgq6mTwI/MrNXCVom54flvgD8IhzMbux8mJS+CtSEA8zrgC+H2/+N4C/858JzdniCoKtqVTjwfB9QHsb2TwRjMZ148BS4PwAvhPX9M0HCeTfwUvj5W4Af9jBuGaI0e6yIiKSlFoWIiKSlRCEiImkpUYiISFpKFCIikpYShYiIpKVEISIiaSlRiIhIWkoUIiKS1v8Hd1ow3ocr83kAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as pyplot\n",
    "%matplotlib inline  \n",
    "\n",
    "pyplot.plot(np.sort(CFW_weithing.D_best))\n",
    "pyplot.ylabel('Weight')\n",
    "pyplot.xlabel('Sorted features')\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Something to think about... do we select features with the 'highest' weight, hence excluding the negative ones, or do we select those that have the 'highest absolute' value, accounting also for the strong negative ones?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Selected 3031/10106 features, weights: 3.26E-03 to 2.73E-01\n",
      "ItemKNNCBFRecommender: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "ItemKNNCBFRecommender: ICM Detected 3298 (30.9%) items with no features.\n",
      "Similarity column 10681 (100.0%), 8924.71 column/sec. Elapsed time 1.20 sec\n",
      "EvaluatorHoldout: Processed 69805 (100.0%) in 2.34 min. Users per second: 497\n",
      "Selected 5053/10106 features, weights: 0.00E+00 to 2.73E-01\n",
      "ItemKNNCBFRecommender: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "ItemKNNCBFRecommender: ICM Detected 3140 (29.4%) items with no features.\n",
      "Similarity column 10681 (100.0%), 8929.66 column/sec. Elapsed time 1.20 sec\n",
      "EvaluatorHoldout: Processed 69805 (100.0%) in 2.14 min. Users per second: 545\n",
      "Selected 7074/10106 features, weights: 0.00E+00 to 2.73E-01\n",
      "ItemKNNCBFRecommender: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "ItemKNNCBFRecommender: ICM Detected 3130 (29.3%) items with no features.\n",
      "Similarity column 10681 (100.0%), 9009.28 column/sec. Elapsed time 1.19 sec\n",
      "EvaluatorHoldout: Processed 69805 (100.0%) in 2.14 min. Users per second: 543\n",
      "Selected 9095/10106 features, weights: -1.46E-03 to 2.73E-01\n",
      "ItemKNNCBFRecommender: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "ItemKNNCBFRecommender: ICM Detected 3106 (29.1%) items with no features.\n",
      "Similarity column 10681 (100.0%), 8960.42 column/sec. Elapsed time 1.19 sec\n",
      "EvaluatorHoldout: Processed 69805 (100.0%) in 2.15 min. Users per second: 541\n"
     ]
    }
   ],
   "source": [
    "argsort_features = np.argsort(-CFW_weithing.D_best)\n",
    "\n",
    "map_list = []\n",
    "selection_quota_list = [0.3, 0.5, 0.7, 0.9]\n",
    "\n",
    "for selection_quota in selection_quota_list:\n",
    "    \n",
    "    n_to_select = int(selection_quota*len(argsort_features))\n",
    "    selected_features = argsort_features[:n_to_select]\n",
    "    \n",
    "    ICM_selected = ICM_tags[:,selected_features]\n",
    "    \n",
    "    print(\"Selected {}/{} features, weights: {:.2E} to {:.2E}\".format(n_to_select, len(argsort_features),\n",
    "                                                                min(CFW_weithing.D_best[selected_features]),\n",
    "                                                                max(CFW_weithing.D_best[selected_features])))\n",
    "    \n",
    "    itemKNNCBF = ItemKNNCBFRecommender(URM_train, ICM_selected)\n",
    "    itemKNNCBF.fit(**best_hyperparams_ItemKNNCBF)\n",
    "    \n",
    "    results_df, _ = evaluator_test.evaluateRecommender(itemKNNCBF)\n",
    "    map_list.append(results_df.loc[10][\"MAP\"])\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEWCAYAAABIVsEJAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA2G0lEQVR4nO3deZwU1bn/8c93ZthXlWFH1hkVFRFHxF1UiJgFY9Ro4h6DqBij5iZmuVlvfjfXxDUYt8SoNxo1uZJr1KugIkoUcVBAUYFh32SVHWR7fn/UGW2anpmeZnqqe+Z5v1796u6qU1XP6arqp2s5p2VmOOecc7VVEHcAzjnn8pMnEOeccxnxBOKccy4jnkCcc85lxBOIc865jHgCcc45lxFPII2UpMslTc6BOF6VdFUV4yTpz5I+kTS1vmNLRdJCSWfW0bxMUr8qxn1T0vg055MT6zITkg6WtFlSYRple4XPrKg+YkuHpNMkLa2jeeXdemwUCSTs9NvChlr56FoH86yTL5J8VQ879EnAMKC7mQ3O0jKqJOlhSf9R38sFMLPHzGx4HMuuVF2CqytmttjMWpvZ7v2dl6SfS/pLDWX2a7+tj88kbrXZrxtFAgm+HDbUysfyOIPJpV9ROawnsNDMtqQa6Z+hczEzswb/ABYCZ6YY3g74E7ACWAb8B1AYxvUFXgHWAmuAx4D2Ydx/A3uAbcBm4PvAacDSqpYL/Bz4O/AXYCNwVQ3L7wdMAjaE5T9ZRd2ah3muBdYDbwOd0qjf5cDkhPkcCkwA1gGzgQsSxrUAbgMWhXgmh2GLAQufwWbg+FD+SuBD4BPgRaBnwryGAR+F+YwNdbwqRb2+BWwHdod5/6LyMwZ+AHwc1kMz4E5geXjcCTQL86gs/31gVfgczgHOBuaEuv6ois91FLAT2BGW/8+Edfo9YGaow5NA84TpvgRMD+viDWBANdulAaOBueGzugdQFetneFgvG4A/JH5ulWWB34X5LABGpLmdp9zOgNdCfFtC/b+eIv5FwDHh9cWhfP/w/irgH+F1AXALMI9oO30KODCM6xWmKwrve4dlbwJeCp/JX5LKXka07a0BfhzGnRXW1c4Q74wU8e6z34bhXwFmhXX2KnBYFetrn8+Ez7exm/l8G7siYZpmYb0sBlYC9wEtqph/8jo/gWh/3hCeTwjDhwLvJZR7CZia8H4ycE543RX4H2B12C6+k1BuMFBO9H20Erg9DE+5X6eMOa4v9fp8UHUC+QdwP9AK6AhMBa5O2LGGhQ2gOGw8d1Y1T9JLIDuJvsAKiL6Aq1v+X4Efh7LNgZOqqNvVwD+BlkAhcAzQNo36fbaxhvFLgCuAImAQ0c55eBh/D9GO1S0s44TwufQiYecPZc8BKoDDwrx+ArwRxnUIG+t5QBPgRmAXKRJIFTvUaaH8f4XltwB+CUwJ9Ssm+tL+VVL5n4blfZtoR3ocaAMcTpSk+lSx/IeB/0ixTqcS7ZgHEiXK0WHcIKIvkePC53RZKN+sivkb8CzQHjg4xHZWivVT+bmdGz7TG4i2pcQEsjPUrxC4hiiZKo3toMrtLMTXr5r96lHg5vD6AaIEcU3CuBvD6++GddQ9rLf7gb+Gcb3YO4G8SfSF25ToFOZG9k0gD4Z1fxTwKeELn2gf+0ttvguAUqKEMCxsI98n2n6bVrPO+iW8P41oG/tlmP5sYCtwQBh/J/AM0bbShmhf/c+atvdQ/hPgkrDOLwrvDwrraVvYLoqIfkwtD/NvEcYdFNbpNKLtvynQB5gPfCHhs74kvG4NDEm1Tqr9POvrSzzOR9hoNhP9wlhPtEN1Chtfi4RyFwETq5jHOcC71WyIp1FzAnktYVy1yyfaAR8gOv9fXd2uJMUv3TTmn7ixfh14PWn6+4GfhY1wG3BUimXvs6EB/wd8K+F9AdEO1RO4FJiSME5Ev95qk0B2sPcv/nnA2Qnvv0B02quy/DY+/7XdJsR7XEL5aYRfaymW/zCpE8jFCe9vBe4Lr+8lJK+E8bOBU6uYv7H3F/ZTwC0p1s+lwJtJn9sS9k4gFQnjW4Z5d96f7YyaE8i3gGfC6w+JjjqeCO8XAYMSxp2RMF0XooRXlLgNESXRXUDLhLJ/Yd8E0j1h/FTgwoR9rLYJ5N+Bp5K212XAadWss+QEso2994FVwJCwnrYAfRPGHQ8sqGl7J0ocU5PGvwlcHl6/TvSDYggwPmw7ZxEdncwMZY4DFifN44fAn8Pr14iO7DvUtF9X9WhM55DPMbOXKt9IGkz0i2GFpMrBBUQ7JpI6AncDJxN98RQQ/QLYH0sSXvesbvlEv4R+BUyV9Alwm5k9lGKe/w30AJ6Q1J5oh/txGvNP1BM4TtL6hGFFYd4diH7xzEuzjj2BuyTdljBMREcvXROXb2YmKVU81VltZtsT3ncl+rKqtCgMq7TWPr9Auy08r0wYv43o11dtfJzwemvC8noCl0m6PmF806R4appXqlhSfW7Jd/58nDB+a1jnrYl+ydbFdpbKJOB3kjoTHfk8CfxMUi+i02bTQ7mewDhJexKm3U2U3JLruc7MtiYMW0K0faesK1V/Zunaa/sxsz1hm+xWi3msNbNdKWIqJkrm0xI+exF9VrWKK1iUENckPj99Nonou+lUoh8Lk0KZnkDXpP26kCj5QPQD4JfAR5IWAL8ws2fTiO0zjSmBJFtC9GF3SFr5lf6TKAsPMLO1ks4hOmdfyZLKbyHaWAAItyUWJ5VJnKba5ZvZx0SnJJB0EvCSpNfMrCKp3E6iXxG/CDvu80S/ep+voX6JlgCTzGxY8ghJBUSnefoCM6qpT+K8fm1mj6WYVwkJXwaK9qrkL4eaJC9zOdGOMiu8PzgMqwup6ledyrr/uo6WX2kF0ekf4LPPrXvVxfeJab+3s1TMrELSVuA7REfXmyR9THT9aLKZVSaMJcCVZvav5HmEbbbSCuBASS0Tkkhtto901leq7efIhHgqt8lltVhuVdYQ/UA53MxqO7/K7TrRwcAL4fUkouuSi4HfECWQB4nW9T2hzBKio52SVAsws7nARWEfPxf4u6SDqMV235juwtqLma0gOvS7TVJbSQWS+ko6NRRpQzjtJakb8G9Js1hJdE6x0hyguaQvSmpCdO6/WabLl3S+pMoviU+IVuo+tzpKGirpyJCwNhKdGtidRv0SPQuUSrpEUpPwOFbSYeFL4CHgdkldJRVKOl5SM6Jz9nuSPof7gB9KOjzE107S+WHcc8Dhks4Nd1B9h+g0y/74K/ATScWSOhCd7632Vs5aSF7HNXkQGC3pOEVahe2hzX7G8RxwpKRzwud2HWl+bvu5naVT/0nAGD7/1ftq0nuItolfS+oZllksaWSKWBcRXdT9uaSmko4HvpxOPRPi7RW+EKsrk1inp4AvSjoj7Lc3E30Jv5Hm9FUK+86DwB3hjAaSukn6QhqTP0+0T35DUpGkrwP9ifZVQnyHEF0In2pmswhnEohOTUF0em+jpB9IahH23SMkHRtiuVhScYhzfZhmN6n365QabQIJLiU6xfAB0c7zd6LzsxD9qh9EdAfEc8DTSdP+J9EX13pJ3zOzDcC1wB+Jfr1sITq8zHT5xwJvSdpMdBHuBjNbkGIencN0G4nONU/i8y/Q6ub/GTPbRHSXz4VEv3w+5vML1RDddfQe0Z0g68K4gvAr8dfAv8LnMMTMxoXxT0jaCLwPjAjLWQOcT/SLaS1QAuzzq7SW/oPoS2dmiPGdMKwu/AnoH+r2j5oKm1k50a/5sUSfdwXRee39kvC53Ur0ufUnqvOnac4i0+3s58Ajof4XVDHvSUQ/tl6r4j3AXWHe4yVtIrqgflwV8/sm0XWCtUTr8cla1PNv4XmtpHeqKJO8384muoPs90RHDF8muuV/RxXT/5yaP5NEPyDaDqaE/eEloi/+apnZWqI7+m4m+iy+D3wpbAtYdGv7O8CshFjfBBaZ2apQZneoz0CiO7DWEH0/tQvlzwJmhXV/F9G1pO2p9uuq4qy8S8M5lyfCL+ylwDfNbGLc8WSTpCeBj8zsZ3HH4vbV2I9AnMsLkr4gqX04dfgjoouxU2IOq86FU6d9w6m2s4CRRHdNuhzUmC+iO5dPjidqv1J5KuocM9tW/SR5qTPR6eKDiI6yrjGzd+MNyVXFT2E555zLiJ/Ccs45l5FGcQqrQ4cO1qtXr7jDcM65vDJt2rQ1Zpbcnu0zWU0g4SLYXUStH/9oZr9JGq8wvrL/mMvN7J0wrj3RLWdHEN2bfqWZvSnpQKJb+3oRdUtwgZlV20K8V69elJeX113FnHOuEZCU3Bp+L1k7hRUatt1D1AagP1GLx/5JxUYQtQUoIWq9em/CuLuAF8zsUKJO0z4Mw28BXg6tK18O751zztWzbF4DGUzUwdv80NDlCaJb8hKNBB61yBSgvaQuktoCpxA15MLMdpjZ+oRpHgmvHyHq5NA551w9y2YC6cbeHfctZd8Oyqoq04eoOf2fJb0r6Y+SWoUynUL3DJXdNHTMRvDOOeeql80EohTDku8ZrqpM5X9S3GtmRxN1C1KrU1WSRkkql1S+evXq2kzqnHMuDdlMIEvZuyfN7uzbS2pVZZYS/bfGW2H434kSCsBKSV0AwvOqVAs3swfMrMzMyoqLq7yJwDnnXIaymUDeBkok9ZbUlKijvmeSyjwDXBp6Lh0CbDCzFaGL6SWSKjsdO4Oo9W3lNJeF15cB/5vFOjjnnKtC1m7jNbNdksYQ/Sd2IfCQmc2SNDqMv4+oy+KziXqr3Er0l6qVrgceC8lnfsK43wBPSfoWUV/45+Occ67eNYquTMrKyiyTdiBvzlvL9CXruea0vlmIyjnncpukaWZWVtV478qkGhNnr+K3L35ExarNcYfinHM5xxNINa4+pQ/NmxRy50tz4g7FOedyjieQahzUuhlXntibZ2eu4MMVG+MOxznncoonkBp8++Q+tGlexB0T/CjEOecSeQKpQbuWTfj2yX0Y/8FK3lu6Ie5wnHMuZ3gCScMVJ/aifcsm3DZhdtyhOOdczvAEkoY2zZsw+tS+vDp7NdMWrYs7HOecywmeQNJ06fE96dC6GbeN92shzjkHnkDS1rJpEdee1pc35q3ljXlr4g7HOedi5wmkFr5x3MF0btuc28fPoTG04HfOuep4AqmF5k0KGXN6P8oXfcKkOd5FvHOucfMEUksXlPWg+wEtuH2CH4U45xo3TyC11LSogO+cUcLMpRuY8MHKuMNxzrnYeALJwLlHd6N3h1bcPmEOe/b4UYhzrnHyBJKBosICvntmCR99vInn318RdzjOORcLTyAZ+tKArpR2as0dE+aw249CnHONkCeQDBUWiBvPLGXe6i387/RlcYfjnHP1zhPIfvjC4Z05vGtb7np5Ljt374k7HOecq1eeQPZDQYG4aVgpi9Zu5X+mLY07HOecq1eeQPbT6Yd2ZGCP9vz+lQo+3bU77nCcc67eeALZT5K4eXgpy9Zv48m3l8QdjnPO1RtPIHXgpH4dGNz7QMa+UsH2nX4U4pxrHLKaQCSdJWm2pApJt6QYL0l3h/EzJQ1KGLdQ0nuSpksqTxh+lKQ3w7h/SmqbzTqkQxI3Dytl1aZP+cuURXGH45xz9SJrCURSIXAPMALoD1wkqX9SsRFASXiMAu5NGj/UzAaaWVnCsD8Ct5jZkcA44N+yEX9tHdfnIE4u6cAfXp3Hlk93xR2Oc85lXTaPQAYDFWY238x2AE8AI5PKjAQetcgUoL2kLjXM9xDgtfB6AvC1ugx6f9w0rJR1W3bw8BsL4w7FOeeyLpsJpBuQeFV5aRiWbhkDxkuaJmlUQpn3ga+E1+cDPVItXNIoSeWSylevrp+u148++ADOOLQjD7w2n43bd9bLMp1zLi7ZTCBKMSy5z4/qypxoZoOITnNdJ+mUMPzK8H4a0AbYkWrhZvaAmZWZWVlxcXHto8/QjcNK2bBtJ396fUG9LdM55+KQzQSylL2PDroDy9MtY2aVz6uIrnUMDu8/MrPhZnYM8FdgXlaiz9AR3dox4ojO/GnyAj7ZkjK3Oedcg5DNBPI2UCKpt6SmwIXAM0llngEuDXdjDQE2mNkKSa0ktQGQ1AoYTnTqCkkdw3MB8BPgvizWISM3Ditly45dPPD6/LhDcc65rMlaAjGzXcAY4EXgQ+ApM5slabSk0aHY88B8oAJ4ELg2DO8ETJY0A5gKPGdmL4RxF0maA3xEdLTy52zVIVOlndrwlaO68vC/FrJ606dxh+Occ1mhxvC3rGVlZVZeXl5zwTo0f/Vmht3xGpef0It//1Ly3cvOOZf7JE1LakaxF2+JniV9iltz7tHd+O8pi/h4w/a4w3HOuTrnCSSLvnNGCWbGPRMr4g7FOefqnCeQLOpxYEsuKOvBE28vZuknW+MOxznn6pQnkCwbc3o/JPH7l/0oxDnXsHgCybIu7VrwzeMO5u/vLGXhmi1xh+Occ3XGE0g9uOa0vjQtLOCul+fGHYpzztUZTyD1oGOb5lx2Qi/+MX0Zc1duijsc55yrE55A6snVp/ShVdMi7nhpTtyhOOdcnfAEUk8OaNWUK0/qzfPvfcys5RviDsc55/abJ5B69K2TetO2eRF3TPCjEOdc/vMEUo/atWjC1af25aUPVzF9yfq4w3HOuf3iCaSeXX5CLw5s1ZTbxs+OOxTnnNsvnkDqWatmRVxzal9en7uGqQvWxR2Oc85lzBNIDC4e0pPiNs24bfxsGkNvyM65hskTSAxaNC1kzNB+vLVgHW/MWxt3OM45lxFPIDG5cHAPurZrzu/8KMQ5l6c8gcSkWVEh159RwruL1/Pq7NVxh+Occ7XmCSRG5x3TnYMPbMltE/woxDmXfzyBxKhJYQE3nFHC+8s28uKslXGH45xzteIJJGbnHN2NPsWtuH3CbHbv8aMQ51z+8AQSs8ICceOZpcxZuZlnZy6POxznnEtbVhOIpLMkzZZUIemWFOMl6e4wfqakQQnjFkp6T9J0SeUJwwdKmlI5XNLgbNahPnzxyC4c2rkNd700l12798QdjnPOpSVrCURSIXAPMALoD1wkqX9SsRFASXiMAu5NGj/UzAaaWVnCsFuBX5jZQOCn4X1eKygQNw4rZf6aLYx7d1nc4TjnXFqyeQQyGKgws/lmtgN4AhiZVGYk8KhFpgDtJXWpYb4GtA2v2wEN4rzP8P6dOLJbO+5+ZS47dvlRiHMu92UzgXQDliS8XxqGpVvGgPGSpkkalVDmu8BvJS0Bfgf8MNXCJY0Kp7jKV6/O/XYWkrhpeClL1m3jb9OW1DyBc87FLJsJRCmGJd9mVF2ZE81sENFprusknRKGXwPcaGY9gBuBP6VauJk9YGZlZlZWXFxc++hjcFppMcf0PICxr1SwfefuuMNxzrlqZTOBLAV6JLzvzr6nm6osY2aVz6uAcUSnxAAuA54Or/+WMDzvSeLmYaWs2LCdv05dHHc4zjlXrWwmkLeBEkm9JTUFLgSeSSrzDHBpuBtrCLDBzFZIaiWpDYCkVsBw4P0wzXLg1PD6dGBuFutQ707o14Hj+xzEPRPnsW2HH4U453JX1hKIme0CxgAvAh8CT5nZLEmjJY0OxZ4H5gMVwIPAtWF4J2CypBnAVOA5M3shjPs2cFsY9/+I7t5qUG4eXsqazZ/y6JsL4w7FOeeqpMbQB1NZWZmVl5fXXDCHXPbQVGYuXc/rPzid1s2K4g7HOdcISZqW1IxiL94SPUfdNKyUT7bu5M+TF8QdinPOpeQJJEcd1aM9w/p34oHX57Nh6864w3HOuX14AslhNw0rZdP2Xfxx8vy4Q3HOuX14Aslhh3VpyxcHdOGhyQtYt2VH3OE459xePIHkuBvPLGHbzt3cP2le3KE459xePIHkuH4d23DOwG488uZCVm3aHnc4zjn3GU8geeCGM0vYudv4w0Q/CnHO5Q5PIHmg50GtOP+Y7jz+1mKWr98WdzjOOQd4Askb159RAsDYiRUxR+KccxFPIHmiW/sWXDi4B0+9vYTFa7fGHY5zznkCySfXDe1HYYG4+5UG1X+kcy5PeQLJI53aNueSIT15+p2lzFu9Oe5wnHONnCeQPDP6tL40b1LIXS/5UYhzLl6eQPJMh9bNuPyEXvxz5nJmf7wp7nCcc42YJ5A8NOqUPrRuWsQdE+bEHYpzrhHzBJKH2rdsyrdO7s0Lsz7m/WUb4g7HOddIeQLJU1ee1Jv2LZtwux+FOOdi4gkkT7Vt3oRRp/ThlY9WMW3RJ3GH45xrhDyB5LHLT+hFh9ZN/VqIcy4WnkDyWMumRYw+tS+TK9bw5ry1cYfjnGtkPIHkuYuH9KRT22bcPmE2ZhZ3OM65RiSrCUTSWZJmS6qQdEuK8ZJ0dxg/U9KghHELJb0nabqk8oThT4Zh00OZ6dmsQ65r3qSQMaeX8PbCT3h97pq4w3HONSJZSyCSCoF7gBFAf+AiSf2Tio0ASsJjFHBv0vihZjbQzMoqB5jZ18OwgcD/AE9nqQp54+tlPejWvgW3jfejEOdc/cnmEchgoMLM5pvZDuAJYGRSmZHAoxaZArSX1CWdmUsScAHw17oMOh81LSrghjNKmLF0Ay9/uCrucJxzjUQ2E0g3YEnC+6VhWLplDBgvaZqkUSnmfzKw0sxSdgolaZSkcknlq1evzqgC+eTcQd3odVBLbpswhz17/CjEOZd92UwgSjEs+ZutujInmtkgotNc10k6JancRVRz9GFmD5hZmZmVFRcXpxtz3ioqLOC7Z5by4YqNvDDr47jDcc41AtlMIEuBHgnvuwPL0y1jZpXPq4BxRKfEAJBUBJwLPFnnUeexLx/VlZKOrbl9whx2+1GIcy7LsplA3gZKJPWW1BS4EHgmqcwzwKXhbqwhwAYzWyGplaQ2AJJaAcOB9xOmOxP4yMyWZjH+vFNYIG4cVkrFqs38c0ZyrnbOubqVtQRiZruAMcCLwIfAU2Y2S9JoSaNDseeB+UAF8CBwbRjeCZgsaQYwFXjOzF5ImP2F+MXzlM46vDOHdWnLnS/NYdfuPXGH45xrwNQYbvssKyuz8vLymgs2EC99sJKrHi3n1q8N4IJje9Q8gXPOpSBpWmIzimRpHYFI6lB3IblsO+OwjhzVoz13vTyXHbv8KMQ5lx3VJhBJX5a0GnhP0lJJJ9RTXG4/SOLmYaUsW7+NJ8uX1DyBc85loKYjkF8DJ5tZF+BrwH9mPyRXF04u6cCxvQ5g7Ctz2b5zd9zhOOcaoJoSyC4z+wjAzN4C2mQ/JFcXJHHz8ENYufFT/jJlUdzhOOcaoKIaxneUdFNV783s9uyE5erCkD4HcWK/g7hv0jwuGnwwrZrVtLqdcy59NR2BPEh01FH5SHzfOruhubpw07BDWLN5B4+8uTDuUJxzDUy1P0nN7BdVjZN0bN2H4+raMT0P4PRDO3L/pPlcPKQnbZs3iTsk51wDUauGhJL6S/qlpLns2/W6y1E3DStlw7adPDR5QdyhOOcakBpPikvqSdRx4UXALqAnUGZmC7MbmqsrR3Rrx1mHd+ZPry/g8hN60b5l07hDcs41ADW1A3mDqLuRJsB5ZnYMsMmTR/65cVgpm3fs4oHX5scdinOugajpFNZqogvmnYDKPtEbft8nDdAhndvw5QFdefiNhazZ/Gnc4TjnGoBqE4iZjQSOBN4BfiFpAXCApMHVTedy0w1nlrB9527ue3Ve3KE45xqAGi+im9kGM3vIzIYBQ4CfAXdK8j4y8kzf4tacO6g7/z1lESs3bo87HOdcnqvVXVhmttLM7jazE4CTshSTy6Ibzihh9x7jnokVcYfinMtz1d6FJSn5D6CSfaUOY3H1oMeBLbng2B78depiRp3Sh+4HtIw7JOdcnqrpNt7jgSVEf970Fqn/w9zlmTFD+/H38qWMfaWC33xtQNzhOOfyVE2nsDoDPwKOAO4ChgFrzGySmU3KdnAuO7q2b8E3jjuYv01bysI1W+IOxzmXp2q6C2u3mb1gZpcRXUCvAF6VdH29ROey5tqhfWlSKO5+eW7coTjn8lSNF9ElNZN0LvAX4DrgbuDpbAfmsqtjm+Zcdnwvxk1fRsWqTXGH45zLQzW1RH8EeAMYBPzCzI41s1+Z2bJ6ic5l1dWn9qVlk0LueMmPQpxztVfTEcglQClwA/CGpI3hsUnSxuyH57LpwFZNufKk3jw3cwUfLPfV6ZyrnZqugRSYWZvwaJvwaGNmbWuauaSzJM2WVCHplhTjJenuMH6mpEEJ4xZKek/SdEnlSdNdH+Y7S9Kttamw29tVJ/ehbfMi7nhpTtyhOOfyTK0aEtaGpELgHmAE0B+4SFL/pGIjgJLwGMW+XcQPNbOBZlaWMN+hwEhggJkdDvwuS1VoFNq1aMK3T+7DhA9WMmPJ+rjDcc7lkawlEGAwUGFm881sB/AE0Rd/opHAoxaZArSX1KWG+V4D/MbMPgUws1V1HXhjc8VJvTmgZRNun+BHIc659GUzgXQjaoRYaWkYlm4ZA8ZLmiZpVEKZUuBkSW9JmlTVPyNKGiWpXFL56tWr96siDV3rZkWMPrUvk+aspnzhurjDcc7liWwmkFSt1pO7gq+uzIlmNojoNNd1kk4Jw4uAA4japfwb8JSkfeZjZg+YWZmZlRUXFyePdkkuPb4XHVo347bxfhTinEtPNhPIUqBHwvvuwPJ0y5hZ5fMqYBzRKbHKaZ4Op72mAnuADnUefSPTomkh1w3ty5vz1/JGxZq4w3HO5YFsJpC3gRJJvSU1BS4EkjtnfAa4NNyNNQTYYGYrJLWS1AZAUitgOPB+mOYfwOlhXCnQFPBvvDpw0eCD6dKuObdNmIOZ/2+Yc/lu+87dWZ1/1hKIme0CxgAvAh8CT5nZLEmjJY0OxZ4H5hN1kfIgcG0Y3gmYLGkGMBV4zsxeCOMeAvpIep/owvxl5t92daJ5k0LGnN6PaYs+4dU5ft3IuXw28aNVnHzrRN5d/EnWlqHG8N1bVlZm5eXlNRd07Ni1hzNuf5X2LZryzJgTSXF5yTmXw3bvMe6YMIexEys4rEtb7rt4ED0PapXRvCRNS2xGkSybp7BcHmpaVMB3Ti/hvWUbGP/ByrjDcc7VwtrNn3LpQ28xdmIFF5R1Z9y1J2ScPNLhCcTt46tHd6NPh1bcPn4Oe/Y0/CNU5xqCaYvW8cW7J1O+8BNu/doAbj3vKJo3KczqMj2BuH0UFRZww5klzF65iefeWxF3OM65apgZf5q8gK/fP4VmTQp4+toTuODYHjVPWAc8gbiUvjygK4d0asMdL81h1+49cYfjnEth0/adjHn8XX717AecfmhHnhlzEod3bVdvy/cE4lIqKBA3Dith/uot/O/05OY7zrm4zf54EyPH/osXZn3MD0ccyv2XHEO7Fk3qNQZPIK5KXzi8M4d3bctdL89lpx+FOJczxr27lHPu+RebPt3FY1cdx9Wn9o3ljklPIK5Kkvje8ENYvG4rf5+2NO5wnGv0Pt21mx+Pe48bn5zBkd3b8dz1JzGkz0GxxeMJxFXrtEOKOfrg9vz+5bl8uiu7rVqdc1Vbsm4r59/3Jo+9tZirT+3D41cdR8e2zWONyROIq1blUcjyDdt5YuqSmidwztW5iR+t4ku/n8yCNVt44JJj+OGIwygqjP/rO/4IXM47oe9BHNf7QMZOrGDbDj8Kca6+7N5j/O7F2Vzx8Nt0bd+CZ68/ieGHd447rM94AnE1ksTNww9h9aZP+cuURXGH41yjUN+tyjPhCcSlZXDvAzm5pAP3TprH5k93xR2Ocw1aHK3KM+EJxKXt5uGHsG7LDh55Y2HcoTjXIMXZqjwTnkBc2gb2aM+Zh3Xk/knz2LBtZ9zhONegJLYqHxpDq/JMeAJxtXLjsFI2bt/FnyYviDsU5xqMxFblt4w4lAdiaFWeCU8grlYO79qOs4/szEOTF7Buy464w3Eu7yW3Kh8dU6vyTHgCcbV245mlbNmxi/tfmxd3KM7lrVxrVZ4JTyCu1ko6tWHkUV155I2FrNq0Pe5wnMs7udiqPBOeQFxGbjizlJ27jXtf9aMQ52ojV1uVZyI/o3ax692hFV8b1I3H3lrMig3b4g7HuZyX663KM+EJxGXs+tNLMDPGvlIRdyjO5bQ1edCqPBNZTSCSzpI0W1KFpFtSjJeku8P4mZIGJYxbKOk9SdMllScM/7mkZWH4dElnZ7MOrmo9DmzJhccezFPlS1iybmvc4TiXk6YtWseX8qBVeSaylkAkFQL3ACOA/sBFkvonFRsBlITHKODepPFDzWygmZUlDb8jDB9oZs9nIXyXpuuG9kMSd788N+5QnMsp+daqPBPZPAIZDFSY2Xwz2wE8AYxMKjMSeNQiU4D2krpkMSZXxzq3a84lQ3ry9LvLmL96c9zhOJcT8rFVeSaymUC6AYl/ILE0DEu3jAHjJU2TNCppujHhlNdDkg5ItXBJoySVSypfvXp15rVwNbrmtL40LSzgLj8KcS5vW5VnIpsJJFVTSqtFmRPNbBDRaa7rJJ0Sht8L9AUGAiuA21It3MweMLMyMysrLi6ubeyuFjq0bsblJ/bimRnLmbNyU9zhOBebfG5VnolsJpClQOIJv+7A8nTLmFnl8ypgHNEpMcxspZntNrM9wIOVw128Rp3ch1ZNi7hjwpy4Q3Gu3jWEVuWZyGYCeRsokdRbUlPgQuCZpDLPAJeGu7GGABvMbIWkVpLaAEhqBQwH3g/vE6+RfLVyuIvXAa2a8q2TevN/73/M+8s2xB2Oc/WmobQqz0TWEoiZ7QLGAC8CHwJPmdksSaMljQ7FngfmAxVERxPXhuGdgMmSZgBTgefM7IUw7tZwe+9MYChwY7bq4GrnWyf3pl2LJn4U4hqNhtSqPBNF2Zx5uMX2+aRh9yW8NuC6FNPNB46qYp6X1HGYro60bd6EUaf04bcvzuadxZ8w6OCU9zc4l/d27zHumDCHsRMrOKxLW+67eFCDaBhYW40nVbp6cfkJvTioVVM/CnENVkNtVZ4JTyCuTrVqVsQ1p/Xl9blreGv+2rjDca5ONeRW5ZnwBOLq3MVDetKxTTNuGz+H6Cylc/ktsVV506KG2ao8E55AXJ1r3qSQMaf3Y+rCdUyuWBN3OM7tl+RW5f+8vmG2Ks+EJxCXFV8/tgdd2zX3oxCX1xpTq/JMeAJxWdGsqJDvnFHC9CXreeWjVXGH41ytNbZW5ZnwBOKy5mvHdKfnQS25fcIc9uzxoxCXHxprq/JMeAJxWdOksIAbzihh1vKNvDjr47jDca5GjblVeSY8gbisGjmwG32LW3HHS3PY7UchLocltiq/vxG2Ks+EfzouqwoLxI3DSpmzcjPPzkzuS9O5+KX6r/Iv5Pl/ldcXTyAu684+oguHdm7DnS/NZdfuPXGH49xnvFX5/vEE4rKuoEDcNKyUBWu28PS7y+IOxznAW5XXBU8grl4M69+JAd3bcffLc9mxy49CXHy8VXnd8QTi6oUUHYUs/WQbT5UvqXkC57LAW5XXLU8grt6cWlpMWc8DGPtKBdt37o47HNfIeKvyuucJxNUbSdw0vJSPN27n8bcWxx2Oa0S8VXl2eAJx9eqEvh04oe9B/OHVCrbu2BV3OK6B275zNz/yVuVZ4wnE1bubh5eyZvMOHn1zUdyhuAasslX5496qPGs8gbh6d0zPAzntkGLumzSPTdt3xh2Oa4AqW5UvXOutyrPJP1EXi5uGlbJ+607+/K+FcYfiGhBvVV6/PIG4WAzo3p7h/Tvx4Ovz2bDVj0Lc/vNW5fUvqwlE0lmSZkuqkHRLivGSdHcYP1PSoIRxCyW9J2m6pPIU035PkknqkM06uOy5aXgpmz/dxYOvz487FJfnvFV5PLKWQCQVAvcAI4D+wEWS+icVGwGUhMco4N6k8UPNbKCZlSXNuwcwDPB7QfPYoZ3b8sUju/DQvxawdvOncYfj8pC3Ko9XNo9ABgMVZjbfzHYATwAjk8qMBB61yBSgvaQuacz7DuD7gPcPnue+e2Yp23fu5v7X/CjE1Y63Ko9fNhNINyCxz4qlYVi6ZQwYL2mapFGVBSR9BVhmZjPqPmRX3/p1bM05R3fjkTcWsmrj9rjDcXnCW5XnhmwmkFTNPJOPGKorc6KZDSI6zXWdpFMktQR+DPy0xoVLoySVSypfvXp1beJ29eyGM0rYvcf4w6vz4g7F5QFvVZ47splAlgKJJyO7A8n/KFRlGTOrfF4FjCM6JdYX6A3MkLQwlH9H0j736ZnZA2ZWZmZlxcXFdVIhlx09D2rF+WXdefytxSxbvy3ucFyO8lbluSebCeRtoERSb0lNgQuBZ5LKPANcGu7GGgJsMLMVklpJagMgqRUwHHjfzN4zs45m1svMehEloEFm5n+4nefGnF4CwNhXKmKOxOWivVqVn+KtynNFUbZmbGa7JI0BXgQKgYfMbJak0WH8fcDzwNlABbAVuCJM3gkYFw5Li4DHzeyFbMXq4tetfQsuGtyDx95azOhT+/j9++4zEz9axXefnM4eM+6/5BhvGJhDZNbwb2QqKyuz8vJ9mpK4HLNq43ZOvnUiXxzQhdsvGBh3OC5mu/cYd0yYw9iJFRzWpS33XTzIf1jUM0nTkptRJPKW6C5ndGzbnEuP78k/3l1GxarNcYfjYuStyvODJxCXU0af2pfmTQq586U5cYfiYuKtyvOHJxCXUw5q3YwrTuzFszNX8OGKjXGH4+qRtyrPP55AXM4ZdXJf2jQv4o4JfhTSWHir8vzkCcTlnHYtm3DVSX0Y/8FK3lu6Ie5wXJZ5q/L85QnE5aQrT+pF+5ZNuG3C7LhDcVnkrcrzmycQl5PaNG/C6FP78urs1UxbtC7ucFwd81blDYMnEJezLj2+Jx1aN+W28X4tpCHxVuUNhycQl7NaNi3i2tP68ca8tbwxb03c4bg6sM9/lZ/t/1Wez3zNuZz2jeMOpnPb5tw+fg6NodeEhsr/q7xh8gTiclrzJoWMOb0f5Ys+4bW5fhSSj7xVecPlCcTlvAvKetD9gBbcNn62H4XkGW9V3rB5AnE5r2lRAd85o4SZSzcw4YOVcYfj0uCtyhsHTyAuL5x7dDd6d2jF7RPmsGePH4XkMm9V3nh4AnF5oaiwgO+eWcJHH2/i+fdXxB2Oq4K3Km9csvaHUs7VtS8N6MrYVyq4Y8IcTi0tpkCistGyiF4kN2KuarzgsxbPSi7rLaEzMu7dpfzo6fdp1ayIx646zhsGNgKeQFzeKCwQNw0r5ZrH3uHIn4+vt+WmlXRIKJQ0PlXyIsW89p1WNU6jpImj5aWOrap5JdYz3WkSFytgj8GCNVsY3PtAxl50tDcMbCQ8gbi8ctYRnbn1vAGs37qDyhuyKq+ImIGFd8k3a1XevZXONEbCyDTLfv5+7xFWw7ITp0mM2VIsO1VZ22u6pHEppkkn3uTl7xv33rFVvjjvmO5cfUofbxjYiHgCcXlFEheU+d08zuUC/6ngnHMuI55AnHPOZSSrCUTSWZJmS6qQdEuK8ZJ0dxg/U9KghHELJb0nabqk8oThvwplp0saL6lrNuvgnHMutawlEEmFwD3ACKA/cJGk/knFRgAl4TEKuDdp/FAzG2hmZQnDfmtmA8xsIPAs8NNsxO+cc6562TwCGQxUmNl8M9sBPAGMTCozEnjUIlOA9pK6VDdTM9uY8LYVCTeDOOecqz/ZTCDdgCUJ75eGYemWMWC8pGmSRiVOJOnXkpYA38SPQJxzLhbZTCCpmvMmHy1UV+ZEMxtEdJrrOkmnfFbA7Mdm1gN4DBiTcuHSKEnlkspXr15d++idc85VK5sJZCmQeMN+d2B5umXMrPJ5FTCO6JRYsseBr6VauJk9YGZlZlZWXFycUQWcc85VLZsNCd8GSiT1BpYBFwLfSCrzDDBG0hPAccAGM1shqRVQYGabwuvhwC8BJJWY2dww/VeAj2oKZNq0aWskLcqwHh2AhvJPRl6X3NNQ6gFel1y1P3XpWd3IrCUQM9slaQzwIlAIPGRmsySNDuPvA54HzgYqgK3AFWHyTsC40O9OEfC4mb0Qxv1G0iHAHmARMDqNWDI+BJFUnnQXWN7yuuSehlIP8LrkqmzWJatdmZjZ80RJInHYfQmvDbguxXTzgaOqmGfKU1bOOefql7dEd845lxFPIDV7IO4A6pDXJfc0lHqA1yVXZa0usuR+r51zzrk0+BGIc865jHgCcc45lxFPIEEaPQePTOgFuFzSSXHEmY6a6pJQ7lhJuyWdV5/xpSuNdXKapA1hnUyXlLPd2qSzTkJ9pkuaJWlSfceYrjTWy78lrJP3wzZ2YByxVieNerST9E9JM8I6uSLVfHJBGnU5QNK48B02VdIRdbJgM2v0D6J2KvOAPkBTYAbQP6lMaz6/ZjQA+CjuuDOtS0K5V4husz4v7rgzXCenAc/GHWsd1aU98AFwcHjfMe6492f7Sij/ZeCVuOPOcJ38CPiv8LoYWAc0jTv2DOvyW+Bn4fWhwMt1sWw/AonU2HOwmW228OmT270Ap9MLMsD1wP8Aq+ozuFpItx75IJ26fAN42swWw2dd+OSi2q6Xi4C/1ktktZNOPQxoo6hFc2uiBLKrfsNMSzp16Q+8DGBmHwG9JHXa3wV7Aomk03Mwkr4q6SPgOeDKeoqttmqsi6RuwFeB+8hdaa0T4PhwiuH/JB1eP6HVWjp1KQUOkPRq6IH60nqLrnbSXS9IagmcRfRDJdekU4+xwGFE/fO9B9xgZnvqJ7xaSacuM4BzASQNJuqipPv+LtgTSCSdnoMxs3FmdihwDvCrbAeVoXTqcifwAzPbnf1wMpZOPd4BeprZUcDvgX9kO6gMpVOXIuAY4IvAF4B/l1Sa7cAykNa+EnwZ+JeZrctiPJlKpx5fAKYDXYGBwFhJbbMbVkbSqctviH6gTCc6+/AudXA0ldWuTPJIOj0Hf8bMXpPUV1IHM8u1DtfSqUsZ8EToa6wDcLakXWb2j3qJMD011sMS/lzMzJ6X9Ic8XidLgTVmtgXYIuk1ou585tRPiGmrzb5yIbl5+grSq8cVwG/CqesKSQuIrh9MrZ8Q05buvnIFRH8lDiwIj/0T9wWgXHgQJdL5QG8+vwh1eFKZfnx+EX0QUQ/Dijv2TOqSVP5hcvMiejrrpHPCOhkMLM7XdUJ0quTlULYl8D5wRNyxZ7p9Ae2Irhm0ijvm/Vgn9wI/D687hX2+Q9yxZ1iX9oQbAIBvE/0T7H4v249ASLvn4K8Bl0raCWwDvm5hbeSSNOuS89Ksx3nANZJ2Ea2TC/N1nZjZh5JeAGYS9TT9RzN7P76oU6vF9vVVYLxFR1Q5J816/Ap4WNJ7RKeJfmC5d3Sbbl0OAx6VtJvobr9v1cWyvSsT55xzGfGL6M455zLiCcQ551xGPIE455zLiCcQ55xzGfEE4pxzLiOeQFyDI+nHoffUyt6Tj6uh/MOZ9EgsqZekb2QwXcrlSTo0xPuupL4ZzPe7ofsQ5+qFJxDXoEg6HvgSMMjMBgBnsnc/QXWpF1EniHXlHOB/zexoM5uXwfTfJWqEmDZJ3hbMZcwTiGtouhB1CfIpgJmtMbPlAJKOkTQpdFb4oqQuyRNXVUZSP0kvhY4b3wlHCL8BTg5HDTdKKpT0W0lvh6Ofq8O0kjRW0geSngM6plju2UQJ4CpJE8Owi8N/N0yXdL+kwjD8XkX/STNL0i/CsO8Q9dk0MWH6zQnzP0/Sw+H1w5JuD+X+K3TL80Ko8+uSDg3lzlf0fx4zQtcqzu0t7mb4/vBHXT6Iut2eTtSH1B+AU8PwJsAbQHF4/3WiFrsQunOpocxbwFfD6+ZEv/RPI+H/SIBRwE/C62ZAOVH3EucCE4haCXcF1pOi+xjg58D3wuvDgH8CTcL7PwCXhtcHhudC4FVgQHi/kISuNoDNCa/PAx5OqO+zQGF4/zJQEl4fR/j/DqIeaLuF1+3jXrf+yL2HH766BsXMNks6BjgZGAo8qegf2sqBI4AJoRPJQmBF0uSHpCojqQ3RF+m4sIztAKFMouHAgITrG+2AEuAU4K8W9X68XNIraVTlDKLeed8Oy2nB5//dcoGkUUR9IHUh+q+HmWnMM9HfzGy3pNbACcDfEurTLDz/i6grj6eAp2s5f9cIeAJxDU74on4VeDX0Y3QZMA2YZWbHVzOpUpWpRRfeAq43sxeTpj+b2v8BmYBHzOyHSfPqDXwPONbMPgmnpZpXMY/EZSaXqeyjqgBYb2YD95nYbHS4AeGLwHRJA81sbS3r4RowvwbiGhRJh0gqSRg0EFgEzAaKw0V2JDXRvn9AlbKMRV1hL5V0ThjeLNzttAlokzD9i0SdOzYJ5UoltQJeAy4M10i6EB0Z1eRl4DxJHcO8DpTUE2hL9OW/QdE/yo1ImCY5npWSDpNUQNS54T5C3RZIOj8sR5KOCq/7mtlbZvZTYA17dxnunB+BuAanNfB7Se2J/jCnAhhlZjvCqaW7JbUj2vbvBGZVTlhDmUuA+yX9EtgJnE902miXpBlE1xXuIroz6x1F54NWE91ZNQ44neiawhxgUk2VMLMPJP0EGB8SwE7gOjObIundENN8otNMlR4A/k/SCjMbCtxCdK1jCVH38K2rWNw3gXvD8poQ/SXqDOC3IRmLKKHNqClu17h4b7zOOecy4qewnHPOZcQTiHPOuYx4AnHOOZcRTyDOOecy4gnEOedcRjyBOOecy4gnEOeccxn5/3vvsBlZ4/0QAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pyplot.plot(selection_quota_list, map_list)\n",
    "pyplot.ylabel('MAP')\n",
    "pyplot.xlabel('Selected features')\n",
    "pyplot.title(\"Features selected from the highest weight to the lowest\")\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Selected 3031/10106 features, weights: 0.00E+00 to 0.00E+00\n",
      "ItemKNNCBFRecommender: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "ItemKNNCBFRecommender: ICM Detected 8696 (81.4%) items with no features.\n",
      "Similarity column 10681 (100.0%), 15491.99 column/sec. Elapsed time 0.69 sec\n",
      "EvaluatorHoldout: Processed 69805 (100.0%) in 1.60 min. Users per second: 726\n",
      "Selected 5053/10106 features, weights: 0.00E+00 to 0.00E+00\n",
      "ItemKNNCBFRecommender: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "ItemKNNCBFRecommender: ICM Detected 7993 (74.8%) items with no features.\n",
      "Similarity column 10681 (100.0%), 15773.41 column/sec. Elapsed time 0.68 sec\n",
      "EvaluatorHoldout: Processed 69805 (100.0%) in 1.61 min. Users per second: 722\n",
      "Selected 7074/10106 features, weights: -1.11E-02 to 1.11E-02\n",
      "ItemKNNCBFRecommender: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "ItemKNNCBFRecommender: ICM Detected 4005 (37.5%) items with no features.\n",
      "Similarity column 10681 (100.0%), 9578.82 column/sec. Elapsed time 1.12 sec\n",
      "EvaluatorHoldout: Processed 69805 (100.0%) in 2.13 min. Users per second: 547\n",
      "Selected 9095/10106 features, weights: -4.83E-02 to 4.85E-02\n",
      "ItemKNNCBFRecommender: URM Detected 46 ( 0.4%) items with no interactions.\n",
      "ItemKNNCBFRecommender: ICM Detected 3289 (30.8%) items with no features.\n",
      "Similarity column 10681 (100.0%), 8981.49 column/sec. Elapsed time 1.19 sec\n",
      "EvaluatorHoldout: Processed 69805 (100.0%) in 2.15 min. Users per second: 542\n"
     ]
    }
   ],
   "source": [
    "argsort_features = np.argsort(np.abs(CFW_weithing.D_best))\n",
    "\n",
    "map_list = []\n",
    "selection_quota_list = [0.3, 0.5, 0.7, 0.9]\n",
    "\n",
    "for selection_quota in selection_quota_list:\n",
    "    \n",
    "    n_to_select = int(selection_quota*len(argsort_features))\n",
    "    selected_features = argsort_features[:n_to_select]\n",
    "    \n",
    "    ICM_selected = ICM_tags[:,selected_features]\n",
    "    \n",
    "    print(\"Selected {}/{} features, weights: {:.2E} to {:.2E}\".format(n_to_select, len(argsort_features),\n",
    "                                                                min(CFW_weithing.D_best[selected_features]),\n",
    "                                                                max(CFW_weithing.D_best[selected_features])))\n",
    "    \n",
    "    itemKNNCBF = ItemKNNCBFRecommender(URM_train, ICM_selected)\n",
    "    itemKNNCBF.fit(**best_hyperparams_ItemKNNCBF)\n",
    "    \n",
    "    results_df, _ = evaluator_test.evaluateRecommender(itemKNNCBF)\n",
    "    map_list.append(results_df.loc[10][\"MAP\"])\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaUAAAEWCAYAAADGjIh1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAxSElEQVR4nO3deZgcZbn38e9vJpOFrEAWspGEbBCQNWwisodFERRUcEFxQRRQEPBwPOc9rx71VQ+L7CACIi6g5yAeRDSBQFhlCTsBkkw2EhLIAknIRjIz9/tH1YROpyfTM0lP9fT8PtfV13RVPVV1P7Xd/VTVVCkiMDMzKwdVWQdgZmbWyEnJzMzKhpOSmZmVDSclMzMrG05KZmZWNpyUzMysbDgptSFJX5b0WBnEMUXS15oYJkm/lvSupKfbOrZCJM2VdPQ2mlZIGtXEsM9LmlTkdMpiXebblssqZ5q3SfrxtpxmKbX39SjpB5J+t42m1a7WHZQwKaU7x1pJq3I+g7bBNLfpDtfeSBqeHlg7lWgWHwGOAYZExAElmkeTstyJIuL3ETEhi3k32lLSbA/K4SC4Ldfjln7ApcO3an+UdLikBa2PsH1oSfIvdUvpxIjokfNZWOL5bVEJD+SVZBgwNyJWFxroZWhmpdTmp+8k9ZZ0i6RFkt6U9GNJ1emwkZIelLRM0lJJv5fUJx32W2Bn4K9pq+t7hX5l5Lam0mbw/0j6naSVwJebmf8oSQ9LWpHO/49N1KFrOs1lkpZLekbSgObqV2A6u0q6X9I7kqZL+kzOsG6SLpc0L43nMUndgEfSIsvT5XBwWv4rkl5LT7tNlDQsZ1rHSHo9nc61gJqI56vAzcDB6bR/2LiMJf2LpLeAX0vqIulKSQvTz5WSuqTTaCz/PUmL0+VwsqQTJM1I6/r9JuZ/FvB54Hvp/P+aM3hvSS+ldfijpK45431c0gvpunhC0p6Fpp/jaEkz02V1nSSl09nk15ykCel6WSHp+nTb2ORXs6TL0unMkXR8Tv8Wb2eSGtfti2n9P1tgGTW5j+TYX9KraVy/blxWkvpKujddTu9IelRSVTpsNyWtguWSpkn6RKEFl7+M0n6R1qng+pM0SNJdkpaky+nbTUx7RDr/xphulrQ4Z/jvJJ1fxPLdJutR0k+AQ4Fr0/pcWyDszfZHSVWS/l3JvrtY0u2Seheob3fg78AgbX42qXM63nvp+hifM15Ry7OJZfx1SbXp+r+ncX5K9vVr0u81klZL+q+0u5ukdZK2T7sPUrKfLZf0oqTDc6b/ZUmz07jnKDmVuhtwIx8cV5ZvMciIKMkHmAscXaD/X4BfAt2B/sDTwDfSYaNITh11AfqRrPArm5omcDiwoKn5Aj8ANgAnkyTgbs3M/w7g39KyXYGPNFG3bwB/BbYDqoH9gF5F1O/LwGPp9+7AfOBMoBOwL7AU2D0dfh0wBRiczuPD6XIZDgTQKSeek4FaYLd0Wv8OPJEO6wusBE4FaoALgDrga03UbWOMOcu4Dvh5Ov9uwH8CT6b16wc8Afwor/x/pPP7OrAE+APQE9gdWAfs0sT8bwN+XGCdPg0MAnYAXgPOToftCywGDkyX05fS8l2amH4A9wJ9SH7kLAGOK7B+Gpfbp9Jl+h2SbelrOWU3pPWrBr4JLARUxHbQ5HaWxjdqC/tVMfvIK8DQdFk93rg8gZ+SHBxq0s+hJD9Qaki2n+8DnYEjgfeAsfnrhLztIz/m/PWX1vHZdHvoDOwCzAaObaJ+bwD7pd+np2V3yxm2Twv3s61dj1NoYl9Jhw9n8/3xK+ny3AXoAfwZ+G0T4x/O5sewH5DsIyekMf0UeLKVyzN33R1JcozZl2T7uQZ4JGfYy+n3DwOzgKdyhr2Yfh8MLEtjqyLZFpeRbIvd02XduN0M5IPj2cZ10mzuKKZQaz4kO8cqYHn6+QswAHgf6JZT7nTgoSamcTLwfN40W5qUHskZtsX5A7cDN5FcT9lS3b5CciDeM69/c9PfuGKAzwKP5o3/S+D/pit7LbBXkTvB34Gv5h0I1pCcijuDdINOhwlYQMuS0nqga06/WcAJOd3Hkpzyayy/FqhOu3um8R6YU/5Z4OTmdqK8dfqFnO7/Am5Mv99AmhBzhk8HDmti+sGmSeBPwCUF1s8ZwD/zltt8Nj2Y1eYM3y6d9k5bs53RTFIqch85O6f7BGBW+v0/gf/Nnz5JcnoLqMrpdwfwg/x1kr995Mecv/5Ifiy8kVf+X4FfN1Gf3wLfTZfj9HRdnw2MIDmOVBWxfLfJeky7p9DypDQZ+FZO91iSxNepwPiHUzgpPZDTPQ5Y28rlmbvubgH+K2dYjzSu4SQ/NtcBOwKXkPxAWZCW+SFwdTrOv5CXYIGJJD8Gu6fr6JTcddPUdtPUp9TXB06OiAcaOyQdQPKrbJG08QxSFclGgqT+wNUkO0nPdNi7WxnD/Jzvw7Y0f+B7wI+ApyW9C1weEbcWmOZvSX6J3pmeOvkdyS/f5qafaxhwYF5TtlM67b4kv6BnFVnHYcBVki7P6SeSXzWDcucfESGpUDxbsiQi1uV0DwLm5XTPS/s1WhYR9en3tenft3OGryXZ2FvirZzva3LmNwz4kqTzcoZ3zounuWkViqXQcsu/IP1WzvA16TrvQdJC2Rbb2WaK3Edy12/uurmU5IA3KY3rpoj4WWNdI6Ihb7zBxcTUjGEkp6eW5/SrBh5tovzDwCdIDoiPkCSFL5IcMB+NiAYlp6aL3c+2Zj22VqH9oxNJMn2zyGnkb6NdlVzPbenyzI/rucaOiFglaRkwOCLmSpoKHAZ8FPgJsDdwSNrvmnS0YcCnJZ2YM90akh8Eq5Wccr4IuEXS48CFEfF6kXUGKHlSyjef5BdO34ioKzD8pyS/OvaMiGWSTgZyz+NGXvnVJL9sAEjPKffLK5M7zhbnHxFvkTTjkfQR4AFJj0REbV65DSS/Hn4oaThwH8mvuvuaqV+u+cDDEXFM/oD0nPo6YCTw4hbqkzutn0TE7wtMazRJAm3sVm53kfLnuZBk45yWdu+c9tsWCtVvSxrr/pNtNP9Gi4AhjR3pchvSdPHNYtrq7awJze0jsOn63bhuIuI94ELgQkm7Aw9JeiYdPlRSVU5i2hmYUWD++fvcTvnVy+ueD8yJiNFF1A2SpHQpSVJ6GHiM5JTjurS7cZrF7mdbsx6h+e2x0PDG/aPRziSntN8uULY123tLlmeTcaXXtHbkg0T5MMmpun2AZ9LuY4ED+ODa2XySltLXC80gIiYCE5Vc//4x8CuSH1BF17NNb3SIiEXAJOBySb3SC4IjJR2WFulJespP0mDg4rxJvE1yDrXRDJJfEB+TVENyLaVLa+cv6dOSGjfYd0kWZH3+dCQdIelDaRJcSdIEri+ifrnuBcZI+mJ6YbFG0v6SdksPDLcCV6QXNauVXEDtQnINpCFvOdwI/Gt6oGm8CPzpdNjfgN0lfSr9pfVtklMjW+MO4N8l9ZPUl+T89jb5vwo2X8fN+RVwtqQDleiebg89tzKOvwEfUnKTRifgHIpcblu5nTVX/+b2EYBzJA2RtAPJaZjGGyk+ruSGBJFst/Xp5ymSZPO9dDs8HDgRuLPAtF8k2Z72VnIDxQ/yhufH/zSwUsmNMt3SbXkPSfsXqlxEzCRpSX+B5NT7ynSap5AmpRbuZ61ej03UJ1+h/fEO4AIlN270AP4f8McmEujbwI4qcCNEE1q0PPP8ATgzXXdd0rieioi56fCHSU53vhoR60lPXZIkwSVpmd8BJ0o6Np13VyU3Nw2RNEDSJ9Jk9z7Jdpq7XQ+R1Lm5ILP459kzSE6vvEqyQ/4PyQUxSFof+wIrSDamP+eN+1OSg+FySRdFxArgWyR3jL1JsmM1d8//lua/P/CUpFXAPcB3ImJOgWnslI63kuSi+8N8cFDe0vQ3Sn+1TgBOI/kF8xYf3EwASRP4ZZJfLO+kw6oiYg1J0/rxdDkcFBF3p8PvVHKX4SvA8el8lgKfBn5GckFyNMnF763xY2Aq8FIa43Npv23hFmBcWre/NFc4IqaStDquJVnetSTnr7dKznL7L5LlNo6kzu8XOYnWbmc/AH6T1v8zbK65fQSSg88kkgvgs/lg3YwGHiA5WPwTuD4ipqQHoE+QbDNLgeuBMwqddomIGSTXph4AZpK0ZHJtsv7S07gnkpwKmpNO/2ZgSwfhh0lOAb+R0y3g+Zwyxe5nW7serwJOVXJn3tUFpr/Z/kjyg/K3JK2LOSStvPPyx03Hf50kic1Ox9/i/3K2cnk2jjsZ+D/AXSQtyJEkx59GT5BcW2psFb2axv5IzjTmAyeR/NhZQtJyupgkl1SRtMQXkhyzDiM5PgM8SHJm5S1JS7cUZ+MdJma2BUpOqS4APh8RD2Udj7WO12P582OGzJqQnqLok57q+D7Jr/UnMw7LWsjrsX1xUjJr2sEkd0AuJTllcnJErN3yKFaGvB7bEZ++MzOzsuGWkpmZlY2Kerhm3759Y/jw4VmHYWbWbjz77LNLIyL//zszU1FJafjw4UydOjXrMMzM2g1J85ov1XZKevpO0nFKns5bK+mSAsMl6ep0+EuS9s0Z1kfJE75fV/L064NLGauZmWWvlC/5qyZ50vXxJP+wdrqkcXnFjif5h77RwFkkD9dsdBXwj4jYFdiL5J9UzcysgpWypXQAydN3Z6f/MX4nyX8C5zoJuD0STwJ9JA2U1IvkoYC3AETE+ohYXsJYzcysDJQyKQ1m06f2LmDzpw43VWYXkkdY/FrS80pe9tW90EwknSVpqqSpS5YsKVTEzMzaiVImpUJvN83/p6imyjS+9O6GiNiH5Jl2m12TAoiImyJifESM79evbG4gMTOzVihlUlrApo/QH8LmrzdoqswCkhdfPZX2/x+SJGVmZhWslEnpGWB0+vj2ziRPo70nr8w9wBnpXXgHASsiYlH6vpn5ksam5Y4ieWKtmZlVsJL9n1JE1Ek6l+RVudXArRExTdLZ6fAbSV6KdwLJ6wbWAGfmTOI84PdpQpudN8zMrF2JCOobgg31wfr6Bjakn7rc7rrke119Axvqgw31DZuUbey3oa5h43Tq6oMuNVWcfdjIrKu4TVTUs+/Gjx8f/udZs46h0EG+Lv9AvoWD/KbJYNODfGMS2LxspGUbCpSNJhJNOl5DA6U63Pbv2YWn/+3oVo0r6dmIGL+NQ2q1inqig5mV1uKV61iwfO0HB9q8g3OhA/GGuk0P8puW/eAgv6Eh53vOQb4xoWwy7YZkWKkO8tVVoqZa1FRXpZ/ke+fG7k6iU1XS3aWmih5dO21SrvHTOe3ulPO9plNVgbLaOO1Ojd+bKVfTqYpOVcmw6qpC94y1T05KZlaUuUtXc9xVj7BuQ0OLx/VB3orlpGRmRfnFAzMAuOmL+9GjSycf5K0knJTMrFnT33qPe15cyDc+OpIJu++UdThWwfw+JTNr1uWTptOjcyfOPmyXrEOxCuekZGZb9ML85Ux69W2+dugu9Nmuc9bhWIVzUjKzLbp80nR26N6Zrx46IutQrANwUjKzJv1z1jIenbmUbx42kh5dfAnaSs9JycwKiggumzSdAb268MWDh2UdjnUQTkpmVtCU6Ut4dt67nHfkaLrWVGcdjnUQTkpmtpmGhqSVNHSHbnxm/NDmRzDbRpyUzGwzf3/lLaYtXMn5R42hcycfJqzteGszs03UNwRX3D+dUf17cPI++S+LNistJyUz28Tdz7/JrCWrufCYMX48kLU5JyUz22h9XQNXPjCDPQb34rg9/Dgha3tOSma20R+feYMF767lwgljkdxKsrbnpGRmAKxdX881D9ay//DtOXxMv6zDsQ7KScnMALj9n3NZ/N77XORWkmXIScnMeG/dBm54eBaHju7LgbvsmHU41oE5KZkZtzw2h+VrNnDxsWOzDsU6OCclsw7u3dXrufnRORy7+wD2HNIn63Csg3NSMuvgbnx4FqvX13HhBLeSLHtOSmYd2OKV6/jNP+dy8t6DGTOgZ9bhmDkpmXVk1z5US119cP7Ro7MOxQxwUjLrsOa/s4Y7nn6DT48fyrAdu2cdjhngpGTWYV01eSaS+PZRo7IOxWyjkiYlScdJmi6pVtIlBYZL0tXp8Jck7ZszbK6klyW9IGlqKeM062hqF6/iz88t4IsHDWNg725Zh2O2UadSTVhSNXAdcAywAHhG0j0R8WpOseOB0ennQOCG9G+jIyJiaaliNOuofnH/DLrWVPPNw0dmHYrZJkrZUjoAqI2I2RGxHrgTOCmvzEnA7ZF4EugjaWAJYzLr8F55cwV/e3kRX/3ICPr26JJ1OGabKGVSGgzMz+lekPYrtkwAkyQ9K+mspmYi6SxJUyVNXbJkyTYI26yyXXH/DHp17cTXDt0l61DMNlPKpFToiY7RgjKHRMS+JKf4zpH00UIziYibImJ8RIzv189PNjbbkmfnvcODry/mG4eNpHe3mqzDMdtMKZPSAmBoTvcQYGGxZSKi8e9i4G6S04Fm1koRwaUTp9O3R2fOPGR41uGYFVTKpPQMMFrSCEmdgdOAe/LK3AOckd6FdxCwIiIWSeouqSeApO7ABOCVEsZqVvEer13Gk7Pf4ZwjRrFd55Ld42S2VUq2ZUZEnaRzgYlANXBrREyTdHY6/EbgPuAEoBZYA5yZjj4AuDt9p0sn4A8R8Y9SxWpW6ZJW0usM6t2Vzx24c9bhmDWppD+XIuI+ksST2+/GnO8BnFNgvNnAXqWMzawjuf/Vt3lxwQp+fsqH6NKpOutwzJrkJzqYVbiGhuDySTMY0bc7p+w7JOtwzLbIScmswv31pYVMf/s9zj96NJ2qvctbefMWalbBNtQ38Iv7Z7DrTj05cc9BWYdj1iwnJbMKdtezC5i7bA0XThhLVVWhfws0Ky9OSmYVat2Geq6aPJO9hvbh6N36Zx2OWVGclMwq1B+eeoNFK9bxvWPHkv57hVnZc1Iyq0Br1tdx/ZRaDt5lRw4Z1TfrcMyK5qRkVoF+/fhclq5az0XHjs06FLMWcVIyqzAr1m7glw/P4qhd+7PfsO2zDsesRZyUzCrMrx6Zzcp1dXx3wpisQzFrMSclswqydNX73Pr4HD6250B2H9Q763DMWsxJyayCXP/QLNZtqOe7x7iVZO2Tk5JZhVi0Yi2/e2oep+w7hJH9emQdjlmrOCmZVYirJ9cSEXz7qNFZh2LWak5KZhVg7tLV/GnqfE4/YGeG7rBd1uGYtZqTklkFuPKBGdRUi3OPGJV1KGZbxUnJrJ2b/tZ7/O+LC/nSh4fTv1fXrMMx2ypOSmbt3OWTptOjcyfO/ujIrEMx22pOSmbt2IvzlzPp1bf52qG7sH33zlmHY7bVnJTM2rHLJk1n++1q+MpHhmcditk24aRk1k49OXsZj85cyjcPH0nPrjVZh2O2TTgpmbVDEcFlE6czoFcXzjh4eNbhmG0zTkpm7dCUGUuYOu9dzj1yNF1rqrMOx2ybcVIya2caGpJW0pDtu/HZ8UOzDsdsm3JSMmtn/jHtLaYtXMkFR4+hcyfvwlZZSrpFSzpO0nRJtZIuKTBckq5Oh78kad+84dWSnpd0bynjNGsv6huCyydNZ1T/Hpy8z+CswzHb5kqWlCRVA9cBxwPjgNMljcsrdjwwOv2cBdyQN/w7wGulitGsvbn7+TeZtWQ13z1mDNVVyjocs22ulC2lA4DaiJgdEeuBO4GT8sqcBNweiSeBPpIGAkgaAnwMuLmEMZq1G+vrGrjygRnsMbgXx+2+U9bhmJVEKZPSYGB+TveCtF+xZa4Evgc0bGkmks6SNFXS1CVLlmxVwGbl7I9T57Pg3bVcOGEsVW4lWYUqZVIqtNdEMWUkfRxYHBHPNjeTiLgpIsZHxPh+/fq1Jk6zsrd2fT3XTJ7J+GHbc/gYb+dWuUqZlBYAuferDgEWFlnmEOATkuaSnPY7UtLvSheqWXn77ZNzWfze+1x87Fgkt5KscpUyKT0DjJY0QlJn4DTgnrwy9wBnpHfhHQSsiIhFEfGvETEkIoan4z0YEV8oYaxmZeu9dRu4YcosDh3dlwN32THrcMxKqlOpJhwRdZLOBSYC1cCtETFN0tnp8BuB+4ATgFpgDXBmqeIxa69ufWwu767ZwEUTxmYdilnJlSwpAUTEfSSJJ7ffjTnfAzinmWlMAaaUIDyzsvfu6vX86tHZHLv7APYa2ifrcMxKzv8OblbGbnxkFqvX13GhW0nWQTgpmZWpxSvX8Zsn5nLSXoMYM6Bn1uGYtQknJbMyde1DtdTVB+cfPSbrUMzajJOSWRma/84a7nj6DT49fijD+3bPOhyzNuOkZFaGrp48E0l8+6hRWYdi1qaclMzKTO3iVdz13AK+eNAwBvbulnU4Zm3KScmszPzigRl0ranmm4ePzDoUszbnpGRWRqYtXMHfXlrEVw4ZQd8eXbIOx6zNOSmZlZHLJ82gV9dOfP2ju2QdilkmnJTMysSz897lwdcX843DRtK7W03W4ZhlwknJrAxEBJdOfJ2+PTpz5iHDsw7HLDNOSmZl4PHaZTw5+x3OOWIU23Uu6SMpzcqak5JZxiKCSydNZ1DvrnzuwJ2zDscsU05KZhl74LXFvDh/Od8+ajRdOlVnHY5ZppyUzDLU0BBcPmk6I/p255T9hmQdjlnmnJTMMvTXlxby+lvvcf7Ro6mp9u5o5r3ALCMb6hv4xf0z2HWnnpy456CswzErC05KZhm569kFzF22hu8eM4aqKmUdjllZcFIyy8D7dfVcPXkmew3twzHjBmQdjlnZcFIyy8AfnnqDhSvWcfGEsUhuJZk1clIya2Nr1tdx3UO1HLTLDhwyaseswzErK0UlJUl9Sx2IWUfx68fnsnTVei4+1q0ks3xbTEqSTpS0BHhZ0gJJH26juMwq0oq1G/jlw7M4ctf+7Ddsh6zDMSs7zbWUfgIcGhEDgVOAn5Y+JLPKdfOjs1m5ro4LJ4zJOhSzstRcUqqLiNcBIuIpoGfpQzKrTEtXvc8tj83hY3sOZPdBvbMOx6wsNfc44v6SvttUd0RcUZqwzCrPDVNmsW5DPRcc7VaSWVOaayn9iqR11PjJ7e7R3MQlHSdpuqRaSZcUGC5JV6fDX5K0b9q/q6SnJb0oaZqkH7a0YmblZNGKtfz2yXl8at8hjOrf7K5j1mFtsaUUEU0mA0n7b2lcSdXAdcAxwALgGUn3RMSrOcWOB0annwOBG9K/7wNHRsQqSTXAY5L+HhFPFlEns7Jz9eRaIoLvHDU661DMylqL/k9J0jhJ/ylpJkkC2ZIDgNqImB0R64E7gZPyypwE3B6JJ4E+kgam3avSMjXpJ1oSq1m5mLdsNf89dT6nH7AzQ3fYLutwzMpas6+4lDQMOD391AHDgPERMbeZUQcD83O6F5C0gporMxhYlLa0ngVGAdelN1oUiu8s4CyAnXf2C9Ks/Fz5wEw6VYtzjxiVdShmZa+5/1N6AriPpKVyakTsB7xXREICKPRfgfmtnSbLRER9ROwNDAEOkLRHoZlExE0RMT4ixvfr16+IsMzazvS33uMvL7zJlz48nP69umYdjlnZa+703RKSmxoGAI1H/GJPoy0AhuZ0DwEWtrRMRCwHpgDHFTlfs7Jxxf3T6dG5E2d/dGTWoZi1C1tMShFxEvAh4Dngh5LmANtLOqCIaT8DjJY0QlJn4DTgnrwy9wBnpHfhHQSsiIhFkvpJ6gMgqRtwNPB6SypmlrUX5y9n4rS3+eqhI9i+e+eswzFrF5q9phQRK4BbgVslDQA+C1wpaWhEDN3CeHWSzgUmAtXArRExTdLZ6fAbSU4NngDUAmuAM9PRBwK/Sa8rVQF/ioh7W1tJsyxcNmk6229Xw1c/MiLrUMzajWaTUq6IeBu4Grg6vQGiufL3kSSe3H435nwP4JwC470E7NOS2MzKyVOzl/HozKV8/4Rd6dm1JutwzNqNLSYlSfmn2/J9YhvGYlYRIoLLJk2nf88unHHw8KzDMWtXmmspHUxyy/YdwFMUvlvOzHJMmbGEZ+a+y49O3oOuNdVZh2PWrjSXlHYieSLD6cDngL8Bd0TEtFIHZtYeRQSXT5rOkO278dnxTV5yNbMmNHf3XX1E/CMivgQcRHJDwhRJ57VJdGbtzD9eeYtX3lzJ+UePoXMnv9jZrKWKeaJDF+BjJK2l4SQ3Ovy5tGGZtT/1DcHl989gVP8efHKfwVmHY9YuNXejw2+APYC/Az+MiFfaJCqzdugvz79J7eJVXP/5famu8uVXs9ZorqX0RWA1MAb4trRxRxPJHd29ShibWbuxvq6BKyfPYPdBvThu952yDses3Wru1RU+KW5WhD9Nnc/8d9by6zP3oMqtJLNWc9Ix20rrNtRzzYMzGT9sew4f44cCm20NJyWzrfTbf87j7ZXvc9GxY8k5xW1mreCkZLYV3lu3geun1HLo6L4ctMuOWYdj1u45KZlthVsfm8u7azZw0YSxWYdiVhGclMxaafma9dz86GwmjBvAXkP7ZB2OWUVwUjJrpRsfns2q9XVc6FaS2TbjpGTWCotXruO2J+Zw0l6DGLtTz6zDMasYTkpmrXDdQ7VsqA/OP3pM1qGYVRQnJbMWWvDuGv7w9Bt8ZvwQhvftnnU4ZhXFScmsha56YCaSOO/I0VmHYlZxnJTMWmDWklXc9dwCvnDgMAb16ZZ1OGYVx0nJrAV+cf8MutZU860jRmYdillFclIyK9K0hSu496VFfOWQEfTt0SXrcMwqkpOSWZGumDSDXl078fWP7pJ1KGYVy0nJrAjPznuXya8v5huHjaR3t5qswzGrWE5KZkW4bOJ0+vbozJc/PDzrUMwqmpOSWTMer13KP2cv41uHj6J7l+Ze1mxmW6OkSUnScZKmS6qVdEmB4ZJ0dTr8JUn7pv2HSnpI0muSpkn6TinjNGtKRHDpxOkM7N2Vzx24c9bhmFW8kiUlSdXAdcDxwDjgdEnj8oodD4xOP2cBN6T964ALI2I34CDgnALjmpXcA68t5oX5y/nOUaPpWlOddThmFa+ULaUDgNqImB0R64E7gZPyypwE3B6JJ4E+kgZGxKKIeA4gIt4DXgMGlzBWs800NASXT5rO8B2345T9hmQdjlmHUMqkNBiYn9O9gM0TS7NlJA0H9gGeKjQTSWdJmipp6pIlS7Y2ZrON7n15Ea+/9R4XHDOGmmpffjVrC6Xc01SgX7SkjKQewF3A+RGxstBMIuKmiBgfEeP79evX6mDNctXVN/CL+2ew6049OXHPQVmHY9ZhlDIpLQCG5nQPARYWW0ZSDUlC+n1E/LmEcZpt5q7nFjBn6Wq+e8wYqqoK/XYys1IoZVJ6BhgtaYSkzsBpwD15Ze4BzkjvwjsIWBERiyQJuAV4LSKuKGGMZpt5v66eqyfXstfQPhwzbkDW4Zh1KCX7p4uIqJN0LjARqAZujYhpks5Oh98I3AecANQCa4Az09EPAb4IvCzphbTf9yPivlLFa9bojqfe4M3la/n5KXuS/D4ys7ZS0v8ETJPIfXn9bsz5HsA5BcZ7jMLXm8xKas36Oq59qJaDdtmBQ0btmHU4Zh2Obykyy3HbE3NZumo9Fx871q0ksww4KZmlVqzdwC8fns2Ru/Znv2E7ZB2OWYfkpGSWuvnR2axYu4ELJ4zJOhSzDstJyQxYtup9bn1sDh/70EB2H9Q763DMOiwnJTPghimzWLuhnguOcSvJLEtOStbhLVqxltufnMen9h3CqP49sg7HrENzUrIO75oHa4kIvnPU6KxDMevwnJSsQ5u3bDV/emY+px+wM0N32C7rcMw6PCcl69CufGAmnarFuUeMyjoUM8NJyTqwGW+/x19eeJMvHTyc/r26Zh2OmeGkZB3YFZNm0L1zJ84+bGTWoZhZyknJOqSXFiznH9Pe4muHjmD77p2zDsfMUk5K1iFdNmkG229Xw1c/MiLrUMwsh5OSdThPzV7GIzOW8M3DR9Kza03W4ZhZDicl61AigssmTad/zy6ccfDwrMMxszxOStahPDxjCc/MfZfzjhxF15rqrMMxszxOStZhNLaShmzfjc/uv3PW4ZhZAU5K1mH845W3eOXNlZx/9Bg6d/Kmb1aOvGdah1DfEFx+/wxG9uvOJ/cZnHU4ZtYEJyXrEP73hTepXbyK7x4zluoqv+bcrFw5KVnFW1/XwJUPzGT3Qb04fo+dsg7HzLbASckq3p+mzueNd9Zw0YSxVLmVZFbWnJSsoq3bUM81D85kv2Hbc/jYflmHY2bNcFKyiva7J+fx9sr3ufjYsUhuJZmVOyclq1ir3q/j+imzOHR0Xw7aZceswzGzIjgpWcW69bE5vLN6PRdOGJt1KGZWpJImJUnHSZouqVbSJQWGS9LV6fCXJO2bM+xWSYslvVLKGK0yLV+znl89MpsJ4waw99A+WYdjZkUqWVKSVA1cBxwPjANOlzQur9jxwOj0cxZwQ86w24DjShWfVbYbH57NqvV1biWZtTOlbCkdANRGxOyIWA/cCZyUV+Yk4PZIPAn0kTQQICIeAd4pYXxWoRa/t47bnpjDJ/YaxNidemYdjpm1QCmT0mBgfk73grRfS8tskaSzJE2VNHXJkiWtCtQqy/UPzWJDfXDB0WOyDsXMWqiUSanQ/bfRijJbFBE3RcT4iBjfr5//D6WjW/DuGn7/1Dw+M34Iw/t2zzocM2uhUialBcDQnO4hwMJWlDEr2tWTZyLEeUeOzjoUM2uFUialZ4DRkkZI6gycBtyTV+Ye4Iz0LryDgBURsaiEMVkFm7VkFXc99yZfOGgYg/p0yzocM2uFkiWliKgDzgUmAq8Bf4qIaZLOlnR2Wuw+YDZQC/wK+Fbj+JLuAP4JjJW0QNJXSxWrVYZf3D+DLp2q+NYRI7MOxcxaqVMpJx4R95Ekntx+N+Z8D+CcJsY9vZSxWWV5deFK7n1pEeccMZK+PbpkHY6ZtZKf6GAV4Yr7p9OrayfOOtStJLP2zEnJ2r3n3niXB15bzDcOG0nv7WqyDsfMtoKTkrV7l02cTt8enfnyh4dnHYqZbSUnJWvXHq9dyhOzlvGtw0fRvUtJL5GaWRtwUrJ2KyK4dOJ0BvbuyucO3DnrcMxsG3BSsnZr8muLeWH+cr591Gi61lRnHY6ZbQNOStYuNTQEl02azvAdt+PU/YZkHY6ZbSNOStYu/e3lRbz+1ntccMwYaqq9GZtVCu/N1u7U1Tfwi/tnMHZAT07cc1DW4ZjZNuSkZO3On597k9lLV3PhhDFUVRV60LyZtVdOStauvF9Xz1WTZ7LX0D4cM25A1uGY2TbmpGTtyh1PvcGby9dy0YQxSG4lmVUaJyVrN9asr+Pah2Zx0C478JFRfbMOx8xKwEnJ2o3fPDGPpave5+Jjx7qVZFahnJSsXVixdgM3PjyLI8b2Y79hO2QdjpmViJOStQu3PDqbFWs3cOGEsVmHYmYl5KRkZW/Zqve55bE5fOxDA9ljcO+swzGzEnJSsrJ3w5RZrN1QzwXHjMk6FDMrMSclK2tvrVjH7U/O45P7DGFU/x5Zh2NmJeakZGXtmgdnEhGcf/TorEMxszbgpGRl641la/jjM/M5bf+dGbrDdlmHY2ZtwEnJytaVD8ygukqce+SorEMxszbi90dnLCJoiJy/BBEkHz4YFkA0bNqvsTzBxu8byxeYRkMA6fQ3lm9oep6blW9unnww7y3OMx0G0JBXvjGG1e/Xc/cLb3LWobswoFfXjNaOmbU1JyXg49c8ytr19elBtYkEkdev8WC7ycF644G5yCQTmVa77PXZroazDxuZdRhm1oaclIBR/XqwoT6QQBJVApF8l0Ck/Rq/VwE5/aqkpssX6NdYno3zar48bBpbVVUz88wtnzONqiLLN/bLL09O/B/EULi8cuu7yXIqrnyvrjV07+JN1KwjKekeL+k44CqgGrg5In6WN1zp8BOANcCXI+K5Ysbdlq48bZ9STdrMzFqgZDc6SKoGrgOOB8YBp0sal1fseGB0+jkLuKEF45qZWYUp5d13BwC1ETE7ItYDdwIn5ZU5Cbg9Ek8CfSQNLHJcMzOrMKVMSoOB+TndC9J+xZQpZlwAJJ0laaqkqUuWLNnqoM3MLDulTEqFXniTf79ZU2WKGTfpGXFTRIyPiPH9+vVrYYhmZlZOSnmjwwJgaE73EGBhkWU6FzGumZlVmFK2lJ4BRksaIakzcBpwT16Ze4AzlDgIWBERi4oc18zMKkzJWkoRUSfpXGAiyW3dt0bENElnp8NvBO4juR28luSW8DO3NG6pYjUzs/KgqKDHCowfPz6mTp2adRhmZu2GpGcjYnzWcTSqqKQkaQkwr5Wj9wWWbsNwslQpdamUeoDrUo4qpR6wdXUZFhFlc5dYRSWlrSFpajn9WtgalVKXSqkHuC7lqFLqAZVVF7+6wszMyoaTkpmZlQ0npQ/clHUA21Cl1KVS6gGuSzmqlHpABdXF15TMzKxsuKVkZmZlw0nJzMzKRodKSpKOkzRdUq2kSwoMP0nSS5JeSJ88/pEs4ixGc3XJKbe/pHpJp7ZlfC1RxHo5XNKKdL28IOk/soizGMWsl7Q+L0iaJunhto6xGEWsk4tz1scr6Ta2QxaxNqeIuvSW9FdJL6br5Mws4ixGEXXZXtLd6XHsaUl7ZBHnVomIDvEheVzRLGAXkge+vgiMyyvTgw+us+0JvJ513K2tS065B0ke53Rq1nFvxXo5HLg361i3UV36AK8CO6fd/bOOu7XbV075E4EHs457K9bJ94Gfp9/7Ae8AnbOOvZV1uRT4v+n3XYHJWcfd0k9Haik1++LAiFgV6doEutPE6zLKQLEvQTwPuAtY3JbBtVAlvdCxmLp8DvhzRLwBEBHluG5auk5OB+5ok8harpi6BNBTkkh+mL4D1LVtmEUppi7jgMkAEfE6MFzSgLYNc+t0pKRU1IsDJX1S0uvA34CvtFFsLdVsXSQNBj4J3NiGcbVGsS90PDg9vfJ3Sbu3TWgtVkxdxgDbS5oi6VlJZ7RZdMVryUs2twOOI/nxU46Kqcu1wG4kr8d5GfhORDS0TXgtUkxdXgQ+BSDpAGAYyat/2o2OlJSKenFgRNwdEbsCJwM/KnVQrVRMXa4E/iUi6ksfzlYppi7PkTyfay/gGuAvpQ6qlYqpSydgP+BjwLHA/5E0ptSBtVDRL9kkOXX3eES8U8J4tkYxdTkWeAEYBOwNXCupV2nDapVi6vIzkh89L5CcKXme8mz1NamUL/krN8W8dHCjiHhE0khJfSOi3B7aWExdxgN3Jmck6AucIKkuIv7SJhEWr9m6RMTKnO/3Sbq+Ha+XBcDSiFgNrJb0CLAXMKNtQixKS/aV0yjfU3dQXF3OBH6WnrqvlTSH5HrM020TYtGK3VfOBEhPR85JP+1H1he12upDkoBnAyP44CLh7nllRvHBjQ77Am82dpfTp5i65JW/jfK90aGY9bJTzno5AHijva4XktNEk9Oy2wGvAHtkHXtrti+gN8n1l+5Zx7yV6+QG4Afp9wHpft8369hbWZc+pDdpAF8Hbs867pZ+OkxLKYp76eApJG/C3QCsBT4b6dotJ0XWpV0osi6nAt+UVEeyXk5rr+slIl6T9A/gJaABuDkiXsku6s21YPv6JDApklZfWSqyLj8CbpP0Mskpsn+J8muFF1uX3YDbJdWT3OX51cwCbiU/ZsjMzMpGR7rRwczMypyTkpmZlQ0nJTMzKxtOSmZmVjaclMzMrGw4KVm7Jenf0qc6Nz7Z/cBmyt/WmqelSxou6XOtGK/g/CTtmsb7vKSRrZju+enjfcwqjpOStUuSDgY+DuwbEXsCR7Ppc8G2peEkD1LdVk4G/jci9omIWa0Y/3ySf7wtmqQO8z+J1r45KVl7NZDkcT3vA0TE0ohYCCBpP0kPpw88nShpYP7ITZWRNErSA+nDX59LWzI/Aw5NWzcXSKqWdKmkZ9JW2jfScSXpWkmvSvob0L/AfE8gSSpfk/RQ2u8L6btvXpD0S0nVaf8blLzXa5qkH6b9vk3yjLaHcsZflTP9UyXdln6/TdIVabmfp4/N+kda50cl7ZqW+7SSdyK9mD72yCw7WT9Swh9/WvMhecXACyTPjLseOCztXwM8AfRLuz9L8p/vkD5uqZkyTwGfTL93JWmRHE7O+5yAs4B/T793AaaSPPrlU8D9JP9tPwhYToHHOwE/AC5Kv+8G/BWoSbuvB85Iv++Q/q0GpgB7pt1zyXkMDrAq5/upwG059b0XqE67JwOj0+8Hkr4DieTJ2IPT732yXrf+dOyPm/TWLkXEKkn7AYcCRwB/VPImzqnAHsD96cNoq4FFeaOPLVRGUk+Sg/Pd6TzWAaRlck0A9sy5XtQbGA18FLgjkiezL5T0YBFVOYrkqeHPpPPpxgfvv/qMpLNInnk2kORdOS8VMc1c/x0R9ZJ6AB8G/junPl3Sv4+TPGbnT8CfWzh9s23KScnarfTgPwWYkj637EvAs8C0iDh4C6OqUJkWvK5AwHkRMTFv/BNo+YshBfwmIv41b1ojgIuA/SPi3fSUXNcmppE7z/wyjc+lqwKWR8Tem40ccXZ6k8jHgBck7R0Ry1pYD7NtwteUrF2SNFbS6JxeewPzgOlAv/RGCCTVaPOXAhYsE8lj/xdIOjnt3yW9y+09oGfO+BNJHhBbk5YbI6k78AhwWnrNaSBJC645k4FTJfVPp7WDpGFAL5KEskLJm0OPzxknP563Je0mqYrkIambSes2R9Kn0/lI0l7p95ER8VRE/AewlE1fj2DWptxSsvaqB3CNpD4kLzGrBc6KiPXpabWrJfUm2cavBKY1jthMmS8Cv5T0n8AG4NMkp8zqJL1Icp3mKpI78p5Tci5sCckddXcDR5Jco5kBPNxcJSLiVUn/DkxKk8oG4JyIeFLS82lMs0lOsTW6Cfi7pEURcQRwCcm1o/kkr8Lo0cTsPg/ckM6vhuR12i8Cl6YJXiRJ8sXm4jYrFT8l3MzMyoZP35mZWdlwUjIzs7LhpGRmZmXDScnMzMqGk5KZmZUNJyUzMysbTkpmZlY2/j/oZUA0KLdvawAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pyplot.plot(selection_quota_list, map_list)\n",
    "pyplot.ylabel('MAP')\n",
    "pyplot.xlabel('Selected features')\n",
    "pyplot.title(\"Features selected from the highest absolute weight to the lowest\")\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}