Practice 09 - Hyperparameter optimization with Skopt.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recommender Systems 2021/22\n",
    "\n",
    "### Practice - Hyperparameter optimization with Skopt\n",
    "\n",
    "### Hyperparameter optimization is essential to achieve the best recommendation quality!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How does it work\n",
    "* Split the data in training, validation and test. Ensure that the distribution of the three sets is similar (e.g., the validation data you use should be split with the same approach used for the test data; if you know the test data contains cold items or users, ensure the validation data does too...)\n",
    "* Choose a recommender model, identify its hyperparameters and choose a value range and distribution for each.\n",
    "* Select a hyperparameter configuration and fit your model using the training data. Evaluate it using the validation data.\n",
    "* Repeat the previous step by exploring many possible hyperparameter configurations. Several exploration strategies are possible.\n",
    "* Select the hyperparameter configuration with the best recommendation quality on the validation data. Use that configuration to fit the model on the union of training and validation data. Evaluate the final model on the test data and report that result (or submit to the course challenge).\n",
    "\n",
    "Using directly or indirectly any information on the composition of the test data at any stage of the training or optimization process will result in *information leakage* and cause you to overestimate the quality of your model.\n",
    "\n",
    "\n",
    "\n",
    "## Hyperparameter optimization strategies\n",
    "\n",
    "### Grid-search\n",
    "For each hyperparameter define a list of possible values, then explore *all* possible combinations. For example, in a KNN using tversky similarity you could choose these values:\n",
    "* number of neighbors: [10, 50, 100, 150, 200, 250]\n",
    "* shrink term: [0, 10, 20, 50, 100]\n",
    "* alpha: [0.1, 0.3, 0.5, 0.7, 0.9, 1.1, 1.3, 1.5]\n",
    "* beta: [0.1, 0.3, 0.5, 0.7, 0.9, 1.1, 1.3, 1.5]\n",
    "\n",
    "You end up with *1920* possible hyperparameter configurations. Furthermore, you cannot easily fine-tune. Say that you found the optimal number of neighbors is 200, but in that area of the search space you were using a step of 50. What if there is an even better result at 215? You have to define a *new* hyperparameter range around the optimal one you found and use a smaller step, e.g., [180, 190, 200, 210, 220] and so on. \n",
    "\n",
    "Overall, grid-search is very sensitive to the range and distribution you choose, very rigid, and often unpractically slow due to how the number of cases grows combinatorially. It has been known for more than 20 years that it is generally a bad idea.\n",
    "\n",
    "### Random-search\n",
    "In a random search you define the range and distribution of possible values, and then you pick at random from it each hyperparameter configuration you explore.\n",
    "For example, in a KNN using tversky similarity you could choose these values:\n",
    "* number of neighbors: uniform random from 10 to 500\n",
    "* shrink term: uniform random from 0 to 500\n",
    "* alpha: uniform random from 0.1 to 1.5\n",
    "* beta: uniform random from 0.1 to 1.5\n",
    "\n",
    "It is very parallelizable and effective (definitely better than Grid-search).\n",
    "\n",
    "\n",
    "### Bayesian-search\n",
    "More advanced strategy that uses a gaussian process to try to model the interdependencies between hyperparameters based on how they affect the result. It contains a part of random search where the search space is explored, then a second part that uses the gaussian process to choose the next hyperparameter configuration. It combines exploration with exploitation. \n",
    "\n",
    "Bayesian-search is less parallelizable (the gaussian process is sequential) but has a good exploration-exploitation tradeoff. It is the strategy used in our research activity.\n",
    "\n",
    "\n",
    "### Other strategies\n",
    "There is no shortage of hyperparameter optimization strategies, for example the Tree-Structured Parzen Estimator, as well as techniques that use previous optimization runs to attempt transfer learning between models and datasets. All those are beyond the scope of the course.\n",
    "\n",
    "\n",
    "\n",
    "### In the course repository you will find a BayesianSearch object in the HyperparameterTuning folder. That is a simple wrapper of another library and its purpose is to provide a very simple way to tune some of the most common hyperparameters. \n",
    "\n",
    "\n",
    "\n",
    "\n",
    "## Hyperparameter sensitivity\n",
    "\n",
    "One of the most important problems you encounter when doing hyperparameter tuning is how to select the range and distribution of the hyperparameters. There is no universal rule and some of those decisions are based on experience. Generally the suggestions we can give you are:\n",
    "* Keep the search space on the larger side, better have more room to manouver than less.\n",
    "* Hyperparameters like the number of neighbors, the shrink term, the number of latent dimensions of matrix factorization models etc. work well with a uniform random distribution\n",
    "* Hyperparameters like the learning rate and l1 l2 regularizations work better with a log-uniform distribution as the training tends to be affected by the orders of magnitude rather than the absolute values. You need to be able to explore maybe from 10^-9 to 10^-3, but is less important to choose the specific value.\n",
    "\n",
    "### In the course repository you will find a script called run_hyperparameter_search that contains a list of commonly used hyperparameters with the corresponding range and distribution.\n",
    "\n",
    "### Hint: If you see that the optimization yelds a hyperparameter that has a value at either end of the range (min or max) this may indicate that in that scenarion you need to expand the search space.\n",
    "\n",
    "\n",
    "\n",
    "## Early-stopping\n",
    "\n",
    "A special hyperparameter is the number of epochs you should train a machine learning model for.\n",
    "It is possible to put this number as a hyperparameter, but that is not a super-effective strategy. A common strategy is *early-stopping*. It works as follows:\n",
    "* Select a maximum number of epochs, say 500\n",
    "* Train the model for a certain number of epochs, say 5\n",
    "* Evaluate the recommendation quality of the model on the validation data. Create a clone of the model.\n",
    "* Continue training and evaluate the model periodically. Every time you find a better recommendation quality update the model clone, which will then represent the \"best\" model you have.\n",
    "* If the recommendation quality does not improve for a certain number of consecutive validation steps, say 5, stop the training. If you reach the maximum number of epochs, stop the training.\n",
    "* Use the \"best\" model clone to generate the recommendations on the validation data.\n",
    "* When you train the model on the union of training and validation data, use the optimal number of epochs selected by the early-stopping at the previous step.\n",
    "\n",
    "Usually early-stopping allows to save a lot of computational time and fine-tune the optimal number of epochs, unless the validation step takes a very large amount of time. In those cases, it can be better to either use early-stopping on the algorithm loss function (which is however a different problem then recommendation and does not guarantee you that the best absolute loss will correspond to the model with the best recommendation quality) or just select the number of epochs as any other hyperparameter.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example on the course repository"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Movielens1M: Verifying data consistency...\n",
      "Movielens1M: Verifying data consistency... Passed!\n",
      "DataReader: current dataset is: <class 'Data_manager.Dataset.Dataset'>\n",
      "\tNumber of items: 3883\n",
      "\tNumber of users: 6040\n",
      "\tNumber of interactions in URM_all: 1000209\n",
      "\tValue range in URM_all: 1.00-5.00\n",
      "\tInteraction density: 4.26E-02\n",
      "\tInteractions per user:\n",
      "\t\t Min: 2.00E+01\n",
      "\t\t Avg: 1.66E+02\n",
      "\t\t Max: 2.31E+03\n",
      "\tInteractions per item:\n",
      "\t\t Min: 0.00E+00\n",
      "\t\t Avg: 2.58E+02\n",
      "\t\t Max: 3.43E+03\n",
      "\tGini Index: 0.53\n",
      "\n",
      "\tICM name: ICM_genres, Value range: 1.00 / 1.00, Num features: 18, feature occurrences: 6408, density 9.17E-02\n",
      "\tICM name: ICM_year, Value range: 1.92E+03 / 2.00E+03, Num features: 1, feature occurrences: 3883, density 1.00E+00\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from Data_manager.split_functions.split_train_validation_random_holdout import split_train_in_two_percentage_global_sample\n",
    "from Data_manager.Movielens.Movielens1MReader import Movielens1MReader\n",
    "\n",
    "data_reader = Movielens1MReader()\n",
    "data_loaded = data_reader.load_data()\n",
    "\n",
    "URM_all = data_loaded.get_URM_all()\n",
    "ICM_all = data_loaded.get_ICM_from_name(\"ICM_genres\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How do we perform hyperparameter optimization?\n",
    "* Split the data in three *disjoint* sets: training, validation and testing data\n",
    "* Define a set of hyperparameters with the range and distribution\n",
    "* Explore hyperparameter space and select those with the best recommendation quality on the *validation* data (including the number of epochs for ML algorithms)\n",
    "* Given the best hyperparameters, fit the model again using the union of training and validation data.\n",
    "* Evaluate this last model on the testing data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Split the data and create the evaluator objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Warning: 4 (0.07 %) of 6040 users have no sampled items\n",
      "Warning: 16 (0.26 %) of 6040 users have no sampled items\n",
      "EvaluatorHoldout: Ignoring 6024 ( 0.3%) Users that have less than 1 test interactions\n",
      "EvaluatorHoldout: Ignoring 6036 ( 0.1%) Users that have less than 1 test interactions\n"
     ]
    }
   ],
   "source": [
    "from Evaluation.Evaluator import EvaluatorHoldout\n",
    "\n",
    "URM_train_validation, URM_test = split_train_in_two_percentage_global_sample(URM_all, train_percentage = 0.8)\n",
    "URM_train, URM_validation = split_train_in_two_percentage_global_sample(URM_train_validation, train_percentage = 0.8)\n",
    "\n",
    "evaluator_validation = EvaluatorHoldout(URM_validation, cutoff_list=[10])\n",
    "evaluator_test = EvaluatorHoldout(URM_test, cutoff_list=[10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Define hyperparameter set for the desired model, in this case ItemKNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from skopt.space import Real, Integer, Categorical\n",
    "\n",
    "hyperparameters_range_dictionary = {\n",
    "    \"topK\": Integer(5, 1000),\n",
    "    \"shrink\": Integer(0, 1000),\n",
    "    \"similarity\": Categorical([\"cosine\"]),\n",
    "    \"normalize\": Categorical([True, False]),\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: Create SearchBayesianSkopt object, providing the desired recommender class and evaluator objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from Recommenders.KNN.ItemKNNCFRecommender import ItemKNNCFRecommender\n",
    "from HyperparameterTuning.SearchBayesianSkopt import SearchBayesianSkopt\n",
    "\n",
    "recommender_class = ItemKNNCFRecommender\n",
    "\n",
    "hyperparameterSearch = SearchBayesianSkopt(recommender_class,\n",
    "                                         evaluator_validation=evaluator_validation,\n",
    "                                         evaluator_test=evaluator_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: Provide the data needed to create an instance of the model, one trained only on URM_train, the other on URM_train_validation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from HyperparameterTuning.SearchAbstractClass import SearchInputRecommenderArgs\n",
    "  \n",
    "recommender_input_args = SearchInputRecommenderArgs(\n",
    "    CONSTRUCTOR_POSITIONAL_ARGS = [URM_train],     # For a CBF model simply put [URM_train, ICM_train]\n",
    "    CONSTRUCTOR_KEYWORD_ARGS = {},\n",
    "    FIT_POSITIONAL_ARGS = [],\n",
    "    FIT_KEYWORD_ARGS = {},\n",
    "    EARLYSTOPPING_KEYWORD_ARGS = {},\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "recommender_input_args_last_test = SearchInputRecommenderArgs(\n",
    "    CONSTRUCTOR_POSITIONAL_ARGS = [URM_train_validation],     # For a CBF model simply put [URM_train_validation, ICM_train]\n",
    "    CONSTRUCTOR_KEYWORD_ARGS = {},\n",
    "    FIT_POSITIONAL_ARGS = [],\n",
    "    FIT_KEYWORD_ARGS = {},\n",
    "    EARLYSTOPPING_KEYWORD_ARGS = {},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5: Create a result folder and select the number of cases (50 with 30% random is a good number)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "output_folder_path = \"result_experiments/\"\n",
    "\n",
    "# If directory does not exist, create\n",
    "if not os.path.exists(output_folder_path):\n",
    "    os.makedirs(output_folder_path)\n",
    "    \n",
    "n_cases = 10  # using 10 as an example\n",
    "n_random_starts = int(n_cases*0.3)\n",
    "metric_to_optimize = \"MAP\"   \n",
    "cutoff_to_optimize = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5: Run!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration No: 1 started. Evaluating function at random point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 150, 'shrink': 314, 'similarity': 'cosine', 'normalize': True}\n",
      "ItemKNNCFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 5210.63 column/sec. Elapsed time 0.75 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.41 sec. Users per second: 1114\n",
      "SearchBayesianSkopt: New best config found. Config 0: {'topK': 150, 'shrink': 314, 'similarity': 'cosine', 'normalize': True} - results: PRECISION: 0.2026228, PRECISION_RECALL_MIN_DEN: 0.2288963, RECALL: 0.1204475, MAP: 0.1059938, MAP_MIN_DEN: 0.1170157, MRR: 0.4239689, NDCG: 0.1923506, F1: 0.1510842, HIT_RATE: 0.7981408, ARHR_ALL_HITS: 0.6578372, NOVELTY: 0.0241524, AVERAGE_POPULARITY: 0.4900907, DIVERSITY_MEAN_INTER_LIST: 0.9322565, DIVERSITY_HERFINDAHL: 0.9932102, COVERAGE_ITEM: 0.2722122, COVERAGE_ITEM_CORRECT: 0.1738347, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.7960265, DIVERSITY_GINI: 0.0539589, SHANNON_ENTROPY: 8.0347933, RATIO_DIVERSITY_HERFINDAHL: 0.9940216, RATIO_DIVERSITY_GINI: 0.1543243, RATIO_SHANNON_ENTROPY: 0.7430767, RATIO_AVERAGE_POPULARITY: 1.7743035, RATIO_NOVELTY: 0.0818374, \n",
      "\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 6.22 sec. Users per second: 971\n",
      "SearchBayesianSkopt: Config evaluated with evaluator_test. Config: {'topK': 150, 'shrink': 314, 'similarity': 'cosine', 'normalize': True} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.2523857, PRECISION_RECALL_MIN_DEN: 0.2714357, RECALL: 0.1208663, MAP: 0.1453211, MAP_MIN_DEN: 0.1541244, MRR: 0.4957808, NDCG: 0.2289295, F1: 0.1634548, HIT_RATE: 0.8494036, ARHR_ALL_HITS: 0.8273696, NOVELTY: 0.0241518, AVERAGE_POPULARITY: 0.4901756, DIVERSITY_MEAN_INTER_LIST: 0.9321996, DIVERSITY_HERFINDAHL: 0.9932045, COVERAGE_ITEM: 0.2722122, COVERAGE_ITEM_CORRECT: 0.1833634, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.8488411, DIVERSITY_GINI: 0.0539452, SHANNON_ENTROPY: 8.0341532, RATIO_DIVERSITY_HERFINDAHL: 0.9940159, RATIO_DIVERSITY_GINI: 0.1542851, RATIO_SHANNON_ENTROPY: 0.7430175, RATIO_AVERAGE_POPULARITY: 1.7746107, RATIO_NOVELTY: 0.0818356, \n",
      "\n",
      "\n",
      "Iteration No: 1 ended. Evaluation done at random point.\n",
      "Time taken: 12.7578\n",
      "Function value obtained: -0.1060\n",
      "Current minimum: -0.1060\n",
      "Iteration No: 2 started. Evaluating function at random point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 383, 'shrink': 664, 'similarity': 'cosine', 'normalize': False}\n",
      "ItemKNNCFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 2696.60 column/sec. Elapsed time 1.44 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 7.99 sec. Users per second: 754\n",
      "SearchBayesianSkopt: Config 1 is suboptimal. Config: {'topK': 383, 'shrink': 664, 'similarity': 'cosine', 'normalize': False} - results: PRECISION: 0.1454349, PRECISION_RECALL_MIN_DEN: 0.1626623, RECALL: 0.0800106, MAP: 0.0754515, MAP_MIN_DEN: 0.0829858, MRR: 0.3540884, NDCG: 0.1474995, F1: 0.1032297, HIT_RATE: 0.6782869, ARHR_ALL_HITS: 0.5083588, NOVELTY: 0.0221713, AVERAGE_POPULARITY: 0.7508613, DIVERSITY_MEAN_INTER_LIST: 0.5417790, DIVERSITY_HERFINDAHL: 0.9541689, COVERAGE_ITEM: 0.0267834, COVERAGE_ITEM_CORRECT: 0.0172547, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.6764901, DIVERSITY_GINI: 0.0053783, SHANNON_ENTROPY: 4.7003642, RATIO_DIVERSITY_HERFINDAHL: 0.9549484, RATIO_DIVERSITY_GINI: 0.0153820, RATIO_SHANNON_ENTROPY: 0.4347008, RATIO_AVERAGE_POPULARITY: 2.7183862, RATIO_NOVELTY: 0.0751250, \n",
      "\n",
      "Iteration No: 2 ended. Evaluation done at random point.\n",
      "Time taken: 9.6579\n",
      "Function value obtained: -0.0755\n",
      "Current minimum: -0.1060\n",
      "Iteration No: 3 started. Evaluating function at random point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 159, 'shrink': 547, 'similarity': 'cosine', 'normalize': True}\n",
      "ItemKNNCFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 2351.03 column/sec. Elapsed time 1.65 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 6.32 sec. Users per second: 954\n",
      "SearchBayesianSkopt: New best config found. Config 2: {'topK': 159, 'shrink': 547, 'similarity': 'cosine', 'normalize': True} - results: PRECISION: 0.2036023, PRECISION_RECALL_MIN_DEN: 0.2295790, RECALL: 0.1205202, MAP: 0.1070413, MAP_MIN_DEN: 0.1183787, MRR: 0.4255832, NDCG: 0.1944625, F1: 0.1514131, HIT_RATE: 0.8002988, ARHR_ALL_HITS: 0.6623607, NOVELTY: 0.0240605, AVERAGE_POPULARITY: 0.4990041, DIVERSITY_MEAN_INTER_LIST: 0.9283310, DIVERSITY_HERFINDAHL: 0.9928177, COVERAGE_ITEM: 0.2557301, COVERAGE_ITEM_CORRECT: 0.1640484, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.7981788, DIVERSITY_GINI: 0.0503057, SHANNON_ENTROPY: 7.9383512, RATIO_DIVERSITY_HERFINDAHL: 0.9936288, RATIO_DIVERSITY_GINI: 0.1438759, RATIO_SHANNON_ENTROPY: 0.7341576, RATIO_AVERAGE_POPULARITY: 1.8065733, RATIO_NOVELTY: 0.0815261, \n",
      "\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 6.37 sec. Users per second: 948\n",
      "SearchBayesianSkopt: Config evaluated with evaluator_test. Config: {'topK': 159, 'shrink': 547, 'similarity': 'cosine', 'normalize': True} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.2515242, PRECISION_RECALL_MIN_DEN: 0.2704378, RECALL: 0.1205935, MAP: 0.1447856, MAP_MIN_DEN: 0.1534835, MRR: 0.4959557, NDCG: 0.2289127, F1: 0.1630247, HIT_RATE: 0.8477469, ARHR_ALL_HITS: 0.8266837, NOVELTY: 0.0240602, AVERAGE_POPULARITY: 0.4990650, DIVERSITY_MEAN_INTER_LIST: 0.9282903, DIVERSITY_HERFINDAHL: 0.9928137, COVERAGE_ITEM: 0.2559876, COVERAGE_ITEM_CORRECT: 0.1735771, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.8471854, DIVERSITY_GINI: 0.0503039, SHANNON_ENTROPY: 7.9380232, RATIO_DIVERSITY_HERFINDAHL: 0.9936247, RATIO_DIVERSITY_GINI: 0.1438707, RATIO_SHANNON_ENTROPY: 0.7341272, RATIO_AVERAGE_POPULARITY: 1.8067937, RATIO_NOVELTY: 0.0815253, \n",
      "\n",
      "\n",
      "Iteration No: 3 ended. Evaluation done at random point.\n",
      "Time taken: 15.3053\n",
      "Function value obtained: -0.1070\n",
      "Current minimum: -0.1070\n",
      "Iteration No: 4 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 300, 'shrink': 1000, 'similarity': 'cosine', 'normalize': True}\n",
      "ItemKNNCFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 2779.89 column/sec. Elapsed time 1.40 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 7.33 sec. Users per second: 822\n",
      "SearchBayesianSkopt: New best config found. Config 3: {'topK': 300, 'shrink': 1000, 'similarity': 'cosine', 'normalize': True} - results: PRECISION: 0.2021248, PRECISION_RECALL_MIN_DEN: 0.2284658, RECALL: 0.1200458, MAP: 0.1075338, MAP_MIN_DEN: 0.1193987, MRR: 0.4290071, NDCG: 0.1978795, F1: 0.1506298, HIT_RATE: 0.7946547, ARHR_ALL_HITS: 0.6670684, NOVELTY: 0.0236151, AVERAGE_POPULARITY: 0.5470590, DIVERSITY_MEAN_INTER_LIST: 0.8979181, DIVERSITY_HERFINDAHL: 0.9897769, COVERAGE_ITEM: 0.1895442, COVERAGE_ITEM_CORRECT: 0.1269637, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.7925497, DIVERSITY_GINI: 0.0339016, SHANNON_ENTROPY: 7.3782931, RATIO_DIVERSITY_HERFINDAHL: 0.9905855, RATIO_DIVERSITY_GINI: 0.0969599, RATIO_SHANNON_ENTROPY: 0.6823621, RATIO_AVERAGE_POPULARITY: 1.9805489, RATIO_NOVELTY: 0.0800169, \n",
      "\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 7.31 sec. Users per second: 826\n",
      "SearchBayesianSkopt: Config evaluated with evaluator_test. Config: {'topK': 300, 'shrink': 1000, 'similarity': 'cosine', 'normalize': True} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.2507124, PRECISION_RECALL_MIN_DEN: 0.2704143, RECALL: 0.1208514, MAP: 0.1447280, MAP_MIN_DEN: 0.1536129, MRR: 0.4952294, NDCG: 0.2296621, F1: 0.1630888, HIT_RATE: 0.8510603, ARHR_ALL_HITS: 0.8261940, NOVELTY: 0.0236151, AVERAGE_POPULARITY: 0.5470495, DIVERSITY_MEAN_INTER_LIST: 0.8979027, DIVERSITY_HERFINDAHL: 0.9897754, COVERAGE_ITEM: 0.1892866, COVERAGE_ITEM_CORRECT: 0.1359773, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.8504967, DIVERSITY_GINI: 0.0338857, SHANNON_ENTROPY: 7.3778664, RATIO_DIVERSITY_HERFINDAHL: 0.9905840, RATIO_DIVERSITY_GINI: 0.0969141, RATIO_SHANNON_ENTROPY: 0.6823226, RATIO_AVERAGE_POPULARITY: 1.9805145, RATIO_NOVELTY: 0.0800171, \n",
      "\n",
      "\n",
      "Iteration No: 4 ended. Search finished for the next optimal point.\n",
      "Time taken: 16.9670\n",
      "Function value obtained: -0.1075\n",
      "Current minimum: -0.1075\n",
      "Iteration No: 5 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 783, 'shrink': 0, 'similarity': 'cosine', 'normalize': False}\n",
      "ItemKNNCFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Similarity column 3883 (100.0%), 2689.85 column/sec. Elapsed time 1.44 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 9.61 sec. Users per second: 627\n",
      "SearchBayesianSkopt: Config 4 is suboptimal. Config: {'topK': 783, 'shrink': 0, 'similarity': 'cosine', 'normalize': False} - results: PRECISION: 0.1420983, PRECISION_RECALL_MIN_DEN: 0.1581309, RECALL: 0.0767443, MAP: 0.0734144, MAP_MIN_DEN: 0.0803848, MRR: 0.3458051, NDCG: 0.1431956, F1: 0.0996628, HIT_RATE: 0.6636786, ARHR_ALL_HITS: 0.4954128, NOVELTY: 0.0221433, AVERAGE_POPULARITY: 0.7557058, DIVERSITY_MEAN_INTER_LIST: 0.5132387, DIVERSITY_HERFINDAHL: 0.9513154, COVERAGE_ITEM: 0.0159670, COVERAGE_ITEM_CORRECT: 0.0121040, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.6619205, DIVERSITY_GINI: 0.0050268, SHANNON_ENTROPY: 4.5819229, RATIO_DIVERSITY_HERFINDAHL: 0.9520925, RATIO_DIVERSITY_GINI: 0.0143769, RATIO_SHANNON_ENTROPY: 0.4237471, RATIO_AVERAGE_POPULARITY: 2.7359250, RATIO_NOVELTY: 0.0750299, \n",
      "\n",
      "Iteration No: 5 ended. Search finished for the next optimal point.\n",
      "Time taken: 11.9365\n",
      "Function value obtained: -0.0734\n",
      "Current minimum: -0.1075\n",
      "Iteration No: 6 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 998, 'shrink': 2, 'similarity': 'cosine', 'normalize': True}\n",
      "ItemKNNCFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 2406.42 column/sec. Elapsed time 1.61 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 9.06 sec. Users per second: 665\n",
      "SearchBayesianSkopt: Config 5 is suboptimal. Config: {'topK': 998, 'shrink': 2, 'similarity': 'cosine', 'normalize': True} - results: PRECISION: 0.1936753, PRECISION_RECALL_MIN_DEN: 0.2178269, RECALL: 0.1128921, MAP: 0.1027232, MAP_MIN_DEN: 0.1137113, MRR: 0.4175427, NDCG: 0.1869757, F1: 0.1426402, HIT_RATE: 0.7790505, ARHR_ALL_HITS: 0.6442132, NOVELTY: 0.0233580, AVERAGE_POPULARITY: 0.5784849, DIVERSITY_MEAN_INTER_LIST: 0.8537787, DIVERSITY_HERFINDAHL: 0.9853637, COVERAGE_ITEM: 0.1712593, COVERAGE_ITEM_CORRECT: 0.1127994, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.7769868, DIVERSITY_GINI: 0.0246646, SHANNON_ENTROPY: 6.8894945, RATIO_DIVERSITY_HERFINDAHL: 0.9861687, RATIO_DIVERSITY_GINI: 0.0705417, RATIO_SHANNON_ENTROPY: 0.6371568, RATIO_AVERAGE_POPULARITY: 2.0943220, RATIO_NOVELTY: 0.0791457, \n",
      "\n",
      "Iteration No: 6 ended. Search finished for the next optimal point.\n",
      "Time taken: 11.7376\n",
      "Function value obtained: -0.1027\n",
      "Current minimum: -0.1075\n",
      "Iteration No: 7 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 5, 'shrink': 1000, 'similarity': 'cosine', 'normalize': True}\n",
      "ItemKNNCFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 3196.02 column/sec. Elapsed time 1.21 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.66 sec. Users per second: 1294\n",
      "SearchBayesianSkopt: Config 6 is suboptimal. Config: {'topK': 5, 'shrink': 1000, 'similarity': 'cosine', 'normalize': True} - results: PRECISION: 0.1576693, PRECISION_RECALL_MIN_DEN: 0.1791103, RECALL: 0.0932141, MAP: 0.0768604, MAP_MIN_DEN: 0.0849079, MRR: 0.3548091, NDCG: 0.1457877, F1: 0.1171620, HIT_RATE: 0.7284197, ARHR_ALL_HITS: 0.5152010, NOVELTY: 0.0247104, AVERAGE_POPULARITY: 0.4321230, DIVERSITY_MEAN_INTER_LIST: 0.9558861, DIVERSITY_HERFINDAHL: 0.9955727, COVERAGE_ITEM: 0.3427762, COVERAGE_ITEM_CORRECT: 0.2011331, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.7264901, DIVERSITY_GINI: 0.0802931, SHANNON_ENTROPY: 8.6270450, RATIO_DIVERSITY_HERFINDAHL: 0.9963861, RATIO_DIVERSITY_GINI: 0.2296408, RATIO_SHANNON_ENTROPY: 0.7978496, RATIO_AVERAGE_POPULARITY: 1.5644395, RATIO_NOVELTY: 0.0837282, \n",
      "\n",
      "Iteration No: 7 ended. Search finished for the next optimal point.\n",
      "Time taken: 6.8914\n",
      "Function value obtained: -0.0769\n",
      "Current minimum: -0.1075\n",
      "Iteration No: 8 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 693, 'shrink': 122, 'similarity': 'cosine', 'normalize': True}\n",
      "ItemKNNCFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 2690.28 column/sec. Elapsed time 1.44 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 8.50 sec. Users per second: 708\n",
      "SearchBayesianSkopt: Config 7 is suboptimal. Config: {'topK': 693, 'shrink': 122, 'similarity': 'cosine', 'normalize': True} - results: PRECISION: 0.1978254, PRECISION_RECALL_MIN_DEN: 0.2228480, RECALL: 0.1159950, MAP: 0.1051192, MAP_MIN_DEN: 0.1162784, MRR: 0.4233456, NDCG: 0.1916339, F1: 0.1462413, HIT_RATE: 0.7863546, ARHR_ALL_HITS: 0.6554997, NOVELTY: 0.0234803, AVERAGE_POPULARITY: 0.5639580, DIVERSITY_MEAN_INTER_LIST: 0.8741722, DIVERSITY_HERFINDAHL: 0.9874027, COVERAGE_ITEM: 0.1877414, COVERAGE_ITEM_CORRECT: 0.1241308, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.7842715, DIVERSITY_GINI: 0.0288496, SHANNON_ENTROPY: 7.1190464, RATIO_DIVERSITY_HERFINDAHL: 0.9882094, RATIO_DIVERSITY_GINI: 0.0825109, RATIO_SHANNON_ENTROPY: 0.6583863, RATIO_AVERAGE_POPULARITY: 2.0417293, RATIO_NOVELTY: 0.0795602, \n",
      "\n",
      "Iteration No: 8 ended. Search finished for the next optimal point.\n",
      "Time taken: 10.6864\n",
      "Function value obtained: -0.1051\n",
      "Current minimum: -0.1075\n",
      "Iteration No: 9 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 513, 'shrink': 46, 'similarity': 'cosine', 'normalize': True}\n",
      "ItemKNNCFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 2239.30 column/sec. Elapsed time 1.73 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 7.48 sec. Users per second: 805\n",
      "SearchBayesianSkopt: Config 8 is suboptimal. Config: {'topK': 513, 'shrink': 46, 'similarity': 'cosine', 'normalize': True} - results: PRECISION: 0.2008300, PRECISION_RECALL_MIN_DEN: 0.2281633, RECALL: 0.1207647, MAP: 0.1059469, MAP_MIN_DEN: 0.1177003, MRR: 0.4245576, NDCG: 0.1942422, F1: 0.1508307, HIT_RATE: 0.7973108, ARHR_ALL_HITS: 0.6600295, NOVELTY: 0.0236755, AVERAGE_POPULARITY: 0.5418831, DIVERSITY_MEAN_INTER_LIST: 0.8951789, DIVERSITY_HERFINDAHL: 0.9895030, COVERAGE_ITEM: 0.2129797, COVERAGE_ITEM_CORRECT: 0.1395828, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.7951987, DIVERSITY_GINI: 0.0351638, SHANNON_ENTROPY: 7.4010801, RATIO_DIVERSITY_HERFINDAHL: 0.9903114, RATIO_DIVERSITY_GINI: 0.1005697, RATIO_SHANNON_ENTROPY: 0.6844694, RATIO_AVERAGE_POPULARITY: 1.9618104, RATIO_NOVELTY: 0.0802215, \n",
      "\n",
      "Iteration No: 9 ended. Search finished for the next optimal point.\n",
      "Time taken: 9.9217\n",
      "Function value obtained: -0.1059\n",
      "Current minimum: -0.1075\n",
      "Iteration No: 10 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 5, 'shrink': 182, 'similarity': 'cosine', 'normalize': False}\n",
      "ItemKNNCFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 2422.47 column/sec. Elapsed time 1.60 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.93 sec. Users per second: 1221\n",
      "SearchBayesianSkopt: Config 9 is suboptimal. Config: {'topK': 5, 'shrink': 182, 'similarity': 'cosine', 'normalize': False} - results: PRECISION: 0.1389276, PRECISION_RECALL_MIN_DEN: 0.1543000, RECALL: 0.0747277, MAP: 0.0695650, MAP_MIN_DEN: 0.0761220, MRR: 0.3256610, NDCG: 0.1342073, F1: 0.0971821, HIT_RATE: 0.6508964, ARHR_ALL_HITS: 0.4692094, NOVELTY: 0.0232710, AVERAGE_POPULARITY: 0.5967606, DIVERSITY_MEAN_INTER_LIST: 0.8555182, DIVERSITY_HERFINDAHL: 0.9855376, COVERAGE_ITEM: 0.2052537, COVERAGE_ITEM_CORRECT: 0.0695339, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.6491722, DIVERSITY_GINI: 0.0240534, SHANNON_ENTROPY: 6.8259713, RATIO_DIVERSITY_HERFINDAHL: 0.9863428, RATIO_DIVERSITY_GINI: 0.0687935, RATIO_SHANNON_ENTROPY: 0.6312820, RATIO_AVERAGE_POPULARITY: 2.1604866, RATIO_NOVELTY: 0.0788511, \n",
      "\n",
      "Iteration No: 10 ended. Search finished for the next optimal point.\n",
      "Time taken: 7.3489\n",
      "Function value obtained: -0.0696\n",
      "Current minimum: -0.1075\n",
      "SearchBayesianSkopt: Search complete. Best config is 3: {'topK': 300, 'shrink': 1000, 'similarity': 'cosine', 'normalize': True}\n",
      "\n",
      "ItemKNNCFRecommender: URM Detected 212 ( 5.5%) items with no interactions.\n",
      "SearchBayesianSkopt: Evaluation with constructor data for final test. Using best config: {'topK': 300, 'shrink': 1000, 'similarity': 'cosine', 'normalize': True}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Similarity column 3883 (100.0%), 1733.86 column/sec. Elapsed time 2.24 sec\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 8.20 sec. Users per second: 736\n",
      "SearchBayesianSkopt: Best config evaluated with evaluator_test with constructor data for final test. Config: {'topK': 300, 'shrink': 1000, 'similarity': 'cosine', 'normalize': True} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.3289430, PRECISION_RECALL_MIN_DEN: 0.3507929, RECALL: 0.1443527, MAP: 0.2372513, MAP_MIN_DEN: 0.2481392, MRR: 0.6085585, NDCG: 0.3006104, F1: 0.2006518, HIT_RATE: 0.8727634, ARHR_ALL_HITS: 1.1395252, NOVELTY: 0.0237508, AVERAGE_POPULARITY: 0.5261314, DIVERSITY_MEAN_INTER_LIST: 0.9116772, DIVERSITY_HERFINDAHL: 0.9911526, COVERAGE_ITEM: 0.1964976, COVERAGE_ITEM_CORRECT: 0.1529745, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.8721854, DIVERSITY_GINI: 0.0382841, SHANNON_ENTROPY: 7.5692657, RATIO_DIVERSITY_HERFINDAHL: 0.9919632, RATIO_DIVERSITY_GINI: 0.1094867, RATIO_SHANNON_ENTROPY: 0.7000265, RATIO_AVERAGE_POPULARITY: 1.9152924, RATIO_NOVELTY: 0.0643818, \n",
      "\n",
      "\n",
      "SearchBayesianSkopt: Saving model in result_experiments/ItemKNNCFRecommender\n",
      "\n",
      "ItemKNNCFRecommender: Saving model in file 'result_experiments/ItemKNNCFRecommender_best_model_last'\n",
      "ItemKNNCFRecommender: Saving complete\n"
     ]
    }
   ],
   "source": [
    "hyperparameterSearch.search(recommender_input_args,\n",
    "                       recommender_input_args_last_test = recommender_input_args_last_test,\n",
    "                       hyperparameter_search_space = hyperparameters_range_dictionary,\n",
    "                       n_cases = n_cases,\n",
    "                       n_random_starts = n_random_starts,\n",
    "                       save_model = \"last\",\n",
    "                       output_folder_path = output_folder_path, # Where to save the results\n",
    "                       output_file_name_root = recommender_class.RECOMMENDER_NAME, # How to call the files\n",
    "                       metric_to_optimize = metric_to_optimize,\n",
    "                       cutoff_to_optimize = cutoff_to_optimize,\n",
    "                      )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The metadata.zip file contains details on the search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['result_on_test_best', 'time_df', 'result_on_last', 'time_on_last_df', 'time_on_validation_avg', 'cutoff_to_optimize', 'metric_to_optimize', 'time_on_validation_total', 'algorithm_name_recommender', 'hyperparameters_df', 'result_on_validation_best', 'time_on_train_total', 'time_on_train_avg', 'result_on_test_df', 'algorithm_name_search', 'time_on_test_avg', 'hyperparameters_best', 'hyperparameters_best_index', 'result_on_validation_df', 'exception_list', 'time_on_test_total'])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from Recommenders.DataIO import DataIO\n",
    "\n",
    "data_loader = DataIO(folder_path = output_folder_path)\n",
    "search_metadata = data_loader.load_data(recommender_class.RECOMMENDER_NAME + \"_metadata.zip\")\n",
    "\n",
    "search_metadata.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>topK</th>\n",
       "      <th>shrink</th>\n",
       "      <th>similarity</th>\n",
       "      <th>normalize</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>150</td>\n",
       "      <td>314</td>\n",
       "      <td>cosine</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>383</td>\n",
       "      <td>664</td>\n",
       "      <td>cosine</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>159</td>\n",
       "      <td>547</td>\n",
       "      <td>cosine</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>300</td>\n",
       "      <td>1000</td>\n",
       "      <td>cosine</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>783</td>\n",
       "      <td>0</td>\n",
       "      <td>cosine</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>998</td>\n",
       "      <td>2</td>\n",
       "      <td>cosine</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>5</td>\n",
       "      <td>1000</td>\n",
       "      <td>cosine</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>693</td>\n",
       "      <td>122</td>\n",
       "      <td>cosine</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>513</td>\n",
       "      <td>46</td>\n",
       "      <td>cosine</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>5</td>\n",
       "      <td>182</td>\n",
       "      <td>cosine</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  topK shrink similarity normalize\n",
       "0  150    314     cosine      True\n",
       "1  383    664     cosine     False\n",
       "2  159    547     cosine      True\n",
       "3  300   1000     cosine      True\n",
       "4  783      0     cosine     False\n",
       "5  998      2     cosine      True\n",
       "6    5   1000     cosine      True\n",
       "7  693    122     cosine      True\n",
       "8  513     46     cosine      True\n",
       "9    5    182     cosine     False"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hyperparameters_df = search_metadata[\"hyperparameters_df\"]\n",
    "hyperparameters_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>PRECISION</th>\n",
       "      <th>PRECISION_RECALL_MIN_DEN</th>\n",
       "      <th>RECALL</th>\n",
       "      <th>MAP</th>\n",
       "      <th>MAP_MIN_DEN</th>\n",
       "      <th>MRR</th>\n",
       "      <th>NDCG</th>\n",
       "      <th>F1</th>\n",
       "      <th>HIT_RATE</th>\n",
       "      <th>ARHR_ALL_HITS</th>\n",
       "      <th>...</th>\n",
       "      <th>COVERAGE_ITEM_CORRECT</th>\n",
       "      <th>COVERAGE_USER</th>\n",
       "      <th>COVERAGE_USER_CORRECT</th>\n",
       "      <th>DIVERSITY_GINI</th>\n",
       "      <th>SHANNON_ENTROPY</th>\n",
       "      <th>RATIO_DIVERSITY_HERFINDAHL</th>\n",
       "      <th>RATIO_DIVERSITY_GINI</th>\n",
       "      <th>RATIO_SHANNON_ENTROPY</th>\n",
       "      <th>RATIO_AVERAGE_POPULARITY</th>\n",
       "      <th>RATIO_NOVELTY</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>cutoff</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <th>10</th>\n",
       "      <td>0.202623</td>\n",
       "      <td>0.228896</td>\n",
       "      <td>0.120448</td>\n",
       "      <td>0.105994</td>\n",
       "      <td>0.117016</td>\n",
       "      <td>0.423969</td>\n",
       "      <td>0.192351</td>\n",
       "      <td>0.151084</td>\n",
       "      <td>0.798141</td>\n",
       "      <td>0.657837</td>\n",
       "      <td>...</td>\n",
       "      <td>0.173835</td>\n",
       "      <td>0.997351</td>\n",
       "      <td>0.796026</td>\n",
       "      <td>0.053959</td>\n",
       "      <td>8.034793</td>\n",
       "      <td>0.994022</td>\n",
       "      <td>0.154324</td>\n",
       "      <td>0.743077</td>\n",
       "      <td>1.774303</td>\n",
       "      <td>0.081837</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <th>10</th>\n",
       "      <td>0.145435</td>\n",
       "      <td>0.162662</td>\n",
       "      <td>0.080011</td>\n",
       "      <td>0.075451</td>\n",
       "      <td>0.082986</td>\n",
       "      <td>0.354088</td>\n",
       "      <td>0.147499</td>\n",
       "      <td>0.10323</td>\n",
       "      <td>0.678287</td>\n",
       "      <td>0.508359</td>\n",
       "      <td>...</td>\n",
       "      <td>0.017255</td>\n",
       "      <td>0.997351</td>\n",
       "      <td>0.67649</td>\n",
       "      <td>0.005378</td>\n",
       "      <td>4.700364</td>\n",
       "      <td>0.954948</td>\n",
       "      <td>0.015382</td>\n",
       "      <td>0.434701</td>\n",
       "      <td>2.718386</td>\n",
       "      <td>0.075125</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <th>10</th>\n",
       "      <td>0.203602</td>\n",
       "      <td>0.229579</td>\n",
       "      <td>0.12052</td>\n",
       "      <td>0.107041</td>\n",
       "      <td>0.118379</td>\n",
       "      <td>0.425583</td>\n",
       "      <td>0.194463</td>\n",
       "      <td>0.151413</td>\n",
       "      <td>0.800299</td>\n",
       "      <td>0.662361</td>\n",
       "      <td>...</td>\n",
       "      <td>0.164048</td>\n",
       "      <td>0.997351</td>\n",
       "      <td>0.798179</td>\n",
       "      <td>0.050306</td>\n",
       "      <td>7.938351</td>\n",
       "      <td>0.993629</td>\n",
       "      <td>0.143876</td>\n",
       "      <td>0.734158</td>\n",
       "      <td>1.806573</td>\n",
       "      <td>0.081526</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <th>10</th>\n",
       "      <td>0.202125</td>\n",
       "      <td>0.228466</td>\n",
       "      <td>0.120046</td>\n",
       "      <td>0.107534</td>\n",
       "      <td>0.119399</td>\n",
       "      <td>0.429007</td>\n",
       "      <td>0.197879</td>\n",
       "      <td>0.15063</td>\n",
       "      <td>0.794655</td>\n",
       "      <td>0.667068</td>\n",
       "      <td>...</td>\n",
       "      <td>0.126964</td>\n",
       "      <td>0.997351</td>\n",
       "      <td>0.79255</td>\n",
       "      <td>0.033902</td>\n",
       "      <td>7.378293</td>\n",
       "      <td>0.990586</td>\n",
       "      <td>0.09696</td>\n",
       "      <td>0.682362</td>\n",
       "      <td>1.980549</td>\n",
       "      <td>0.080017</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <th>10</th>\n",
       "      <td>0.142098</td>\n",
       "      <td>0.158131</td>\n",
       "      <td>0.076744</td>\n",
       "      <td>0.073414</td>\n",
       "      <td>0.080385</td>\n",
       "      <td>0.345805</td>\n",
       "      <td>0.143196</td>\n",
       "      <td>0.099663</td>\n",
       "      <td>0.663679</td>\n",
       "      <td>0.495413</td>\n",
       "      <td>...</td>\n",
       "      <td>0.012104</td>\n",
       "      <td>0.997351</td>\n",
       "      <td>0.661921</td>\n",
       "      <td>0.005027</td>\n",
       "      <td>4.581923</td>\n",
       "      <td>0.952093</td>\n",
       "      <td>0.014377</td>\n",
       "      <td>0.423747</td>\n",
       "      <td>2.735925</td>\n",
       "      <td>0.07503</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <th>10</th>\n",
       "      <td>0.193675</td>\n",
       "      <td>0.217827</td>\n",
       "      <td>0.112892</td>\n",
       "      <td>0.102723</td>\n",
       "      <td>0.113711</td>\n",
       "      <td>0.417543</td>\n",
       "      <td>0.186976</td>\n",
       "      <td>0.14264</td>\n",
       "      <td>0.77905</td>\n",
       "      <td>0.644213</td>\n",
       "      <td>...</td>\n",
       "      <td>0.112799</td>\n",
       "      <td>0.997351</td>\n",
       "      <td>0.776987</td>\n",
       "      <td>0.024665</td>\n",
       "      <td>6.889494</td>\n",
       "      <td>0.986169</td>\n",
       "      <td>0.070542</td>\n",
       "      <td>0.637157</td>\n",
       "      <td>2.094322</td>\n",
       "      <td>0.079146</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <th>10</th>\n",
       "      <td>0.157669</td>\n",
       "      <td>0.17911</td>\n",
       "      <td>0.093214</td>\n",
       "      <td>0.07686</td>\n",
       "      <td>0.084908</td>\n",
       "      <td>0.354809</td>\n",
       "      <td>0.145788</td>\n",
       "      <td>0.117162</td>\n",
       "      <td>0.72842</td>\n",
       "      <td>0.515201</td>\n",
       "      <td>...</td>\n",
       "      <td>0.201133</td>\n",
       "      <td>0.997351</td>\n",
       "      <td>0.72649</td>\n",
       "      <td>0.080293</td>\n",
       "      <td>8.627045</td>\n",
       "      <td>0.996386</td>\n",
       "      <td>0.229641</td>\n",
       "      <td>0.79785</td>\n",
       "      <td>1.564439</td>\n",
       "      <td>0.083728</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <th>10</th>\n",
       "      <td>0.197825</td>\n",
       "      <td>0.222848</td>\n",
       "      <td>0.115995</td>\n",
       "      <td>0.105119</td>\n",
       "      <td>0.116278</td>\n",
       "      <td>0.423346</td>\n",
       "      <td>0.191634</td>\n",
       "      <td>0.146241</td>\n",
       "      <td>0.786355</td>\n",
       "      <td>0.6555</td>\n",
       "      <td>...</td>\n",
       "      <td>0.124131</td>\n",
       "      <td>0.997351</td>\n",
       "      <td>0.784272</td>\n",
       "      <td>0.02885</td>\n",
       "      <td>7.119046</td>\n",
       "      <td>0.988209</td>\n",
       "      <td>0.082511</td>\n",
       "      <td>0.658386</td>\n",
       "      <td>2.041729</td>\n",
       "      <td>0.07956</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <th>10</th>\n",
       "      <td>0.20083</td>\n",
       "      <td>0.228163</td>\n",
       "      <td>0.120765</td>\n",
       "      <td>0.105947</td>\n",
       "      <td>0.1177</td>\n",
       "      <td>0.424558</td>\n",
       "      <td>0.194242</td>\n",
       "      <td>0.150831</td>\n",
       "      <td>0.797311</td>\n",
       "      <td>0.660029</td>\n",
       "      <td>...</td>\n",
       "      <td>0.139583</td>\n",
       "      <td>0.997351</td>\n",
       "      <td>0.795199</td>\n",
       "      <td>0.035164</td>\n",
       "      <td>7.40108</td>\n",
       "      <td>0.990311</td>\n",
       "      <td>0.10057</td>\n",
       "      <td>0.684469</td>\n",
       "      <td>1.96181</td>\n",
       "      <td>0.080221</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <td>0.138928</td>\n",
       "      <td>0.1543</td>\n",
       "      <td>0.074728</td>\n",
       "      <td>0.069565</td>\n",
       "      <td>0.076122</td>\n",
       "      <td>0.325661</td>\n",
       "      <td>0.134207</td>\n",
       "      <td>0.097182</td>\n",
       "      <td>0.650896</td>\n",
       "      <td>0.469209</td>\n",
       "      <td>...</td>\n",
       "      <td>0.069534</td>\n",
       "      <td>0.997351</td>\n",
       "      <td>0.649172</td>\n",
       "      <td>0.024053</td>\n",
       "      <td>6.825971</td>\n",
       "      <td>0.986343</td>\n",
       "      <td>0.068793</td>\n",
       "      <td>0.631282</td>\n",
       "      <td>2.160487</td>\n",
       "      <td>0.078851</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 25 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         PRECISION PRECISION_RECALL_MIN_DEN    RECALL       MAP MAP_MIN_DEN  \\\n",
       "  cutoff                                                                      \n",
       "0 10      0.202623                 0.228896  0.120448  0.105994    0.117016   \n",
       "1 10      0.145435                 0.162662  0.080011  0.075451    0.082986   \n",
       "2 10      0.203602                 0.229579   0.12052  0.107041    0.118379   \n",
       "3 10      0.202125                 0.228466  0.120046  0.107534    0.119399   \n",
       "4 10      0.142098                 0.158131  0.076744  0.073414    0.080385   \n",
       "5 10      0.193675                 0.217827  0.112892  0.102723    0.113711   \n",
       "6 10      0.157669                  0.17911  0.093214   0.07686    0.084908   \n",
       "7 10      0.197825                 0.222848  0.115995  0.105119    0.116278   \n",
       "8 10       0.20083                 0.228163  0.120765  0.105947      0.1177   \n",
       "9 10      0.138928                   0.1543  0.074728  0.069565    0.076122   \n",
       "\n",
       "               MRR      NDCG        F1  HIT_RATE ARHR_ALL_HITS  ...  \\\n",
       "  cutoff                                                        ...   \n",
       "0 10      0.423969  0.192351  0.151084  0.798141      0.657837  ...   \n",
       "1 10      0.354088  0.147499   0.10323  0.678287      0.508359  ...   \n",
       "2 10      0.425583  0.194463  0.151413  0.800299      0.662361  ...   \n",
       "3 10      0.429007  0.197879   0.15063  0.794655      0.667068  ...   \n",
       "4 10      0.345805  0.143196  0.099663  0.663679      0.495413  ...   \n",
       "5 10      0.417543  0.186976   0.14264   0.77905      0.644213  ...   \n",
       "6 10      0.354809  0.145788  0.117162   0.72842      0.515201  ...   \n",
       "7 10      0.423346  0.191634  0.146241  0.786355        0.6555  ...   \n",
       "8 10      0.424558  0.194242  0.150831  0.797311      0.660029  ...   \n",
       "9 10      0.325661  0.134207  0.097182  0.650896      0.469209  ...   \n",
       "\n",
       "         COVERAGE_ITEM_CORRECT COVERAGE_USER COVERAGE_USER_CORRECT  \\\n",
       "  cutoff                                                             \n",
       "0 10                  0.173835      0.997351              0.796026   \n",
       "1 10                  0.017255      0.997351               0.67649   \n",
       "2 10                  0.164048      0.997351              0.798179   \n",
       "3 10                  0.126964      0.997351               0.79255   \n",
       "4 10                  0.012104      0.997351              0.661921   \n",
       "5 10                  0.112799      0.997351              0.776987   \n",
       "6 10                  0.201133      0.997351               0.72649   \n",
       "7 10                  0.124131      0.997351              0.784272   \n",
       "8 10                  0.139583      0.997351              0.795199   \n",
       "9 10                  0.069534      0.997351              0.649172   \n",
       "\n",
       "         DIVERSITY_GINI SHANNON_ENTROPY RATIO_DIVERSITY_HERFINDAHL  \\\n",
       "  cutoff                                                             \n",
       "0 10           0.053959        8.034793                   0.994022   \n",
       "1 10           0.005378        4.700364                   0.954948   \n",
       "2 10           0.050306        7.938351                   0.993629   \n",
       "3 10           0.033902        7.378293                   0.990586   \n",
       "4 10           0.005027        4.581923                   0.952093   \n",
       "5 10           0.024665        6.889494                   0.986169   \n",
       "6 10           0.080293        8.627045                   0.996386   \n",
       "7 10            0.02885        7.119046                   0.988209   \n",
       "8 10           0.035164         7.40108                   0.990311   \n",
       "9 10           0.024053        6.825971                   0.986343   \n",
       "\n",
       "         RATIO_DIVERSITY_GINI RATIO_SHANNON_ENTROPY RATIO_AVERAGE_POPULARITY  \\\n",
       "  cutoff                                                                       \n",
       "0 10                 0.154324              0.743077                 1.774303   \n",
       "1 10                 0.015382              0.434701                 2.718386   \n",
       "2 10                 0.143876              0.734158                 1.806573   \n",
       "3 10                  0.09696              0.682362                 1.980549   \n",
       "4 10                 0.014377              0.423747                 2.735925   \n",
       "5 10                 0.070542              0.637157                 2.094322   \n",
       "6 10                 0.229641               0.79785                 1.564439   \n",
       "7 10                 0.082511              0.658386                 2.041729   \n",
       "8 10                  0.10057              0.684469                  1.96181   \n",
       "9 10                 0.068793              0.631282                 2.160487   \n",
       "\n",
       "         RATIO_NOVELTY  \n",
       "  cutoff                \n",
       "0 10          0.081837  \n",
       "1 10          0.075125  \n",
       "2 10          0.081526  \n",
       "3 10          0.080017  \n",
       "4 10           0.07503  \n",
       "5 10          0.079146  \n",
       "6 10          0.083728  \n",
       "7 10           0.07956  \n",
       "8 10          0.080221  \n",
       "9 10          0.078851  \n",
       "\n",
       "[10 rows x 25 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result_on_validation_df = search_metadata[\"result_on_validation_df\"]\n",
    "result_on_validation_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PRECISION</th>\n",
       "      <th>PRECISION_RECALL_MIN_DEN</th>\n",
       "      <th>RECALL</th>\n",
       "      <th>MAP</th>\n",
       "      <th>MAP_MIN_DEN</th>\n",
       "      <th>MRR</th>\n",
       "      <th>NDCG</th>\n",
       "      <th>F1</th>\n",
       "      <th>HIT_RATE</th>\n",
       "      <th>ARHR_ALL_HITS</th>\n",
       "      <th>...</th>\n",
       "      <th>COVERAGE_ITEM_CORRECT</th>\n",
       "      <th>COVERAGE_USER</th>\n",
       "      <th>COVERAGE_USER_CORRECT</th>\n",
       "      <th>DIVERSITY_GINI</th>\n",
       "      <th>SHANNON_ENTROPY</th>\n",
       "      <th>RATIO_DIVERSITY_HERFINDAHL</th>\n",
       "      <th>RATIO_DIVERSITY_GINI</th>\n",
       "      <th>RATIO_SHANNON_ENTROPY</th>\n",
       "      <th>RATIO_AVERAGE_POPULARITY</th>\n",
       "      <th>RATIO_NOVELTY</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>cutoff</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.328943</td>\n",
       "      <td>0.350793</td>\n",
       "      <td>0.144353</td>\n",
       "      <td>0.237251</td>\n",
       "      <td>0.248139</td>\n",
       "      <td>0.608559</td>\n",
       "      <td>0.30061</td>\n",
       "      <td>0.200652</td>\n",
       "      <td>0.872763</td>\n",
       "      <td>1.139525</td>\n",
       "      <td>...</td>\n",
       "      <td>0.152975</td>\n",
       "      <td>0.999338</td>\n",
       "      <td>0.872185</td>\n",
       "      <td>0.038284</td>\n",
       "      <td>7.569266</td>\n",
       "      <td>0.991963</td>\n",
       "      <td>0.109487</td>\n",
       "      <td>0.700027</td>\n",
       "      <td>1.915292</td>\n",
       "      <td>0.064382</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 25 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       PRECISION PRECISION_RECALL_MIN_DEN    RECALL       MAP MAP_MIN_DEN  \\\n",
       "cutoff                                                                      \n",
       "10      0.328943                 0.350793  0.144353  0.237251    0.248139   \n",
       "\n",
       "             MRR     NDCG        F1  HIT_RATE ARHR_ALL_HITS  ...  \\\n",
       "cutoff                                                       ...   \n",
       "10      0.608559  0.30061  0.200652  0.872763      1.139525  ...   \n",
       "\n",
       "       COVERAGE_ITEM_CORRECT COVERAGE_USER COVERAGE_USER_CORRECT  \\\n",
       "cutoff                                                             \n",
       "10                  0.152975      0.999338              0.872185   \n",
       "\n",
       "       DIVERSITY_GINI SHANNON_ENTROPY RATIO_DIVERSITY_HERFINDAHL  \\\n",
       "cutoff                                                             \n",
       "10           0.038284        7.569266                   0.991963   \n",
       "\n",
       "       RATIO_DIVERSITY_GINI RATIO_SHANNON_ENTROPY RATIO_AVERAGE_POPULARITY  \\\n",
       "cutoff                                                                       \n",
       "10                 0.109487              0.700027                 1.915292   \n",
       "\n",
       "       RATIO_NOVELTY  \n",
       "cutoff                \n",
       "10          0.064382  \n",
       "\n",
       "[1 rows x 25 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result_best_on_test = search_metadata[\"result_on_last\"]\n",
    "result_best_on_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'topK': 300, 'shrink': 1000, 'similarity': 'cosine', 'normalize': True}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_hyperparameters = search_metadata[\"hyperparameters_best\"]\n",
    "best_hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>train</th>\n",
       "      <th>validation</th>\n",
       "      <th>test</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.790614</td>\n",
       "      <td>5.419285</td>\n",
       "      <td>6.228587</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.550529</td>\n",
       "      <td>8.002338</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.721653</td>\n",
       "      <td>6.326061</td>\n",
       "      <td>6.375081</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.495981</td>\n",
       "      <td>7.337181</td>\n",
       "      <td>7.332524</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.628996</td>\n",
       "      <td>9.625446</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1.810539</td>\n",
       "      <td>9.066864</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1.304739</td>\n",
       "      <td>4.667664</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1.568717</td>\n",
       "      <td>8.511135</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>1.85302</td>\n",
       "      <td>7.490336</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1.650328</td>\n",
       "      <td>4.953592</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      train validation      test\n",
       "0  0.790614   5.419285  6.228587\n",
       "1  1.550529   8.002338       NaN\n",
       "2  1.721653   6.326061  6.375081\n",
       "3  1.495981   7.337181  7.332524\n",
       "4  1.628996   9.625446       NaN\n",
       "5  1.810539   9.066864       NaN\n",
       "6  1.304739   4.667664       NaN\n",
       "7  1.568717   8.511135       NaN\n",
       "8   1.85302   7.490336       NaN\n",
       "9  1.650328   4.953592       NaN"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "time_df = search_metadata[\"time_df\"]\n",
    "time_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[None, None, None, None, None, None, None, None, None, None]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "exception_list = search_metadata[\"exception_list\"]\n",
    "exception_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## An example with earlystopping, for FunKSVD"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "hyperparameters_range_dictionary = {\n",
    "    \"epochs\": Categorical([500]),\n",
    "    \"num_factors\": Integer(1, 200),\n",
    "    \"sgd_mode\": Categorical([\"sgd\", \"adagrad\", \"adam\"]),\n",
    "    \"batch_size\": Categorical([1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024]),\n",
    "    \"item_reg\": Real(low = 1e-5, high = 1e-2, prior = 'log-uniform'),\n",
    "    \"user_reg\": Real(low = 1e-5, high = 1e-2, prior = 'log-uniform'),\n",
    "    \"learning_rate\": Real(low = 1e-4, high = 1e-1, prior = 'log-uniform'),\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "earlystopping_keywargs = {\"validation_every_n\": 5,\n",
    "                          \"stop_on_validation\": True,\n",
    "                          \"evaluator_object\": evaluator_validation,\n",
    "                          \"lower_validations_allowed\": 5,\n",
    "                          \"validation_metric\": metric_to_optimize,\n",
    "                          }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "recommender_input_args = SearchInputRecommenderArgs(\n",
    "    CONSTRUCTOR_POSITIONAL_ARGS = [URM_train],     # For a CBF model simply put [URM_train, ICM_train]\n",
    "    CONSTRUCTOR_KEYWORD_ARGS = {},\n",
    "    FIT_POSITIONAL_ARGS = [],\n",
    "    FIT_KEYWORD_ARGS = {},\n",
    "    EARLYSTOPPING_KEYWORD_ARGS = earlystopping_keywargs,     # Additional hyperparameters for the fit function\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "recommender_input_args_last_test = SearchInputRecommenderArgs(\n",
    "    CONSTRUCTOR_POSITIONAL_ARGS = [URM_train_validation],     # For a CBF model simply put [URM_train_validation, ICM_train]\n",
    "    CONSTRUCTOR_KEYWORD_ARGS = {},\n",
    "    FIT_POSITIONAL_ARGS = [],\n",
    "    FIT_KEYWORD_ARGS = {},\n",
    "    EARLYSTOPPING_KEYWORD_ARGS = earlystopping_keywargs,     # Additional hyperparameters for the fit function\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from Recommenders.MatrixFactorization.Cython.MatrixFactorization_Cython import MatrixFactorization_FunkSVD_Cython\n",
    "\n",
    "recommender_class = MatrixFactorization_FunkSVD_Cython\n",
    "\n",
    "hyperparameterSearch = SearchBayesianSkopt(recommender_class,\n",
    "                                         evaluator_validation=evaluator_validation,\n",
    "                                         evaluator_test=evaluator_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration No: 1 started. Evaluating function at random point.\n",
      "SearchBayesianSkopt: Testing config: {'epochs': 500, 'num_factors': 53, 'sgd_mode': 'sgd', 'batch_size': 8, 'item_reg': 0.00012885833828867778, 'user_reg': 1.3327983000192677e-05, 'learning_rate': 0.0011498156311265017}\n",
      "MatrixFactorization_FunkSVD_Cython_Recommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 4.02 sec. MSE loss 1.28E+00. Sample per second: 159340\n",
      "FUNK_SVD: Epoch 1 of 500. Elapsed time 3.79 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.80 sec. MSE loss 1.17E+00. Sample per second: 168655\n",
      "FUNK_SVD: Epoch 2 of 500. Elapsed time 7.57 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.67 sec. MSE loss 1.14E+00. Sample per second: 174457\n",
      "FUNK_SVD: Epoch 3 of 500. Elapsed time 10.45 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.58 sec. MSE loss 1.11E+00. Sample per second: 178704\n",
      "FUNK_SVD: Epoch 4 of 500. Elapsed time 13.36 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.04 sec. MSE loss 1.10E+00. Sample per second: 210568\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.25 sec. Users per second: 1854\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0995850, PRECISION_RECALL_MIN_DEN: 0.1104261, RECALL: 0.0514475, MAP: 0.0439052, MAP_MIN_DEN: 0.0474488, MRR: 0.2259337, NDCG: 0.0962809, F1: 0.0678450, HIT_RATE: 0.5385126, ARHR_ALL_HITS: 0.3113147, NOVELTY: 0.0226039, AVERAGE_POPULARITY: 0.6804943, DIVERSITY_MEAN_INTER_LIST: 0.4728648, DIVERSITY_HERFINDAHL: 0.9472786, COVERAGE_ITEM: 0.0252382, COVERAGE_ITEM_CORRECT: 0.0182848, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5370861, DIVERSITY_GINI: 0.0049817, SHANNON_ENTROPY: 4.6032759, RATIO_DIVERSITY_HERFINDAHL: 0.9480525, RATIO_DIVERSITY_GINI: 0.0142478, RATIO_SHANNON_ENTROPY: 0.4257219, RATIO_AVERAGE_POPULARITY: 2.4636325, RATIO_NOVELTY: 0.0765906, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 5 of 500. Elapsed time 19.09 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.27 sec. MSE loss 1.08E+00. Sample per second: 195734\n",
      "FUNK_SVD: Epoch 6 of 500. Elapsed time 22.05 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.36 sec. MSE loss 1.06E+00. Sample per second: 190382\n",
      "FUNK_SVD: Epoch 7 of 500. Elapsed time 25.14 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.34 sec. MSE loss 1.05E+00. Sample per second: 191870\n",
      "FUNK_SVD: Epoch 8 of 500. Elapsed time 28.11 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.78 sec. MSE loss 1.04E+00. Sample per second: 230588\n",
      "FUNK_SVD: Epoch 9 of 500. Elapsed time 30.55 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.70 sec. MSE loss 1.03E+00. Sample per second: 172864\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.49 sec. Users per second: 1726\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0913347, PRECISION_RECALL_MIN_DEN: 0.1008718, RECALL: 0.0465364, MAP: 0.0390606, MAP_MIN_DEN: 0.0420836, MRR: 0.2072957, NDCG: 0.0877196, F1: 0.0616574, HIT_RATE: 0.5127822, ARHR_ALL_HITS: 0.2820357, NOVELTY: 0.0228993, AVERAGE_POPULARITY: 0.6381434, DIVERSITY_MEAN_INTER_LIST: 0.4837307, DIVERSITY_HERFINDAHL: 0.9483650, COVERAGE_ITEM: 0.0224054, COVERAGE_ITEM_CORRECT: 0.0167396, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5114238, DIVERSITY_GINI: 0.0049389, SHANNON_ENTROPY: 4.5910589, RATIO_DIVERSITY_HERFINDAHL: 0.9491398, RATIO_DIVERSITY_GINI: 0.0141253, RATIO_SHANNON_ENTROPY: 0.4245920, RATIO_AVERAGE_POPULARITY: 2.3103071, RATIO_NOVELTY: 0.0775914, \n",
      "\n",
      "FUNK_SVD: Epoch 10 of 500. Elapsed time 36.98 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.13 sec. MSE loss 1.02E+00. Sample per second: 204393\n",
      "FUNK_SVD: Epoch 11 of 500. Elapsed time 39.91 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.90 sec. MSE loss 1.01E+00. Sample per second: 220435\n",
      "FUNK_SVD: Epoch 12 of 500. Elapsed time 42.68 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.20 sec. MSE loss 1.00E+00. Sample per second: 199823\n",
      "FUNK_SVD: Epoch 13 of 500. Elapsed time 44.98 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.10 sec. MSE loss 9.97E-01. Sample per second: 206393\n",
      "FUNK_SVD: Epoch 14 of 500. Elapsed time 47.88 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.91 sec. MSE loss 9.83E-01. Sample per second: 219605\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.53 sec. Users per second: 1709\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0828021, PRECISION_RECALL_MIN_DEN: 0.0911446, RECALL: 0.0417413, MAP: 0.0347651, MAP_MIN_DEN: 0.0374511, MRR: 0.1926540, NDCG: 0.0800399, F1: 0.0555031, HIT_RATE: 0.4800797, ARHR_ALL_HITS: 0.2565495, NOVELTY: 0.0232793, AVERAGE_POPULARITY: 0.5883789, DIVERSITY_MEAN_INTER_LIST: 0.5139654, DIVERSITY_HERFINDAHL: 0.9513880, COVERAGE_ITEM: 0.0218903, COVERAGE_ITEM_CORRECT: 0.0182848, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4788079, DIVERSITY_GINI: 0.0051763, SHANNON_ENTROPY: 4.6549817, RATIO_DIVERSITY_HERFINDAHL: 0.9521653, RATIO_DIVERSITY_GINI: 0.0148043, RATIO_SHANNON_ENTROPY: 0.4305037, RATIO_AVERAGE_POPULARITY: 2.1301418, RATIO_NOVELTY: 0.0788792, \n",
      "\n",
      "FUNK_SVD: Epoch 15 of 500. Elapsed time 54.24 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.08 sec. MSE loss 9.78E-01. Sample per second: 207631\n",
      "FUNK_SVD: Epoch 16 of 500. Elapsed time 56.87 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.59 sec. MSE loss 9.72E-01. Sample per second: 247293\n",
      "FUNK_SVD: Epoch 17 of 500. Elapsed time 59.37 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.47 sec. MSE loss 9.65E-01. Sample per second: 184642\n",
      "FUNK_SVD: Epoch 18 of 500. Elapsed time 1.04 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.31 sec. MSE loss 9.61E-01. Sample per second: 193593\n",
      "FUNK_SVD: Epoch 19 of 500. Elapsed time 1.08 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.08 sec. MSE loss 9.53E-01. Sample per second: 207838\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.32 sec. Users per second: 1815\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0744024, PRECISION_RECALL_MIN_DEN: 0.0817089, RECALL: 0.0371753, MAP: 0.0305854, MAP_MIN_DEN: 0.0328968, MRR: 0.1775507, NDCG: 0.0718049, F1: 0.0495786, HIT_RATE: 0.4490372, ARHR_ALL_HITS: 0.2307945, NOVELTY: 0.0236565, AVERAGE_POPULARITY: 0.5414978, DIVERSITY_MEAN_INTER_LIST: 0.5254793, DIVERSITY_HERFINDAHL: 0.9525392, COVERAGE_ITEM: 0.0218903, COVERAGE_ITEM_CORRECT: 0.0167396, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4478477, DIVERSITY_GINI: 0.0053229, SHANNON_ENTROPY: 4.6940442, RATIO_DIVERSITY_HERFINDAHL: 0.9533174, RATIO_DIVERSITY_GINI: 0.0152236, RATIO_SHANNON_ENTROPY: 0.4341163, RATIO_AVERAGE_POPULARITY: 1.9604155, RATIO_NOVELTY: 0.0801574, \n",
      "\n",
      "FUNK_SVD: Epoch 20 of 500. Elapsed time 1.19 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.07 sec. MSE loss 9.52E-01. Sample per second: 208574\n",
      "FUNK_SVD: Epoch 21 of 500. Elapsed time 1.23 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.97 sec. MSE loss 9.45E-01. Sample per second: 215532\n",
      "FUNK_SVD: Epoch 22 of 500. Elapsed time 1.28 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.82 sec. MSE loss 9.44E-01. Sample per second: 167528\n",
      "FUNK_SVD: Epoch 23 of 500. Elapsed time 1.33 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.42 sec. MSE loss 9.34E-01. Sample per second: 186898\n",
      "FUNK_SVD: Epoch 24 of 500. Elapsed time 1.37 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.87 sec. MSE loss 9.26E-01. Sample per second: 223117\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.56 sec. Users per second: 1693\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0694555, PRECISION_RECALL_MIN_DEN: 0.0760500, RECALL: 0.0342082, MAP: 0.0278753, MAP_MIN_DEN: 0.0299479, MRR: 0.1665358, NDCG: 0.0668470, F1: 0.0458395, HIT_RATE: 0.4299469, ARHR_ALL_HITS: 0.2135130, NOVELTY: 0.0239443, AVERAGE_POPULARITY: 0.5103564, DIVERSITY_MEAN_INTER_LIST: 0.5388758, DIVERSITY_HERFINDAHL: 0.9538786, COVERAGE_ITEM: 0.0221478, COVERAGE_ITEM_CORRECT: 0.0169972, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4288079, DIVERSITY_GINI: 0.0054790, SHANNON_ENTROPY: 4.7409399, RATIO_DIVERSITY_HERFINDAHL: 0.9546579, RATIO_DIVERSITY_GINI: 0.0156702, RATIO_SHANNON_ENTROPY: 0.4384534, RATIO_AVERAGE_POPULARITY: 1.8476725, RATIO_NOVELTY: 0.0811323, \n",
      "\n",
      "FUNK_SVD: Epoch 25 of 500. Elapsed time 1.47 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.31 sec. MSE loss 9.24E-01. Sample per second: 193529\n",
      "FUNK_SVD: Epoch 26 of 500. Elapsed time 1.52 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Processed 640136 (100.0%) in 3.25 sec. MSE loss 9.18E-01. Sample per second: 197039\n",
      "FUNK_SVD: Epoch 27 of 500. Elapsed time 1.57 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.77 sec. MSE loss 9.17E-01. Sample per second: 230949\n",
      "FUNK_SVD: Epoch 28 of 500. Elapsed time 1.61 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.35 sec. MSE loss 9.12E-01. Sample per second: 191178\n",
      "FUNK_SVD: Epoch 29 of 500. Elapsed time 1.65 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.27 sec. MSE loss 9.09E-01. Sample per second: 195660\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.59 sec. Users per second: 1676\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0633798, PRECISION_RECALL_MIN_DEN: 0.0693543, RECALL: 0.0307000, MAP: 0.0248637, MAP_MIN_DEN: 0.0267379, MRR: 0.1524233, NDCG: 0.0606391, F1: 0.0413640, HIT_RATE: 0.4025564, ARHR_ALL_HITS: 0.1929833, NOVELTY: 0.0242611, AVERAGE_POPULARITY: 0.4746087, DIVERSITY_MEAN_INTER_LIST: 0.5401015, DIVERSITY_HERFINDAHL: 0.9540012, COVERAGE_ITEM: 0.0226629, COVERAGE_ITEM_CORRECT: 0.0164821, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4014901, DIVERSITY_GINI: 0.0054967, SHANNON_ENTROPY: 4.7614317, RATIO_DIVERSITY_HERFINDAHL: 0.9547806, RATIO_DIVERSITY_GINI: 0.0157209, RATIO_SHANNON_ENTROPY: 0.4403485, RATIO_AVERAGE_POPULARITY: 1.7182529, RATIO_NOVELTY: 0.0822060, \n",
      "\n",
      "FUNK_SVD: Convergence reached! Terminating at epoch 30. Best value for 'MAP' at epoch 5 is 0.0439. Elapsed time 1.76 min\n",
      "FUNK_SVD: Epoch 30 of 500. Elapsed time 1.76 min\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.54 sec. Users per second: 1702\n",
      "SearchBayesianSkopt: New best config found. Config 0: {'epochs': 5, 'num_factors': 53, 'sgd_mode': 'sgd', 'batch_size': 8, 'item_reg': 0.00012885833828867778, 'user_reg': 1.3327983000192677e-05, 'learning_rate': 0.0011498156311265017} - results: PRECISION: 0.0995850, PRECISION_RECALL_MIN_DEN: 0.1104261, RECALL: 0.0514475, MAP: 0.0439052, MAP_MIN_DEN: 0.0474488, MRR: 0.2259337, NDCG: 0.0962809, F1: 0.0678450, HIT_RATE: 0.5385126, ARHR_ALL_HITS: 0.3113147, NOVELTY: 0.0226039, AVERAGE_POPULARITY: 0.6804943, DIVERSITY_MEAN_INTER_LIST: 0.4728648, DIVERSITY_HERFINDAHL: 0.9472786, COVERAGE_ITEM: 0.0252382, COVERAGE_ITEM_CORRECT: 0.0182848, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5370861, DIVERSITY_GINI: 0.0049817, SHANNON_ENTROPY: 4.6032759, RATIO_DIVERSITY_HERFINDAHL: 0.9480525, RATIO_DIVERSITY_GINI: 0.0142478, RATIO_SHANNON_ENTROPY: 0.4257219, RATIO_AVERAGE_POPULARITY: 2.4636325, RATIO_NOVELTY: 0.0765906, \n",
      "\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 3.32 sec. Users per second: 1817\n",
      "SearchBayesianSkopt: Config evaluated with evaluator_test. Config: {'epochs': 5, 'num_factors': 53, 'sgd_mode': 'sgd', 'batch_size': 8, 'item_reg': 0.00012885833828867778, 'user_reg': 1.3327983000192677e-05, 'learning_rate': 0.0011498156311265017} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.1252154, PRECISION_RECALL_MIN_DEN: 0.1334045, RECALL: 0.0529349, MAP: 0.0593963, MAP_MIN_DEN: 0.0621646, MRR: 0.2734886, NDCG: 0.1170758, F1: 0.0744121, HIT_RATE: 0.6068588, ARHR_ALL_HITS: 0.3959720, NOVELTY: 0.0226037, AVERAGE_POPULARITY: 0.6805272, DIVERSITY_MEAN_INTER_LIST: 0.4727202, DIVERSITY_HERFINDAHL: 0.9472642, COVERAGE_ITEM: 0.0252382, COVERAGE_ITEM_CORRECT: 0.0180273, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.6064570, DIVERSITY_GINI: 0.0049807, SHANNON_ENTROPY: 4.6029113, RATIO_DIVERSITY_HERFINDAHL: 0.9480381, RATIO_DIVERSITY_GINI: 0.0142449, RATIO_SHANNON_ENTROPY: 0.4256882, RATIO_AVERAGE_POPULARITY: 2.4637517, RATIO_NOVELTY: 0.0765900, \n",
      "\n",
      "\n",
      "Iteration No: 1 ended. Evaluation done at random point.\n",
      "Time taken: 112.8247\n",
      "Function value obtained: -0.0439\n",
      "Current minimum: -0.0439\n",
      "Iteration No: 2 started. Evaluating function at random point.\n",
      "SearchBayesianSkopt: Testing config: {'epochs': 500, 'num_factors': 40, 'sgd_mode': 'adagrad', 'batch_size': 16, 'item_reg': 0.0003358028682286283, 'user_reg': 2.129854424829547e-05, 'learning_rate': 0.0023556938742367067}\n",
      "MatrixFactorization_FunkSVD_Cython_Recommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 3.46 sec. MSE loss 1.03E+01. Sample per second: 184925\n",
      "FUNK_SVD: Epoch 1 of 500. Elapsed time 2.48 sec\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 2.88 sec. MSE loss 7.29E+00. Sample per second: 222252\n",
      "FUNK_SVD: Epoch 2 of 500. Elapsed time 4.90 sec\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 4.32 sec. MSE loss 5.75E+00. Sample per second: 148050\n",
      "FUNK_SVD: Epoch 3 of 500. Elapsed time 8.35 sec\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 3.32 sec. MSE loss 4.74E+00. Sample per second: 192584\n",
      "FUNK_SVD: Epoch 4 of 500. Elapsed time 11.35 sec\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 4.64 sec. MSE loss 4.00E+00. Sample per second: 137998\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.70 sec. Users per second: 1283\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1027556, PRECISION_RECALL_MIN_DEN: 0.1145107, RECALL: 0.0535569, MAP: 0.0467209, MAP_MIN_DEN: 0.0512382, MRR: 0.2509092, NDCG: 0.0950765, F1: 0.0704137, HIT_RATE: 0.5610890, ARHR_ALL_HITS: 0.3374159, NOVELTY: 0.0225629, AVERAGE_POPULARITY: 0.6888377, DIVERSITY_MEAN_INTER_LIST: 0.7339976, DIVERSITY_HERFINDAHL: 0.9733876, COVERAGE_ITEM: 0.0821530, COVERAGE_ITEM_CORRECT: 0.0381149, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5596026, DIVERSITY_GINI: 0.0106715, SHANNON_ENTROPY: 5.7467934, RATIO_DIVERSITY_HERFINDAHL: 0.9741828, RATIO_DIVERSITY_GINI: 0.0305209, RATIO_SHANNON_ENTROPY: 0.5314771, RATIO_AVERAGE_POPULARITY: 2.4938387, RATIO_NOVELTY: 0.0764518, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 5 of 500. Elapsed time 20.37 sec\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 4.66 sec. MSE loss 3.45E+00. Sample per second: 137358\n",
      "FUNK_SVD: Epoch 6 of 500. Elapsed time 24.69 sec\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 4.90 sec. MSE loss 3.02E+00. Sample per second: 130536\n",
      "FUNK_SVD: Epoch 7 of 500. Elapsed time 28.93 sec\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 5.19 sec. MSE loss 2.68E+00. Sample per second: 123266\n",
      "FUNK_SVD: Epoch 8 of 500. Elapsed time 33.22 sec\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 4.42 sec. MSE loss 2.40E+00. Sample per second: 144779\n",
      "FUNK_SVD: Epoch 9 of 500. Elapsed time 37.45 sec\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 4.64 sec. MSE loss 2.18E+00. Sample per second: 137998\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.99 sec. Users per second: 1207\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1080511, PRECISION_RECALL_MIN_DEN: 0.1208182, RECALL: 0.0572309, MAP: 0.0495202, MAP_MIN_DEN: 0.0542981, MRR: 0.2612348, NDCG: 0.1017560, F1: 0.0748281, HIT_RATE: 0.5780212, ARHR_ALL_HITS: 0.3541615, NOVELTY: 0.0224086, AVERAGE_POPULARITY: 0.7117494, DIVERSITY_MEAN_INTER_LIST: 0.6738402, DIVERSITY_HERFINDAHL: 0.9673728, COVERAGE_ITEM: 0.0471285, COVERAGE_ITEM_CORRECT: 0.0265259, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5764901, DIVERSITY_GINI: 0.0079973, SHANNON_ENTROPY: 5.3408490, RATIO_DIVERSITY_HERFINDAHL: 0.9681631, RATIO_DIVERSITY_GINI: 0.0228725, RATIO_SHANNON_ENTROPY: 0.4939344, RATIO_AVERAGE_POPULARITY: 2.5767870, RATIO_NOVELTY: 0.0759288, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 10 of 500. Elapsed time 46.67 sec\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 4.89 sec. MSE loss 1.99E+00. Sample per second: 130855\n",
      "FUNK_SVD: Epoch 11 of 500. Elapsed time 50.92 sec\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 5.09 sec. MSE loss 1.85E+00. Sample per second: 125716\n",
      "FUNK_SVD: Epoch 12 of 500. Elapsed time 55.12 sec\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 4.32 sec. MSE loss 1.72E+00. Sample per second: 148194\n",
      "FUNK_SVD: Epoch 13 of 500. Elapsed time 59.34 sec\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 4.68 sec. MSE loss 1.62E+00. Sample per second: 136872\n",
      "FUNK_SVD: Epoch 14 of 500. Elapsed time 1.06 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 4.96 sec. MSE loss 1.53E+00. Sample per second: 128959\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.71 sec. Users per second: 1278\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1094954, PRECISION_RECALL_MIN_DEN: 0.1222020, RECALL: 0.0578714, MAP: 0.0502259, MAP_MIN_DEN: 0.0550159, MRR: 0.2617675, NDCG: 0.1038678, F1: 0.0757218, HIT_RATE: 0.5795153, ARHR_ALL_HITS: 0.3568285, NOVELTY: 0.0223773, AVERAGE_POPULARITY: 0.7164640, DIVERSITY_MEAN_INTER_LIST: 0.6439891, DIVERSITY_HERFINDAHL: 0.9643882, COVERAGE_ITEM: 0.0370847, COVERAGE_ITEM_CORRECT: 0.0234355, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5779801, DIVERSITY_GINI: 0.0072164, SHANNON_ENTROPY: 5.1835927, RATIO_DIVERSITY_HERFINDAHL: 0.9651761, RATIO_DIVERSITY_GINI: 0.0206391, RATIO_SHANNON_ENTROPY: 0.4793909, RATIO_AVERAGE_POPULARITY: 2.5938558, RATIO_NOVELTY: 0.0758227, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 15 of 500. Elapsed time 1.21 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Processed 640144 (100.0%) in 5.01 sec. MSE loss 1.46E+00. Sample per second: 127688\n",
      "FUNK_SVD: Epoch 16 of 500. Elapsed time 1.28 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 4.32 sec. MSE loss 1.40E+00. Sample per second: 148304\n",
      "FUNK_SVD: Epoch 17 of 500. Elapsed time 1.36 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 4.56 sec. MSE loss 1.34E+00. Sample per second: 140246\n",
      "FUNK_SVD: Epoch 18 of 500. Elapsed time 1.43 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 4.82 sec. MSE loss 1.30E+00. Sample per second: 132573\n",
      "FUNK_SVD: Epoch 19 of 500. Elapsed time 1.50 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 5.06 sec. MSE loss 1.26E+00. Sample per second: 126492\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.83 sec. Users per second: 1248\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1099104, PRECISION_RECALL_MIN_DEN: 0.1221991, RECALL: 0.0575444, MAP: 0.0504870, MAP_MIN_DEN: 0.0551992, MRR: 0.2610717, NDCG: 0.1046849, F1: 0.0755395, HIT_RATE: 0.5761952, ARHR_ALL_HITS: 0.3572360, NOVELTY: 0.0223789, AVERAGE_POPULARITY: 0.7159351, DIVERSITY_MEAN_INTER_LIST: 0.6252631, DIVERSITY_HERFINDAHL: 0.9625159, COVERAGE_ITEM: 0.0324491, COVERAGE_ITEM_CORRECT: 0.0231780, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5746689, DIVERSITY_GINI: 0.0068256, SHANNON_ENTROPY: 5.0945282, RATIO_DIVERSITY_HERFINDAHL: 0.9633023, RATIO_DIVERSITY_GINI: 0.0195214, RATIO_SHANNON_ENTROPY: 0.4711541, RATIO_AVERAGE_POPULARITY: 2.5919410, RATIO_NOVELTY: 0.0758283, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 20 of 500. Elapsed time 1.65 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 5.18 sec. MSE loss 1.23E+00. Sample per second: 123527\n",
      "FUNK_SVD: Epoch 21 of 500. Elapsed time 1.72 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 4.36 sec. MSE loss 1.20E+00. Sample per second: 146688\n",
      "FUNK_SVD: Epoch 22 of 500. Elapsed time 1.79 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 4.52 sec. MSE loss 1.18E+00. Sample per second: 141500\n",
      "FUNK_SVD: Epoch 23 of 500. Elapsed time 1.86 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 4.77 sec. MSE loss 1.16E+00. Sample per second: 134035\n",
      "FUNK_SVD: Epoch 24 of 500. Elapsed time 1.93 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 5.04 sec. MSE loss 1.14E+00. Sample per second: 126812\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.75 sec. Users per second: 1268\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1102424, PRECISION_RECALL_MIN_DEN: 0.1225121, RECALL: 0.0576475, MAP: 0.0503446, MAP_MIN_DEN: 0.0551128, MRR: 0.2604658, NDCG: 0.1053723, F1: 0.0757068, HIT_RATE: 0.5765272, ARHR_ALL_HITS: 0.3562619, NOVELTY: 0.0223946, AVERAGE_POPULARITY: 0.7131571, DIVERSITY_MEAN_INTER_LIST: 0.6134726, DIVERSITY_HERFINDAHL: 0.9613371, COVERAGE_ITEM: 0.0296163, COVERAGE_ITEM_CORRECT: 0.0224054, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5750000, DIVERSITY_GINI: 0.0066085, SHANNON_ENTROPY: 5.0451072, RATIO_DIVERSITY_HERFINDAHL: 0.9621225, RATIO_DIVERSITY_GINI: 0.0189007, RATIO_SHANNON_ENTROPY: 0.4665835, RATIO_AVERAGE_POPULARITY: 2.5818835, RATIO_NOVELTY: 0.0758815, \n",
      "\n",
      "FUNK_SVD: Epoch 25 of 500. Elapsed time 2.08 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 5.12 sec. MSE loss 1.13E+00. Sample per second: 125029\n",
      "FUNK_SVD: Epoch 26 of 500. Elapsed time 2.15 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 4.36 sec. MSE loss 1.11E+00. Sample per second: 146753\n",
      "FUNK_SVD: Epoch 27 of 500. Elapsed time 2.22 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 4.61 sec. MSE loss 1.10E+00. Sample per second: 138678\n",
      "FUNK_SVD: Epoch 28 of 500. Elapsed time 2.29 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 3.57 sec. MSE loss 1.09E+00. Sample per second: 178810\n",
      "FUNK_SVD: Epoch 29 of 500. Elapsed time 2.34 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 4.80 sec. MSE loss 1.08E+00. Sample per second: 133253\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.80 sec. Users per second: 1254\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1093459, PRECISION_RECALL_MIN_DEN: 0.1213804, RECALL: 0.0569141, MAP: 0.0500390, MAP_MIN_DEN: 0.0546856, MRR: 0.2598237, NDCG: 0.1052829, F1: 0.0748626, HIT_RATE: 0.5728752, ARHR_ALL_HITS: 0.3547250, NOVELTY: 0.0224156, AVERAGE_POPULARITY: 0.7095926, DIVERSITY_MEAN_INTER_LIST: 0.6048597, DIVERSITY_HERFINDAHL: 0.9604759, COVERAGE_ITEM: 0.0293587, COVERAGE_ITEM_CORRECT: 0.0213752, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5713576, DIVERSITY_GINI: 0.0064582, SHANNON_ENTROPY: 5.0127956, RATIO_DIVERSITY_HERFINDAHL: 0.9612606, RATIO_DIVERSITY_GINI: 0.0184707, RATIO_SHANNON_ENTROPY: 0.4635952, RATIO_AVERAGE_POPULARITY: 2.5689788, RATIO_NOVELTY: 0.0759526, \n",
      "\n",
      "FUNK_SVD: Epoch 30 of 500. Elapsed time 2.49 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 4.92 sec. MSE loss 1.08E+00. Sample per second: 130046\n",
      "FUNK_SVD: Epoch 31 of 500. Elapsed time 2.57 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 5.34 sec. MSE loss 1.06E+00. Sample per second: 119936\n",
      "FUNK_SVD: Epoch 32 of 500. Elapsed time 2.64 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 4.73 sec. MSE loss 1.06E+00. Sample per second: 135196\n",
      "FUNK_SVD: Epoch 33 of 500. Elapsed time 2.71 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 4.96 sec. MSE loss 1.05E+00. Sample per second: 128855\n",
      "FUNK_SVD: Epoch 34 of 500. Elapsed time 2.78 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 5.22 sec. MSE loss 1.04E+00. Sample per second: 122647\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.71 sec. Users per second: 1278\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1088811, PRECISION_RECALL_MIN_DEN: 0.1207895, RECALL: 0.0566129, MAP: 0.0495226, MAP_MIN_DEN: 0.0540868, MRR: 0.2570477, NDCG: 0.1047677, F1: 0.0744931, HIT_RATE: 0.5708831, ARHR_ALL_HITS: 0.3510591, NOVELTY: 0.0224440, AVERAGE_POPULARITY: 0.7048254, DIVERSITY_MEAN_INTER_LIST: 0.6002132, DIVERSITY_HERFINDAHL: 0.9600114, COVERAGE_ITEM: 0.0306464, COVERAGE_ITEM_CORRECT: 0.0208602, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5693709, DIVERSITY_GINI: 0.0063993, SHANNON_ENTROPY: 5.0002326, RATIO_DIVERSITY_HERFINDAHL: 0.9607957, RATIO_DIVERSITY_GINI: 0.0183021, RATIO_SHANNON_ENTROPY: 0.4624334, RATIO_AVERAGE_POPULARITY: 2.5517198, RATIO_NOVELTY: 0.0760490, \n",
      "\n",
      "FUNK_SVD: Epoch 35 of 500. Elapsed time 2.93 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 5.39 sec. MSE loss 1.04E+00. Sample per second: 118741\n",
      "FUNK_SVD: Epoch 36 of 500. Elapsed time 3.01 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 4.55 sec. MSE loss 1.03E+00. Sample per second: 140567\n",
      "FUNK_SVD: Epoch 37 of 500. Elapsed time 3.08 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 4.91 sec. MSE loss 1.03E+00. Sample per second: 130463\n",
      "FUNK_SVD: Epoch 38 of 500. Elapsed time 3.15 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 5.15 sec. MSE loss 1.02E+00. Sample per second: 124222\n",
      "FUNK_SVD: Epoch 39 of 500. Elapsed time 3.22 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 4.41 sec. MSE loss 1.02E+00. Sample per second: 145222\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.70 sec. Users per second: 1281\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1082503, PRECISION_RECALL_MIN_DEN: 0.1198101, RECALL: 0.0557961, MAP: 0.0489527, MAP_MIN_DEN: 0.0533424, MRR: 0.2537135, NDCG: 0.1038715, F1: 0.0736370, HIT_RATE: 0.5668991, ARHR_ALL_HITS: 0.3467386, NOVELTY: 0.0224693, AVERAGE_POPULARITY: 0.7007147, DIVERSITY_MEAN_INTER_LIST: 0.5969070, DIVERSITY_HERFINDAHL: 0.9596808, COVERAGE_ITEM: 0.0303889, COVERAGE_ITEM_CORRECT: 0.0213752, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5653974, DIVERSITY_GINI: 0.0063799, SHANNON_ENTROPY: 4.9946795, RATIO_DIVERSITY_HERFINDAHL: 0.9604648, RATIO_DIVERSITY_GINI: 0.0182469, RATIO_SHANNON_ENTROPY: 0.4619198, RATIO_AVERAGE_POPULARITY: 2.5368376, RATIO_NOVELTY: 0.0761345, \n",
      "\n",
      "FUNK_SVD: Epoch 40 of 500. Elapsed time 3.37 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 4.46 sec. MSE loss 1.02E+00. Sample per second: 143275\n",
      "FUNK_SVD: Epoch 41 of 500. Elapsed time 3.44 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 4.87 sec. MSE loss 1.02E+00. Sample per second: 131410\n",
      "FUNK_SVD: Epoch 42 of 500. Elapsed time 3.51 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 5.18 sec. MSE loss 1.01E+00. Sample per second: 123465\n",
      "FUNK_SVD: Epoch 43 of 500. Elapsed time 3.59 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 4.48 sec. MSE loss 1.01E+00. Sample per second: 143010\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Epoch 44 of 500. Elapsed time 3.66 min\n",
      "FUNK_SVD: Processed 640144 (100.0%) in 4.63 sec. MSE loss 1.00E+00. Sample per second: 138161\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.87 sec. Users per second: 1238\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1073373, PRECISION_RECALL_MIN_DEN: 0.1186354, RECALL: 0.0550678, MAP: 0.0482923, MAP_MIN_DEN: 0.0525795, MRR: 0.2509920, NDCG: 0.1027217, F1: 0.0727912, HIT_RATE: 0.5634130, ARHR_ALL_HITS: 0.3425592, NOVELTY: 0.0224971, AVERAGE_POPULARITY: 0.6962233, DIVERSITY_MEAN_INTER_LIST: 0.5957626, DIVERSITY_HERFINDAHL: 0.9595664, COVERAGE_ITEM: 0.0309039, COVERAGE_ITEM_CORRECT: 0.0218903, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5619205, DIVERSITY_GINI: 0.0063850, SHANNON_ENTROPY: 4.9959996, RATIO_DIVERSITY_HERFINDAHL: 0.9603503, RATIO_DIVERSITY_GINI: 0.0182612, RATIO_SHANNON_ENTROPY: 0.4620419, RATIO_AVERAGE_POPULARITY: 2.5205771, RATIO_NOVELTY: 0.0762288, \n",
      "\n",
      "FUNK_SVD: Convergence reached! Terminating at epoch 45. Best value for 'MAP' at epoch 20 is 0.0505. Elapsed time 3.81 min\n",
      "FUNK_SVD: Epoch 45 of 500. Elapsed time 3.81 min\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.64 sec. Users per second: 1299\n",
      "SearchBayesianSkopt: New best config found. Config 1: {'epochs': 20, 'num_factors': 40, 'sgd_mode': 'adagrad', 'batch_size': 16, 'item_reg': 0.0003358028682286283, 'user_reg': 2.129854424829547e-05, 'learning_rate': 0.0023556938742367067} - results: PRECISION: 0.1099104, PRECISION_RECALL_MIN_DEN: 0.1221991, RECALL: 0.0575444, MAP: 0.0504870, MAP_MIN_DEN: 0.0551992, MRR: 0.2610717, NDCG: 0.1046849, F1: 0.0755395, HIT_RATE: 0.5761952, ARHR_ALL_HITS: 0.3572360, NOVELTY: 0.0223789, AVERAGE_POPULARITY: 0.7159351, DIVERSITY_MEAN_INTER_LIST: 0.6252631, DIVERSITY_HERFINDAHL: 0.9625159, COVERAGE_ITEM: 0.0324491, COVERAGE_ITEM_CORRECT: 0.0231780, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5746689, DIVERSITY_GINI: 0.0068256, SHANNON_ENTROPY: 5.0945282, RATIO_DIVERSITY_HERFINDAHL: 0.9633023, RATIO_DIVERSITY_GINI: 0.0195214, RATIO_SHANNON_ENTROPY: 0.4711541, RATIO_AVERAGE_POPULARITY: 2.5919410, RATIO_NOVELTY: 0.0758283, \n",
      "\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 4.71 sec. Users per second: 1282\n",
      "SearchBayesianSkopt: Config evaluated with evaluator_test. Config: {'epochs': 20, 'num_factors': 40, 'sgd_mode': 'adagrad', 'batch_size': 16, 'item_reg': 0.0003358028682286283, 'user_reg': 2.129854424829547e-05, 'learning_rate': 0.0023556938742367067} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.1345593, PRECISION_RECALL_MIN_DEN: 0.1436526, RECALL: 0.0564271, MAP: 0.0656972, MAP_MIN_DEN: 0.0689043, MRR: 0.2943474, NDCG: 0.1197241, F1: 0.0795113, HIT_RATE: 0.6267396, ARHR_ALL_HITS: 0.4302729, NOVELTY: 0.0223787, AVERAGE_POPULARITY: 0.7159753, DIVERSITY_MEAN_INTER_LIST: 0.6250699, DIVERSITY_HERFINDAHL: 0.9624966, COVERAGE_ITEM: 0.0324491, COVERAGE_ITEM_CORRECT: 0.0231780, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.6263245, DIVERSITY_GINI: 0.0068226, SHANNON_ENTROPY: 5.0938915, RATIO_DIVERSITY_HERFINDAHL: 0.9632830, RATIO_DIVERSITY_GINI: 0.0195128, RATIO_SHANNON_ENTROPY: 0.4710952, RATIO_AVERAGE_POPULARITY: 2.5920863, RATIO_NOVELTY: 0.0758275, \n",
      "\n",
      "\n",
      "Iteration No: 2 ended. Evaluation done at random point.\n",
      "Time taken: 238.1170\n",
      "Function value obtained: -0.0505\n",
      "Current minimum: -0.0505\n",
      "Iteration No: 3 started. Evaluating function at random point.\n",
      "SearchBayesianSkopt: Testing config: {'epochs': 500, 'num_factors': 122, 'sgd_mode': 'sgd', 'batch_size': 256, 'item_reg': 0.006261745893648571, 'user_reg': 0.0002345781923232971, 'learning_rate': 0.00016732948647785252}\n",
      "MatrixFactorization_FunkSVD_Cython_Recommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.21 sec. MSE loss 1.05E+01. Sample per second: 69517\n",
      "FUNK_SVD: Epoch 1 of 500. Elapsed time 9.06 sec\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.53 sec. MSE loss 5.25E+00. Sample per second: 67145\n",
      "FUNK_SVD: Epoch 2 of 500. Elapsed time 18.38 sec\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.07 sec. MSE loss 2.98E+00. Sample per second: 63596\n",
      "FUNK_SVD: Epoch 3 of 500. Elapsed time 27.92 sec\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 7.55 sec. MSE loss 1.99E+00. Sample per second: 84852\n",
      "FUNK_SVD: Epoch 4 of 500. Elapsed time 35.40 sec\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.96 sec. MSE loss 1.56E+00. Sample per second: 64300\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.50 sec. Users per second: 1338\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0080677, PRECISION_RECALL_MIN_DEN: 0.0086186, RECALL: 0.0030444, MAP: 0.0025898, MAP_MIN_DEN: 0.0028020, MRR: 0.0235581, NDCG: 0.0051050, F1: 0.0044207, HIT_RATE: 0.0737052, ARHR_ALL_HITS: 0.0246030, NOVELTY: 0.0319039, AVERAGE_POPULARITY: 0.0751165, DIVERSITY_MEAN_INTER_LIST: 0.9966035, DIVERSITY_HERFINDAHL: 0.9996438, COVERAGE_ITEM: 0.9989699, COVERAGE_ITEM_CORRECT: 0.0988926, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.0735099, DIVERSITY_GINI: 0.6690264, SHANNON_ENTROPY: 11.6650728, RATIO_DIVERSITY_HERFINDAHL: 1.0004605, RATIO_DIVERSITY_GINI: 1.9134378, RATIO_SHANNON_ENTROPY: 1.0788136, RATIO_AVERAGE_POPULARITY: 0.2719484, RATIO_NOVELTY: 0.1081026, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 5 of 500. Elapsed time 49.34 sec\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.86 sec. MSE loss 1.38E+00. Sample per second: 64909\n",
      "FUNK_SVD: Epoch 6 of 500. Elapsed time 58.71 sec\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.01 sec. MSE loss 1.30E+00. Sample per second: 63965\n",
      "FUNK_SVD: Epoch 7 of 500. Elapsed time 1.13 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.62 sec. MSE loss 1.27E+00. Sample per second: 66568\n",
      "FUNK_SVD: Epoch 8 of 500. Elapsed time 1.29 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.30 sec. MSE loss 1.25E+00. Sample per second: 62183\n",
      "FUNK_SVD: Epoch 9 of 500. Elapsed time 1.45 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.33 sec. MSE loss 1.25E+00. Sample per second: 68642\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.69 sec. Users per second: 1284\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0084163, PRECISION_RECALL_MIN_DEN: 0.0090419, RECALL: 0.0032413, MAP: 0.0026727, MAP_MIN_DEN: 0.0028927, MRR: 0.0243155, NDCG: 0.0053616, F1: 0.0046801, HIT_RATE: 0.0768592, ARHR_ALL_HITS: 0.0253964, NOVELTY: 0.0318698, AVERAGE_POPULARITY: 0.0767448, DIVERSITY_MEAN_INTER_LIST: 0.9965990, DIVERSITY_HERFINDAHL: 0.9996434, COVERAGE_ITEM: 0.9989699, COVERAGE_ITEM_CORRECT: 0.1004378, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.0766556, DIVERSITY_GINI: 0.6683354, SHANNON_ENTROPY: 11.6641172, RATIO_DIVERSITY_HERFINDAHL: 1.0004600, RATIO_DIVERSITY_GINI: 1.9114615, RATIO_SHANNON_ENTROPY: 1.0787252, RATIO_AVERAGE_POPULARITY: 0.2778436, RATIO_NOVELTY: 0.1079870, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 10 of 500. Elapsed time 1.68 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.29 sec. MSE loss 1.24E+00. Sample per second: 68943\n",
      "FUNK_SVD: Epoch 11 of 500. Elapsed time 1.84 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.67 sec. MSE loss 1.24E+00. Sample per second: 66203\n",
      "FUNK_SVD: Epoch 12 of 500. Elapsed time 1.99 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.67 sec. MSE loss 1.24E+00. Sample per second: 66197\n",
      "FUNK_SVD: Epoch 13 of 500. Elapsed time 2.14 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.97 sec. MSE loss 1.24E+00. Sample per second: 64180\n",
      "FUNK_SVD: Epoch 14 of 500. Elapsed time 2.30 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.23 sec. MSE loss 1.24E+00. Sample per second: 62574\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.63 sec. Users per second: 1300\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0086321, PRECISION_RECALL_MIN_DEN: 0.0093172, RECALL: 0.0034080, MAP: 0.0027178, MAP_MIN_DEN: 0.0029489, MRR: 0.0247333, NDCG: 0.0055601, F1: 0.0048868, HIT_RATE: 0.0788513, ARHR_ALL_HITS: 0.0258305, NOVELTY: 0.0318539, AVERAGE_POPULARITY: 0.0780011, DIVERSITY_MEAN_INTER_LIST: 0.9965934, DIVERSITY_HERFINDAHL: 0.9996428, COVERAGE_ITEM: 0.9989699, COVERAGE_ITEM_CORRECT: 0.1004378, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.0786424, DIVERSITY_GINI: 0.6677292, SHANNON_ENTROPY: 11.6631055, RATIO_DIVERSITY_HERFINDAHL: 1.0004595, RATIO_DIVERSITY_GINI: 1.9097278, RATIO_SHANNON_ENTROPY: 1.0786317, RATIO_AVERAGE_POPULARITY: 0.2823920, RATIO_NOVELTY: 0.1079333, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 15 of 500. Elapsed time 2.53 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Processed 640256 (100.0%) in 8.26 sec. MSE loss 1.24E+00. Sample per second: 77544\n",
      "FUNK_SVD: Epoch 16 of 500. Elapsed time 2.65 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.38 sec. MSE loss 1.24E+00. Sample per second: 68258\n",
      "FUNK_SVD: Epoch 17 of 500. Elapsed time 2.80 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.75 sec. MSE loss 1.24E+00. Sample per second: 65654\n",
      "FUNK_SVD: Epoch 18 of 500. Elapsed time 2.96 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.05 sec. MSE loss 1.24E+00. Sample per second: 63694\n",
      "FUNK_SVD: Epoch 19 of 500. Elapsed time 3.12 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 8.93 sec. MSE loss 1.24E+00. Sample per second: 71661\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.50 sec. Users per second: 1339\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0087317, PRECISION_RECALL_MIN_DEN: 0.0094239, RECALL: 0.0034604, MAP: 0.0027641, MAP_MIN_DEN: 0.0029988, MRR: 0.0251002, NDCG: 0.0057061, F1: 0.0049565, HIT_RATE: 0.0796813, ARHR_ALL_HITS: 0.0262422, NOVELTY: 0.0318371, AVERAGE_POPULARITY: 0.0791954, DIVERSITY_MEAN_INTER_LIST: 0.9965875, DIVERSITY_HERFINDAHL: 0.9996422, COVERAGE_ITEM: 0.9989699, COVERAGE_ITEM_CORRECT: 0.0991501, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.0794702, DIVERSITY_GINI: 0.6669591, SHANNON_ENTROPY: 11.6617951, RATIO_DIVERSITY_HERFINDAHL: 1.0004589, RATIO_DIVERSITY_GINI: 1.9075254, RATIO_SHANNON_ENTROPY: 1.0785105, RATIO_AVERAGE_POPULARITY: 0.2867156, RATIO_NOVELTY: 0.1078764, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 20 of 500. Elapsed time 3.34 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.43 sec. MSE loss 1.24E+00. Sample per second: 67882\n",
      "FUNK_SVD: Epoch 21 of 500. Elapsed time 3.49 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.48 sec. MSE loss 1.24E+00. Sample per second: 67481\n",
      "FUNK_SVD: Epoch 22 of 500. Elapsed time 3.64 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.11 sec. MSE loss 1.23E+00. Sample per second: 63292\n",
      "FUNK_SVD: Epoch 23 of 500. Elapsed time 3.80 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.18 sec. MSE loss 1.24E+00. Sample per second: 69779\n",
      "FUNK_SVD: Epoch 24 of 500. Elapsed time 3.95 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.28 sec. MSE loss 1.23E+00. Sample per second: 68956\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.56 sec. Users per second: 1320\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0090471, PRECISION_RECALL_MIN_DEN: 0.0098417, RECALL: 0.0037027, MAP: 0.0028382, MAP_MIN_DEN: 0.0030959, MRR: 0.0257799, NDCG: 0.0059788, F1: 0.0052548, HIT_RATE: 0.0826693, ARHR_ALL_HITS: 0.0269433, NOVELTY: 0.0318193, AVERAGE_POPULARITY: 0.0806006, DIVERSITY_MEAN_INTER_LIST: 0.9965753, DIVERSITY_HERFINDAHL: 0.9996410, COVERAGE_ITEM: 0.9989699, COVERAGE_ITEM_CORRECT: 0.0996652, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.0824503, DIVERSITY_GINI: 0.6656656, SHANNON_ENTROPY: 11.6594838, RATIO_DIVERSITY_HERFINDAHL: 1.0004577, RATIO_DIVERSITY_GINI: 1.9038260, RATIO_SHANNON_ENTROPY: 1.0782967, RATIO_AVERAGE_POPULARITY: 0.2918029, RATIO_NOVELTY: 0.1078158, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 25 of 500. Elapsed time 4.18 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.03 sec. MSE loss 1.24E+00. Sample per second: 63818\n",
      "FUNK_SVD: Epoch 26 of 500. Elapsed time 4.33 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.39 sec. MSE loss 1.24E+00. Sample per second: 68151\n",
      "FUNK_SVD: Epoch 27 of 500. Elapsed time 4.49 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 8.84 sec. MSE loss 1.24E+00. Sample per second: 72415\n",
      "FUNK_SVD: Epoch 28 of 500. Elapsed time 4.63 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.28 sec. MSE loss 1.23E+00. Sample per second: 68963\n",
      "FUNK_SVD: Epoch 29 of 500. Elapsed time 4.77 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.57 sec. MSE loss 1.23E+00. Sample per second: 66861\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.52 sec. Users per second: 1333\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0091799, PRECISION_RECALL_MIN_DEN: 0.0099855, RECALL: 0.0037595, MAP: 0.0029067, MAP_MIN_DEN: 0.0031748, MRR: 0.0264318, NDCG: 0.0061775, F1: 0.0053344, HIT_RATE: 0.0838313, ARHR_ALL_HITS: 0.0276045, NOVELTY: 0.0317986, AVERAGE_POPULARITY: 0.0821318, DIVERSITY_MEAN_INTER_LIST: 0.9965536, DIVERSITY_HERFINDAHL: 0.9996388, COVERAGE_ITEM: 0.9987123, COVERAGE_ITEM_CORRECT: 0.0988926, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.0836093, DIVERSITY_GINI: 0.6636632, SHANNON_ENTROPY: 11.6555622, RATIO_DIVERSITY_HERFINDAHL: 1.0004555, RATIO_DIVERSITY_GINI: 1.8980991, RATIO_SHANNON_ENTROPY: 1.0779340, RATIO_AVERAGE_POPULARITY: 0.2973463, RATIO_NOVELTY: 0.1077459, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 30 of 500. Elapsed time 5.00 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.14 sec. MSE loss 1.23E+00. Sample per second: 70085\n",
      "FUNK_SVD: Epoch 31 of 500. Elapsed time 5.15 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.26 sec. MSE loss 1.23E+00. Sample per second: 69111\n",
      "FUNK_SVD: Epoch 32 of 500. Elapsed time 5.30 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.71 sec. MSE loss 1.23E+00. Sample per second: 65951\n",
      "FUNK_SVD: Epoch 33 of 500. Elapsed time 5.46 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.26 sec. MSE loss 1.23E+00. Sample per second: 62411\n",
      "FUNK_SVD: Epoch 34 of 500. Elapsed time 5.62 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.23 sec. MSE loss 1.23E+00. Sample per second: 69322\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.43 sec. Users per second: 1360\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0094622, PRECISION_RECALL_MIN_DEN: 0.0103583, RECALL: 0.0040328, MAP: 0.0029775, MAP_MIN_DEN: 0.0032638, MRR: 0.0271042, NDCG: 0.0064338, F1: 0.0056553, HIT_RATE: 0.0864874, ARHR_ALL_HITS: 0.0282946, NOVELTY: 0.0317756, AVERAGE_POPULARITY: 0.0837789, DIVERSITY_MEAN_INTER_LIST: 0.9965281, DIVERSITY_HERFINDAHL: 0.9996363, COVERAGE_ITEM: 0.9984548, COVERAGE_ITEM_CORRECT: 0.0988926, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.0862583, DIVERSITY_GINI: 0.6617070, SHANNON_ENTROPY: 11.6514832, RATIO_DIVERSITY_HERFINDAHL: 1.0004529, RATIO_DIVERSITY_GINI: 1.8925042, RATIO_SHANNON_ENTROPY: 1.0775568, RATIO_AVERAGE_POPULARITY: 0.3033097, RATIO_NOVELTY: 0.1076677, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 35 of 500. Elapsed time 5.84 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.25 sec. MSE loss 1.23E+00. Sample per second: 62428\n",
      "FUNK_SVD: Epoch 36 of 500. Elapsed time 6.00 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.60 sec. MSE loss 1.24E+00. Sample per second: 66707\n",
      "FUNK_SVD: Epoch 37 of 500. Elapsed time 6.16 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.59 sec. MSE loss 1.23E+00. Sample per second: 66761\n",
      "FUNK_SVD: Epoch 38 of 500. Elapsed time 6.31 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.06 sec. MSE loss 1.23E+00. Sample per second: 63662\n",
      "FUNK_SVD: Epoch 39 of 500. Elapsed time 6.47 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.87 sec. MSE loss 1.23E+00. Sample per second: 64836\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.95 sec. Users per second: 1527\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0096116, PRECISION_RECALL_MIN_DEN: 0.0105372, RECALL: 0.0041754, MAP: 0.0030282, MAP_MIN_DEN: 0.0033320, MRR: 0.0275668, NDCG: 0.0066261, F1: 0.0058217, HIT_RATE: 0.0879814, ARHR_ALL_HITS: 0.0287770, NOVELTY: 0.0317411, AVERAGE_POPULARITY: 0.0854771, DIVERSITY_MEAN_INTER_LIST: 0.9965061, DIVERSITY_HERFINDAHL: 0.9996341, COVERAGE_ITEM: 0.9984548, COVERAGE_ITEM_CORRECT: 0.0999227, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.0877483, DIVERSITY_GINI: 0.6603308, SHANNON_ENTROPY: 11.6483381, RATIO_DIVERSITY_HERFINDAHL: 1.0004507, RATIO_DIVERSITY_GINI: 1.8885682, RATIO_SHANNON_ENTROPY: 1.0772659, RATIO_AVERAGE_POPULARITY: 0.3094576, RATIO_NOVELTY: 0.1075509, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 40 of 500. Elapsed time 6.69 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 7.01 sec. MSE loss 1.23E+00. Sample per second: 91348\n",
      "FUNK_SVD: Epoch 41 of 500. Elapsed time 6.80 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.54 sec. MSE loss 1.23E+00. Sample per second: 97968\n",
      "FUNK_SVD: Epoch 42 of 500. Elapsed time 6.91 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 7.28 sec. MSE loss 1.23E+00. Sample per second: 87965\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Epoch 43 of 500. Elapsed time 7.02 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.30 sec. MSE loss 1.23E+00. Sample per second: 101564\n",
      "FUNK_SVD: Epoch 44 of 500. Elapsed time 7.12 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 7.25 sec. MSE loss 1.23E+00. Sample per second: 88293\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.19 sec. Users per second: 1438\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0098440, PRECISION_RECALL_MIN_DEN: 0.0107738, RECALL: 0.0042859, MAP: 0.0031124, MAP_MIN_DEN: 0.0034256, MRR: 0.0283008, NDCG: 0.0068696, F1: 0.0059718, HIT_RATE: 0.0901394, ARHR_ALL_HITS: 0.0295647, NOVELTY: 0.0316974, AVERAGE_POPULARITY: 0.0877446, DIVERSITY_MEAN_INTER_LIST: 0.9964656, DIVERSITY_HERFINDAHL: 0.9996300, COVERAGE_ITEM: 0.9984548, COVERAGE_ITEM_CORRECT: 0.0994077, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.0899007, DIVERSITY_GINI: 0.6577621, SHANNON_ENTROPY: 11.6426352, RATIO_DIVERSITY_HERFINDAHL: 1.0004467, RATIO_DIVERSITY_GINI: 1.8812215, RATIO_SHANNON_ENTROPY: 1.0767385, RATIO_AVERAGE_POPULARITY: 0.3176670, RATIO_NOVELTY: 0.1074028, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 45 of 500. Elapsed time 7.31 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 7.02 sec. MSE loss 1.23E+00. Sample per second: 91191\n",
      "FUNK_SVD: Epoch 46 of 500. Elapsed time 7.41 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.72 sec. MSE loss 1.23E+00. Sample per second: 95298\n",
      "FUNK_SVD: Epoch 47 of 500. Elapsed time 7.53 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.76 sec. MSE loss 1.23E+00. Sample per second: 94720\n",
      "FUNK_SVD: Epoch 48 of 500. Elapsed time 7.63 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 7.18 sec. MSE loss 1.23E+00. Sample per second: 89162\n",
      "FUNK_SVD: Epoch 49 of 500. Elapsed time 7.73 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.65 sec. MSE loss 1.23E+00. Sample per second: 96232\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.42 sec. Users per second: 1762\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0101262, PRECISION_RECALL_MIN_DEN: 0.0110726, RECALL: 0.0044073, MAP: 0.0032353, MAP_MIN_DEN: 0.0035573, MRR: 0.0294336, NDCG: 0.0072336, F1: 0.0061415, HIT_RATE: 0.0926295, ARHR_ALL_HITS: 0.0307445, NOVELTY: 0.0316683, AVERAGE_POPULARITY: 0.0895678, DIVERSITY_MEAN_INTER_LIST: 0.9964187, DIVERSITY_HERFINDAHL: 0.9996253, COVERAGE_ITEM: 0.9981973, COVERAGE_ITEM_CORRECT: 0.1001803, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.0923841, DIVERSITY_GINI: 0.6553121, SHANNON_ENTROPY: 11.6367232, RATIO_DIVERSITY_HERFINDAHL: 1.0004420, RATIO_DIVERSITY_GINI: 1.8742146, RATIO_SHANNON_ENTROPY: 1.0761918, RATIO_AVERAGE_POPULARITY: 0.3242675, RATIO_NOVELTY: 0.1073044, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 50 of 500. Elapsed time 7.90 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 5.75 sec. MSE loss 1.22E+00. Sample per second: 111380\n",
      "FUNK_SVD: Epoch 51 of 500. Elapsed time 7.99 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 7.27 sec. MSE loss 1.23E+00. Sample per second: 88082\n",
      "FUNK_SVD: Epoch 52 of 500. Elapsed time 8.10 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.47 sec. MSE loss 1.23E+00. Sample per second: 98923\n",
      "FUNK_SVD: Epoch 53 of 500. Elapsed time 8.21 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.68 sec. MSE loss 1.23E+00. Sample per second: 95790\n",
      "FUNK_SVD: Epoch 54 of 500. Elapsed time 8.31 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.85 sec. MSE loss 1.23E+00. Sample per second: 93517\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.04 sec. Users per second: 1981\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0103586, PRECISION_RECALL_MIN_DEN: 0.0113068, RECALL: 0.0045126, MAP: 0.0033322, MAP_MIN_DEN: 0.0036704, MRR: 0.0302713, NDCG: 0.0074894, F1: 0.0062865, HIT_RATE: 0.0944555, ARHR_ALL_HITS: 0.0316454, NOVELTY: 0.0316332, AVERAGE_POPULARITY: 0.0917162, DIVERSITY_MEAN_INTER_LIST: 0.9963523, DIVERSITY_HERFINDAHL: 0.9996187, COVERAGE_ITEM: 0.9979397, COVERAGE_ITEM_CORRECT: 0.1004378, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.0942053, DIVERSITY_GINI: 0.6518547, SHANNON_ENTROPY: 11.6284139, RATIO_DIVERSITY_HERFINDAHL: 1.0004353, RATIO_DIVERSITY_GINI: 1.8643263, RATIO_SHANNON_ENTROPY: 1.0754233, RATIO_AVERAGE_POPULARITY: 0.3320455, RATIO_NOVELTY: 0.1071855, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 55 of 500. Elapsed time 8.46 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 5.43 sec. MSE loss 1.23E+00. Sample per second: 117910\n",
      "FUNK_SVD: Epoch 56 of 500. Elapsed time 8.54 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 7.96 sec. MSE loss 1.23E+00. Sample per second: 80440\n",
      "FUNK_SVD: Epoch 57 of 500. Elapsed time 8.66 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.25 sec. MSE loss 1.23E+00. Sample per second: 62439\n",
      "FUNK_SVD: Epoch 58 of 500. Elapsed time 8.82 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.50 sec. MSE loss 1.23E+00. Sample per second: 67374\n",
      "FUNK_SVD: Epoch 59 of 500. Elapsed time 8.97 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.81 sec. MSE loss 1.23E+00. Sample per second: 65253\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.64 sec. Users per second: 1299\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0106740, PRECISION_RECALL_MIN_DEN: 0.0116841, RECALL: 0.0047533, MAP: 0.0034528, MAP_MIN_DEN: 0.0038020, MRR: 0.0314240, NDCG: 0.0078858, F1: 0.0065775, HIT_RATE: 0.0972776, ARHR_ALL_HITS: 0.0328332, NOVELTY: 0.0315861, AVERAGE_POPULARITY: 0.0944463, DIVERSITY_MEAN_INTER_LIST: 0.9962659, DIVERSITY_HERFINDAHL: 0.9996101, COVERAGE_ITEM: 0.9979397, COVERAGE_ITEM_CORRECT: 0.0996652, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.0970199, DIVERSITY_GINI: 0.6486981, SHANNON_ENTROPY: 11.6197636, RATIO_DIVERSITY_HERFINDAHL: 1.0004267, RATIO_DIVERSITY_GINI: 1.8552983, RATIO_SHANNON_ENTROPY: 1.0746233, RATIO_AVERAGE_POPULARITY: 0.3419293, RATIO_NOVELTY: 0.1070257, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 60 of 500. Elapsed time 9.21 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.97 sec. MSE loss 1.23E+00. Sample per second: 64212\n",
      "FUNK_SVD: Epoch 61 of 500. Elapsed time 9.36 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.64 sec. MSE loss 1.23E+00. Sample per second: 60200\n",
      "FUNK_SVD: Epoch 62 of 500. Elapsed time 9.52 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.11 sec. MSE loss 1.23E+00. Sample per second: 63352\n",
      "FUNK_SVD: Epoch 63 of 500. Elapsed time 9.68 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.49 sec. MSE loss 1.22E+00. Sample per second: 67448\n",
      "FUNK_SVD: Epoch 64 of 500. Elapsed time 9.84 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.67 sec. MSE loss 1.22E+00. Sample per second: 66238\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.49 sec. Users per second: 1341\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0110724, PRECISION_RECALL_MIN_DEN: 0.0121104, RECALL: 0.0049326, MAP: 0.0035746, MAP_MIN_DEN: 0.0039308, MRR: 0.0324318, NDCG: 0.0082277, F1: 0.0068248, HIT_RATE: 0.1005976, ARHR_ALL_HITS: 0.0339462, NOVELTY: 0.0315488, AVERAGE_POPULARITY: 0.0967166, DIVERSITY_MEAN_INTER_LIST: 0.9961793, DIVERSITY_HERFINDAHL: 0.9996014, COVERAGE_ITEM: 0.9976822, COVERAGE_ITEM_CORRECT: 0.1001803, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.1003311, DIVERSITY_GINI: 0.6457192, SHANNON_ENTROPY: 11.6112606, RATIO_DIVERSITY_HERFINDAHL: 1.0004180, RATIO_DIVERSITY_GINI: 1.8467784, RATIO_SHANNON_ENTROPY: 1.0738369, RATIO_AVERAGE_POPULARITY: 0.3501488, RATIO_NOVELTY: 0.1068993, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 65 of 500. Elapsed time 10.07 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.41 sec. MSE loss 1.22E+00. Sample per second: 68039\n",
      "FUNK_SVD: Epoch 66 of 500. Elapsed time 10.22 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.74 sec. MSE loss 1.22E+00. Sample per second: 65681\n",
      "FUNK_SVD: Epoch 67 of 500. Elapsed time 10.38 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.95 sec. MSE loss 1.22E+00. Sample per second: 64342\n",
      "FUNK_SVD: Epoch 68 of 500. Elapsed time 10.53 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.07 sec. MSE loss 1.23E+00. Sample per second: 63591\n",
      "FUNK_SVD: Epoch 69 of 500. Elapsed time 10.68 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 8.44 sec. MSE loss 1.22E+00. Sample per second: 75880\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.12 sec. Users per second: 1461\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0115206, PRECISION_RECALL_MIN_DEN: 0.0125946, RECALL: 0.0051836, MAP: 0.0037291, MAP_MIN_DEN: 0.0040975, MRR: 0.0337290, NDCG: 0.0086734, F1: 0.0071500, HIT_RATE: 0.1040837, ARHR_ALL_HITS: 0.0353673, NOVELTY: 0.0315051, AVERAGE_POPULARITY: 0.0996354, DIVERSITY_MEAN_INTER_LIST: 0.9960504, DIVERSITY_HERFINDAHL: 0.9995885, COVERAGE_ITEM: 0.9976822, COVERAGE_ITEM_CORRECT: 0.1012104, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.1038079, DIVERSITY_GINI: 0.6422826, SHANNON_ENTROPY: 11.6005743, RATIO_DIVERSITY_HERFINDAHL: 1.0004051, RATIO_DIVERSITY_GINI: 1.8369498, RATIO_SHANNON_ENTROPY: 1.0728486, RATIO_AVERAGE_POPULARITY: 0.3607159, RATIO_NOVELTY: 0.1067512, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 70 of 500. Elapsed time 10.89 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.77 sec. MSE loss 1.22E+00. Sample per second: 65546\n",
      "FUNK_SVD: Epoch 71 of 500. Elapsed time 11.04 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.08 sec. MSE loss 1.23E+00. Sample per second: 63467\n",
      "FUNK_SVD: Epoch 72 of 500. Elapsed time 11.20 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.40 sec. MSE loss 1.22E+00. Sample per second: 68084\n",
      "FUNK_SVD: Epoch 73 of 500. Elapsed time 11.35 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.70 sec. MSE loss 1.22E+00. Sample per second: 65986\n",
      "FUNK_SVD: Epoch 74 of 500. Elapsed time 11.51 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.93 sec. MSE loss 1.22E+00. Sample per second: 64489\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.64 sec. Users per second: 1299\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0119356, PRECISION_RECALL_MIN_DEN: 0.0130308, RECALL: 0.0053357, MAP: 0.0039124, MAP_MIN_DEN: 0.0042923, MRR: 0.0352715, NDCG: 0.0091079, F1: 0.0073746, HIT_RATE: 0.1072377, ARHR_ALL_HITS: 0.0370549, NOVELTY: 0.0314525, AVERAGE_POPULARITY: 0.1023352, DIVERSITY_MEAN_INTER_LIST: 0.9959289, DIVERSITY_HERFINDAHL: 0.9995764, COVERAGE_ITEM: 0.9976822, COVERAGE_ITEM_CORRECT: 0.1012104, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.1069536, DIVERSITY_GINI: 0.6386127, SHANNON_ENTROPY: 11.5895378, RATIO_DIVERSITY_HERFINDAHL: 1.0003930, RATIO_DIVERSITY_GINI: 1.8264536, RATIO_SHANNON_ENTROPY: 1.0718279, RATIO_AVERAGE_POPULARITY: 0.3704899, RATIO_NOVELTY: 0.1065731, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 75 of 500. Elapsed time 11.74 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.85 sec. MSE loss 1.22E+00. Sample per second: 64999\n",
      "FUNK_SVD: Epoch 76 of 500. Elapsed time 11.89 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.00 sec. MSE loss 1.22E+00. Sample per second: 64005\n",
      "FUNK_SVD: Epoch 77 of 500. Elapsed time 12.05 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.29 sec. MSE loss 1.22E+00. Sample per second: 68934\n",
      "FUNK_SVD: Epoch 78 of 500. Elapsed time 12.20 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.63 sec. MSE loss 1.22E+00. Sample per second: 66501\n",
      "FUNK_SVD: Epoch 79 of 500. Elapsed time 12.36 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.89 sec. MSE loss 1.22E+00. Sample per second: 64698\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.65 sec. Users per second: 1296\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0122178, PRECISION_RECALL_MIN_DEN: 0.0133561, RECALL: 0.0055067, MAP: 0.0041005, MAP_MIN_DEN: 0.0045054, MRR: 0.0370972, NDCG: 0.0095560, F1: 0.0075917, HIT_RATE: 0.1100598, ARHR_ALL_HITS: 0.0389165, NOVELTY: 0.0314004, AVERAGE_POPULARITY: 0.1053742, DIVERSITY_MEAN_INTER_LIST: 0.9957723, DIVERSITY_HERFINDAHL: 0.9995607, COVERAGE_ITEM: 0.9976822, COVERAGE_ITEM_CORRECT: 0.1009529, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.1097682, DIVERSITY_GINI: 0.6348512, SHANNON_ENTROPY: 11.5771828, RATIO_DIVERSITY_HERFINDAHL: 1.0003773, RATIO_DIVERSITY_GINI: 1.8156958, RATIO_SHANNON_ENTROPY: 1.0706853, RATIO_AVERAGE_POPULARITY: 0.3814923, RATIO_NOVELTY: 0.1063966, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 80 of 500. Elapsed time 12.59 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.80 sec. MSE loss 1.22E+00. Sample per second: 65332\n",
      "FUNK_SVD: Epoch 81 of 500. Elapsed time 12.74 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.19 sec. MSE loss 1.22E+00. Sample per second: 62795\n",
      "FUNK_SVD: Epoch 82 of 500. Elapsed time 12.90 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 7.70 sec. MSE loss 1.22E+00. Sample per second: 83190\n",
      "FUNK_SVD: Epoch 83 of 500. Elapsed time 13.03 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.90 sec. MSE loss 1.22E+00. Sample per second: 64664\n",
      "FUNK_SVD: Epoch 84 of 500. Elapsed time 13.18 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.01 sec. MSE loss 1.22E+00. Sample per second: 63967\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.79 sec. Users per second: 1257\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0126660, PRECISION_RECALL_MIN_DEN: 0.0138405, RECALL: 0.0057304, MAP: 0.0042615, MAP_MIN_DEN: 0.0046770, MRR: 0.0384415, NDCG: 0.0099570, F1: 0.0078908, HIT_RATE: 0.1137118, ARHR_ALL_HITS: 0.0403920, NOVELTY: 0.0313502, AVERAGE_POPULARITY: 0.1085461, DIVERSITY_MEAN_INTER_LIST: 0.9955714, DIVERSITY_HERFINDAHL: 0.9995406, COVERAGE_ITEM: 0.9974247, COVERAGE_ITEM_CORRECT: 0.1009529, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.1134106, DIVERSITY_GINI: 0.6311567, SHANNON_ENTROPY: 11.5632465, RATIO_DIVERSITY_HERFINDAHL: 1.0003572, RATIO_DIVERSITY_GINI: 1.8051294, RATIO_SHANNON_ENTROPY: 1.0693965, RATIO_AVERAGE_POPULARITY: 0.3929757, RATIO_NOVELTY: 0.1062264, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 85 of 500. Elapsed time 13.41 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.98 sec. MSE loss 1.22E+00. Sample per second: 64181\n",
      "FUNK_SVD: Epoch 86 of 500. Elapsed time 13.56 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.22 sec. MSE loss 1.21E+00. Sample per second: 62623\n",
      "FUNK_SVD: Epoch 87 of 500. Elapsed time 13.72 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.46 sec. MSE loss 1.22E+00. Sample per second: 67659\n",
      "FUNK_SVD: Epoch 88 of 500. Elapsed time 13.87 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.86 sec. MSE loss 1.22E+00. Sample per second: 64928\n",
      "FUNK_SVD: Epoch 89 of 500. Elapsed time 14.03 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.01 sec. MSE loss 1.22E+00. Sample per second: 63934\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.65 sec. Users per second: 1295\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0130478, PRECISION_RECALL_MIN_DEN: 0.0141596, RECALL: 0.0057677, MAP: 0.0044609, MAP_MIN_DEN: 0.0049119, MRR: 0.0400734, NDCG: 0.0104035, F1: 0.0079993, HIT_RATE: 0.1163679, ARHR_ALL_HITS: 0.0421802, NOVELTY: 0.0313000, AVERAGE_POPULARITY: 0.1119376, DIVERSITY_MEAN_INTER_LIST: 0.9953538, DIVERSITY_HERFINDAHL: 0.9995189, COVERAGE_ITEM: 0.9974247, COVERAGE_ITEM_CORRECT: 0.1019830, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.1160596, DIVERSITY_GINI: 0.6268629, SHANNON_ENTROPY: 11.5478485, RATIO_DIVERSITY_HERFINDAHL: 1.0003354, RATIO_DIVERSITY_GINI: 1.7928488, RATIO_SHANNON_ENTROPY: 1.0679724, RATIO_AVERAGE_POPULARITY: 0.4052540, RATIO_NOVELTY: 0.1060563, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 90 of 500. Elapsed time 14.26 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.02 sec. MSE loss 1.21E+00. Sample per second: 63846\n",
      "FUNK_SVD: Epoch 91 of 500. Elapsed time 14.41 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.31 sec. MSE loss 1.22E+00. Sample per second: 68738\n",
      "FUNK_SVD: Epoch 92 of 500. Elapsed time 14.57 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.50 sec. MSE loss 1.22E+00. Sample per second: 67396\n",
      "FUNK_SVD: Epoch 93 of 500. Elapsed time 14.72 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.76 sec. MSE loss 1.22E+00. Sample per second: 65630\n",
      "FUNK_SVD: Epoch 94 of 500. Elapsed time 14.88 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.80 sec. MSE loss 1.22E+00. Sample per second: 65328\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.86 sec. Users per second: 1240\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0134296, PRECISION_RECALL_MIN_DEN: 0.0145845, RECALL: 0.0060078, MAP: 0.0046398, MAP_MIN_DEN: 0.0051621, MRR: 0.0417410, NDCG: 0.0109688, F1: 0.0083018, HIT_RATE: 0.1200199, ARHR_ALL_HITS: 0.0439065, NOVELTY: 0.0312352, AVERAGE_POPULARITY: 0.1159007, DIVERSITY_MEAN_INTER_LIST: 0.9950484, DIVERSITY_HERFINDAHL: 0.9994883, COVERAGE_ITEM: 0.9974247, COVERAGE_ITEM_CORRECT: 0.1017255, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.1197020, DIVERSITY_GINI: 0.6215670, SHANNON_ENTROPY: 11.5283229, RATIO_DIVERSITY_HERFINDAHL: 1.0003049, RATIO_DIVERSITY_GINI: 1.7777025, RATIO_SHANNON_ENTROPY: 1.0661666, RATIO_AVERAGE_POPULARITY: 0.4196020, RATIO_NOVELTY: 0.1058369, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 95 of 500. Elapsed time 15.11 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Processed 640256 (100.0%) in 8.54 sec. MSE loss 1.22E+00. Sample per second: 74952\n",
      "FUNK_SVD: Epoch 96 of 500. Elapsed time 15.24 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.29 sec. MSE loss 1.22E+00. Sample per second: 62232\n",
      "FUNK_SVD: Epoch 97 of 500. Elapsed time 15.40 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.50 sec. MSE loss 1.22E+00. Sample per second: 67379\n",
      "FUNK_SVD: Epoch 98 of 500. Elapsed time 15.56 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.31 sec. MSE loss 1.22E+00. Sample per second: 62065\n",
      "FUNK_SVD: Epoch 99 of 500. Elapsed time 15.72 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.87 sec. MSE loss 1.22E+00. Sample per second: 64889\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.61 sec. Users per second: 1306\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0138280, PRECISION_RECALL_MIN_DEN: 0.0150423, RECALL: 0.0062747, MAP: 0.0048458, MAP_MIN_DEN: 0.0053886, MRR: 0.0435508, NDCG: 0.0114577, F1: 0.0086323, HIT_RATE: 0.1235060, ARHR_ALL_HITS: 0.0458277, NOVELTY: 0.0311713, AVERAGE_POPULARITY: 0.1198536, DIVERSITY_MEAN_INTER_LIST: 0.9947268, DIVERSITY_HERFINDAHL: 0.9994562, COVERAGE_ITEM: 0.9974247, COVERAGE_ITEM_CORRECT: 0.1006953, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.1231788, DIVERSITY_GINI: 0.6160898, SHANNON_ENTROPY: 11.5079653, RATIO_DIVERSITY_HERFINDAHL: 1.0002727, RATIO_DIVERSITY_GINI: 1.7620376, RATIO_SHANNON_ENTROPY: 1.0642839, RATIO_AVERAGE_POPULARITY: 0.4339129, RATIO_NOVELTY: 0.1056203, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 100 of 500. Elapsed time 15.96 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.89 sec. MSE loss 1.22E+00. Sample per second: 64751\n",
      "FUNK_SVD: Epoch 101 of 500. Elapsed time 16.11 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.01 sec. MSE loss 1.22E+00. Sample per second: 63945\n",
      "FUNK_SVD: Epoch 102 of 500. Elapsed time 16.26 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.21 sec. MSE loss 1.21E+00. Sample per second: 69508\n",
      "FUNK_SVD: Epoch 103 of 500. Elapsed time 16.42 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.44 sec. MSE loss 1.22E+00. Sample per second: 67788\n",
      "FUNK_SVD: Epoch 104 of 500. Elapsed time 16.57 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.67 sec. MSE loss 1.22E+00. Sample per second: 66197\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.71 sec. Users per second: 1278\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0145584, PRECISION_RECALL_MIN_DEN: 0.0159753, RECALL: 0.0068283, MAP: 0.0051239, MAP_MIN_DEN: 0.0057328, MRR: 0.0458906, NDCG: 0.0122905, F1: 0.0092964, HIT_RATE: 0.1296481, ARHR_ALL_HITS: 0.0483881, NOVELTY: 0.0311169, AVERAGE_POPULARITY: 0.1238535, DIVERSITY_MEAN_INTER_LIST: 0.9943313, DIVERSITY_HERFINDAHL: 0.9994166, COVERAGE_ITEM: 0.9969096, COVERAGE_ITEM_CORRECT: 0.1006953, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.1293046, DIVERSITY_GINI: 0.6109131, SHANNON_ENTROPY: 11.4862195, RATIO_DIVERSITY_HERFINDAHL: 1.0002331, RATIO_DIVERSITY_GINI: 1.7472319, RATIO_SHANNON_ENTROPY: 1.0622728, RATIO_AVERAGE_POPULARITY: 0.4483940, RATIO_NOVELTY: 0.1054360, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 105 of 500. Elapsed time 16.80 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.95 sec. MSE loss 1.22E+00. Sample per second: 64331\n",
      "FUNK_SVD: Epoch 106 of 500. Elapsed time 16.96 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.25 sec. MSE loss 1.22E+00. Sample per second: 62451\n",
      "FUNK_SVD: Epoch 107 of 500. Elapsed time 17.12 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.67 sec. MSE loss 1.21E+00. Sample per second: 66170\n",
      "FUNK_SVD: Epoch 108 of 500. Elapsed time 17.28 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 8.22 sec. MSE loss 1.21E+00. Sample per second: 77928\n",
      "FUNK_SVD: Epoch 109 of 500. Elapsed time 17.40 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.77 sec. MSE loss 1.21E+00. Sample per second: 65523\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.92 sec. Users per second: 1225\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0150896, PRECISION_RECALL_MIN_DEN: 0.0165371, RECALL: 0.0071217, MAP: 0.0053714, MAP_MIN_DEN: 0.0060011, MRR: 0.0480653, NDCG: 0.0129074, F1: 0.0096764, HIT_RATE: 0.1341301, ARHR_ALL_HITS: 0.0506944, NOVELTY: 0.0310363, AVERAGE_POPULARITY: 0.1283558, DIVERSITY_MEAN_INTER_LIST: 0.9938632, DIVERSITY_HERFINDAHL: 0.9993698, COVERAGE_ITEM: 0.9969096, COVERAGE_ITEM_CORRECT: 0.1001803, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.1337748, DIVERSITY_GINI: 0.6046448, SHANNON_ENTROPY: 11.4610822, RATIO_DIVERSITY_HERFINDAHL: 1.0001863, RATIO_DIVERSITY_GINI: 1.7293044, RATIO_SHANNON_ENTROPY: 1.0599481, RATIO_AVERAGE_POPULARITY: 0.4646939, RATIO_NOVELTY: 0.1051628, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 110 of 500. Elapsed time 17.64 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.95 sec. MSE loss 1.22E+00. Sample per second: 64346\n",
      "FUNK_SVD: Epoch 111 of 500. Elapsed time 17.80 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.13 sec. MSE loss 1.21E+00. Sample per second: 63184\n",
      "FUNK_SVD: Epoch 112 of 500. Elapsed time 17.95 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.56 sec. MSE loss 1.21E+00. Sample per second: 66945\n",
      "FUNK_SVD: Epoch 113 of 500. Elapsed time 18.11 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.93 sec. MSE loss 1.22E+00. Sample per second: 64469\n",
      "FUNK_SVD: Epoch 114 of 500. Elapsed time 18.26 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.28 sec. MSE loss 1.22E+00. Sample per second: 62276\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.72 sec. Users per second: 1277\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0158201, PRECISION_RECALL_MIN_DEN: 0.0173251, RECALL: 0.0074851, MAP: 0.0056847, MAP_MIN_DEN: 0.0063368, MRR: 0.0508117, NDCG: 0.0137531, F1: 0.0101621, HIT_RATE: 0.1402722, ARHR_ALL_HITS: 0.0536300, NOVELTY: 0.0309676, AVERAGE_POPULARITY: 0.1329960, DIVERSITY_MEAN_INTER_LIST: 0.9933382, DIVERSITY_HERFINDAHL: 0.9993173, COVERAGE_ITEM: 0.9966521, COVERAGE_ITEM_CORRECT: 0.1004378, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.1399007, DIVERSITY_GINI: 0.5980245, SHANNON_ENTROPY: 11.4342295, RATIO_DIVERSITY_HERFINDAHL: 1.0001337, RATIO_DIVERSITY_GINI: 1.7103700, RATIO_SHANNON_ENTROPY: 1.0574647, RATIO_AVERAGE_POPULARITY: 0.4814929, RATIO_NOVELTY: 0.1049301, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 115 of 500. Elapsed time 18.50 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.48 sec. MSE loss 1.21E+00. Sample per second: 67481\n",
      "FUNK_SVD: Epoch 116 of 500. Elapsed time 18.66 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.80 sec. MSE loss 1.21E+00. Sample per second: 65304\n",
      "FUNK_SVD: Epoch 117 of 500. Elapsed time 18.81 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.95 sec. MSE loss 1.22E+00. Sample per second: 64328\n",
      "FUNK_SVD: Epoch 118 of 500. Elapsed time 18.96 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.25 sec. MSE loss 1.21E+00. Sample per second: 62430\n",
      "FUNK_SVD: Epoch 119 of 500. Elapsed time 19.12 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.40 sec. MSE loss 1.22E+00. Sample per second: 68108\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.78 sec. Users per second: 1261\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0166335, PRECISION_RECALL_MIN_DEN: 0.0182111, RECALL: 0.0079351, MAP: 0.0059701, MAP_MIN_DEN: 0.0066451, MRR: 0.0531837, NDCG: 0.0145659, F1: 0.0107445, HIT_RATE: 0.1472444, ARHR_ALL_HITS: 0.0562264, NOVELTY: 0.0308898, AVERAGE_POPULARITY: 0.1377053, DIVERSITY_MEAN_INTER_LIST: 0.9927618, DIVERSITY_HERFINDAHL: 0.9992597, COVERAGE_ITEM: 0.9963945, COVERAGE_ITEM_CORRECT: 0.1004378, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.1468543, DIVERSITY_GINI: 0.5918456, SHANNON_ENTROPY: 11.4067113, RATIO_DIVERSITY_HERFINDAHL: 1.0000761, RATIO_DIVERSITY_GINI: 1.6926982, RATIO_SHANNON_ENTROPY: 1.0549197, RATIO_AVERAGE_POPULARITY: 0.4985425, RATIO_NOVELTY: 0.1046663, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 120 of 500. Elapsed time 19.35 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 8.54 sec. MSE loss 1.22E+00. Sample per second: 74970\n",
      "FUNK_SVD: Epoch 121 of 500. Elapsed time 19.49 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.11 sec. MSE loss 1.21E+00. Sample per second: 104787\n",
      "FUNK_SVD: Epoch 122 of 500. Elapsed time 19.58 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.30 sec. MSE loss 1.21E+00. Sample per second: 101551\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Epoch 123 of 500. Elapsed time 19.69 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.52 sec. MSE loss 1.21E+00. Sample per second: 98236\n",
      "FUNK_SVD: Epoch 124 of 500. Elapsed time 19.79 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.39 sec. MSE loss 1.21E+00. Sample per second: 100096\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.45 sec. Users per second: 1747\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0172643, PRECISION_RECALL_MIN_DEN: 0.0189638, RECALL: 0.0083705, MAP: 0.0062853, MAP_MIN_DEN: 0.0070220, MRR: 0.0559150, NDCG: 0.0154109, F1: 0.0112746, HIT_RATE: 0.1523904, ARHR_ALL_HITS: 0.0591447, NOVELTY: 0.0308105, AVERAGE_POPULARITY: 0.1429024, DIVERSITY_MEAN_INTER_LIST: 0.9920456, DIVERSITY_HERFINDAHL: 0.9991881, COVERAGE_ITEM: 0.9958795, COVERAGE_ITEM_CORRECT: 0.0994077, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.1519868, DIVERSITY_GINI: 0.5851221, SHANNON_ENTROPY: 11.3752785, RATIO_DIVERSITY_HERFINDAHL: 1.0000044, RATIO_DIVERSITY_GINI: 1.6734688, RATIO_SHANNON_ENTROPY: 1.0520127, RATIO_AVERAGE_POPULARITY: 0.5173577, RATIO_NOVELTY: 0.1043976, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 125 of 500. Elapsed time 19.95 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 7.04 sec. MSE loss 1.22E+00. Sample per second: 90984\n",
      "FUNK_SVD: Epoch 126 of 500. Elapsed time 20.05 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.34 sec. MSE loss 1.21E+00. Sample per second: 100993\n",
      "FUNK_SVD: Epoch 127 of 500. Elapsed time 20.15 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.37 sec. MSE loss 1.21E+00. Sample per second: 100434\n",
      "FUNK_SVD: Epoch 128 of 500. Elapsed time 20.25 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.15 sec. MSE loss 1.21E+00. Sample per second: 104041\n",
      "FUNK_SVD: Epoch 129 of 500. Elapsed time 20.35 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.40 sec. MSE loss 1.21E+00. Sample per second: 100027\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.48 sec. Users per second: 1730\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0178785, PRECISION_RECALL_MIN_DEN: 0.0196986, RECALL: 0.0087688, MAP: 0.0065512, MAP_MIN_DEN: 0.0073270, MRR: 0.0579801, NDCG: 0.0160742, F1: 0.0117666, HIT_RATE: 0.1572045, ARHR_ALL_HITS: 0.0614856, NOVELTY: 0.0307396, AVERAGE_POPULARITY: 0.1479403, DIVERSITY_MEAN_INTER_LIST: 0.9912994, DIVERSITY_HERFINDAHL: 0.9991135, COVERAGE_ITEM: 0.9948493, COVERAGE_ITEM_CORRECT: 0.0996652, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.1567881, DIVERSITY_GINI: 0.5783786, SHANNON_ENTROPY: 11.3434728, RATIO_DIVERSITY_HERFINDAHL: 0.9999297, RATIO_DIVERSITY_GINI: 1.6541822, RATIO_SHANNON_ENTROPY: 1.0490713, RATIO_AVERAGE_POPULARITY: 0.5355969, RATIO_NOVELTY: 0.1041576, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 130 of 500. Elapsed time 20.51 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 7.00 sec. MSE loss 1.21E+00. Sample per second: 91361\n",
      "FUNK_SVD: Epoch 131 of 500. Elapsed time 20.61 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 5.87 sec. MSE loss 1.21E+00. Sample per second: 109113\n",
      "FUNK_SVD: Epoch 132 of 500. Elapsed time 20.71 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 7.24 sec. MSE loss 1.21E+00. Sample per second: 88371\n",
      "FUNK_SVD: Epoch 133 of 500. Elapsed time 20.82 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.52 sec. MSE loss 1.21E+00. Sample per second: 98235\n",
      "FUNK_SVD: Epoch 134 of 500. Elapsed time 20.92 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.37 sec. MSE loss 1.21E+00. Sample per second: 100480\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.21 sec. Users per second: 1879\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0186753, PRECISION_RECALL_MIN_DEN: 0.0205671, RECALL: 0.0091908, MAP: 0.0068986, MAP_MIN_DEN: 0.0077096, MRR: 0.0609182, NDCG: 0.0169699, F1: 0.0123190, HIT_RATE: 0.1638446, ARHR_ALL_HITS: 0.0646904, NOVELTY: 0.0306549, AVERAGE_POPULARITY: 0.1533945, DIVERSITY_MEAN_INTER_LIST: 0.9904295, DIVERSITY_HERFINDAHL: 0.9990265, COVERAGE_ITEM: 0.9943343, COVERAGE_ITEM_CORRECT: 0.0991501, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.1634106, DIVERSITY_GINI: 0.5714852, SHANNON_ENTROPY: 11.3093566, RATIO_DIVERSITY_HERFINDAHL: 0.9998427, RATIO_DIVERSITY_GINI: 1.6344669, RATIO_SHANNON_ENTROPY: 1.0459161, RATIO_AVERAGE_POPULARITY: 0.5553429, RATIO_NOVELTY: 0.1038704, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 135 of 500. Elapsed time 21.07 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.43 sec. MSE loss 1.21E+00. Sample per second: 99620\n",
      "FUNK_SVD: Epoch 136 of 500. Elapsed time 21.17 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 4.91 sec. MSE loss 1.21E+00. Sample per second: 130269\n",
      "FUNK_SVD: Epoch 137 of 500. Elapsed time 21.25 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 7.24 sec. MSE loss 1.21E+00. Sample per second: 88406\n",
      "FUNK_SVD: Epoch 138 of 500. Elapsed time 21.35 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.44 sec. MSE loss 1.21E+00. Sample per second: 67795\n",
      "FUNK_SVD: Epoch 139 of 500. Elapsed time 21.50 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.63 sec. MSE loss 1.21E+00. Sample per second: 66500\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.61 sec. Users per second: 1307\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0195717, PRECISION_RECALL_MIN_DEN: 0.0216574, RECALL: 0.0097231, MAP: 0.0073253, MAP_MIN_DEN: 0.0081772, MRR: 0.0642113, NDCG: 0.0180130, F1: 0.0129919, HIT_RATE: 0.1698207, ARHR_ALL_HITS: 0.0684310, NOVELTY: 0.0305566, AVERAGE_POPULARITY: 0.1593287, DIVERSITY_MEAN_INTER_LIST: 0.9893969, DIVERSITY_HERFINDAHL: 0.9989233, COVERAGE_ITEM: 0.9938192, COVERAGE_ITEM_CORRECT: 0.0988926, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.1693709, DIVERSITY_GINI: 0.5638308, SHANNON_ENTROPY: 11.2706302, RATIO_DIVERSITY_HERFINDAHL: 0.9997393, RATIO_DIVERSITY_GINI: 1.6125749, RATIO_SHANNON_ENTROPY: 1.0423346, RATIO_AVERAGE_POPULARITY: 0.5768270, RATIO_NOVELTY: 0.1035373, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 140 of 500. Elapsed time 21.74 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.74 sec. MSE loss 1.21E+00. Sample per second: 65715\n",
      "FUNK_SVD: Epoch 141 of 500. Elapsed time 21.89 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.02 sec. MSE loss 1.21E+00. Sample per second: 63872\n",
      "FUNK_SVD: Epoch 142 of 500. Elapsed time 22.05 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.32 sec. MSE loss 1.21E+00. Sample per second: 68732\n",
      "FUNK_SVD: Epoch 143 of 500. Elapsed time 22.20 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.51 sec. MSE loss 1.21E+00. Sample per second: 67315\n",
      "FUNK_SVD: Epoch 144 of 500. Elapsed time 22.36 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.58 sec. MSE loss 1.21E+00. Sample per second: 66857\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.60 sec. Users per second: 1310\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0203685, PRECISION_RECALL_MIN_DEN: 0.0225570, RECALL: 0.0102087, MAP: 0.0077865, MAP_MIN_DEN: 0.0087008, MRR: 0.0678232, NDCG: 0.0190791, F1: 0.0136007, HIT_RATE: 0.1754648, ARHR_ALL_HITS: 0.0725305, NOVELTY: 0.0304702, AVERAGE_POPULARITY: 0.1650146, DIVERSITY_MEAN_INTER_LIST: 0.9883166, DIVERSITY_HERFINDAHL: 0.9988153, COVERAGE_ITEM: 0.9938192, COVERAGE_ITEM_CORRECT: 0.0981200, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.1750000, DIVERSITY_GINI: 0.5563133, SHANNON_ENTROPY: 11.2320766, RATIO_DIVERSITY_HERFINDAHL: 0.9996312, RATIO_DIVERSITY_GINI: 1.5910746, RATIO_SHANNON_ENTROPY: 1.0387691, RATIO_AVERAGE_POPULARITY: 0.5974116, RATIO_NOVELTY: 0.1032445, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 145 of 500. Elapsed time 22.58 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.33 sec. MSE loss 1.21E+00. Sample per second: 68604\n",
      "FUNK_SVD: Epoch 146 of 500. Elapsed time 22.74 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.70 sec. MSE loss 1.20E+00. Sample per second: 66017\n",
      "FUNK_SVD: Epoch 147 of 500. Elapsed time 22.89 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.01 sec. MSE loss 1.21E+00. Sample per second: 63956\n",
      "FUNK_SVD: Epoch 148 of 500. Elapsed time 23.05 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 8.97 sec. MSE loss 1.21E+00. Sample per second: 71402\n",
      "FUNK_SVD: Epoch 149 of 500. Elapsed time 23.20 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.28 sec. MSE loss 1.20E+00. Sample per second: 62282\n",
      "FUNK_SVD: Validation begins...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.32 sec. Users per second: 1395\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0211155, PRECISION_RECALL_MIN_DEN: 0.0233704, RECALL: 0.0105701, MAP: 0.0082179, MAP_MIN_DEN: 0.0091919, MRR: 0.0712811, NDCG: 0.0201063, F1: 0.0140880, HIT_RATE: 0.1807769, ARHR_ALL_HITS: 0.0764042, NOVELTY: 0.0303728, AVERAGE_POPULARITY: 0.1709661, DIVERSITY_MEAN_INTER_LIST: 0.9871562, DIVERSITY_HERFINDAHL: 0.9986992, COVERAGE_ITEM: 0.9935617, COVERAGE_ITEM_CORRECT: 0.0978625, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.1802980, DIVERSITY_GINI: 0.5488731, SHANNON_ENTROPY: 11.1916019, RATIO_DIVERSITY_HERFINDAHL: 0.9995151, RATIO_DIVERSITY_GINI: 1.5697955, RATIO_SHANNON_ENTROPY: 1.0350259, RATIO_AVERAGE_POPULARITY: 0.6189584, RATIO_NOVELTY: 0.1029147, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 150 of 500. Elapsed time 23.42 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 7.74 sec. MSE loss 1.20E+00. Sample per second: 82635\n",
      "FUNK_SVD: Epoch 151 of 500. Elapsed time 23.54 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.10 sec. MSE loss 1.21E+00. Sample per second: 63367\n",
      "FUNK_SVD: Epoch 152 of 500. Elapsed time 23.70 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.26 sec. MSE loss 1.21E+00. Sample per second: 69134\n",
      "FUNK_SVD: Epoch 153 of 500. Elapsed time 23.85 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.29 sec. MSE loss 1.21E+00. Sample per second: 68889\n",
      "FUNK_SVD: Epoch 154 of 500. Elapsed time 24.00 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.48 sec. MSE loss 1.21E+00. Sample per second: 67502\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.50 sec. Users per second: 1340\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0218459, PRECISION_RECALL_MIN_DEN: 0.0241904, RECALL: 0.0109924, MAP: 0.0085475, MAP_MIN_DEN: 0.0095583, MRR: 0.0737446, NDCG: 0.0209950, F1: 0.0146256, HIT_RATE: 0.1859230, ARHR_ALL_HITS: 0.0792502, NOVELTY: 0.0302699, AVERAGE_POPULARITY: 0.1770371, DIVERSITY_MEAN_INTER_LIST: 0.9859224, DIVERSITY_HERFINDAHL: 0.9985759, COVERAGE_ITEM: 0.9930466, COVERAGE_ITEM_CORRECT: 0.0976049, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.1854305, DIVERSITY_GINI: 0.5408248, SHANNON_ENTROPY: 11.1487162, RATIO_DIVERSITY_HERFINDAHL: 0.9993917, RATIO_DIVERSITY_GINI: 1.5467772, RATIO_SHANNON_ENTROPY: 1.0310597, RATIO_AVERAGE_POPULARITY: 0.6409377, RATIO_NOVELTY: 0.1025659, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 155 of 500. Elapsed time 24.23 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.30 sec. MSE loss 1.21E+00. Sample per second: 68878\n",
      "FUNK_SVD: Epoch 156 of 500. Elapsed time 24.39 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.61 sec. MSE loss 1.21E+00. Sample per second: 66635\n",
      "FUNK_SVD: Epoch 157 of 500. Elapsed time 24.54 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.72 sec. MSE loss 1.20E+00. Sample per second: 65881\n",
      "FUNK_SVD: Epoch 158 of 500. Elapsed time 24.69 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.95 sec. MSE loss 1.21E+00. Sample per second: 64335\n",
      "FUNK_SVD: Epoch 159 of 500. Elapsed time 24.85 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.29 sec. MSE loss 1.20E+00. Sample per second: 62185\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.66 sec. Users per second: 1294\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0227922, PRECISION_RECALL_MIN_DEN: 0.0252425, RECALL: 0.0115053, MAP: 0.0090207, MAP_MIN_DEN: 0.0100851, MRR: 0.0773846, NDCG: 0.0221218, F1: 0.0152916, HIT_RATE: 0.1928951, ARHR_ALL_HITS: 0.0833914, NOVELTY: 0.0301716, AVERAGE_POPULARITY: 0.1831927, DIVERSITY_MEAN_INTER_LIST: 0.9845460, DIVERSITY_HERFINDAHL: 0.9984383, COVERAGE_ITEM: 0.9925315, COVERAGE_ITEM_CORRECT: 0.0973474, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.1923841, DIVERSITY_GINI: 0.5325635, SHANNON_ENTROPY: 11.1034252, RATIO_DIVERSITY_HERFINDAHL: 0.9992539, RATIO_DIVERSITY_GINI: 1.5231495, RATIO_SHANNON_ENTROPY: 1.0268711, RATIO_AVERAGE_POPULARITY: 0.6632229, RATIO_NOVELTY: 0.1022328, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 160 of 500. Elapsed time 25.08 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.92 sec. MSE loss 1.20E+00. Sample per second: 64545\n",
      "FUNK_SVD: Epoch 161 of 500. Elapsed time 25.23 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.99 sec. MSE loss 1.20E+00. Sample per second: 64106\n",
      "FUNK_SVD: Epoch 162 of 500. Elapsed time 25.38 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 8.86 sec. MSE loss 1.20E+00. Sample per second: 72224\n",
      "FUNK_SVD: Epoch 163 of 500. Elapsed time 25.51 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.75 sec. MSE loss 1.20E+00. Sample per second: 65659\n",
      "FUNK_SVD: Epoch 164 of 500. Elapsed time 25.66 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.80 sec. MSE loss 1.21E+00. Sample per second: 65354\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.69 sec. Users per second: 1286\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0235392, PRECISION_RECALL_MIN_DEN: 0.0262555, RECALL: 0.0120659, MAP: 0.0095018, MAP_MIN_DEN: 0.0106403, MRR: 0.0812141, NDCG: 0.0232458, F1: 0.0159540, HIT_RATE: 0.1980412, ARHR_ALL_HITS: 0.0877131, NOVELTY: 0.0300692, AVERAGE_POPULARITY: 0.1895551, DIVERSITY_MEAN_INTER_LIST: 0.9830268, DIVERSITY_HERFINDAHL: 0.9982864, COVERAGE_ITEM: 0.9925315, COVERAGE_ITEM_CORRECT: 0.0968323, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.1975166, DIVERSITY_GINI: 0.5241568, SHANNON_ENTROPY: 11.0561470, RATIO_DIVERSITY_HERFINDAHL: 0.9991019, RATIO_DIVERSITY_GINI: 1.4991061, RATIO_SHANNON_ENTROPY: 1.0224987, RATIO_AVERAGE_POPULARITY: 0.6862572, RATIO_NOVELTY: 0.1018859, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 165 of 500. Elapsed time 25.89 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.79 sec. MSE loss 1.20E+00. Sample per second: 65359\n",
      "FUNK_SVD: Epoch 166 of 500. Elapsed time 26.04 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.25 sec. MSE loss 1.20E+00. Sample per second: 62470\n",
      "FUNK_SVD: Epoch 167 of 500. Elapsed time 26.20 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.24 sec. MSE loss 1.21E+00. Sample per second: 69292\n",
      "FUNK_SVD: Epoch 168 of 500. Elapsed time 26.35 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.63 sec. MSE loss 1.20E+00. Sample per second: 66417\n",
      "FUNK_SVD: Epoch 169 of 500. Elapsed time 26.51 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.93 sec. MSE loss 1.20E+00. Sample per second: 64463\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.52 sec. Users per second: 1332\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0243194, PRECISION_RECALL_MIN_DEN: 0.0271669, RECALL: 0.0125882, MAP: 0.0099133, MAP_MIN_DEN: 0.0111003, MRR: 0.0843233, NDCG: 0.0243030, F1: 0.0165894, HIT_RATE: 0.2038513, ARHR_ALL_HITS: 0.0912559, NOVELTY: 0.0299724, AVERAGE_POPULARITY: 0.1956156, DIVERSITY_MEAN_INTER_LIST: 0.9814070, DIVERSITY_HERFINDAHL: 0.9981244, COVERAGE_ITEM: 0.9920165, COVERAGE_ITEM_CORRECT: 0.0960597, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.2033113, DIVERSITY_GINI: 0.5160349, SHANNON_ENTROPY: 11.0094067, RATIO_DIVERSITY_HERFINDAHL: 0.9989398, RATIO_DIVERSITY_GINI: 1.4758771, RATIO_SHANNON_ENTROPY: 1.0181760, RATIO_AVERAGE_POPULARITY: 0.7081983, RATIO_NOVELTY: 0.1015578, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 170 of 500. Elapsed time 26.74 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.71 sec. MSE loss 1.20E+00. Sample per second: 65943\n",
      "FUNK_SVD: Epoch 171 of 500. Elapsed time 26.89 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.44 sec. MSE loss 1.21E+00. Sample per second: 61355\n",
      "FUNK_SVD: Epoch 172 of 500. Elapsed time 27.05 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.71 sec. MSE loss 1.20E+00. Sample per second: 65938\n",
      "FUNK_SVD: Epoch 173 of 500. Elapsed time 27.21 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.72 sec. MSE loss 1.20E+00. Sample per second: 65839\n",
      "FUNK_SVD: Epoch 174 of 500. Elapsed time 27.36 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.14 sec. MSE loss 1.21E+00. Sample per second: 63135\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.59 sec. Users per second: 1680\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0253652, PRECISION_RECALL_MIN_DEN: 0.0282979, RECALL: 0.0131097, MAP: 0.0104013, MAP_MIN_DEN: 0.0116276, MRR: 0.0882076, NDCG: 0.0254771, F1: 0.0172856, HIT_RATE: 0.2111554, ARHR_ALL_HITS: 0.0955967, NOVELTY: 0.0298634, AVERAGE_POPULARITY: 0.2024528, DIVERSITY_MEAN_INTER_LIST: 0.9796238, DIVERSITY_HERFINDAHL: 0.9979461, COVERAGE_ITEM: 0.9907288, COVERAGE_ITEM_CORRECT: 0.0963173, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.2105960, DIVERSITY_GINI: 0.5071169, SHANNON_ENTROPY: 10.9565060, RATIO_DIVERSITY_HERFINDAHL: 0.9987614, RATIO_DIVERSITY_GINI: 1.4503714, RATIO_SHANNON_ENTROPY: 1.0132837, RATIO_AVERAGE_POPULARITY: 0.7329515, RATIO_NOVELTY: 0.1011887, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 175 of 500. Elapsed time 27.58 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Processed 640256 (100.0%) in 9.64 sec. MSE loss 1.20E+00. Sample per second: 66446\n",
      "FUNK_SVD: Epoch 176 of 500. Elapsed time 27.72 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.56 sec. MSE loss 1.20E+00. Sample per second: 66996\n",
      "FUNK_SVD: Epoch 177 of 500. Elapsed time 27.87 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.63 sec. MSE loss 1.20E+00. Sample per second: 66444\n",
      "FUNK_SVD: Epoch 178 of 500. Elapsed time 28.02 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.15 sec. MSE loss 1.20E+00. Sample per second: 63103\n",
      "FUNK_SVD: Epoch 179 of 500. Elapsed time 28.18 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.37 sec. MSE loss 1.20E+00. Sample per second: 68313\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.62 sec. Users per second: 1305\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0262616, PRECISION_RECALL_MIN_DEN: 0.0293764, RECALL: 0.0136763, MAP: 0.0108567, MAP_MIN_DEN: 0.0121548, MRR: 0.0914941, NDCG: 0.0266015, F1: 0.0179860, HIT_RATE: 0.2172975, ARHR_ALL_HITS: 0.0994776, NOVELTY: 0.0297554, AVERAGE_POPULARITY: 0.2095415, DIVERSITY_MEAN_INTER_LIST: 0.9775572, DIVERSITY_HERFINDAHL: 0.9977395, COVERAGE_ITEM: 0.9896987, COVERAGE_ITEM_CORRECT: 0.0960597, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.2167219, DIVERSITY_GINI: 0.4979033, SHANNON_ENTROPY: 10.9000505, RATIO_DIVERSITY_HERFINDAHL: 0.9985546, RATIO_DIVERSITY_GINI: 1.4240201, RATIO_SHANNON_ENTROPY: 1.0080625, RATIO_AVERAGE_POPULARITY: 0.7586153, RATIO_NOVELTY: 0.1008228, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 180 of 500. Elapsed time 28.41 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.24 sec. MSE loss 1.20E+00. Sample per second: 69282\n",
      "FUNK_SVD: Epoch 181 of 500. Elapsed time 28.57 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.88 sec. MSE loss 1.20E+00. Sample per second: 64821\n",
      "FUNK_SVD: Epoch 182 of 500. Elapsed time 28.73 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.03 sec. MSE loss 1.20E+00. Sample per second: 63847\n",
      "FUNK_SVD: Epoch 183 of 500. Elapsed time 28.88 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 8.99 sec. MSE loss 1.20E+00. Sample per second: 71172\n",
      "FUNK_SVD: Epoch 184 of 500. Elapsed time 29.03 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.16 sec. MSE loss 1.20E+00. Sample per second: 63015\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.49 sec. Users per second: 1341\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0272908, PRECISION_RECALL_MIN_DEN: 0.0305121, RECALL: 0.0142571, MAP: 0.0113425, MAP_MIN_DEN: 0.0127033, MRR: 0.0948924, NDCG: 0.0276575, F1: 0.0187296, HIT_RATE: 0.2234396, ARHR_ALL_HITS: 0.1035590, NOVELTY: 0.0296407, AVERAGE_POPULARITY: 0.2163199, DIVERSITY_MEAN_INTER_LIST: 0.9756175, DIVERSITY_HERFINDAHL: 0.9975455, COVERAGE_ITEM: 0.9896987, COVERAGE_ITEM_CORRECT: 0.0950296, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.2228477, DIVERSITY_GINI: 0.4886859, SHANNON_ENTROPY: 10.8455829, RATIO_DIVERSITY_HERFINDAHL: 0.9983605, RATIO_DIVERSITY_GINI: 1.3976579, RATIO_SHANNON_ENTROPY: 1.0030252, RATIO_AVERAGE_POPULARITY: 0.7831553, RATIO_NOVELTY: 0.1004341, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 185 of 500. Elapsed time 29.26 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.03 sec. MSE loss 1.20E+00. Sample per second: 63843\n",
      "FUNK_SVD: Epoch 186 of 500. Elapsed time 29.41 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.15 sec. MSE loss 1.20E+00. Sample per second: 70000\n",
      "FUNK_SVD: Epoch 187 of 500. Elapsed time 29.57 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 7.41 sec. MSE loss 1.20E+00. Sample per second: 86408\n",
      "FUNK_SVD: Epoch 188 of 500. Elapsed time 29.69 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.74 sec. MSE loss 1.20E+00. Sample per second: 65735\n",
      "FUNK_SVD: Epoch 189 of 500. Elapsed time 29.84 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.06 sec. MSE loss 1.20E+00. Sample per second: 63638\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.58 sec. Users per second: 1314\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0280378, PRECISION_RECALL_MIN_DEN: 0.0314307, RECALL: 0.0147774, MAP: 0.0118083, MAP_MIN_DEN: 0.0132447, MRR: 0.0983432, NDCG: 0.0287412, F1: 0.0193541, HIT_RATE: 0.2292497, ARHR_ALL_HITS: 0.1075987, NOVELTY: 0.0295478, AVERAGE_POPULARITY: 0.2230431, DIVERSITY_MEAN_INTER_LIST: 0.9735188, DIVERSITY_HERFINDAHL: 0.9973357, COVERAGE_ITEM: 0.9889261, COVERAGE_ITEM_CORRECT: 0.0945146, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.2286424, DIVERSITY_GINI: 0.4803537, SHANNON_ENTROPY: 10.7909421, RATIO_DIVERSITY_HERFINDAHL: 0.9981505, RATIO_DIVERSITY_GINI: 1.3738277, RATIO_SHANNON_ENTROPY: 0.9979719, RATIO_AVERAGE_POPULARITY: 0.8074955, RATIO_NOVELTY: 0.1001192, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 190 of 500. Elapsed time 30.08 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.56 sec. MSE loss 1.20E+00. Sample per second: 66940\n",
      "FUNK_SVD: Epoch 191 of 500. Elapsed time 30.22 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.73 sec. MSE loss 1.20E+00. Sample per second: 65809\n",
      "FUNK_SVD: Epoch 192 of 500. Elapsed time 30.38 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.11 sec. MSE loss 1.20E+00. Sample per second: 63323\n",
      "FUNK_SVD: Epoch 193 of 500. Elapsed time 30.53 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.23 sec. MSE loss 1.20E+00. Sample per second: 69378\n",
      "FUNK_SVD: Epoch 194 of 500. Elapsed time 30.68 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.18 sec. MSE loss 1.19E+00. Sample per second: 69716\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.73 sec. Users per second: 1273\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0289841, PRECISION_RECALL_MIN_DEN: 0.0325162, RECALL: 0.0153593, MAP: 0.0123751, MAP_MIN_DEN: 0.0138536, MRR: 0.1026188, NDCG: 0.0299833, F1: 0.0200786, HIT_RATE: 0.2357238, ARHR_ALL_HITS: 0.1125294, NOVELTY: 0.0294338, AVERAGE_POPULARITY: 0.2302276, DIVERSITY_MEAN_INTER_LIST: 0.9712067, DIVERSITY_HERFINDAHL: 0.9971045, COVERAGE_ITEM: 0.9881535, COVERAGE_ITEM_CORRECT: 0.0937420, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.2350993, DIVERSITY_GINI: 0.4713524, SHANNON_ENTROPY: 10.7323092, RATIO_DIVERSITY_HERFINDAHL: 0.9979191, RATIO_DIVERSITY_GINI: 1.3480836, RATIO_SHANNON_ENTROPY: 0.9925494, RATIO_AVERAGE_POPULARITY: 0.8335060, RATIO_NOVELTY: 0.0997330, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 195 of 500. Elapsed time 30.91 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.07 sec. MSE loss 1.20E+00. Sample per second: 63568\n",
      "FUNK_SVD: Epoch 196 of 500. Elapsed time 31.07 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.46 sec. MSE loss 1.20E+00. Sample per second: 67688\n",
      "FUNK_SVD: Epoch 197 of 500. Elapsed time 31.22 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.58 sec. MSE loss 1.20E+00. Sample per second: 66851\n",
      "FUNK_SVD: Epoch 198 of 500. Elapsed time 31.37 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.52 sec. MSE loss 1.20E+00. Sample per second: 67257\n",
      "FUNK_SVD: Epoch 199 of 500. Elapsed time 31.52 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 8.70 sec. MSE loss 1.20E+00. Sample per second: 73628\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.23 sec. Users per second: 1865\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0301129, PRECISION_RECALL_MIN_DEN: 0.0338508, RECALL: 0.0161361, MAP: 0.0129730, MAP_MIN_DEN: 0.0145361, MRR: 0.1070315, NDCG: 0.0314718, F1: 0.0210125, HIT_RATE: 0.2445219, ARHR_ALL_HITS: 0.1176169, NOVELTY: 0.0293198, AVERAGE_POPULARITY: 0.2375022, DIVERSITY_MEAN_INTER_LIST: 0.9687067, DIVERSITY_HERFINDAHL: 0.9968546, COVERAGE_ITEM: 0.9863508, COVERAGE_ITEM_CORRECT: 0.0932269, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.2438742, DIVERSITY_GINI: 0.4619230, SHANNON_ENTROPY: 10.6707444, RATIO_DIVERSITY_HERFINDAHL: 0.9976690, RATIO_DIVERSITY_GINI: 1.3211152, RATIO_SHANNON_ENTROPY: 0.9868558, RATIO_AVERAGE_POPULARITY: 0.8598429, RATIO_NOVELTY: 0.0993467, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 200 of 500. Elapsed time 31.71 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.46 sec. MSE loss 1.20E+00. Sample per second: 99159\n",
      "FUNK_SVD: Epoch 201 of 500. Elapsed time 31.81 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.62 sec. MSE loss 1.20E+00. Sample per second: 96695\n",
      "FUNK_SVD: Epoch 202 of 500. Elapsed time 31.91 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.64 sec. MSE loss 1.20E+00. Sample per second: 96364\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Epoch 203 of 500. Elapsed time 32.01 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.41 sec. MSE loss 1.20E+00. Sample per second: 99853\n",
      "FUNK_SVD: Epoch 204 of 500. Elapsed time 32.10 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.01 sec. MSE loss 1.19E+00. Sample per second: 106528\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.34 sec. Users per second: 1801\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0311421, PRECISION_RECALL_MIN_DEN: 0.0350174, RECALL: 0.0167639, MAP: 0.0134718, MAP_MIN_DEN: 0.0151011, MRR: 0.1106827, NDCG: 0.0326634, F1: 0.0217953, HIT_RATE: 0.2516600, ARHR_ALL_HITS: 0.1218831, NOVELTY: 0.0291942, AVERAGE_POPULARITY: 0.2450652, DIVERSITY_MEAN_INTER_LIST: 0.9659987, DIVERSITY_HERFINDAHL: 0.9965838, COVERAGE_ITEM: 0.9860932, COVERAGE_ITEM_CORRECT: 0.0921968, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.2509934, DIVERSITY_GINI: 0.4530622, SHANNON_ENTROPY: 10.6078227, RATIO_DIVERSITY_HERFINDAHL: 0.9973980, RATIO_DIVERSITY_GINI: 1.2957730, RATIO_SHANNON_ENTROPY: 0.9810366, RATIO_AVERAGE_POPULARITY: 0.8872235, RATIO_NOVELTY: 0.0989213, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 205 of 500. Elapsed time 32.25 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.41 sec. MSE loss 1.19E+00. Sample per second: 99843\n",
      "FUNK_SVD: Epoch 206 of 500. Elapsed time 32.35 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.44 sec. MSE loss 1.19E+00. Sample per second: 99410\n",
      "FUNK_SVD: Epoch 207 of 500. Elapsed time 32.45 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 5.80 sec. MSE loss 1.19E+00. Sample per second: 110290\n",
      "FUNK_SVD: Epoch 208 of 500. Elapsed time 32.54 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.92 sec. MSE loss 1.19E+00. Sample per second: 92558\n",
      "FUNK_SVD: Epoch 209 of 500. Elapsed time 32.65 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.93 sec. MSE loss 1.19E+00. Sample per second: 92337\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.37 sec. Users per second: 1786\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0320385, PRECISION_RECALL_MIN_DEN: 0.0363131, RECALL: 0.0176338, MAP: 0.0139783, MAP_MIN_DEN: 0.0157150, MRR: 0.1144460, NDCG: 0.0339994, F1: 0.0227475, HIT_RATE: 0.2579681, ARHR_ALL_HITS: 0.1262382, NOVELTY: 0.0290816, AVERAGE_POPULARITY: 0.2522695, DIVERSITY_MEAN_INTER_LIST: 0.9634351, DIVERSITY_HERFINDAHL: 0.9963275, COVERAGE_ITEM: 0.9850631, COVERAGE_ITEM_CORRECT: 0.0909091, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.2572848, DIVERSITY_GINI: 0.4440637, SHANNON_ENTROPY: 10.5461653, RATIO_DIVERSITY_HERFINDAHL: 0.9971415, RATIO_DIVERSITY_GINI: 1.2700371, RATIO_SHANNON_ENTROPY: 0.9753344, RATIO_AVERAGE_POPULARITY: 0.9133059, RATIO_NOVELTY: 0.0985394, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 210 of 500. Elapsed time 32.80 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 5.81 sec. MSE loss 1.19E+00. Sample per second: 110086\n",
      "FUNK_SVD: Epoch 211 of 500. Elapsed time 32.89 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.68 sec. MSE loss 1.19E+00. Sample per second: 95778\n",
      "FUNK_SVD: Epoch 212 of 500. Elapsed time 32.99 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.71 sec. MSE loss 1.20E+00. Sample per second: 95419\n",
      "FUNK_SVD: Epoch 213 of 500. Elapsed time 33.09 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.80 sec. MSE loss 1.20E+00. Sample per second: 94086\n",
      "FUNK_SVD: Epoch 214 of 500. Elapsed time 33.19 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 5.79 sec. MSE loss 1.20E+00. Sample per second: 110535\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 2.64 sec. Users per second: 2283\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0328353, PRECISION_RECALL_MIN_DEN: 0.0372506, RECALL: 0.0181014, MAP: 0.0144382, MAP_MIN_DEN: 0.0161862, MRR: 0.1174425, NDCG: 0.0348805, F1: 0.0233374, HIT_RATE: 0.2629482, ARHR_ALL_HITS: 0.1299534, NOVELTY: 0.0289631, AVERAGE_POPULARITY: 0.2597045, DIVERSITY_MEAN_INTER_LIST: 0.9605391, DIVERSITY_HERFINDAHL: 0.9960380, COVERAGE_ITEM: 0.9840330, COVERAGE_ITEM_CORRECT: 0.0896214, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.2622517, DIVERSITY_GINI: 0.4346517, SHANNON_ENTROPY: 10.4802281, RATIO_DIVERSITY_HERFINDAHL: 0.9968517, RATIO_DIVERSITY_GINI: 1.2431184, RATIO_SHANNON_ENTROPY: 0.9692364, RATIO_AVERAGE_POPULARITY: 0.9402231, RATIO_NOVELTY: 0.0981380, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 215 of 500. Elapsed time 33.32 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.55 sec. MSE loss 1.19E+00. Sample per second: 97701\n",
      "FUNK_SVD: Epoch 216 of 500. Elapsed time 33.42 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 8.74 sec. MSE loss 1.19E+00. Sample per second: 73223\n",
      "FUNK_SVD: Epoch 217 of 500. Elapsed time 33.56 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.21 sec. MSE loss 1.19E+00. Sample per second: 62717\n",
      "FUNK_SVD: Epoch 218 of 500. Elapsed time 33.72 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.44 sec. MSE loss 1.19E+00. Sample per second: 67842\n",
      "FUNK_SVD: Epoch 219 of 500. Elapsed time 33.87 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.60 sec. MSE loss 1.20E+00. Sample per second: 66696\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.78 sec. Users per second: 1261\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0338479, PRECISION_RECALL_MIN_DEN: 0.0383765, RECALL: 0.0187081, MAP: 0.0149293, MAP_MIN_DEN: 0.0167693, MRR: 0.1206129, NDCG: 0.0361810, F1: 0.0240973, HIT_RATE: 0.2700863, ARHR_ALL_HITS: 0.1339095, NOVELTY: 0.0288399, AVERAGE_POPULARITY: 0.2671290, DIVERSITY_MEAN_INTER_LIST: 0.9573579, DIVERSITY_HERFINDAHL: 0.9957199, COVERAGE_ITEM: 0.9822302, COVERAGE_ITEM_CORRECT: 0.0888488, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.2693709, DIVERSITY_GINI: 0.4246571, SHANNON_ENTROPY: 10.4108453, RATIO_DIVERSITY_HERFINDAHL: 0.9965334, RATIO_DIVERSITY_GINI: 1.2145336, RATIO_SHANNON_ENTROPY: 0.9628197, RATIO_AVERAGE_POPULARITY: 0.9671025, RATIO_NOVELTY: 0.0977205, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 220 of 500. Elapsed time 34.10 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.79 sec. MSE loss 1.19E+00. Sample per second: 65411\n",
      "FUNK_SVD: Epoch 221 of 500. Elapsed time 34.26 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.17 sec. MSE loss 1.19E+00. Sample per second: 62943\n",
      "FUNK_SVD: Epoch 222 of 500. Elapsed time 34.42 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.20 sec. MSE loss 1.19E+00. Sample per second: 69628\n",
      "FUNK_SVD: Epoch 223 of 500. Elapsed time 34.57 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.31 sec. MSE loss 1.19E+00. Sample per second: 68744\n",
      "FUNK_SVD: Epoch 224 of 500. Elapsed time 34.72 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.62 sec. MSE loss 1.19E+00. Sample per second: 66499\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.69 sec. Users per second: 1285\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0354416, PRECISION_RECALL_MIN_DEN: 0.0402807, RECALL: 0.0197647, MAP: 0.0155135, MAP_MIN_DEN: 0.0174817, MRR: 0.1244903, NDCG: 0.0378179, F1: 0.0253772, HIT_RATE: 0.2813745, ARHR_ALL_HITS: 0.1386409, NOVELTY: 0.0287093, AVERAGE_POPULARITY: 0.2752076, DIVERSITY_MEAN_INTER_LIST: 0.9538420, DIVERSITY_HERFINDAHL: 0.9953684, COVERAGE_ITEM: 0.9812001, COVERAGE_ITEM_CORRECT: 0.0883338, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.2806291, DIVERSITY_GINI: 0.4145078, SHANNON_ENTROPY: 10.3370571, RATIO_DIVERSITY_HERFINDAHL: 0.9961815, RATIO_DIVERSITY_GINI: 1.1855063, RATIO_SHANNON_ENTROPY: 0.9559956, RATIO_AVERAGE_POPULARITY: 0.9963499, RATIO_NOVELTY: 0.0972780, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 225 of 500. Elapsed time 34.95 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.59 sec. MSE loss 1.19E+00. Sample per second: 66777\n",
      "FUNK_SVD: Epoch 226 of 500. Elapsed time 35.11 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.69 sec. MSE loss 1.19E+00. Sample per second: 66101\n",
      "FUNK_SVD: Epoch 227 of 500. Elapsed time 35.26 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.89 sec. MSE loss 1.19E+00. Sample per second: 64751\n",
      "FUNK_SVD: Epoch 228 of 500. Elapsed time 35.41 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 8.34 sec. MSE loss 1.19E+00. Sample per second: 76723\n",
      "FUNK_SVD: Epoch 229 of 500. Elapsed time 35.54 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.80 sec. MSE loss 1.19E+00. Sample per second: 65345\n",
      "FUNK_SVD: Validation begins...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.55 sec. Users per second: 1324\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0364708, PRECISION_RECALL_MIN_DEN: 0.0414284, RECALL: 0.0203618, MAP: 0.0160436, MAP_MIN_DEN: 0.0180564, MRR: 0.1280440, NDCG: 0.0390786, F1: 0.0261333, HIT_RATE: 0.2886786, ARHR_ALL_HITS: 0.1430159, NOVELTY: 0.0285949, AVERAGE_POPULARITY: 0.2824691, DIVERSITY_MEAN_INTER_LIST: 0.9507271, DIVERSITY_HERFINDAHL: 0.9950569, COVERAGE_ITEM: 0.9799124, COVERAGE_ITEM_CORRECT: 0.0878187, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.2879139, DIVERSITY_GINI: 0.4051828, SHANNON_ENTROPY: 10.2708056, RATIO_DIVERSITY_HERFINDAHL: 0.9958699, RATIO_DIVERSITY_GINI: 1.1588363, RATIO_SHANNON_ENTROPY: 0.9498685, RATIO_AVERAGE_POPULARITY: 1.0226391, RATIO_NOVELTY: 0.0968903, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 230 of 500. Elapsed time 35.77 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.46 sec. MSE loss 1.19E+00. Sample per second: 67652\n",
      "FUNK_SVD: Epoch 231 of 500. Elapsed time 35.92 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.01 sec. MSE loss 1.19E+00. Sample per second: 63952\n",
      "FUNK_SVD: Epoch 232 of 500. Elapsed time 36.08 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.15 sec. MSE loss 1.19E+00. Sample per second: 69941\n",
      "FUNK_SVD: Epoch 233 of 500. Elapsed time 36.23 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.36 sec. MSE loss 1.19E+00. Sample per second: 68403\n",
      "FUNK_SVD: Epoch 234 of 500. Elapsed time 36.39 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.30 sec. MSE loss 1.19E+00. Sample per second: 68849\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.62 sec. Users per second: 1305\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0371846, PRECISION_RECALL_MIN_DEN: 0.0422355, RECALL: 0.0207891, MAP: 0.0164762, MAP_MIN_DEN: 0.0184887, MRR: 0.1306053, NDCG: 0.0399920, F1: 0.0266685, HIT_RATE: 0.2921647, ARHR_ALL_HITS: 0.1463887, NOVELTY: 0.0284869, AVERAGE_POPULARITY: 0.2898473, DIVERSITY_MEAN_INTER_LIST: 0.9473438, DIVERSITY_HERFINDAHL: 0.9947187, COVERAGE_ITEM: 0.9786248, COVERAGE_ITEM_CORRECT: 0.0873036, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.2913907, DIVERSITY_GINI: 0.3966991, SHANNON_ENTROPY: 10.2030856, RATIO_DIVERSITY_HERFINDAHL: 0.9955313, RATIO_DIVERSITY_GINI: 1.1345727, RATIO_SHANNON_ENTROPY: 0.9436056, RATIO_AVERAGE_POPULARITY: 1.0493508, RATIO_NOVELTY: 0.0965246, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 235 of 500. Elapsed time 36.61 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.44 sec. MSE loss 1.19E+00. Sample per second: 61349\n",
      "FUNK_SVD: Epoch 236 of 500. Elapsed time 36.77 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.74 sec. MSE loss 1.19E+00. Sample per second: 65732\n",
      "FUNK_SVD: Epoch 237 of 500. Elapsed time 36.93 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.94 sec. MSE loss 1.19E+00. Sample per second: 64408\n",
      "FUNK_SVD: Epoch 238 of 500. Elapsed time 37.08 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.94 sec. MSE loss 1.19E+00. Sample per second: 64419\n",
      "FUNK_SVD: Epoch 239 of 500. Elapsed time 37.23 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.97 sec. MSE loss 1.19E+00. Sample per second: 64154\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.55 sec. Users per second: 1324\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0381972, PRECISION_RECALL_MIN_DEN: 0.0433459, RECALL: 0.0212800, MAP: 0.0170599, MAP_MIN_DEN: 0.0191377, MRR: 0.1345816, NDCG: 0.0412900, F1: 0.0273327, HIT_RATE: 0.2984728, ARHR_ALL_HITS: 0.1512048, NOVELTY: 0.0283854, AVERAGE_POPULARITY: 0.2968837, DIVERSITY_MEAN_INTER_LIST: 0.9438971, DIVERSITY_HERFINDAHL: 0.9943740, COVERAGE_ITEM: 0.9768220, COVERAGE_ITEM_CORRECT: 0.0865310, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.2976821, DIVERSITY_GINI: 0.3875595, SHANNON_ENTROPY: 10.1338732, RATIO_DIVERSITY_HERFINDAHL: 0.9951864, RATIO_DIVERSITY_GINI: 1.1084331, RATIO_SHANNON_ENTROPY: 0.9372046, RATIO_AVERAGE_POPULARITY: 1.0748251, RATIO_NOVELTY: 0.0961806, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 240 of 500. Elapsed time 37.46 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 8.47 sec. MSE loss 1.19E+00. Sample per second: 75601\n",
      "FUNK_SVD: Epoch 241 of 500. Elapsed time 37.59 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.28 sec. MSE loss 1.19E+00. Sample per second: 68964\n",
      "FUNK_SVD: Epoch 242 of 500. Elapsed time 37.74 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.39 sec. MSE loss 1.20E+00. Sample per second: 68172\n",
      "FUNK_SVD: Epoch 243 of 500. Elapsed time 37.89 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.52 sec. MSE loss 1.19E+00. Sample per second: 67251\n",
      "FUNK_SVD: Epoch 244 of 500. Elapsed time 38.04 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.74 sec. MSE loss 1.19E+00. Sample per second: 65687\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.62 sec. Users per second: 1305\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0393094, PRECISION_RECALL_MIN_DEN: 0.0445722, RECALL: 0.0218753, MAP: 0.0176265, MAP_MIN_DEN: 0.0198223, MRR: 0.1385359, NDCG: 0.0427154, F1: 0.0281085, HIT_RATE: 0.3054449, ARHR_ALL_HITS: 0.1559277, NOVELTY: 0.0282598, AVERAGE_POPULARITY: 0.3044725, DIVERSITY_MEAN_INTER_LIST: 0.9402907, DIVERSITY_HERFINDAHL: 0.9940135, COVERAGE_ITEM: 0.9760494, COVERAGE_ITEM_CORRECT: 0.0865310, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3046358, DIVERSITY_GINI: 0.3786365, SHANNON_ENTROPY: 10.0620735, RATIO_DIVERSITY_HERFINDAHL: 0.9948255, RATIO_DIVERSITY_GINI: 1.0829131, RATIO_SHANNON_ENTROPY: 0.9305644, RATIO_AVERAGE_POPULARITY: 1.1022994, RATIO_NOVELTY: 0.0957551, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 245 of 500. Elapsed time 38.27 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.62 sec. MSE loss 1.19E+00. Sample per second: 66534\n",
      "FUNK_SVD: Epoch 246 of 500. Elapsed time 38.42 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.69 sec. MSE loss 1.19E+00. Sample per second: 66088\n",
      "FUNK_SVD: Epoch 247 of 500. Elapsed time 38.58 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.64 sec. MSE loss 1.18E+00. Sample per second: 66362\n",
      "FUNK_SVD: Epoch 248 of 500. Elapsed time 38.72 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.72 sec. MSE loss 1.19E+00. Sample per second: 65878\n",
      "FUNK_SVD: Epoch 249 of 500. Elapsed time 38.88 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.32 sec. MSE loss 1.19E+00. Sample per second: 61995\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.61 sec. Users per second: 1307\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0402888, PRECISION_RECALL_MIN_DEN: 0.0457090, RECALL: 0.0224613, MAP: 0.0181830, MAP_MIN_DEN: 0.0204661, MRR: 0.1424255, NDCG: 0.0439754, F1: 0.0288426, HIT_RATE: 0.3109230, ARHR_ALL_HITS: 0.1605945, NOVELTY: 0.0281446, AVERAGE_POPULARITY: 0.3118614, DIVERSITY_MEAN_INTER_LIST: 0.9365145, DIVERSITY_HERFINDAHL: 0.9936359, COVERAGE_ITEM: 0.9745042, COVERAGE_ITEM_CORRECT: 0.0865310, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3100993, DIVERSITY_GINI: 0.3695057, SHANNON_ENTROPY: 9.9898158, RATIO_DIVERSITY_HERFINDAHL: 0.9944477, RATIO_DIVERSITY_GINI: 1.0567986, RATIO_SHANNON_ENTROPY: 0.9238819, RATIO_AVERAGE_POPULARITY: 1.1290496, RATIO_NOVELTY: 0.0953646, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 250 of 500. Elapsed time 39.11 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.06 sec. MSE loss 1.19E+00. Sample per second: 63633\n",
      "FUNK_SVD: Epoch 251 of 500. Elapsed time 39.27 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.05 sec. MSE loss 1.19E+00. Sample per second: 70762\n",
      "FUNK_SVD: Epoch 252 of 500. Elapsed time 39.41 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.41 sec. MSE loss 1.18E+00. Sample per second: 68025\n",
      "FUNK_SVD: Epoch 253 of 500. Elapsed time 39.57 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 7.69 sec. MSE loss 1.19E+00. Sample per second: 83150\n",
      "FUNK_SVD: Epoch 254 of 500. Elapsed time 39.69 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.88 sec. MSE loss 1.19E+00. Sample per second: 64750\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.77 sec. Users per second: 1264\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0414841, PRECISION_RECALL_MIN_DEN: 0.0471564, RECALL: 0.0232470, MAP: 0.0187253, MAP_MIN_DEN: 0.0210656, MRR: 0.1455253, NDCG: 0.0452353, F1: 0.0297965, HIT_RATE: 0.3168991, ARHR_ALL_HITS: 0.1647471, NOVELTY: 0.0280204, AVERAGE_POPULARITY: 0.3192699, DIVERSITY_MEAN_INTER_LIST: 0.9326665, DIVERSITY_HERFINDAHL: 0.9932512, COVERAGE_ITEM: 0.9737317, COVERAGE_ITEM_CORRECT: 0.0862735, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3160596, DIVERSITY_GINI: 0.3605224, SHANNON_ENTROPY: 9.9168843, RATIO_DIVERSITY_HERFINDAHL: 0.9940626, RATIO_DIVERSITY_GINI: 1.0311062, RATIO_SHANNON_ENTROPY: 0.9171370, RATIO_AVERAGE_POPULARITY: 1.1558712, RATIO_NOVELTY: 0.0949437, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 255 of 500. Elapsed time 39.93 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Processed 640256 (100.0%) in 9.95 sec. MSE loss 1.19E+00. Sample per second: 64327\n",
      "FUNK_SVD: Epoch 256 of 500. Elapsed time 40.08 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.28 sec. MSE loss 1.19E+00. Sample per second: 62259\n",
      "FUNK_SVD: Epoch 257 of 500. Elapsed time 40.24 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.57 sec. MSE loss 1.19E+00. Sample per second: 66874\n",
      "FUNK_SVD: Epoch 258 of 500. Elapsed time 40.39 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.58 sec. MSE loss 1.19E+00. Sample per second: 66816\n",
      "FUNK_SVD: Epoch 259 of 500. Elapsed time 40.54 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.77 sec. MSE loss 1.18E+00. Sample per second: 65515\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.51 sec. Users per second: 1334\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0425963, PRECISION_RECALL_MIN_DEN: 0.0486847, RECALL: 0.0242810, MAP: 0.0193130, MAP_MIN_DEN: 0.0217682, MRR: 0.1492216, NDCG: 0.0467004, F1: 0.0309307, HIT_RATE: 0.3243692, ARHR_ALL_HITS: 0.1694330, NOVELTY: 0.0279071, AVERAGE_POPULARITY: 0.3264652, DIVERSITY_MEAN_INTER_LIST: 0.9289703, DIVERSITY_HERFINDAHL: 0.9928816, COVERAGE_ITEM: 0.9729591, COVERAGE_ITEM_CORRECT: 0.0865310, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3235099, DIVERSITY_GINI: 0.3519633, SHANNON_ENTROPY: 9.8467104, RATIO_DIVERSITY_HERFINDAHL: 0.9936928, RATIO_DIVERSITY_GINI: 1.0066268, RATIO_SHANNON_ENTROPY: 0.9106471, RATIO_AVERAGE_POPULARITY: 1.1819206, RATIO_NOVELTY: 0.0945598, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 260 of 500. Elapsed time 40.77 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.63 sec. MSE loss 1.19E+00. Sample per second: 66511\n",
      "FUNK_SVD: Epoch 261 of 500. Elapsed time 40.92 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.86 sec. MSE loss 1.18E+00. Sample per second: 64909\n",
      "FUNK_SVD: Epoch 262 of 500. Elapsed time 41.08 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.96 sec. MSE loss 1.19E+00. Sample per second: 64281\n",
      "FUNK_SVD: Epoch 263 of 500. Elapsed time 41.23 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.10 sec. MSE loss 1.19E+00. Sample per second: 63375\n",
      "FUNK_SVD: Epoch 264 of 500. Elapsed time 41.38 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.34 sec. MSE loss 1.18E+00. Sample per second: 68521\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.44 sec. Users per second: 1356\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0436089, PRECISION_RECALL_MIN_DEN: 0.0499142, RECALL: 0.0249150, MAP: 0.0198512, MAP_MIN_DEN: 0.0223955, MRR: 0.1526885, NDCG: 0.0479010, F1: 0.0317120, HIT_RATE: 0.3303453, ARHR_ALL_HITS: 0.1737379, NOVELTY: 0.0277868, AVERAGE_POPULARITY: 0.3340065, DIVERSITY_MEAN_INTER_LIST: 0.9248661, DIVERSITY_HERFINDAHL: 0.9924713, COVERAGE_ITEM: 0.9708988, COVERAGE_ITEM_CORRECT: 0.0867886, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3294702, DIVERSITY_GINI: 0.3425877, SHANNON_ENTROPY: 9.7708756, RATIO_DIVERSITY_HERFINDAHL: 0.9932821, RATIO_DIVERSITY_GINI: 0.9798122, RATIO_SHANNON_ENTROPY: 0.9036338, RATIO_AVERAGE_POPULARITY: 1.2092227, RATIO_NOVELTY: 0.0941523, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 265 of 500. Elapsed time 41.61 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 8.59 sec. MSE loss 1.18E+00. Sample per second: 74512\n",
      "FUNK_SVD: Epoch 266 of 500. Elapsed time 41.74 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.50 sec. MSE loss 1.19E+00. Sample per second: 67416\n",
      "FUNK_SVD: Epoch 267 of 500. Elapsed time 41.89 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.43 sec. MSE loss 1.18E+00. Sample per second: 67884\n",
      "FUNK_SVD: Epoch 268 of 500. Elapsed time 42.04 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.75 sec. MSE loss 1.19E+00. Sample per second: 65627\n",
      "FUNK_SVD: Epoch 269 of 500. Elapsed time 42.19 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.42 sec. MSE loss 1.18E+00. Sample per second: 61427\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.66 sec. Users per second: 1294\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0447543, PRECISION_RECALL_MIN_DEN: 0.0511433, RECALL: 0.0255081, MAP: 0.0203915, MAP_MIN_DEN: 0.0230055, MRR: 0.1558372, NDCG: 0.0491938, F1: 0.0324952, HIT_RATE: 0.3371514, ARHR_ALL_HITS: 0.1778578, NOVELTY: 0.0276665, AVERAGE_POPULARITY: 0.3412368, DIVERSITY_MEAN_INTER_LIST: 0.9206939, DIVERSITY_HERFINDAHL: 0.9920541, COVERAGE_ITEM: 0.9690961, COVERAGE_ITEM_CORRECT: 0.0862735, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3362583, DIVERSITY_GINI: 0.3336074, SHANNON_ENTROPY: 9.6963350, RATIO_DIVERSITY_HERFINDAHL: 0.9928646, RATIO_DIVERSITY_GINI: 0.9541284, RATIO_SHANNON_ENTROPY: 0.8967401, RATIO_AVERAGE_POPULARITY: 1.2353992, RATIO_NOVELTY: 0.0937447, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 270 of 500. Elapsed time 42.43 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.19 sec. MSE loss 1.18E+00. Sample per second: 69689\n",
      "FUNK_SVD: Epoch 271 of 500. Elapsed time 42.58 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.07 sec. MSE loss 1.18E+00. Sample per second: 70603\n",
      "FUNK_SVD: Epoch 272 of 500. Elapsed time 42.73 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.30 sec. MSE loss 1.18E+00. Sample per second: 68844\n",
      "FUNK_SVD: Epoch 273 of 500. Elapsed time 42.89 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.67 sec. MSE loss 1.18E+00. Sample per second: 66142\n",
      "FUNK_SVD: Epoch 274 of 500. Elapsed time 43.04 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.09 sec. MSE loss 1.19E+00. Sample per second: 63473\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.52 sec. Users per second: 1332\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0458333, PRECISION_RECALL_MIN_DEN: 0.0523763, RECALL: 0.0261413, MAP: 0.0209133, MAP_MIN_DEN: 0.0236224, MRR: 0.1589398, NDCG: 0.0504495, F1: 0.0332935, HIT_RATE: 0.3436255, ARHR_ALL_HITS: 0.1818605, NOVELTY: 0.0275591, AVERAGE_POPULARITY: 0.3486444, DIVERSITY_MEAN_INTER_LIST: 0.9164485, DIVERSITY_HERFINDAHL: 0.9916296, COVERAGE_ITEM: 0.9660057, COVERAGE_ITEM_CORRECT: 0.0857584, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3427152, DIVERSITY_GINI: 0.3246766, SHANNON_ENTROPY: 9.6194072, RATIO_DIVERSITY_HERFINDAHL: 0.9924398, RATIO_DIVERSITY_GINI: 0.9285860, RATIO_SHANNON_ENTROPY: 0.8896256, RATIO_AVERAGE_POPULARITY: 1.2622171, RATIO_NOVELTY: 0.0933809, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 275 of 500. Elapsed time 43.27 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.71 sec. MSE loss 1.18E+00. Sample per second: 65951\n",
      "FUNK_SVD: Epoch 276 of 500. Elapsed time 43.43 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.92 sec. MSE loss 1.18E+00. Sample per second: 64490\n",
      "FUNK_SVD: Epoch 277 of 500. Elapsed time 43.58 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.28 sec. MSE loss 1.18E+00. Sample per second: 62268\n",
      "FUNK_SVD: Epoch 278 of 500. Elapsed time 43.74 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.31 sec. MSE loss 1.18E+00. Sample per second: 101445\n",
      "FUNK_SVD: Epoch 279 of 500. Elapsed time 43.84 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 5.86 sec. MSE loss 1.18E+00. Sample per second: 109179\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.24 sec. Users per second: 1861\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0470452, PRECISION_RECALL_MIN_DEN: 0.0537278, RECALL: 0.0268309, MAP: 0.0214558, MAP_MIN_DEN: 0.0242079, MRR: 0.1622729, NDCG: 0.0517324, F1: 0.0341725, HIT_RATE: 0.3512616, ARHR_ALL_HITS: 0.1860940, NOVELTY: 0.0274473, AVERAGE_POPULARITY: 0.3561177, DIVERSITY_MEAN_INTER_LIST: 0.9119886, DIVERSITY_HERFINDAHL: 0.9911837, COVERAGE_ITEM: 0.9644605, COVERAGE_ITEM_CORRECT: 0.0847283, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3503311, DIVERSITY_GINI: 0.3154311, SHANNON_ENTROPY: 9.5406725, RATIO_DIVERSITY_HERFINDAHL: 0.9919935, RATIO_DIVERSITY_GINI: 0.9021436, RATIO_SHANNON_ENTROPY: 0.8823440, RATIO_AVERAGE_POPULARITY: 1.2892734, RATIO_NOVELTY: 0.0930019, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 280 of 500. Elapsed time 43.98 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.20 sec. MSE loss 1.18E+00. Sample per second: 103294\n",
      "FUNK_SVD: Epoch 281 of 500. Elapsed time 44.08 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.29 sec. MSE loss 1.18E+00. Sample per second: 101737\n",
      "FUNK_SVD: Epoch 282 of 500. Elapsed time 44.19 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.24 sec. MSE loss 1.19E+00. Sample per second: 102474\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Epoch 283 of 500. Elapsed time 44.28 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 5.68 sec. MSE loss 1.18E+00. Sample per second: 112653\n",
      "FUNK_SVD: Epoch 284 of 500. Elapsed time 44.38 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.74 sec. MSE loss 1.18E+00. Sample per second: 94932\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.40 sec. Users per second: 1770\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0479416, PRECISION_RECALL_MIN_DEN: 0.0547067, RECALL: 0.0273321, MAP: 0.0219584, MAP_MIN_DEN: 0.0247636, MRR: 0.1649711, NDCG: 0.0527168, F1: 0.0348155, HIT_RATE: 0.3564077, ARHR_ALL_HITS: 0.1898037, NOVELTY: 0.0273270, AVERAGE_POPULARITY: 0.3634751, DIVERSITY_MEAN_INTER_LIST: 0.9073447, DIVERSITY_HERFINDAHL: 0.9907194, COVERAGE_ITEM: 0.9626577, COVERAGE_ITEM_CORRECT: 0.0839557, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3554636, DIVERSITY_GINI: 0.3065650, SHANNON_ENTROPY: 9.4614627, RATIO_DIVERSITY_HERFINDAHL: 0.9915288, RATIO_DIVERSITY_GINI: 0.8767861, RATIO_SHANNON_ENTROPY: 0.8750185, RATIO_AVERAGE_POPULARITY: 1.3159097, RATIO_NOVELTY: 0.0925944, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 285 of 500. Elapsed time 44.53 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.24 sec. MSE loss 1.18E+00. Sample per second: 102623\n",
      "FUNK_SVD: Epoch 286 of 500. Elapsed time 44.63 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 5.76 sec. MSE loss 1.18E+00. Sample per second: 111002\n",
      "FUNK_SVD: Epoch 287 of 500. Elapsed time 44.73 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.63 sec. MSE loss 1.18E+00. Sample per second: 96526\n",
      "FUNK_SVD: Epoch 288 of 500. Elapsed time 44.82 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.72 sec. MSE loss 1.18E+00. Sample per second: 95256\n",
      "FUNK_SVD: Epoch 289 of 500. Elapsed time 44.93 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.77 sec. MSE loss 1.18E+00. Sample per second: 94580\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.36 sec. Users per second: 1790\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0488546, PRECISION_RECALL_MIN_DEN: 0.0556397, RECALL: 0.0277246, MAP: 0.0224622, MAP_MIN_DEN: 0.0253300, MRR: 0.1676693, NDCG: 0.0537830, F1: 0.0353745, HIT_RATE: 0.3602258, ARHR_ALL_HITS: 0.1935283, NOVELTY: 0.0272230, AVERAGE_POPULARITY: 0.3705566, DIVERSITY_MEAN_INTER_LIST: 0.9029593, DIVERSITY_HERFINDAHL: 0.9902809, COVERAGE_ITEM: 0.9598249, COVERAGE_ITEM_CORRECT: 0.0831831, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3592715, DIVERSITY_GINI: 0.2978692, SHANNON_ENTROPY: 9.3855936, RATIO_DIVERSITY_HERFINDAHL: 0.9910900, RATIO_DIVERSITY_GINI: 0.8519158, RATIO_SHANNON_ENTROPY: 0.8680020, RATIO_AVERAGE_POPULARITY: 1.3415474, RATIO_NOVELTY: 0.0922419, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 290 of 500. Elapsed time 45.08 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 5.60 sec. MSE loss 1.18E+00. Sample per second: 114235\n",
      "FUNK_SVD: Epoch 291 of 500. Elapsed time 45.17 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.66 sec. MSE loss 1.18E+00. Sample per second: 96090\n",
      "FUNK_SVD: Epoch 292 of 500. Elapsed time 45.28 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.34 sec. MSE loss 1.18E+00. Sample per second: 100915\n",
      "FUNK_SVD: Epoch 293 of 500. Elapsed time 45.37 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 4.72 sec. MSE loss 1.18E+00. Sample per second: 135668\n",
      "FUNK_SVD: Epoch 294 of 500. Elapsed time 45.44 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.78 sec. MSE loss 1.18E+00. Sample per second: 94394\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.12 sec. Users per second: 1461\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0497012, PRECISION_RECALL_MIN_DEN: 0.0565894, RECALL: 0.0282071, MAP: 0.0229817, MAP_MIN_DEN: 0.0259121, MRR: 0.1702736, NDCG: 0.0548015, F1: 0.0359891, HIT_RATE: 0.3642098, ARHR_ALL_HITS: 0.1972507, NOVELTY: 0.0271214, AVERAGE_POPULARITY: 0.3768711, DIVERSITY_MEAN_INTER_LIST: 0.8988859, DIVERSITY_HERFINDAHL: 0.9898737, COVERAGE_ITEM: 0.9564770, COVERAGE_ITEM_CORRECT: 0.0824105, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3632450, DIVERSITY_GINI: 0.2904431, SHANNON_ENTROPY: 9.3161992, RATIO_DIVERSITY_HERFINDAHL: 0.9906824, RATIO_DIVERSITY_GINI: 0.8306771, RATIO_SHANNON_ENTROPY: 0.8615842, RATIO_AVERAGE_POPULARITY: 1.3644080, RATIO_NOVELTY: 0.0918978, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 295 of 500. Elapsed time 45.61 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.22 sec. MSE loss 1.18E+00. Sample per second: 62663\n",
      "FUNK_SVD: Epoch 296 of 500. Elapsed time 45.77 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.32 sec. MSE loss 1.18E+00. Sample per second: 68726\n",
      "FUNK_SVD: Epoch 297 of 500. Elapsed time 45.92 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.34 sec. MSE loss 1.18E+00. Sample per second: 68574\n",
      "FUNK_SVD: Epoch 298 of 500. Elapsed time 46.07 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.50 sec. MSE loss 1.18E+00. Sample per second: 67371\n",
      "FUNK_SVD: Epoch 299 of 500. Elapsed time 46.22 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.78 sec. MSE loss 1.18E+00. Sample per second: 65477\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.91 sec. Users per second: 1226\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0508466, PRECISION_RECALL_MIN_DEN: 0.0579577, RECALL: 0.0289286, MAP: 0.0235416, MAP_MIN_DEN: 0.0265269, MRR: 0.1734234, NDCG: 0.0560248, F1: 0.0368766, HIT_RATE: 0.3705179, ARHR_ALL_HITS: 0.2014370, NOVELTY: 0.0270056, AVERAGE_POPULARITY: 0.3840473, DIVERSITY_MEAN_INTER_LIST: 0.8942657, DIVERSITY_HERFINDAHL: 0.9894117, COVERAGE_ITEM: 0.9551893, COVERAGE_ITEM_CORRECT: 0.0826680, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3695364, DIVERSITY_GINI: 0.2816587, SHANNON_ENTROPY: 9.2378580, RATIO_DIVERSITY_HERFINDAHL: 0.9902200, RATIO_DIVERSITY_GINI: 0.8055532, RATIO_SHANNON_ENTROPY: 0.8543390, RATIO_AVERAGE_POPULARITY: 1.3903884, RATIO_NOVELTY: 0.0915052, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 300 of 500. Elapsed time 46.46 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.16 sec. MSE loss 1.18E+00. Sample per second: 63025\n",
      "FUNK_SVD: Epoch 301 of 500. Elapsed time 46.62 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.42 sec. MSE loss 1.18E+00. Sample per second: 67968\n",
      "FUNK_SVD: Epoch 302 of 500. Elapsed time 46.77 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.65 sec. MSE loss 1.18E+00. Sample per second: 66307\n",
      "FUNK_SVD: Epoch 303 of 500. Elapsed time 46.93 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.65 sec. MSE loss 1.18E+00. Sample per second: 66335\n",
      "FUNK_SVD: Epoch 304 of 500. Elapsed time 47.07 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.78 sec. MSE loss 1.18E+00. Sample per second: 65444\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.71 sec. Users per second: 1279\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0516268, PRECISION_RECALL_MIN_DEN: 0.0589155, RECALL: 0.0294285, MAP: 0.0239811, MAP_MIN_DEN: 0.0270134, MRR: 0.1757004, NDCG: 0.0568877, F1: 0.0374880, HIT_RATE: 0.3738380, ARHR_ALL_HITS: 0.2046462, NOVELTY: 0.0268984, AVERAGE_POPULARITY: 0.3910366, DIVERSITY_MEAN_INTER_LIST: 0.8897353, DIVERSITY_HERFINDAHL: 0.9889588, COVERAGE_ITEM: 0.9515838, COVERAGE_ITEM_CORRECT: 0.0818954, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3728477, DIVERSITY_GINI: 0.2732293, SHANNON_ENTROPY: 9.1603455, RATIO_DIVERSITY_HERFINDAHL: 0.9897667, RATIO_DIVERSITY_GINI: 0.7814449, RATIO_SHANNON_ENTROPY: 0.8471705, RATIO_AVERAGE_POPULARITY: 1.4156922, RATIO_NOVELTY: 0.0911420, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 305 of 500. Elapsed time 47.31 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.95 sec. MSE loss 1.18E+00. Sample per second: 64314\n",
      "FUNK_SVD: Epoch 306 of 500. Elapsed time 47.46 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.25 sec. MSE loss 1.18E+00. Sample per second: 69224\n",
      "FUNK_SVD: Epoch 307 of 500. Elapsed time 47.60 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 8.09 sec. MSE loss 1.18E+00. Sample per second: 79168\n",
      "FUNK_SVD: Epoch 308 of 500. Elapsed time 47.73 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.84 sec. MSE loss 1.18E+00. Sample per second: 65040\n",
      "FUNK_SVD: Epoch 309 of 500. Elapsed time 47.89 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.06 sec. MSE loss 1.17E+00. Sample per second: 63593\n",
      "FUNK_SVD: Validation begins...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.67 sec. Users per second: 1289\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0526062, PRECISION_RECALL_MIN_DEN: 0.0601665, RECALL: 0.0301585, MAP: 0.0244564, MAP_MIN_DEN: 0.0275536, MRR: 0.1782749, NDCG: 0.0579454, F1: 0.0383382, HIT_RATE: 0.3794821, ARHR_ALL_HITS: 0.2081897, NOVELTY: 0.0267930, AVERAGE_POPULARITY: 0.3975933, DIVERSITY_MEAN_INTER_LIST: 0.8852871, DIVERSITY_HERFINDAHL: 0.9885140, COVERAGE_ITEM: 0.9492660, COVERAGE_ITEM_CORRECT: 0.0816379, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3784768, DIVERSITY_GINI: 0.2653890, SHANNON_ENTROPY: 9.0865984, RATIO_DIVERSITY_HERFINDAHL: 0.9893216, RATIO_DIVERSITY_GINI: 0.7590215, RATIO_SHANNON_ENTROPY: 0.8403502, RATIO_AVERAGE_POPULARITY: 1.4394299, RATIO_NOVELTY: 0.0907850, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 310 of 500. Elapsed time 48.13 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.84 sec. MSE loss 1.17E+00. Sample per second: 65074\n",
      "FUNK_SVD: Epoch 311 of 500. Elapsed time 48.28 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.93 sec. MSE loss 1.18E+00. Sample per second: 64445\n",
      "FUNK_SVD: Epoch 312 of 500. Elapsed time 48.43 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.19 sec. MSE loss 1.18E+00. Sample per second: 62807\n",
      "FUNK_SVD: Epoch 313 of 500. Elapsed time 48.58 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.45 sec. MSE loss 1.18E+00. Sample per second: 67709\n",
      "FUNK_SVD: Epoch 314 of 500. Elapsed time 48.74 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.53 sec. MSE loss 1.18E+00. Sample per second: 67206\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.61 sec. Users per second: 1305\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0536521, PRECISION_RECALL_MIN_DEN: 0.0613774, RECALL: 0.0307188, MAP: 0.0249391, MAP_MIN_DEN: 0.0280782, MRR: 0.1806455, NDCG: 0.0590603, F1: 0.0390687, HIT_RATE: 0.3849602, ARHR_ALL_HITS: 0.2116183, NOVELTY: 0.0266880, AVERAGE_POPULARITY: 0.4046560, DIVERSITY_MEAN_INTER_LIST: 0.8802840, DIVERSITY_HERFINDAHL: 0.9880138, COVERAGE_ITEM: 0.9451455, COVERAGE_ITEM_CORRECT: 0.0811228, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3839404, DIVERSITY_GINI: 0.2571977, SHANNON_ENTROPY: 9.0060678, RATIO_DIVERSITY_HERFINDAHL: 0.9888210, RATIO_DIVERSITY_GINI: 0.7355939, RATIO_SHANNON_ENTROPY: 0.8329025, RATIO_AVERAGE_POPULARITY: 1.4649994, RATIO_NOVELTY: 0.0904290, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 315 of 500. Elapsed time 48.97 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.24 sec. MSE loss 1.18E+00. Sample per second: 69254\n",
      "FUNK_SVD: Epoch 316 of 500. Elapsed time 49.12 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.66 sec. MSE loss 1.18E+00. Sample per second: 66279\n",
      "FUNK_SVD: Epoch 317 of 500. Elapsed time 49.28 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.14 sec. MSE loss 1.18E+00. Sample per second: 63117\n",
      "FUNK_SVD: Epoch 318 of 500. Elapsed time 49.43 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.43 sec. MSE loss 1.18E+00. Sample per second: 67853\n",
      "FUNK_SVD: Epoch 319 of 500. Elapsed time 49.59 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 8.18 sec. MSE loss 1.18E+00. Sample per second: 78242\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.02 sec. Users per second: 1201\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0545817, PRECISION_RECALL_MIN_DEN: 0.0624194, RECALL: 0.0312710, MAP: 0.0254647, MAP_MIN_DEN: 0.0286233, MRR: 0.1836664, NDCG: 0.0601071, F1: 0.0397617, HIT_RATE: 0.3899402, ARHR_ALL_HITS: 0.2156072, NOVELTY: 0.0265830, AVERAGE_POPULARITY: 0.4114839, DIVERSITY_MEAN_INTER_LIST: 0.8751351, DIVERSITY_HERFINDAHL: 0.9874990, COVERAGE_ITEM: 0.9407674, COVERAGE_ITEM_CORRECT: 0.0803502, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3889073, DIVERSITY_GINI: 0.2490360, SHANNON_ENTROPY: 8.9270523, RATIO_DIVERSITY_HERFINDAHL: 0.9883057, RATIO_DIVERSITY_GINI: 0.7122514, RATIO_SHANNON_ENTROPY: 0.8255950, RATIO_AVERAGE_POPULARITY: 1.4897188, RATIO_NOVELTY: 0.0900733, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 320 of 500. Elapsed time 49.80 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.45 sec. MSE loss 1.18E+00. Sample per second: 67732\n",
      "FUNK_SVD: Epoch 321 of 500. Elapsed time 49.96 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.81 sec. MSE loss 1.17E+00. Sample per second: 65276\n",
      "FUNK_SVD: Epoch 322 of 500. Elapsed time 50.11 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.43 sec. MSE loss 1.18E+00. Sample per second: 61370\n",
      "FUNK_SVD: Epoch 323 of 500. Elapsed time 50.27 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.44 sec. MSE loss 1.17E+00. Sample per second: 67854\n",
      "FUNK_SVD: Epoch 324 of 500. Elapsed time 50.42 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.81 sec. MSE loss 1.18E+00. Sample per second: 65226\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.89 sec. Users per second: 1233\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0556607, PRECISION_RECALL_MIN_DEN: 0.0636100, RECALL: 0.0318302, MAP: 0.0258785, MAP_MIN_DEN: 0.0290549, MRR: 0.1853372, NDCG: 0.0610413, F1: 0.0405000, HIT_RATE: 0.3950863, ARHR_ALL_HITS: 0.2183485, NOVELTY: 0.0264677, AVERAGE_POPULARITY: 0.4184108, DIVERSITY_MEAN_INTER_LIST: 0.8700821, DIVERSITY_HERFINDAHL: 0.9869938, COVERAGE_ITEM: 0.9369045, COVERAGE_ITEM_CORRECT: 0.0806078, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3940397, DIVERSITY_GINI: 0.2408211, SHANNON_ENTROPY: 8.8465926, RATIO_DIVERSITY_HERFINDAHL: 0.9878001, RATIO_DIVERSITY_GINI: 0.6887565, RATIO_SHANNON_ENTROPY: 0.8181539, RATIO_AVERAGE_POPULARITY: 1.5147966, RATIO_NOVELTY: 0.0896827, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 325 of 500. Elapsed time 50.66 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.14 sec. MSE loss 1.17E+00. Sample per second: 63120\n",
      "FUNK_SVD: Epoch 326 of 500. Elapsed time 50.82 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.36 sec. MSE loss 1.18E+00. Sample per second: 68380\n",
      "FUNK_SVD: Epoch 327 of 500. Elapsed time 50.97 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.38 sec. MSE loss 1.17E+00. Sample per second: 68235\n",
      "FUNK_SVD: Epoch 328 of 500. Elapsed time 51.12 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.52 sec. MSE loss 1.18E+00. Sample per second: 67258\n",
      "FUNK_SVD: Epoch 329 of 500. Elapsed time 51.27 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.74 sec. MSE loss 1.18E+00. Sample per second: 65708\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.60 sec. Users per second: 1309\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0563413, PRECISION_RECALL_MIN_DEN: 0.0643364, RECALL: 0.0321732, MAP: 0.0263236, MAP_MIN_DEN: 0.0295286, MRR: 0.1875295, NDCG: 0.0618734, F1: 0.0409578, HIT_RATE: 0.3982404, ARHR_ALL_HITS: 0.2215072, NOVELTY: 0.0263656, AVERAGE_POPULARITY: 0.4248062, DIVERSITY_MEAN_INTER_LIST: 0.8653624, DIVERSITY_HERFINDAHL: 0.9865219, COVERAGE_ITEM: 0.9317538, COVERAGE_ITEM_CORRECT: 0.0790626, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3971854, DIVERSITY_GINI: 0.2330702, SHANNON_ENTROPY: 8.7702715, RATIO_DIVERSITY_HERFINDAHL: 0.9873278, RATIO_DIVERSITY_GINI: 0.6665885, RATIO_SHANNON_ENTROPY: 0.8110955, RATIO_AVERAGE_POPULARITY: 1.5379501, RATIO_NOVELTY: 0.0893367, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 330 of 500. Elapsed time 51.50 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.55 sec. MSE loss 1.17E+00. Sample per second: 67039\n",
      "FUNK_SVD: Epoch 331 of 500. Elapsed time 51.66 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 7.94 sec. MSE loss 1.18E+00. Sample per second: 80598\n",
      "FUNK_SVD: Epoch 332 of 500. Elapsed time 51.78 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.47 sec. MSE loss 1.17E+00. Sample per second: 61174\n",
      "FUNK_SVD: Epoch 333 of 500. Elapsed time 51.94 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.65 sec. MSE loss 1.17E+00. Sample per second: 66378\n",
      "FUNK_SVD: Epoch 334 of 500. Elapsed time 52.09 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.61 sec. MSE loss 1.18E+00. Sample per second: 66631\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.62 sec. Users per second: 1303\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0574037, PRECISION_RECALL_MIN_DEN: 0.0654991, RECALL: 0.0326456, MAP: 0.0267896, MAP_MIN_DEN: 0.0300651, MRR: 0.1892668, NDCG: 0.0628789, F1: 0.0416211, HIT_RATE: 0.4017264, ARHR_ALL_HITS: 0.2244461, NOVELTY: 0.0262567, AVERAGE_POPULARITY: 0.4317579, DIVERSITY_MEAN_INTER_LIST: 0.8603147, DIVERSITY_HERFINDAHL: 0.9860172, COVERAGE_ITEM: 0.9263456, COVERAGE_ITEM_CORRECT: 0.0770023, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4006623, DIVERSITY_GINI: 0.2247204, SHANNON_ENTROPY: 8.6886308, RATIO_DIVERSITY_HERFINDAHL: 0.9868227, RATIO_DIVERSITY_GINI: 0.6427080, RATIO_SHANNON_ENTROPY: 0.8035452, RATIO_AVERAGE_POPULARITY: 1.5631177, RATIO_NOVELTY: 0.0889677, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 335 of 500. Elapsed time 52.32 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Processed 640256 (100.0%) in 9.41 sec. MSE loss 1.17E+00. Sample per second: 67997\n",
      "FUNK_SVD: Epoch 336 of 500. Elapsed time 52.47 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.75 sec. MSE loss 1.17E+00. Sample per second: 65688\n",
      "FUNK_SVD: Epoch 337 of 500. Elapsed time 52.63 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.05 sec. MSE loss 1.18E+00. Sample per second: 63727\n",
      "FUNK_SVD: Epoch 338 of 500. Elapsed time 52.78 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.20 sec. MSE loss 1.17E+00. Sample per second: 69548\n",
      "FUNK_SVD: Epoch 339 of 500. Elapsed time 52.93 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.40 sec. MSE loss 1.17E+00. Sample per second: 68112\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.00 sec. Users per second: 1205\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0581839, PRECISION_RECALL_MIN_DEN: 0.0662999, RECALL: 0.0329649, MAP: 0.0271893, MAP_MIN_DEN: 0.0304932, MRR: 0.1910058, NDCG: 0.0636941, F1: 0.0420856, HIT_RATE: 0.4050465, ARHR_ALL_HITS: 0.2271611, NOVELTY: 0.0261529, AVERAGE_POPULARITY: 0.4379252, DIVERSITY_MEAN_INTER_LIST: 0.8556630, DIVERSITY_HERFINDAHL: 0.9855521, COVERAGE_ITEM: 0.9219676, COVERAGE_ITEM_CORRECT: 0.0767448, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4039735, DIVERSITY_GINI: 0.2177506, SHANNON_ENTROPY: 8.6157042, RATIO_DIVERSITY_HERFINDAHL: 0.9863573, RATIO_DIVERSITY_GINI: 0.6227740, RATIO_SHANNON_ENTROPY: 0.7968008, RATIO_AVERAGE_POPULARITY: 1.5854456, RATIO_NOVELTY: 0.0886160, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 340 of 500. Elapsed time 53.17 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.59 sec. MSE loss 1.18E+00. Sample per second: 66745\n",
      "FUNK_SVD: Epoch 341 of 500. Elapsed time 53.32 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.52 sec. MSE loss 1.17E+00. Sample per second: 67218\n",
      "FUNK_SVD: Epoch 342 of 500. Elapsed time 53.47 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.64 sec. MSE loss 1.17E+00. Sample per second: 66362\n",
      "FUNK_SVD: Epoch 343 of 500. Elapsed time 53.62 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.00 sec. MSE loss 1.17E+00. Sample per second: 64021\n",
      "FUNK_SVD: Epoch 344 of 500. Elapsed time 53.78 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 7.23 sec. MSE loss 1.17E+00. Sample per second: 88571\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.64 sec. Users per second: 1299\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0592131, PRECISION_RECALL_MIN_DEN: 0.0675741, RECALL: 0.0336490, MAP: 0.0276219, MAP_MIN_DEN: 0.0310035, MRR: 0.1931764, NDCG: 0.0647844, F1: 0.0429123, HIT_RATE: 0.4100266, ARHR_ALL_HITS: 0.2302188, NOVELTY: 0.0260555, AVERAGE_POPULARITY: 0.4443346, DIVERSITY_MEAN_INTER_LIST: 0.8504122, DIVERSITY_HERFINDAHL: 0.9850271, COVERAGE_ITEM: 0.9188772, COVERAGE_ITEM_CORRECT: 0.0759722, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4089404, DIVERSITY_GINI: 0.2107572, SHANNON_ENTROPY: 8.5391006, RATIO_DIVERSITY_HERFINDAHL: 0.9858318, RATIO_DIVERSITY_GINI: 0.6027727, RATIO_SHANNON_ENTROPY: 0.7897163, RATIO_AVERAGE_POPULARITY: 1.6086500, RATIO_NOVELTY: 0.0882861, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 345 of 500. Elapsed time 53.98 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.08 sec. MSE loss 1.17E+00. Sample per second: 63493\n",
      "FUNK_SVD: Epoch 346 of 500. Elapsed time 54.13 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.29 sec. MSE loss 1.17E+00. Sample per second: 68904\n",
      "FUNK_SVD: Epoch 347 of 500. Elapsed time 54.29 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.56 sec. MSE loss 1.17E+00. Sample per second: 66945\n",
      "FUNK_SVD: Epoch 348 of 500. Elapsed time 54.44 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.61 sec. MSE loss 1.17E+00. Sample per second: 66649\n",
      "FUNK_SVD: Epoch 349 of 500. Elapsed time 54.59 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.02 sec. MSE loss 1.17E+00. Sample per second: 63880\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.59 sec. Users per second: 1311\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0601926, PRECISION_RECALL_MIN_DEN: 0.0686333, RECALL: 0.0341308, MAP: 0.0279745, MAP_MIN_DEN: 0.0313966, MRR: 0.1945664, NDCG: 0.0656812, F1: 0.0435612, HIT_RATE: 0.4151726, ARHR_ALL_HITS: 0.2324837, NOVELTY: 0.0259660, AVERAGE_POPULARITY: 0.4503628, DIVERSITY_MEAN_INTER_LIST: 0.8456926, DIVERSITY_HERFINDAHL: 0.9845552, COVERAGE_ITEM: 0.9137265, COVERAGE_ITEM_CORRECT: 0.0749421, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4140728, DIVERSITY_GINI: 0.2040524, SHANNON_ENTROPY: 8.4658178, RATIO_DIVERSITY_HERFINDAHL: 0.9853596, RATIO_DIVERSITY_GINI: 0.5835967, RATIO_SHANNON_ENTROPY: 0.7829389, RATIO_AVERAGE_POPULARITY: 1.6304743, RATIO_NOVELTY: 0.0879828, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 350 of 500. Elapsed time 54.82 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.99 sec. MSE loss 1.17E+00. Sample per second: 64077\n",
      "FUNK_SVD: Epoch 351 of 500. Elapsed time 54.98 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.07 sec. MSE loss 1.17E+00. Sample per second: 70544\n",
      "FUNK_SVD: Epoch 352 of 500. Elapsed time 55.13 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.13 sec. MSE loss 1.17E+00. Sample per second: 70111\n",
      "FUNK_SVD: Epoch 353 of 500. Elapsed time 55.28 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.11 sec. MSE loss 1.17E+00. Sample per second: 70298\n",
      "FUNK_SVD: Epoch 354 of 500. Elapsed time 55.43 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.31 sec. MSE loss 1.17E+00. Sample per second: 68726\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.63 sec. Users per second: 1300\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0611886, PRECISION_RECALL_MIN_DEN: 0.0697604, RECALL: 0.0347175, MAP: 0.0284284, MAP_MIN_DEN: 0.0319015, MRR: 0.1962757, NDCG: 0.0666782, F1: 0.0442999, HIT_RATE: 0.4193227, ARHR_ALL_HITS: 0.2354079, NOVELTY: 0.0258681, AVERAGE_POPULARITY: 0.4562549, DIVERSITY_MEAN_INTER_LIST: 0.8409742, DIVERSITY_HERFINDAHL: 0.9840835, COVERAGE_ITEM: 0.9103786, COVERAGE_ITEM_CORRECT: 0.0739119, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4182119, DIVERSITY_GINI: 0.1973815, SHANNON_ENTROPY: 8.3939574, RATIO_DIVERSITY_HERFINDAHL: 0.9848874, RATIO_DIVERSITY_GINI: 0.5645176, RATIO_SHANNON_ENTROPY: 0.7762931, RATIO_AVERAGE_POPULARITY: 1.6518057, RATIO_NOVELTY: 0.0876512, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 355 of 500. Elapsed time 55.66 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.26 sec. MSE loss 1.17E+00. Sample per second: 62391\n",
      "FUNK_SVD: Epoch 356 of 500. Elapsed time 55.82 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.28 sec. MSE loss 1.17E+00. Sample per second: 101893\n",
      "FUNK_SVD: Epoch 357 of 500. Elapsed time 55.92 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.07 sec. MSE loss 1.17E+00. Sample per second: 105306\n",
      "FUNK_SVD: Epoch 358 of 500. Elapsed time 56.02 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 5.64 sec. MSE loss 1.17E+00. Sample per second: 113532\n",
      "FUNK_SVD: Epoch 359 of 500. Elapsed time 56.11 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.71 sec. MSE loss 1.17E+00. Sample per second: 95438\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.37 sec. Users per second: 1786\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0619024, PRECISION_RECALL_MIN_DEN: 0.0705367, RECALL: 0.0350250, MAP: 0.0288257, MAP_MIN_DEN: 0.0323084, MRR: 0.1979374, NDCG: 0.0673739, F1: 0.0447373, HIT_RATE: 0.4221448, ARHR_ALL_HITS: 0.2380462, NOVELTY: 0.0257755, AVERAGE_POPULARITY: 0.4619254, DIVERSITY_MEAN_INTER_LIST: 0.8363696, DIVERSITY_HERFINDAHL: 0.9836231, COVERAGE_ITEM: 0.9062580, COVERAGE_ITEM_CORRECT: 0.0728818, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4210265, DIVERSITY_GINI: 0.1908464, SHANNON_ENTROPY: 8.3234185, RATIO_DIVERSITY_HERFINDAHL: 0.9844267, RATIO_DIVERSITY_GINI: 0.5458271, RATIO_SHANNON_ENTROPY: 0.7697695, RATIO_AVERAGE_POPULARITY: 1.6723351, RATIO_NOVELTY: 0.0873371, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 360 of 500. Elapsed time 56.27 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.13 sec. MSE loss 1.17E+00. Sample per second: 104503\n",
      "FUNK_SVD: Epoch 361 of 500. Elapsed time 56.37 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 5.66 sec. MSE loss 1.17E+00. Sample per second: 113078\n",
      "FUNK_SVD: Epoch 362 of 500. Elapsed time 56.46 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.47 sec. MSE loss 1.17E+00. Sample per second: 98964\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Epoch 363 of 500. Elapsed time 56.56 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.51 sec. MSE loss 1.17E+00. Sample per second: 98362\n",
      "FUNK_SVD: Epoch 364 of 500. Elapsed time 56.66 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.58 sec. MSE loss 1.17E+00. Sample per second: 97299\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.39 sec. Users per second: 1779\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0625996, PRECISION_RECALL_MIN_DEN: 0.0712829, RECALL: 0.0353843, MAP: 0.0292478, MAP_MIN_DEN: 0.0327627, MRR: 0.1995559, NDCG: 0.0681370, F1: 0.0452124, HIT_RATE: 0.4244688, ARHR_ALL_HITS: 0.2406832, NOVELTY: 0.0257049, AVERAGE_POPULARITY: 0.4671956, DIVERSITY_MEAN_INTER_LIST: 0.8318940, DIVERSITY_HERFINDAHL: 0.9831756, COVERAGE_ITEM: 0.9013649, COVERAGE_ITEM_CORRECT: 0.0713366, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4233444, DIVERSITY_GINI: 0.1850793, SHANNON_ENTROPY: 8.2581284, RATIO_DIVERSITY_HERFINDAHL: 0.9839788, RATIO_DIVERSITY_GINI: 0.5293331, RATIO_SHANNON_ENTROPY: 0.7637313, RATIO_AVERAGE_POPULARITY: 1.6914150, RATIO_NOVELTY: 0.0870981, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 365 of 500. Elapsed time 56.81 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.42 sec. MSE loss 1.17E+00. Sample per second: 99783\n",
      "FUNK_SVD: Epoch 366 of 500. Elapsed time 56.90 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.35 sec. MSE loss 1.17E+00. Sample per second: 100824\n",
      "FUNK_SVD: Epoch 367 of 500. Elapsed time 57.00 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.41 sec. MSE loss 1.17E+00. Sample per second: 99800\n",
      "FUNK_SVD: Epoch 368 of 500. Elapsed time 57.10 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.42 sec. MSE loss 1.17E+00. Sample per second: 99746\n",
      "FUNK_SVD: Epoch 369 of 500. Elapsed time 57.20 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 5.95 sec. MSE loss 1.17E+00. Sample per second: 107671\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.22 sec. Users per second: 1869\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0637450, PRECISION_RECALL_MIN_DEN: 0.0725560, RECALL: 0.0359620, MAP: 0.0297193, MAP_MIN_DEN: 0.0332835, MRR: 0.2012655, NDCG: 0.0691594, F1: 0.0459827, HIT_RATE: 0.4292829, ARHR_ALL_HITS: 0.2435856, NOVELTY: 0.0256035, AVERAGE_POPULARITY: 0.4733062, DIVERSITY_MEAN_INTER_LIST: 0.8268008, DIVERSITY_HERFINDAHL: 0.9826664, COVERAGE_ITEM: 0.8954417, COVERAGE_ITEM_CORRECT: 0.0708215, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4281457, DIVERSITY_GINI: 0.1781477, SHANNON_ENTROPY: 8.1806337, RATIO_DIVERSITY_HERFINDAHL: 0.9834692, RATIO_DIVERSITY_GINI: 0.5095083, RATIO_SHANNON_ENTROPY: 0.7565644, RATIO_AVERAGE_POPULARITY: 1.7135377, RATIO_NOVELTY: 0.0867543, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 370 of 500. Elapsed time 57.35 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 5.94 sec. MSE loss 1.17E+00. Sample per second: 107846\n",
      "FUNK_SVD: Epoch 371 of 500. Elapsed time 57.45 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 5.24 sec. MSE loss 1.17E+00. Sample per second: 122082\n",
      "FUNK_SVD: Epoch 372 of 500. Elapsed time 57.52 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.47 sec. MSE loss 1.17E+00. Sample per second: 99020\n",
      "FUNK_SVD: Epoch 373 of 500. Elapsed time 57.62 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.64 sec. MSE loss 1.17E+00. Sample per second: 66389\n",
      "FUNK_SVD: Epoch 374 of 500. Elapsed time 57.77 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.90 sec. MSE loss 1.17E+00. Sample per second: 64649\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.62 sec. Users per second: 1304\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0646746, PRECISION_RECALL_MIN_DEN: 0.0735343, RECALL: 0.0363750, MAP: 0.0300670, MAP_MIN_DEN: 0.0336508, MRR: 0.2025154, NDCG: 0.0699033, F1: 0.0465620, HIT_RATE: 0.4327689, ARHR_ALL_HITS: 0.2457540, NOVELTY: 0.0255160, AVERAGE_POPULARITY: 0.4788738, DIVERSITY_MEAN_INTER_LIST: 0.8219281, DIVERSITY_HERFINDAHL: 0.9821792, COVERAGE_ITEM: 0.8910636, COVERAGE_ITEM_CORRECT: 0.0697914, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4316225, DIVERSITY_GINI: 0.1720924, SHANNON_ENTROPY: 8.1099913, RATIO_DIVERSITY_HERFINDAHL: 0.9829816, RATIO_DIVERSITY_GINI: 0.4921900, RATIO_SHANNON_ENTROPY: 0.7500312, RATIO_AVERAGE_POPULARITY: 1.7336941, RATIO_NOVELTY: 0.0864579, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 375 of 500. Elapsed time 58.01 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.73 sec. MSE loss 1.17E+00. Sample per second: 65781\n",
      "FUNK_SVD: Epoch 376 of 500. Elapsed time 58.16 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.72 sec. MSE loss 1.17E+00. Sample per second: 65872\n",
      "FUNK_SVD: Epoch 377 of 500. Elapsed time 58.31 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.79 sec. MSE loss 1.17E+00. Sample per second: 65366\n",
      "FUNK_SVD: Epoch 378 of 500. Elapsed time 58.46 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.97 sec. MSE loss 1.17E+00. Sample per second: 64231\n",
      "FUNK_SVD: Epoch 379 of 500. Elapsed time 58.61 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.78 sec. MSE loss 1.17E+00. Sample per second: 59349\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.76 sec. Users per second: 1266\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0656873, PRECISION_RECALL_MIN_DEN: 0.0745808, RECALL: 0.0367737, MAP: 0.0304874, MAP_MIN_DEN: 0.0340847, MRR: 0.2040026, NDCG: 0.0706496, F1: 0.0471509, HIT_RATE: 0.4364210, ARHR_ALL_HITS: 0.2483292, NOVELTY: 0.0254328, AVERAGE_POPULARITY: 0.4844050, DIVERSITY_MEAN_INTER_LIST: 0.8169648, DIVERSITY_HERFINDAHL: 0.9816829, COVERAGE_ITEM: 0.8853979, COVERAGE_ITEM_CORRECT: 0.0687613, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4352649, DIVERSITY_GINI: 0.1658527, SHANNON_ENTROPY: 8.0380776, RATIO_DIVERSITY_HERFINDAHL: 0.9824849, RATIO_DIVERSITY_GINI: 0.4743444, RATIO_SHANNON_ENTROPY: 0.7433805, RATIO_AVERAGE_POPULARITY: 1.7537191, RATIO_NOVELTY: 0.0861761, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 380 of 500. Elapsed time 58.86 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.63 sec. MSE loss 1.17E+00. Sample per second: 66483\n",
      "FUNK_SVD: Epoch 381 of 500. Elapsed time 59.01 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.95 sec. MSE loss 1.17E+00. Sample per second: 64309\n",
      "FUNK_SVD: Epoch 382 of 500. Elapsed time 59.16 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.29 sec. MSE loss 1.17E+00. Sample per second: 62191\n",
      "FUNK_SVD: Epoch 383 of 500. Elapsed time 59.32 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.43 sec. MSE loss 1.17E+00. Sample per second: 67876\n",
      "FUNK_SVD: Epoch 384 of 500. Elapsed time 59.47 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.96 sec. MSE loss 1.16E+00. Sample per second: 64231\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.76 sec. Users per second: 1602\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0663347, PRECISION_RECALL_MIN_DEN: 0.0753038, RECALL: 0.0370942, MAP: 0.0308574, MAP_MIN_DEN: 0.0344856, MRR: 0.2055046, NDCG: 0.0713577, F1: 0.0475812, HIT_RATE: 0.4384130, ARHR_ALL_HITS: 0.2506948, NOVELTY: 0.0253589, AVERAGE_POPULARITY: 0.4893261, DIVERSITY_MEAN_INTER_LIST: 0.8127145, DIVERSITY_HERFINDAHL: 0.9812580, COVERAGE_ITEM: 0.8812774, COVERAGE_ITEM_CORRECT: 0.0690188, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4372517, DIVERSITY_GINI: 0.1604229, SHANNON_ENTROPY: 7.9756670, RATIO_DIVERSITY_HERFINDAHL: 0.9820596, RATIO_DIVERSITY_GINI: 0.4588148, RATIO_SHANNON_ENTROPY: 0.7376086, RATIO_AVERAGE_POPULARITY: 1.7715353, RATIO_NOVELTY: 0.0859256, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 385 of 500. Elapsed time 59.69 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.87 sec. MSE loss 1.17E+00. Sample per second: 64850\n",
      "FUNK_SVD: Epoch 386 of 500. Elapsed time 59.85 min\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.32 sec. MSE loss 1.17E+00. Sample per second: 62033\n",
      "FUNK_SVD: Epoch 387 of 500. Elapsed time 1.00 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.64 sec. MSE loss 1.17E+00. Sample per second: 66437\n",
      "FUNK_SVD: Epoch 388 of 500. Elapsed time 1.00 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.08 sec. MSE loss 1.16E+00. Sample per second: 63535\n",
      "FUNK_SVD: Epoch 389 of 500. Elapsed time 1.01 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.55 sec. MSE loss 1.17E+00. Sample per second: 67019\n",
      "FUNK_SVD: Validation begins...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.84 sec. Users per second: 1244\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0669655, PRECISION_RECALL_MIN_DEN: 0.0759362, RECALL: 0.0373173, MAP: 0.0311894, MAP_MIN_DEN: 0.0348183, MRR: 0.2064615, NDCG: 0.0719500, F1: 0.0479268, HIT_RATE: 0.4404050, ARHR_ALL_HITS: 0.2526715, NOVELTY: 0.0252663, AVERAGE_POPULARITY: 0.4951144, DIVERSITY_MEAN_INTER_LIST: 0.8075010, DIVERSITY_HERFINDAHL: 0.9807367, COVERAGE_ITEM: 0.8748390, COVERAGE_ITEM_CORRECT: 0.0679887, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4392384, DIVERSITY_GINI: 0.1543429, SHANNON_ENTROPY: 7.9011821, RATIO_DIVERSITY_HERFINDAHL: 0.9815379, RATIO_DIVERSITY_GINI: 0.4414257, RATIO_SHANNON_ENTROPY: 0.7307201, RATIO_AVERAGE_POPULARITY: 1.7924911, RATIO_NOVELTY: 0.0856120, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 390 of 500. Elapsed time 1.01 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.66 sec. MSE loss 1.17E+00. Sample per second: 66304\n",
      "FUNK_SVD: Epoch 391 of 500. Elapsed time 1.01 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.90 sec. MSE loss 1.17E+00. Sample per second: 64648\n",
      "FUNK_SVD: Epoch 392 of 500. Elapsed time 1.01 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.56 sec. MSE loss 1.16E+00. Sample per second: 60622\n",
      "FUNK_SVD: Epoch 393 of 500. Elapsed time 1.02 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.73 sec. MSE loss 1.16E+00. Sample per second: 65789\n",
      "FUNK_SVD: Epoch 394 of 500. Elapsed time 1.02 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.88 sec. MSE loss 1.17E+00. Sample per second: 64777\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.99 sec. Users per second: 1207\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0677291, PRECISION_RECALL_MIN_DEN: 0.0767487, RECALL: 0.0376680, MAP: 0.0314878, MAP_MIN_DEN: 0.0351299, MRR: 0.2074884, NDCG: 0.0726186, F1: 0.0484116, HIT_RATE: 0.4433931, ARHR_ALL_HITS: 0.2545531, NOVELTY: 0.0251826, AVERAGE_POPULARITY: 0.5003465, DIVERSITY_MEAN_INTER_LIST: 0.8029507, DIVERSITY_HERFINDAHL: 0.9802817, COVERAGE_ITEM: 0.8678857, COVERAGE_ITEM_CORRECT: 0.0669585, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4422185, DIVERSITY_GINI: 0.1489150, SHANNON_ENTROPY: 7.8340246, RATIO_DIVERSITY_HERFINDAHL: 0.9810826, RATIO_DIVERSITY_GINI: 0.4259018, RATIO_SHANNON_ENTROPY: 0.7245092, RATIO_AVERAGE_POPULARITY: 1.8114332, RATIO_NOVELTY: 0.0853281, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 395 of 500. Elapsed time 1.02 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.12 sec. MSE loss 1.17E+00. Sample per second: 63255\n",
      "FUNK_SVD: Epoch 396 of 500. Elapsed time 1.03 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.77 sec. MSE loss 1.17E+00. Sample per second: 65493\n",
      "FUNK_SVD: Epoch 397 of 500. Elapsed time 1.03 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.27 sec. MSE loss 1.16E+00. Sample per second: 62338\n",
      "FUNK_SVD: Epoch 398 of 500. Elapsed time 1.03 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 7.93 sec. MSE loss 1.16E+00. Sample per second: 80670\n",
      "FUNK_SVD: Epoch 399 of 500. Elapsed time 1.03 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.17 sec. MSE loss 1.16E+00. Sample per second: 62934\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.91 sec. Users per second: 1226\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0685425, PRECISION_RECALL_MIN_DEN: 0.0775911, RECALL: 0.0380255, MAP: 0.0317854, MAP_MIN_DEN: 0.0354187, MRR: 0.2081783, NDCG: 0.0732052, F1: 0.0489146, HIT_RATE: 0.4472112, ARHR_ALL_HITS: 0.2561041, NOVELTY: 0.0251046, AVERAGE_POPULARITY: 0.5053454, DIVERSITY_MEAN_INTER_LIST: 0.7982438, DIVERSITY_HERFINDAHL: 0.9798111, COVERAGE_ITEM: 0.8624775, COVERAGE_ITEM_CORRECT: 0.0661859, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4460265, DIVERSITY_GINI: 0.1434756, SHANNON_ENTROPY: 7.7667346, RATIO_DIVERSITY_HERFINDAHL: 0.9806116, RATIO_DIVERSITY_GINI: 0.4103450, RATIO_SHANNON_ENTROPY: 0.7182860, RATIO_AVERAGE_POPULARITY: 1.8295310, RATIO_NOVELTY: 0.0850639, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 400 of 500. Elapsed time 1.04 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.52 sec. MSE loss 1.16E+00. Sample per second: 67259\n",
      "FUNK_SVD: Epoch 401 of 500. Elapsed time 1.04 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.93 sec. MSE loss 1.17E+00. Sample per second: 64490\n",
      "FUNK_SVD: Epoch 402 of 500. Elapsed time 1.04 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.25 sec. MSE loss 1.16E+00. Sample per second: 62403\n",
      "FUNK_SVD: Epoch 403 of 500. Elapsed time 1.05 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.74 sec. MSE loss 1.16E+00. Sample per second: 65708\n",
      "FUNK_SVD: Epoch 404 of 500. Elapsed time 1.05 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.01 sec. MSE loss 1.17E+00. Sample per second: 63935\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.85 sec. Users per second: 1241\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0691899, PRECISION_RECALL_MIN_DEN: 0.0783491, RECALL: 0.0384432, MAP: 0.0321150, MAP_MIN_DEN: 0.0357772, MRR: 0.2095896, NDCG: 0.0738875, F1: 0.0494250, HIT_RATE: 0.4500332, ARHR_ALL_HITS: 0.2582427, NOVELTY: 0.0250305, AVERAGE_POPULARITY: 0.5102144, DIVERSITY_MEAN_INTER_LIST: 0.7937886, DIVERSITY_HERFINDAHL: 0.9793657, COVERAGE_ITEM: 0.8550090, COVERAGE_ITEM_CORRECT: 0.0656709, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4488411, DIVERSITY_GINI: 0.1380677, SHANNON_ENTROPY: 7.7017107, RATIO_DIVERSITY_HERFINDAHL: 0.9801658, RATIO_DIVERSITY_GINI: 0.3948781, RATIO_SHANNON_ENTROPY: 0.7122725, RATIO_AVERAGE_POPULARITY: 1.8471584, RATIO_NOVELTY: 0.0848128, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 405 of 500. Elapsed time 1.05 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.51 sec. MSE loss 1.16E+00. Sample per second: 60918\n",
      "FUNK_SVD: Epoch 406 of 500. Elapsed time 1.05 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.70 sec. MSE loss 1.16E+00. Sample per second: 65952\n",
      "FUNK_SVD: Epoch 407 of 500. Elapsed time 1.06 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.99 sec. MSE loss 1.17E+00. Sample per second: 64052\n",
      "FUNK_SVD: Epoch 408 of 500. Elapsed time 1.06 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.23 sec. MSE loss 1.16E+00. Sample per second: 62525\n",
      "FUNK_SVD: Epoch 409 of 500. Elapsed time 1.06 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.74 sec. MSE loss 1.17E+00. Sample per second: 65751\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.96 sec. Users per second: 1214\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0699867, PRECISION_RECALL_MIN_DEN: 0.0791906, RECALL: 0.0388116, MAP: 0.0324295, MAP_MIN_DEN: 0.0361136, MRR: 0.2107686, NDCG: 0.0745258, F1: 0.0499327, HIT_RATE: 0.4538513, ARHR_ALL_HITS: 0.2602539, NOVELTY: 0.0249515, AVERAGE_POPULARITY: 0.5148334, DIVERSITY_MEAN_INTER_LIST: 0.7895402, DIVERSITY_HERFINDAHL: 0.9789409, COVERAGE_ITEM: 0.8470255, COVERAGE_ITEM_CORRECT: 0.0651558, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4526490, DIVERSITY_GINI: 0.1331331, SHANNON_ENTROPY: 7.6392541, RATIO_DIVERSITY_HERFINDAHL: 0.9797407, RATIO_DIVERSITY_GINI: 0.3807651, RATIO_SHANNON_ENTROPY: 0.7064963, RATIO_AVERAGE_POPULARITY: 1.8638807, RATIO_NOVELTY: 0.0845453, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 410 of 500. Elapsed time 1.07 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.13 sec. MSE loss 1.16E+00. Sample per second: 63188\n",
      "FUNK_SVD: Epoch 411 of 500. Elapsed time 1.07 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 7.62 sec. MSE loss 1.16E+00. Sample per second: 84024\n",
      "FUNK_SVD: Epoch 412 of 500. Elapsed time 1.07 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.93 sec. MSE loss 1.16E+00. Sample per second: 64464\n",
      "FUNK_SVD: Epoch 413 of 500. Elapsed time 1.07 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.45 sec. MSE loss 1.16E+00. Sample per second: 61274\n",
      "FUNK_SVD: Epoch 414 of 500. Elapsed time 1.08 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.07 sec. MSE loss 1.16E+00. Sample per second: 63594\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.18 sec. Users per second: 1162\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0707005, PRECISION_RECALL_MIN_DEN: 0.0800015, RECALL: 0.0392186, MAP: 0.0327010, MAP_MIN_DEN: 0.0364091, MRR: 0.2117544, NDCG: 0.0751596, F1: 0.0504512, HIT_RATE: 0.4580013, ARHR_ALL_HITS: 0.2619393, NOVELTY: 0.0248739, AVERAGE_POPULARITY: 0.5196629, DIVERSITY_MEAN_INTER_LIST: 0.7848525, DIVERSITY_HERFINDAHL: 0.9784722, COVERAGE_ITEM: 0.8387844, COVERAGE_ITEM_CORRECT: 0.0648983, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4567881, DIVERSITY_GINI: 0.1277897, SHANNON_ENTROPY: 7.5724215, RATIO_DIVERSITY_HERFINDAHL: 0.9792716, RATIO_DIVERSITY_GINI: 0.3654827, RATIO_SHANNON_ENTROPY: 0.7003155, RATIO_AVERAGE_POPULARITY: 1.8813654, RATIO_NOVELTY: 0.0842824, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 415 of 500. Elapsed time 1.08 hour\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Processed 640256 (100.0%) in 9.66 sec. MSE loss 1.16E+00. Sample per second: 66271\n",
      "FUNK_SVD: Epoch 416 of 500. Elapsed time 1.08 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.17 sec. MSE loss 1.16E+00. Sample per second: 62964\n",
      "FUNK_SVD: Epoch 417 of 500. Elapsed time 1.09 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.26 sec. MSE loss 1.16E+00. Sample per second: 69125\n",
      "FUNK_SVD: Epoch 418 of 500. Elapsed time 1.09 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.48 sec. MSE loss 1.16E+00. Sample per second: 67519\n",
      "FUNK_SVD: Epoch 419 of 500. Elapsed time 1.09 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.41 sec. MSE loss 1.16E+00. Sample per second: 61517\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.82 sec. Users per second: 1249\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0712981, PRECISION_RECALL_MIN_DEN: 0.0806413, RECALL: 0.0395003, MAP: 0.0329616, MAP_MIN_DEN: 0.0366821, MRR: 0.2125624, NDCG: 0.0756297, F1: 0.0508364, HIT_RATE: 0.4603254, ARHR_ALL_HITS: 0.2634440, NOVELTY: 0.0248185, AVERAGE_POPULARITY: 0.5238683, DIVERSITY_MEAN_INTER_LIST: 0.7807564, DIVERSITY_HERFINDAHL: 0.9780627, COVERAGE_ITEM: 0.8323461, COVERAGE_ITEM_CORRECT: 0.0646407, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4591060, DIVERSITY_GINI: 0.1234431, SHANNON_ENTROPY: 7.5150994, RATIO_DIVERSITY_HERFINDAHL: 0.9788617, RATIO_DIVERSITY_GINI: 0.3530515, RATIO_SHANNON_ENTROPY: 0.6950142, RATIO_AVERAGE_POPULARITY: 1.8965904, RATIO_NOVELTY: 0.0840944, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 420 of 500. Elapsed time 1.09 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.46 sec. MSE loss 1.16E+00. Sample per second: 67711\n",
      "FUNK_SVD: Epoch 421 of 500. Elapsed time 1.10 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.59 sec. MSE loss 1.16E+00. Sample per second: 66721\n",
      "FUNK_SVD: Epoch 422 of 500. Elapsed time 1.10 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.81 sec. MSE loss 1.16E+00. Sample per second: 65253\n",
      "FUNK_SVD: Epoch 423 of 500. Elapsed time 1.10 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.93 sec. MSE loss 1.17E+00. Sample per second: 64412\n",
      "FUNK_SVD: Epoch 424 of 500. Elapsed time 1.10 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 8.21 sec. MSE loss 1.16E+00. Sample per second: 77966\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.78 sec. Users per second: 1261\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0719954, PRECISION_RECALL_MIN_DEN: 0.0814201, RECALL: 0.0399108, MAP: 0.0332588, MAP_MIN_DEN: 0.0370011, MRR: 0.2136262, NDCG: 0.0763033, F1: 0.0513536, HIT_RATE: 0.4626494, ARHR_ALL_HITS: 0.2652691, NOVELTY: 0.0247489, AVERAGE_POPULARITY: 0.5283114, DIVERSITY_MEAN_INTER_LIST: 0.7763911, DIVERSITY_HERFINDAHL: 0.9776262, COVERAGE_ITEM: 0.8253927, COVERAGE_ITEM_CORRECT: 0.0636106, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4614238, DIVERSITY_GINI: 0.1186763, SHANNON_ENTROPY: 7.4531645, RATIO_DIVERSITY_HERFINDAHL: 0.9784249, RATIO_DIVERSITY_GINI: 0.3394183, RATIO_SHANNON_ENTROPY: 0.6892863, RATIO_AVERAGE_POPULARITY: 1.9126760, RATIO_NOVELTY: 0.0838587, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 425 of 500. Elapsed time 1.11 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.30 sec. MSE loss 1.16E+00. Sample per second: 68879\n",
      "FUNK_SVD: Epoch 426 of 500. Elapsed time 1.11 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.78 sec. MSE loss 1.16E+00. Sample per second: 65437\n",
      "FUNK_SVD: Epoch 427 of 500. Elapsed time 1.11 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.83 sec. MSE loss 1.16E+00. Sample per second: 65151\n",
      "FUNK_SVD: Epoch 428 of 500. Elapsed time 1.12 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.84 sec. MSE loss 1.16E+00. Sample per second: 65093\n",
      "FUNK_SVD: Epoch 429 of 500. Elapsed time 1.12 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.00 sec. MSE loss 1.16E+00. Sample per second: 63974\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.62 sec. Users per second: 1304\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0726926, PRECISION_RECALL_MIN_DEN: 0.0822012, RECALL: 0.0402843, MAP: 0.0335022, MAP_MIN_DEN: 0.0372543, MRR: 0.2142140, NDCG: 0.0768303, F1: 0.0518402, HIT_RATE: 0.4654714, ARHR_ALL_HITS: 0.2665870, NOVELTY: 0.0246876, AVERAGE_POPULARITY: 0.5324844, DIVERSITY_MEAN_INTER_LIST: 0.7720625, DIVERSITY_HERFINDAHL: 0.9771934, COVERAGE_ITEM: 0.8184394, COVERAGE_ITEM_CORRECT: 0.0633531, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4642384, DIVERSITY_GINI: 0.1144736, SHANNON_ENTROPY: 7.3945647, RATIO_DIVERSITY_HERFINDAHL: 0.9779918, RATIO_DIVERSITY_GINI: 0.3273982, RATIO_SHANNON_ENTROPY: 0.6838669, RATIO_AVERAGE_POPULARITY: 1.9277838, RATIO_NOVELTY: 0.0836512, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 430 of 500. Elapsed time 1.12 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.89 sec. MSE loss 1.16E+00. Sample per second: 64722\n",
      "FUNK_SVD: Epoch 431 of 500. Elapsed time 1.12 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.02 sec. MSE loss 1.16E+00. Sample per second: 63867\n",
      "FUNK_SVD: Epoch 432 of 500. Elapsed time 1.13 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.07 sec. MSE loss 1.16E+00. Sample per second: 70591\n",
      "FUNK_SVD: Epoch 433 of 500. Elapsed time 1.13 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.09 sec. MSE loss 1.16E+00. Sample per second: 70416\n",
      "FUNK_SVD: Epoch 434 of 500. Elapsed time 1.13 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.36 sec. MSE loss 1.16E+00. Sample per second: 68374\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.65 sec. Users per second: 1295\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0734230, PRECISION_RECALL_MIN_DEN: 0.0830192, RECALL: 0.0406540, MAP: 0.0337849, MAP_MIN_DEN: 0.0375517, MRR: 0.2151600, NDCG: 0.0774369, F1: 0.0523320, HIT_RATE: 0.4686255, ARHR_ALL_HITS: 0.2682466, NOVELTY: 0.0246223, AVERAGE_POPULARITY: 0.5364975, DIVERSITY_MEAN_INTER_LIST: 0.7680579, DIVERSITY_HERFINDAHL: 0.9767930, COVERAGE_ITEM: 0.8114860, COVERAGE_ITEM_CORRECT: 0.0630955, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4673841, DIVERSITY_GINI: 0.1101687, SHANNON_ENTROPY: 7.3374326, RATIO_DIVERSITY_HERFINDAHL: 0.9775910, RATIO_DIVERSITY_GINI: 0.3150863, RATIO_SHANNON_ENTROPY: 0.6785832, RATIO_AVERAGE_POPULARITY: 1.9423126, RATIO_NOVELTY: 0.0834299, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 435 of 500. Elapsed time 1.14 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.49 sec. MSE loss 1.16E+00. Sample per second: 67481\n",
      "FUNK_SVD: Epoch 436 of 500. Elapsed time 1.14 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.85 sec. MSE loss 1.16E+00. Sample per second: 93474\n",
      "FUNK_SVD: Epoch 437 of 500. Elapsed time 1.14 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.85 sec. MSE loss 1.16E+00. Sample per second: 93518\n",
      "FUNK_SVD: Epoch 438 of 500. Elapsed time 1.14 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.92 sec. MSE loss 1.16E+00. Sample per second: 92568\n",
      "FUNK_SVD: Epoch 439 of 500. Elapsed time 1.14 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.98 sec. MSE loss 1.16E+00. Sample per second: 91784\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.41 sec. Users per second: 1768\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0739874, PRECISION_RECALL_MIN_DEN: 0.0837556, RECALL: 0.0410313, MAP: 0.0340588, MAP_MIN_DEN: 0.0378528, MRR: 0.2158197, NDCG: 0.0779676, F1: 0.0527880, HIT_RATE: 0.4706175, ARHR_ALL_HITS: 0.2696865, NOVELTY: 0.0245602, AVERAGE_POPULARITY: 0.5403740, DIVERSITY_MEAN_INTER_LIST: 0.7641511, DIVERSITY_HERFINDAHL: 0.9764024, COVERAGE_ITEM: 0.8016997, COVERAGE_ITEM_CORRECT: 0.0615503, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4693709, DIVERSITY_GINI: 0.1059687, SHANNON_ENTROPY: 7.2818704, RATIO_DIVERSITY_HERFINDAHL: 0.9772001, RATIO_DIVERSITY_GINI: 0.3030739, RATIO_SHANNON_ENTROPY: 0.6734446, RATIO_AVERAGE_POPULARITY: 1.9563471, RATIO_NOVELTY: 0.0832194, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 440 of 500. Elapsed time 1.15 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 5.80 sec. MSE loss 1.16E+00. Sample per second: 110329\n",
      "FUNK_SVD: Epoch 441 of 500. Elapsed time 1.15 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.84 sec. MSE loss 1.16E+00. Sample per second: 93651\n",
      "FUNK_SVD: Epoch 442 of 500. Elapsed time 1.15 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.87 sec. MSE loss 1.16E+00. Sample per second: 93249\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Epoch 443 of 500. Elapsed time 1.15 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.89 sec. MSE loss 1.16E+00. Sample per second: 92878\n",
      "FUNK_SVD: Epoch 444 of 500. Elapsed time 1.15 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.41 sec. MSE loss 1.16E+00. Sample per second: 99904\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.27 sec. Users per second: 1840\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0746680, PRECISION_RECALL_MIN_DEN: 0.0844733, RECALL: 0.0413233, MAP: 0.0343516, MAP_MIN_DEN: 0.0381705, MRR: 0.2169731, NDCG: 0.0786267, F1: 0.0532027, HIT_RATE: 0.4732736, ARHR_ALL_HITS: 0.2715508, NOVELTY: 0.0245021, AVERAGE_POPULARITY: 0.5444604, DIVERSITY_MEAN_INTER_LIST: 0.7597723, DIVERSITY_HERFINDAHL: 0.9759646, COVERAGE_ITEM: 0.7950039, COVERAGE_ITEM_CORRECT: 0.0615503, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4720199, DIVERSITY_GINI: 0.1018547, SHANNON_ENTROPY: 7.2237119, RATIO_DIVERSITY_HERFINDAHL: 0.9767619, RATIO_DIVERSITY_GINI: 0.2913079, RATIO_SHANNON_ENTROPY: 0.6680660, RATIO_AVERAGE_POPULARITY: 1.9711413, RATIO_NOVELTY: 0.0830226, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 445 of 500. Elapsed time 1.16 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.78 sec. MSE loss 1.16E+00. Sample per second: 94484\n",
      "FUNK_SVD: Epoch 446 of 500. Elapsed time 1.16 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.78 sec. MSE loss 1.16E+00. Sample per second: 94462\n",
      "FUNK_SVD: Epoch 447 of 500. Elapsed time 1.16 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.83 sec. MSE loss 1.16E+00. Sample per second: 93775\n",
      "FUNK_SVD: Epoch 448 of 500. Elapsed time 1.16 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.30 sec. MSE loss 1.15E+00. Sample per second: 101564\n",
      "FUNK_SVD: Epoch 449 of 500. Elapsed time 1.16 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.20 sec. MSE loss 1.16E+00. Sample per second: 103313\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.35 sec. Users per second: 1799\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0752490, PRECISION_RECALL_MIN_DEN: 0.0850958, RECALL: 0.0415607, MAP: 0.0345535, MAP_MIN_DEN: 0.0383714, MRR: 0.2171605, NDCG: 0.0789587, F1: 0.0535470, HIT_RATE: 0.4742696, ARHR_ALL_HITS: 0.2724480, NOVELTY: 0.0244407, AVERAGE_POPULARITY: 0.5483954, DIVERSITY_MEAN_INTER_LIST: 0.7557299, DIVERSITY_HERFINDAHL: 0.9755604, COVERAGE_ITEM: 0.7849601, COVERAGE_ITEM_CORRECT: 0.0610353, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4730132, DIVERSITY_GINI: 0.0974747, SHANNON_ENTROPY: 7.1654401, RATIO_DIVERSITY_HERFINDAHL: 0.9763574, RATIO_DIVERSITY_GINI: 0.2787809, RATIO_SHANNON_ENTROPY: 0.6626769, RATIO_AVERAGE_POPULARITY: 1.9853872, RATIO_NOVELTY: 0.0828143, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 450 of 500. Elapsed time 1.16 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 6.63 sec. MSE loss 1.16E+00. Sample per second: 96544\n",
      "FUNK_SVD: Epoch 451 of 500. Elapsed time 1.17 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 5.71 sec. MSE loss 1.16E+00. Sample per second: 112114\n",
      "FUNK_SVD: Epoch 452 of 500. Elapsed time 1.17 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 7.27 sec. MSE loss 1.16E+00. Sample per second: 88044\n",
      "FUNK_SVD: Epoch 453 of 500. Elapsed time 1.17 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.09 sec. MSE loss 1.15E+00. Sample per second: 70455\n",
      "FUNK_SVD: Epoch 454 of 500. Elapsed time 1.17 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.78 sec. MSE loss 1.15E+00. Sample per second: 65486\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.11 sec. Users per second: 1178\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0759628, PRECISION_RECALL_MIN_DEN: 0.0858051, RECALL: 0.0418463, MAP: 0.0348589, MAP_MIN_DEN: 0.0386822, MRR: 0.2180000, NDCG: 0.0794697, F1: 0.0539646, HIT_RATE: 0.4770916, ARHR_ALL_HITS: 0.2740732, NOVELTY: 0.0243836, AVERAGE_POPULARITY: 0.5522509, DIVERSITY_MEAN_INTER_LIST: 0.7516674, DIVERSITY_HERFINDAHL: 0.9751543, COVERAGE_ITEM: 0.7792944, COVERAGE_ITEM_CORRECT: 0.0612928, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4758278, DIVERSITY_GINI: 0.0936167, SHANNON_ENTROPY: 7.1085975, RATIO_DIVERSITY_HERFINDAHL: 0.9759509, RATIO_DIVERSITY_GINI: 0.2677468, RATIO_SHANNON_ENTROPY: 0.6574200, RATIO_AVERAGE_POPULARITY: 1.9993456, RATIO_NOVELTY: 0.0826210, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 455 of 500. Elapsed time 1.18 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.10 sec. MSE loss 1.16E+00. Sample per second: 63415\n",
      "FUNK_SVD: Epoch 456 of 500. Elapsed time 1.18 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.81 sec. MSE loss 1.16E+00. Sample per second: 65276\n",
      "FUNK_SVD: Epoch 457 of 500. Elapsed time 1.18 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.36 sec. MSE loss 1.15E+00. Sample per second: 61777\n",
      "FUNK_SVD: Epoch 458 of 500. Elapsed time 1.18 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.67 sec. MSE loss 1.16E+00. Sample per second: 66223\n",
      "FUNK_SVD: Epoch 459 of 500. Elapsed time 1.19 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.72 sec. MSE loss 1.16E+00. Sample per second: 65850\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.84 sec. Users per second: 1244\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0764940, PRECISION_RECALL_MIN_DEN: 0.0863683, RECALL: 0.0420568, MAP: 0.0350933, MAP_MIN_DEN: 0.0389301, MRR: 0.2187810, NDCG: 0.0799722, F1: 0.0542736, HIT_RATE: 0.4790837, ARHR_ALL_HITS: 0.2754842, NOVELTY: 0.0243326, AVERAGE_POPULARITY: 0.5557718, DIVERSITY_MEAN_INTER_LIST: 0.7475992, DIVERSITY_HERFINDAHL: 0.9747475, COVERAGE_ITEM: 0.7695081, COVERAGE_ITEM_CORRECT: 0.0605202, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4778146, DIVERSITY_GINI: 0.0900689, SHANNON_ENTROPY: 7.0562742, RATIO_DIVERSITY_HERFINDAHL: 0.9755438, RATIO_DIVERSITY_GINI: 0.2576002, RATIO_SHANNON_ENTROPY: 0.6525810, RATIO_AVERAGE_POPULARITY: 2.0120925, RATIO_NOVELTY: 0.0824482, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 460 of 500. Elapsed time 1.19 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.84 sec. MSE loss 1.15E+00. Sample per second: 65014\n",
      "FUNK_SVD: Epoch 461 of 500. Elapsed time 1.19 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.04 sec. MSE loss 1.16E+00. Sample per second: 63776\n",
      "FUNK_SVD: Epoch 462 of 500. Elapsed time 1.20 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.15 sec. MSE loss 1.16E+00. Sample per second: 69966\n",
      "FUNK_SVD: Epoch 463 of 500. Elapsed time 1.20 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.27 sec. MSE loss 1.16E+00. Sample per second: 69087\n",
      "FUNK_SVD: Epoch 464 of 500. Elapsed time 1.20 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.53 sec. MSE loss 1.15E+00. Sample per second: 67155\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.66 sec. Users per second: 1293\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0770584, PRECISION_RECALL_MIN_DEN: 0.0869438, RECALL: 0.0422682, MAP: 0.0353307, MAP_MIN_DEN: 0.0391671, MRR: 0.2192993, NDCG: 0.0804280, F1: 0.0545917, HIT_RATE: 0.4807437, ARHR_ALL_HITS: 0.2767422, NOVELTY: 0.0242753, AVERAGE_POPULARITY: 0.5594433, DIVERSITY_MEAN_INTER_LIST: 0.7432714, DIVERSITY_HERFINDAHL: 0.9743148, COVERAGE_ITEM: 0.7571465, COVERAGE_ITEM_CORRECT: 0.0594901, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4794702, DIVERSITY_GINI: 0.0863099, SHANNON_ENTROPY: 7.0008670, RATIO_DIVERSITY_HERFINDAHL: 0.9751108, RATIO_DIVERSITY_GINI: 0.2468493, RATIO_SHANNON_ENTROPY: 0.6474568, RATIO_AVERAGE_POPULARITY: 2.0253846, RATIO_NOVELTY: 0.0822539, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 465 of 500. Elapsed time 1.20 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 8.27 sec. MSE loss 1.16E+00. Sample per second: 77434\n",
      "FUNK_SVD: Epoch 466 of 500. Elapsed time 1.21 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.29 sec. MSE loss 1.15E+00. Sample per second: 68942\n",
      "FUNK_SVD: Epoch 467 of 500. Elapsed time 1.21 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.47 sec. MSE loss 1.16E+00. Sample per second: 67566\n",
      "FUNK_SVD: Epoch 468 of 500. Elapsed time 1.21 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.77 sec. MSE loss 1.15E+00. Sample per second: 65519\n",
      "FUNK_SVD: Epoch 469 of 500. Elapsed time 1.21 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.02 sec. MSE loss 1.15E+00. Sample per second: 63911\n",
      "FUNK_SVD: Validation begins...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.75 sec. Users per second: 1269\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0775896, PRECISION_RECALL_MIN_DEN: 0.0875525, RECALL: 0.0425381, MAP: 0.0355919, MAP_MIN_DEN: 0.0394462, MRR: 0.2201345, NDCG: 0.0810179, F1: 0.0549501, HIT_RATE: 0.4824037, ARHR_ALL_HITS: 0.2783166, NOVELTY: 0.0242273, AVERAGE_POPULARITY: 0.5625959, DIVERSITY_MEAN_INTER_LIST: 0.7397524, DIVERSITY_HERFINDAHL: 0.9739630, COVERAGE_ITEM: 0.7465877, COVERAGE_ITEM_CORRECT: 0.0592326, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4811258, DIVERSITY_GINI: 0.0830703, SHANNON_ENTROPY: 6.9532279, RATIO_DIVERSITY_HERFINDAHL: 0.9747587, RATIO_DIVERSITY_GINI: 0.2375837, RATIO_SHANNON_ENTROPY: 0.6430510, RATIO_AVERAGE_POPULARITY: 2.0367983, RATIO_NOVELTY: 0.0820914, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 470 of 500. Elapsed time 1.22 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.04 sec. MSE loss 1.15E+00. Sample per second: 63745\n",
      "FUNK_SVD: Epoch 471 of 500. Elapsed time 1.22 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.38 sec. MSE loss 1.15E+00. Sample per second: 68282\n",
      "FUNK_SVD: Epoch 472 of 500. Elapsed time 1.22 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.67 sec. MSE loss 1.15E+00. Sample per second: 66206\n",
      "FUNK_SVD: Epoch 473 of 500. Elapsed time 1.23 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.88 sec. MSE loss 1.16E+00. Sample per second: 64832\n",
      "FUNK_SVD: Epoch 474 of 500. Elapsed time 1.23 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.05 sec. MSE loss 1.15E+00. Sample per second: 63694\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.84 sec. Users per second: 1244\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0782371, PRECISION_RECALL_MIN_DEN: 0.0882536, RECALL: 0.0428377, MAP: 0.0358980, MAP_MIN_DEN: 0.0397693, MRR: 0.2209463, NDCG: 0.0816045, F1: 0.0553624, HIT_RATE: 0.4835657, ARHR_ALL_HITS: 0.2800205, NOVELTY: 0.0241732, AVERAGE_POPULARITY: 0.5661076, DIVERSITY_MEAN_INTER_LIST: 0.7358795, DIVERSITY_HERFINDAHL: 0.9735757, COVERAGE_ITEM: 0.7365439, COVERAGE_ITEM_CORRECT: 0.0584600, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4822848, DIVERSITY_GINI: 0.0795567, SHANNON_ENTROPY: 6.9002142, RATIO_DIVERSITY_HERFINDAHL: 0.9743711, RATIO_DIVERSITY_GINI: 0.2275347, RATIO_SHANNON_ENTROPY: 0.6381482, RATIO_AVERAGE_POPULARITY: 2.0495117, RATIO_NOVELTY: 0.0819079, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 475 of 500. Elapsed time 1.23 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.05 sec. MSE loss 1.15E+00. Sample per second: 63701\n",
      "FUNK_SVD: Epoch 476 of 500. Elapsed time 1.24 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.36 sec. MSE loss 1.15E+00. Sample per second: 68345\n",
      "FUNK_SVD: Epoch 477 of 500. Elapsed time 1.24 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.87 sec. MSE loss 1.16E+00. Sample per second: 64856\n",
      "FUNK_SVD: Epoch 478 of 500. Elapsed time 1.24 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 8.65 sec. MSE loss 1.15E+00. Sample per second: 74013\n",
      "FUNK_SVD: Epoch 479 of 500. Elapsed time 1.24 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.91 sec. MSE loss 1.15E+00. Sample per second: 64620\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.88 sec. Users per second: 1235\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0786521, PRECISION_RECALL_MIN_DEN: 0.0887724, RECALL: 0.0431755, MAP: 0.0360486, MAP_MIN_DEN: 0.0399322, MRR: 0.2213851, NDCG: 0.0819742, F1: 0.0557483, HIT_RATE: 0.4860558, ARHR_ALL_HITS: 0.2809439, NOVELTY: 0.0241219, AVERAGE_POPULARITY: 0.5693201, DIVERSITY_MEAN_INTER_LIST: 0.7323170, DIVERSITY_HERFINDAHL: 0.9732195, COVERAGE_ITEM: 0.7270152, COVERAGE_ITEM_CORRECT: 0.0574298, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4847682, DIVERSITY_GINI: 0.0764993, SHANNON_ENTROPY: 6.8514318, RATIO_DIVERSITY_HERFINDAHL: 0.9740146, RATIO_DIVERSITY_GINI: 0.2187904, RATIO_SHANNON_ENTROPY: 0.6336367, RATIO_AVERAGE_POPULARITY: 2.0611420, RATIO_NOVELTY: 0.0817343, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 480 of 500. Elapsed time 1.25 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.12 sec. MSE loss 1.16E+00. Sample per second: 63248\n",
      "FUNK_SVD: Epoch 481 of 500. Elapsed time 1.25 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.51 sec. MSE loss 1.16E+00. Sample per second: 67310\n",
      "FUNK_SVD: Epoch 482 of 500. Elapsed time 1.25 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.99 sec. MSE loss 1.15E+00. Sample per second: 64103\n",
      "FUNK_SVD: Epoch 483 of 500. Elapsed time 1.25 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.28 sec. MSE loss 1.16E+00. Sample per second: 62272\n",
      "FUNK_SVD: Epoch 484 of 500. Elapsed time 1.26 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.57 sec. MSE loss 1.15E+00. Sample per second: 66893\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.98 sec. Users per second: 1211\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0792663, PRECISION_RECALL_MIN_DEN: 0.0894340, RECALL: 0.0434767, MAP: 0.0362479, MAP_MIN_DEN: 0.0401363, MRR: 0.2219500, NDCG: 0.0823845, F1: 0.0561537, HIT_RATE: 0.4880478, ARHR_ALL_HITS: 0.2820396, NOVELTY: 0.0240720, AVERAGE_POPULARITY: 0.5725705, DIVERSITY_MEAN_INTER_LIST: 0.7283346, DIVERSITY_HERFINDAHL: 0.9728214, COVERAGE_ITEM: 0.7187741, COVERAGE_ITEM_CORRECT: 0.0569148, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4867550, DIVERSITY_GINI: 0.0735609, SHANNON_ENTROPY: 6.8016897, RATIO_DIVERSITY_HERFINDAHL: 0.9736161, RATIO_DIVERSITY_GINI: 0.2103867, RATIO_SHANNON_ENTROPY: 0.6290364, RATIO_AVERAGE_POPULARITY: 2.0729098, RATIO_NOVELTY: 0.0815650, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 485 of 500. Elapsed time 1.26 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.01 sec. MSE loss 1.15E+00. Sample per second: 63923\n",
      "FUNK_SVD: Epoch 486 of 500. Elapsed time 1.26 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.33 sec. MSE loss 1.15E+00. Sample per second: 68600\n",
      "FUNK_SVD: Epoch 487 of 500. Elapsed time 1.27 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.56 sec. MSE loss 1.16E+00. Sample per second: 66973\n",
      "FUNK_SVD: Epoch 488 of 500. Elapsed time 1.27 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.80 sec. MSE loss 1.15E+00. Sample per second: 65287\n",
      "FUNK_SVD: Epoch 489 of 500. Elapsed time 1.27 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.21 sec. MSE loss 1.15E+00. Sample per second: 62717\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.76 sec. Users per second: 1264\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0797477, PRECISION_RECALL_MIN_DEN: 0.0900418, RECALL: 0.0437941, MAP: 0.0364841, MAP_MIN_DEN: 0.0403928, MRR: 0.2226962, NDCG: 0.0828209, F1: 0.0565392, HIT_RATE: 0.4900398, ARHR_ALL_HITS: 0.2833456, NOVELTY: 0.0240284, AVERAGE_POPULARITY: 0.5755296, DIVERSITY_MEAN_INTER_LIST: 0.7250196, DIVERSITY_HERFINDAHL: 0.9724899, COVERAGE_ITEM: 0.7092454, COVERAGE_ITEM_CORRECT: 0.0563997, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4887417, DIVERSITY_GINI: 0.0706667, SHANNON_ENTROPY: 6.7553892, RATIO_DIVERSITY_HERFINDAHL: 0.9732844, RATIO_DIVERSITY_GINI: 0.2021091, RATIO_SHANNON_ENTROPY: 0.6247544, RATIO_AVERAGE_POPULARITY: 2.0836229, RATIO_NOVELTY: 0.0814174, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 490 of 500. Elapsed time 1.28 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.23 sec. MSE loss 1.15E+00. Sample per second: 69399\n",
      "FUNK_SVD: Epoch 491 of 500. Elapsed time 1.28 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 7.67 sec. MSE loss 1.15E+00. Sample per second: 83467\n",
      "FUNK_SVD: Epoch 492 of 500. Elapsed time 1.28 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.80 sec. MSE loss 1.15E+00. Sample per second: 65333\n",
      "FUNK_SVD: Epoch 493 of 500. Elapsed time 1.28 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.04 sec. MSE loss 1.15E+00. Sample per second: 63720\n",
      "FUNK_SVD: Epoch 494 of 500. Elapsed time 1.28 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.31 sec. MSE loss 1.15E+00. Sample per second: 68753\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.72 sec. Users per second: 1276\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0803785, PRECISION_RECALL_MIN_DEN: 0.0907590, RECALL: 0.0440961, MAP: 0.0367211, MAP_MIN_DEN: 0.0406357, MRR: 0.2233955, NDCG: 0.0832880, F1: 0.0569495, HIT_RATE: 0.4928619, ARHR_ALL_HITS: 0.2846664, NOVELTY: 0.0239847, AVERAGE_POPULARITY: 0.5784467, DIVERSITY_MEAN_INTER_LIST: 0.7213530, DIVERSITY_HERFINDAHL: 0.9721233, COVERAGE_ITEM: 0.6999742, COVERAGE_ITEM_CORRECT: 0.0558846, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4915563, DIVERSITY_GINI: 0.0679097, SHANNON_ENTROPY: 6.7088987, RATIO_DIVERSITY_HERFINDAHL: 0.9729175, RATIO_DIVERSITY_GINI: 0.1942241, RATIO_SHANNON_ENTROPY: 0.6204549, RATIO_AVERAGE_POPULARITY: 2.0941838, RATIO_NOVELTY: 0.0812692, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 495 of 500. Elapsed time 1.29 hour\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Processed 640256 (100.0%) in 9.26 sec. MSE loss 1.15E+00. Sample per second: 69121\n",
      "FUNK_SVD: Epoch 496 of 500. Elapsed time 1.29 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.48 sec. MSE loss 1.15E+00. Sample per second: 67557\n",
      "FUNK_SVD: Epoch 497 of 500. Elapsed time 1.29 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.70 sec. MSE loss 1.15E+00. Sample per second: 65980\n",
      "FUNK_SVD: Epoch 498 of 500. Elapsed time 1.30 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 10.19 sec. MSE loss 1.15E+00. Sample per second: 62810\n",
      "FUNK_SVD: Epoch 499 of 500. Elapsed time 1.30 hour\n",
      "FUNK_SVD: Processed 640256 (100.0%) in 9.41 sec. MSE loss 1.15E+00. Sample per second: 68065\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.74 sec. Users per second: 1272\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0808433, PRECISION_RECALL_MIN_DEN: 0.0912505, RECALL: 0.0442404, MAP: 0.0369268, MAP_MIN_DEN: 0.0408399, MRR: 0.2237674, NDCG: 0.0836874, F1: 0.0571864, HIT_RATE: 0.4938579, ARHR_ALL_HITS: 0.2856503, NOVELTY: 0.0239375, AVERAGE_POPULARITY: 0.5813469, DIVERSITY_MEAN_INTER_LIST: 0.7178521, DIVERSITY_HERFINDAHL: 0.9717733, COVERAGE_ITEM: 0.6912181, COVERAGE_ITEM_CORRECT: 0.0548545, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4925497, DIVERSITY_GINI: 0.0653199, SHANNON_ENTROPY: 6.6634003, RATIO_DIVERSITY_HERFINDAHL: 0.9725672, RATIO_DIVERSITY_GINI: 0.1868171, RATIO_SHANNON_ENTROPY: 0.6162471, RATIO_AVERAGE_POPULARITY: 2.1046835, RATIO_NOVELTY: 0.0811093, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 500 of 500. Elapsed time 1.30 hour\n",
      "FUNK_SVD: Terminating at epoch 500. Best value for 'MAP' at epoch 500 is 0.0369. Elapsed time 1.30 hour\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.73 sec. Users per second: 1274\n",
      "SearchBayesianSkopt: Config 2 is suboptimal. Config: {'epochs': 500, 'num_factors': 122, 'sgd_mode': 'sgd', 'batch_size': 256, 'item_reg': 0.006261745893648571, 'user_reg': 0.0002345781923232971, 'learning_rate': 0.00016732948647785252} - results: PRECISION: 0.0808433, PRECISION_RECALL_MIN_DEN: 0.0912505, RECALL: 0.0442404, MAP: 0.0369268, MAP_MIN_DEN: 0.0408399, MRR: 0.2237674, NDCG: 0.0836874, F1: 0.0571864, HIT_RATE: 0.4938579, ARHR_ALL_HITS: 0.2856503, NOVELTY: 0.0239375, AVERAGE_POPULARITY: 0.5813469, DIVERSITY_MEAN_INTER_LIST: 0.7178521, DIVERSITY_HERFINDAHL: 0.9717733, COVERAGE_ITEM: 0.6912181, COVERAGE_ITEM_CORRECT: 0.0548545, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4925497, DIVERSITY_GINI: 0.0653199, SHANNON_ENTROPY: 6.6634003, RATIO_DIVERSITY_HERFINDAHL: 0.9725672, RATIO_DIVERSITY_GINI: 0.1868171, RATIO_SHANNON_ENTROPY: 0.6162471, RATIO_AVERAGE_POPULARITY: 2.1046835, RATIO_NOVELTY: 0.0811093, \n",
      "\n",
      "Iteration No: 3 ended. Evaluation done at random point.\n",
      "Time taken: 4696.7959\n",
      "Function value obtained: -0.0369\n",
      "Current minimum: -0.0505\n",
      "Iteration No: 4 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'epochs': 500, 'num_factors': 48, 'sgd_mode': 'sgd', 'batch_size': 2, 'item_reg': 0.00015736081658333225, 'user_reg': 2.585551202307481e-05, 'learning_rate': 0.008652038142191442}\n",
      "MatrixFactorization_FunkSVD_Cython_Recommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 4.65 sec. MSE loss 1.02E+00. Sample per second: 137583\n",
      "FUNK_SVD: Epoch 1 of 500. Elapsed time 3.73 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 4.11 sec. MSE loss 8.74E-01. Sample per second: 155837\n",
      "FUNK_SVD: Epoch 2 of 500. Elapsed time 7.18 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.82 sec. MSE loss 8.16E-01. Sample per second: 167575\n",
      "FUNK_SVD: Epoch 3 of 500. Elapsed time 10.90 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 4.29 sec. MSE loss 7.77E-01. Sample per second: 149008\n",
      "FUNK_SVD: Epoch 4 of 500. Elapsed time 14.37 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.84 sec. MSE loss 7.44E-01. Sample per second: 166709\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.76 sec. Users per second: 1266\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0492032, PRECISION_RECALL_MIN_DEN: 0.0539265, RECALL: 0.0241321, MAP: 0.0207454, MAP_MIN_DEN: 0.0225452, MRR: 0.1425381, NDCG: 0.0502701, F1: 0.0323821, HIT_RATE: 0.3378154, ARHR_ALL_HITS: 0.1703717, NOVELTY: 0.0256007, AVERAGE_POPULARITY: 0.3766898, DIVERSITY_MEAN_INTER_LIST: 0.7186340, DIVERSITY_HERFINDAHL: 0.9718515, COVERAGE_ITEM: 0.0463559, COVERAGE_ITEM_CORRECT: 0.0252382, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3369205, DIVERSITY_GINI: 0.0099193, SHANNON_ENTROPY: 5.6623843, RATIO_DIVERSITY_HERFINDAHL: 0.9726454, RATIO_DIVERSITY_GINI: 0.0283694, RATIO_SHANNON_ENTROPY: 0.5236707, RATIO_AVERAGE_POPULARITY: 1.3637518, RATIO_NOVELTY: 0.0867449, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 5 of 500. Elapsed time 22.68 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 4.34 sec. MSE loss 7.05E-01. Sample per second: 147399\n",
      "FUNK_SVD: Epoch 6 of 500. Elapsed time 26.42 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.74 sec. MSE loss 6.63E-01. Sample per second: 170824\n",
      "FUNK_SVD: Epoch 7 of 500. Elapsed time 29.82 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 4.43 sec. MSE loss 6.25E-01. Sample per second: 144638\n",
      "FUNK_SVD: Epoch 8 of 500. Elapsed time 33.50 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.90 sec. MSE loss 5.89E-01. Sample per second: 164277\n",
      "FUNK_SVD: Epoch 9 of 500. Elapsed time 36.97 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.04 sec. MSE loss 5.54E-01. Sample per second: 210021\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.66 sec. Users per second: 1292\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0583665, PRECISION_RECALL_MIN_DEN: 0.0644439, RECALL: 0.0298767, MAP: 0.0256909, MAP_MIN_DEN: 0.0277896, MRR: 0.1698463, NDCG: 0.0596481, F1: 0.0395226, HIT_RATE: 0.3897742, ARHR_ALL_HITS: 0.2070506, NOVELTY: 0.0248794, AVERAGE_POPULARITY: 0.4402573, DIVERSITY_MEAN_INTER_LIST: 0.8564625, DIVERSITY_HERFINDAHL: 0.9856320, COVERAGE_ITEM: 0.0839557, COVERAGE_ITEM_CORRECT: 0.0417203, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3887417, DIVERSITY_GINI: 0.0196775, SHANNON_ENTROPY: 6.6673205, RATIO_DIVERSITY_HERFINDAHL: 0.9864373, RATIO_DIVERSITY_GINI: 0.0562784, RATIO_SHANNON_ENTROPY: 0.6166096, RATIO_AVERAGE_POPULARITY: 1.5938886, RATIO_NOVELTY: 0.0843008, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 10 of 500. Elapsed time 43.80 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 4.45 sec. MSE loss 5.24E-01. Sample per second: 143715\n",
      "FUNK_SVD: Epoch 11 of 500. Elapsed time 47.53 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.94 sec. MSE loss 4.94E-01. Sample per second: 162308\n",
      "FUNK_SVD: Epoch 12 of 500. Elapsed time 51.02 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 4.50 sec. MSE loss 4.66E-01. Sample per second: 142281\n",
      "FUNK_SVD: Epoch 13 of 500. Elapsed time 54.57 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 4.20 sec. MSE loss 4.46E-01. Sample per second: 152247\n",
      "FUNK_SVD: Epoch 14 of 500. Elapsed time 58.28 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.65 sec. MSE loss 4.23E-01. Sample per second: 175094\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.78 sec. Users per second: 1261\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0592961, PRECISION_RECALL_MIN_DEN: 0.0646817, RECALL: 0.0289759, MAP: 0.0255706, MAP_MIN_DEN: 0.0275113, MRR: 0.1668970, NDCG: 0.0587116, F1: 0.0389287, HIT_RATE: 0.3944223, ARHR_ALL_HITS: 0.2047902, NOVELTY: 0.0248007, AVERAGE_POPULARITY: 0.4369532, DIVERSITY_MEAN_INTER_LIST: 0.9090424, DIVERSITY_HERFINDAHL: 0.9908891, COVERAGE_ITEM: 0.1210404, COVERAGE_ITEM_CORRECT: 0.0574298, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3933775, DIVERSITY_GINI: 0.0305121, SHANNON_ENTROPY: 7.3095671, RATIO_DIVERSITY_HERFINDAHL: 0.9916987, RATIO_DIVERSITY_GINI: 0.0872657, RATIO_SHANNON_ENTROPY: 0.6760061, RATIO_AVERAGE_POPULARITY: 1.5819266, RATIO_NOVELTY: 0.0840342, \n",
      "\n",
      "FUNK_SVD: Epoch 15 of 500. Elapsed time 1.11 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 4.11 sec. MSE loss 4.05E-01. Sample per second: 155930\n",
      "FUNK_SVD: Epoch 16 of 500. Elapsed time 1.17 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.54 sec. MSE loss 3.84E-01. Sample per second: 180532\n",
      "FUNK_SVD: Epoch 17 of 500. Elapsed time 1.23 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 4.27 sec. MSE loss 3.70E-01. Sample per second: 149843\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Epoch 18 of 500. Elapsed time 1.29 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.69 sec. MSE loss 3.55E-01. Sample per second: 173198\n",
      "FUNK_SVD: Epoch 19 of 500. Elapsed time 1.35 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 4.30 sec. MSE loss 3.43E-01. Sample per second: 148969\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.83 sec. Users per second: 1246\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0600764, PRECISION_RECALL_MIN_DEN: 0.0649755, RECALL: 0.0278546, MAP: 0.0255623, MAP_MIN_DEN: 0.0274350, MRR: 0.1640084, NDCG: 0.0582670, F1: 0.0380618, HIT_RATE: 0.3929283, ARHR_ALL_HITS: 0.2027927, NOVELTY: 0.0248901, AVERAGE_POPULARITY: 0.4223439, DIVERSITY_MEAN_INTER_LIST: 0.9304402, DIVERSITY_HERFINDAHL: 0.9930286, COVERAGE_ITEM: 0.1560649, COVERAGE_ITEM_CORRECT: 0.0715941, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3918874, DIVERSITY_GINI: 0.0401247, SHANNON_ENTROPY: 7.7060191, RATIO_DIVERSITY_HERFINDAHL: 0.9938398, RATIO_DIVERSITY_GINI: 0.1147581, RATIO_SHANNON_ENTROPY: 0.7126709, RATIO_AVERAGE_POPULARITY: 1.5290358, RATIO_NOVELTY: 0.0843371, \n",
      "\n",
      "FUNK_SVD: Epoch 20 of 500. Elapsed time 1.49 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.85 sec. MSE loss 3.29E-01. Sample per second: 166459\n",
      "FUNK_SVD: Epoch 21 of 500. Elapsed time 1.55 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 4.65 sec. MSE loss 3.18E-01. Sample per second: 137526\n",
      "FUNK_SVD: Epoch 22 of 500. Elapsed time 1.61 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 4.12 sec. MSE loss 3.09E-01. Sample per second: 155382\n",
      "FUNK_SVD: Epoch 23 of 500. Elapsed time 1.67 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.93 sec. MSE loss 3.02E-01. Sample per second: 163051\n",
      "FUNK_SVD: Epoch 24 of 500. Elapsed time 1.73 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 4.36 sec. MSE loss 2.92E-01. Sample per second: 146763\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.84 sec. Users per second: 1244\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0598108, PRECISION_RECALL_MIN_DEN: 0.0644520, RECALL: 0.0269541, MAP: 0.0253722, MAP_MIN_DEN: 0.0270736, MRR: 0.1606931, NDCG: 0.0571023, F1: 0.0371613, HIT_RATE: 0.3854582, ARHR_ALL_HITS: 0.2002638, NOVELTY: 0.0249617, AVERAGE_POPULARITY: 0.4119398, DIVERSITY_MEAN_INTER_LIST: 0.9410804, DIVERSITY_HERFINDAHL: 0.9940924, COVERAGE_ITEM: 0.1820757, COVERAGE_ITEM_CORRECT: 0.0793201, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3844371, DIVERSITY_GINI: 0.0475839, SHANNON_ENTROPY: 7.9520013, RATIO_DIVERSITY_HERFINDAHL: 0.9949046, RATIO_DIVERSITY_GINI: 0.1360915, RATIO_SHANNON_ENTROPY: 0.7354199, RATIO_AVERAGE_POPULARITY: 1.4913692, RATIO_NOVELTY: 0.0845798, \n",
      "\n",
      "FUNK_SVD: Epoch 25 of 500. Elapsed time 1.87 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.96 sec. MSE loss 2.85E-01. Sample per second: 161490\n",
      "FUNK_SVD: Epoch 26 of 500. Elapsed time 1.93 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 4.36 sec. MSE loss 2.76E-01. Sample per second: 146731\n",
      "FUNK_SVD: Epoch 27 of 500. Elapsed time 1.99 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 4.15 sec. MSE loss 2.70E-01. Sample per second: 154068\n",
      "FUNK_SVD: Epoch 28 of 500. Elapsed time 2.05 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.60 sec. MSE loss 2.61E-01. Sample per second: 177423\n",
      "FUNK_SVD: Epoch 29 of 500. Elapsed time 2.11 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 4.11 sec. MSE loss 2.59E-01. Sample per second: 155652\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.68 sec. Users per second: 1287\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0600598, PRECISION_RECALL_MIN_DEN: 0.0644535, RECALL: 0.0260629, MAP: 0.0256583, MAP_MIN_DEN: 0.0271755, MRR: 0.1611190, NDCG: 0.0564760, F1: 0.0363513, HIT_RATE: 0.3831341, ARHR_ALL_HITS: 0.2012345, NOVELTY: 0.0250618, AVERAGE_POPULARITY: 0.4004860, DIVERSITY_MEAN_INTER_LIST: 0.9489353, DIVERSITY_HERFINDAHL: 0.9948778, COVERAGE_ITEM: 0.2047386, COVERAGE_ITEM_CORRECT: 0.0901365, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3821192, DIVERSITY_GINI: 0.0556759, SHANNON_ENTROPY: 8.1757245, RATIO_DIVERSITY_HERFINDAHL: 0.9956906, RATIO_DIVERSITY_GINI: 0.1592350, RATIO_SHANNON_ENTROPY: 0.7561104, RATIO_AVERAGE_POPULARITY: 1.4499025, RATIO_NOVELTY: 0.0849190, \n",
      "\n",
      "FUNK_SVD: Epoch 30 of 500. Elapsed time 2.25 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 4.44 sec. MSE loss 2.51E-01. Sample per second: 144154\n",
      "FUNK_SVD: Epoch 31 of 500. Elapsed time 2.31 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.94 sec. MSE loss 2.46E-01. Sample per second: 162315\n",
      "FUNK_SVD: Epoch 32 of 500. Elapsed time 2.37 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 4.63 sec. MSE loss 2.42E-01. Sample per second: 138195\n",
      "FUNK_SVD: Epoch 33 of 500. Elapsed time 2.43 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 4.03 sec. MSE loss 2.36E-01. Sample per second: 158701\n",
      "FUNK_SVD: Epoch 34 of 500. Elapsed time 2.49 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.83 sec. MSE loss 2.32E-01. Sample per second: 167036\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.72 sec. Users per second: 1275\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0580843, PRECISION_RECALL_MIN_DEN: 0.0623747, RECALL: 0.0250401, MAP: 0.0249955, MAP_MIN_DEN: 0.0264777, MRR: 0.1587107, NDCG: 0.0547246, F1: 0.0349942, HIT_RATE: 0.3740040, ARHR_ALL_HITS: 0.1972053, NOVELTY: 0.0252138, AVERAGE_POPULARITY: 0.3878089, DIVERSITY_MEAN_INTER_LIST: 0.9549357, DIVERSITY_HERFINDAHL: 0.9954777, COVERAGE_ITEM: 0.2230234, COVERAGE_ITEM_CORRECT: 0.0950296, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3730132, DIVERSITY_GINI: 0.0621928, SHANNON_ENTROPY: 8.3390780, RATIO_DIVERSITY_HERFINDAHL: 0.9962910, RATIO_DIVERSITY_GINI: 0.1778734, RATIO_SHANNON_ENTROPY: 0.7712177, RATIO_AVERAGE_POPULARITY: 1.4040067, RATIO_NOVELTY: 0.0854339, \n",
      "\n",
      "FUNK_SVD: Convergence reached! Terminating at epoch 35. Best value for 'MAP' at epoch 10 is 0.0257. Elapsed time 2.63 min\n",
      "FUNK_SVD: Epoch 35 of 500. Elapsed time 2.63 min\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.60 sec. Users per second: 1311\n",
      "SearchBayesianSkopt: Config 3 is suboptimal. Config: {'epochs': 10, 'num_factors': 48, 'sgd_mode': 'sgd', 'batch_size': 2, 'item_reg': 0.00015736081658333225, 'user_reg': 2.585551202307481e-05, 'learning_rate': 0.008652038142191442} - results: PRECISION: 0.0583665, PRECISION_RECALL_MIN_DEN: 0.0644439, RECALL: 0.0298767, MAP: 0.0256909, MAP_MIN_DEN: 0.0277896, MRR: 0.1698463, NDCG: 0.0596481, F1: 0.0395226, HIT_RATE: 0.3897742, ARHR_ALL_HITS: 0.2070506, NOVELTY: 0.0248794, AVERAGE_POPULARITY: 0.4402573, DIVERSITY_MEAN_INTER_LIST: 0.8564625, DIVERSITY_HERFINDAHL: 0.9856320, COVERAGE_ITEM: 0.0839557, COVERAGE_ITEM_CORRECT: 0.0417203, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3887417, DIVERSITY_GINI: 0.0196775, SHANNON_ENTROPY: 6.6673205, RATIO_DIVERSITY_HERFINDAHL: 0.9864373, RATIO_DIVERSITY_GINI: 0.0562784, RATIO_SHANNON_ENTROPY: 0.6166096, RATIO_AVERAGE_POPULARITY: 1.5938886, RATIO_NOVELTY: 0.0843008, \n",
      "\n",
      "Iteration No: 4 ended. Search finished for the next optimal point.\n",
      "Time taken: 163.5325\n",
      "Function value obtained: -0.0257\n",
      "Current minimum: -0.0505\n",
      "Iteration No: 5 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'epochs': 500, 'num_factors': 132, 'sgd_mode': 'adagrad', 'batch_size': 32, 'item_reg': 1.5654635614454078e-05, 'user_reg': 0.001534238191444969, 'learning_rate': 0.09091098040947913}\n",
      "MatrixFactorization_FunkSVD_Cython_Recommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.84 sec. MSE loss 6.61E-01. Sample per second: 59026\n",
      "FUNK_SVD: Epoch 1 of 500. Elapsed time 10.36 sec\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.86 sec. MSE loss 2.75E-01. Sample per second: 58924\n",
      "FUNK_SVD: Epoch 2 of 500. Elapsed time 20.38 sec\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.64 sec. MSE loss 1.92E-01. Sample per second: 60170\n",
      "FUNK_SVD: Epoch 3 of 500. Elapsed time 30.15 sec\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.63 sec. MSE loss 1.50E-01. Sample per second: 60213\n",
      "FUNK_SVD: Epoch 4 of 500. Elapsed time 40.15 sec\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.39 sec. MSE loss 1.23E-01. Sample per second: 61586\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.37 sec. Users per second: 1787\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0454183, PRECISION_RECALL_MIN_DEN: 0.0491130, RECALL: 0.0195329, MAP: 0.0181815, MAP_MIN_DEN: 0.0194547, MRR: 0.1210822, NDCG: 0.0417997, F1: 0.0273174, HIT_RATE: 0.3102590, ARHR_ALL_HITS: 0.1473141, NOVELTY: 0.0268393, AVERAGE_POPULARITY: 0.2888284, DIVERSITY_MEAN_INTER_LIST: 0.9762929, DIVERSITY_HERFINDAHL: 0.9976131, COVERAGE_ITEM: 0.4975534, COVERAGE_ITEM_CORRECT: 0.1439609, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3094371, DIVERSITY_GINI: 0.1423153, SHANNON_ENTROPY: 9.4788694, RATIO_DIVERSITY_HERFINDAHL: 0.9984281, RATIO_DIVERSITY_GINI: 0.4070265, RATIO_SHANNON_ENTROPY: 0.8766283, RATIO_AVERAGE_POPULARITY: 1.0456621, RATIO_NOVELTY: 0.0909418, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 5 of 500. Elapsed time 53.30 sec\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Processed 640160 (100.0%) in 10.57 sec. MSE loss 1.07E-01. Sample per second: 60590\n",
      "FUNK_SVD: Epoch 6 of 500. Elapsed time 1.05 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.22 sec. MSE loss 9.43E-02. Sample per second: 62665\n",
      "FUNK_SVD: Epoch 7 of 500. Elapsed time 1.21 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 9.95 sec. MSE loss 8.41E-02. Sample per second: 64322\n",
      "FUNK_SVD: Epoch 8 of 500. Elapsed time 1.37 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.38 sec. MSE loss 7.60E-02. Sample per second: 61649\n",
      "FUNK_SVD: Epoch 9 of 500. Elapsed time 1.53 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 9.95 sec. MSE loss 7.07E-02. Sample per second: 64313\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.21 sec. Users per second: 1879\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0450863, PRECISION_RECALL_MIN_DEN: 0.0484788, RECALL: 0.0188861, MAP: 0.0178595, MAP_MIN_DEN: 0.0190478, MRR: 0.1209602, NDCG: 0.0409930, F1: 0.0266210, HIT_RATE: 0.3139110, ARHR_ALL_HITS: 0.1456131, NOVELTY: 0.0270146, AVERAGE_POPULARITY: 0.2772685, DIVERSITY_MEAN_INTER_LIST: 0.9791945, DIVERSITY_HERFINDAHL: 0.9979032, COVERAGE_ITEM: 0.5099150, COVERAGE_ITEM_CORRECT: 0.1581252, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3130795, DIVERSITY_GINI: 0.1570454, SHANNON_ENTROPY: 9.6301167, RATIO_DIVERSITY_HERFINDAHL: 0.9987184, RATIO_DIVERSITY_GINI: 0.4491552, RATIO_SHANNON_ENTROPY: 0.8906160, RATIO_AVERAGE_POPULARITY: 1.0038110, RATIO_NOVELTY: 0.0915357, \n",
      "\n",
      "FUNK_SVD: Epoch 10 of 500. Elapsed time 1.74 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 8.54 sec. MSE loss 6.46E-02. Sample per second: 74965\n",
      "FUNK_SVD: Epoch 11 of 500. Elapsed time 1.88 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.74 sec. MSE loss 6.04E-02. Sample per second: 54528\n",
      "FUNK_SVD: Epoch 12 of 500. Elapsed time 2.07 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 13.52 sec. MSE loss 5.68E-02. Sample per second: 47321\n",
      "FUNK_SVD: Epoch 13 of 500. Elapsed time 2.28 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 13.10 sec. MSE loss 5.38E-02. Sample per second: 48852\n",
      "FUNK_SVD: Epoch 14 of 500. Elapsed time 2.49 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 12.96 sec. MSE loss 5.08E-02. Sample per second: 49392\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.71 sec. Users per second: 1279\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0434761, PRECISION_RECALL_MIN_DEN: 0.0468943, RECALL: 0.0183587, MAP: 0.0174041, MAP_MIN_DEN: 0.0186504, MRR: 0.1168468, NDCG: 0.0393122, F1: 0.0258160, HIT_RATE: 0.3022908, ARHR_ALL_HITS: 0.1413041, NOVELTY: 0.0271326, AVERAGE_POPULARITY: 0.2697434, DIVERSITY_MEAN_INTER_LIST: 0.9807277, DIVERSITY_HERFINDAHL: 0.9980565, COVERAGE_ITEM: 0.5194437, COVERAGE_ITEM_CORRECT: 0.1568375, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3014901, DIVERSITY_GINI: 0.1660730, SHANNON_ENTROPY: 9.7157710, RATIO_DIVERSITY_HERFINDAHL: 0.9988719, RATIO_DIVERSITY_GINI: 0.4749742, RATIO_SHANNON_ENTROPY: 0.8985375, RATIO_AVERAGE_POPULARITY: 0.9765675, RATIO_NOVELTY: 0.0919358, \n",
      "\n",
      "FUNK_SVD: Epoch 15 of 500. Elapsed time 2.79 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 13.68 sec. MSE loss 4.85E-02. Sample per second: 46768\n",
      "FUNK_SVD: Epoch 16 of 500. Elapsed time 3.00 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 13.47 sec. MSE loss 4.63E-02. Sample per second: 47509\n",
      "FUNK_SVD: Epoch 17 of 500. Elapsed time 3.22 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 13.46 sec. MSE loss 4.39E-02. Sample per second: 47562\n",
      "FUNK_SVD: Epoch 18 of 500. Elapsed time 3.43 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 13.37 sec. MSE loss 4.27E-02. Sample per second: 47859\n",
      "FUNK_SVD: Epoch 19 of 500. Elapsed time 3.65 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 12.76 sec. MSE loss 4.07E-02. Sample per second: 50161\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.74 sec. Users per second: 1270\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0423639, PRECISION_RECALL_MIN_DEN: 0.0459275, RECALL: 0.0183175, MAP: 0.0169947, MAP_MIN_DEN: 0.0183759, MRR: 0.1155812, NDCG: 0.0388140, F1: 0.0255762, HIT_RATE: 0.2998008, ARHR_ALL_HITS: 0.1391281, NOVELTY: 0.0272026, AVERAGE_POPULARITY: 0.2653676, DIVERSITY_MEAN_INTER_LIST: 0.9815015, DIVERSITY_HERFINDAHL: 0.9981339, COVERAGE_ITEM: 0.5217615, COVERAGE_ITEM_CORRECT: 0.1632758, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.2990066, DIVERSITY_GINI: 0.1714190, SHANNON_ENTROPY: 9.7634197, RATIO_DIVERSITY_HERFINDAHL: 0.9989493, RATIO_DIVERSITY_GINI: 0.4902640, RATIO_SHANNON_ENTROPY: 0.9029442, RATIO_AVERAGE_POPULARITY: 0.9607257, RATIO_NOVELTY: 0.0921729, \n",
      "\n",
      "FUNK_SVD: Epoch 20 of 500. Elapsed time 3.93 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 12.16 sec. MSE loss 3.93E-02. Sample per second: 52661\n",
      "FUNK_SVD: Epoch 21 of 500. Elapsed time 4.13 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 12.63 sec. MSE loss 3.81E-02. Sample per second: 50652\n",
      "FUNK_SVD: Epoch 22 of 500. Elapsed time 4.34 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 13.24 sec. MSE loss 3.65E-02. Sample per second: 48355\n",
      "FUNK_SVD: Epoch 23 of 500. Elapsed time 4.55 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 13.25 sec. MSE loss 3.56E-02. Sample per second: 48292\n",
      "FUNK_SVD: Epoch 24 of 500. Elapsed time 4.76 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 13.19 sec. MSE loss 3.46E-02. Sample per second: 48537\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.79 sec. Users per second: 1259\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0420983, PRECISION_RECALL_MIN_DEN: 0.0459718, RECALL: 0.0187852, MAP: 0.0167501, MAP_MIN_DEN: 0.0181653, MRR: 0.1135262, NDCG: 0.0384525, F1: 0.0259783, HIT_RATE: 0.2969788, ARHR_ALL_HITS: 0.1365006, NOVELTY: 0.0272287, AVERAGE_POPULARITY: 0.2637340, DIVERSITY_MEAN_INTER_LIST: 0.9815998, DIVERSITY_HERFINDAHL: 0.9981437, COVERAGE_ITEM: 0.5199588, COVERAGE_ITEM_CORRECT: 0.1625032, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.2961921, DIVERSITY_GINI: 0.1734094, SHANNON_ENTROPY: 9.7774156, RATIO_DIVERSITY_HERFINDAHL: 0.9989591, RATIO_DIVERSITY_GINI: 0.4959567, RATIO_SHANNON_ENTROPY: 0.9042386, RATIO_AVERAGE_POPULARITY: 0.9548114, RATIO_NOVELTY: 0.0922613, \n",
      "\n",
      "FUNK_SVD: Epoch 25 of 500. Elapsed time 5.06 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 12.38 sec. MSE loss 3.36E-02. Sample per second: 51698\n",
      "FUNK_SVD: Epoch 26 of 500. Elapsed time 5.27 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 12.92 sec. MSE loss 3.25E-02. Sample per second: 49539\n",
      "FUNK_SVD: Epoch 27 of 500. Elapsed time 5.47 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 13.64 sec. MSE loss 3.12E-02. Sample per second: 46938\n",
      "FUNK_SVD: Epoch 28 of 500. Elapsed time 5.69 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 13.11 sec. MSE loss 3.08E-02. Sample per second: 48816\n",
      "FUNK_SVD: Epoch 29 of 500. Elapsed time 5.89 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 12.64 sec. MSE loss 2.98E-02. Sample per second: 50630\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.62 sec. Users per second: 1304\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0425631, PRECISION_RECALL_MIN_DEN: 0.0466155, RECALL: 0.0191371, MAP: 0.0169071, MAP_MIN_DEN: 0.0184653, MRR: 0.1133844, NDCG: 0.0388772, F1: 0.0264030, HIT_RATE: 0.2951527, ARHR_ALL_HITS: 0.1369044, NOVELTY: 0.0272666, AVERAGE_POPULARITY: 0.2614013, DIVERSITY_MEAN_INTER_LIST: 0.9819572, DIVERSITY_HERFINDAHL: 0.9981794, COVERAGE_ITEM: 0.5248519, COVERAGE_ITEM_CORRECT: 0.1625032, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.2943709, DIVERSITY_GINI: 0.1759764, SHANNON_ENTROPY: 9.7999014, RATIO_DIVERSITY_HERFINDAHL: 0.9989949, RATIO_DIVERSITY_GINI: 0.5032985, RATIO_SHANNON_ENTROPY: 0.9063181, RATIO_AVERAGE_POPULARITY: 0.9463662, RATIO_NOVELTY: 0.0923898, \n",
      "\n",
      "FUNK_SVD: Convergence reached! Terminating at epoch 30. Best value for 'MAP' at epoch 5 is 0.0182. Elapsed time 6.18 min\n",
      "FUNK_SVD: Epoch 30 of 500. Elapsed time 6.18 min\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.36 sec. Users per second: 1380\n",
      "SearchBayesianSkopt: Config 4 is suboptimal. Config: {'epochs': 5, 'num_factors': 132, 'sgd_mode': 'adagrad', 'batch_size': 32, 'item_reg': 1.5654635614454078e-05, 'user_reg': 0.001534238191444969, 'learning_rate': 0.09091098040947913} - results: PRECISION: 0.0454183, PRECISION_RECALL_MIN_DEN: 0.0491130, RECALL: 0.0195329, MAP: 0.0181815, MAP_MIN_DEN: 0.0194547, MRR: 0.1210822, NDCG: 0.0417997, F1: 0.0273174, HIT_RATE: 0.3102590, ARHR_ALL_HITS: 0.1473141, NOVELTY: 0.0268393, AVERAGE_POPULARITY: 0.2888284, DIVERSITY_MEAN_INTER_LIST: 0.9762929, DIVERSITY_HERFINDAHL: 0.9976131, COVERAGE_ITEM: 0.4975534, COVERAGE_ITEM_CORRECT: 0.1439609, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3094371, DIVERSITY_GINI: 0.1423153, SHANNON_ENTROPY: 9.4788694, RATIO_DIVERSITY_HERFINDAHL: 0.9984281, RATIO_DIVERSITY_GINI: 0.4070265, RATIO_SHANNON_ENTROPY: 0.8766283, RATIO_AVERAGE_POPULARITY: 1.0456621, RATIO_NOVELTY: 0.0909418, \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration No: 5 ended. Search finished for the next optimal point.\n",
      "Time taken: 375.9501\n",
      "Function value obtained: -0.0182\n",
      "Current minimum: -0.0505\n",
      "Iteration No: 6 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'epochs': 500, 'num_factors': 108, 'sgd_mode': 'adagrad', 'batch_size': 32, 'item_reg': 0.0035797717161438407, 'user_reg': 1.3811120614703808e-05, 'learning_rate': 0.001228995115651476}\n",
      "MatrixFactorization_FunkSVD_Cython_Recommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.16 sec. MSE loss 1.29E+01. Sample per second: 63008\n",
      "FUNK_SVD: Epoch 1 of 500. Elapsed time 9.77 sec\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.45 sec. MSE loss 1.14E+01. Sample per second: 61248\n",
      "FUNK_SVD: Epoch 2 of 500. Elapsed time 20.06 sec\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.89 sec. MSE loss 1.05E+01. Sample per second: 58763\n",
      "FUNK_SVD: Epoch 3 of 500. Elapsed time 30.50 sec\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.94 sec. MSE loss 9.75E+00. Sample per second: 58473\n",
      "FUNK_SVD: Epoch 4 of 500. Elapsed time 40.56 sec\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.23 sec. MSE loss 9.16E+00. Sample per second: 57014\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.59 sec. Users per second: 1313\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0449203, PRECISION_RECALL_MIN_DEN: 0.0494539, RECALL: 0.0218718, MAP: 0.0185970, MAP_MIN_DEN: 0.0203903, MRR: 0.1334094, NDCG: 0.0379305, F1: 0.0294193, HIT_RATE: 0.3256972, ARHR_ALL_HITS: 0.1568043, NOVELTY: 0.0263297, AVERAGE_POPULARITY: 0.3592626, DIVERSITY_MEAN_INTER_LIST: 0.9704396, DIVERSITY_HERFINDAHL: 0.9970279, COVERAGE_ITEM: 0.8488282, COVERAGE_ITEM_CORRECT: 0.1295390, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3248344, DIVERSITY_GINI: 0.2140313, SHANNON_ENTROPY: 9.7791661, RATIO_DIVERSITY_HERFINDAHL: 0.9978424, RATIO_DIVERSITY_GINI: 0.6121368, RATIO_SHANNON_ENTROPY: 0.9044005, RATIO_AVERAGE_POPULARITY: 1.3006589, RATIO_NOVELTY: 0.0892152, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 5 of 500. Elapsed time 55.44 sec\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.54 sec. MSE loss 8.68E+00. Sample per second: 55459\n",
      "FUNK_SVD: Epoch 6 of 500. Elapsed time 1.10 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.53 sec. MSE loss 8.23E+00. Sample per second: 60759\n",
      "FUNK_SVD: Epoch 7 of 500. Elapsed time 1.27 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.65 sec. MSE loss 7.85E+00. Sample per second: 60053\n",
      "FUNK_SVD: Epoch 8 of 500. Elapsed time 1.44 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.09 sec. MSE loss 7.51E+00. Sample per second: 57693\n",
      "FUNK_SVD: Epoch 9 of 500. Elapsed time 1.61 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.35 sec. MSE loss 7.19E+00. Sample per second: 61872\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.60 sec. Users per second: 1310\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0658035, PRECISION_RECALL_MIN_DEN: 0.0728409, RECALL: 0.0329741, MAP: 0.0281650, MAP_MIN_DEN: 0.0308474, MRR: 0.1808621, NDCG: 0.0576789, F1: 0.0439333, HIT_RATE: 0.4266268, ARHR_ALL_HITS: 0.2239649, NOVELTY: 0.0244670, AVERAGE_POPULARITY: 0.4884825, DIVERSITY_MEAN_INTER_LIST: 0.9267168, DIVERSITY_HERFINDAHL: 0.9926563, COVERAGE_ITEM: 0.5614216, COVERAGE_ITEM_CORRECT: 0.1043008, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4254967, DIVERSITY_GINI: 0.0867223, SHANNON_ENTROPY: 8.3857993, RATIO_DIVERSITY_HERFINDAHL: 0.9934673, RATIO_DIVERSITY_GINI: 0.2480287, RATIO_SHANNON_ENTROPY: 0.7755386, RATIO_AVERAGE_POPULARITY: 1.7684812, RATIO_NOVELTY: 0.0829036, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 10 of 500. Elapsed time 1.86 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.92 sec. MSE loss 6.90E+00. Sample per second: 58610\n",
      "FUNK_SVD: Epoch 11 of 500. Elapsed time 2.03 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 9.83 sec. MSE loss 6.65E+00. Sample per second: 65137\n",
      "FUNK_SVD: Epoch 12 of 500. Elapsed time 2.17 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.08 sec. MSE loss 6.40E+00. Sample per second: 57755\n",
      "FUNK_SVD: Epoch 13 of 500. Elapsed time 2.34 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.47 sec. MSE loss 6.16E+00. Sample per second: 61107\n",
      "FUNK_SVD: Epoch 14 of 500. Elapsed time 2.52 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.59 sec. MSE loss 5.95E+00. Sample per second: 60415\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.71 sec. Users per second: 1280\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0774236, PRECISION_RECALL_MIN_DEN: 0.0862436, RECALL: 0.0396507, MAP: 0.0337393, MAP_MIN_DEN: 0.0369729, MRR: 0.2035700, NDCG: 0.0684063, F1: 0.0524436, HIT_RATE: 0.4731076, ARHR_ALL_HITS: 0.2596168, NOVELTY: 0.0237086, AVERAGE_POPULARITY: 0.5539259, DIVERSITY_MEAN_INTER_LIST: 0.8893387, DIVERSITY_HERFINDAHL: 0.9889191, COVERAGE_ITEM: 0.3734226, COVERAGE_ITEM_CORRECT: 0.0875612, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4718543, DIVERSITY_GINI: 0.0475323, SHANNON_ENTROPY: 7.5976477, RATIO_DIVERSITY_HERFINDAHL: 0.9897270, RATIO_DIVERSITY_GINI: 0.1359439, RATIO_SHANNON_ENTROPY: 0.7026485, RATIO_AVERAGE_POPULARITY: 2.0054098, RATIO_NOVELTY: 0.0803339, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 15 of 500. Elapsed time 2.77 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.51 sec. MSE loss 5.75E+00. Sample per second: 60901\n",
      "FUNK_SVD: Epoch 16 of 500. Elapsed time 2.94 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.67 sec. MSE loss 5.56E+00. Sample per second: 59962\n",
      "FUNK_SVD: Epoch 17 of 500. Elapsed time 3.10 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.15 sec. MSE loss 5.38E+00. Sample per second: 57403\n",
      "FUNK_SVD: Epoch 18 of 500. Elapsed time 3.28 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.54 sec. MSE loss 5.21E+00. Sample per second: 60690\n",
      "FUNK_SVD: Epoch 19 of 500. Elapsed time 3.45 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.96 sec. MSE loss 5.06E+00. Sample per second: 58388\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.68 sec. Users per second: 1286\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0843459, PRECISION_RECALL_MIN_DEN: 0.0940520, RECALL: 0.0434872, MAP: 0.0371566, MAP_MIN_DEN: 0.0407395, MRR: 0.2148519, NDCG: 0.0751327, F1: 0.0573868, HIT_RATE: 0.4970120, ARHR_ALL_HITS: 0.2796379, NOVELTY: 0.0233352, AVERAGE_POPULARITY: 0.5918065, DIVERSITY_MEAN_INTER_LIST: 0.8599635, DIVERSITY_HERFINDAHL: 0.9859821, COVERAGE_ITEM: 0.2675766, COVERAGE_ITEM_CORRECT: 0.0777749, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4956954, DIVERSITY_GINI: 0.0322148, SHANNON_ENTROPY: 7.1166454, RATIO_DIVERSITY_HERFINDAHL: 0.9867876, RATIO_DIVERSITY_GINI: 0.0921353, RATIO_SHANNON_ENTROPY: 0.6581642, RATIO_AVERAGE_POPULARITY: 2.1425508, RATIO_NOVELTY: 0.0790687, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 20 of 500. Elapsed time 3.70 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.09 sec. MSE loss 4.90E+00. Sample per second: 57707\n",
      "FUNK_SVD: Epoch 21 of 500. Elapsed time 3.88 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.63 sec. MSE loss 4.76E+00. Sample per second: 60166\n",
      "FUNK_SVD: Epoch 22 of 500. Elapsed time 4.05 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 9.76 sec. MSE loss 4.63E+00. Sample per second: 65608\n",
      "FUNK_SVD: Epoch 23 of 500. Elapsed time 4.21 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.89 sec. MSE loss 4.50E+00. Sample per second: 58774\n",
      "FUNK_SVD: Epoch 24 of 500. Elapsed time 4.38 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.34 sec. MSE loss 4.38E+00. Sample per second: 56452\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.77 sec. Users per second: 1263\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0893924, PRECISION_RECALL_MIN_DEN: 0.0996339, RECALL: 0.0462505, MAP: 0.0395153, MAP_MIN_DEN: 0.0433553, MRR: 0.2218277, NDCG: 0.0799546, F1: 0.0609607, HIT_RATE: 0.5141102, ARHR_ALL_HITS: 0.2929909, NOVELTY: 0.0231187, AVERAGE_POPULARITY: 0.6160984, DIVERSITY_MEAN_INTER_LIST: 0.8367701, DIVERSITY_HERFINDAHL: 0.9836631, COVERAGE_ITEM: 0.2086016, COVERAGE_ITEM_CORRECT: 0.0672161, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5127483, DIVERSITY_GINI: 0.0246350, SHANNON_ENTROPY: 6.7936611, RATIO_DIVERSITY_HERFINDAHL: 0.9844667, RATIO_DIVERSITY_GINI: 0.0704569, RATIO_SHANNON_ENTROPY: 0.6282939, RATIO_AVERAGE_POPULARITY: 2.2304964, RATIO_NOVELTY: 0.0783349, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 25 of 500. Elapsed time 4.63 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Processed 640160 (100.0%) in 10.12 sec. MSE loss 4.25E+00. Sample per second: 63287\n",
      "FUNK_SVD: Epoch 26 of 500. Elapsed time 4.80 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.62 sec. MSE loss 4.13E+00. Sample per second: 60258\n",
      "FUNK_SVD: Epoch 27 of 500. Elapsed time 4.97 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.90 sec. MSE loss 4.03E+00. Sample per second: 58709\n",
      "FUNK_SVD: Epoch 28 of 500. Elapsed time 5.14 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.99 sec. MSE loss 3.93E+00. Sample per second: 58240\n",
      "FUNK_SVD: Epoch 29 of 500. Elapsed time 5.31 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.23 sec. MSE loss 3.83E+00. Sample per second: 56995\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.77 sec. Users per second: 1263\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0932603, PRECISION_RECALL_MIN_DEN: 0.1040418, RECALL: 0.0481212, MAP: 0.0416013, MAP_MIN_DEN: 0.0455969, MRR: 0.2284368, NDCG: 0.0836111, F1: 0.0634849, HIT_RATE: 0.5242364, ARHR_ALL_HITS: 0.3044445, NOVELTY: 0.0229778, AVERAGE_POPULARITY: 0.6328706, DIVERSITY_MEAN_INTER_LIST: 0.8182649, DIVERSITY_HERFINDAHL: 0.9818129, COVERAGE_ITEM: 0.1722895, COVERAGE_ITEM_CORRECT: 0.0620654, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5228477, DIVERSITY_GINI: 0.0203457, SHANNON_ENTROPY: 6.5621477, RATIO_DIVERSITY_HERFINDAHL: 0.9826150, RATIO_DIVERSITY_GINI: 0.0581894, RATIO_SHANNON_ENTROPY: 0.6068830, RATIO_AVERAGE_POPULARITY: 2.2912178, RATIO_NOVELTY: 0.0778576, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 30 of 500. Elapsed time 5.56 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.30 sec. MSE loss 3.73E+00. Sample per second: 62127\n",
      "FUNK_SVD: Epoch 31 of 500. Elapsed time 5.73 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.52 sec. MSE loss 3.65E+00. Sample per second: 60806\n",
      "FUNK_SVD: Epoch 32 of 500. Elapsed time 5.90 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.53 sec. MSE loss 3.56E+00. Sample per second: 60803\n",
      "FUNK_SVD: Epoch 33 of 500. Elapsed time 6.07 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 9.40 sec. MSE loss 3.49E+00. Sample per second: 68114\n",
      "FUNK_SVD: Epoch 34 of 500. Elapsed time 6.22 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 8.00 sec. MSE loss 3.39E+00. Sample per second: 79974\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.28 sec. Users per second: 1839\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0958499, PRECISION_RECALL_MIN_DEN: 0.1068667, RECALL: 0.0494174, MAP: 0.0430485, MAP_MIN_DEN: 0.0472651, MRR: 0.2333514, NDCG: 0.0865035, F1: 0.0652129, HIT_RATE: 0.5313745, ARHR_ALL_HITS: 0.3126483, NOVELTY: 0.0228823, AVERAGE_POPULARITY: 0.6449462, DIVERSITY_MEAN_INTER_LIST: 0.8030970, DIVERSITY_HERFINDAHL: 0.9802964, COVERAGE_ITEM: 0.1503992, COVERAGE_ITEM_CORRECT: 0.0566572, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5299669, DIVERSITY_GINI: 0.0177037, SHANNON_ENTROPY: 6.3919090, RATIO_DIVERSITY_HERFINDAHL: 0.9810972, RATIO_DIVERSITY_GINI: 0.0506331, RATIO_SHANNON_ENTROPY: 0.5911389, RATIO_AVERAGE_POPULARITY: 2.3349356, RATIO_NOVELTY: 0.0775339, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 35 of 500. Elapsed time 6.40 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 8.35 sec. MSE loss 3.31E+00. Sample per second: 76670\n",
      "FUNK_SVD: Epoch 36 of 500. Elapsed time 6.53 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 7.79 sec. MSE loss 3.24E+00. Sample per second: 82164\n",
      "FUNK_SVD: Epoch 37 of 500. Elapsed time 6.66 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 8.27 sec. MSE loss 3.17E+00. Sample per second: 77421\n",
      "FUNK_SVD: Epoch 38 of 500. Elapsed time 6.78 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 8.14 sec. MSE loss 3.10E+00. Sample per second: 78606\n",
      "FUNK_SVD: Epoch 39 of 500. Elapsed time 6.91 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 7.81 sec. MSE loss 3.03E+00. Sample per second: 81925\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.42 sec. Users per second: 1759\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0974602, PRECISION_RECALL_MIN_DEN: 0.1086348, RECALL: 0.0503748, MAP: 0.0437966, MAP_MIN_DEN: 0.0481230, MRR: 0.2356686, NDCG: 0.0881891, F1: 0.0664192, HIT_RATE: 0.5383466, ARHR_ALL_HITS: 0.3170749, NOVELTY: 0.0228126, AVERAGE_POPULARITY: 0.6538972, DIVERSITY_MEAN_INTER_LIST: 0.7907053, DIVERSITY_HERFINDAHL: 0.9790574, COVERAGE_ITEM: 0.1313417, COVERAGE_ITEM_CORRECT: 0.0530518, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5369205, DIVERSITY_GINI: 0.0159144, SHANNON_ENTROPY: 6.2611967, RATIO_DIVERSITY_HERFINDAHL: 0.9798573, RATIO_DIVERSITY_GINI: 0.0455157, RATIO_SHANNON_ENTROPY: 0.5790503, RATIO_AVERAGE_POPULARITY: 2.3673414, RATIO_NOVELTY: 0.0772977, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 40 of 500. Elapsed time 7.10 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 7.74 sec. MSE loss 2.97E+00. Sample per second: 82699\n",
      "FUNK_SVD: Epoch 41 of 500. Elapsed time 7.22 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 8.15 sec. MSE loss 2.91E+00. Sample per second: 78530\n",
      "FUNK_SVD: Epoch 42 of 500. Elapsed time 7.35 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 8.08 sec. MSE loss 2.85E+00. Sample per second: 79150\n",
      "FUNK_SVD: Epoch 43 of 500. Elapsed time 7.48 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 7.60 sec. MSE loss 2.79E+00. Sample per second: 84177\n",
      "FUNK_SVD: Epoch 44 of 500. Elapsed time 7.60 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 8.08 sec. MSE loss 2.73E+00. Sample per second: 79230\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.90 sec. Users per second: 1543\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0992198, PRECISION_RECALL_MIN_DEN: 0.1106321, RECALL: 0.0513940, MAP: 0.0446558, MAP_MIN_DEN: 0.0491070, MRR: 0.2379849, NDCG: 0.0899547, F1: 0.0677136, HIT_RATE: 0.5431607, ARHR_ALL_HITS: 0.3214349, NOVELTY: 0.0227555, AVERAGE_POPULARITY: 0.6613285, DIVERSITY_MEAN_INTER_LIST: 0.7794365, DIVERSITY_HERFINDAHL: 0.9779307, COVERAGE_ITEM: 0.1164048, COVERAGE_ITEM_CORRECT: 0.0507340, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5417219, DIVERSITY_GINI: 0.0144632, SHANNON_ENTROPY: 6.1446510, RATIO_DIVERSITY_HERFINDAHL: 0.9787296, RATIO_DIVERSITY_GINI: 0.0413653, RATIO_SHANNON_ENTROPY: 0.5682719, RATIO_AVERAGE_POPULARITY: 2.3942454, RATIO_NOVELTY: 0.0771045, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 45 of 500. Elapsed time 7.79 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 9.11 sec. MSE loss 2.68E+00. Sample per second: 70232\n",
      "FUNK_SVD: Epoch 46 of 500. Elapsed time 7.95 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.36 sec. MSE loss 2.63E+00. Sample per second: 61792\n",
      "FUNK_SVD: Epoch 47 of 500. Elapsed time 8.12 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.69 sec. MSE loss 2.58E+00. Sample per second: 59851\n",
      "FUNK_SVD: Epoch 48 of 500. Elapsed time 8.29 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.16 sec. MSE loss 2.53E+00. Sample per second: 57348\n",
      "FUNK_SVD: Epoch 49 of 500. Elapsed time 8.46 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.46 sec. MSE loss 2.49E+00. Sample per second: 61187\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.57 sec. Users per second: 1317\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1004814, PRECISION_RECALL_MIN_DEN: 0.1120311, RECALL: 0.0519814, MAP: 0.0453638, MAP_MIN_DEN: 0.0499230, MRR: 0.2408289, NDCG: 0.0915464, F1: 0.0685172, HIT_RATE: 0.5479748, ARHR_ALL_HITS: 0.3257058, NOVELTY: 0.0227139, AVERAGE_POPULARITY: 0.6669948, DIVERSITY_MEAN_INTER_LIST: 0.7699175, DIVERSITY_HERFINDAHL: 0.9769790, COVERAGE_ITEM: 0.1081638, COVERAGE_ITEM_CORRECT: 0.0479011, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5465232, DIVERSITY_GINI: 0.0134987, SHANNON_ENTROPY: 6.0562093, RATIO_DIVERSITY_HERFINDAHL: 0.9777771, RATIO_DIVERSITY_GINI: 0.0386066, RATIO_SHANNON_ENTROPY: 0.5600926, RATIO_AVERAGE_POPULARITY: 2.4147594, RATIO_NOVELTY: 0.0769634, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 50 of 500. Elapsed time 8.71 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.28 sec. MSE loss 2.44E+00. Sample per second: 62272\n",
      "FUNK_SVD: Epoch 51 of 500. Elapsed time 8.88 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.73 sec. MSE loss 2.39E+00. Sample per second: 59659\n",
      "FUNK_SVD: Epoch 52 of 500. Elapsed time 9.06 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.07 sec. MSE loss 2.36E+00. Sample per second: 57816\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Epoch 53 of 500. Elapsed time 9.23 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.58 sec. MSE loss 2.31E+00. Sample per second: 60507\n",
      "FUNK_SVD: Epoch 54 of 500. Elapsed time 9.40 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.76 sec. MSE loss 2.27E+00. Sample per second: 59484\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.65 sec. Users per second: 1296\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1016932, PRECISION_RECALL_MIN_DEN: 0.1132565, RECALL: 0.0524341, MAP: 0.0459877, MAP_MIN_DEN: 0.0506099, MRR: 0.2428992, NDCG: 0.0929215, F1: 0.0691920, HIT_RATE: 0.5529548, ARHR_ALL_HITS: 0.3292225, NOVELTY: 0.0226775, AVERAGE_POPULARITY: 0.6719359, DIVERSITY_MEAN_INTER_LIST: 0.7607519, DIVERSITY_HERFINDAHL: 0.9760626, COVERAGE_ITEM: 0.0999227, COVERAGE_ITEM_CORRECT: 0.0460984, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5514901, DIVERSITY_GINI: 0.0126639, SHANNON_ENTROPY: 5.9743211, RATIO_DIVERSITY_HERFINDAHL: 0.9768600, RATIO_DIVERSITY_GINI: 0.0362191, RATIO_SHANNON_ENTROPY: 0.5525194, RATIO_AVERAGE_POPULARITY: 2.4326481, RATIO_NOVELTY: 0.0768400, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 55 of 500. Elapsed time 9.65 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.62 sec. MSE loss 2.23E+00. Sample per second: 60279\n",
      "FUNK_SVD: Epoch 56 of 500. Elapsed time 9.82 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 9.86 sec. MSE loss 2.19E+00. Sample per second: 64949\n",
      "FUNK_SVD: Epoch 57 of 500. Elapsed time 9.97 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.39 sec. MSE loss 2.16E+00. Sample per second: 56174\n",
      "FUNK_SVD: Epoch 58 of 500. Elapsed time 10.15 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.87 sec. MSE loss 2.13E+00. Sample per second: 58911\n",
      "FUNK_SVD: Epoch 59 of 500. Elapsed time 10.32 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.40 sec. MSE loss 2.09E+00. Sample per second: 56147\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.62 sec. Users per second: 1305\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1026228, PRECISION_RECALL_MIN_DEN: 0.1143655, RECALL: 0.0531085, MAP: 0.0464263, MAP_MIN_DEN: 0.0511300, MRR: 0.2441354, NDCG: 0.0940347, F1: 0.0699942, HIT_RATE: 0.5559429, ARHR_ALL_HITS: 0.3315282, NOVELTY: 0.0226495, AVERAGE_POPULARITY: 0.6757118, DIVERSITY_MEAN_INTER_LIST: 0.7533885, DIVERSITY_HERFINDAHL: 0.9753263, COVERAGE_ITEM: 0.0927118, COVERAGE_ITEM_CORRECT: 0.0435230, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5544702, DIVERSITY_GINI: 0.0120035, SHANNON_ENTROPY: 5.9069102, RATIO_DIVERSITY_HERFINDAHL: 0.9761231, RATIO_DIVERSITY_GINI: 0.0343303, RATIO_SHANNON_ENTROPY: 0.5462851, RATIO_AVERAGE_POPULARITY: 2.4463182, RATIO_NOVELTY: 0.0767453, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 60 of 500. Elapsed time 10.58 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.46 sec. MSE loss 2.06E+00. Sample per second: 61178\n",
      "FUNK_SVD: Epoch 61 of 500. Elapsed time 10.75 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.01 sec. MSE loss 2.03E+00. Sample per second: 58112\n",
      "FUNK_SVD: Epoch 62 of 500. Elapsed time 10.93 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.16 sec. MSE loss 2.00E+00. Sample per second: 63008\n",
      "FUNK_SVD: Epoch 63 of 500. Elapsed time 11.10 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.18 sec. MSE loss 1.97E+00. Sample per second: 62906\n",
      "FUNK_SVD: Epoch 64 of 500. Elapsed time 11.26 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.47 sec. MSE loss 1.94E+00. Sample per second: 61131\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.79 sec. Users per second: 1257\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1033201, PRECISION_RECALL_MIN_DEN: 0.1151373, RECALL: 0.0534788, MAP: 0.0468058, MAP_MIN_DEN: 0.0515334, MRR: 0.2454200, NDCG: 0.0950655, F1: 0.0704780, HIT_RATE: 0.5572709, ARHR_ALL_HITS: 0.3337331, NOVELTY: 0.0226289, AVERAGE_POPULARITY: 0.6785484, DIVERSITY_MEAN_INTER_LIST: 0.7470242, DIVERSITY_HERFINDAHL: 0.9746900, COVERAGE_ITEM: 0.0855009, COVERAGE_ITEM_CORRECT: 0.0414628, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5557947, DIVERSITY_GINI: 0.0115400, SHANNON_ENTROPY: 5.8554885, RATIO_DIVERSITY_HERFINDAHL: 0.9754863, RATIO_DIVERSITY_GINI: 0.0330049, RATIO_SHANNON_ENTROPY: 0.5415295, RATIO_AVERAGE_POPULARITY: 2.4565875, RATIO_NOVELTY: 0.0766752, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 65 of 500. Elapsed time 11.51 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.49 sec. MSE loss 1.91E+00. Sample per second: 61027\n",
      "FUNK_SVD: Epoch 66 of 500. Elapsed time 11.68 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.62 sec. MSE loss 1.88E+00. Sample per second: 60292\n",
      "FUNK_SVD: Epoch 67 of 500. Elapsed time 11.85 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 9.59 sec. MSE loss 1.86E+00. Sample per second: 66773\n",
      "FUNK_SVD: Epoch 68 of 500. Elapsed time 12.00 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.93 sec. MSE loss 1.83E+00. Sample per second: 58537\n",
      "FUNK_SVD: Epoch 69 of 500. Elapsed time 12.18 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.33 sec. MSE loss 1.81E+00. Sample per second: 56504\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.68 sec. Users per second: 1288\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1037517, PRECISION_RECALL_MIN_DEN: 0.1155867, RECALL: 0.0536473, MAP: 0.0471557, MAP_MIN_DEN: 0.0519042, MRR: 0.2465597, NDCG: 0.0957530, F1: 0.0707247, HIT_RATE: 0.5577689, ARHR_ALL_HITS: 0.3356186, NOVELTY: 0.0226138, AVERAGE_POPULARITY: 0.6807194, DIVERSITY_MEAN_INTER_LIST: 0.7413774, DIVERSITY_HERFINDAHL: 0.9741254, COVERAGE_ITEM: 0.0829256, COVERAGE_ITEM_CORRECT: 0.0409477, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5562914, DIVERSITY_GINI: 0.0111856, SHANNON_ENTROPY: 5.8127060, RATIO_DIVERSITY_HERFINDAHL: 0.9749213, RATIO_DIVERSITY_GINI: 0.0319913, RATIO_SHANNON_ENTROPY: 0.5375728, RATIO_AVERAGE_POPULARITY: 2.4644474, RATIO_NOVELTY: 0.0766242, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 70 of 500. Elapsed time 12.43 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.19 sec. MSE loss 1.78E+00. Sample per second: 62810\n",
      "FUNK_SVD: Epoch 71 of 500. Elapsed time 12.60 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.76 sec. MSE loss 1.76E+00. Sample per second: 59494\n",
      "FUNK_SVD: Epoch 72 of 500. Elapsed time 12.77 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.22 sec. MSE loss 1.74E+00. Sample per second: 57039\n",
      "FUNK_SVD: Epoch 73 of 500. Elapsed time 12.95 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.61 sec. MSE loss 1.71E+00. Sample per second: 60336\n",
      "FUNK_SVD: Epoch 74 of 500. Elapsed time 13.12 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.97 sec. MSE loss 1.69E+00. Sample per second: 58324\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.69 sec. Users per second: 1285\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1042995, PRECISION_RECALL_MIN_DEN: 0.1161744, RECALL: 0.0539493, MAP: 0.0474135, MAP_MIN_DEN: 0.0522038, MRR: 0.2475778, NDCG: 0.0964336, F1: 0.0711144, HIT_RATE: 0.5594290, ARHR_ALL_HITS: 0.3372457, NOVELTY: 0.0226008, AVERAGE_POPULARITY: 0.6824518, DIVERSITY_MEAN_INTER_LIST: 0.7364517, DIVERSITY_HERFINDAHL: 0.9736329, COVERAGE_ITEM: 0.0775174, COVERAGE_ITEM_CORRECT: 0.0399176, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5579470, DIVERSITY_GINI: 0.0108593, SHANNON_ENTROPY: 5.7739006, RATIO_DIVERSITY_HERFINDAHL: 0.9744284, RATIO_DIVERSITY_GINI: 0.0310581, RATIO_SHANNON_ENTROPY: 0.5339840, RATIO_AVERAGE_POPULARITY: 2.4707194, RATIO_NOVELTY: 0.0765803, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 75 of 500. Elapsed time 13.37 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.76 sec. MSE loss 1.68E+00. Sample per second: 59498\n",
      "FUNK_SVD: Epoch 76 of 500. Elapsed time 13.54 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.65 sec. MSE loss 1.65E+00. Sample per second: 60082\n",
      "FUNK_SVD: Epoch 77 of 500. Elapsed time 13.70 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.08 sec. MSE loss 1.63E+00. Sample per second: 57783\n",
      "FUNK_SVD: Epoch 78 of 500. Elapsed time 13.88 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.16 sec. MSE loss 1.61E+00. Sample per second: 63008\n",
      "FUNK_SVD: Epoch 79 of 500. Elapsed time 14.05 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 9.85 sec. MSE loss 1.60E+00. Sample per second: 64964\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.68 sec. Users per second: 1286\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1051461, PRECISION_RECALL_MIN_DEN: 0.1171060, RECALL: 0.0544593, MAP: 0.0477506, MAP_MIN_DEN: 0.0525700, MRR: 0.2485348, NDCG: 0.0972552, F1: 0.0717543, HIT_RATE: 0.5624170, ARHR_ALL_HITS: 0.3390866, NOVELTY: 0.0225907, AVERAGE_POPULARITY: 0.6838491, DIVERSITY_MEAN_INTER_LIST: 0.7319107, DIVERSITY_HERFINDAHL: 0.9731789, COVERAGE_ITEM: 0.0741695, COVERAGE_ITEM_CORRECT: 0.0381149, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5609272, DIVERSITY_GINI: 0.0105938, SHANNON_ENTROPY: 5.7405084, RATIO_DIVERSITY_HERFINDAHL: 0.9739740, RATIO_DIVERSITY_GINI: 0.0302986, RATIO_SHANNON_ENTROPY: 0.5308958, RATIO_AVERAGE_POPULARITY: 2.4757779, RATIO_NOVELTY: 0.0765461, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 80 of 500. Elapsed time 14.29 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.80 sec. MSE loss 1.58E+00. Sample per second: 59274\n",
      "FUNK_SVD: Epoch 81 of 500. Elapsed time 14.46 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.74 sec. MSE loss 1.56E+00. Sample per second: 59583\n",
      "FUNK_SVD: Epoch 82 of 500. Elapsed time 14.62 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.90 sec. MSE loss 1.54E+00. Sample per second: 58710\n",
      "FUNK_SVD: Epoch 83 of 500. Elapsed time 14.79 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.10 sec. MSE loss 1.52E+00. Sample per second: 57672\n",
      "FUNK_SVD: Epoch 84 of 500. Elapsed time 14.96 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.24 sec. MSE loss 1.51E+00. Sample per second: 62467\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.62 sec. Users per second: 1303\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1052789, PRECISION_RECALL_MIN_DEN: 0.1172267, RECALL: 0.0545059, MAP: 0.0478012, MAP_MIN_DEN: 0.0526143, MRR: 0.2482451, NDCG: 0.0975930, F1: 0.0718256, HIT_RATE: 0.5627490, ARHR_ALL_HITS: 0.3390202, NOVELTY: 0.0225828, AVERAGE_POPULARITY: 0.6848991, DIVERSITY_MEAN_INTER_LIST: 0.7280631, DIVERSITY_HERFINDAHL: 0.9727942, COVERAGE_ITEM: 0.0708215, COVERAGE_ITEM_CORRECT: 0.0368272, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5612583, DIVERSITY_GINI: 0.0103684, SHANNON_ENTROPY: 5.7115064, RATIO_DIVERSITY_HERFINDAHL: 0.9735890, RATIO_DIVERSITY_GINI: 0.0296539, RATIO_SHANNON_ENTROPY: 0.5282137, RATIO_AVERAGE_POPULARITY: 2.4795795, RATIO_NOVELTY: 0.0765192, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 85 of 500. Elapsed time 15.21 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.16 sec. MSE loss 1.49E+00. Sample per second: 57357\n",
      "FUNK_SVD: Epoch 86 of 500. Elapsed time 15.38 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.49 sec. MSE loss 1.48E+00. Sample per second: 60991\n",
      "FUNK_SVD: Epoch 87 of 500. Elapsed time 15.55 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.55 sec. MSE loss 1.46E+00. Sample per second: 60644\n",
      "FUNK_SVD: Epoch 88 of 500. Elapsed time 15.72 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.77 sec. MSE loss 1.45E+00. Sample per second: 59440\n",
      "FUNK_SVD: Epoch 89 of 500. Elapsed time 15.89 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.97 sec. MSE loss 1.43E+00. Sample per second: 58345\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.97 sec. Users per second: 1516\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1056939, PRECISION_RECALL_MIN_DEN: 0.1176031, RECALL: 0.0545886, MAP: 0.0480431, MAP_MIN_DEN: 0.0528574, MRR: 0.2484481, NDCG: 0.0980808, F1: 0.0719939, HIT_RATE: 0.5625830, ARHR_ALL_HITS: 0.3399149, NOVELTY: 0.0225769, AVERAGE_POPULARITY: 0.6856804, DIVERSITY_MEAN_INTER_LIST: 0.7240333, DIVERSITY_HERFINDAHL: 0.9723913, COVERAGE_ITEM: 0.0685037, COVERAGE_ITEM_CORRECT: 0.0363121, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5610927, DIVERSITY_GINI: 0.0101774, SHANNON_ENTROPY: 5.6854394, RATIO_DIVERSITY_HERFINDAHL: 0.9731857, RATIO_DIVERSITY_GINI: 0.0291078, RATIO_SHANNON_ENTROPY: 0.5258029, RATIO_AVERAGE_POPULARITY: 2.4824079, RATIO_NOVELTY: 0.0764991, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 90 of 500. Elapsed time 16.13 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.86 sec. MSE loss 1.42E+00. Sample per second: 58934\n",
      "FUNK_SVD: Epoch 91 of 500. Elapsed time 16.29 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.30 sec. MSE loss 1.41E+00. Sample per second: 56635\n",
      "FUNK_SVD: Epoch 92 of 500. Elapsed time 16.47 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.04 sec. MSE loss 1.39E+00. Sample per second: 57999\n",
      "FUNK_SVD: Epoch 93 of 500. Elapsed time 16.64 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.38 sec. MSE loss 1.38E+00. Sample per second: 61669\n",
      "FUNK_SVD: Epoch 94 of 500. Elapsed time 16.82 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.61 sec. MSE loss 1.37E+00. Sample per second: 60302\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.09 sec. Users per second: 1184\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1061255, PRECISION_RECALL_MIN_DEN: 0.1180260, RECALL: 0.0547781, MAP: 0.0480661, MAP_MIN_DEN: 0.0528728, MRR: 0.2484375, NDCG: 0.0984729, F1: 0.0722589, HIT_RATE: 0.5642430, ARHR_ALL_HITS: 0.3400309, NOVELTY: 0.0225722, AVERAGE_POPULARITY: 0.6863162, DIVERSITY_MEAN_INTER_LIST: 0.7207278, DIVERSITY_HERFINDAHL: 0.9720608, COVERAGE_ITEM: 0.0661859, COVERAGE_ITEM_CORRECT: 0.0357971, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5627483, DIVERSITY_GINI: 0.0100132, SHANNON_ENTROPY: 5.6628398, RATIO_DIVERSITY_HERFINDAHL: 0.9728549, RATIO_DIVERSITY_GINI: 0.0286381, RATIO_SHANNON_ENTROPY: 0.5237129, RATIO_AVERAGE_POPULARITY: 2.4847098, RATIO_NOVELTY: 0.0764832, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 95 of 500. Elapsed time 17.07 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.20 sec. MSE loss 1.36E+00. Sample per second: 57169\n",
      "FUNK_SVD: Epoch 96 of 500. Elapsed time 17.25 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.54 sec. MSE loss 1.35E+00. Sample per second: 60724\n",
      "FUNK_SVD: Epoch 97 of 500. Elapsed time 17.42 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.85 sec. MSE loss 1.34E+00. Sample per second: 58979\n",
      "FUNK_SVD: Epoch 98 of 500. Elapsed time 17.59 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.10 sec. MSE loss 1.33E+00. Sample per second: 57631\n",
      "FUNK_SVD: Epoch 99 of 500. Elapsed time 17.76 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.26 sec. MSE loss 1.32E+00. Sample per second: 62414\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.64 sec. Users per second: 1298\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1066899, PRECISION_RECALL_MIN_DEN: 0.1186356, RECALL: 0.0550898, MAP: 0.0481856, MAP_MIN_DEN: 0.0529944, MRR: 0.2483064, NDCG: 0.0989752, F1: 0.0726608, HIT_RATE: 0.5665671, ARHR_ALL_HITS: 0.3404281, NOVELTY: 0.0225697, AVERAGE_POPULARITY: 0.6866038, DIVERSITY_MEAN_INTER_LIST: 0.7174303, DIVERSITY_HERFINDAHL: 0.9717311, COVERAGE_ITEM: 0.0643832, COVERAGE_ITEM_CORRECT: 0.0355395, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5650662, DIVERSITY_GINI: 0.0098646, SHANNON_ENTROPY: 5.6415784, RATIO_DIVERSITY_HERFINDAHL: 0.9725250, RATIO_DIVERSITY_GINI: 0.0282131, RATIO_SHANNON_ENTROPY: 0.5217466, RATIO_AVERAGE_POPULARITY: 2.4857512, RATIO_NOVELTY: 0.0764747, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 100 of 500. Elapsed time 18.01 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.63 sec. MSE loss 1.30E+00. Sample per second: 60211\n",
      "FUNK_SVD: Epoch 101 of 500. Elapsed time 18.17 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 8.38 sec. MSE loss 1.29E+00. Sample per second: 76372\n",
      "FUNK_SVD: Epoch 102 of 500. Elapsed time 18.30 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 7.88 sec. MSE loss 1.29E+00. Sample per second: 81222\n",
      "FUNK_SVD: Epoch 103 of 500. Elapsed time 18.42 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 8.43 sec. MSE loss 1.27E+00. Sample per second: 75968\n",
      "FUNK_SVD: Epoch 104 of 500. Elapsed time 18.55 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 8.30 sec. MSE loss 1.27E+00. Sample per second: 77084\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.21 sec. Users per second: 1877\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1067729, PRECISION_RECALL_MIN_DEN: 0.1187962, RECALL: 0.0552655, MAP: 0.0481710, MAP_MIN_DEN: 0.0529691, MRR: 0.2483556, NDCG: 0.0992672, F1: 0.0728328, HIT_RATE: 0.5670651, ARHR_ALL_HITS: 0.3404274, NOVELTY: 0.0225681, AVERAGE_POPULARITY: 0.6867575, DIVERSITY_MEAN_INTER_LIST: 0.7146155, DIVERSITY_HERFINDAHL: 0.9714497, COVERAGE_ITEM: 0.0618079, COVERAGE_ITEM_CORRECT: 0.0345094, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5655629, DIVERSITY_GINI: 0.0097403, SHANNON_ENTROPY: 5.6240993, RATIO_DIVERSITY_HERFINDAHL: 0.9722433, RATIO_DIVERSITY_GINI: 0.0278575, RATIO_SHANNON_ENTROPY: 0.5201300, RATIO_AVERAGE_POPULARITY: 2.4863076, RATIO_NOVELTY: 0.0764694, \n",
      "\n",
      "FUNK_SVD: Epoch 105 of 500. Elapsed time 18.74 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Processed 640160 (100.0%) in 8.14 sec. MSE loss 1.26E+00. Sample per second: 78656\n",
      "FUNK_SVD: Epoch 106 of 500. Elapsed time 18.86 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 7.56 sec. MSE loss 1.25E+00. Sample per second: 84647\n",
      "FUNK_SVD: Epoch 107 of 500. Elapsed time 18.99 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 8.09 sec. MSE loss 1.24E+00. Sample per second: 79072\n",
      "FUNK_SVD: Epoch 108 of 500. Elapsed time 19.11 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 7.97 sec. MSE loss 1.23E+00. Sample per second: 80342\n",
      "FUNK_SVD: Epoch 109 of 500. Elapsed time 19.24 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 8.40 sec. MSE loss 1.22E+00. Sample per second: 76236\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.42 sec. Users per second: 1764\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1070717, PRECISION_RECALL_MIN_DEN: 0.1190805, RECALL: 0.0553969, MAP: 0.0483031, MAP_MIN_DEN: 0.0531144, MRR: 0.2483022, NDCG: 0.0995412, F1: 0.0730164, HIT_RATE: 0.5667331, ARHR_ALL_HITS: 0.3407529, NOVELTY: 0.0225680, AVERAGE_POPULARITY: 0.6866818, DIVERSITY_MEAN_INTER_LIST: 0.7121497, DIVERSITY_HERFINDAHL: 0.9712031, COVERAGE_ITEM: 0.0607777, COVERAGE_ITEM_CORRECT: 0.0345094, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5652318, DIVERSITY_GINI: 0.0096371, SHANNON_ENTROPY: 5.6089913, RATIO_DIVERSITY_HERFINDAHL: 0.9719966, RATIO_DIVERSITY_GINI: 0.0275625, RATIO_SHANNON_ENTROPY: 0.5187328, RATIO_AVERAGE_POPULARITY: 2.4860335, RATIO_NOVELTY: 0.0764689, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 110 of 500. Elapsed time 19.42 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 8.30 sec. MSE loss 1.22E+00. Sample per second: 77097\n",
      "FUNK_SVD: Epoch 111 of 500. Elapsed time 19.55 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 8.15 sec. MSE loss 1.21E+00. Sample per second: 78528\n",
      "FUNK_SVD: Epoch 112 of 500. Elapsed time 19.68 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 8.19 sec. MSE loss 1.20E+00. Sample per second: 78176\n",
      "FUNK_SVD: Epoch 113 of 500. Elapsed time 19.81 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 9.97 sec. MSE loss 1.19E+00. Sample per second: 64179\n",
      "FUNK_SVD: Epoch 114 of 500. Elapsed time 19.98 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.21 sec. MSE loss 1.18E+00. Sample per second: 57114\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.72 sec. Users per second: 1276\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1074535, PRECISION_RECALL_MIN_DEN: 0.1194035, RECALL: 0.0555308, MAP: 0.0483598, MAP_MIN_DEN: 0.0531502, MRR: 0.2484545, NDCG: 0.0997872, F1: 0.0732215, HIT_RATE: 0.5678951, ARHR_ALL_HITS: 0.3411344, NOVELTY: 0.0225700, AVERAGE_POPULARITY: 0.6862987, DIVERSITY_MEAN_INTER_LIST: 0.7101337, DIVERSITY_HERFINDAHL: 0.9710016, COVERAGE_ITEM: 0.0610353, COVERAGE_ITEM_CORRECT: 0.0345094, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5663907, DIVERSITY_GINI: 0.0095610, SHANNON_ENTROPY: 5.5976326, RATIO_DIVERSITY_HERFINDAHL: 0.9717949, RATIO_DIVERSITY_GINI: 0.0273448, RATIO_SHANNON_ENTROPY: 0.5176823, RATIO_AVERAGE_POPULARITY: 2.4846465, RATIO_NOVELTY: 0.0764759, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 115 of 500. Elapsed time 20.23 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.36 sec. MSE loss 1.18E+00. Sample per second: 56355\n",
      "FUNK_SVD: Epoch 116 of 500. Elapsed time 20.40 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.47 sec. MSE loss 1.17E+00. Sample per second: 61113\n",
      "FUNK_SVD: Epoch 117 of 500. Elapsed time 20.57 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.61 sec. MSE loss 1.17E+00. Sample per second: 60324\n",
      "FUNK_SVD: Epoch 118 of 500. Elapsed time 20.74 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.07 sec. MSE loss 1.16E+00. Sample per second: 57845\n",
      "FUNK_SVD: Epoch 119 of 500. Elapsed time 20.91 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.14 sec. MSE loss 1.15E+00. Sample per second: 63108\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.73 sec. Users per second: 1274\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1074203, PRECISION_RECALL_MIN_DEN: 0.1194445, RECALL: 0.0555594, MAP: 0.0484041, MAP_MIN_DEN: 0.0531864, MRR: 0.2487290, NDCG: 0.0999831, F1: 0.0732386, HIT_RATE: 0.5672311, ARHR_ALL_HITS: 0.3414505, NOVELTY: 0.0225731, AVERAGE_POPULARITY: 0.6857761, DIVERSITY_MEAN_INTER_LIST: 0.7084019, DIVERSITY_HERFINDAHL: 0.9708284, COVERAGE_ITEM: 0.0597476, COVERAGE_ITEM_CORRECT: 0.0337368, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5657285, DIVERSITY_GINI: 0.0094915, SHANNON_ENTROPY: 5.5880472, RATIO_DIVERSITY_HERFINDAHL: 0.9716216, RATIO_DIVERSITY_GINI: 0.0271460, RATIO_SHANNON_ENTROPY: 0.5167959, RATIO_AVERAGE_POPULARITY: 2.4827546, RATIO_NOVELTY: 0.0764862, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 120 of 500. Elapsed time 21.16 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.26 sec. MSE loss 1.14E+00. Sample per second: 56832\n",
      "FUNK_SVD: Epoch 121 of 500. Elapsed time 21.33 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.50 sec. MSE loss 1.14E+00. Sample per second: 60996\n",
      "FUNK_SVD: Epoch 122 of 500. Elapsed time 21.50 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.73 sec. MSE loss 1.13E+00. Sample per second: 59650\n",
      "FUNK_SVD: Epoch 123 of 500. Elapsed time 21.67 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.11 sec. MSE loss 1.13E+00. Sample per second: 57642\n",
      "FUNK_SVD: Epoch 124 of 500. Elapsed time 21.85 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 8.98 sec. MSE loss 1.12E+00. Sample per second: 71293\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.69 sec. Users per second: 1286\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1070385, PRECISION_RECALL_MIN_DEN: 0.1188306, RECALL: 0.0551456, MAP: 0.0482876, MAP_MIN_DEN: 0.0530368, MRR: 0.2488489, NDCG: 0.0999155, F1: 0.0727902, HIT_RATE: 0.5660691, ARHR_ALL_HITS: 0.3410583, NOVELTY: 0.0225781, AVERAGE_POPULARITY: 0.6849712, DIVERSITY_MEAN_INTER_LIST: 0.7072533, DIVERSITY_HERFINDAHL: 0.9707136, COVERAGE_ITEM: 0.0592326, COVERAGE_ITEM_CORRECT: 0.0334793, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5645695, DIVERSITY_GINI: 0.0094607, SHANNON_ENTROPY: 5.5834881, RATIO_DIVERSITY_HERFINDAHL: 0.9715066, RATIO_DIVERSITY_GINI: 0.0270580, RATIO_SHANNON_ENTROPY: 0.5163742, RATIO_AVERAGE_POPULARITY: 2.4798406, RATIO_NOVELTY: 0.0765034, \n",
      "\n",
      "FUNK_SVD: Epoch 125 of 500. Elapsed time 22.07 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.96 sec. MSE loss 1.12E+00. Sample per second: 58406\n",
      "FUNK_SVD: Epoch 126 of 500. Elapsed time 22.24 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.08 sec. MSE loss 1.11E+00. Sample per second: 57796\n",
      "FUNK_SVD: Epoch 127 of 500. Elapsed time 22.41 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.38 sec. MSE loss 1.11E+00. Sample per second: 61665\n",
      "FUNK_SVD: Epoch 128 of 500. Elapsed time 22.58 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.66 sec. MSE loss 1.10E+00. Sample per second: 60059\n",
      "FUNK_SVD: Epoch 129 of 500. Elapsed time 22.75 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.72 sec. MSE loss 1.10E+00. Sample per second: 59717\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.80 sec. Users per second: 1256\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1065571, PRECISION_RECALL_MIN_DEN: 0.1182689, RECALL: 0.0548588, MAP: 0.0481005, MAP_MIN_DEN: 0.0528331, MRR: 0.2482404, NDCG: 0.0997651, F1: 0.0724290, HIT_RATE: 0.5642430, ARHR_ALL_HITS: 0.3400698, NOVELTY: 0.0225839, AVERAGE_POPULARITY: 0.6840265, DIVERSITY_MEAN_INTER_LIST: 0.7061860, DIVERSITY_HERFINDAHL: 0.9706069, COVERAGE_ITEM: 0.0587175, COVERAGE_ITEM_CORRECT: 0.0327067, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5627483, DIVERSITY_GINI: 0.0094272, SHANNON_ENTROPY: 5.5786911, RATIO_DIVERSITY_HERFINDAHL: 0.9713998, RATIO_DIVERSITY_GINI: 0.0269620, RATIO_SHANNON_ENTROPY: 0.5159306, RATIO_AVERAGE_POPULARITY: 2.4764203, RATIO_NOVELTY: 0.0765230, \n",
      "\n",
      "FUNK_SVD: Epoch 130 of 500. Elapsed time 23.00 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.64 sec. MSE loss 1.09E+00. Sample per second: 60152\n",
      "FUNK_SVD: Epoch 131 of 500. Elapsed time 23.17 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.08 sec. MSE loss 1.08E+00. Sample per second: 57795\n",
      "FUNK_SVD: Epoch 132 of 500. Elapsed time 23.34 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.44 sec. MSE loss 1.08E+00. Sample per second: 61295\n",
      "FUNK_SVD: Epoch 133 of 500. Elapsed time 23.52 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Processed 640160 (100.0%) in 10.55 sec. MSE loss 1.08E+00. Sample per second: 60667\n",
      "FUNK_SVD: Epoch 134 of 500. Elapsed time 23.69 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.04 sec. MSE loss 1.07E+00. Sample per second: 57964\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.58 sec. Users per second: 1316\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1063579, PRECISION_RECALL_MIN_DEN: 0.1179115, RECALL: 0.0545429, MAP: 0.0480363, MAP_MIN_DEN: 0.0527462, MRR: 0.2481029, NDCG: 0.0996293, F1: 0.0721074, HIT_RATE: 0.5625830, ARHR_ALL_HITS: 0.3398114, NOVELTY: 0.0225893, AVERAGE_POPULARITY: 0.6831954, DIVERSITY_MEAN_INTER_LIST: 0.7049758, DIVERSITY_HERFINDAHL: 0.9704859, COVERAGE_ITEM: 0.0592326, COVERAGE_ITEM_CORRECT: 0.0321916, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5610927, DIVERSITY_GINI: 0.0093991, SHANNON_ENTROPY: 5.5742194, RATIO_DIVERSITY_HERFINDAHL: 0.9712787, RATIO_DIVERSITY_GINI: 0.0268817, RATIO_SHANNON_ENTROPY: 0.5155170, RATIO_AVERAGE_POPULARITY: 2.4734116, RATIO_NOVELTY: 0.0765410, \n",
      "\n",
      "FUNK_SVD: Epoch 135 of 500. Elapsed time 23.94 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 9.66 sec. MSE loss 1.07E+00. Sample per second: 66215\n",
      "FUNK_SVD: Epoch 136 of 500. Elapsed time 24.09 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.21 sec. MSE loss 1.06E+00. Sample per second: 57107\n",
      "FUNK_SVD: Epoch 137 of 500. Elapsed time 24.26 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.82 sec. MSE loss 1.06E+00. Sample per second: 59121\n",
      "FUNK_SVD: Epoch 138 of 500. Elapsed time 24.44 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.06 sec. MSE loss 1.06E+00. Sample per second: 57859\n",
      "FUNK_SVD: Epoch 139 of 500. Elapsed time 24.61 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.21 sec. MSE loss 1.05E+00. Sample per second: 62687\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.68 sec. Users per second: 1287\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1062417, PRECISION_RECALL_MIN_DEN: 0.1177683, RECALL: 0.0544081, MAP: 0.0479462, MAP_MIN_DEN: 0.0526034, MRR: 0.2477698, NDCG: 0.0996447, F1: 0.0719629, HIT_RATE: 0.5610890, ARHR_ALL_HITS: 0.3393030, NOVELTY: 0.0225944, AVERAGE_POPULARITY: 0.6823567, DIVERSITY_MEAN_INTER_LIST: 0.7039985, DIVERSITY_HERFINDAHL: 0.9703882, COVERAGE_ITEM: 0.0584600, COVERAGE_ITEM_CORRECT: 0.0321916, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5596026, DIVERSITY_GINI: 0.0093707, SHANNON_ENTROPY: 5.5699380, RATIO_DIVERSITY_HERFINDAHL: 0.9711809, RATIO_DIVERSITY_GINI: 0.0268004, RATIO_SHANNON_ENTROPY: 0.5151211, RATIO_AVERAGE_POPULARITY: 2.4703752, RATIO_NOVELTY: 0.0765584, \n",
      "\n",
      "FUNK_SVD: Epoch 140 of 500. Elapsed time 24.86 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.26 sec. MSE loss 1.05E+00. Sample per second: 56862\n",
      "FUNK_SVD: Epoch 141 of 500. Elapsed time 25.03 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.69 sec. MSE loss 1.04E+00. Sample per second: 59851\n",
      "FUNK_SVD: Epoch 142 of 500. Elapsed time 25.21 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.93 sec. MSE loss 1.04E+00. Sample per second: 58583\n",
      "FUNK_SVD: Epoch 143 of 500. Elapsed time 25.38 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 11.11 sec. MSE loss 1.04E+00. Sample per second: 57608\n",
      "FUNK_SVD: Epoch 144 of 500. Elapsed time 25.55 min\n",
      "FUNK_SVD: Processed 640160 (100.0%) in 10.88 sec. MSE loss 1.03E+00. Sample per second: 58816\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.94 sec. Users per second: 1220\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1061587, PRECISION_RECALL_MIN_DEN: 0.1177099, RECALL: 0.0544000, MAP: 0.0478650, MAP_MIN_DEN: 0.0525093, MRR: 0.2474166, NDCG: 0.0997581, F1: 0.0719368, HIT_RATE: 0.5602590, ARHR_ALL_HITS: 0.3387552, NOVELTY: 0.0226018, AVERAGE_POPULARITY: 0.6812441, DIVERSITY_MEAN_INTER_LIST: 0.7034052, DIVERSITY_HERFINDAHL: 0.9703288, COVERAGE_ITEM: 0.0584600, COVERAGE_ITEM_CORRECT: 0.0316765, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5587748, DIVERSITY_GINI: 0.0093719, SHANNON_ENTROPY: 5.5698837, RATIO_DIVERSITY_HERFINDAHL: 0.9711216, RATIO_DIVERSITY_GINI: 0.0268039, RATIO_SHANNON_ENTROPY: 0.5151161, RATIO_AVERAGE_POPULARITY: 2.4663470, RATIO_NOVELTY: 0.0765834, \n",
      "\n",
      "FUNK_SVD: Convergence reached! Terminating at epoch 145. Best value for 'MAP' at epoch 120 is 0.0484. Elapsed time 25.81 min\n",
      "FUNK_SVD: Epoch 145 of 500. Elapsed time 25.81 min\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.90 sec. Users per second: 1228\n",
      "SearchBayesianSkopt: Config 5 is suboptimal. Config: {'epochs': 120, 'num_factors': 108, 'sgd_mode': 'adagrad', 'batch_size': 32, 'item_reg': 0.0035797717161438407, 'user_reg': 1.3811120614703808e-05, 'learning_rate': 0.001228995115651476} - results: PRECISION: 0.1074203, PRECISION_RECALL_MIN_DEN: 0.1194445, RECALL: 0.0555594, MAP: 0.0484041, MAP_MIN_DEN: 0.0531864, MRR: 0.2487290, NDCG: 0.0999831, F1: 0.0732386, HIT_RATE: 0.5672311, ARHR_ALL_HITS: 0.3414505, NOVELTY: 0.0225731, AVERAGE_POPULARITY: 0.6857761, DIVERSITY_MEAN_INTER_LIST: 0.7084019, DIVERSITY_HERFINDAHL: 0.9708284, COVERAGE_ITEM: 0.0597476, COVERAGE_ITEM_CORRECT: 0.0337368, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5657285, DIVERSITY_GINI: 0.0094915, SHANNON_ENTROPY: 5.5880472, RATIO_DIVERSITY_HERFINDAHL: 0.9716216, RATIO_DIVERSITY_GINI: 0.0271460, RATIO_SHANNON_ENTROPY: 0.5167959, RATIO_AVERAGE_POPULARITY: 2.4827546, RATIO_NOVELTY: 0.0764862, \n",
      "\n",
      "Iteration No: 6 ended. Search finished for the next optimal point.\n",
      "Time taken: 1554.1905\n",
      "Function value obtained: -0.0484\n",
      "Current minimum: -0.0505\n",
      "Iteration No: 7 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'epochs': 500, 'num_factors': 20, 'sgd_mode': 'adagrad', 'batch_size': 4, 'item_reg': 1.4509981505329461e-05, 'user_reg': 0.0021138581529642794, 'learning_rate': 0.00045300762673031024}\n",
      "MatrixFactorization_FunkSVD_Cython_Recommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.94 sec. MSE loss 1.31E+01. Sample per second: 217529\n",
      "FUNK_SVD: Epoch 1 of 500. Elapsed time 2.37 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.32 sec. MSE loss 1.17E+01. Sample per second: 193019\n",
      "FUNK_SVD: Epoch 2 of 500. Elapsed time 4.75 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.85 sec. MSE loss 1.09E+01. Sample per second: 224976\n",
      "FUNK_SVD: Epoch 3 of 500. Elapsed time 7.29 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.19 sec. MSE loss 1.02E+01. Sample per second: 200791\n",
      "FUNK_SVD: Epoch 4 of 500. Elapsed time 9.62 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.36 sec. MSE loss 9.71E+00. Sample per second: 271461\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.77 sec. Users per second: 1598\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0420153, PRECISION_RECALL_MIN_DEN: 0.0460530, RECALL: 0.0202226, MAP: 0.0168203, MAP_MIN_DEN: 0.0180770, MRR: 0.1221405, NDCG: 0.0344136, F1: 0.0273036, HIT_RATE: 0.3109230, ARHR_ALL_HITS: 0.1425794, NOVELTY: 0.0267964, AVERAGE_POPULARITY: 0.3336866, DIVERSITY_MEAN_INTER_LIST: 0.9692717, DIVERSITY_HERFINDAHL: 0.9969111, COVERAGE_ITEM: 0.5887201, COVERAGE_ITEM_CORRECT: 0.1125418, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3100993, DIVERSITY_GINI: 0.1515091, SHANNON_ENTROPY: 9.4434524, RATIO_DIVERSITY_HERFINDAHL: 0.9977255, RATIO_DIVERSITY_GINI: 0.4333210, RATIO_SHANNON_ENTROPY: 0.8733529, RATIO_AVERAGE_POPULARITY: 1.2080646, RATIO_NOVELTY: 0.0907963, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 5 of 500. Elapsed time 15.59 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.51 sec. MSE loss 9.25E+00. Sample per second: 255222\n",
      "FUNK_SVD: Epoch 6 of 500. Elapsed time 17.94 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.81 sec. MSE loss 8.87E+00. Sample per second: 228024\n",
      "FUNK_SVD: Epoch 7 of 500. Elapsed time 20.24 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.12 sec. MSE loss 8.49E+00. Sample per second: 205130\n",
      "FUNK_SVD: Epoch 8 of 500. Elapsed time 22.55 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.51 sec. MSE loss 8.17E+00. Sample per second: 254513\n",
      "FUNK_SVD: Epoch 9 of 500. Elapsed time 24.95 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.95 sec. MSE loss 7.87E+00. Sample per second: 216848\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.14 sec. Users per second: 1171\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0615704, PRECISION_RECALL_MIN_DEN: 0.0682427, RECALL: 0.0312979, MAP: 0.0264669, MAP_MIN_DEN: 0.0285956, MRR: 0.1732234, NDCG: 0.0535249, F1: 0.0415002, HIT_RATE: 0.4090305, ARHR_ALL_HITS: 0.2127078, NOVELTY: 0.0248514, AVERAGE_POPULARITY: 0.4667952, DIVERSITY_MEAN_INTER_LIST: 0.9278163, DIVERSITY_HERFINDAHL: 0.9927662, COVERAGE_ITEM: 0.4238990, COVERAGE_ITEM_CORRECT: 0.0963173, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4079470, DIVERSITY_GINI: 0.0752955, SHANNON_ENTROPY: 8.3102684, RATIO_DIVERSITY_HERFINDAHL: 0.9935773, RATIO_DIVERSITY_GINI: 0.2153478, RATIO_SHANNON_ENTROPY: 0.7685533, RATIO_AVERAGE_POPULARITY: 1.6899655, RATIO_NOVELTY: 0.0842059, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 10 of 500. Elapsed time 32.54 sec\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Processed 640136 (100.0%) in 2.52 sec. MSE loss 7.59E+00. Sample per second: 254156\n",
      "FUNK_SVD: Epoch 11 of 500. Elapsed time 34.95 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.82 sec. MSE loss 7.34E+00. Sample per second: 226678\n",
      "FUNK_SVD: Epoch 12 of 500. Elapsed time 37.26 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.15 sec. MSE loss 7.11E+00. Sample per second: 202834\n",
      "FUNK_SVD: Epoch 13 of 500. Elapsed time 39.59 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.59 sec. MSE loss 6.90E+00. Sample per second: 247272\n",
      "FUNK_SVD: Epoch 14 of 500. Elapsed time 42.02 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.03 sec. MSE loss 6.69E+00. Sample per second: 210879\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.86 sec. Users per second: 1241\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0746016, PRECISION_RECALL_MIN_DEN: 0.0827230, RECALL: 0.0380684, MAP: 0.0325137, MAP_MIN_DEN: 0.0351956, MRR: 0.1993404, NDCG: 0.0655356, F1: 0.0504121, HIT_RATE: 0.4638114, ARHR_ALL_HITS: 0.2522563, NOVELTY: 0.0239452, AVERAGE_POPULARITY: 0.5406393, DIVERSITY_MEAN_INTER_LIST: 0.8889076, DIVERSITY_HERFINDAHL: 0.9888760, COVERAGE_ITEM: 0.3165079, COVERAGE_ITEM_CORRECT: 0.0829256, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4625828, DIVERSITY_GINI: 0.0452592, SHANNON_ENTROPY: 7.5746230, RATIO_DIVERSITY_HERFINDAHL: 0.9896839, RATIO_DIVERSITY_GINI: 0.1294427, RATIO_SHANNON_ENTROPY: 0.7005191, RATIO_AVERAGE_POPULARITY: 1.9573076, RATIO_NOVELTY: 0.0811354, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 15 of 500. Elapsed time 49.35 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.38 sec. MSE loss 6.50E+00. Sample per second: 189016\n",
      "FUNK_SVD: Epoch 16 of 500. Elapsed time 51.82 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.80 sec. MSE loss 6.31E+00. Sample per second: 228663\n",
      "FUNK_SVD: Epoch 17 of 500. Elapsed time 54.23 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.22 sec. MSE loss 6.14E+00. Sample per second: 198733\n",
      "FUNK_SVD: Epoch 18 of 500. Elapsed time 56.65 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.55 sec. MSE loss 5.97E+00. Sample per second: 250653\n",
      "FUNK_SVD: Epoch 19 of 500. Elapsed time 58.99 sec\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.05 sec. MSE loss 5.82E+00. Sample per second: 209648\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.62 sec. Users per second: 1303\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0823705, PRECISION_RECALL_MIN_DEN: 0.0913555, RECALL: 0.0421051, MAP: 0.0364629, MAP_MIN_DEN: 0.0394728, MRR: 0.2156853, NDCG: 0.0731862, F1: 0.0557253, HIT_RATE: 0.4936919, ARHR_ALL_HITS: 0.2775489, NOVELTY: 0.0234662, AVERAGE_POPULARITY: 0.5857242, DIVERSITY_MEAN_INTER_LIST: 0.8565692, DIVERSITY_HERFINDAHL: 0.9856427, COVERAGE_ITEM: 0.2438836, COVERAGE_ITEM_CORRECT: 0.0723667, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4923841, DIVERSITY_GINI: 0.0311075, SHANNON_ENTROPY: 7.0805500, RATIO_DIVERSITY_HERFINDAHL: 0.9864479, RATIO_DIVERSITY_GINI: 0.0889686, RATIO_SHANNON_ENTROPY: 0.6548261, RATIO_AVERAGE_POPULARITY: 2.1205307, RATIO_NOVELTY: 0.0795126, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 20 of 500. Elapsed time 1.10 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.13 sec. MSE loss 5.68E+00. Sample per second: 204542\n",
      "FUNK_SVD: Epoch 21 of 500. Elapsed time 1.14 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.75 sec. MSE loss 5.53E+00. Sample per second: 232750\n",
      "FUNK_SVD: Epoch 22 of 500. Elapsed time 1.19 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.21 sec. MSE loss 5.39E+00. Sample per second: 199297\n",
      "FUNK_SVD: Epoch 23 of 500. Elapsed time 1.23 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.66 sec. MSE loss 5.27E+00. Sample per second: 240294\n",
      "FUNK_SVD: Epoch 24 of 500. Elapsed time 1.27 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.04 sec. MSE loss 5.14E+00. Sample per second: 210433\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.74 sec. Users per second: 1271\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0877988, PRECISION_RECALL_MIN_DEN: 0.0979271, RECALL: 0.0456359, MAP: 0.0391193, MAP_MIN_DEN: 0.0424710, MRR: 0.2255221, NDCG: 0.0786859, F1: 0.0600560, HIT_RATE: 0.5137782, ARHR_ALL_HITS: 0.2939049, NOVELTY: 0.0231825, AVERAGE_POPULARITY: 0.6150063, DIVERSITY_MEAN_INTER_LIST: 0.8303446, DIVERSITY_HERFINDAHL: 0.9830207, COVERAGE_ITEM: 0.2016482, COVERAGE_ITEM_CORRECT: 0.0628380, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5124172, DIVERSITY_GINI: 0.0238182, SHANNON_ENTROPY: 6.7408325, RATIO_DIVERSITY_HERFINDAHL: 0.9838238, RATIO_DIVERSITY_GINI: 0.0681207, RATIO_SHANNON_ENTROPY: 0.6234082, RATIO_AVERAGE_POPULARITY: 2.2265427, RATIO_NOVELTY: 0.0785511, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 25 of 500. Elapsed time 1.39 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.23 sec. MSE loss 5.03E+00. Sample per second: 198311\n",
      "FUNK_SVD: Epoch 26 of 500. Elapsed time 1.43 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.56 sec. MSE loss 4.92E+00. Sample per second: 250047\n",
      "FUNK_SVD: Epoch 27 of 500. Elapsed time 1.47 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.90 sec. MSE loss 4.80E+00. Sample per second: 220787\n",
      "FUNK_SVD: Epoch 28 of 500. Elapsed time 1.51 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.33 sec. MSE loss 4.70E+00. Sample per second: 192007\n",
      "FUNK_SVD: Epoch 29 of 500. Elapsed time 1.55 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.69 sec. MSE loss 4.60E+00. Sample per second: 237504\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.06 sec. Users per second: 1192\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0921481, PRECISION_RECALL_MIN_DEN: 0.1028868, RECALL: 0.0479683, MAP: 0.0412928, MAP_MIN_DEN: 0.0448497, MRR: 0.2327793, NDCG: 0.0826980, F1: 0.0630931, HIT_RATE: 0.5277224, ARHR_ALL_HITS: 0.3065280, NOVELTY: 0.0229995, AVERAGE_POPULARITY: 0.6354450, DIVERSITY_MEAN_INTER_LIST: 0.8086618, DIVERSITY_HERFINDAHL: 0.9808528, COVERAGE_ITEM: 0.1740922, COVERAGE_ITEM_CORRECT: 0.0571723, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5263245, DIVERSITY_GINI: 0.0195329, SHANNON_ENTROPY: 6.4931209, RATIO_DIVERSITY_HERFINDAHL: 0.9816541, RATIO_DIVERSITY_GINI: 0.0558647, RATIO_SHANNON_ENTROPY: 0.6004992, RATIO_AVERAGE_POPULARITY: 2.3005381, RATIO_NOVELTY: 0.0779312, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 30 of 500. Elapsed time 1.67 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.24 sec. MSE loss 4.51E+00. Sample per second: 197523\n",
      "FUNK_SVD: Epoch 31 of 500. Elapsed time 1.71 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.63 sec. MSE loss 4.41E+00. Sample per second: 242847\n",
      "FUNK_SVD: Epoch 32 of 500. Elapsed time 1.75 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.06 sec. MSE loss 4.33E+00. Sample per second: 209195\n",
      "FUNK_SVD: Epoch 33 of 500. Elapsed time 1.79 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.51 sec. MSE loss 4.24E+00. Sample per second: 254964\n",
      "FUNK_SVD: Epoch 34 of 500. Elapsed time 1.83 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.87 sec. MSE loss 4.16E+00. Sample per second: 223016\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.87 sec. Users per second: 1238\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0951859, PRECISION_RECALL_MIN_DEN: 0.1063510, RECALL: 0.0496690, MAP: 0.0428098, MAP_MIN_DEN: 0.0465196, MRR: 0.2376980, NDCG: 0.0856087, F1: 0.0652762, HIT_RATE: 0.5361886, ARHR_ALL_HITS: 0.3152375, NOVELTY: 0.0228670, AVERAGE_POPULARITY: 0.6511495, DIVERSITY_MEAN_INTER_LIST: 0.7897667, DIVERSITY_HERFINDAHL: 0.9789636, COVERAGE_ITEM: 0.1496266, COVERAGE_ITEM_CORRECT: 0.0512490, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5347682, DIVERSITY_GINI: 0.0166498, SHANNON_ENTROPY: 6.2956777, RATIO_DIVERSITY_HERFINDAHL: 0.9797633, RATIO_DIVERSITY_GINI: 0.0476191, RATIO_SHANNON_ENTROPY: 0.5822392, RATIO_AVERAGE_POPULARITY: 2.3573938, RATIO_NOVELTY: 0.0774823, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 35 of 500. Elapsed time 1.95 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.24 sec. MSE loss 4.08E+00. Sample per second: 197315\n",
      "FUNK_SVD: Epoch 36 of 500. Elapsed time 1.99 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.65 sec. MSE loss 3.99E+00. Sample per second: 241164\n",
      "FUNK_SVD: Epoch 37 of 500. Elapsed time 2.03 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.08 sec. MSE loss 3.92E+00. Sample per second: 207570\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Epoch 38 of 500. Elapsed time 2.08 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.55 sec. MSE loss 3.85E+00. Sample per second: 250448\n",
      "FUNK_SVD: Epoch 39 of 500. Elapsed time 2.12 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.95 sec. MSE loss 3.78E+00. Sample per second: 216814\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.02 sec. Users per second: 1199\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0970120, PRECISION_RECALL_MIN_DEN: 0.1082082, RECALL: 0.0504449, MAP: 0.0439043, MAP_MIN_DEN: 0.0477554, MRR: 0.2416044, NDCG: 0.0878581, F1: 0.0663755, HIT_RATE: 0.5418327, ARHR_ALL_HITS: 0.3217275, NOVELTY: 0.0227753, AVERAGE_POPULARITY: 0.6623323, DIVERSITY_MEAN_INTER_LIST: 0.7748141, DIVERSITY_HERFINDAHL: 0.9774685, COVERAGE_ITEM: 0.1300541, COVERAGE_ITEM_CORRECT: 0.0471285, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5403974, DIVERSITY_GINI: 0.0147258, SHANNON_ENTROPY: 6.1450618, RATIO_DIVERSITY_HERFINDAHL: 0.9782671, RATIO_DIVERSITY_GINI: 0.0421162, RATIO_SHANNON_ENTROPY: 0.5683099, RATIO_AVERAGE_POPULARITY: 2.3978796, RATIO_NOVELTY: 0.0771713, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 40 of 500. Elapsed time 2.24 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.32 sec. MSE loss 3.71E+00. Sample per second: 275926\n",
      "FUNK_SVD: Epoch 41 of 500. Elapsed time 2.28 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.75 sec. MSE loss 3.65E+00. Sample per second: 231936\n",
      "FUNK_SVD: Epoch 42 of 500. Elapsed time 2.32 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.24 sec. MSE loss 3.57E+00. Sample per second: 197088\n",
      "FUNK_SVD: Epoch 43 of 500. Elapsed time 2.36 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.72 sec. MSE loss 3.52E+00. Sample per second: 235436\n",
      "FUNK_SVD: Epoch 44 of 500. Elapsed time 2.40 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.07 sec. MSE loss 3.46E+00. Sample per second: 208184\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.86 sec. Users per second: 1238\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.0985392, PRECISION_RECALL_MIN_DEN: 0.1097852, RECALL: 0.0509849, MAP: 0.0448391, MAP_MIN_DEN: 0.0487653, MRR: 0.2442778, NDCG: 0.0894660, F1: 0.0672000, HIT_RATE: 0.5446547, ARHR_ALL_HITS: 0.3269565, NOVELTY: 0.0227048, AVERAGE_POPULARITY: 0.6713165, DIVERSITY_MEAN_INTER_LIST: 0.7614361, DIVERSITY_HERFINDAHL: 0.9761310, COVERAGE_ITEM: 0.1156322, COVERAGE_ITEM_CORRECT: 0.0448107, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5432119, DIVERSITY_GINI: 0.0133795, SHANNON_ENTROPY: 6.0241497, RATIO_DIVERSITY_HERFINDAHL: 0.9769284, RATIO_DIVERSITY_GINI: 0.0382658, RATIO_SHANNON_ENTROPY: 0.5571277, RATIO_AVERAGE_POPULARITY: 2.4304058, RATIO_NOVELTY: 0.0769326, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 45 of 500. Elapsed time 2.52 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.89 sec. MSE loss 3.40E+00. Sample per second: 221488\n",
      "FUNK_SVD: Epoch 46 of 500. Elapsed time 2.56 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.24 sec. MSE loss 3.35E+00. Sample per second: 285180\n",
      "FUNK_SVD: Epoch 47 of 500. Elapsed time 2.58 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.74 sec. MSE loss 3.29E+00. Sample per second: 233241\n",
      "FUNK_SVD: Epoch 48 of 500. Elapsed time 2.62 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.31 sec. MSE loss 3.24E+00. Sample per second: 193169\n",
      "FUNK_SVD: Epoch 49 of 500. Elapsed time 2.66 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.73 sec. MSE loss 3.19E+00. Sample per second: 233619\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.12 sec. Users per second: 1176\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1001992, PRECISION_RECALL_MIN_DEN: 0.1116686, RECALL: 0.0518367, MAP: 0.0456437, MAP_MIN_DEN: 0.0496363, MRR: 0.2462791, NDCG: 0.0909853, F1: 0.0683259, HIT_RATE: 0.5491368, ARHR_ALL_HITS: 0.3310789, NOVELTY: 0.0226499, AVERAGE_POPULARITY: 0.6785160, DIVERSITY_MEAN_INTER_LIST: 0.7495817, DIVERSITY_HERFINDAHL: 0.9749457, COVERAGE_ITEM: 0.1024981, COVERAGE_ITEM_CORRECT: 0.0406902, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5476821, DIVERSITY_GINI: 0.0123206, SHANNON_ENTROPY: 5.9210507, RATIO_DIVERSITY_HERFINDAHL: 0.9757422, RATIO_DIVERSITY_GINI: 0.0352372, RATIO_SHANNON_ENTROPY: 0.5475928, RATIO_AVERAGE_POPULARITY: 2.4564704, RATIO_NOVELTY: 0.0767466, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 50 of 500. Elapsed time 2.79 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.39 sec. MSE loss 3.14E+00. Sample per second: 189043\n",
      "FUNK_SVD: Epoch 51 of 500. Elapsed time 2.83 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.76 sec. MSE loss 3.08E+00. Sample per second: 232320\n",
      "FUNK_SVD: Epoch 52 of 500. Elapsed time 2.87 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.08 sec. MSE loss 3.04E+00. Sample per second: 207301\n",
      "FUNK_SVD: Epoch 53 of 500. Elapsed time 2.91 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.47 sec. MSE loss 2.99E+00. Sample per second: 258540\n",
      "FUNK_SVD: Epoch 54 of 500. Elapsed time 2.95 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.93 sec. MSE loss 2.95E+00. Sample per second: 218305\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.95 sec. Users per second: 1217\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1009628, PRECISION_RECALL_MIN_DEN: 0.1124573, RECALL: 0.0521801, MAP: 0.0461284, MAP_MIN_DEN: 0.0501631, MRR: 0.2473077, NDCG: 0.0919013, F1: 0.0688017, HIT_RATE: 0.5517928, ARHR_ALL_HITS: 0.3333225, NOVELTY: 0.0226076, AVERAGE_POPULARITY: 0.6843534, DIVERSITY_MEAN_INTER_LIST: 0.7393032, DIVERSITY_HERFINDAHL: 0.9739180, COVERAGE_ITEM: 0.0945146, COVERAGE_ITEM_CORRECT: 0.0396601, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5503311, DIVERSITY_GINI: 0.0115498, SHANNON_ENTROPY: 5.8372999, RATIO_DIVERSITY_HERFINDAHL: 0.9747137, RATIO_DIVERSITY_GINI: 0.0330328, RATIO_SHANNON_ENTROPY: 0.5398473, RATIO_AVERAGE_POPULARITY: 2.4776040, RATIO_NOVELTY: 0.0766032, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 55 of 500. Elapsed time 3.07 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.21 sec. MSE loss 2.91E+00. Sample per second: 199148\n",
      "FUNK_SVD: Epoch 56 of 500. Elapsed time 3.11 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.52 sec. MSE loss 2.86E+00. Sample per second: 254312\n",
      "FUNK_SVD: Epoch 57 of 500. Elapsed time 3.15 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.93 sec. MSE loss 2.82E+00. Sample per second: 218653\n",
      "FUNK_SVD: Epoch 58 of 500. Elapsed time 3.19 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.28 sec. MSE loss 2.78E+00. Sample per second: 194692\n",
      "FUNK_SVD: Epoch 59 of 500. Elapsed time 3.23 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.59 sec. MSE loss 2.75E+00. Sample per second: 246588\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.70 sec. Users per second: 1282\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1019422, PRECISION_RECALL_MIN_DEN: 0.1136315, RECALL: 0.0528600, MAP: 0.0466752, MAP_MIN_DEN: 0.0508007, MRR: 0.2487759, NDCG: 0.0930423, F1: 0.0696200, HIT_RATE: 0.5539509, ARHR_ALL_HITS: 0.3362022, NOVELTY: 0.0225688, AVERAGE_POPULARITY: 0.6896862, DIVERSITY_MEAN_INTER_LIST: 0.7293789, DIVERSITY_HERFINDAHL: 0.9729258, COVERAGE_ITEM: 0.0860160, COVERAGE_ITEM_CORRECT: 0.0370847, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5524834, DIVERSITY_GINI: 0.0108786, SHANNON_ENTROPY: 5.7586974, RATIO_DIVERSITY_HERFINDAHL: 0.9737206, RATIO_DIVERSITY_GINI: 0.0311131, RATIO_SHANNON_ENTROPY: 0.5325780, RATIO_AVERAGE_POPULARITY: 2.4969106, RATIO_NOVELTY: 0.0764717, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 60 of 500. Elapsed time 3.35 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.77 sec. MSE loss 2.71E+00. Sample per second: 231329\n",
      "FUNK_SVD: Epoch 61 of 500. Elapsed time 3.39 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.21 sec. MSE loss 2.67E+00. Sample per second: 199296\n",
      "FUNK_SVD: Epoch 62 of 500. Elapsed time 3.43 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.64 sec. MSE loss 2.64E+00. Sample per second: 242096\n",
      "FUNK_SVD: Epoch 63 of 500. Elapsed time 3.47 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.02 sec. MSE loss 2.60E+00. Sample per second: 211687\n",
      "FUNK_SVD: Epoch 64 of 500. Elapsed time 3.51 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.50 sec. MSE loss 2.56E+00. Sample per second: 256057\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.65 sec. Users per second: 1294\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1031541, PRECISION_RECALL_MIN_DEN: 0.1149100, RECALL: 0.0534253, MAP: 0.0472375, MAP_MIN_DEN: 0.0514044, MRR: 0.2508450, NDCG: 0.0941957, F1: 0.0703929, HIT_RATE: 0.5589309, ARHR_ALL_HITS: 0.3394688, NOVELTY: 0.0225398, AVERAGE_POPULARITY: 0.6937215, DIVERSITY_MEAN_INTER_LIST: 0.7211373, DIVERSITY_HERFINDAHL: 0.9721018, COVERAGE_ITEM: 0.0803502, COVERAGE_ITEM_CORRECT: 0.0352820, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5574503, DIVERSITY_GINI: 0.0103738, SHANNON_ENTROPY: 5.6957925, RATIO_DIVERSITY_HERFINDAHL: 0.9728959, RATIO_DIVERSITY_GINI: 0.0296694, RATIO_SHANNON_ENTROPY: 0.5267604, RATIO_AVERAGE_POPULARITY: 2.5115196, RATIO_NOVELTY: 0.0763736, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 65 of 500. Elapsed time 3.63 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.57 sec. MSE loss 2.53E+00. Sample per second: 249109\n",
      "FUNK_SVD: Epoch 66 of 500. Elapsed time 3.67 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.85 sec. MSE loss 2.51E+00. Sample per second: 224191\n",
      "FUNK_SVD: Epoch 67 of 500. Elapsed time 3.70 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.17 sec. MSE loss 2.47E+00. Sample per second: 201570\n",
      "FUNK_SVD: Epoch 68 of 500. Elapsed time 3.74 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.61 sec. MSE loss 2.45E+00. Sample per second: 245079\n",
      "FUNK_SVD: Epoch 69 of 500. Elapsed time 3.78 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.96 sec. MSE loss 2.41E+00. Sample per second: 215864\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.60 sec. Users per second: 1310\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1038679, PRECISION_RECALL_MIN_DEN: 0.1155947, RECALL: 0.0536422, MAP: 0.0476763, MAP_MIN_DEN: 0.0518529, MRR: 0.2520376, NDCG: 0.0949702, F1: 0.0707472, HIT_RATE: 0.5599270, ARHR_ALL_HITS: 0.3417539, NOVELTY: 0.0225142, AVERAGE_POPULARITY: 0.6973293, DIVERSITY_MEAN_INTER_LIST: 0.7135398, DIVERSITY_HERFINDAHL: 0.9713421, COVERAGE_ITEM: 0.0749421, COVERAGE_ITEM_CORRECT: 0.0334793, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5584437, DIVERSITY_GINI: 0.0099419, SHANNON_ENTROPY: 5.6386781, RATIO_DIVERSITY_HERFINDAHL: 0.9721357, RATIO_DIVERSITY_GINI: 0.0284341, RATIO_SHANNON_ENTROPY: 0.5214783, RATIO_AVERAGE_POPULARITY: 2.5245813, RATIO_NOVELTY: 0.0762869, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 70 of 500. Elapsed time 3.90 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.87 sec. MSE loss 2.39E+00. Sample per second: 222781\n",
      "FUNK_SVD: Epoch 71 of 500. Elapsed time 3.94 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.25 sec. MSE loss 2.36E+00. Sample per second: 196363\n",
      "FUNK_SVD: Epoch 72 of 500. Elapsed time 3.98 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.62 sec. MSE loss 2.33E+00. Sample per second: 243846\n",
      "FUNK_SVD: Epoch 73 of 500. Elapsed time 4.02 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.95 sec. MSE loss 2.30E+00. Sample per second: 216558\n",
      "FUNK_SVD: Epoch 74 of 500. Elapsed time 4.06 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.38 sec. MSE loss 2.27E+00. Sample per second: 189476\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.72 sec. Users per second: 1277\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1045817, PRECISION_RECALL_MIN_DEN: 0.1165155, RECALL: 0.0542201, MAP: 0.0481075, MAP_MIN_DEN: 0.0523278, MRR: 0.2534875, NDCG: 0.0958182, F1: 0.0714152, HIT_RATE: 0.5619190, ARHR_ALL_HITS: 0.3442146, NOVELTY: 0.0224945, AVERAGE_POPULARITY: 0.7001880, DIVERSITY_MEAN_INTER_LIST: 0.7067899, DIVERSITY_HERFINDAHL: 0.9706673, COVERAGE_ITEM: 0.0710791, COVERAGE_ITEM_CORRECT: 0.0319341, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5604305, DIVERSITY_GINI: 0.0095973, SHANNON_ENTROPY: 5.5901640, RATIO_DIVERSITY_HERFINDAHL: 0.9714603, RATIO_DIVERSITY_GINI: 0.0274486, RATIO_SHANNON_ENTROPY: 0.5169916, RATIO_AVERAGE_POPULARITY: 2.5349306, RATIO_NOVELTY: 0.0762201, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 75 of 500. Elapsed time 4.18 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.49 sec. MSE loss 2.25E+00. Sample per second: 256589\n",
      "FUNK_SVD: Epoch 76 of 500. Elapsed time 4.22 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.87 sec. MSE loss 2.22E+00. Sample per second: 223393\n",
      "FUNK_SVD: Epoch 77 of 500. Elapsed time 4.25 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.18 sec. MSE loss 2.20E+00. Sample per second: 201474\n",
      "FUNK_SVD: Epoch 78 of 500. Elapsed time 4.29 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.46 sec. MSE loss 2.18E+00. Sample per second: 260145\n",
      "FUNK_SVD: Epoch 79 of 500. Elapsed time 4.33 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.73 sec. MSE loss 2.15E+00. Sample per second: 234462\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.74 sec. Users per second: 1271\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1051793, PRECISION_RECALL_MIN_DEN: 0.1171682, RECALL: 0.0544396, MAP: 0.0483456, MAP_MIN_DEN: 0.0525749, MRR: 0.2540410, NDCG: 0.0965337, F1: 0.0717449, HIT_RATE: 0.5642430, ARHR_ALL_HITS: 0.3452621, NOVELTY: 0.0224763, AVERAGE_POPULARITY: 0.7027630, DIVERSITY_MEAN_INTER_LIST: 0.7006027, DIVERSITY_HERFINDAHL: 0.9700486, COVERAGE_ITEM: 0.0672161, COVERAGE_ITEM_CORRECT: 0.0311615, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5627483, DIVERSITY_GINI: 0.0092945, SHANNON_ENTROPY: 5.5462078, RATIO_DIVERSITY_HERFINDAHL: 0.9708411, RATIO_DIVERSITY_GINI: 0.0265826, RATIO_SHANNON_ENTROPY: 0.5129265, RATIO_AVERAGE_POPULARITY: 2.5442531, RATIO_NOVELTY: 0.0761582, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 80 of 500. Elapsed time 4.45 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.87 sec. MSE loss 2.13E+00. Sample per second: 223135\n",
      "FUNK_SVD: Epoch 81 of 500. Elapsed time 4.49 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.21 sec. MSE loss 2.11E+00. Sample per second: 199299\n",
      "FUNK_SVD: Epoch 82 of 500. Elapsed time 4.53 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.52 sec. MSE loss 2.09E+00. Sample per second: 253626\n",
      "FUNK_SVD: Epoch 83 of 500. Elapsed time 4.57 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.81 sec. MSE loss 2.07E+00. Sample per second: 227842\n",
      "FUNK_SVD: Epoch 84 of 500. Elapsed time 4.60 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.19 sec. MSE loss 2.06E+00. Sample per second: 200660\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.33 sec. Users per second: 1390\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1055611, PRECISION_RECALL_MIN_DEN: 0.1174888, RECALL: 0.0544498, MAP: 0.0486277, MAP_MIN_DEN: 0.0528832, MRR: 0.2545033, NDCG: 0.0970852, F1: 0.0718423, HIT_RATE: 0.5640770, ARHR_ALL_HITS: 0.3464108, NOVELTY: 0.0224612, AVERAGE_POPULARITY: 0.7049362, DIVERSITY_MEAN_INTER_LIST: 0.6949457, DIVERSITY_HERFINDAHL: 0.9694830, COVERAGE_ITEM: 0.0641257, COVERAGE_ITEM_CORRECT: 0.0301313, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5625828, DIVERSITY_GINI: 0.0090456, SHANNON_ENTROPY: 5.5083571, RATIO_DIVERSITY_HERFINDAHL: 0.9702751, RATIO_DIVERSITY_GINI: 0.0258709, RATIO_SHANNON_ENTROPY: 0.5094259, RATIO_AVERAGE_POPULARITY: 2.5521209, RATIO_NOVELTY: 0.0761072, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 85 of 500. Elapsed time 4.72 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.71 sec. MSE loss 2.03E+00. Sample per second: 236232\n",
      "FUNK_SVD: Epoch 86 of 500. Elapsed time 4.75 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.05 sec. MSE loss 2.01E+00. Sample per second: 312261\n",
      "FUNK_SVD: Epoch 87 of 500. Elapsed time 4.77 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.11 sec. MSE loss 1.99E+00. Sample per second: 303299\n",
      "FUNK_SVD: Epoch 88 of 500. Elapsed time 4.81 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 1.50 sec. MSE loss 1.97E+00. Sample per second: 427040\n",
      "FUNK_SVD: Epoch 89 of 500. Elapsed time 4.83 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.55 sec. MSE loss 1.96E+00. Sample per second: 250883\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.49 sec. Users per second: 1724\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1061089, PRECISION_RECALL_MIN_DEN: 0.1180532, RECALL: 0.0547042, MAP: 0.0488990, MAP_MIN_DEN: 0.0532127, MRR: 0.2551478, NDCG: 0.0977740, F1: 0.0721907, HIT_RATE: 0.5655710, ARHR_ALL_HITS: 0.3477942, NOVELTY: 0.0224483, AVERAGE_POPULARITY: 0.7067790, DIVERSITY_MEAN_INTER_LIST: 0.6897066, DIVERSITY_HERFINDAHL: 0.9689592, COVERAGE_ITEM: 0.0615503, COVERAGE_ITEM_CORRECT: 0.0296163, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5640728, DIVERSITY_GINI: 0.0088192, SHANNON_ENTROPY: 5.4732014, RATIO_DIVERSITY_HERFINDAHL: 0.9697508, RATIO_DIVERSITY_GINI: 0.0252231, RATIO_SHANNON_ENTROPY: 0.5061746, RATIO_AVERAGE_POPULARITY: 2.5587924, RATIO_NOVELTY: 0.0760633, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 90 of 500. Elapsed time 4.93 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Processed 640136 (100.0%) in 1.46 sec. MSE loss 1.94E+00. Sample per second: 436962\n",
      "FUNK_SVD: Epoch 91 of 500. Elapsed time 4.95 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.45 sec. MSE loss 1.92E+00. Sample per second: 260963\n",
      "FUNK_SVD: Epoch 92 of 500. Elapsed time 4.98 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 1.85 sec. MSE loss 1.90E+00. Sample per second: 346836\n",
      "FUNK_SVD: Epoch 93 of 500. Elapsed time 5.00 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.82 sec. MSE loss 1.88E+00. Sample per second: 226834\n",
      "FUNK_SVD: Epoch 94 of 500. Elapsed time 5.04 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.20 sec. MSE loss 1.87E+00. Sample per second: 291385\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.53 sec. Users per second: 1707\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1066567, PRECISION_RECALL_MIN_DEN: 0.1187991, RECALL: 0.0551608, MAP: 0.0491069, MAP_MIN_DEN: 0.0534290, MRR: 0.2556876, NDCG: 0.0982044, F1: 0.0727149, HIT_RATE: 0.5667331, ARHR_ALL_HITS: 0.3488573, NOVELTY: 0.0224366, AVERAGE_POPULARITY: 0.7084729, DIVERSITY_MEAN_INTER_LIST: 0.6846985, DIVERSITY_HERFINDAHL: 0.9684585, COVERAGE_ITEM: 0.0602627, COVERAGE_ITEM_CORRECT: 0.0288437, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5652318, DIVERSITY_GINI: 0.0086228, SHANNON_ENTROPY: 5.4410236, RATIO_DIVERSITY_HERFINDAHL: 0.9692497, RATIO_DIVERSITY_GINI: 0.0246614, RATIO_SHANNON_ENTROPY: 0.5031988, RATIO_AVERAGE_POPULARITY: 2.5649252, RATIO_NOVELTY: 0.0760238, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 95 of 500. Elapsed time 5.12 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.54 sec. MSE loss 1.85E+00. Sample per second: 251630\n",
      "FUNK_SVD: Epoch 96 of 500. Elapsed time 5.15 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.12 sec. MSE loss 1.84E+00. Sample per second: 301966\n",
      "FUNK_SVD: Epoch 97 of 500. Elapsed time 5.18 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 1.88 sec. MSE loss 1.83E+00. Sample per second: 340640\n",
      "FUNK_SVD: Epoch 98 of 500. Elapsed time 5.21 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.47 sec. MSE loss 1.81E+00. Sample per second: 258916\n",
      "FUNK_SVD: Epoch 99 of 500. Elapsed time 5.23 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.23 sec. MSE loss 1.80E+00. Sample per second: 286463\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.28 sec. Users per second: 1838\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1068891, PRECISION_RECALL_MIN_DEN: 0.1189585, RECALL: 0.0552210, MAP: 0.0492986, MAP_MIN_DEN: 0.0537009, MRR: 0.2566390, NDCG: 0.0986444, F1: 0.0728212, HIT_RATE: 0.5672311, ARHR_ALL_HITS: 0.3501518, NOVELTY: 0.0224260, AVERAGE_POPULARITY: 0.7099929, DIVERSITY_MEAN_INTER_LIST: 0.6802624, DIVERSITY_HERFINDAHL: 0.9680149, COVERAGE_ITEM: 0.0571723, COVERAGE_ITEM_CORRECT: 0.0285861, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5657285, DIVERSITY_GINI: 0.0084384, SHANNON_ENTROPY: 5.4109340, RATIO_DIVERSITY_HERFINDAHL: 0.9688058, RATIO_DIVERSITY_GINI: 0.0241340, RATIO_SHANNON_ENTROPY: 0.5004160, RATIO_AVERAGE_POPULARITY: 2.5704279, RATIO_NOVELTY: 0.0759878, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 100 of 500. Elapsed time 5.32 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.31 sec. MSE loss 1.78E+00. Sample per second: 277196\n",
      "FUNK_SVD: Epoch 101 of 500. Elapsed time 5.35 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 1.90 sec. MSE loss 1.77E+00. Sample per second: 336201\n",
      "FUNK_SVD: Epoch 102 of 500. Elapsed time 5.37 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.67 sec. MSE loss 1.76E+00. Sample per second: 240114\n",
      "FUNK_SVD: Epoch 103 of 500. Elapsed time 5.40 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.25 sec. MSE loss 1.74E+00. Sample per second: 283753\n",
      "FUNK_SVD: Epoch 104 of 500. Elapsed time 5.43 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.08 sec. MSE loss 1.73E+00. Sample per second: 307406\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.53 sec. Users per second: 1708\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1073871, PRECISION_RECALL_MIN_DEN: 0.1194387, RECALL: 0.0553709, MAP: 0.0495062, MAP_MIN_DEN: 0.0539076, MRR: 0.2568603, NDCG: 0.0991544, F1: 0.0730670, HIT_RATE: 0.5683931, ARHR_ALL_HITS: 0.3510455, NOVELTY: 0.0224170, AVERAGE_POPULARITY: 0.7112674, DIVERSITY_MEAN_INTER_LIST: 0.6759574, DIVERSITY_HERFINDAHL: 0.9675845, COVERAGE_ITEM: 0.0548545, COVERAGE_ITEM_CORRECT: 0.0272985, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5668874, DIVERSITY_GINI: 0.0082726, SHANNON_ENTROPY: 5.3831524, RATIO_DIVERSITY_HERFINDAHL: 0.9683750, RATIO_DIVERSITY_GINI: 0.0236599, RATIO_SHANNON_ENTROPY: 0.4978467, RATIO_AVERAGE_POPULARITY: 2.5750420, RATIO_NOVELTY: 0.0759574, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 105 of 500. Elapsed time 5.52 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.07 sec. MSE loss 1.72E+00. Sample per second: 308948\n",
      "FUNK_SVD: Epoch 106 of 500. Elapsed time 5.54 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.05 sec. MSE loss 1.70E+00. Sample per second: 312523\n",
      "FUNK_SVD: Epoch 107 of 500. Elapsed time 5.57 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 1.40 sec. MSE loss 1.70E+00. Sample per second: 457045\n",
      "FUNK_SVD: Epoch 108 of 500. Elapsed time 5.60 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.46 sec. MSE loss 1.68E+00. Sample per second: 259915\n",
      "FUNK_SVD: Epoch 109 of 500. Elapsed time 5.63 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 1.82 sec. MSE loss 1.67E+00. Sample per second: 351784\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.50 sec. Users per second: 1721\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1077523, PRECISION_RECALL_MIN_DEN: 0.1199388, RECALL: 0.0556074, MAP: 0.0496755, MAP_MIN_DEN: 0.0541054, MRR: 0.2568597, NDCG: 0.0995091, F1: 0.0733575, HIT_RATE: 0.5687251, ARHR_ALL_HITS: 0.3515398, NOVELTY: 0.0224085, AVERAGE_POPULARITY: 0.7125150, DIVERSITY_MEAN_INTER_LIST: 0.6713630, DIVERSITY_HERFINDAHL: 0.9671252, COVERAGE_ITEM: 0.0530518, COVERAGE_ITEM_CORRECT: 0.0270409, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5672185, DIVERSITY_GINI: 0.0081194, SHANNON_ENTROPY: 5.3560102, RATIO_DIVERSITY_HERFINDAHL: 0.9679153, RATIO_DIVERSITY_GINI: 0.0232218, RATIO_SHANNON_ENTROPY: 0.4953365, RATIO_AVERAGE_POPULARITY: 2.5795590, RATIO_NOVELTY: 0.0759286, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 110 of 500. Elapsed time 5.71 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.39 sec. MSE loss 1.66E+00. Sample per second: 267654\n",
      "FUNK_SVD: Epoch 111 of 500. Elapsed time 5.75 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 1.76 sec. MSE loss 1.65E+00. Sample per second: 364368\n",
      "FUNK_SVD: Epoch 112 of 500. Elapsed time 5.77 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.76 sec. MSE loss 1.63E+00. Sample per second: 231878\n",
      "FUNK_SVD: Epoch 113 of 500. Elapsed time 5.80 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.14 sec. MSE loss 1.63E+00. Sample per second: 299806\n",
      "FUNK_SVD: Epoch 114 of 500. Elapsed time 5.83 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.10 sec. MSE loss 1.61E+00. Sample per second: 304653\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.38 sec. Users per second: 1780\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1077855, PRECISION_RECALL_MIN_DEN: 0.1199382, RECALL: 0.0555642, MAP: 0.0497972, MAP_MIN_DEN: 0.0542348, MRR: 0.2571883, NDCG: 0.0997991, F1: 0.0733276, HIT_RATE: 0.5685591, ARHR_ALL_HITS: 0.3522321, NOVELTY: 0.0224015, AVERAGE_POPULARITY: 0.7135761, DIVERSITY_MEAN_INTER_LIST: 0.6671995, DIVERSITY_HERFINDAHL: 0.9667089, COVERAGE_ITEM: 0.0512490, COVERAGE_ITEM_CORRECT: 0.0262683, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5670530, DIVERSITY_GINI: 0.0079846, SHANNON_ENTROPY: 5.3318355, RATIO_DIVERSITY_HERFINDAHL: 0.9674986, RATIO_DIVERSITY_GINI: 0.0228362, RATIO_SHANNON_ENTROPY: 0.4931008, RATIO_AVERAGE_POPULARITY: 2.5834004, RATIO_NOVELTY: 0.0759048, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 115 of 500. Elapsed time 5.92 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.12 sec. MSE loss 1.61E+00. Sample per second: 302215\n",
      "FUNK_SVD: Epoch 116 of 500. Elapsed time 5.94 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 1.91 sec. MSE loss 1.60E+00. Sample per second: 334475\n",
      "FUNK_SVD: Epoch 117 of 500. Elapsed time 5.97 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.51 sec. MSE loss 1.59E+00. Sample per second: 254879\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Epoch 118 of 500. Elapsed time 6.00 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.28 sec. MSE loss 1.57E+00. Sample per second: 280767\n",
      "FUNK_SVD: Epoch 119 of 500. Elapsed time 6.03 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 1.89 sec. MSE loss 1.57E+00. Sample per second: 338934\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.43 sec. Users per second: 1754\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1078353, PRECISION_RECALL_MIN_DEN: 0.1199836, RECALL: 0.0555464, MAP: 0.0498711, MAP_MIN_DEN: 0.0543150, MRR: 0.2579400, NDCG: 0.1000591, F1: 0.0733235, HIT_RATE: 0.5693891, ARHR_ALL_HITS: 0.3530325, NOVELTY: 0.0223953, AVERAGE_POPULARITY: 0.7144836, DIVERSITY_MEAN_INTER_LIST: 0.6634879, DIVERSITY_HERFINDAHL: 0.9663378, COVERAGE_ITEM: 0.0499614, COVERAGE_ITEM_CORRECT: 0.0262683, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5678808, DIVERSITY_GINI: 0.0078667, SHANNON_ENTROPY: 5.3099273, RATIO_DIVERSITY_HERFINDAHL: 0.9671272, RATIO_DIVERSITY_GINI: 0.0224990, RATIO_SHANNON_ENTROPY: 0.4910747, RATIO_AVERAGE_POPULARITY: 2.5866858, RATIO_NOVELTY: 0.0758839, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 120 of 500. Elapsed time 6.11 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 1.89 sec. MSE loss 1.56E+00. Sample per second: 338708\n",
      "FUNK_SVD: Epoch 121 of 500. Elapsed time 6.14 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.74 sec. MSE loss 1.55E+00. Sample per second: 233876\n",
      "FUNK_SVD: Epoch 122 of 500. Elapsed time 6.17 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.27 sec. MSE loss 1.54E+00. Sample per second: 282159\n",
      "FUNK_SVD: Epoch 123 of 500. Elapsed time 6.20 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.22 sec. MSE loss 1.53E+00. Sample per second: 288144\n",
      "FUNK_SVD: Epoch 124 of 500. Elapsed time 6.23 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 1.72 sec. MSE loss 1.53E+00. Sample per second: 371321\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.34 sec. Users per second: 1806\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1081839, PRECISION_RECALL_MIN_DEN: 0.1204318, RECALL: 0.0558565, MAP: 0.0500041, MAP_MIN_DEN: 0.0544492, MRR: 0.2584738, NDCG: 0.1005022, F1: 0.0736742, HIT_RATE: 0.5702191, ARHR_ALL_HITS: 0.3539038, NOVELTY: 0.0223887, AVERAGE_POPULARITY: 0.7154455, DIVERSITY_MEAN_INTER_LIST: 0.6596230, DIVERSITY_HERFINDAHL: 0.9659514, COVERAGE_ITEM: 0.0486737, COVERAGE_ITEM_CORRECT: 0.0262683, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5687086, DIVERSITY_GINI: 0.0077442, SHANNON_ENTROPY: 5.2869724, RATIO_DIVERSITY_HERFINDAHL: 0.9667405, RATIO_DIVERSITY_GINI: 0.0221487, RATIO_SHANNON_ENTROPY: 0.4889517, RATIO_AVERAGE_POPULARITY: 2.5901682, RATIO_NOVELTY: 0.0758616, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 125 of 500. Elapsed time 6.31 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.15 sec. MSE loss 1.52E+00. Sample per second: 297580\n",
      "FUNK_SVD: Epoch 126 of 500. Elapsed time 6.34 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 1.59 sec. MSE loss 1.51E+00. Sample per second: 402024\n",
      "FUNK_SVD: Epoch 127 of 500. Elapsed time 6.37 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.22 sec. MSE loss 1.50E+00. Sample per second: 288304\n",
      "FUNK_SVD: Epoch 128 of 500. Elapsed time 6.39 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.63 sec. MSE loss 1.49E+00. Sample per second: 242949\n",
      "FUNK_SVD: Epoch 129 of 500. Elapsed time 6.43 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.37 sec. MSE loss 1.49E+00. Sample per second: 270377\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.96 sec. Users per second: 1522\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1083997, PRECISION_RECALL_MIN_DEN: 0.1206504, RECALL: 0.0559379, MAP: 0.0501497, MAP_MIN_DEN: 0.0545969, MRR: 0.2586421, NDCG: 0.1007958, F1: 0.0737951, HIT_RATE: 0.5695551, ARHR_ALL_HITS: 0.3545300, NOVELTY: 0.0223845, AVERAGE_POPULARITY: 0.7160730, DIVERSITY_MEAN_INTER_LIST: 0.6563896, DIVERSITY_HERFINDAHL: 0.9656281, COVERAGE_ITEM: 0.0466134, COVERAGE_ITEM_CORRECT: 0.0254958, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5680464, DIVERSITY_GINI: 0.0076473, SHANNON_ENTROPY: 5.2686797, RATIO_DIVERSITY_HERFINDAHL: 0.9664170, RATIO_DIVERSITY_GINI: 0.0218715, RATIO_SHANNON_ENTROPY: 0.4872600, RATIO_AVERAGE_POPULARITY: 2.5924400, RATIO_NOVELTY: 0.0758471, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 130 of 500. Elapsed time 6.53 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.91 sec. MSE loss 1.48E+00. Sample per second: 218927\n",
      "FUNK_SVD: Epoch 131 of 500. Elapsed time 6.57 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.29 sec. MSE loss 1.47E+00. Sample per second: 194451\n",
      "FUNK_SVD: Epoch 132 of 500. Elapsed time 6.61 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.71 sec. MSE loss 1.47E+00. Sample per second: 236111\n",
      "FUNK_SVD: Epoch 133 of 500. Elapsed time 6.65 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.09 sec. MSE loss 1.46E+00. Sample per second: 206999\n",
      "FUNK_SVD: Epoch 134 of 500. Elapsed time 6.69 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.50 sec. MSE loss 1.45E+00. Sample per second: 256428\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.79 sec. Users per second: 1258\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1086155, PRECISION_RECALL_MIN_DEN: 0.1208810, RECALL: 0.0560898, MAP: 0.0502754, MAP_MIN_DEN: 0.0547094, MRR: 0.2590549, NDCG: 0.1010960, F1: 0.0739772, HIT_RATE: 0.5695551, ARHR_ALL_HITS: 0.3551699, NOVELTY: 0.0223795, AVERAGE_POPULARITY: 0.7167724, DIVERSITY_MEAN_INTER_LIST: 0.6528385, DIVERSITY_HERFINDAHL: 0.9652730, COVERAGE_ITEM: 0.0453258, COVERAGE_ITEM_CORRECT: 0.0249807, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5680464, DIVERSITY_GINI: 0.0075344, SHANNON_ENTROPY: 5.2469390, RATIO_DIVERSITY_HERFINDAHL: 0.9660616, RATIO_DIVERSITY_GINI: 0.0215487, RATIO_SHANNON_ENTROPY: 0.4852494, RATIO_AVERAGE_POPULARITY: 2.5949721, RATIO_NOVELTY: 0.0758303, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 135 of 500. Elapsed time 6.81 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.69 sec. MSE loss 1.45E+00. Sample per second: 237819\n",
      "FUNK_SVD: Epoch 136 of 500. Elapsed time 6.85 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.02 sec. MSE loss 1.44E+00. Sample per second: 212061\n",
      "FUNK_SVD: Epoch 137 of 500. Elapsed time 6.89 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.37 sec. MSE loss 1.43E+00. Sample per second: 269999\n",
      "FUNK_SVD: Epoch 138 of 500. Elapsed time 6.93 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.70 sec. MSE loss 1.43E+00. Sample per second: 237351\n",
      "FUNK_SVD: Epoch 139 of 500. Elapsed time 6.97 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.13 sec. MSE loss 1.42E+00. Sample per second: 204540\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.63 sec. Users per second: 1302\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1085491, PRECISION_RECALL_MIN_DEN: 0.1209861, RECALL: 0.0562567, MAP: 0.0502905, MAP_MIN_DEN: 0.0547632, MRR: 0.2592685, NDCG: 0.1013008, F1: 0.0741068, HIT_RATE: 0.5692231, ARHR_ALL_HITS: 0.3553255, NOVELTY: 0.0223759, AVERAGE_POPULARITY: 0.7172913, DIVERSITY_MEAN_INTER_LIST: 0.6497458, DIVERSITY_HERFINDAHL: 0.9649638, COVERAGE_ITEM: 0.0448107, COVERAGE_ITEM_CORRECT: 0.0244656, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5677152, DIVERSITY_GINI: 0.0074512, SHANNON_ENTROPY: 5.2303942, RATIO_DIVERSITY_HERFINDAHL: 0.9657521, RATIO_DIVERSITY_GINI: 0.0213105, RATIO_SHANNON_ENTROPY: 0.4837193, RATIO_AVERAGE_POPULARITY: 2.5968508, RATIO_NOVELTY: 0.0758181, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 140 of 500. Elapsed time 7.09 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.21 sec. MSE loss 1.42E+00. Sample per second: 199297\n",
      "FUNK_SVD: Epoch 141 of 500. Elapsed time 7.13 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.67 sec. MSE loss 1.41E+00. Sample per second: 239413\n",
      "FUNK_SVD: Epoch 142 of 500. Elapsed time 7.17 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.06 sec. MSE loss 1.40E+00. Sample per second: 209484\n",
      "FUNK_SVD: Epoch 143 of 500. Elapsed time 7.21 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.38 sec. MSE loss 1.40E+00. Sample per second: 268496\n",
      "FUNK_SVD: Epoch 144 of 500. Elapsed time 7.25 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.73 sec. MSE loss 1.39E+00. Sample per second: 234554\n",
      "FUNK_SVD: Validation begins...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.68 sec. Users per second: 1287\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1086653, PRECISION_RECALL_MIN_DEN: 0.1211343, RECALL: 0.0563226, MAP: 0.0504157, MAP_MIN_DEN: 0.0549187, MRR: 0.2596839, NDCG: 0.1016023, F1: 0.0741910, HIT_RATE: 0.5690571, ARHR_ALL_HITS: 0.3560143, NOVELTY: 0.0223732, AVERAGE_POPULARITY: 0.7176940, DIVERSITY_MEAN_INTER_LIST: 0.6467894, DIVERSITY_HERFINDAHL: 0.9646682, COVERAGE_ITEM: 0.0442956, COVERAGE_ITEM_CORRECT: 0.0244656, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5675497, DIVERSITY_GINI: 0.0073780, SHANNON_ENTROPY: 5.2156390, RATIO_DIVERSITY_HERFINDAHL: 0.9654563, RATIO_DIVERSITY_GINI: 0.0211013, RATIO_SHANNON_ENTROPY: 0.4823547, RATIO_AVERAGE_POPULARITY: 2.5983087, RATIO_NOVELTY: 0.0758089, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 145 of 500. Elapsed time 7.36 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.80 sec. MSE loss 1.39E+00. Sample per second: 228544\n",
      "FUNK_SVD: Epoch 146 of 500. Elapsed time 7.40 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.14 sec. MSE loss 1.38E+00. Sample per second: 203602\n",
      "FUNK_SVD: Epoch 147 of 500. Elapsed time 7.44 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.54 sec. MSE loss 1.38E+00. Sample per second: 251632\n",
      "FUNK_SVD: Epoch 148 of 500. Elapsed time 7.48 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.85 sec. MSE loss 1.37E+00. Sample per second: 224522\n",
      "FUNK_SVD: Epoch 149 of 500. Elapsed time 7.52 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.41 sec. MSE loss 1.37E+00. Sample per second: 187607\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.81 sec. Users per second: 1253\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1089309, PRECISION_RECALL_MIN_DEN: 0.1214642, RECALL: 0.0565319, MAP: 0.0505256, MAP_MIN_DEN: 0.0550330, MRR: 0.2596515, NDCG: 0.1018794, F1: 0.0744345, HIT_RATE: 0.5690571, ARHR_ALL_HITS: 0.3563353, NOVELTY: 0.0223697, AVERAGE_POPULARITY: 0.7182066, DIVERSITY_MEAN_INTER_LIST: 0.6437482, DIVERSITY_HERFINDAHL: 0.9643641, COVERAGE_ITEM: 0.0430080, COVERAGE_ITEM_CORRECT: 0.0244656, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5675497, DIVERSITY_GINI: 0.0072931, SHANNON_ENTROPY: 5.1987441, RATIO_DIVERSITY_HERFINDAHL: 0.9651520, RATIO_DIVERSITY_GINI: 0.0208585, RATIO_SHANNON_ENTROPY: 0.4807922, RATIO_AVERAGE_POPULARITY: 2.6001647, RATIO_NOVELTY: 0.0757970, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 150 of 500. Elapsed time 7.64 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.55 sec. MSE loss 1.36E+00. Sample per second: 250710\n",
      "FUNK_SVD: Epoch 151 of 500. Elapsed time 7.68 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.97 sec. MSE loss 1.36E+00. Sample per second: 215175\n",
      "FUNK_SVD: Epoch 152 of 500. Elapsed time 7.72 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.32 sec. MSE loss 1.36E+00. Sample per second: 192791\n",
      "FUNK_SVD: Epoch 153 of 500. Elapsed time 7.76 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.66 sec. MSE loss 1.35E+00. Sample per second: 240268\n",
      "FUNK_SVD: Epoch 154 of 500. Elapsed time 7.80 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.02 sec. MSE loss 1.35E+00. Sample per second: 211793\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.67 sec. Users per second: 1290\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1089973, PRECISION_RECALL_MIN_DEN: 0.1215882, RECALL: 0.0566625, MAP: 0.0505987, MAP_MIN_DEN: 0.0551694, MRR: 0.2600630, NDCG: 0.1021903, F1: 0.0745632, HIT_RATE: 0.5697211, ARHR_ALL_HITS: 0.3568057, NOVELTY: 0.0223673, AVERAGE_POPULARITY: 0.7185401, DIVERSITY_MEAN_INTER_LIST: 0.6410540, DIVERSITY_HERFINDAHL: 0.9640948, COVERAGE_ITEM: 0.0417203, COVERAGE_ITEM_CORRECT: 0.0234355, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5682119, DIVERSITY_GINI: 0.0072241, SHANNON_ENTROPY: 5.1845897, RATIO_DIVERSITY_HERFINDAHL: 0.9648824, RATIO_DIVERSITY_GINI: 0.0206613, RATIO_SHANNON_ENTROPY: 0.4794832, RATIO_AVERAGE_POPULARITY: 2.6013719, RATIO_NOVELTY: 0.0757891, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 155 of 500. Elapsed time 7.92 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.01 sec. MSE loss 1.34E+00. Sample per second: 212247\n",
      "FUNK_SVD: Epoch 156 of 500. Elapsed time 7.96 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.36 sec. MSE loss 1.33E+00. Sample per second: 271396\n",
      "FUNK_SVD: Epoch 157 of 500. Elapsed time 8.00 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.78 sec. MSE loss 1.33E+00. Sample per second: 229605\n",
      "FUNK_SVD: Epoch 158 of 500. Elapsed time 8.04 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.10 sec. MSE loss 1.33E+00. Sample per second: 206227\n",
      "FUNK_SVD: Epoch 159 of 500. Elapsed time 8.08 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.47 sec. MSE loss 1.33E+00. Sample per second: 258712\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.68 sec. Users per second: 1287\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1092961, PRECISION_RECALL_MIN_DEN: 0.1219167, RECALL: 0.0567997, MAP: 0.0507277, MAP_MIN_DEN: 0.0553172, MRR: 0.2600605, NDCG: 0.1024541, F1: 0.0747518, HIT_RATE: 0.5702191, ARHR_ALL_HITS: 0.3571989, NOVELTY: 0.0223651, AVERAGE_POPULARITY: 0.7188596, DIVERSITY_MEAN_INTER_LIST: 0.6384112, DIVERSITY_HERFINDAHL: 0.9638305, COVERAGE_ITEM: 0.0406902, COVERAGE_ITEM_CORRECT: 0.0239506, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5687086, DIVERSITY_GINI: 0.0071595, SHANNON_ENTROPY: 5.1709418, RATIO_DIVERSITY_HERFINDAHL: 0.9646179, RATIO_DIVERSITY_GINI: 0.0204763, RATIO_SHANNON_ENTROPY: 0.4782210, RATIO_AVERAGE_POPULARITY: 2.6025288, RATIO_NOVELTY: 0.0757814, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 160 of 500. Elapsed time 8.19 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.64 sec. MSE loss 1.32E+00. Sample per second: 242796\n",
      "FUNK_SVD: Epoch 161 of 500. Elapsed time 8.23 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.95 sec. MSE loss 1.32E+00. Sample per second: 216733\n",
      "FUNK_SVD: Epoch 162 of 500. Elapsed time 8.27 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.34 sec. MSE loss 1.31E+00. Sample per second: 191474\n",
      "FUNK_SVD: Epoch 163 of 500. Elapsed time 8.31 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.62 sec. MSE loss 1.31E+00. Sample per second: 244109\n",
      "FUNK_SVD: Epoch 164 of 500. Elapsed time 8.35 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.93 sec. MSE loss 1.31E+00. Sample per second: 218031\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.72 sec. Users per second: 1277\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1092131, PRECISION_RECALL_MIN_DEN: 0.1217977, RECALL: 0.0567570, MAP: 0.0507303, MAP_MIN_DEN: 0.0553212, MRR: 0.2603672, NDCG: 0.1025376, F1: 0.0746955, HIT_RATE: 0.5698871, ARHR_ALL_HITS: 0.3574485, NOVELTY: 0.0223632, AVERAGE_POPULARITY: 0.7191360, DIVERSITY_MEAN_INTER_LIST: 0.6358951, DIVERSITY_HERFINDAHL: 0.9635790, COVERAGE_ITEM: 0.0399176, COVERAGE_ITEM_CORRECT: 0.0236930, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5683775, DIVERSITY_GINI: 0.0071031, SHANNON_ENTROPY: 5.1590887, RATIO_DIVERSITY_HERFINDAHL: 0.9643662, RATIO_DIVERSITY_GINI: 0.0203152, RATIO_SHANNON_ENTROPY: 0.4771248, RATIO_AVERAGE_POPULARITY: 2.6035294, RATIO_NOVELTY: 0.0757750, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 165 of 500. Elapsed time 8.47 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.08 sec. MSE loss 1.30E+00. Sample per second: 208094\n",
      "FUNK_SVD: Epoch 166 of 500. Elapsed time 8.51 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.57 sec. MSE loss 1.30E+00. Sample per second: 249087\n",
      "FUNK_SVD: Epoch 167 of 500. Elapsed time 8.55 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.03 sec. MSE loss 1.29E+00. Sample per second: 211354\n",
      "FUNK_SVD: Epoch 168 of 500. Elapsed time 8.59 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.08 sec. MSE loss 1.29E+00. Sample per second: 307712\n",
      "FUNK_SVD: Epoch 169 of 500. Elapsed time 8.63 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 1.24 sec. MSE loss 1.29E+00. Sample per second: 516459\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.60 sec. Users per second: 1311\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1089475, PRECISION_RECALL_MIN_DEN: 0.1213775, RECALL: 0.0564717, MAP: 0.0506442, MAP_MIN_DEN: 0.0552043, MRR: 0.2599432, NDCG: 0.1023914, F1: 0.0743862, HIT_RATE: 0.5682271, ARHR_ALL_HITS: 0.3569130, NOVELTY: 0.0223615, AVERAGE_POPULARITY: 0.7193732, DIVERSITY_MEAN_INTER_LIST: 0.6333159, DIVERSITY_HERFINDAHL: 0.9633211, COVERAGE_ITEM: 0.0391450, COVERAGE_ITEM_CORRECT: 0.0236930, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5667219, DIVERSITY_GINI: 0.0070439, SHANNON_ENTROPY: 5.1461458, RATIO_DIVERSITY_HERFINDAHL: 0.9641081, RATIO_DIVERSITY_GINI: 0.0201457, RATIO_SHANNON_ENTROPY: 0.4759278, RATIO_AVERAGE_POPULARITY: 2.6043879, RATIO_NOVELTY: 0.0757693, \n",
      "\n",
      "FUNK_SVD: Epoch 170 of 500. Elapsed time 8.72 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Processed 640136 (100.0%) in 3.21 sec. MSE loss 1.28E+00. Sample per second: 199369\n",
      "FUNK_SVD: Epoch 171 of 500. Elapsed time 8.76 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.47 sec. MSE loss 1.28E+00. Sample per second: 259459\n",
      "FUNK_SVD: Epoch 172 of 500. Elapsed time 8.80 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.76 sec. MSE loss 1.28E+00. Sample per second: 231263\n",
      "FUNK_SVD: Epoch 173 of 500. Elapsed time 8.84 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.07 sec. MSE loss 1.28E+00. Sample per second: 208126\n",
      "FUNK_SVD: Epoch 174 of 500. Elapsed time 8.88 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.39 sec. MSE loss 1.27E+00. Sample per second: 267603\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.54 sec. Users per second: 1326\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1089143, PRECISION_RECALL_MIN_DEN: 0.1214095, RECALL: 0.0565850, MAP: 0.0506578, MAP_MIN_DEN: 0.0552166, MRR: 0.2603165, NDCG: 0.1025430, F1: 0.0744766, HIT_RATE: 0.5687251, ARHR_ALL_HITS: 0.3572263, NOVELTY: 0.0223599, AVERAGE_POPULARITY: 0.7195888, DIVERSITY_MEAN_INTER_LIST: 0.6308781, DIVERSITY_HERFINDAHL: 0.9630773, COVERAGE_ITEM: 0.0381149, COVERAGE_ITEM_CORRECT: 0.0234355, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5672185, DIVERSITY_GINI: 0.0069853, SHANNON_ENTROPY: 5.1335298, RATIO_DIVERSITY_HERFINDAHL: 0.9638641, RATIO_DIVERSITY_GINI: 0.0199782, RATIO_SHANNON_ENTROPY: 0.4747610, RATIO_AVERAGE_POPULARITY: 2.6051685, RATIO_NOVELTY: 0.0757638, \n",
      "\n",
      "FUNK_SVD: Epoch 175 of 500. Elapsed time 8.99 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.31 sec. MSE loss 1.27E+00. Sample per second: 192904\n",
      "FUNK_SVD: Epoch 176 of 500. Elapsed time 9.03 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.72 sec. MSE loss 1.27E+00. Sample per second: 235000\n",
      "FUNK_SVD: Epoch 177 of 500. Elapsed time 9.07 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.09 sec. MSE loss 1.26E+00. Sample per second: 206764\n",
      "FUNK_SVD: Epoch 178 of 500. Elapsed time 9.11 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.43 sec. MSE loss 1.26E+00. Sample per second: 263220\n",
      "FUNK_SVD: Epoch 179 of 500. Elapsed time 9.15 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.00 sec. MSE loss 1.26E+00. Sample per second: 213316\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.66 sec. Users per second: 1293\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1091965, PRECISION_RECALL_MIN_DEN: 0.1217006, RECALL: 0.0566887, MAP: 0.0507613, MAP_MIN_DEN: 0.0553258, MRR: 0.2600463, NDCG: 0.1028217, F1: 0.0746324, HIT_RATE: 0.5685591, ARHR_ALL_HITS: 0.3574276, NOVELTY: 0.0223591, AVERAGE_POPULARITY: 0.7196939, DIVERSITY_MEAN_INTER_LIST: 0.6285004, DIVERSITY_HERFINDAHL: 0.9628396, COVERAGE_ITEM: 0.0375998, COVERAGE_ITEM_CORRECT: 0.0234355, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5670530, DIVERSITY_GINI: 0.0069387, SHANNON_ENTROPY: 5.1231857, RATIO_DIVERSITY_HERFINDAHL: 0.9636262, RATIO_DIVERSITY_GINI: 0.0198450, RATIO_SHANNON_ENTROPY: 0.4738044, RATIO_AVERAGE_POPULARITY: 2.6055489, RATIO_NOVELTY: 0.0757610, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 180 of 500. Elapsed time 9.27 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.06 sec. MSE loss 1.26E+00. Sample per second: 208652\n",
      "FUNK_SVD: Epoch 181 of 500. Elapsed time 9.31 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.43 sec. MSE loss 1.25E+00. Sample per second: 263212\n",
      "FUNK_SVD: Epoch 182 of 500. Elapsed time 9.35 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.75 sec. MSE loss 1.25E+00. Sample per second: 232964\n",
      "FUNK_SVD: Epoch 183 of 500. Elapsed time 9.39 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.99 sec. MSE loss 1.25E+00. Sample per second: 214377\n",
      "FUNK_SVD: Epoch 184 of 500. Elapsed time 9.42 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.44 sec. MSE loss 1.25E+00. Sample per second: 186173\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.72 sec. Users per second: 1277\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1094788, PRECISION_RECALL_MIN_DEN: 0.1220326, RECALL: 0.0568548, MAP: 0.0508080, MAP_MIN_DEN: 0.0553957, MRR: 0.2600000, NDCG: 0.1030846, F1: 0.0748423, HIT_RATE: 0.5695551, ARHR_ALL_HITS: 0.3575486, NOVELTY: 0.0223579, AVERAGE_POPULARITY: 0.7198550, DIVERSITY_MEAN_INTER_LIST: 0.6263101, DIVERSITY_HERFINDAHL: 0.9626206, COVERAGE_ITEM: 0.0370847, COVERAGE_ITEM_CORRECT: 0.0234355, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5680464, DIVERSITY_GINI: 0.0068911, SHANNON_ENTROPY: 5.1121768, RATIO_DIVERSITY_HERFINDAHL: 0.9634070, RATIO_DIVERSITY_GINI: 0.0197089, RATIO_SHANNON_ENTROPY: 0.4727862, RATIO_AVERAGE_POPULARITY: 2.6061322, RATIO_NOVELTY: 0.0757570, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 185 of 500. Elapsed time 9.54 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.58 sec. MSE loss 1.25E+00. Sample per second: 248390\n",
      "FUNK_SVD: Epoch 186 of 500. Elapsed time 9.58 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.94 sec. MSE loss 1.24E+00. Sample per second: 217142\n",
      "FUNK_SVD: Epoch 187 of 500. Elapsed time 9.62 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.40 sec. MSE loss 1.24E+00. Sample per second: 188374\n",
      "FUNK_SVD: Epoch 188 of 500. Elapsed time 9.66 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.83 sec. MSE loss 1.24E+00. Sample per second: 225960\n",
      "FUNK_SVD: Epoch 189 of 500. Elapsed time 9.70 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.23 sec. MSE loss 1.23E+00. Sample per second: 197918\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.62 sec. Users per second: 1304\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1092463, PRECISION_RECALL_MIN_DEN: 0.1217730, RECALL: 0.0567832, MAP: 0.0507119, MAP_MIN_DEN: 0.0552946, MRR: 0.2597912, NDCG: 0.1029771, F1: 0.0747260, HIT_RATE: 0.5693891, ARHR_ALL_HITS: 0.3570599, NOVELTY: 0.0223572, AVERAGE_POPULARITY: 0.7199321, DIVERSITY_MEAN_INTER_LIST: 0.6241444, DIVERSITY_HERFINDAHL: 0.9624041, COVERAGE_ITEM: 0.0368272, COVERAGE_ITEM_CORRECT: 0.0234355, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5678808, DIVERSITY_GINI: 0.0068472, SHANNON_ENTROPY: 5.1022219, RATIO_DIVERSITY_HERFINDAHL: 0.9631903, RATIO_DIVERSITY_GINI: 0.0195833, RATIO_SHANNON_ENTROPY: 0.4718656, RATIO_AVERAGE_POPULARITY: 2.6064115, RATIO_NOVELTY: 0.0757546, \n",
      "\n",
      "FUNK_SVD: Epoch 190 of 500. Elapsed time 9.82 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.28 sec. MSE loss 1.24E+00. Sample per second: 195129\n",
      "FUNK_SVD: Epoch 191 of 500. Elapsed time 9.86 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.63 sec. MSE loss 1.23E+00. Sample per second: 243655\n",
      "FUNK_SVD: Epoch 192 of 500. Elapsed time 9.90 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.90 sec. MSE loss 1.23E+00. Sample per second: 220521\n",
      "FUNK_SVD: Epoch 193 of 500. Elapsed time 9.94 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.32 sec. MSE loss 1.23E+00. Sample per second: 192583\n",
      "FUNK_SVD: Epoch 194 of 500. Elapsed time 9.98 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.62 sec. MSE loss 1.22E+00. Sample per second: 243840\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.81 sec. Users per second: 1253\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1093127, PRECISION_RECALL_MIN_DEN: 0.1218609, RECALL: 0.0568557, MAP: 0.0507892, MAP_MIN_DEN: 0.0553806, MRR: 0.2602692, NDCG: 0.1031814, F1: 0.0748042, HIT_RATE: 0.5693891, ARHR_ALL_HITS: 0.3576461, NOVELTY: 0.0223563, AVERAGE_POPULARITY: 0.7200250, DIVERSITY_MEAN_INTER_LIST: 0.6220554, DIVERSITY_HERFINDAHL: 0.9621952, COVERAGE_ITEM: 0.0357971, COVERAGE_ITEM_CORRECT: 0.0229204, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5678808, DIVERSITY_GINI: 0.0068048, SHANNON_ENTROPY: 5.0926121, RATIO_DIVERSITY_HERFINDAHL: 0.9629813, RATIO_DIVERSITY_GINI: 0.0194620, RATIO_SHANNON_ENTROPY: 0.4709768, RATIO_AVERAGE_POPULARITY: 2.6067478, RATIO_NOVELTY: 0.0757518, \n",
      "\n",
      "FUNK_SVD: Epoch 195 of 500. Elapsed time 10.10 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.86 sec. MSE loss 1.22E+00. Sample per second: 223657\n",
      "FUNK_SVD: Epoch 196 of 500. Elapsed time 10.14 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.27 sec. MSE loss 1.22E+00. Sample per second: 195918\n",
      "FUNK_SVD: Epoch 197 of 500. Elapsed time 10.18 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.68 sec. MSE loss 1.22E+00. Sample per second: 239095\n",
      "FUNK_SVD: Epoch 198 of 500. Elapsed time 10.22 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.02 sec. MSE loss 1.22E+00. Sample per second: 211947\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Epoch 199 of 500. Elapsed time 10.26 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.41 sec. MSE loss 1.22E+00. Sample per second: 264961\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.83 sec. Users per second: 1248\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1094456, PRECISION_RECALL_MIN_DEN: 0.1219190, RECALL: 0.0568162, MAP: 0.0508366, MAP_MIN_DEN: 0.0554013, MRR: 0.2604169, NDCG: 0.1033511, F1: 0.0748011, HIT_RATE: 0.5695551, ARHR_ALL_HITS: 0.3578970, NOVELTY: 0.0223562, AVERAGE_POPULARITY: 0.7200156, DIVERSITY_MEAN_INTER_LIST: 0.6200192, DIVERSITY_HERFINDAHL: 0.9619916, COVERAGE_ITEM: 0.0357971, COVERAGE_ITEM_CORRECT: 0.0229204, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5680464, DIVERSITY_GINI: 0.0067667, SHANNON_ENTROPY: 5.0838441, RATIO_DIVERSITY_HERFINDAHL: 0.9627775, RATIO_DIVERSITY_GINI: 0.0193531, RATIO_SHANNON_ENTROPY: 0.4701660, RATIO_AVERAGE_POPULARITY: 2.6067138, RATIO_NOVELTY: 0.0757512, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 200 of 500. Elapsed time 10.38 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.59 sec. MSE loss 1.21E+00. Sample per second: 247038\n",
      "FUNK_SVD: Epoch 201 of 500. Elapsed time 10.42 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.85 sec. MSE loss 1.21E+00. Sample per second: 224246\n",
      "FUNK_SVD: Epoch 202 of 500. Elapsed time 10.45 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.25 sec. MSE loss 1.21E+00. Sample per second: 196847\n",
      "FUNK_SVD: Epoch 203 of 500. Elapsed time 10.49 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.63 sec. MSE loss 1.21E+00. Sample per second: 243356\n",
      "FUNK_SVD: Epoch 204 of 500. Elapsed time 10.53 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.95 sec. MSE loss 1.21E+00. Sample per second: 216836\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.00 sec. Users per second: 1204\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1093625, PRECISION_RECALL_MIN_DEN: 0.1218159, RECALL: 0.0567399, MAP: 0.0508143, MAP_MIN_DEN: 0.0553797, MRR: 0.2602377, NDCG: 0.1033108, F1: 0.0747155, HIT_RATE: 0.5693891, ARHR_ALL_HITS: 0.3576764, NOVELTY: 0.0223561, AVERAGE_POPULARITY: 0.7199955, DIVERSITY_MEAN_INTER_LIST: 0.6182407, DIVERSITY_HERFINDAHL: 0.9618138, COVERAGE_ITEM: 0.0355395, COVERAGE_ITEM_CORRECT: 0.0226629, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5678808, DIVERSITY_GINI: 0.0067318, SHANNON_ENTROPY: 5.0757406, RATIO_DIVERSITY_HERFINDAHL: 0.9625996, RATIO_DIVERSITY_GINI: 0.0192531, RATIO_SHANNON_ENTROPY: 0.4694165, RATIO_AVERAGE_POPULARITY: 2.6066410, RATIO_NOVELTY: 0.0757511, \n",
      "\n",
      "FUNK_SVD: Epoch 205 of 500. Elapsed time 10.66 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.33 sec. MSE loss 1.21E+00. Sample per second: 192120\n",
      "FUNK_SVD: Epoch 206 of 500. Elapsed time 10.70 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.28 sec. MSE loss 1.20E+00. Sample per second: 280471\n",
      "FUNK_SVD: Epoch 207 of 500. Elapsed time 10.73 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 1.46 sec. MSE loss 1.20E+00. Sample per second: 439490\n",
      "FUNK_SVD: Epoch 208 of 500. Elapsed time 10.75 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.67 sec. MSE loss 1.20E+00. Sample per second: 239926\n",
      "FUNK_SVD: Epoch 209 of 500. Elapsed time 10.79 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.93 sec. MSE loss 1.20E+00. Sample per second: 218511\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.70 sec. Users per second: 1283\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1094124, PRECISION_RECALL_MIN_DEN: 0.1219074, RECALL: 0.0568277, MAP: 0.0507724, MAP_MIN_DEN: 0.0553411, MRR: 0.2603063, NDCG: 0.1034472, F1: 0.0748033, HIT_RATE: 0.5703851, ARHR_ALL_HITS: 0.3575874, NOVELTY: 0.0223554, AVERAGE_POPULARITY: 0.7200731, DIVERSITY_MEAN_INTER_LIST: 0.6163500, DIVERSITY_HERFINDAHL: 0.9616248, COVERAGE_ITEM: 0.0347669, COVERAGE_ITEM_CORRECT: 0.0218903, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5688742, DIVERSITY_GINI: 0.0066901, SHANNON_ENTROPY: 5.0664410, RATIO_DIVERSITY_HERFINDAHL: 0.9624104, RATIO_DIVERSITY_GINI: 0.0191340, RATIO_SHANNON_ENTROPY: 0.4685565, RATIO_AVERAGE_POPULARITY: 2.6069218, RATIO_NOVELTY: 0.0757486, \n",
      "\n",
      "FUNK_SVD: Epoch 210 of 500. Elapsed time 10.90 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.00 sec. MSE loss 1.20E+00. Sample per second: 213096\n",
      "FUNK_SVD: Epoch 211 of 500. Elapsed time 10.94 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.43 sec. MSE loss 1.20E+00. Sample per second: 262894\n",
      "FUNK_SVD: Epoch 212 of 500. Elapsed time 10.98 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.90 sec. MSE loss 1.20E+00. Sample per second: 220681\n",
      "FUNK_SVD: Epoch 213 of 500. Elapsed time 11.02 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.23 sec. MSE loss 1.19E+00. Sample per second: 197819\n",
      "FUNK_SVD: Epoch 214 of 500. Elapsed time 11.06 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.71 sec. MSE loss 1.19E+00. Sample per second: 235694\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.82 sec. Users per second: 1251\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1093459, PRECISION_RECALL_MIN_DEN: 0.1218675, RECALL: 0.0568504, MAP: 0.0507683, MAP_MIN_DEN: 0.0553363, MRR: 0.2607652, NDCG: 0.1035588, F1: 0.0748074, HIT_RATE: 0.5708831, ARHR_ALL_HITS: 0.3579734, NOVELTY: 0.0223552, AVERAGE_POPULARITY: 0.7200862, DIVERSITY_MEAN_INTER_LIST: 0.6143983, DIVERSITY_HERFINDAHL: 0.9614296, COVERAGE_ITEM: 0.0347669, COVERAGE_ITEM_CORRECT: 0.0221478, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5693709, DIVERSITY_GINI: 0.0066565, SHANNON_ENTROPY: 5.0585023, RATIO_DIVERSITY_HERFINDAHL: 0.9622151, RATIO_DIVERSITY_GINI: 0.0190377, RATIO_SHANNON_ENTROPY: 0.4678223, RATIO_AVERAGE_POPULARITY: 2.6069694, RATIO_NOVELTY: 0.0757480, \n",
      "\n",
      "FUNK_SVD: Epoch 215 of 500. Elapsed time 11.18 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.96 sec. MSE loss 1.19E+00. Sample per second: 216214\n",
      "FUNK_SVD: Epoch 216 of 500. Elapsed time 11.22 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.29 sec. MSE loss 1.19E+00. Sample per second: 194784\n",
      "FUNK_SVD: Epoch 217 of 500. Elapsed time 11.26 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.62 sec. MSE loss 1.19E+00. Sample per second: 244743\n",
      "FUNK_SVD: Epoch 218 of 500. Elapsed time 11.30 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.95 sec. MSE loss 1.19E+00. Sample per second: 217060\n",
      "FUNK_SVD: Epoch 219 of 500. Elapsed time 11.34 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.17 sec. MSE loss 1.19E+00. Sample per second: 201707\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.66 sec. Users per second: 1293\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1094954, PRECISION_RECALL_MIN_DEN: 0.1220618, RECALL: 0.0570266, MAP: 0.0508329, MAP_MIN_DEN: 0.0554135, MRR: 0.2610564, NDCG: 0.1037875, F1: 0.0749949, HIT_RATE: 0.5707171, ARHR_ALL_HITS: 0.3584084, NOVELTY: 0.0223553, AVERAGE_POPULARITY: 0.7200399, DIVERSITY_MEAN_INTER_LIST: 0.6123229, DIVERSITY_HERFINDAHL: 0.9612221, COVERAGE_ITEM: 0.0339943, COVERAGE_ITEM_CORRECT: 0.0221478, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5692053, DIVERSITY_GINI: 0.0066194, SHANNON_ENTROPY: 5.0497087, RATIO_DIVERSITY_HERFINDAHL: 0.9620074, RATIO_DIVERSITY_GINI: 0.0189316, RATIO_SHANNON_ENTROPY: 0.4670090, RATIO_AVERAGE_POPULARITY: 2.6068015, RATIO_NOVELTY: 0.0757483, \n",
      "\n",
      "FUNK_SVD: Epoch 220 of 500. Elapsed time 11.45 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.31 sec. MSE loss 1.19E+00. Sample per second: 193048\n",
      "FUNK_SVD: Epoch 221 of 500. Elapsed time 11.50 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.73 sec. MSE loss 1.18E+00. Sample per second: 234146\n",
      "FUNK_SVD: Epoch 222 of 500. Elapsed time 11.54 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.09 sec. MSE loss 1.18E+00. Sample per second: 206764\n",
      "FUNK_SVD: Epoch 223 of 500. Elapsed time 11.58 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.54 sec. MSE loss 1.18E+00. Sample per second: 252016\n",
      "FUNK_SVD: Epoch 224 of 500. Elapsed time 11.62 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.95 sec. MSE loss 1.18E+00. Sample per second: 216813\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.71 sec. Users per second: 1278\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1097278, PRECISION_RECALL_MIN_DEN: 0.1221446, RECALL: 0.0569601, MAP: 0.0508628, MAP_MIN_DEN: 0.0554340, MRR: 0.2608450, NDCG: 0.1039514, F1: 0.0749917, HIT_RATE: 0.5708831, ARHR_ALL_HITS: 0.3583522, NOVELTY: 0.0223549, AVERAGE_POPULARITY: 0.7200755, DIVERSITY_MEAN_INTER_LIST: 0.6103231, DIVERSITY_HERFINDAHL: 0.9610222, COVERAGE_ITEM: 0.0334793, COVERAGE_ITEM_CORRECT: 0.0218903, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5693709, DIVERSITY_GINI: 0.0065816, SHANNON_ENTROPY: 5.0405423, RATIO_DIVERSITY_HERFINDAHL: 0.9618073, RATIO_DIVERSITY_GINI: 0.0188235, RATIO_SHANNON_ENTROPY: 0.4661613, RATIO_AVERAGE_POPULARITY: 2.6069305, RATIO_NOVELTY: 0.0757469, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 225 of 500. Elapsed time 11.74 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Processed 640136 (100.0%) in 3.04 sec. MSE loss 1.18E+00. Sample per second: 210252\n",
      "FUNK_SVD: Epoch 226 of 500. Elapsed time 11.77 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.41 sec. MSE loss 1.18E+00. Sample per second: 265401\n",
      "FUNK_SVD: Epoch 227 of 500. Elapsed time 11.81 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.82 sec. MSE loss 1.18E+00. Sample per second: 226688\n",
      "FUNK_SVD: Epoch 228 of 500. Elapsed time 11.85 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.12 sec. MSE loss 1.18E+00. Sample per second: 205161\n",
      "FUNK_SVD: Epoch 229 of 500. Elapsed time 11.89 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.45 sec. MSE loss 1.17E+00. Sample per second: 261573\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.65 sec. Users per second: 1296\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1093459, PRECISION_RECALL_MIN_DEN: 0.1217019, RECALL: 0.0567079, MAP: 0.0508657, MAP_MIN_DEN: 0.0554343, MRR: 0.2611518, NDCG: 0.1038655, F1: 0.0746839, HIT_RATE: 0.5690571, ARHR_ALL_HITS: 0.3586402, NOVELTY: 0.0223552, AVERAGE_POPULARITY: 0.7200007, DIVERSITY_MEAN_INTER_LIST: 0.6087416, DIVERSITY_HERFINDAHL: 0.9608641, COVERAGE_ITEM: 0.0329642, COVERAGE_ITEM_CORRECT: 0.0218903, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5675497, DIVERSITY_GINI: 0.0065544, SHANNON_ENTROPY: 5.0340900, RATIO_DIVERSITY_HERFINDAHL: 0.9616490, RATIO_DIVERSITY_GINI: 0.0187457, RATIO_SHANNON_ENTROPY: 0.4655646, RATIO_AVERAGE_POPULARITY: 2.6066597, RATIO_NOVELTY: 0.0757479, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 230 of 500. Elapsed time 12.01 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.45 sec. MSE loss 1.17E+00. Sample per second: 260648\n",
      "FUNK_SVD: Epoch 231 of 500. Elapsed time 12.05 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.78 sec. MSE loss 1.17E+00. Sample per second: 229957\n",
      "FUNK_SVD: Epoch 232 of 500. Elapsed time 12.09 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.16 sec. MSE loss 1.17E+00. Sample per second: 202576\n",
      "FUNK_SVD: Epoch 233 of 500. Elapsed time 12.13 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.62 sec. MSE loss 1.17E+00. Sample per second: 243920\n",
      "FUNK_SVD: Epoch 234 of 500. Elapsed time 12.17 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.10 sec. MSE loss 1.17E+00. Sample per second: 206804\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.56 sec. Users per second: 1321\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1093958, PRECISION_RECALL_MIN_DEN: 0.1216697, RECALL: 0.0565748, MAP: 0.0509167, MAP_MIN_DEN: 0.0554920, MRR: 0.2613087, NDCG: 0.1039390, F1: 0.0745800, HIT_RATE: 0.5692231, ARHR_ALL_HITS: 0.3588577, NOVELTY: 0.0223554, AVERAGE_POPULARITY: 0.7199414, DIVERSITY_MEAN_INTER_LIST: 0.6068328, DIVERSITY_HERFINDAHL: 0.9606732, COVERAGE_ITEM: 0.0327067, COVERAGE_ITEM_CORRECT: 0.0218903, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5677152, DIVERSITY_GINI: 0.0065185, SHANNON_ENTROPY: 5.0258754, RATIO_DIVERSITY_HERFINDAHL: 0.9614580, RATIO_DIVERSITY_GINI: 0.0186431, RATIO_SHANNON_ENTROPY: 0.4648049, RATIO_AVERAGE_POPULARITY: 2.6064453, RATIO_NOVELTY: 0.0757487, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 235 of 500. Elapsed time 12.29 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.04 sec. MSE loss 1.17E+00. Sample per second: 209745\n",
      "FUNK_SVD: Epoch 236 of 500. Elapsed time 12.32 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.42 sec. MSE loss 1.17E+00. Sample per second: 263651\n",
      "FUNK_SVD: Epoch 237 of 500. Elapsed time 12.36 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.88 sec. MSE loss 1.16E+00. Sample per second: 221640\n",
      "FUNK_SVD: Epoch 238 of 500. Elapsed time 12.41 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.20 sec. MSE loss 1.17E+00. Sample per second: 200086\n",
      "FUNK_SVD: Epoch 239 of 500. Elapsed time 12.44 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.45 sec. MSE loss 1.16E+00. Sample per second: 261576\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.58 sec. Users per second: 1315\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1093792, PRECISION_RECALL_MIN_DEN: 0.1216441, RECALL: 0.0565797, MAP: 0.0508646, MAP_MIN_DEN: 0.0554391, MRR: 0.2611547, NDCG: 0.1040079, F1: 0.0745804, HIT_RATE: 0.5695551, ARHR_ALL_HITS: 0.3585872, NOVELTY: 0.0223555, AVERAGE_POPULARITY: 0.7199039, DIVERSITY_MEAN_INTER_LIST: 0.6049720, DIVERSITY_HERFINDAHL: 0.9604872, COVERAGE_ITEM: 0.0321916, COVERAGE_ITEM_CORRECT: 0.0211177, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5680464, DIVERSITY_GINI: 0.0064830, SHANNON_ENTROPY: 5.0177632, RATIO_DIVERSITY_HERFINDAHL: 0.9612718, RATIO_DIVERSITY_GINI: 0.0185417, RATIO_SHANNON_ENTROPY: 0.4640546, RATIO_AVERAGE_POPULARITY: 2.6063092, RATIO_NOVELTY: 0.0757490, \n",
      "\n",
      "FUNK_SVD: Epoch 240 of 500. Elapsed time 12.56 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.35 sec. MSE loss 1.16E+00. Sample per second: 271705\n",
      "FUNK_SVD: Epoch 241 of 500. Elapsed time 12.60 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.71 sec. MSE loss 1.16E+00. Sample per second: 236393\n",
      "FUNK_SVD: Epoch 242 of 500. Elapsed time 12.64 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.21 sec. MSE loss 1.16E+00. Sample per second: 199298\n",
      "FUNK_SVD: Epoch 243 of 500. Elapsed time 12.68 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.85 sec. MSE loss 1.16E+00. Sample per second: 224250\n",
      "FUNK_SVD: Epoch 244 of 500. Elapsed time 12.72 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.46 sec. MSE loss 1.16E+00. Sample per second: 184853\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.35 sec. Users per second: 1384\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1095120, PRECISION_RECALL_MIN_DEN: 0.1217345, RECALL: 0.0565326, MAP: 0.0509166, MAP_MIN_DEN: 0.0554893, MRR: 0.2613399, NDCG: 0.1041118, F1: 0.0745703, HIT_RATE: 0.5693891, ARHR_ALL_HITS: 0.3588450, NOVELTY: 0.0223558, AVERAGE_POPULARITY: 0.7198112, DIVERSITY_MEAN_INTER_LIST: 0.6031729, DIVERSITY_HERFINDAHL: 0.9603073, COVERAGE_ITEM: 0.0314190, COVERAGE_ITEM_CORRECT: 0.0208602, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5678808, DIVERSITY_GINI: 0.0064491, SHANNON_ENTROPY: 5.0097597, RATIO_DIVERSITY_HERFINDAHL: 0.9610918, RATIO_DIVERSITY_GINI: 0.0184446, RATIO_SHANNON_ENTROPY: 0.4633145, RATIO_AVERAGE_POPULARITY: 2.6059736, RATIO_NOVELTY: 0.0757502, \n",
      "\n",
      "FUNK_SVD: Epoch 245 of 500. Elapsed time 12.84 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.02 sec. MSE loss 1.16E+00. Sample per second: 316414\n",
      "FUNK_SVD: Epoch 246 of 500. Elapsed time 12.86 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.32 sec. MSE loss 1.16E+00. Sample per second: 275725\n",
      "FUNK_SVD: Epoch 247 of 500. Elapsed time 12.90 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.62 sec. MSE loss 1.16E+00. Sample per second: 243959\n",
      "FUNK_SVD: Epoch 248 of 500. Elapsed time 12.93 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.04 sec. MSE loss 1.15E+00. Sample per second: 210602\n",
      "FUNK_SVD: Epoch 249 of 500. Elapsed time 12.97 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.44 sec. MSE loss 1.16E+00. Sample per second: 262311\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.54 sec. Users per second: 1327\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1096282, PRECISION_RECALL_MIN_DEN: 0.1219390, RECALL: 0.0566182, MAP: 0.0509676, MAP_MIN_DEN: 0.0555351, MRR: 0.2615158, NDCG: 0.1042832, F1: 0.0746717, HIT_RATE: 0.5692231, ARHR_ALL_HITS: 0.3590724, NOVELTY: 0.0223565, AVERAGE_POPULARITY: 0.7196841, DIVERSITY_MEAN_INTER_LIST: 0.6014478, DIVERSITY_HERFINDAHL: 0.9601348, COVERAGE_ITEM: 0.0314190, COVERAGE_ITEM_CORRECT: 0.0208602, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5677152, DIVERSITY_GINI: 0.0064197, SHANNON_ENTROPY: 5.0027835, RATIO_DIVERSITY_HERFINDAHL: 0.9609192, RATIO_DIVERSITY_GINI: 0.0183607, RATIO_SHANNON_ENTROPY: 0.4626693, RATIO_AVERAGE_POPULARITY: 2.6055137, RATIO_NOVELTY: 0.0757524, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 250 of 500. Elapsed time 13.09 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.35 sec. MSE loss 1.15E+00. Sample per second: 271707\n",
      "FUNK_SVD: Epoch 251 of 500. Elapsed time 13.13 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.81 sec. MSE loss 1.15E+00. Sample per second: 227322\n",
      "FUNK_SVD: Epoch 252 of 500. Elapsed time 13.17 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.32 sec. MSE loss 1.15E+00. Sample per second: 193058\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Epoch 253 of 500. Elapsed time 13.21 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.82 sec. MSE loss 1.15E+00. Sample per second: 226677\n",
      "FUNK_SVD: Epoch 254 of 500. Elapsed time 13.25 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.41 sec. MSE loss 1.15E+00. Sample per second: 187956\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.67 sec. Users per second: 1291\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1095286, PRECISION_RECALL_MIN_DEN: 0.1218028, RECALL: 0.0564752, MAP: 0.0510068, MAP_MIN_DEN: 0.0555633, MRR: 0.2618322, NDCG: 0.1043068, F1: 0.0745241, HIT_RATE: 0.5685591, ARHR_ALL_HITS: 0.3594044, NOVELTY: 0.0223568, AVERAGE_POPULARITY: 0.7196093, DIVERSITY_MEAN_INTER_LIST: 0.5999097, DIVERSITY_HERFINDAHL: 0.9599810, COVERAGE_ITEM: 0.0311615, COVERAGE_ITEM_CORRECT: 0.0206026, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5670530, DIVERSITY_GINI: 0.0063932, SHANNON_ENTROPY: 4.9964090, RATIO_DIVERSITY_HERFINDAHL: 0.9607653, RATIO_DIVERSITY_GINI: 0.0182848, RATIO_SHANNON_ENTROPY: 0.4620798, RATIO_AVERAGE_POPULARITY: 2.6052428, RATIO_NOVELTY: 0.0757534, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 255 of 500. Elapsed time 13.38 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.41 sec. MSE loss 1.15E+00. Sample per second: 265586\n",
      "FUNK_SVD: Epoch 256 of 500. Elapsed time 13.41 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.70 sec. MSE loss 1.15E+00. Sample per second: 237040\n",
      "FUNK_SVD: Epoch 257 of 500. Elapsed time 13.45 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.10 sec. MSE loss 1.15E+00. Sample per second: 206234\n",
      "FUNK_SVD: Epoch 258 of 500. Elapsed time 13.49 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.36 sec. MSE loss 1.15E+00. Sample per second: 270935\n",
      "FUNK_SVD: Epoch 259 of 500. Elapsed time 13.53 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.78 sec. MSE loss 1.15E+00. Sample per second: 229937\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.73 sec. Users per second: 1272\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1093792, PRECISION_RECALL_MIN_DEN: 0.1216099, RECALL: 0.0563530, MAP: 0.0510076, MAP_MIN_DEN: 0.0555461, MRR: 0.2615879, NDCG: 0.1042218, F1: 0.0743832, HIT_RATE: 0.5673971, ARHR_ALL_HITS: 0.3591719, NOVELTY: 0.0223568, AVERAGE_POPULARITY: 0.7195970, DIVERSITY_MEAN_INTER_LIST: 0.5982793, DIVERSITY_HERFINDAHL: 0.9598180, COVERAGE_ITEM: 0.0311615, COVERAGE_ITEM_CORRECT: 0.0206026, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5658940, DIVERSITY_GINI: 0.0063643, SHANNON_ENTROPY: 4.9895884, RATIO_DIVERSITY_HERFINDAHL: 0.9606021, RATIO_DIVERSITY_GINI: 0.0182022, RATIO_SHANNON_ENTROPY: 0.4614490, RATIO_AVERAGE_POPULARITY: 2.6051982, RATIO_NOVELTY: 0.0757534, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 260 of 500. Elapsed time 13.65 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.96 sec. MSE loss 1.15E+00. Sample per second: 215973\n",
      "FUNK_SVD: Epoch 261 of 500. Elapsed time 13.69 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.49 sec. MSE loss 1.15E+00. Sample per second: 183282\n",
      "FUNK_SVD: Epoch 262 of 500. Elapsed time 13.73 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.88 sec. MSE loss 1.15E+00. Sample per second: 222084\n",
      "FUNK_SVD: Epoch 263 of 500. Elapsed time 13.77 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.27 sec. MSE loss 1.14E+00. Sample per second: 195906\n",
      "FUNK_SVD: Epoch 264 of 500. Elapsed time 13.81 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.60 sec. MSE loss 1.15E+00. Sample per second: 245451\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.58 sec. Users per second: 1314\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1092795, PRECISION_RECALL_MIN_DEN: 0.1214008, RECALL: 0.0561082, MAP: 0.0509998, MAP_MIN_DEN: 0.0555090, MRR: 0.2613915, NDCG: 0.1041256, F1: 0.0741468, HIT_RATE: 0.5664011, ARHR_ALL_HITS: 0.3589877, NOVELTY: 0.0223573, AVERAGE_POPULARITY: 0.7194911, DIVERSITY_MEAN_INTER_LIST: 0.5969646, DIVERSITY_HERFINDAHL: 0.9596866, COVERAGE_ITEM: 0.0303889, COVERAGE_ITEM_CORRECT: 0.0203451, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5649007, DIVERSITY_GINI: 0.0063395, SHANNON_ENTROPY: 4.9836880, RATIO_DIVERSITY_HERFINDAHL: 0.9604706, RATIO_DIVERSITY_GINI: 0.0181311, RATIO_SHANNON_ENTROPY: 0.4609033, RATIO_AVERAGE_POPULARITY: 2.6048148, RATIO_NOVELTY: 0.0757552, \n",
      "\n",
      "FUNK_SVD: Epoch 265 of 500. Elapsed time 13.93 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.56 sec. MSE loss 1.14E+00. Sample per second: 249993\n",
      "FUNK_SVD: Epoch 266 of 500. Elapsed time 13.97 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.81 sec. MSE loss 1.14E+00. Sample per second: 227841\n",
      "FUNK_SVD: Epoch 267 of 500. Elapsed time 14.00 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.13 sec. MSE loss 1.14E+00. Sample per second: 204719\n",
      "FUNK_SVD: Epoch 268 of 500. Elapsed time 14.04 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.50 sec. MSE loss 1.14E+00. Sample per second: 254834\n",
      "FUNK_SVD: Epoch 269 of 500. Elapsed time 14.08 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.89 sec. MSE loss 1.14E+00. Sample per second: 221045\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.65 sec. Users per second: 1294\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1092961, PRECISION_RECALL_MIN_DEN: 0.1214162, RECALL: 0.0561377, MAP: 0.0509943, MAP_MIN_DEN: 0.0555174, MRR: 0.2614070, NDCG: 0.1042419, F1: 0.0741763, HIT_RATE: 0.5672311, ARHR_ALL_HITS: 0.3590244, NOVELTY: 0.0223582, AVERAGE_POPULARITY: 0.7193378, DIVERSITY_MEAN_INTER_LIST: 0.5953582, DIVERSITY_HERFINDAHL: 0.9595259, COVERAGE_ITEM: 0.0301313, COVERAGE_ITEM_CORRECT: 0.0206026, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5657285, DIVERSITY_GINI: 0.0063150, SHANNON_ENTROPY: 4.9779701, RATIO_DIVERSITY_HERFINDAHL: 0.9603098, RATIO_DIVERSITY_GINI: 0.0180612, RATIO_SHANNON_ENTROPY: 0.4603745, RATIO_AVERAGE_POPULARITY: 2.6042598, RATIO_NOVELTY: 0.0757581, \n",
      "\n",
      "FUNK_SVD: Epoch 270 of 500. Elapsed time 14.20 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.02 sec. MSE loss 1.14E+00. Sample per second: 211963\n",
      "FUNK_SVD: Epoch 271 of 500. Elapsed time 14.24 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.57 sec. MSE loss 1.14E+00. Sample per second: 248882\n",
      "FUNK_SVD: Epoch 272 of 500. Elapsed time 14.28 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.94 sec. MSE loss 1.14E+00. Sample per second: 217736\n",
      "FUNK_SVD: Epoch 273 of 500. Elapsed time 14.32 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.27 sec. MSE loss 1.14E+00. Sample per second: 195829\n",
      "FUNK_SVD: Epoch 274 of 500. Elapsed time 14.36 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.58 sec. MSE loss 1.14E+00. Sample per second: 248268\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.68 sec. Users per second: 1288\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1092961, PRECISION_RECALL_MIN_DEN: 0.1214063, RECALL: 0.0561274, MAP: 0.0509781, MAP_MIN_DEN: 0.0554957, MRR: 0.2611875, NDCG: 0.1043018, F1: 0.0741673, HIT_RATE: 0.5670651, ARHR_ALL_HITS: 0.3588844, NOVELTY: 0.0223588, AVERAGE_POPULARITY: 0.7192186, DIVERSITY_MEAN_INTER_LIST: 0.5937404, DIVERSITY_HERFINDAHL: 0.9593642, COVERAGE_ITEM: 0.0298738, COVERAGE_ITEM_CORRECT: 0.0206026, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5655629, DIVERSITY_GINI: 0.0062915, SHANNON_ENTROPY: 4.9723564, RATIO_DIVERSITY_HERFINDAHL: 0.9601479, RATIO_DIVERSITY_GINI: 0.0179940, RATIO_SHANNON_ENTROPY: 0.4598553, RATIO_AVERAGE_POPULARITY: 2.6038283, RATIO_NOVELTY: 0.0757603, \n",
      "\n",
      "FUNK_SVD: Epoch 275 of 500. Elapsed time 14.48 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.57 sec. MSE loss 1.14E+00. Sample per second: 249042\n",
      "FUNK_SVD: Epoch 276 of 500. Elapsed time 14.52 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.99 sec. MSE loss 1.14E+00. Sample per second: 214295\n",
      "FUNK_SVD: Epoch 277 of 500. Elapsed time 14.56 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.30 sec. MSE loss 1.14E+00. Sample per second: 194126\n",
      "FUNK_SVD: Epoch 278 of 500. Elapsed time 14.60 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.66 sec. MSE loss 1.13E+00. Sample per second: 240655\n",
      "FUNK_SVD: Epoch 279 of 500. Elapsed time 14.63 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.97 sec. MSE loss 1.14E+00. Sample per second: 215391\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.55 sec. Users per second: 1325\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1091467, PRECISION_RECALL_MIN_DEN: 0.1212640, RECALL: 0.0561020, MAP: 0.0509159, MAP_MIN_DEN: 0.0554406, MRR: 0.2611153, NDCG: 0.1042303, F1: 0.0741107, HIT_RATE: 0.5667331, ARHR_ALL_HITS: 0.3586457, NOVELTY: 0.0223598, AVERAGE_POPULARITY: 0.7190597, DIVERSITY_MEAN_INTER_LIST: 0.5922331, DIVERSITY_HERFINDAHL: 0.9592135, COVERAGE_ITEM: 0.0298738, COVERAGE_ITEM_CORRECT: 0.0206026, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5652318, DIVERSITY_GINI: 0.0062680, SHANNON_ENTROPY: 4.9667569, RATIO_DIVERSITY_HERFINDAHL: 0.9599971, RATIO_DIVERSITY_GINI: 0.0179267, RATIO_SHANNON_ENTROPY: 0.4593375, RATIO_AVERAGE_POPULARITY: 2.6032530, RATIO_NOVELTY: 0.0757636, \n",
      "\n",
      "FUNK_SVD: Epoch 280 of 500. Elapsed time 14.75 min\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Processed 640136 (100.0%) in 2.84 sec. MSE loss 1.14E+00. Sample per second: 225303\n",
      "FUNK_SVD: Epoch 281 of 500. Elapsed time 14.79 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.19 sec. MSE loss 1.14E+00. Sample per second: 200748\n",
      "FUNK_SVD: Epoch 282 of 500. Elapsed time 14.83 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.77 sec. MSE loss 1.14E+00. Sample per second: 231409\n",
      "FUNK_SVD: Epoch 283 of 500. Elapsed time 14.87 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 3.15 sec. MSE loss 1.14E+00. Sample per second: 203249\n",
      "FUNK_SVD: Epoch 284 of 500. Elapsed time 14.91 min\n",
      "FUNK_SVD: Processed 640136 (100.0%) in 2.04 sec. MSE loss 1.13E+00. Sample per second: 314348\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.55 sec. Users per second: 1695\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1091799, PRECISION_RECALL_MIN_DEN: 0.1213124, RECALL: 0.0561255, MAP: 0.0509183, MAP_MIN_DEN: 0.0554402, MRR: 0.2609325, NDCG: 0.1042828, F1: 0.0741389, HIT_RATE: 0.5672311, ARHR_ALL_HITS: 0.3585203, NOVELTY: 0.0223608, AVERAGE_POPULARITY: 0.7188749, DIVERSITY_MEAN_INTER_LIST: 0.5909658, DIVERSITY_HERFINDAHL: 0.9590868, COVERAGE_ITEM: 0.0298738, COVERAGE_ITEM_CORRECT: 0.0203451, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5657285, DIVERSITY_GINI: 0.0062490, SHANNON_ENTROPY: 4.9622442, RATIO_DIVERSITY_HERFINDAHL: 0.9598703, RATIO_DIVERSITY_GINI: 0.0178723, RATIO_SHANNON_ENTROPY: 0.4589201, RATIO_AVERAGE_POPULARITY: 2.6025841, RATIO_NOVELTY: 0.0757670, \n",
      "\n",
      "FUNK_SVD: Convergence reached! Terminating at epoch 285. Best value for 'MAP' at epoch 260 is 0.0510. Elapsed time 15.00 min\n",
      "FUNK_SVD: Epoch 285 of 500. Elapsed time 15.00 min\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.67 sec. Users per second: 1289\n",
      "SearchBayesianSkopt: New best config found. Config 6: {'epochs': 260, 'num_factors': 20, 'sgd_mode': 'adagrad', 'batch_size': 4, 'item_reg': 1.4509981505329461e-05, 'user_reg': 0.0021138581529642794, 'learning_rate': 0.00045300762673031024} - results: PRECISION: 0.1093792, PRECISION_RECALL_MIN_DEN: 0.1216099, RECALL: 0.0563530, MAP: 0.0510076, MAP_MIN_DEN: 0.0555461, MRR: 0.2615879, NDCG: 0.1042218, F1: 0.0743832, HIT_RATE: 0.5673971, ARHR_ALL_HITS: 0.3591719, NOVELTY: 0.0223568, AVERAGE_POPULARITY: 0.7195970, DIVERSITY_MEAN_INTER_LIST: 0.5982793, DIVERSITY_HERFINDAHL: 0.9598180, COVERAGE_ITEM: 0.0311615, COVERAGE_ITEM_CORRECT: 0.0206026, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5658940, DIVERSITY_GINI: 0.0063643, SHANNON_ENTROPY: 4.9895884, RATIO_DIVERSITY_HERFINDAHL: 0.9606021, RATIO_DIVERSITY_GINI: 0.0182022, RATIO_SHANNON_ENTROPY: 0.4614490, RATIO_AVERAGE_POPULARITY: 2.6051982, RATIO_NOVELTY: 0.0757534, \n",
      "\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 4.75 sec. Users per second: 1270\n",
      "SearchBayesianSkopt: Config evaluated with evaluator_test. Config: {'epochs': 260, 'num_factors': 20, 'sgd_mode': 'adagrad', 'batch_size': 4, 'item_reg': 1.4509981505329461e-05, 'user_reg': 0.0021138581529642794, 'learning_rate': 0.00045300762673031024} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.1355368, PRECISION_RECALL_MIN_DEN: 0.1444367, RECALL: 0.0568278, MAP: 0.0663491, MAP_MIN_DEN: 0.0694865, MRR: 0.2969919, NDCG: 0.1222231, F1: 0.0800797, HIT_RATE: 0.6335321, ARHR_ALL_HITS: 0.4334459, NOVELTY: 0.0223566, AVERAGE_POPULARITY: 0.7196236, DIVERSITY_MEAN_INTER_LIST: 0.5982703, DIVERSITY_HERFINDAHL: 0.9598171, COVERAGE_ITEM: 0.0311615, COVERAGE_ITEM_CORRECT: 0.0231780, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.6331126, DIVERSITY_GINI: 0.0063638, SHANNON_ENTROPY: 4.9895218, RATIO_DIVERSITY_HERFINDAHL: 0.9606013, RATIO_DIVERSITY_GINI: 0.0182008, RATIO_SHANNON_ENTROPY: 0.4614428, RATIO_AVERAGE_POPULARITY: 2.6052945, RATIO_NOVELTY: 0.0757528, \n",
      "\n",
      "\n",
      "Iteration No: 7 ended. Search finished for the next optimal point.\n",
      "Time taken: 910.4964\n",
      "Function value obtained: -0.0510\n",
      "Current minimum: -0.0510\n",
      "Iteration No: 8 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'epochs': 500, 'num_factors': 1, 'sgd_mode': 'adagrad', 'batch_size': 64, 'item_reg': 0.01, 'user_reg': 0.01, 'learning_rate': 0.0011332925114552466}\n",
      "MatrixFactorization_FunkSVD_Cython_Recommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 0.62 sec. MSE loss 1.35E+01. Sample per second: 1035274\n",
      "FUNK_SVD: Epoch 1 of 500. Elapsed time 0.56 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 0.99 sec. MSE loss 1.24E+01. Sample per second: 649059\n",
      "FUNK_SVD: Epoch 2 of 500. Elapsed time 0.92 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 1.40 sec. MSE loss 1.17E+01. Sample per second: 458581\n",
      "FUNK_SVD: Epoch 3 of 500. Elapsed time 1.33 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 0.58 sec. MSE loss 1.12E+01. Sample per second: 1103675\n",
      "FUNK_SVD: Epoch 4 of 500. Elapsed time 1.52 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 0.95 sec. MSE loss 1.08E+01. Sample per second: 674600\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.76 sec. Users per second: 1267\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1184927, PRECISION_RECALL_MIN_DEN: 0.1317462, RECALL: 0.0614929, MAP: 0.0566593, MAP_MIN_DEN: 0.0621423, MRR: 0.2823353, NDCG: 0.1116149, F1: 0.0809671, HIT_RATE: 0.5956175, ARHR_ALL_HITS: 0.3927180, NOVELTY: 0.0221177, AVERAGE_POPULARITY: 0.7607312, DIVERSITY_MEAN_INTER_LIST: 0.5033341, DIVERSITY_HERFINDAHL: 0.9503251, COVERAGE_ITEM: 0.0144218, COVERAGE_ITEM_CORRECT: 0.0121040, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5940397, DIVERSITY_GINI: 0.0049249, SHANNON_ENTROPY: 4.5350801, RATIO_DIVERSITY_HERFINDAHL: 0.9511014, RATIO_DIVERSITY_GINI: 0.0140854, RATIO_SHANNON_ENTROPY: 0.4194150, RATIO_AVERAGE_POPULARITY: 2.7541189, RATIO_NOVELTY: 0.0749433, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 5 of 500. Elapsed time 6.65 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 1.15 sec. MSE loss 1.04E+01. Sample per second: 558541\n",
      "FUNK_SVD: Epoch 6 of 500. Elapsed time 7.08 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 0.66 sec. MSE loss 1.00E+01. Sample per second: 970067\n",
      "FUNK_SVD: Epoch 7 of 500. Elapsed time 7.60 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 1.00 sec. MSE loss 9.73E+00. Sample per second: 635139\n",
      "FUNK_SVD: Epoch 8 of 500. Elapsed time 7.95 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 0.34 sec. MSE loss 9.44E+00. Sample per second: 1891968\n",
      "FUNK_SVD: Epoch 9 of 500. Elapsed time 8.28 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 0.59 sec. MSE loss 9.19E+00. Sample per second: 1076276\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.70 sec. Users per second: 1280\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1194389, PRECISION_RECALL_MIN_DEN: 0.1331258, RECALL: 0.0624780, MAP: 0.0570224, MAP_MIN_DEN: 0.0625758, MRR: 0.2831374, NDCG: 0.1128328, F1: 0.0820408, HIT_RATE: 0.5997676, ARHR_ALL_HITS: 0.3943467, NOVELTY: 0.0221026, AVERAGE_POPULARITY: 0.7633340, DIVERSITY_MEAN_INTER_LIST: 0.4782303, DIVERSITY_HERFINDAHL: 0.9478151, COVERAGE_ITEM: 0.0128766, COVERAGE_ITEM_CORRECT: 0.0113314, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5981788, DIVERSITY_GINI: 0.0046908, SHANNON_ENTROPY: 4.4557657, RATIO_DIVERSITY_HERFINDAHL: 0.9485894, RATIO_DIVERSITY_GINI: 0.0134157, RATIO_SHANNON_ENTROPY: 0.4120798, RATIO_AVERAGE_POPULARITY: 2.7635417, RATIO_NOVELTY: 0.0748919, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 10 of 500. Elapsed time 13.25 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 0.62 sec. MSE loss 8.94E+00. Sample per second: 1022523\n",
      "FUNK_SVD: Epoch 11 of 500. Elapsed time 13.56 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 1.07 sec. MSE loss 8.72E+00. Sample per second: 598729\n",
      "FUNK_SVD: Epoch 12 of 500. Elapsed time 14.01 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 0.51 sec. MSE loss 8.50E+00. Sample per second: 1263067\n",
      "FUNK_SVD: Epoch 13 of 500. Elapsed time 14.45 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 0.84 sec. MSE loss 8.30E+00. Sample per second: 765959\n",
      "FUNK_SVD: Epoch 14 of 500. Elapsed time 14.77 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 1.13 sec. MSE loss 8.10E+00. Sample per second: 567856\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.56 sec. Users per second: 1320\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1195385, PRECISION_RECALL_MIN_DEN: 0.1331685, RECALL: 0.0624577, MAP: 0.0571800, MAP_MIN_DEN: 0.0627791, MRR: 0.2831003, NDCG: 0.1131422, F1: 0.0820467, HIT_RATE: 0.5974436, ARHR_ALL_HITS: 0.3948766, NOVELTY: 0.0220971, AVERAGE_POPULARITY: 0.7643235, DIVERSITY_MEAN_INTER_LIST: 0.4660260, DIVERSITY_HERFINDAHL: 0.9465949, COVERAGE_ITEM: 0.0121040, COVERAGE_ITEM_CORRECT: 0.0108164, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5958609, DIVERSITY_GINI: 0.0045929, SHANNON_ENTROPY: 4.4188832, RATIO_DIVERSITY_HERFINDAHL: 0.9473682, RATIO_DIVERSITY_GINI: 0.0131358, RATIO_SHANNON_ENTROPY: 0.4086688, RATIO_AVERAGE_POPULARITY: 2.7671242, RATIO_NOVELTY: 0.0748734, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 15 of 500. Elapsed time 19.63 sec\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Processed 640192 (100.0%) in 0.95 sec. MSE loss 7.94E+00. Sample per second: 674510\n",
      "FUNK_SVD: Epoch 16 of 500. Elapsed time 19.89 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 1.26 sec. MSE loss 7.78E+00. Sample per second: 508086\n",
      "FUNK_SVD: Epoch 17 of 500. Elapsed time 20.20 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 0.73 sec. MSE loss 7.61E+00. Sample per second: 876447\n",
      "FUNK_SVD: Epoch 18 of 500. Elapsed time 20.67 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 1.23 sec. MSE loss 7.45E+00. Sample per second: 521074\n",
      "FUNK_SVD: Epoch 19 of 500. Elapsed time 21.17 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 0.57 sec. MSE loss 7.30E+00. Sample per second: 1123081\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.56 sec. Users per second: 1320\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1189575, PRECISION_RECALL_MIN_DEN: 0.1325547, RECALL: 0.0621542, MAP: 0.0569873, MAP_MIN_DEN: 0.0625789, MRR: 0.2833098, NDCG: 0.1131901, F1: 0.0816480, HIT_RATE: 0.5966135, ARHR_ALL_HITS: 0.3944796, NOVELTY: 0.0220954, AVERAGE_POPULARITY: 0.7646300, DIVERSITY_MEAN_INTER_LIST: 0.4586440, DIVERSITY_HERFINDAHL: 0.9458568, COVERAGE_ITEM: 0.0118465, COVERAGE_ITEM_CORRECT: 0.0105588, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5950331, DIVERSITY_GINI: 0.0045365, SHANNON_ENTROPY: 4.3958656, RATIO_DIVERSITY_HERFINDAHL: 0.9466295, RATIO_DIVERSITY_GINI: 0.0129745, RATIO_SHANNON_ENTROPY: 0.4065401, RATIO_AVERAGE_POPULARITY: 2.7682340, RATIO_NOVELTY: 0.0748676, \n",
      "\n",
      "FUNK_SVD: Epoch 20 of 500. Elapsed time 26.08 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 0.46 sec. MSE loss 7.17E+00. Sample per second: 1384471\n",
      "FUNK_SVD: Epoch 21 of 500. Elapsed time 26.40 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 0.71 sec. MSE loss 7.03E+00. Sample per second: 905069\n",
      "FUNK_SVD: Epoch 22 of 500. Elapsed time 26.65 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 1.06 sec. MSE loss 6.90E+00. Sample per second: 602176\n",
      "FUNK_SVD: Epoch 23 of 500. Elapsed time 27.00 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 0.50 sec. MSE loss 6.78E+00. Sample per second: 1288712\n",
      "FUNK_SVD: Epoch 24 of 500. Elapsed time 27.43 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 1.00 sec. MSE loss 6.65E+00. Sample per second: 637668\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.60 sec. Users per second: 1309\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1192231, PRECISION_RECALL_MIN_DEN: 0.1329839, RECALL: 0.0625774, MAP: 0.0570263, MAP_MIN_DEN: 0.0626347, MRR: 0.2829895, NDCG: 0.1132961, F1: 0.0820754, HIT_RATE: 0.5977756, ARHR_ALL_HITS: 0.3944871, NOVELTY: 0.0220936, AVERAGE_POPULARITY: 0.7649674, DIVERSITY_MEAN_INTER_LIST: 0.4539945, DIVERSITY_HERFINDAHL: 0.9453919, COVERAGE_ITEM: 0.0118465, COVERAGE_ITEM_CORRECT: 0.0105588, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5961921, DIVERSITY_GINI: 0.0045022, SHANNON_ENTROPY: 4.3832346, RATIO_DIVERSITY_HERFINDAHL: 0.9461643, RATIO_DIVERSITY_GINI: 0.0128765, RATIO_SHANNON_ENTROPY: 0.4053719, RATIO_AVERAGE_POPULARITY: 2.7694553, RATIO_NOVELTY: 0.0748616, \n",
      "\n",
      "FUNK_SVD: Epoch 25 of 500. Elapsed time 32.56 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 0.97 sec. MSE loss 6.53E+00. Sample per second: 660782\n",
      "FUNK_SVD: Epoch 26 of 500. Elapsed time 32.91 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 1.27 sec. MSE loss 6.42E+00. Sample per second: 503367\n",
      "FUNK_SVD: Epoch 27 of 500. Elapsed time 33.21 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 0.52 sec. MSE loss 6.32E+00. Sample per second: 1230436\n",
      "FUNK_SVD: Epoch 28 of 500. Elapsed time 33.46 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 0.85 sec. MSE loss 6.21E+00. Sample per second: 748307\n",
      "FUNK_SVD: Epoch 29 of 500. Elapsed time 33.79 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 1.25 sec. MSE loss 6.10E+00. Sample per second: 511941\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.48 sec. Users per second: 1344\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1191069, PRECISION_RECALL_MIN_DEN: 0.1327689, RECALL: 0.0624662, MAP: 0.0570449, MAP_MIN_DEN: 0.0626386, MRR: 0.2829893, NDCG: 0.1134091, F1: 0.0819522, HIT_RATE: 0.5962815, ARHR_ALL_HITS: 0.3945856, NOVELTY: 0.0220927, AVERAGE_POPULARITY: 0.7651364, DIVERSITY_MEAN_INTER_LIST: 0.4504690, DIVERSITY_HERFINDAHL: 0.9450394, COVERAGE_ITEM: 0.0118465, COVERAGE_ITEM_CORRECT: 0.0105588, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5947020, DIVERSITY_GINI: 0.0044777, SHANNON_ENTROPY: 4.3727219, RATIO_DIVERSITY_HERFINDAHL: 0.9458115, RATIO_DIVERSITY_GINI: 0.0128063, RATIO_SHANNON_ENTROPY: 0.4043997, RATIO_AVERAGE_POPULARITY: 2.7700673, RATIO_NOVELTY: 0.0748586, \n",
      "\n",
      "FUNK_SVD: Epoch 30 of 500. Elapsed time 38.69 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 1.29 sec. MSE loss 6.01E+00. Sample per second: 495382\n",
      "FUNK_SVD: Epoch 31 of 500. Elapsed time 39.23 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 0.66 sec. MSE loss 5.91E+00. Sample per second: 973187\n",
      "FUNK_SVD: Epoch 32 of 500. Elapsed time 39.60 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 1.01 sec. MSE loss 5.82E+00. Sample per second: 634875\n",
      "FUNK_SVD: Epoch 33 of 500. Elapsed time 39.95 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 0.19 sec. MSE loss 5.71E+00. Sample per second: 3358772\n",
      "FUNK_SVD: Epoch 34 of 500. Elapsed time 40.13 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 0.60 sec. MSE loss 5.64E+00. Sample per second: 1069513\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.58 sec. Users per second: 1316\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1191567, PRECISION_RECALL_MIN_DEN: 0.1330189, RECALL: 0.0627690, MAP: 0.0570450, MAP_MIN_DEN: 0.0626476, MRR: 0.2829288, NDCG: 0.1135493, F1: 0.0822242, HIT_RATE: 0.5959495, ARHR_ALL_HITS: 0.3945134, NOVELTY: 0.0220923, AVERAGE_POPULARITY: 0.7652122, DIVERSITY_MEAN_INTER_LIST: 0.4476134, DIVERSITY_HERFINDAHL: 0.9447539, COVERAGE_ITEM: 0.0118465, COVERAGE_ITEM_CORRECT: 0.0105588, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5943709, DIVERSITY_GINI: 0.0044582, SHANNON_ENTROPY: 4.3644133, RATIO_DIVERSITY_HERFINDAHL: 0.9455257, RATIO_DIVERSITY_GINI: 0.0127505, RATIO_SHANNON_ENTROPY: 0.4036313, RATIO_AVERAGE_POPULARITY: 2.7703416, RATIO_NOVELTY: 0.0748573, \n",
      "\n",
      "FUNK_SVD: Epoch 35 of 500. Elapsed time 45.13 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 0.65 sec. MSE loss 5.55E+00. Sample per second: 986299\n",
      "FUNK_SVD: Epoch 36 of 500. Elapsed time 45.59 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 1.19 sec. MSE loss 5.47E+00. Sample per second: 539153\n",
      "FUNK_SVD: Epoch 37 of 500. Elapsed time 46.13 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 0.55 sec. MSE loss 5.40E+00. Sample per second: 1168873\n",
      "FUNK_SVD: Epoch 38 of 500. Elapsed time 46.49 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 0.88 sec. MSE loss 5.32E+00. Sample per second: 731010\n",
      "FUNK_SVD: Epoch 39 of 500. Elapsed time 46.81 sec\n",
      "FUNK_SVD: Processed 640192 (100.0%) in 1.18 sec. MSE loss 5.23E+00. Sample per second: 544711\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.67 sec. Users per second: 1291\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1190737, PRECISION_RECALL_MIN_DEN: 0.1329040, RECALL: 0.0627470, MAP: 0.0570944, MAP_MIN_DEN: 0.0626908, MRR: 0.2833472, NDCG: 0.1136457, F1: 0.0821856, HIT_RATE: 0.5966135, ARHR_ALL_HITS: 0.3949782, NOVELTY: 0.0220925, AVERAGE_POPULARITY: 0.7651930, DIVERSITY_MEAN_INTER_LIST: 0.4463860, DIVERSITY_HERFINDAHL: 0.9446312, COVERAGE_ITEM: 0.0118465, COVERAGE_ITEM_CORRECT: 0.0105588, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5950331, DIVERSITY_GINI: 0.0044487, SHANNON_ENTROPY: 4.3602670, RATIO_DIVERSITY_HERFINDAHL: 0.9454029, RATIO_DIVERSITY_GINI: 0.0127234, RATIO_SHANNON_ENTROPY: 0.4032478, RATIO_AVERAGE_POPULARITY: 2.7702721, RATIO_NOVELTY: 0.0748577, \n",
      "\n",
      "FUNK_SVD: Convergence reached! Terminating at epoch 40. Best value for 'MAP' at epoch 15 is 0.0572. Elapsed time 51.79 sec\n",
      "FUNK_SVD: Epoch 40 of 500. Elapsed time 51.79 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.56 sec. Users per second: 1321\n",
      "SearchBayesianSkopt: New best config found. Config 7: {'epochs': 15, 'num_factors': 1, 'sgd_mode': 'adagrad', 'batch_size': 64, 'item_reg': 0.01, 'user_reg': 0.01, 'learning_rate': 0.0011332925114552466} - results: PRECISION: 0.1195385, PRECISION_RECALL_MIN_DEN: 0.1331685, RECALL: 0.0624577, MAP: 0.0571800, MAP_MIN_DEN: 0.0627791, MRR: 0.2831003, NDCG: 0.1131422, F1: 0.0820467, HIT_RATE: 0.5974436, ARHR_ALL_HITS: 0.3948766, NOVELTY: 0.0220971, AVERAGE_POPULARITY: 0.7643235, DIVERSITY_MEAN_INTER_LIST: 0.4660260, DIVERSITY_HERFINDAHL: 0.9465949, COVERAGE_ITEM: 0.0121040, COVERAGE_ITEM_CORRECT: 0.0108164, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5958609, DIVERSITY_GINI: 0.0045929, SHANNON_ENTROPY: 4.4188832, RATIO_DIVERSITY_HERFINDAHL: 0.9473682, RATIO_DIVERSITY_GINI: 0.0131358, RATIO_SHANNON_ENTROPY: 0.4086688, RATIO_AVERAGE_POPULARITY: 2.7671242, RATIO_NOVELTY: 0.0748734, \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EvaluatorHoldout: Processed 6036 (100.0%) in 4.82 sec. Users per second: 1253\n",
      "SearchBayesianSkopt: Config evaluated with evaluator_test. Config: {'epochs': 15, 'num_factors': 1, 'sgd_mode': 'adagrad', 'batch_size': 64, 'item_reg': 0.01, 'user_reg': 0.01, 'learning_rate': 0.0011332925114552466} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.1470345, PRECISION_RECALL_MIN_DEN: 0.1571106, RECALL: 0.0618970, MAP: 0.0736408, MAP_MIN_DEN: 0.0774005, MRR: 0.3169586, NDCG: 0.1292467, F1: 0.0871194, HIT_RATE: 0.6504307, ARHR_ALL_HITS: 0.4713302, NOVELTY: 0.0220970, AVERAGE_POPULARITY: 0.7643492, DIVERSITY_MEAN_INTER_LIST: 0.4658630, DIVERSITY_HERFINDAHL: 0.9465786, COVERAGE_ITEM: 0.0121040, COVERAGE_ITEM_CORRECT: 0.0108164, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.6500000, DIVERSITY_GINI: 0.0045915, SHANNON_ENTROPY: 4.4184519, RATIO_DIVERSITY_HERFINDAHL: 0.9473519, RATIO_DIVERSITY_GINI: 0.0131319, RATIO_SHANNON_ENTROPY: 0.4086289, RATIO_AVERAGE_POPULARITY: 2.7672173, RATIO_NOVELTY: 0.0748730, \n",
      "\n",
      "\n",
      "Iteration No: 8 ended. Search finished for the next optimal point.\n",
      "Time taken: 62.4171\n",
      "Function value obtained: -0.0572\n",
      "Current minimum: -0.0572\n",
      "Iteration No: 9 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'epochs': 500, 'num_factors': 1, 'sgd_mode': 'adagrad', 'batch_size': 1, 'item_reg': 1e-05, 'user_reg': 1e-05, 'learning_rate': 0.001334504569942782}\n",
      "MatrixFactorization_FunkSVD_Cython_Recommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.93 sec. MSE loss 7.26E+00. Sample per second: 680999\n",
      "FUNK_SVD: Epoch 1 of 500. Elapsed time 0.49 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 1.19 sec. MSE loss 3.61E+00. Sample per second: 540102\n",
      "FUNK_SVD: Epoch 2 of 500. Elapsed time 0.71 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.57 sec. MSE loss 2.41E+00. Sample per second: 1103619\n",
      "FUNK_SVD: Epoch 3 of 500. Elapsed time 1.11 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 1.11 sec. MSE loss 1.85E+00. Sample per second: 575660\n",
      "FUNK_SVD: Epoch 4 of 500. Elapsed time 1.64 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.70 sec. MSE loss 1.56E+00. Sample per second: 909289\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.65 sec. Users per second: 1296\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1165007, PRECISION_RECALL_MIN_DEN: 0.1291117, RECALL: 0.0600661, MAP: 0.0550182, MAP_MIN_DEN: 0.0604231, MRR: 0.2758832, NDCG: 0.1118487, F1: 0.0792645, HIT_RATE: 0.5859894, ARHR_ALL_HITS: 0.3822948, NOVELTY: 0.0221626, AVERAGE_POPULARITY: 0.7520878, DIVERSITY_MEAN_INTER_LIST: 0.4508033, DIVERSITY_HERFINDAHL: 0.9450728, COVERAGE_ITEM: 0.0128766, COVERAGE_ITEM_CORRECT: 0.0115890, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5844371, DIVERSITY_GINI: 0.0045158, SHANNON_ENTROPY: 4.4039840, RATIO_DIVERSITY_HERFINDAHL: 0.9458449, RATIO_DIVERSITY_GINI: 0.0129154, RATIO_SHANNON_ENTROPY: 0.4072909, RATIO_AVERAGE_POPULARITY: 2.7228265, RATIO_NOVELTY: 0.0750955, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 5 of 500. Elapsed time 6.89 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.82 sec. MSE loss 1.40E+00. Sample per second: 776436\n",
      "FUNK_SVD: Epoch 6 of 500. Elapsed time 7.35 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 1.10 sec. MSE loss 1.31E+00. Sample per second: 579677\n",
      "FUNK_SVD: Epoch 7 of 500. Elapsed time 7.63 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.48 sec. MSE loss 1.25E+00. Sample per second: 1320201\n",
      "FUNK_SVD: Epoch 8 of 500. Elapsed time 8.01 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 1.02 sec. MSE loss 1.22E+00. Sample per second: 626695\n",
      "FUNK_SVD: Epoch 9 of 500. Elapsed time 8.55 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.59 sec. MSE loss 1.20E+00. Sample per second: 1088726\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.83 sec. Users per second: 1246\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1150398, PRECISION_RECALL_MIN_DEN: 0.1273140, RECALL: 0.0590825, MAP: 0.0539225, MAP_MIN_DEN: 0.0591474, MRR: 0.2726287, NDCG: 0.1114386, F1: 0.0780697, HIT_RATE: 0.5815073, ARHR_ALL_HITS: 0.3758597, NOVELTY: 0.0221955, AVERAGE_POPULARITY: 0.7460864, DIVERSITY_MEAN_INTER_LIST: 0.4282136, DIVERSITY_HERFINDAHL: 0.9428143, COVERAGE_ITEM: 0.0126191, COVERAGE_ITEM_CORRECT: 0.0110739, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5799669, DIVERSITY_GINI: 0.0043473, SHANNON_ENTROPY: 4.3317948, RATIO_DIVERSITY_HERFINDAHL: 0.9435845, RATIO_DIVERSITY_GINI: 0.0124333, RATIO_SHANNON_ENTROPY: 0.4006147, RATIO_AVERAGE_POPULARITY: 2.7010994, RATIO_NOVELTY: 0.0752067, \n",
      "\n",
      "FUNK_SVD: Epoch 10 of 500. Elapsed time 13.96 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.74 sec. MSE loss 1.18E+00. Sample per second: 865784\n",
      "FUNK_SVD: Epoch 11 of 500. Elapsed time 14.27 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 1.09 sec. MSE loss 1.17E+00. Sample per second: 584751\n",
      "FUNK_SVD: Epoch 12 of 500. Elapsed time 14.62 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.52 sec. MSE loss 1.17E+00. Sample per second: 1231095\n",
      "FUNK_SVD: Epoch 13 of 500. Elapsed time 15.05 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 1.18 sec. MSE loss 1.16E+00. Sample per second: 544396\n",
      "FUNK_SVD: Epoch 14 of 500. Elapsed time 15.70 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.59 sec. MSE loss 1.16E+00. Sample per second: 1076914\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.83 sec. Users per second: 1248\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1133466, PRECISION_RECALL_MIN_DEN: 0.1256490, RECALL: 0.0584146, MAP: 0.0528744, MAP_MIN_DEN: 0.0580198, MRR: 0.2699046, NDCG: 0.1108023, F1: 0.0770966, HIT_RATE: 0.5800133, ARHR_ALL_HITS: 0.3705976, NOVELTY: 0.0222272, AVERAGE_POPULARITY: 0.7402497, DIVERSITY_MEAN_INTER_LIST: 0.4160216, DIVERSITY_HERFINDAHL: 0.9415953, COVERAGE_ITEM: 0.0126191, COVERAGE_ITEM_CORRECT: 0.0108164, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5784768, DIVERSITY_GINI: 0.0042826, SHANNON_ENTROPY: 4.3031430, RATIO_DIVERSITY_HERFINDAHL: 0.9423645, RATIO_DIVERSITY_GINI: 0.0122484, RATIO_SHANNON_ENTROPY: 0.3979649, RATIO_AVERAGE_POPULARITY: 2.6799683, RATIO_NOVELTY: 0.0753143, \n",
      "\n",
      "FUNK_SVD: Epoch 15 of 500. Elapsed time 20.95 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.63 sec. MSE loss 1.15E+00. Sample per second: 1009074\n",
      "FUNK_SVD: Epoch 16 of 500. Elapsed time 21.16 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 1.05 sec. MSE loss 1.15E+00. Sample per second: 609343\n",
      "FUNK_SVD: Epoch 17 of 500. Elapsed time 21.58 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.53 sec. MSE loss 1.15E+00. Sample per second: 1212388\n",
      "FUNK_SVD: Epoch 18 of 500. Elapsed time 22.06 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 1.11 sec. MSE loss 1.14E+00. Sample per second: 574156\n",
      "FUNK_SVD: Epoch 19 of 500. Elapsed time 22.64 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.60 sec. MSE loss 1.14E+00. Sample per second: 1063310\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.60 sec. Users per second: 1309\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1116866, PRECISION_RECALL_MIN_DEN: 0.1233766, RECALL: 0.0569712, MAP: 0.0523756, MAP_MIN_DEN: 0.0574357, MRR: 0.2693724, NDCG: 0.1099699, F1: 0.0754536, HIT_RATE: 0.5727092, ARHR_ALL_HITS: 0.3686432, NOVELTY: 0.0222625, AVERAGE_POPULARITY: 0.7339001, DIVERSITY_MEAN_INTER_LIST: 0.4064934, DIVERSITY_HERFINDAHL: 0.9406426, COVERAGE_ITEM: 0.0118465, COVERAGE_ITEM_CORRECT: 0.0100438, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5711921, DIVERSITY_GINI: 0.0042297, SHANNON_ENTROPY: 4.2854074, RATIO_DIVERSITY_HERFINDAHL: 0.9414111, RATIO_DIVERSITY_GINI: 0.0120972, RATIO_SHANNON_ENTROPY: 0.3963246, RATIO_AVERAGE_POPULARITY: 2.6569806, RATIO_NOVELTY: 0.0754340, \n",
      "\n",
      "FUNK_SVD: Epoch 20 of 500. Elapsed time 27.74 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.50 sec. MSE loss 1.14E+00. Sample per second: 1287377\n",
      "FUNK_SVD: Epoch 21 of 500. Elapsed time 28.03 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.89 sec. MSE loss 1.14E+00. Sample per second: 720434\n",
      "FUNK_SVD: Epoch 22 of 500. Elapsed time 28.42 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 1.51 sec. MSE loss 1.14E+00. Sample per second: 423364\n",
      "FUNK_SVD: Epoch 23 of 500. Elapsed time 29.04 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 1.04 sec. MSE loss 1.14E+00. Sample per second: 616978\n",
      "FUNK_SVD: Epoch 24 of 500. Elapsed time 29.57 sec\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Processed 640135 (100.0%) in 0.40 sec. MSE loss 1.13E+00. Sample per second: 1619413\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.61 sec. Users per second: 1306\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1102922, PRECISION_RECALL_MIN_DEN: 0.1217955, RECALL: 0.0561448, MAP: 0.0518545, MAP_MIN_DEN: 0.0568572, MRR: 0.2680191, NDCG: 0.1090101, F1: 0.0744105, HIT_RATE: 0.5678951, ARHR_ALL_HITS: 0.3659392, NOVELTY: 0.0222883, AVERAGE_POPULARITY: 0.7294135, DIVERSITY_MEAN_INTER_LIST: 0.3999889, DIVERSITY_HERFINDAHL: 0.9399922, COVERAGE_ITEM: 0.0121040, COVERAGE_ITEM_CORRECT: 0.0110739, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5663907, DIVERSITY_GINI: 0.0041905, SHANNON_ENTROPY: 4.2718474, RATIO_DIVERSITY_HERFINDAHL: 0.9407602, RATIO_DIVERSITY_GINI: 0.0119851, RATIO_SHANNON_ENTROPY: 0.3950706, RATIO_AVERAGE_POPULARITY: 2.6407375, RATIO_NOVELTY: 0.0755212, \n",
      "\n",
      "FUNK_SVD: Epoch 25 of 500. Elapsed time 34.54 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.35 sec. MSE loss 1.13E+00. Sample per second: 1838340\n",
      "FUNK_SVD: Epoch 26 of 500. Elapsed time 34.88 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.73 sec. MSE loss 1.13E+00. Sample per second: 871865\n",
      "FUNK_SVD: Epoch 27 of 500. Elapsed time 35.26 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 1.37 sec. MSE loss 1.12E+00. Sample per second: 467256\n",
      "FUNK_SVD: Epoch 28 of 500. Elapsed time 35.90 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.81 sec. MSE loss 1.12E+00. Sample per second: 784510\n",
      "FUNK_SVD: Epoch 29 of 500. Elapsed time 36.35 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 1.17 sec. MSE loss 1.12E+00. Sample per second: 549263\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.81 sec. Users per second: 1252\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1098938, PRECISION_RECALL_MIN_DEN: 0.1215035, RECALL: 0.0560757, MAP: 0.0515172, MAP_MIN_DEN: 0.0565258, MRR: 0.2665011, NDCG: 0.1087536, F1: 0.0742591, HIT_RATE: 0.5668991, ARHR_ALL_HITS: 0.3636848, NOVELTY: 0.0223127, AVERAGE_POPULARITY: 0.7252378, DIVERSITY_MEAN_INTER_LIST: 0.3958457, DIVERSITY_HERFINDAHL: 0.9395780, COVERAGE_ITEM: 0.0123616, COVERAGE_ITEM_CORRECT: 0.0105588, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5653974, DIVERSITY_GINI: 0.0041670, SHANNON_ENTROPY: 4.2604205, RATIO_DIVERSITY_HERFINDAHL: 0.9403456, RATIO_DIVERSITY_GINI: 0.0119178, RATIO_SHANNON_ENTROPY: 0.3940138, RATIO_AVERAGE_POPULARITY: 2.6256199, RATIO_NOVELTY: 0.0756040, \n",
      "\n",
      "FUNK_SVD: Convergence reached! Terminating at epoch 30. Best value for 'MAP' at epoch 5 is 0.0550. Elapsed time 41.51 sec\n",
      "FUNK_SVD: Epoch 30 of 500. Elapsed time 41.51 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.54 sec. Users per second: 1326\n",
      "SearchBayesianSkopt: Config 8 is suboptimal. Config: {'epochs': 5, 'num_factors': 1, 'sgd_mode': 'adagrad', 'batch_size': 1, 'item_reg': 1e-05, 'user_reg': 1e-05, 'learning_rate': 0.001334504569942782} - results: PRECISION: 0.1165007, PRECISION_RECALL_MIN_DEN: 0.1291117, RECALL: 0.0600661, MAP: 0.0550182, MAP_MIN_DEN: 0.0604231, MRR: 0.2758832, NDCG: 0.1118487, F1: 0.0792645, HIT_RATE: 0.5859894, ARHR_ALL_HITS: 0.3822948, NOVELTY: 0.0221626, AVERAGE_POPULARITY: 0.7520878, DIVERSITY_MEAN_INTER_LIST: 0.4508033, DIVERSITY_HERFINDAHL: 0.9450728, COVERAGE_ITEM: 0.0128766, COVERAGE_ITEM_CORRECT: 0.0115890, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5844371, DIVERSITY_GINI: 0.0045158, SHANNON_ENTROPY: 4.4039840, RATIO_DIVERSITY_HERFINDAHL: 0.9458449, RATIO_DIVERSITY_GINI: 0.0129154, RATIO_SHANNON_ENTROPY: 0.4072909, RATIO_AVERAGE_POPULARITY: 2.7228265, RATIO_NOVELTY: 0.0750955, \n",
      "\n",
      "Iteration No: 9 ended. Search finished for the next optimal point.\n",
      "Time taken: 47.2636\n",
      "Function value obtained: -0.0550\n",
      "Current minimum: -0.0572\n",
      "Iteration No: 10 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'epochs': 500, 'num_factors': 1, 'sgd_mode': 'adagrad', 'batch_size': 1, 'item_reg': 0.01, 'user_reg': 1e-05, 'learning_rate': 0.0008899431406346556}\n",
      "MatrixFactorization_FunkSVD_Cython_Recommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 1.25 sec. MSE loss 9.24E+00. Sample per second: 512930\n",
      "FUNK_SVD: Epoch 1 of 500. Elapsed time 0.51 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.70 sec. MSE loss 5.87E+00. Sample per second: 917933\n",
      "FUNK_SVD: Epoch 2 of 500. Elapsed time 0.96 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 1.03 sec. MSE loss 4.37E+00. Sample per second: 621005\n",
      "FUNK_SVD: Epoch 3 of 500. Elapsed time 1.30 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.50 sec. MSE loss 3.45E+00. Sample per second: 1288703\n",
      "FUNK_SVD: Epoch 4 of 500. Elapsed time 1.77 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 1.05 sec. MSE loss 2.85E+00. Sample per second: 608509\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.45 sec. Users per second: 1746\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1159363, PRECISION_RECALL_MIN_DEN: 0.1291929, RECALL: 0.0604679, MAP: 0.0548675, MAP_MIN_DEN: 0.0600584, MRR: 0.2735066, NDCG: 0.1095503, F1: 0.0794814, HIT_RATE: 0.5848274, ARHR_ALL_HITS: 0.3801449, NOVELTY: 0.0221574, AVERAGE_POPULARITY: 0.7538559, DIVERSITY_MEAN_INTER_LIST: 0.5012433, DIVERSITY_HERFINDAHL: 0.9501160, COVERAGE_ITEM: 0.0149369, COVERAGE_ITEM_CORRECT: 0.0136492, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5832781, DIVERSITY_GINI: 0.0049283, SHANNON_ENTROPY: 4.5526662, RATIO_DIVERSITY_HERFINDAHL: 0.9508922, RATIO_DIVERSITY_GINI: 0.0140950, RATIO_SHANNON_ENTROPY: 0.4210414, RATIO_AVERAGE_POPULARITY: 2.7292278, RATIO_NOVELTY: 0.0750778, \n",
      "\n",
      "FUNK_SVD: New best model found! Updating.\n",
      "FUNK_SVD: Epoch 5 of 500. Elapsed time 5.77 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.72 sec. MSE loss 2.43E+00. Sample per second: 883368\n",
      "FUNK_SVD: Epoch 6 of 500. Elapsed time 5.99 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.94 sec. MSE loss 2.13E+00. Sample per second: 680012\n",
      "FUNK_SVD: Epoch 7 of 500. Elapsed time 6.21 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 1.16 sec. MSE loss 1.90E+00. Sample per second: 552382\n",
      "FUNK_SVD: Epoch 8 of 500. Elapsed time 6.42 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.35 sec. MSE loss 1.74E+00. Sample per second: 1820482\n",
      "FUNK_SVD: Epoch 9 of 500. Elapsed time 6.62 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.54 sec. MSE loss 1.61E+00. Sample per second: 1178239\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.35 sec. Users per second: 1797\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1152556, PRECISION_RECALL_MIN_DEN: 0.1281252, RECALL: 0.0598491, MAP: 0.0545930, MAP_MIN_DEN: 0.0597581, MRR: 0.2727599, NDCG: 0.1102989, F1: 0.0787865, HIT_RATE: 0.5826693, ARHR_ALL_HITS: 0.3788866, NOVELTY: 0.0221655, AVERAGE_POPULARITY: 0.7520610, DIVERSITY_MEAN_INTER_LIST: 0.4696559, DIVERSITY_HERFINDAHL: 0.9469578, COVERAGE_ITEM: 0.0133917, COVERAGE_ITEM_CORRECT: 0.0121040, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5811258, DIVERSITY_GINI: 0.0046505, SHANNON_ENTROPY: 4.4500865, RATIO_DIVERSITY_HERFINDAHL: 0.9477314, RATIO_DIVERSITY_GINI: 0.0133005, RATIO_SHANNON_ENTROPY: 0.4115546, RATIO_AVERAGE_POPULARITY: 2.7227294, RATIO_NOVELTY: 0.0751053, \n",
      "\n",
      "FUNK_SVD: Epoch 10 of 500. Elapsed time 10.17 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 1.09 sec. MSE loss 1.52E+00. Sample per second: 584736\n",
      "FUNK_SVD: Epoch 11 of 500. Elapsed time 10.36 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.31 sec. MSE loss 1.45E+00. Sample per second: 2075085\n",
      "FUNK_SVD: Epoch 12 of 500. Elapsed time 10.57 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.58 sec. MSE loss 1.39E+00. Sample per second: 1096148\n",
      "FUNK_SVD: Epoch 13 of 500. Elapsed time 10.85 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.97 sec. MSE loss 1.34E+00. Sample per second: 659404\n",
      "FUNK_SVD: Epoch 14 of 500. Elapsed time 11.24 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 1.43 sec. MSE loss 1.31E+00. Sample per second: 445787\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.10 sec. Users per second: 1941\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1149070, PRECISION_RECALL_MIN_DEN: 0.1277776, RECALL: 0.0596349, MAP: 0.0542328, MAP_MIN_DEN: 0.0594672, MRR: 0.2712866, NDCG: 0.1107940, F1: 0.0785195, HIT_RATE: 0.5818393, ARHR_ALL_HITS: 0.3762333, NOVELTY: 0.0221753, AVERAGE_POPULARITY: 0.7502187, DIVERSITY_MEAN_INTER_LIST: 0.4494750, DIVERSITY_HERFINDAHL: 0.9449400, COVERAGE_ITEM: 0.0133917, COVERAGE_ITEM_CORRECT: 0.0115890, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5802980, DIVERSITY_GINI: 0.0045198, SHANNON_ENTROPY: 4.3892455, RATIO_DIVERSITY_HERFINDAHL: 0.9457120, RATIO_DIVERSITY_GINI: 0.0129269, RATIO_SHANNON_ENTROPY: 0.4059278, RATIO_AVERAGE_POPULARITY: 2.7160598, RATIO_NOVELTY: 0.0751384, \n",
      "\n",
      "FUNK_SVD: Epoch 15 of 500. Elapsed time 14.81 sec\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Processed 640135 (100.0%) in 1.00 sec. MSE loss 1.28E+00. Sample per second: 637050\n",
      "FUNK_SVD: Epoch 16 of 500. Elapsed time 15.27 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.46 sec. MSE loss 1.26E+00. Sample per second: 1399274\n",
      "FUNK_SVD: Epoch 17 of 500. Elapsed time 15.72 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.69 sec. MSE loss 1.24E+00. Sample per second: 923494\n",
      "FUNK_SVD: Epoch 18 of 500. Elapsed time 15.96 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.91 sec. MSE loss 1.23E+00. Sample per second: 702973\n",
      "FUNK_SVD: Epoch 19 of 500. Elapsed time 16.18 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 1.13 sec. MSE loss 1.21E+00. Sample per second: 568212\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.43 sec. Users per second: 1756\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1141102, PRECISION_RECALL_MIN_DEN: 0.1266480, RECALL: 0.0590362, MAP: 0.0535163, MAP_MIN_DEN: 0.0586453, MRR: 0.2690154, NDCG: 0.1102873, F1: 0.0778143, HIT_RATE: 0.5808433, ARHR_ALL_HITS: 0.3724712, NOVELTY: 0.0221889, AVERAGE_POPULARITY: 0.7478132, DIVERSITY_MEAN_INTER_LIST: 0.4365560, DIVERSITY_HERFINDAHL: 0.9436484, COVERAGE_ITEM: 0.0126191, COVERAGE_ITEM_CORRECT: 0.0113314, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5793046, DIVERSITY_GINI: 0.0044373, SHANNON_ENTROPY: 4.3531049, RATIO_DIVERSITY_HERFINDAHL: 0.9444193, RATIO_DIVERSITY_GINI: 0.0126910, RATIO_SHANNON_ENTROPY: 0.4025855, RATIO_AVERAGE_POPULARITY: 2.7073510, RATIO_NOVELTY: 0.0751845, \n",
      "\n",
      "FUNK_SVD: Epoch 20 of 500. Elapsed time 19.83 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.78 sec. MSE loss 1.20E+00. Sample per second: 819128\n",
      "FUNK_SVD: Epoch 21 of 500. Elapsed time 20.05 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 1.00 sec. MSE loss 1.19E+00. Sample per second: 640970\n",
      "FUNK_SVD: Epoch 22 of 500. Elapsed time 20.26 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.22 sec. MSE loss 1.19E+00. Sample per second: 2967216\n",
      "FUNK_SVD: Epoch 23 of 500. Elapsed time 20.48 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.41 sec. MSE loss 1.19E+00. Sample per second: 1563805\n",
      "FUNK_SVD: Epoch 24 of 500. Elapsed time 20.67 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.60 sec. MSE loss 1.18E+00. Sample per second: 1061210\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.33 sec. Users per second: 1808\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1135956, PRECISION_RECALL_MIN_DEN: 0.1255429, RECALL: 0.0579606, MAP: 0.0531480, MAP_MIN_DEN: 0.0581652, MRR: 0.2677262, NDCG: 0.1098783, F1: 0.0767570, HIT_RATE: 0.5765272, ARHR_ALL_HITS: 0.3700249, NOVELTY: 0.0222054, AVERAGE_POPULARITY: 0.7446774, DIVERSITY_MEAN_INTER_LIST: 0.4264302, DIVERSITY_HERFINDAHL: 0.9426359, COVERAGE_ITEM: 0.0126191, COVERAGE_ITEM_CORRECT: 0.0110739, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5750000, DIVERSITY_GINI: 0.0043517, SHANNON_ENTROPY: 4.3289752, RATIO_DIVERSITY_HERFINDAHL: 0.9434060, RATIO_DIVERSITY_GINI: 0.0124459, RATIO_SHANNON_ENTROPY: 0.4003539, RATIO_AVERAGE_POPULARITY: 2.6959982, RATIO_NOVELTY: 0.0752403, \n",
      "\n",
      "FUNK_SVD: Epoch 25 of 500. Elapsed time 24.21 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 1.13 sec. MSE loss 1.18E+00. Sample per second: 568949\n",
      "FUNK_SVD: Epoch 26 of 500. Elapsed time 24.39 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.33 sec. MSE loss 1.17E+00. Sample per second: 1960379\n",
      "FUNK_SVD: Epoch 27 of 500. Elapsed time 24.59 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.54 sec. MSE loss 1.17E+00. Sample per second: 1187293\n",
      "FUNK_SVD: Epoch 28 of 500. Elapsed time 24.80 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 0.95 sec. MSE loss 1.17E+00. Sample per second: 672700\n",
      "FUNK_SVD: Epoch 29 of 500. Elapsed time 25.22 sec\n",
      "FUNK_SVD: Processed 640135 (100.0%) in 1.34 sec. MSE loss 1.17E+00. Sample per second: 476172\n",
      "FUNK_SVD: Validation begins...\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.35 sec. Users per second: 1800\n",
      "FUNK_SVD: CUTOFF: 10 - PRECISION: 0.1130644, PRECISION_RECALL_MIN_DEN: 0.1249840, RECALL: 0.0576662, MAP: 0.0528998, MAP_MIN_DEN: 0.0578925, MRR: 0.2672263, NDCG: 0.1098980, F1: 0.0763776, HIT_RATE: 0.5747012, ARHR_ALL_HITS: 0.3688804, NOVELTY: 0.0222283, AVERAGE_POPULARITY: 0.7401877, DIVERSITY_MEAN_INTER_LIST: 0.4180126, DIVERSITY_HERFINDAHL: 0.9417943, COVERAGE_ITEM: 0.0121040, COVERAGE_ITEM_CORRECT: 0.0110739, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5731788, DIVERSITY_GINI: 0.0042950, SHANNON_ENTROPY: 4.3100907, RATIO_DIVERSITY_HERFINDAHL: 0.9425637, RATIO_DIVERSITY_GINI: 0.0122838, RATIO_SHANNON_ENTROPY: 0.3986074, RATIO_AVERAGE_POPULARITY: 2.6797439, RATIO_NOVELTY: 0.0753181, \n",
      "\n",
      "FUNK_SVD: Convergence reached! Terminating at epoch 30. Best value for 'MAP' at epoch 5 is 0.0549. Elapsed time 28.96 sec\n",
      "FUNK_SVD: Epoch 30 of 500. Elapsed time 28.96 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.25 sec. Users per second: 1854\n",
      "SearchBayesianSkopt: Config 9 is suboptimal. Config: {'epochs': 5, 'num_factors': 1, 'sgd_mode': 'adagrad', 'batch_size': 1, 'item_reg': 0.01, 'user_reg': 1e-05, 'learning_rate': 0.0008899431406346556} - results: PRECISION: 0.1159363, PRECISION_RECALL_MIN_DEN: 0.1291929, RECALL: 0.0604679, MAP: 0.0548675, MAP_MIN_DEN: 0.0600584, MRR: 0.2735066, NDCG: 0.1095503, F1: 0.0794814, HIT_RATE: 0.5848274, ARHR_ALL_HITS: 0.3801449, NOVELTY: 0.0221574, AVERAGE_POPULARITY: 0.7538559, DIVERSITY_MEAN_INTER_LIST: 0.5012433, DIVERSITY_HERFINDAHL: 0.9501160, COVERAGE_ITEM: 0.0149369, COVERAGE_ITEM_CORRECT: 0.0136492, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5832781, DIVERSITY_GINI: 0.0049283, SHANNON_ENTROPY: 4.5526662, RATIO_DIVERSITY_HERFINDAHL: 0.9508922, RATIO_DIVERSITY_GINI: 0.0140950, RATIO_SHANNON_ENTROPY: 0.4210414, RATIO_AVERAGE_POPULARITY: 2.7292278, RATIO_NOVELTY: 0.0750778, \n",
      "\n",
      "Iteration No: 10 ended. Search finished for the next optimal point.\n",
      "Time taken: 33.3288\n",
      "Function value obtained: -0.0549\n",
      "Current minimum: -0.0572\n",
      "SearchBayesianSkopt: Search complete. Best config is 7: {'epochs': 15, 'num_factors': 1, 'sgd_mode': 'adagrad', 'batch_size': 64, 'item_reg': 0.01, 'user_reg': 0.01, 'learning_rate': 0.0011332925114552466}\n",
      "\n",
      "MatrixFactorization_FunkSVD_Cython_Recommender: URM Detected 212 ( 5.5%) items with no interactions.\n",
      "SearchBayesianSkopt: Evaluation with constructor data for final test. Using best config: {'epochs': 15, 'num_factors': 1, 'sgd_mode': 'adagrad', 'batch_size': 64, 'item_reg': 0.01, 'user_reg': 0.01, 'learning_rate': 0.0011332925114552466}\n",
      "FUNK_SVD: Processed 800192 (100.0%) in 0.55 sec. MSE loss 1.34E+01. Sample per second: 1453447\n",
      "FUNK_SVD: Epoch 1 of 15. Elapsed time 0.48 sec\n",
      "FUNK_SVD: Processed 800192 (100.0%) in 0.79 sec. MSE loss 1.21E+01. Sample per second: 1008101\n",
      "FUNK_SVD: Epoch 2 of 15. Elapsed time 0.72 sec\n",
      "FUNK_SVD: Processed 800192 (100.0%) in 1.04 sec. MSE loss 1.14E+01. Sample per second: 771845\n",
      "FUNK_SVD: Epoch 3 of 15. Elapsed time 0.97 sec\n",
      "FUNK_SVD: Processed 800192 (100.0%) in 0.28 sec. MSE loss 1.08E+01. Sample per second: 2863640\n",
      "FUNK_SVD: Epoch 4 of 15. Elapsed time 1.21 sec\n",
      "FUNK_SVD: Processed 800192 (100.0%) in 0.52 sec. MSE loss 1.03E+01. Sample per second: 1525463\n",
      "FUNK_SVD: Epoch 5 of 15. Elapsed time 1.45 sec\n",
      "FUNK_SVD: Processed 800192 (100.0%) in 0.75 sec. MSE loss 9.90E+00. Sample per second: 1063621\n",
      "FUNK_SVD: Epoch 6 of 15. Elapsed time 1.68 sec\n",
      "FUNK_SVD: Processed 800192 (100.0%) in 1.07 sec. MSE loss 9.55E+00. Sample per second: 749877\n",
      "FUNK_SVD: Epoch 7 of 15. Elapsed time 2.00 sec\n",
      "FUNK_SVD: Processed 800192 (100.0%) in 0.45 sec. MSE loss 9.21E+00. Sample per second: 1791899\n",
      "FUNK_SVD: Epoch 8 of 15. Elapsed time 2.38 sec\n",
      "FUNK_SVD: Processed 800192 (100.0%) in 0.85 sec. MSE loss 8.91E+00. Sample per second: 939981\n",
      "FUNK_SVD: Epoch 9 of 15. Elapsed time 2.78 sec\n",
      "FUNK_SVD: Processed 800192 (100.0%) in 1.28 sec. MSE loss 8.63E+00. Sample per second: 623203\n",
      "FUNK_SVD: Epoch 10 of 15. Elapsed time 3.21 sec\n",
      "FUNK_SVD: Processed 800192 (100.0%) in 0.82 sec. MSE loss 8.37E+00. Sample per second: 979991\n",
      "FUNK_SVD: Epoch 11 of 15. Elapsed time 3.75 sec\n",
      "FUNK_SVD: Processed 800192 (100.0%) in 1.10 sec. MSE loss 8.14E+00. Sample per second: 729626\n",
      "FUNK_SVD: Epoch 12 of 15. Elapsed time 4.03 sec\n",
      "FUNK_SVD: Processed 800192 (100.0%) in 0.34 sec. MSE loss 7.91E+00. Sample per second: 2358089\n",
      "FUNK_SVD: Epoch 13 of 15. Elapsed time 4.27 sec\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FUNK_SVD: Processed 800192 (100.0%) in 0.58 sec. MSE loss 7.71E+00. Sample per second: 1375515\n",
      "FUNK_SVD: Epoch 14 of 15. Elapsed time 4.51 sec\n",
      "FUNK_SVD: Processed 800192 (100.0%) in 0.82 sec. MSE loss 7.51E+00. Sample per second: 972444\n",
      "FUNK_SVD: Epoch 15 of 15. Elapsed time 4.75 sec\n",
      "FUNK_SVD: Terminating at epoch 15. Elapsed time 4.75 sec\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 3.44 sec. Users per second: 1754\n",
      "SearchBayesianSkopt: Best config evaluated with evaluator_test with constructor data for final test. Config: {'epochs': 15, 'num_factors': 1, 'sgd_mode': 'adagrad', 'batch_size': 64, 'item_reg': 0.01, 'user_reg': 0.01, 'learning_rate': 0.0011332925114552466} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.1763419, PRECISION_RECALL_MIN_DEN: 0.1864401, RECALL: 0.0674992, MAP: 0.1051057, MAP_MIN_DEN: 0.1090986, MRR: 0.3674129, NDCG: 0.1543828, F1: 0.0976286, HIT_RATE: 0.6706428, ARHR_ALL_HITS: 0.5930055, NOVELTY: 0.0221507, AVERAGE_POPULARITY: 0.7488238, DIVERSITY_MEAN_INTER_LIST: 0.5295465, DIVERSITY_HERFINDAHL: 0.9529459, COVERAGE_ITEM: 0.0198300, COVERAGE_ITEM_CORRECT: 0.0185424, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.6701987, DIVERSITY_GINI: 0.0052895, SHANNON_ENTROPY: 4.6633545, RATIO_DIVERSITY_HERFINDAHL: 0.9537253, RATIO_DIVERSITY_GINI: 0.0151270, RATIO_SHANNON_ENTROPY: 0.4312799, RATIO_AVERAGE_POPULARITY: 2.7259662, RATIO_NOVELTY: 0.0600442, \n",
      "\n",
      "\n",
      "SearchBayesianSkopt: Saving model in result_experiments/MatrixFactorization_FunkSVD_Cython_Recommender\n",
      "\n",
      "MatrixFactorization_FunkSVD_Cython_Recommender: Saving model in file 'result_experiments/MatrixFactorization_FunkSVD_Cython_Recommender_best_model_last'\n",
      "MatrixFactorization_FunkSVD_Cython_Recommender: Saving complete\n"
     ]
    }
   ],
   "source": [
    "hyperparameterSearch.search(recommender_input_args,\n",
    "                       recommender_input_args_last_test = recommender_input_args_last_test,\n",
    "                       hyperparameter_search_space = hyperparameters_range_dictionary,\n",
    "                       n_cases = n_cases,\n",
    "                       n_random_starts = n_random_starts,\n",
    "                       save_model = \"last\",\n",
    "                       output_folder_path = output_folder_path, # Where to save the results\n",
    "                       output_file_name_root = recommender_class.RECOMMENDER_NAME, # How to call the files\n",
    "                       metric_to_optimize = metric_to_optimize,\n",
    "                       cutoff_to_optimize = cutoff_to_optimize,\n",
    "                      )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>epochs</th>\n",
       "      <th>num_factors</th>\n",
       "      <th>sgd_mode</th>\n",
       "      <th>batch_size</th>\n",
       "      <th>item_reg</th>\n",
       "      <th>user_reg</th>\n",
       "      <th>learning_rate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5</td>\n",
       "      <td>53</td>\n",
       "      <td>sgd</td>\n",
       "      <td>8</td>\n",
       "      <td>0.000129</td>\n",
       "      <td>0.000013</td>\n",
       "      <td>0.00115</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>20</td>\n",
       "      <td>40</td>\n",
       "      <td>adagrad</td>\n",
       "      <td>16</td>\n",
       "      <td>0.000336</td>\n",
       "      <td>0.000021</td>\n",
       "      <td>0.002356</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>500</td>\n",
       "      <td>122</td>\n",
       "      <td>sgd</td>\n",
       "      <td>256</td>\n",
       "      <td>0.006262</td>\n",
       "      <td>0.000235</td>\n",
       "      <td>0.000167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10</td>\n",
       "      <td>48</td>\n",
       "      <td>sgd</td>\n",
       "      <td>2</td>\n",
       "      <td>0.000157</td>\n",
       "      <td>0.000026</td>\n",
       "      <td>0.008652</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>132</td>\n",
       "      <td>adagrad</td>\n",
       "      <td>32</td>\n",
       "      <td>0.000016</td>\n",
       "      <td>0.001534</td>\n",
       "      <td>0.090911</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>120</td>\n",
       "      <td>108</td>\n",
       "      <td>adagrad</td>\n",
       "      <td>32</td>\n",
       "      <td>0.00358</td>\n",
       "      <td>0.000014</td>\n",
       "      <td>0.001229</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>260</td>\n",
       "      <td>20</td>\n",
       "      <td>adagrad</td>\n",
       "      <td>4</td>\n",
       "      <td>0.000015</td>\n",
       "      <td>0.002114</td>\n",
       "      <td>0.000453</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>15</td>\n",
       "      <td>1</td>\n",
       "      <td>adagrad</td>\n",
       "      <td>64</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.001133</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>adagrad</td>\n",
       "      <td>1</td>\n",
       "      <td>0.00001</td>\n",
       "      <td>0.00001</td>\n",
       "      <td>0.001335</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>adagrad</td>\n",
       "      <td>1</td>\n",
       "      <td>0.01</td>\n",
       "      <td>0.00001</td>\n",
       "      <td>0.00089</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  epochs num_factors sgd_mode batch_size  item_reg  user_reg learning_rate\n",
       "0      5          53      sgd          8  0.000129  0.000013       0.00115\n",
       "1     20          40  adagrad         16  0.000336  0.000021      0.002356\n",
       "2    500         122      sgd        256  0.006262  0.000235      0.000167\n",
       "3     10          48      sgd          2  0.000157  0.000026      0.008652\n",
       "4      5         132  adagrad         32  0.000016  0.001534      0.090911\n",
       "5    120         108  adagrad         32   0.00358  0.000014      0.001229\n",
       "6    260          20  adagrad          4  0.000015  0.002114      0.000453\n",
       "7     15           1  adagrad         64      0.01      0.01      0.001133\n",
       "8      5           1  adagrad          1   0.00001   0.00001      0.001335\n",
       "9      5           1  adagrad          1      0.01   0.00001       0.00089"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "search_metadata = data_loader.load_data(recommender_class.RECOMMENDER_NAME + \"_metadata.zip\")\n",
    "\n",
    "hyperparameters_df = search_metadata[\"hyperparameters_df\"]\n",
    "hyperparameters_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>train</th>\n",
       "      <th>validation</th>\n",
       "      <th>test</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>105.788402</td>\n",
       "      <td>3.551863</td>\n",
       "      <td>3.345683</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>228.568995</td>\n",
       "      <td>4.646658</td>\n",
       "      <td>4.720899</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4691.106388</td>\n",
       "      <td>4.741997</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>157.704445</td>\n",
       "      <td>4.618278</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>370.884716</td>\n",
       "      <td>4.37645</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1548.496231</td>\n",
       "      <td>4.912879</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>900.066335</td>\n",
       "      <td>4.686435</td>\n",
       "      <td>4.761179</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>51.814222</td>\n",
       "      <td>4.571804</td>\n",
       "      <td>4.823095</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>41.532243</td>\n",
       "      <td>4.548487</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>28.977696</td>\n",
       "      <td>3.262919</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         train validation      test\n",
       "0   105.788402   3.551863  3.345683\n",
       "1   228.568995   4.646658  4.720899\n",
       "2  4691.106388   4.741997       NaN\n",
       "3   157.704445   4.618278       NaN\n",
       "4   370.884716    4.37645       NaN\n",
       "5  1548.496231   4.912879       NaN\n",
       "6   900.066335   4.686435  4.761179\n",
       "7    51.814222   4.571804  4.823095\n",
       "8    41.532243   4.548487       NaN\n",
       "9    28.977696   3.262919       NaN"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "time_df = search_metadata[\"time_df\"]\n",
    "time_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'epochs': 15,\n",
       " 'num_factors': 1,\n",
       " 'sgd_mode': 'adagrad',\n",
       " 'batch_size': 64,\n",
       " 'item_reg': 0.01,\n",
       " 'user_reg': 0.01,\n",
       " 'learning_rate': 0.0011332925114552466}"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_hyperparameters = search_metadata[\"hyperparameters_best\"]\n",
    "best_hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to use the predefined hyperparameter ranges\n",
    "\n",
    "Function runParameterSearch_Collaborative takes as input all needed to optimize any of the recommenders in the framework as well as other parameters such as which similarity heuristics to use for the KNNs, whether to parallelize the training of KNNs and so on..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "similarity_type_list = ['cosine', 'jaccard', \"asymmetric\", \"dice\", \"tversky\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SearchSingleCase: Resuming 'TopPopRecommender' Failed, no such file exists.\n",
      "\n",
      "SearchSingleCase: Testing config: {}\n",
      "TopPopRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 2.63 sec. Users per second: 2291\n",
      "SearchSingleCase: New best config found. Config 0: {} - results: PRECISION: 0.1217961, PRECISION_RECALL_MIN_DEN: 0.1349623, RECALL: 0.0622819, MAP: 0.0596391, MAP_MIN_DEN: 0.0653490, MRR: 0.2875275, NDCG: 0.1124584, F1: 0.0824182, HIT_RATE: 0.5946215, ARHR_ALL_HITS: 0.4067459, NOVELTY: 0.0220221, AVERAGE_POPULARITY: 0.7784131, DIVERSITY_MEAN_INTER_LIST: 0.4049413, DIVERSITY_HERFINDAHL: 0.9404874, COVERAGE_ITEM: 0.0113314, COVERAGE_ITEM_CORRECT: 0.0100438, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5930464, DIVERSITY_GINI: 0.0042281, SHANNON_ENTROPY: 4.2698400, RATIO_DIVERSITY_HERFINDAHL: 0.9412558, RATIO_DIVERSITY_GINI: 0.0120924, RATIO_SHANNON_ENTROPY: 0.3948849, RATIO_AVERAGE_POPULARITY: 2.8181335, RATIO_NOVELTY: 0.0746194, \n",
      "\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 2.77 sec. Users per second: 2176\n",
      "SearchSingleCase: Config evaluated with evaluator_test. Config: {} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.1502154, PRECISION_RECALL_MIN_DEN: 0.1594144, RECALL: 0.0611866, MAP: 0.0773867, MAP_MIN_DEN: 0.0809969, MRR: 0.3232669, NDCG: 0.1285539, F1: 0.0869545, HIT_RATE: 0.6434725, ARHR_ALL_HITS: 0.4875103, NOVELTY: 0.0220219, AVERAGE_POPULARITY: 0.7784514, DIVERSITY_MEAN_INTER_LIST: 0.4046388, DIVERSITY_HERFINDAHL: 0.9404572, COVERAGE_ITEM: 0.0113314, COVERAGE_ITEM_CORRECT: 0.0105588, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.6430464, DIVERSITY_GINI: 0.0042263, SHANNON_ENTROPY: 4.2691368, RATIO_DIVERSITY_HERFINDAHL: 0.9412255, RATIO_DIVERSITY_GINI: 0.0120873, RATIO_SHANNON_ENTROPY: 0.3948199, RATIO_AVERAGE_POPULARITY: 2.8182721, RATIO_NOVELTY: 0.0746187, \n",
      "\n",
      "\n",
      "SearchSingleCase: Saving model in result_experiments/TopPopRecommender\n",
      "\n",
      "TopPopRecommender: Saving model in file 'result_experiments/TopPopRecommender_best_model'\n",
      "TopPopRecommender: Saving complete\n",
      "SearchSingleCase: Search complete. Best config is 0: {}\n",
      "\n",
      "TopPopRecommender: URM Detected 212 ( 5.5%) items with no interactions.\n",
      "SearchSingleCase: Evaluation with constructor data for final test. Using best config: {}\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 2.92 sec. Users per second: 2068\n",
      "SearchSingleCase: Best config evaluated with evaluator_test with constructor data for final test. Config: {} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.1823393, PRECISION_RECALL_MIN_DEN: 0.1921090, RECALL: 0.0680471, MAP: 0.1117371, MAP_MIN_DEN: 0.1156208, MRR: 0.3754170, NDCG: 0.1554023, F1: 0.0991081, HIT_RATE: 0.6648443, ARHR_ALL_HITS: 0.6164049, NOVELTY: 0.0220897, AVERAGE_POPULARITY: 0.7596891, DIVERSITY_MEAN_INTER_LIST: 0.4950965, DIVERSITY_HERFINDAHL: 0.9495014, COVERAGE_ITEM: 0.0193150, COVERAGE_ITEM_CORRECT: 0.0190574, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.6644040, DIVERSITY_GINI: 0.0050102, SHANNON_ENTROPY: 4.5866876, RATIO_DIVERSITY_HERFINDAHL: 0.9502780, RATIO_DIVERSITY_GINI: 0.0143283, RATIO_SHANNON_ENTROPY: 0.4241895, RATIO_AVERAGE_POPULARITY: 2.7655196, RATIO_NOVELTY: 0.0598790, \n",
      "\n",
      "\n",
      "SearchSingleCase: Saving model in result_experiments/TopPopRecommender\n",
      "\n",
      "TopPopRecommender: Saving model in file 'result_experiments/TopPopRecommender_best_model_last'\n",
      "TopPopRecommender: Saving complete\n"
     ]
    }
   ],
   "source": [
    "from HyperparameterTuning.run_hyperparameter_search import runHyperparameterSearch_Collaborative, runHyperparameterSearch_Content\n",
    "from Recommenders.NonPersonalizedRecommender import TopPop, Random\n",
    "from Recommenders.GraphBased.P3alphaRecommender import P3alphaRecommender\n",
    "\n",
    "recommender_class = TopPop\n",
    "\n",
    "runHyperparameterSearch_Collaborative(recommender_class,\n",
    "       URM_train = URM_train,\n",
    "       URM_train_last_test = URM_train_validation,\n",
    "       metric_to_optimize = metric_to_optimize,\n",
    "       cutoff_to_optimize = cutoff_to_optimize,\n",
    "       evaluator_validation_earlystopping = evaluator_validation,\n",
    "       evaluator_validation = evaluator_validation,\n",
    "       evaluator_test = evaluator_test,\n",
    "       output_folder_path = output_folder_path,\n",
    "       parallelizeKNN = True,\n",
    "       allow_weighting = True,\n",
    "       resume_from_saved = True,\n",
    "       save_model = \"best\",\n",
    "       similarity_type_list = ['cosine', 'jaccard', \"asymmetric\", \"dice\", \"tversky\"],\n",
    "       n_cases = n_cases,\n",
    "       n_random_starts = n_random_starts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The process is similar for content-based algorithms.\n",
    "* Add the ICM data and its name (it will be used for file names)\n",
    "* Remove the evaluator for earlystopping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SearchBayesianSkopt: Resuming 'ItemKNNCBFRecommender_ICM_genres_cosine' Failed, no such file exists.\n",
      "\n",
      "Iteration No: 1 started. Evaluating function at random point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 454, 'shrink': 169, 'similarity': 'cosine', 'normalize': True, 'feature_weighting': 'BM25'}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "SearchBayesianSkopt: Resuming 'ItemKNNCBFRecommender_ICM_genres_jaccard' Failed, no such file exists.\n",
      "\n",
      "Iteration No: 1 started. Evaluating function at random point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 927, 'shrink': 987, 'similarity': 'jaccard', 'normalize': True}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "SearchBayesianSkopt: Resuming 'ItemKNNCBFRecommender_ICM_genres_dice' Failed, no such file exists.\n",
      "\n",
      "Iteration No: 1 started. Evaluating function at random point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 750, 'shrink': 926, 'similarity': 'dice', 'normalize': True}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "SearchBayesianSkopt: Resuming 'ItemKNNCBFRecommender_ICM_genres_tversky' Failed, no such file exists.\n",
      "Similarity column 3883 (100.0%), 9987.15 column/sec. Elapsed time 0.39 sec\n",
      "\n",
      "Iteration No: 1 started. Evaluating function at random point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 33, 'shrink': 878, 'similarity': 'tversky', 'normalize': True, 'tversky_alpha': 0.7608104848060062, 'tversky_beta': 1.4715251858289065}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 8169.23 column/sec. Elapsed time 0.48 sec\n",
      "Similarity column 3883 (100.0%), 9086.44 column/sec. Elapsed time 0.43 sec\n",
      "Similarity column 3883 (100.0%), 11719.58 column/sec. Elapsed time 0.33 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.22 sec. Users per second: 1428\n",
      "SearchBayesianSkopt: New best config found. Config 0: {'topK': 454, 'shrink': 169, 'similarity': 'cosine', 'normalize': True, 'feature_weighting': 'BM25'} - results: PRECISION: 0.0465471, PRECISION_RECALL_MIN_DEN: 0.0488861, RECALL: 0.0180792, MAP: 0.0192035, MAP_MIN_DEN: 0.0201235, MRR: 0.1202688, NDCG: 0.0302410, F1: 0.0260431, HIT_RATE: 0.2983068, ARHR_ALL_HITS: 0.1505461, NOVELTY: 0.0311873, AVERAGE_POPULARITY: 0.1258822, DIVERSITY_MEAN_INTER_LIST: 0.9749864, DIVERSITY_HERFINDAHL: 0.9974825, COVERAGE_ITEM: 0.4375483, COVERAGE_ITEM_CORRECT: 0.1295390, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.2975166, DIVERSITY_GINI: 0.1221961, SHANNON_ENTROPY: 9.2890479, RATIO_DIVERSITY_HERFINDAHL: 0.9982974, RATIO_DIVERSITY_GINI: 0.3494848, RATIO_SHANNON_ENTROPY: 0.8590732, RATIO_AVERAGE_POPULARITY: 0.4557385, RATIO_NOVELTY: 0.1056744, \n",
      "\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.39 sec. Users per second: 1371\n",
      "SearchBayesianSkopt: New best config found. Config 0: {'topK': 33, 'shrink': 878, 'similarity': 'tversky', 'normalize': True, 'tversky_alpha': 0.7608104848060062, 'tversky_beta': 1.4715251858289065} - results: PRECISION: 0.0503486, PRECISION_RECALL_MIN_DEN: 0.0540731, RECALL: 0.0220650, MAP: 0.0201109, MAP_MIN_DEN: 0.0213148, MRR: 0.1271754, NDCG: 0.0322006, F1: 0.0306832, HIT_RATE: 0.3242032, ARHR_ALL_HITS: 0.1581623, NOVELTY: 0.0298036, AVERAGE_POPULARITY: 0.1756040, DIVERSITY_MEAN_INTER_LIST: 0.9615407, DIVERSITY_HERFINDAHL: 0.9961381, COVERAGE_ITEM: 0.4316250, COVERAGE_ITEM_CORRECT: 0.1022405, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3233444, DIVERSITY_GINI: 0.0939388, SHANNON_ENTROPY: 8.8267257, RATIO_DIVERSITY_HERFINDAHL: 0.9969519, RATIO_DIVERSITY_GINI: 0.2686682, RATIO_SHANNON_ENTROPY: 0.8163165, RATIO_AVERAGE_POPULARITY: 0.6357492, RATIO_NOVELTY: 0.1009861, \n",
      "\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.32 sec. Users per second: 1133\n",
      "SearchBayesianSkopt: New best config found. Config 0: {'topK': 927, 'shrink': 987, 'similarity': 'jaccard', 'normalize': True} - results: PRECISION: 0.0548141, PRECISION_RECALL_MIN_DEN: 0.0591055, RECALL: 0.0245421, MAP: 0.0219297, MAP_MIN_DEN: 0.0233306, MRR: 0.1320240, NDCG: 0.0352106, F1: 0.0339041, HIT_RATE: 0.3441235, ARHR_ALL_HITS: 0.1688202, NOVELTY: 0.0290488, AVERAGE_POPULARITY: 0.2016613, DIVERSITY_MEAN_INTER_LIST: 0.9082928, DIVERSITY_HERFINDAHL: 0.9908142, COVERAGE_ITEM: 0.2459439, COVERAGE_ITEM_CORRECT: 0.0764873, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3432119, DIVERSITY_GINI: 0.0423212, SHANNON_ENTROPY: 7.6652819, RATIO_DIVERSITY_HERFINDAHL: 0.9916237, RATIO_DIVERSITY_GINI: 0.1210401, RATIO_SHANNON_ENTROPY: 0.7089035, RATIO_AVERAGE_POPULARITY: 0.7300859, RATIO_NOVELTY: 0.0984284, \n",
      "\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.52 sec. Users per second: 1091\n",
      "SearchBayesianSkopt: New best config found. Config 0: {'topK': 750, 'shrink': 926, 'similarity': 'dice', 'normalize': True} - results: PRECISION: 0.0552955, PRECISION_RECALL_MIN_DEN: 0.0599494, RECALL: 0.0251147, MAP: 0.0220589, MAP_MIN_DEN: 0.0235834, MRR: 0.1297941, NDCG: 0.0362334, F1: 0.0345412, HIT_RATE: 0.3388114, ARHR_ALL_HITS: 0.1675834, NOVELTY: 0.0292010, AVERAGE_POPULARITY: 0.1970266, DIVERSITY_MEAN_INTER_LIST: 0.9081640, DIVERSITY_HERFINDAHL: 0.9908013, COVERAGE_ITEM: 0.2485192, COVERAGE_ITEM_CORRECT: 0.0749421, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3379139, DIVERSITY_GINI: 0.0403583, SHANNON_ENTROPY: 7.6092254, RATIO_DIVERSITY_HERFINDAHL: 0.9916108, RATIO_DIVERSITY_GINI: 0.1154261, RATIO_SHANNON_ENTROPY: 0.7037192, RATIO_AVERAGE_POPULARITY: 0.7133068, RATIO_NOVELTY: 0.0989440, \n",
      "\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 4.20 sec. Users per second: 1438\n",
      "SearchBayesianSkopt: Config evaluated with evaluator_test. Config: {'topK': 454, 'shrink': 169, 'similarity': 'cosine', 'normalize': True, 'feature_weighting': 'BM25'} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.0556991, PRECISION_RECALL_MIN_DEN: 0.0575504, RECALL: 0.0183909, MAP: 0.0233206, MAP_MIN_DEN: 0.0238810, MRR: 0.1388489, NDCG: 0.0338357, F1: 0.0276517, HIT_RATE: 0.3421140, ARHR_ALL_HITS: 0.1776638, NOVELTY: 0.0311887, AVERAGE_POPULARITY: 0.1258597, DIVERSITY_MEAN_INTER_LIST: 0.9750136, DIVERSITY_HERFINDAHL: 0.9974852, COVERAGE_ITEM: 0.4375483, COVERAGE_ITEM_CORRECT: 0.1439609, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.3418874, DIVERSITY_GINI: 0.1222896, SHANNON_ENTROPY: 9.2902543, RATIO_DIVERSITY_HERFINDAHL: 0.9983001, RATIO_DIVERSITY_GINI: 0.3497523, RATIO_SHANNON_ENTROPY: 0.8591848, RATIO_AVERAGE_POPULARITY: 0.4556569, RATIO_NOVELTY: 0.1056792, \n",
      "\n",
      "\n",
      "SearchBayesianSkopt: Saving model in result_experiments/ItemKNNCBFRecommender_ICM_genres_cosine\n",
      "\n",
      "ItemKNNCBFRecommender: Saving model in file 'result_experiments/ItemKNNCBFRecommender_ICM_genres_cosine_best_model'\n",
      "ItemKNNCBFRecommender: Saving complete\n",
      "Iteration No: 1 ended. Evaluation done at random point.EvaluatorHoldout: Processed 6036 (100.0%) in 4.40 sec. Users per second: 1372\n",
      "\n",
      "Time taken: 9.7457\n",
      "Function value obtained: -0.0192\n",
      "Current minimum: -0.0192\n",
      "Iteration No: 2 started. Evaluating function at random point.\n",
      "SearchBayesianSkopt: Config evaluated with evaluator_test. Config: {'topK': 33, 'shrink': 878, 'similarity': 'tversky', 'normalize': True, 'tversky_alpha': 0.7608104848060062, 'tversky_beta': 1.4715251858289065} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.0663685, PRECISION_RECALL_MIN_DEN: 0.0693178, RECALL: 0.0233792, MAP: 0.0276472, MAP_MIN_DEN: 0.0286242, MRR: 0.1569787, NDCG: 0.0399850, F1: 0.0345779, HIT_RATE: 0.3858516, ARHR_ALL_HITS: 0.2050564, NOVELTY: 0.0298042, AVERAGE_POPULARITY: 0.1755816, DIVERSITY_MEAN_INTER_LIST: 0.9615541, DIVERSITY_HERFINDAHL: 0.9961395, COVERAGE_ITEM: 0.4316250, COVERAGE_ITEM_CORRECT: 0.1146021, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.3855960, DIVERSITY_GINI: 0.0939482, SHANNON_ENTROPY: 8.8270112, RATIO_DIVERSITY_HERFINDAHL: 0.9969533, RATIO_DIVERSITY_GINI: 0.2686950, RATIO_SHANNON_ENTROPY: 0.8163429, RATIO_AVERAGE_POPULARITY: 0.6356682, RATIO_NOVELTY: 0.1009880, \n",
      "\n",
      "\n",
      "SearchBayesianSkopt: Testing config: {'topK': 605, 'shrink': 807, 'similarity': 'cosine', 'normalize': True, 'feature_weighting': 'none'}\n",
      "SearchBayesianSkopt: Saving model in result_experiments/ItemKNNCBFRecommender_ICM_genres_tversky\n",
      "\n",
      "ItemKNNCBFRecommender: Saving model in file 'result_experiments/ItemKNNCBFRecommender_ICM_genres_tversky_best_model'\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "ItemKNNCBFRecommender: Saving complete\n",
      "Iteration No: 1 ended. Evaluation done at random point.\n",
      "Time taken: 9.6693\n",
      "Function value obtained: -0.0201\n",
      "Current minimum: -0.0201\n",
      "Iteration No: 2 started. Evaluating function at random point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 347, 'shrink': 379, 'similarity': 'tversky', 'normalize': True, 'tversky_alpha': 0.8564596806756939, 'tversky_beta': 0.09091909290431067}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 6774.56 column/sec. Elapsed time 0.57 sec\n",
      "Similarity column 3883 (100.0%), 17871.11 column/sec. Elapsed time 0.22 sec\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 5.28 sec. Users per second: 1143\n",
      "SearchBayesianSkopt: Config evaluated with evaluator_test. Config: {'topK': 750, 'shrink': 926, 'similarity': 'dice', 'normalize': True} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.0655070, PRECISION_RECALL_MIN_DEN: 0.0684475, RECALL: 0.0236318, MAP: 0.0273778, MAP_MIN_DEN: 0.0282739, MRR: 0.1510087, NDCG: 0.0408456, F1: 0.0347335, HIT_RATE: 0.3769052, ARHR_ALL_HITS: 0.1998998, NOVELTY: 0.0291988, AVERAGE_POPULARITY: 0.1970941, DIVERSITY_MEAN_INTER_LIST: 0.9082098, DIVERSITY_HERFINDAHL: 0.9908059, COVERAGE_ITEM: 0.2485192, COVERAGE_ITEM_CORRECT: 0.0780324, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.3766556, DIVERSITY_GINI: 0.0404100, SHANNON_ENTROPY: 7.6109465, RATIO_DIVERSITY_HERFINDAHL: 0.9916154, RATIO_DIVERSITY_GINI: 0.1155740, RATIO_SHANNON_ENTROPY: 0.7038784, RATIO_AVERAGE_POPULARITY: 0.7135510, RATIO_NOVELTY: 0.0989366, \n",
      "\n",
      "\n",
      "SearchBayesianSkopt: Saving model in result_experiments/ItemKNNCBFRecommender_ICM_genres_dice\n",
      "\n",
      "ItemKNNCBFRecommender: Saving model in file 'result_experiments/ItemKNNCBFRecommender_ICM_genres_dice_best_model'\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 5.68 sec. Users per second: 1062\n",
      "SearchBayesianSkopt: Config evaluated with evaluator_test. Config: {'topK': 927, 'shrink': 987, 'similarity': 'jaccard', 'normalize': True} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.0666170, PRECISION_RECALL_MIN_DEN: 0.0693224, RECALL: 0.0230769, MAP: 0.0279215, MAP_MIN_DEN: 0.0288067, MRR: 0.1547764, NDCG: 0.0398138, F1: 0.0342791, HIT_RATE: 0.3840292, ARHR_ALL_HITS: 0.2048934, NOVELTY: 0.0290487, AVERAGE_POPULARITY: 0.2016761, DIVERSITY_MEAN_INTER_LIST: 0.9083925, DIVERSITY_HERFINDAHL: 0.9908242, COVERAGE_ITEM: 0.2469740, COVERAGE_ITEM_CORRECT: 0.0800927, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.3837748, DIVERSITY_GINI: 0.0424003, SHANNON_ENTROPY: 7.6676174, RATIO_DIVERSITY_HERFINDAHL: 0.9916337, RATIO_DIVERSITY_GINI: 0.1212663, RATIO_SHANNON_ENTROPY: 0.7091194, RATIO_AVERAGE_POPULARITY: 0.7301394, RATIO_NOVELTY: 0.0984279, \n",
      "\n",
      "\n",
      "SearchBayesianSkopt: Saving model in result_experiments/ItemKNNCBFRecommender_ICM_genres_jaccard\n",
      "\n",
      "ItemKNNCBFRecommender: Saving model in file 'result_experiments/ItemKNNCBFRecommender_ICM_genres_jaccard_best_model'\n",
      "ItemKNNCBFRecommender: Saving complete\n",
      "Iteration No: 1 ended. Evaluation done at random point.\n",
      "Time taken: 12.2250\n",
      "Function value obtained: -0.0221\n",
      "Current minimum: -0.0221\n",
      "Iteration No: 2 started. Evaluating function at random point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 749, 'shrink': 191, 'similarity': 'dice', 'normalize': False}\n",
      "ItemKNNCBFRecommender: Saving completeItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "\n",
      "Iteration No: 1 ended. Evaluation done at random point.\n",
      "Time taken: 12.9111\n",
      "Function value obtained: -0.0219\n",
      "Current minimum: -0.0219\n",
      "Iteration No: 2 started. Evaluating function at random point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 789, 'shrink': 207, 'similarity': 'jaccard', 'normalize': False}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 9071.96 column/sec. Elapsed time 0.43 sec\n",
      "Similarity column 3883 (100.0%), 9183.65 column/sec. Elapsed time 0.42 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.16 sec. Users per second: 1449\n",
      "SearchBayesianSkopt: New best config found. Config 1: {'topK': 347, 'shrink': 379, 'similarity': 'tversky', 'normalize': True, 'tversky_alpha': 0.8564596806756939, 'tversky_beta': 0.09091909290431067} - results: PRECISION: 0.0599436, PRECISION_RECALL_MIN_DEN: 0.0650443, RECALL: 0.0272549, MAP: 0.0242426, MAP_MIN_DEN: 0.0258937, MRR: 0.1412799, NDCG: 0.0388804, F1: 0.0374721, HIT_RATE: 0.3650398, ARHR_ALL_HITS: 0.1833293, NOVELTY: 0.0282844, AVERAGE_POPULARITY: 0.2122619, DIVERSITY_MEAN_INTER_LIST: 0.9016601, DIVERSITY_HERFINDAHL: 0.9901510, COVERAGE_ITEM: 0.3036312, COVERAGE_ITEM_CORRECT: 0.0803502, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3640728, DIVERSITY_GINI: 0.0467863, SHANNON_ENTROPY: 7.7405016, RATIO_DIVERSITY_HERFINDAHL: 0.9909600, RATIO_DIVERSITY_GINI: 0.1338105, RATIO_SHANNON_ENTROPY: 0.7158600, RATIO_AVERAGE_POPULARITY: 0.7684641, RATIO_NOVELTY: 0.0958382, \n",
      "\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.08 sec. Users per second: 1185\n",
      "SearchBayesianSkopt: New best config found. Config 1: {'topK': 605, 'shrink': 807, 'similarity': 'cosine', 'normalize': True, 'feature_weighting': 'none'} - results: PRECISION: 0.0545485, PRECISION_RECALL_MIN_DEN: 0.0590429, RECALL: 0.0247167, MAP: 0.0217823, MAP_MIN_DEN: 0.0232770, MRR: 0.1315512, NDCG: 0.0358779, F1: 0.0340189, HIT_RATE: 0.3414675, ARHR_ALL_HITS: 0.1675772, NOVELTY: 0.0290028, AVERAGE_POPULARITY: 0.1970485, DIVERSITY_MEAN_INTER_LIST: 0.8961830, DIVERSITY_HERFINDAHL: 0.9896034, COVERAGE_ITEM: 0.2634561, COVERAGE_ITEM_CORRECT: 0.0754571, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3405629, DIVERSITY_GINI: 0.0383337, SHANNON_ENTROPY: 7.4860608, RATIO_DIVERSITY_HERFINDAHL: 0.9904119, RATIO_DIVERSITY_GINI: 0.1096357, RATIO_SHANNON_ENTROPY: 0.6923287, RATIO_AVERAGE_POPULARITY: 0.7133859, RATIO_NOVELTY: 0.0982725, \n",
      "\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.84 sec. Users per second: 1244\n",
      "SearchBayesianSkopt: Config 1 is suboptimal. Config: {'topK': 789, 'shrink': 207, 'similarity': 'jaccard', 'normalize': False} - results: PRECISION: 0.0542331, PRECISION_RECALL_MIN_DEN: 0.0585734, RECALL: 0.0244325, MAP: 0.0214121, MAP_MIN_DEN: 0.0229229, MRR: 0.1269021, NDCG: 0.0348917, F1: 0.0336882, HIT_RATE: 0.3361554, ARHR_ALL_HITS: 0.1632170, NOVELTY: 0.0292104, AVERAGE_POPULARITY: 0.1961664, DIVERSITY_MEAN_INTER_LIST: 0.9105736, DIVERSITY_HERFINDAHL: 0.9910422, COVERAGE_ITEM: 0.2400206, COVERAGE_ITEM_CORRECT: 0.0754571, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3352649, DIVERSITY_GINI: 0.0414981, SHANNON_ENTROPY: 7.6566016, RATIO_DIVERSITY_HERFINDAHL: 0.9918519, RATIO_DIVERSITY_GINI: 0.1186861, RATIO_SHANNON_ENTROPY: 0.7081007, RATIO_AVERAGE_POPULARITY: 0.7101926, RATIO_NOVELTY: 0.0989760, \n",
      "\n",
      "Iteration No: 2 ended. Evaluation done at random point.\n",
      "Time taken: 5.6067\n",
      "Function value obtained: -0.0214\n",
      "Current minimum: -0.0219\n",
      "Iteration No: 3 started. Evaluating function at random point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 945, 'shrink': 954, 'similarity': 'jaccard', 'normalize': False}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 9579.54 column/sec. Elapsed time 0.41 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 6.13 sec. Users per second: 983\n",
      "SearchBayesianSkopt: Config 1 is suboptimal. Config: {'topK': 749, 'shrink': 191, 'similarity': 'dice', 'normalize': False} - results: PRECISION: 0.0546813, PRECISION_RECALL_MIN_DEN: 0.0592707, RECALL: 0.0249019, MAP: 0.0217874, MAP_MIN_DEN: 0.0233107, MRR: 0.1295724, NDCG: 0.0356134, F1: 0.0342200, HIT_RATE: 0.3379814, ARHR_ALL_HITS: 0.1663098, NOVELTY: 0.0292336, AVERAGE_POPULARITY: 0.1957778, DIVERSITY_MEAN_INTER_LIST: 0.9094742, DIVERSITY_HERFINDAHL: 0.9909323, COVERAGE_ITEM: 0.2480041, COVERAGE_ITEM_CORRECT: 0.0764873, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3370861, DIVERSITY_GINI: 0.0407932, SHANNON_ENTROPY: 7.6269982, RATIO_DIVERSITY_HERFINDAHL: 0.9917419, RATIO_DIVERSITY_GINI: 0.1166700, RATIO_SHANNON_ENTROPY: 0.7053629, RATIO_AVERAGE_POPULARITY: 0.7087855, RATIO_NOVELTY: 0.0990545, \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Iteration No: 2 ended. Evaluation done at random point.\n",
      "Time taken: 6.8641\n",
      "Function value obtained: -0.0218\n",
      "Current minimum: -0.0221\n",
      "Iteration No: 3 started. Evaluating function at random point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 6, 'shrink': 608, 'similarity': 'dice', 'normalize': False}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 20947.92 column/sec. Elapsed time 0.19 sec\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 5.33 sec. Users per second: 1132\n",
      "SearchBayesianSkopt: Config evaluated with evaluator_test. Config: {'topK': 347, 'shrink': 379, 'similarity': 'tversky', 'normalize': True, 'tversky_alpha': 0.8564596806756939, 'tversky_beta': 0.09091909290431067} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.0723492, PRECISION_RECALL_MIN_DEN: 0.0751475, RECALL: 0.0251246, MAP: 0.0311188, MAP_MIN_DEN: 0.0319933, MRR: 0.1671991, NDCG: 0.0438749, F1: 0.0372971, HIT_RATE: 0.4037442, ARHR_ALL_HITS: 0.2239745, NOVELTY: 0.0282863, AVERAGE_POPULARITY: 0.2122193, DIVERSITY_MEAN_INTER_LIST: 0.9016580, DIVERSITY_HERFINDAHL: 0.9901509, COVERAGE_ITEM: 0.3036312, COVERAGE_ITEM_CORRECT: 0.0885913, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.4034768, DIVERSITY_GINI: 0.0468006, SHANNON_ENTROPY: 7.7408503, RATIO_DIVERSITY_HERFINDAHL: 0.9909598, RATIO_DIVERSITY_GINI: 0.1338512, RATIO_SHANNON_ENTROPY: 0.7158922, RATIO_AVERAGE_POPULARITY: 0.7683096, RATIO_NOVELTY: 0.0958447, \n",
      "\n",
      "\n",
      "SearchBayesianSkopt: Saving model in result_experiments/ItemKNNCBFRecommender_ICM_genres_tversky\n",
      "\n",
      "ItemKNNCBFRecommender: Saving model in file 'result_experiments/ItemKNNCBFRecommender_ICM_genres_tversky_best_model'\n",
      "ItemKNNCBFRecommender: Saving complete\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 4.91 sec. Users per second: 1228\n",
      "SearchBayesianSkopt: Config evaluated with evaluator_test. Config: {'topK': 605, 'shrink': 807, 'similarity': 'cosine', 'normalize': True, 'feature_weighting': 'none'} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.0650099, PRECISION_RECALL_MIN_DEN: 0.0680306, RECALL: 0.0236035, MAP: 0.0268142, MAP_MIN_DEN: 0.0277041, MRR: 0.1519913, NDCG: 0.0410864, F1: 0.0346327, HIT_RATE: 0.3848575, ARHR_ALL_HITS: 0.1988475, NOVELTY: 0.0290018, AVERAGE_POPULARITY: 0.1971117, DIVERSITY_MEAN_INTER_LIST: 0.8962370, DIVERSITY_HERFINDAHL: 0.9896088, COVERAGE_ITEM: 0.2644862, COVERAGE_ITEM_CORRECT: 0.0808653, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.3846026, DIVERSITY_GINI: 0.0384471, SHANNON_ENTROPY: 7.4890026, RATIO_DIVERSITY_HERFINDAHL: 0.9904173, RATIO_DIVERSITY_GINI: 0.1099600, RATIO_SHANNON_ENTROPY: 0.6926007, RATIO_AVERAGE_POPULARITY: 0.7136148, RATIO_NOVELTY: 0.0982692, \n",
      "\n",
      "\n",
      "SearchBayesianSkopt: Saving model in result_experiments/ItemKNNCBFRecommender_ICM_genres_cosine\n",
      "\n",
      "ItemKNNCBFRecommender: Saving model in file 'result_experiments/ItemKNNCBFRecommender_ICM_genres_cosine_best_model'\n",
      "Iteration No: 2 ended. Evaluation done at random point.\n",
      "Time taken: 10.2983\n",
      "Function value obtained: -0.0242\n",
      "Current minimum: -0.0242\n",
      "Iteration No: 3 started. Evaluating function at random point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 681, 'shrink': 926, 'similarity': 'tversky', 'normalize': True, 'tversky_alpha': 0.22099231365753497, 'tversky_beta': 1.7661510026858884}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 9355.02 column/sec. Elapsed time 0.42 sec\n",
      "ItemKNNCBFRecommender: Saving complete\n",
      "Iteration No: 2 ended. Evaluation done at random point.\n",
      "Time taken: 11.4060\n",
      "Function value obtained: -0.0218\n",
      "Current minimum: -0.0218\n",
      "Iteration No: 3 started. Evaluating function at random point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 355, 'shrink': 176, 'similarity': 'cosine', 'normalize': False, 'feature_weighting': 'TF-IDF'}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 11675.00 column/sec. Elapsed time 0.33 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.60 sec. Users per second: 1672\n",
      "SearchBayesianSkopt: Config 2 is suboptimal. Config: {'topK': 6, 'shrink': 608, 'similarity': 'dice', 'normalize': False} - results: PRECISION: 0.0481408, PRECISION_RECALL_MIN_DEN: 0.0525982, RECALL: 0.0227087, MAP: 0.0193374, MAP_MIN_DEN: 0.0208013, MRR: 0.1272041, NDCG: 0.0341993, F1: 0.0308602, HIT_RATE: 0.3261952, ARHR_ALL_HITS: 0.1553382, NOVELTY: 0.0294362, AVERAGE_POPULARITY: 0.1849578, DIVERSITY_MEAN_INTER_LIST: 0.9538457, DIVERSITY_HERFINDAHL: 0.9953687, COVERAGE_ITEM: 0.3674994, COVERAGE_ITEM_CORRECT: 0.1076487, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3253311, DIVERSITY_GINI: 0.0771679, SHANNON_ENTROPY: 8.5593144, RATIO_DIVERSITY_HERFINDAHL: 0.9961819, RATIO_DIVERSITY_GINI: 0.2207027, RATIO_SHANNON_ENTROPY: 0.7915857, RATIO_AVERAGE_POPULARITY: 0.6696134, RATIO_NOVELTY: 0.0997410, \n",
      "\n",
      "Iteration No: 3 ended. Evaluation done at random point.\n",
      "Time taken: 4.4967\n",
      "Function value obtained: -0.0193\n",
      "Current minimum: -0.0221\n",
      "Iteration No: 4 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 792, 'shrink': 1000, 'similarity': 'dice', 'normalize': True}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 15241.55 column/sec. Elapsed time 0.25 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.61 sec. Users per second: 1074\n",
      "SearchBayesianSkopt: Config 2 is suboptimal. Config: {'topK': 945, 'shrink': 954, 'similarity': 'jaccard', 'normalize': False} - results: PRECISION: 0.0545485, PRECISION_RECALL_MIN_DEN: 0.0586532, RECALL: 0.0240721, MAP: 0.0217651, MAP_MIN_DEN: 0.0231384, MRR: 0.1307500, NDCG: 0.0345164, F1: 0.0334034, HIT_RATE: 0.3411355, ARHR_ALL_HITS: 0.1673417, NOVELTY: 0.0289641, AVERAGE_POPULARITY: 0.2019670, DIVERSITY_MEAN_INTER_LIST: 0.9085758, DIVERSITY_HERFINDAHL: 0.9908425, COVERAGE_ITEM: 0.2456863, COVERAGE_ITEM_CORRECT: 0.0749421, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3402318, DIVERSITY_GINI: 0.0424794, SHANNON_ENTROPY: 7.6703550, RATIO_DIVERSITY_HERFINDAHL: 0.9916520, RATIO_DIVERSITY_GINI: 0.1214926, RATIO_SHANNON_ENTROPY: 0.7093726, RATIO_AVERAGE_POPULARITY: 0.7311928, RATIO_NOVELTY: 0.0981416, \n",
      "\n",
      "Iteration No: 3 ended. Evaluation done at random point.\n",
      "Time taken: 6.6030\n",
      "Function value obtained: -0.0218\n",
      "Current minimum: -0.0219\n",
      "Iteration No: 4 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 5, 'shrink': 1000, 'similarity': 'jaccard', 'normalize': True}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 17408.09 column/sec. Elapsed time 0.22 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.32 sec. Users per second: 1133\n",
      "SearchBayesianSkopt: Config 2 is suboptimal. Config: {'topK': 681, 'shrink': 926, 'similarity': 'tversky', 'normalize': True, 'tversky_alpha': 0.22099231365753497, 'tversky_beta': 1.7661510026858884} - results: PRECISION: 0.0533699, PRECISION_RECALL_MIN_DEN: 0.0578951, RECALL: 0.0245501, MAP: 0.0215469, MAP_MIN_DEN: 0.0231153, MRR: 0.1302994, NDCG: 0.0356513, F1: 0.0336303, HIT_RATE: 0.3341633, ARHR_ALL_HITS: 0.1657022, NOVELTY: 0.0292705, AVERAGE_POPULARITY: 0.1935514, DIVERSITY_MEAN_INTER_LIST: 0.9006011, DIVERSITY_HERFINDAHL: 0.9900452, COVERAGE_ITEM: 0.2547000, COVERAGE_ITEM_CORRECT: 0.0770023, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3332781, DIVERSITY_GINI: 0.0385299, SHANNON_ENTROPY: 7.5183687, RATIO_DIVERSITY_HERFINDAHL: 0.9908540, RATIO_DIVERSITY_GINI: 0.1101968, RATIO_SHANNON_ENTROPY: 0.6953166, RATIO_AVERAGE_POPULARITY: 0.7007252, RATIO_NOVELTY: 0.0991795, \n",
      "\n",
      "Iteration No: 3 ended. Evaluation done at random point.\n",
      "Time taken: 6.9314\n",
      "Function value obtained: -0.0215\n",
      "Current minimum: -0.0242\n",
      "Iteration No: 4 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 981, 'shrink': 272, 'similarity': 'tversky', 'normalize': True, 'tversky_alpha': 1.3634708469496153, 'tversky_beta': 0.4058944254393818}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.88 sec. Users per second: 1025\n",
      "SearchBayesianSkopt: New best config found. Config 2: {'topK': 355, 'shrink': 176, 'similarity': 'cosine', 'normalize': False, 'feature_weighting': 'TF-IDF'} - results: PRECISION: 0.0606242, PRECISION_RECALL_MIN_DEN: 0.0654620, RECALL: 0.0271190, MAP: 0.0248456, MAP_MIN_DEN: 0.0263914, MRR: 0.1475194, NDCG: 0.0412371, F1: 0.0374746, HIT_RATE: 0.3748340, ARHR_ALL_HITS: 0.1894949, NOVELTY: 0.0283543, AVERAGE_POPULARITY: 0.2234652, DIVERSITY_MEAN_INTER_LIST: 0.8944077, DIVERSITY_HERFINDAHL: 0.9894259, COVERAGE_ITEM: 0.1898017, COVERAGE_ITEM_CORRECT: 0.0754571, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3738411, DIVERSITY_GINI: 0.0359121, SHANNON_ENTROPY: 7.4249245, RATIO_DIVERSITY_HERFINDAHL: 0.9902342, RATIO_DIVERSITY_GINI: 0.1027099, RATIO_SHANNON_ENTROPY: 0.6866746, RATIO_AVERAGE_POPULARITY: 0.8090238, RATIO_NOVELTY: 0.0960753, \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Similarity column 3883 (100.0%), 14454.75 column/sec. Elapsed time 0.27 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.82 sec. Users per second: 1576\n",
      "SearchBayesianSkopt: Config 3 is suboptimal. Config: {'topK': 5, 'shrink': 1000, 'similarity': 'jaccard', 'normalize': True} - results: PRECISION: 0.0497178, PRECISION_RECALL_MIN_DEN: 0.0540287, RECALL: 0.0230467, MAP: 0.0199906, MAP_MIN_DEN: 0.0214813, MRR: 0.1305180, NDCG: 0.0347948, F1: 0.0314943, HIT_RATE: 0.3343293, ARHR_ALL_HITS: 0.1601152, NOVELTY: 0.0292292, AVERAGE_POPULARITY: 0.1910602, DIVERSITY_MEAN_INTER_LIST: 0.9525647, DIVERSITY_HERFINDAHL: 0.9952407, COVERAGE_ITEM: 0.3757404, COVERAGE_ITEM_CORRECT: 0.1079063, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3334437, DIVERSITY_GINI: 0.0765684, SHANNON_ENTROPY: 8.5402188, RATIO_DIVERSITY_HERFINDAHL: 0.9960537, RATIO_DIVERSITY_GINI: 0.2189882, RATIO_SHANNON_ENTROPY: 0.7898197, RATIO_AVERAGE_POPULARITY: 0.6917064, RATIO_NOVELTY: 0.0990397, \n",
      "\n",
      "Iteration No: 4 ended. Search finished for the next optimal point.\n",
      "Time taken: 4.8543\n",
      "Function value obtained: -0.0200\n",
      "Current minimum: -0.0219\n",
      "Iteration No: 5 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 735, 'shrink': 92, 'similarity': 'jaccard', 'normalize': False}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 6.04 sec. Users per second: 998\n",
      "SearchBayesianSkopt: Config 3 is suboptimal. Config: {'topK': 792, 'shrink': 1000, 'similarity': 'dice', 'normalize': True} - results: PRECISION: 0.0542829, PRECISION_RECALL_MIN_DEN: 0.0584543, RECALL: 0.0242720, MAP: 0.0214516, MAP_MIN_DEN: 0.0229371, MRR: 0.1276176, NDCG: 0.0348781, F1: 0.0335449, HIT_RATE: 0.3381474, ARHR_ALL_HITS: 0.1637692, NOVELTY: 0.0292056, AVERAGE_POPULARITY: 0.1966866, DIVERSITY_MEAN_INTER_LIST: 0.9102080, DIVERSITY_HERFINDAHL: 0.9910057, COVERAGE_ITEM: 0.2356425, COVERAGE_ITEM_CORRECT: 0.0757147, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3372517, DIVERSITY_GINI: 0.0412863, SHANNON_ENTROPY: 7.6487916, RATIO_DIVERSITY_HERFINDAHL: 0.9918153, RATIO_DIVERSITY_GINI: 0.1180802, RATIO_SHANNON_ENTROPY: 0.7073784, RATIO_AVERAGE_POPULARITY: 0.7120758, RATIO_NOVELTY: 0.0989596, \n",
      "\n",
      "Similarity column 3883 (100.0%), 8584.48 column/sec. Elapsed time 0.45 sec\n",
      "Iteration No: 4 ended. Search finished for the next optimal point.\n",
      "Time taken: 7.4211\n",
      "Function value obtained: -0.0215\n",
      "Current minimum: -0.0221\n",
      "Iteration No: 5 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 513, 'shrink': 992, 'similarity': 'dice', 'normalize': True}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 4.48 sec. Users per second: 1348\n",
      "Similarity column 3883 (100.0%), 9848.62 column/sec. Elapsed time 0.39 secSearchBayesianSkopt: Config evaluated with evaluator_test. Config: {'topK': 355, 'shrink': 176, 'similarity': 'cosine', 'normalize': False, 'feature_weighting': 'TF-IDF'} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.0751657, PRECISION_RECALL_MIN_DEN: 0.0781466, RECALL: 0.0269322, MAP: 0.0328370, MAP_MIN_DEN: 0.0338437, MRR: 0.1828481, NDCG: 0.0480270, F1: 0.0396557, HIT_RATE: 0.4324056, ARHR_ALL_HITS: 0.2418377, NOVELTY: 0.0283548, AVERAGE_POPULARITY: 0.2234509, DIVERSITY_MEAN_INTER_LIST: 0.8944616, DIVERSITY_HERFINDAHL: 0.9894313, COVERAGE_ITEM: 0.1898017, COVERAGE_ITEM_CORRECT: 0.0813804, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.4321192, DIVERSITY_GINI: 0.0359416, SHANNON_ENTROPY: 7.4259199, RATIO_DIVERSITY_HERFINDAHL: 0.9902397, RATIO_DIVERSITY_GINI: 0.1027940, RATIO_SHANNON_ENTROPY: 0.6867667, RATIO_AVERAGE_POPULARITY: 0.8089721, RATIO_NOVELTY: 0.0960768, \n",
      "\n",
      "\n",
      "\n",
      "SearchBayesianSkopt: Saving model in result_experiments/ItemKNNCBFRecommender_ICM_genres_cosine\n",
      "\n",
      "ItemKNNCBFRecommender: Saving model in file 'result_experiments/ItemKNNCBFRecommender_ICM_genres_cosine_best_model'\n",
      "ItemKNNCBFRecommender: Saving complete\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.14 sec. Users per second: 1172\n",
      "SearchBayesianSkopt: Config 3 is suboptimal. Config: {'topK': 981, 'shrink': 272, 'similarity': 'tversky', 'normalize': True, 'tversky_alpha': 1.3634708469496153, 'tversky_beta': 0.4058944254393818} - results: PRECISION: 0.0552955, PRECISION_RECALL_MIN_DEN: 0.0592452, RECALL: 0.0240721, MAP: 0.0218278, MAP_MIN_DEN: 0.0231694, MRR: 0.1304056, NDCG: 0.0347913, F1: 0.0335421, HIT_RATE: 0.3461155, ARHR_ALL_HITS: 0.1672440, NOVELTY: 0.0289746, AVERAGE_POPULARITY: 0.2014496, DIVERSITY_MEAN_INTER_LIST: 0.9098139, DIVERSITY_HERFINDAHL: 0.9909663, COVERAGE_ITEM: 0.2503219, COVERAGE_ITEM_CORRECT: 0.0782900, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3451987, DIVERSITY_GINI: 0.0441167, SHANNON_ENTROPY: 7.7187730, RATIO_DIVERSITY_HERFINDAHL: 0.9917759, RATIO_DIVERSITY_GINI: 0.1261752, RATIO_SHANNON_ENTROPY: 0.7138504, RATIO_AVERAGE_POPULARITY: 0.7293195, RATIO_NOVELTY: 0.0981769, \n",
      "\n",
      "Iteration No: 3 ended. Evaluation done at random point.\n",
      "Time taken: 11.9448\n",
      "Function value obtained: -0.0248\n",
      "Current minimum: -0.0248\n",
      "Iteration No: 4 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 236, 'shrink': 897, 'similarity': 'cosine', 'normalize': False, 'feature_weighting': 'TF-IDF'}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Iteration No: 4 ended. Search finished for the next optimal point.\n",
      "Time taken: 6.1256\n",
      "Function value obtained: -0.0218\n",
      "Current minimum: -0.0242\n",
      "Iteration No: 5 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 163, 'shrink': 414, 'similarity': 'tversky', 'normalize': True, 'tversky_alpha': 0.6695033591018915, 'tversky_beta': 0.6675348484565986}\n",
      "Similarity column 3883 (100.0%), 12672.20 column/sec. Elapsed time 0.31 sec\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 11908.78 column/sec. Elapsed time 0.33 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.27 sec. Users per second: 1143\n",
      "SearchBayesianSkopt: New best config found. Config 4: {'topK': 735, 'shrink': 92, 'similarity': 'jaccard', 'normalize': False} - results: PRECISION: 0.0539011, PRECISION_RECALL_MIN_DEN: 0.0583439, RECALL: 0.0244156, MAP: 0.0219594, MAP_MIN_DEN: 0.0234760, MRR: 0.1324468, NDCG: 0.0358900, F1: 0.0336078, HIT_RATE: 0.3354914, ARHR_ALL_HITS: 0.1687006, NOVELTY: 0.0293535, AVERAGE_POPULARITY: 0.1940465, DIVERSITY_MEAN_INTER_LIST: 0.9088274, DIVERSITY_HERFINDAHL: 0.9908677, COVERAGE_ITEM: 0.2464589, COVERAGE_ITEM_CORRECT: 0.0777749, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3346026, DIVERSITY_GINI: 0.0406334, SHANNON_ENTROPY: 7.6171534, RATIO_DIVERSITY_HERFINDAHL: 0.9916772, RATIO_DIVERSITY_GINI: 0.1162128, RATIO_SHANNON_ENTROPY: 0.7044524, RATIO_AVERAGE_POPULARITY: 0.7025177, RATIO_NOVELTY: 0.0994610, \n",
      "\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.98 sec. Users per second: 1210\n",
      "SearchBayesianSkopt: New best config found. Config 4: {'topK': 513, 'shrink': 992, 'similarity': 'dice', 'normalize': True} - results: PRECISION: 0.0557105, PRECISION_RECALL_MIN_DEN: 0.0603075, RECALL: 0.0252026, MAP: 0.0224027, MAP_MIN_DEN: 0.0240268, MRR: 0.1351250, NDCG: 0.0375778, F1: 0.0347052, HIT_RATE: 0.3471116, ARHR_ALL_HITS: 0.1723372, NOVELTY: 0.0288026, AVERAGE_POPULARITY: 0.2037959, DIVERSITY_MEAN_INTER_LIST: 0.8936065, DIVERSITY_HERFINDAHL: 0.9893458, COVERAGE_ITEM: 0.2611383, COVERAGE_ITEM_CORRECT: 0.0795776, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3461921, DIVERSITY_GINI: 0.0388285, SHANNON_ENTROPY: 7.4935914, RATIO_DIVERSITY_HERFINDAHL: 0.9901541, RATIO_DIVERSITY_GINI: 0.1110507, RATIO_SHANNON_ENTROPY: 0.6930251, RATIO_AVERAGE_POPULARITY: 0.7378138, RATIO_NOVELTY: 0.0975941, \n",
      "\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.09 sec. Users per second: 1472\n",
      "SearchBayesianSkopt: Config 4 is suboptimal. Config: {'topK': 163, 'shrink': 414, 'similarity': 'tversky', 'normalize': True, 'tversky_alpha': 0.6695033591018915, 'tversky_beta': 0.6675348484565986} - results: PRECISION: 0.0577689, PRECISION_RECALL_MIN_DEN: 0.0621432, RECALL: 0.0248402, MAP: 0.0231171, MAP_MIN_DEN: 0.0245389, MRR: 0.1361836, NDCG: 0.0359246, F1: 0.0347417, HIT_RATE: 0.3542497, ARHR_ALL_HITS: 0.1749467, NOVELTY: 0.0288271, AVERAGE_POPULARITY: 0.1995167, DIVERSITY_MEAN_INTER_LIST: 0.9305414, DIVERSITY_HERFINDAHL: 0.9930387, COVERAGE_ITEM: 0.4141128, COVERAGE_ITEM_CORRECT: 0.1104816, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3533113, DIVERSITY_GINI: 0.0839415, SHANNON_ENTROPY: 8.5063360, RATIO_DIVERSITY_HERFINDAHL: 0.9938500, RATIO_DIVERSITY_GINI: 0.2400754, RATIO_SHANNON_ENTROPY: 0.7866861, RATIO_AVERAGE_POPULARITY: 0.7223219, RATIO_NOVELTY: 0.0976772, \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Iteration No: 5 ended. Search finished for the next optimal point.\n",
      "Time taken: 4.8693\n",
      "Function value obtained: -0.0231\n",
      "Current minimum: -0.0242\n",
      "Iteration No: 6 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 365, 'shrink': 0, 'similarity': 'tversky', 'normalize': True, 'tversky_alpha': 0.9451994341143322, 'tversky_beta': 1.9034035449301236}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.04 sec. Users per second: 1195\n",
      "SearchBayesianSkopt: Config 3 is suboptimal. Config: {'topK': 236, 'shrink': 897, 'similarity': 'cosine', 'normalize': False, 'feature_weighting': 'TF-IDF'} - results: PRECISION: 0.0589475, PRECISION_RECALL_MIN_DEN: 0.0637218, RECALL: 0.0265087, MAP: 0.0242811, MAP_MIN_DEN: 0.0258535, MRR: 0.1472971, NDCG: 0.0402735, F1: 0.0365713, HIT_RATE: 0.3670319, ARHR_ALL_HITS: 0.1871194, NOVELTY: 0.0286932, AVERAGE_POPULARITY: 0.2084592, DIVERSITY_MEAN_INTER_LIST: 0.9175837, DIVERSITY_HERFINDAHL: 0.9917431, COVERAGE_ITEM: 0.2613958, COVERAGE_ITEM_CORRECT: 0.0818954, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3660596, DIVERSITY_GINI: 0.0448756, SHANNON_ENTROPY: 7.7558838, RATIO_DIVERSITY_HERFINDAHL: 0.9925534, RATIO_DIVERSITY_GINI: 0.1283457, RATIO_SHANNON_ENTROPY: 0.7172825, RATIO_AVERAGE_POPULARITY: 0.7546970, RATIO_NOVELTY: 0.0972234, \n",
      "\n",
      "Similarity column 3883 (100.0%), 11134.82 column/sec. Elapsed time 0.35 sec\n",
      "Iteration No: 4 ended. Search finished for the next optimal point.\n",
      "Time taken: 6.1603\n",
      "Function value obtained: -0.0243\n",
      "Current minimum: -0.0248\n",
      "Iteration No: 5 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 1000, 'shrink': 0, 'similarity': 'cosine', 'normalize': False, 'feature_weighting': 'none'}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 9323.15 column/sec. Elapsed time 0.42 sec\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 4.76 sec. Users per second: 1269\n",
      "SearchBayesianSkopt: Config evaluated with evaluator_test. Config: {'topK': 513, 'shrink': 992, 'similarity': 'dice', 'normalize': True} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.0669649, PRECISION_RECALL_MIN_DEN: 0.0698461, RECALL: 0.0240596, MAP: 0.0281851, MAP_MIN_DEN: 0.0291254, MRR: 0.1605903, NDCG: 0.0426305, F1: 0.0354003, HIT_RATE: 0.3952949, ARHR_ALL_HITS: 0.2094763, NOVELTY: 0.0288017, AVERAGE_POPULARITY: 0.2038084, DIVERSITY_MEAN_INTER_LIST: 0.8936655, DIVERSITY_HERFINDAHL: 0.9893517, COVERAGE_ITEM: 0.2616534, COVERAGE_ITEM_CORRECT: 0.0816379, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.3950331, DIVERSITY_GINI: 0.0389457, SHANNON_ENTROPY: 7.4966691, RATIO_DIVERSITY_HERFINDAHL: 0.9901600, RATIO_DIVERSITY_GINI: 0.1113861, RATIO_SHANNON_ENTROPY: 0.6933097, RATIO_AVERAGE_POPULARITY: 0.7378591, RATIO_NOVELTY: 0.0975910, \n",
      "\n",
      "\n",
      "SearchBayesianSkopt: Saving model in result_experiments/ItemKNNCBFRecommender_ICM_genres_dice\n",
      "\n",
      "ItemKNNCBFRecommender: Saving model in file 'result_experiments/ItemKNNCBFRecommender_ICM_genres_dice_best_model'\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 6.26 sec. Users per second: 964\n",
      "SearchBayesianSkopt: Config evaluated with evaluator_test. Config: {'topK': 735, 'shrink': 92, 'similarity': 'jaccard', 'normalize': False} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.0639828, PRECISION_RECALL_MIN_DEN: 0.0667611, RECALL: 0.0232852, MAP: 0.0272119, MAP_MIN_DEN: 0.0280720, MRR: 0.1539675, NDCG: 0.0405080, F1: 0.0341443, HIT_RATE: 0.3750828, ARHR_ALL_HITS: 0.2016042, NOVELTY: 0.0293515, AVERAGE_POPULARITY: 0.1940854, DIVERSITY_MEAN_INTER_LIST: 0.9088904, DIVERSITY_HERFINDAHL: 0.9908740, COVERAGE_ITEM: 0.2464589, COVERAGE_ITEM_CORRECT: 0.0793201, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.3748344, DIVERSITY_GINI: 0.0406945, SHANNON_ENTROPY: 7.6191741, RATIO_DIVERSITY_HERFINDAHL: 0.9916835, RATIO_DIVERSITY_GINI: 0.1163876, RATIO_SHANNON_ENTROPY: 0.7046393, RATIO_AVERAGE_POPULARITY: 0.7026583, RATIO_NOVELTY: 0.0994540, \n",
      "\n",
      "\n",
      "SearchBayesianSkopt: Saving model in result_experiments/ItemKNNCBFRecommender_ICM_genres_jaccard\n",
      "\n",
      "ItemKNNCBFRecommender: Saving model in file 'result_experiments/ItemKNNCBFRecommender_ICM_genres_jaccard_best_model'\n",
      "ItemKNNCBFRecommender: Saving complete\n",
      "Iteration No: 5 ended. Search finished for the next optimal point.ItemKNNCBFRecommender: Saving complete\n",
      "\n",
      "Time taken: 11.5526\n",
      "Function value obtained: -0.0224\n",
      "Current minimum: -0.0224\n",
      "Iteration No: 6 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 287, 'shrink': 52, 'similarity': 'dice', 'normalize': False}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 14882.88 column/sec. Elapsed time 0.26 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.11 sec. Users per second: 1179\n",
      "SearchBayesianSkopt: Config 5 is suboptimal. Config: {'topK': 365, 'shrink': 0, 'similarity': 'tversky', 'normalize': True, 'tversky_alpha': 0.9451994341143322, 'tversky_beta': 1.9034035449301236} - results: PRECISION: 0.0442563, PRECISION_RECALL_MIN_DEN: 0.0482084, RECALL: 0.0212000, MAP: 0.0179184, MAP_MIN_DEN: 0.0191855, MRR: 0.1097268, NDCG: 0.0286517, F1: 0.0286675, HIT_RATE: 0.2817065, ARHR_ALL_HITS: 0.1383780, NOVELTY: 0.0302992, AVERAGE_POPULARITY: 0.1293653, DIVERSITY_MEAN_INTER_LIST: 0.9727176, DIVERSITY_HERFINDAHL: 0.9972556, COVERAGE_ITEM: 0.7190317, COVERAGE_ITEM_CORRECT: 0.1452485, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.2809603, DIVERSITY_GINI: 0.2178837, SHANNON_ENTROPY: 9.8875999, RATIO_DIVERSITY_HERFINDAHL: 0.9980703, RATIO_DIVERSITY_GINI: 0.6231548, RATIO_SHANNON_ENTROPY: 0.9144287, RATIO_AVERAGE_POPULARITY: 0.4683485, RATIO_NOVELTY: 0.1026652, \n",
      "\n",
      "Iteration No: 5 ended. Search finished for the next optimal point.\n",
      "Time taken: 14.1182\n",
      "Function value obtained: -0.0220\n",
      "Current minimum: -0.0220\n",
      "Iteration No: 6 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 283, 'shrink': 1, 'similarity': 'jaccard', 'normalize': False}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 11278.84 column/sec. Elapsed time 0.34 sec\n",
      "Iteration No: 6 ended. Search finished for the next optimal point.\n",
      "Time taken: 6.6821\n",
      "Function value obtained: -0.0179\n",
      "Current minimum: -0.0242\n",
      "Iteration No: 7 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 9, 'shrink': 967, 'similarity': 'tversky', 'normalize': True, 'tversky_alpha': 0.03618537990293192, 'tversky_beta': 0.008877465435235267}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 21519.74 column/sec. Elapsed time 0.18 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 6.12 sec. Users per second: 984\n",
      "SearchBayesianSkopt: Config 4 is suboptimal. Config: {'topK': 1000, 'shrink': 0, 'similarity': 'cosine', 'normalize': False, 'feature_weighting': 'none'} - results: PRECISION: 0.0536355, PRECISION_RECALL_MIN_DEN: 0.0584220, RECALL: 0.0248232, MAP: 0.0220464, MAP_MIN_DEN: 0.0236169, MRR: 0.1342199, NDCG: 0.0354345, F1: 0.0339390, HIT_RATE: 0.3343293, ARHR_ALL_HITS: 0.1699463, NOVELTY: 0.0290489, AVERAGE_POPULARITY: 0.2005970, DIVERSITY_MEAN_INTER_LIST: 0.8822767, DIVERSITY_HERFINDAHL: 0.9882130, COVERAGE_ITEM: 0.2480041, COVERAGE_ITEM_CORRECT: 0.0697914, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3334437, DIVERSITY_GINI: 0.0383266, SHANNON_ENTROPY: 7.4380894, RATIO_DIVERSITY_HERFINDAHL: 0.9890204, RATIO_DIVERSITY_GINI: 0.1096154, RATIO_SHANNON_ENTROPY: 0.6878922, RATIO_AVERAGE_POPULARITY: 0.7262327, RATIO_NOVELTY: 0.0984288, \n",
      "\n",
      "Iteration No: 5 ended. Search finished for the next optimal point.\n",
      "Time taken: 7.2717\n",
      "Function value obtained: -0.0220\n",
      "Current minimum: -0.0248\n",
      "Iteration No: 6 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 916, 'shrink': 812, 'similarity': 'cosine', 'normalize': False, 'feature_weighting': 'BM25'}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 14910.42 column/sec. Elapsed time 0.26 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 2.96 sec. Users per second: 2038\n",
      "SearchBayesianSkopt: Config 6 is suboptimal. Config: {'topK': 9, 'shrink': 967, 'similarity': 'tversky', 'normalize': True, 'tversky_alpha': 0.03618537990293192, 'tversky_beta': 0.008877465435235267} - results: PRECISION: 0.0498174, PRECISION_RECALL_MIN_DEN: 0.0543617, RECALL: 0.0234052, MAP: 0.0195411, MAP_MIN_DEN: 0.0209707, MRR: 0.1237085, NDCG: 0.0338970, F1: 0.0318477, HIT_RATE: 0.3310093, ARHR_ALL_HITS: 0.1535940, NOVELTY: 0.0295618, AVERAGE_POPULARITY: 0.1806798, DIVERSITY_MEAN_INTER_LIST: 0.9587595, DIVERSITY_HERFINDAHL: 0.9958600, COVERAGE_ITEM: 0.3780582, COVERAGE_ITEM_CORRECT: 0.1127994, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3301325, DIVERSITY_GINI: 0.0844340, SHANNON_ENTROPY: 8.7045558, RATIO_DIVERSITY_HERFINDAHL: 0.9966736, RATIO_DIVERSITY_GINI: 0.2414842, RATIO_SHANNON_ENTROPY: 0.8050180, RATIO_AVERAGE_POPULARITY: 0.6541256, RATIO_NOVELTY: 0.1001666, \n",
      "\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.76 sec. Users per second: 1045\n",
      "SearchBayesianSkopt: New best config found. Config 5: {'topK': 287, 'shrink': 52, 'similarity': 'dice', 'normalize': False} - results: PRECISION: 0.0600266, PRECISION_RECALL_MIN_DEN: 0.0647196, RECALL: 0.0267254, MAP: 0.0255744, MAP_MIN_DEN: 0.0272247, MRR: 0.1527863, NDCG: 0.0403366, F1: 0.0369844, HIT_RATE: 0.3648738, ARHR_ALL_HITS: 0.1955431, NOVELTY: 0.0285218, AVERAGE_POPULARITY: 0.2066722, DIVERSITY_MEAN_INTER_LIST: 0.9261509, DIVERSITY_HERFINDAHL: 0.9925997, COVERAGE_ITEM: 0.3443214, COVERAGE_ITEM_CORRECT: 0.0885913, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3639073, DIVERSITY_GINI: 0.0582015, SHANNON_ENTROPY: 8.0879046, RATIO_DIVERSITY_HERFINDAHL: 0.9934106, RATIO_DIVERSITY_GINI: 0.1664584, RATIO_SHANNON_ENTROPY: 0.7479886, RATIO_AVERAGE_POPULARITY: 0.7482272, RATIO_NOVELTY: 0.0966426, \n",
      "\n",
      "Iteration No: 7 ended. Search finished for the next optimal point.\n",
      "Time taken: 3.9996\n",
      "Function value obtained: -0.0195\n",
      "Current minimum: -0.0242\n",
      "Iteration No: 8 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 858, 'shrink': 550, 'similarity': 'tversky', 'normalize': True, 'tversky_alpha': 0.8601419391283047, 'tversky_beta': 1.5240656544517568}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 8254.07 column/sec. Elapsed time 0.47 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.46 sec. Users per second: 1102\n",
      "SearchBayesianSkopt: Config 5 is suboptimal. Config: {'topK': 283, 'shrink': 1, 'similarity': 'jaccard', 'normalize': False} - results: PRECISION: 0.0481574, PRECISION_RECALL_MIN_DEN: 0.0519393, RECALL: 0.0213857, MAP: 0.0185962, MAP_MIN_DEN: 0.0198599, MRR: 0.1147441, NDCG: 0.0304723, F1: 0.0296185, HIT_RATE: 0.3077689, ARHR_ALL_HITS: 0.1445215, NOVELTY: 0.0306637, AVERAGE_POPULARITY: 0.1444000, DIVERSITY_MEAN_INTER_LIST: 0.9697372, DIVERSITY_HERFINDAHL: 0.9969576, COVERAGE_ITEM: 0.6579964, COVERAGE_ITEM_CORRECT: 0.1359773, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3069536, DIVERSITY_GINI: 0.1538749, SHANNON_ENTROPY: 9.4260833, RATIO_DIVERSITY_HERFINDAHL: 0.9977721, RATIO_DIVERSITY_GINI: 0.4400873, RATIO_SHANNON_ENTROPY: 0.8717465, RATIO_AVERAGE_POPULARITY: 0.5227796, RATIO_NOVELTY: 0.1039005, \n",
      "\n",
      "Iteration No: 6 ended. Search finished for the next optimal point.\n",
      "Time taken: 6.3949\n",
      "Function value obtained: -0.0186\n",
      "Current minimum: -0.0220\n",
      "Iteration No: 7 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 569, 'shrink': 999, 'similarity': 'jaccard', 'normalize': True}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 15970.87 column/sec. Elapsed time 0.24 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 6.53 sec. Users per second: 922\n",
      "SearchBayesianSkopt: Config 5 is suboptimal. Config: {'topK': 916, 'shrink': 812, 'similarity': 'cosine', 'normalize': False, 'feature_weighting': 'BM25'} - results: PRECISION: 0.0487550, PRECISION_RECALL_MIN_DEN: 0.0513934, RECALL: 0.0196372, MAP: 0.0192443, MAP_MIN_DEN: 0.0201858, MRR: 0.1183040, NDCG: 0.0299800, F1: 0.0279977, HIT_RATE: 0.3099270, ARHR_ALL_HITS: 0.1491704, NOVELTY: 0.0308297, AVERAGE_POPULARITY: 0.1394305, DIVERSITY_MEAN_INTER_LIST: 0.9705394, DIVERSITY_HERFINDAHL: 0.9970378, COVERAGE_ITEM: 0.4486222, COVERAGE_ITEM_CORRECT: 0.1367499, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3091060, DIVERSITY_GINI: 0.1187917, SHANNON_ENTROPY: 9.1984988, RATIO_DIVERSITY_HERFINDAHL: 0.9978524, RATIO_DIVERSITY_GINI: 0.3397483, RATIO_SHANNON_ENTROPY: 0.8506990, RATIO_AVERAGE_POPULARITY: 0.5047881, RATIO_NOVELTY: 0.1044627, \n",
      "\n",
      "Iteration No: 6 ended. Search finished for the next optimal point.\n",
      "Time taken: 7.8500\n",
      "Function value obtained: -0.0192\n",
      "Current minimum: -0.0248\n",
      "Iteration No: 7 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 1000, 'shrink': 0, 'similarity': 'cosine', 'normalize': True, 'feature_weighting': 'TF-IDF'}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 5.45 sec. Users per second: 1107\n",
      "SearchBayesianSkopt: Config evaluated with evaluator_test. Config: {'topK': 287, 'shrink': 52, 'similarity': 'dice', 'normalize': False} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.0725646, PRECISION_RECALL_MIN_DEN: 0.0757406, RECALL: 0.0258105, MAP: 0.0329706, MAP_MIN_DEN: 0.0339820, MRR: 0.1795169, NDCG: 0.0457005, F1: 0.0380773, HIT_RATE: 0.4075547, ARHR_ALL_HITS: 0.2392056, NOVELTY: 0.0285228, AVERAGE_POPULARITY: 0.2067020, DIVERSITY_MEAN_INTER_LIST: 0.9261470, DIVERSITY_HERFINDAHL: 0.9925994, COVERAGE_ITEM: 0.3443214, COVERAGE_ITEM_CORRECT: 0.0958022, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.4072848, DIVERSITY_GINI: 0.0582346, SHANNON_ENTROPY: 8.0884383, RATIO_DIVERSITY_HERFINDAHL: 0.9934103, RATIO_DIVERSITY_GINI: 0.1665528, RATIO_SHANNON_ENTROPY: 0.7480380, RATIO_AVERAGE_POPULARITY: 0.7483352, RATIO_NOVELTY: 0.0966461, \n",
      "\n",
      "\n",
      "SearchBayesianSkopt: Saving model in result_experiments/ItemKNNCBFRecommender_ICM_genres_dice\n",
      "\n",
      "ItemKNNCBFRecommender: Saving model in file 'result_experiments/ItemKNNCBFRecommender_ICM_genres_dice_best_model'\n",
      "Similarity column 3883 (100.0%), 12979.49 column/sec. Elapsed time 0.30 sec\n",
      "ItemKNNCBFRecommender: Saving complete\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.77 sec. Users per second: 1044Iteration No: 6 ended. Search finished for the next optimal point.\n",
      "\n",
      "Time taken: 12.6301\n",
      "SearchBayesianSkopt: Config 7 is suboptimal. Config: {'topK': 858, 'shrink': 550, 'similarity': 'tversky', 'normalize': True, 'tversky_alpha': 0.8601419391283047, 'tversky_beta': 1.5240656544517568} - results: PRECISION: 0.0547477, PRECISION_RECALL_MIN_DEN: 0.0587860, RECALL: 0.0239854, MAP: 0.0214813, MAP_MIN_DEN: 0.0228294, MRR: 0.1281529, NDCG: 0.0348191, F1: 0.0333568, HIT_RATE: 0.3411355, ARHR_ALL_HITS: 0.1645351, NOVELTY: 0.0290980, AVERAGE_POPULARITY: 0.1995948, DIVERSITY_MEAN_INTER_LIST: 0.9111096, DIVERSITY_HERFINDAHL: 0.9910958, COVERAGE_ITEM: 0.2487767, COVERAGE_ITEM_CORRECT: 0.0746845, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3402318, DIVERSITY_GINI: 0.0420367, SHANNON_ENTROPY: 7.6711548, RATIO_DIVERSITY_HERFINDAHL: 0.9919055, RATIO_DIVERSITY_GINI: 0.1202262, RATIO_SHANNON_ENTROPY: 0.7094466, RATIO_AVERAGE_POPULARITY: 0.7226045, RATIO_NOVELTY: 0.0985952, \n",
      "Function value obtained: -0.0256\n",
      "\n",
      "Current minimum: -0.0256\n",
      "Iteration No: 7 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 1000, 'shrink': 684, 'similarity': 'dice', 'normalize': True}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Iteration No: 8 ended. Search finished for the next optimal point.\n",
      "Time taken: 6.8434\n",
      "Function value obtained: -0.0215\n",
      "Current minimum: -0.0242\n",
      "Iteration No: 9 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 12, 'shrink': 233, 'similarity': 'tversky', 'normalize': True, 'tversky_alpha': 1.1406450287840182, 'tversky_beta': 1.4812523429702427}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Similarity column 3883 (100.0%), 8014.72 column/sec. Elapsed time 0.48 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.49 sec. Users per second: 1098\n",
      "SearchBayesianSkopt: Config 6 is suboptimal. Config: {'topK': 569, 'shrink': 999, 'similarity': 'jaccard', 'normalize': True} - results: PRECISION: 0.0538679, PRECISION_RECALL_MIN_DEN: 0.0583835, RECALL: 0.0247511, MAP: 0.0215132, MAP_MIN_DEN: 0.0229863, MRR: 0.1305522, NDCG: 0.0362937, F1: 0.0339177, HIT_RATE: 0.3389774, ARHR_ALL_HITS: 0.1661128, NOVELTY: 0.0290451, AVERAGE_POPULARITY: 0.1962369, DIVERSITY_MEAN_INTER_LIST: 0.8982177, DIVERSITY_HERFINDAHL: 0.9898069, COVERAGE_ITEM: 0.2547000, COVERAGE_ITEM_CORRECT: 0.0767448, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3380795, DIVERSITY_GINI: 0.0385445, SHANNON_ENTROPY: 7.5063410, RATIO_DIVERSITY_HERFINDAHL: 0.9906155, RATIO_DIVERSITY_GINI: 0.1102384, RATIO_SHANNON_ENTROPY: 0.6942042, RATIO_AVERAGE_POPULARITY: 0.7104476, RATIO_NOVELTY: 0.0984159, \n",
      "\n",
      "Similarity column 3883 (100.0%), 12461.69 column/sec. Elapsed time 0.31 sec\n",
      "Iteration No: 7 ended. Search finished for the next optimal point.\n",
      "Time taken: 6.7118\n",
      "Function value obtained: -0.0215\n",
      "Current minimum: -0.0220\n",
      "Iteration No: 8 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 531, 'shrink': 2, 'similarity': 'jaccard', 'normalize': False}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 10202.49 column/sec. Elapsed time 0.38 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.50 sec. Users per second: 1722\n",
      "SearchBayesianSkopt: Config 8 is suboptimal. Config: {'topK': 12, 'shrink': 233, 'similarity': 'tversky', 'normalize': True, 'tversky_alpha': 1.1406450287840182, 'tversky_beta': 1.4812523429702427} - results: PRECISION: 0.0486388, PRECISION_RECALL_MIN_DEN: 0.0533100, RECALL: 0.0228542, MAP: 0.0190485, MAP_MIN_DEN: 0.0205423, MRR: 0.1220675, NDCG: 0.0332500, F1: 0.0310967, HIT_RATE: 0.3212151, ARHR_ALL_HITS: 0.1513546, NOVELTY: 0.0296623, AVERAGE_POPULARITY: 0.1751131, DIVERSITY_MEAN_INTER_LIST: 0.9575611, DIVERSITY_HERFINDAHL: 0.9957402, COVERAGE_ITEM: 0.3947978, COVERAGE_ITEM_CORRECT: 0.1099665, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3203642, DIVERSITY_GINI: 0.0894254, SHANNON_ENTROPY: 8.7626686, RATIO_DIVERSITY_HERFINDAHL: 0.9965537, RATIO_DIVERSITY_GINI: 0.2557596, RATIO_SHANNON_ENTROPY: 0.8103924, RATIO_AVERAGE_POPULARITY: 0.6339720, RATIO_NOVELTY: 0.1005074, \n",
      "\n",
      "Iteration No: 9 ended. Search finished for the next optimal point.\n",
      "Time taken: 4.5888\n",
      "Function value obtained: -0.0190\n",
      "Current minimum: -0.0242\n",
      "Iteration No: 10 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 210, 'shrink': 622, 'similarity': 'tversky', 'normalize': True, 'tversky_alpha': 1.974842434156474, 'tversky_beta': 0.023003328412736227}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.90 sec. Users per second: 1021\n",
      "SearchBayesianSkopt: Config 6 is suboptimal. Config: {'topK': 1000, 'shrink': 0, 'similarity': 'cosine', 'normalize': True, 'feature_weighting': 'TF-IDF'} - results: PRECISION: 0.0374336, PRECISION_RECALL_MIN_DEN: 0.0405500, RECALL: 0.0172952, MAP: 0.0141138, MAP_MIN_DEN: 0.0151664, MRR: 0.0897379, NDCG: 0.0228558, F1: 0.0236593, HIT_RATE: 0.2481740, ARHR_ALL_HITS: 0.1120802, NOVELTY: 0.0317408, AVERAGE_POPULARITY: 0.1120494, DIVERSITY_MEAN_INTER_LIST: 0.9840417, DIVERSITY_HERFINDAHL: 0.9983878, COVERAGE_ITEM: 0.8977595, COVERAGE_ITEM_CORRECT: 0.1784703, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.2475166, DIVERSITY_GINI: 0.3065159, SHANNON_ENTROPY: 10.4390068, RATIO_DIVERSITY_HERFINDAHL: 0.9992035, RATIO_DIVERSITY_GINI: 0.8766456, RATIO_SHANNON_ENTROPY: 0.9654241, RATIO_AVERAGE_POPULARITY: 0.4056589, RATIO_NOVELTY: 0.1075501, \n",
      "\n",
      "Similarity column 3883 (100.0%), 10860.07 column/sec. Elapsed time 0.36 sec\n",
      "Iteration No: 7 ended. Search finished for the next optimal point.\n",
      "Time taken: 7.3062\n",
      "Function value obtained: -0.0141\n",
      "Current minimum: -0.0248\n",
      "Iteration No: 8 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 650, 'shrink': 1000, 'similarity': 'cosine', 'normalize': False, 'feature_weighting': 'none'}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 17757.81 column/sec. Elapsed time 0.22 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.31 sec. Users per second: 1398\n",
      "SearchBayesianSkopt: Config 7 is suboptimal. Config: {'topK': 531, 'shrink': 2, 'similarity': 'jaccard', 'normalize': False} - results: PRECISION: 0.0493526, PRECISION_RECALL_MIN_DEN: 0.0533356, RECALL: 0.0222774, MAP: 0.0196026, MAP_MIN_DEN: 0.0210074, MRR: 0.1213105, NDCG: 0.0318987, F1: 0.0306980, HIT_RATE: 0.3155710, ARHR_ALL_HITS: 0.1525717, NOVELTY: 0.0302332, AVERAGE_POPULARITY: 0.1542081, DIVERSITY_MEAN_INTER_LIST: 0.9563435, DIVERSITY_HERFINDAHL: 0.9956185, COVERAGE_ITEM: 0.6355910, COVERAGE_ITEM_CORRECT: 0.1267062, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3147351, DIVERSITY_GINI: 0.1184219, SHANNON_ENTROPY: 8.9377315, RATIO_DIVERSITY_HERFINDAHL: 0.9964319, RATIO_DIVERSITY_GINI: 0.3386907, RATIO_SHANNON_ENTROPY: 0.8265826, RATIO_AVERAGE_POPULARITY: 0.5582885, RATIO_NOVELTY: 0.1024417, \n",
      "\n",
      "Iteration No: 8 ended. Search finished for the next optimal point.\n",
      "Time taken: 5.6149\n",
      "Function value obtained: -0.0196\n",
      "Current minimum: -0.0220\n",
      "Iteration No: 9 started. Searching for the next optimal point.\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 6.73 sec. Users per second: 895\n",
      "SearchBayesianSkopt: Testing config: {'topK': 996, 'shrink': 11, 'similarity': 'jaccard', 'normalize': True}\n",
      "SearchBayesianSkopt: Config 6 is suboptimal. Config: {'topK': 1000, 'shrink': 684, 'similarity': 'dice', 'normalize': True} - results: PRECISION: 0.0546149, PRECISION_RECALL_MIN_DEN: 0.0585994, RECALL: 0.0237289, MAP: 0.0214386, MAP_MIN_DEN: 0.0227544, MRR: 0.1268898, NDCG: 0.0337818, F1: 0.0330837, HIT_RATE: 0.3383134, ARHR_ALL_HITS: 0.1636748, NOVELTY: 0.0289961, AVERAGE_POPULARITY: 0.2012151, DIVERSITY_MEAN_INTER_LIST: 0.9078116, DIVERSITY_HERFINDAHL: 0.9907661, COVERAGE_ITEM: 0.2462014, COVERAGE_ITEM_CORRECT: 0.0777749, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3374172, DIVERSITY_GINI: 0.0432721, SHANNON_ENTROPY: 7.6905227, RATIO_DIVERSITY_HERFINDAHL: 0.9915755, RATIO_DIVERSITY_GINI: 0.1237598, RATIO_SHANNON_ENTROPY: 0.7112378, RATIO_AVERAGE_POPULARITY: 0.7284707, RATIO_NOVELTY: 0.0982500, \n",
      "\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 7896.93 column/sec. Elapsed time 0.49 sec\n",
      "Iteration No: 7 ended. Search finished for the next optimal point.\n",
      "Time taken: 7.9999\n",
      "Function value obtained: -0.0214\n",
      "Current minimum: -0.0256\n",
      "Iteration No: 8 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 171, 'shrink': 853, 'similarity': 'dice', 'normalize': True}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 11405.87 column/sec. Elapsed time 0.34 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.55 sec. Users per second: 1086\n",
      "SearchBayesianSkopt: New best config found. Config 9: {'topK': 210, 'shrink': 622, 'similarity': 'tversky', 'normalize': True, 'tversky_alpha': 1.974842434156474, 'tversky_beta': 0.023003328412736227} - results: PRECISION: 0.0571879, PRECISION_RECALL_MIN_DEN: 0.0618280, RECALL: 0.0252993, MAP: 0.0242456, MAP_MIN_DEN: 0.0258985, MRR: 0.1482522, NDCG: 0.0382549, F1: 0.0350797, HIT_RATE: 0.3567397, ARHR_ALL_HITS: 0.1873327, NOVELTY: 0.0290207, AVERAGE_POPULARITY: 0.1978623, DIVERSITY_MEAN_INTER_LIST: 0.9378360, DIVERSITY_HERFINDAHL: 0.9937680, COVERAGE_ITEM: 0.3814061, COVERAGE_ITEM_CORRECT: 0.1035282, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3557947, DIVERSITY_GINI: 0.0791638, SHANNON_ENTROPY: 8.4870603, RATIO_DIVERSITY_HERFINDAHL: 0.9945799, RATIO_DIVERSITY_GINI: 0.2264113, RATIO_SHANNON_ENTROPY: 0.7849035, RATIO_AVERAGE_POPULARITY: 0.7163323, RATIO_NOVELTY: 0.0983332, \n",
      "\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.59 sec. Users per second: 1078\n",
      "SearchBayesianSkopt: Config 7 is suboptimal. Config: {'topK': 650, 'shrink': 1000, 'similarity': 'cosine', 'normalize': False, 'feature_weighting': 'none'} - results: PRECISION: 0.0537849, PRECISION_RECALL_MIN_DEN: 0.0586032, RECALL: 0.0249778, MAP: 0.0217764, MAP_MIN_DEN: 0.0234478, MRR: 0.1277486, NDCG: 0.0355572, F1: 0.0341133, HIT_RATE: 0.3253652, ARHR_ALL_HITS: 0.1650309, NOVELTY: 0.0291673, AVERAGE_POPULARITY: 0.1981315, DIVERSITY_MEAN_INTER_LIST: 0.8899143, DIVERSITY_HERFINDAHL: 0.9889767, COVERAGE_ITEM: 0.2165851, COVERAGE_ITEM_CORRECT: 0.0731393, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3245033, DIVERSITY_GINI: 0.0331547, SHANNON_ENTROPY: 7.3217202, RATIO_DIVERSITY_HERFINDAHL: 0.9897846, RATIO_DIVERSITY_GINI: 0.0948234, RATIO_SHANNON_ENTROPY: 0.6771301, RATIO_AVERAGE_POPULARITY: 0.7173069, RATIO_NOVELTY: 0.0988298, \n",
      "\n",
      "Iteration No: 8 ended. Search finished for the next optimal point.\n",
      "Time taken: 6.6680\n",
      "Function value obtained: -0.0218\n",
      "Current minimum: -0.0248\n",
      "Iteration No: 9 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 1000, 'shrink': 0, 'similarity': 'cosine', 'normalize': False, 'feature_weighting': 'TF-IDF'}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 19624.32 column/sec. Elapsed time 0.20 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.87 sec. Users per second: 1236\n",
      "SearchBayesianSkopt: Config 7 is suboptimal. Config: {'topK': 171, 'shrink': 853, 'similarity': 'dice', 'normalize': True} - results: PRECISION: 0.0578021, PRECISION_RECALL_MIN_DEN: 0.0623528, RECALL: 0.0249780, MAP: 0.0231849, MAP_MIN_DEN: 0.0247793, MRR: 0.1353929, NDCG: 0.0369000, F1: 0.0348824, HIT_RATE: 0.3537517, ARHR_ALL_HITS: 0.1743523, NOVELTY: 0.0288782, AVERAGE_POPULARITY: 0.1989794, DIVERSITY_MEAN_INTER_LIST: 0.9326233, DIVERSITY_HERFINDAHL: 0.9932469, COVERAGE_ITEM: 0.4048416, COVERAGE_ITEM_CORRECT: 0.1084213, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3528146, DIVERSITY_GINI: 0.0845645, SHANNON_ENTROPY: 8.5329742, RATIO_DIVERSITY_HERFINDAHL: 0.9940583, RATIO_DIVERSITY_GINI: 0.2418573, RATIO_SHANNON_ENTROPY: 0.7891497, RATIO_AVERAGE_POPULARITY: 0.7203765, RATIO_NOVELTY: 0.0978504, \n",
      "\n",
      "Iteration No: 8 ended. Search finished for the next optimal point.\n",
      "Time taken: 5.8125\n",
      "Function value obtained: -0.0232\n",
      "Current minimum: -0.0256\n",
      "Iteration No: 9 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 391, 'shrink': 884, 'similarity': 'dice', 'normalize': True}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 17989.49 column/sec. Elapsed time 0.22 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 6.50 sec. Users per second: 926\n",
      "SearchBayesianSkopt: Config 8 is suboptimal. Config: {'topK': 996, 'shrink': 11, 'similarity': 'jaccard', 'normalize': True} - results: PRECISION: 0.0517430, PRECISION_RECALL_MIN_DEN: 0.0558704, RECALL: 0.0230354, MAP: 0.0212127, MAP_MIN_DEN: 0.0227128, MRR: 0.1286602, NDCG: 0.0328193, F1: 0.0318787, HIT_RATE: 0.3261952, ARHR_ALL_HITS: 0.1637475, NOVELTY: 0.0295737, AVERAGE_POPULARITY: 0.1828375, DIVERSITY_MEAN_INTER_LIST: 0.9375178, DIVERSITY_HERFINDAHL: 0.9937362, COVERAGE_ITEM: 0.3672418, COVERAGE_ITEM_CORRECT: 0.0963173, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3253311, DIVERSITY_GINI: 0.0677207, SHANNON_ENTROPY: 8.3141187, RATIO_DIVERSITY_HERFINDAHL: 0.9945481, RATIO_DIVERSITY_GINI: 0.1936835, RATIO_SHANNON_ENTROPY: 0.7689094, RATIO_AVERAGE_POPULARITY: 0.6619371, RATIO_NOVELTY: 0.1002069, \n",
      "\n",
      "Iteration No: 9 ended. Search finished for the next optimal point.\n",
      "Time taken: 8.0128\n",
      "Function value obtained: -0.0212\n",
      "Current minimum: -0.0220\n",
      "Iteration No: 10 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 763, 'shrink': 885, 'similarity': 'jaccard', 'normalize': True}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 4.35 sec. Users per second: 1388\n",
      "SearchBayesianSkopt: Config evaluated with evaluator_test. Config: {'topK': 210, 'shrink': 622, 'similarity': 'tversky', 'normalize': True, 'tversky_alpha': 1.974842434156474, 'tversky_beta': 0.023003328412736227} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.0698476, PRECISION_RECALL_MIN_DEN: 0.0725877, RECALL: 0.0242441, MAP: 0.0298015, MAP_MIN_DEN: 0.0307752, MRR: 0.1630850, NDCG: 0.0424014, F1: 0.0359946, HIT_RATE: 0.3972830, ARHR_ALL_HITS: 0.2173189, NOVELTY: 0.0290197, AVERAGE_POPULARITY: 0.1979430, DIVERSITY_MEAN_INTER_LIST: 0.9378690, DIVERSITY_HERFINDAHL: 0.9937714, COVERAGE_ITEM: 0.3816637, COVERAGE_ITEM_CORRECT: 0.1086789, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.3970199, DIVERSITY_GINI: 0.0791639, SHANNON_ENTROPY: 8.4872346, RATIO_DIVERSITY_HERFINDAHL: 0.9945832, RATIO_DIVERSITY_GINI: 0.2264113, RATIO_SHANNON_ENTROPY: 0.7849196, RATIO_AVERAGE_POPULARITY: 0.7166246, RATIO_NOVELTY: 0.0983299, \n",
      "\n",
      "\n",
      "SearchBayesianSkopt: Saving model in result_experiments/ItemKNNCBFRecommender_ICM_genres_tversky\n",
      "\n",
      "ItemKNNCBFRecommender: Saving model in file 'result_experiments/ItemKNNCBFRecommender_ICM_genres_tversky_best_model'\n",
      "ItemKNNCBFRecommender: Saving complete\n",
      "Similarity column 3883 (100.0%), 7946.37 column/sec. Elapsed time 0.49 sec\n",
      "Iteration No: 10 ended. Search finished for the next optimal point.\n",
      "Time taken: 11.8317\n",
      "Function value obtained: -0.0242\n",
      "Current minimum: -0.0242\n",
      "SearchBayesianSkopt: Search complete. Best config is 9: {'topK': 210, 'shrink': 622, 'similarity': 'tversky', 'normalize': True, 'tversky_alpha': 1.974842434156474, 'tversky_beta': 0.023003328412736227}\n",
      "\n",
      "ItemKNNCBFRecommender: URM Detected 212 ( 5.5%) items with no interactions.\n",
      "SearchBayesianSkopt: Evaluation with constructor data for final test. Using best config: {'topK': 210, 'shrink': 622, 'similarity': 'tversky', 'normalize': True, 'tversky_alpha': 1.974842434156474, 'tversky_beta': 0.023003328412736227}\n",
      "Similarity column 3883 (100.0%), 10076.08 column/sec. Elapsed time 0.39 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 6.28 sec. Users per second: 960\n",
      "SearchBayesianSkopt: Config 8 is suboptimal. Config: {'topK': 1000, 'shrink': 0, 'similarity': 'cosine', 'normalize': False, 'feature_weighting': 'TF-IDF'} - results: PRECISION: 0.0580013, PRECISION_RECALL_MIN_DEN: 0.0626265, RECALL: 0.0261697, MAP: 0.0223903, MAP_MIN_DEN: 0.0239071, MRR: 0.1333892, NDCG: 0.0374745, F1: 0.0360665, HIT_RATE: 0.3647078, ARHR_ALL_HITS: 0.1712780, NOVELTY: 0.0280406, AVERAGE_POPULARITY: 0.2306591, DIVERSITY_MEAN_INTER_LIST: 0.8335125, DIVERSITY_HERFINDAHL: 0.9833374, COVERAGE_ITEM: 0.1398403, COVERAGE_ITEM_CORRECT: 0.0594901, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3637417, DIVERSITY_GINI: 0.0227165, SHANNON_ENTROPY: 6.7635655, RATIO_DIVERSITY_HERFINDAHL: 0.9841408, RATIO_DIVERSITY_GINI: 0.0649699, RATIO_SHANNON_ENTROPY: 0.6255106, RATIO_AVERAGE_POPULARITY: 0.8350685, RATIO_NOVELTY: 0.0950121, \n",
      "\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.71 sec. Users per second: 1056\n",
      "SearchBayesianSkopt: Config 8 is suboptimal. Config: {'topK': 391, 'shrink': 884, 'similarity': 'dice', 'normalize': True} - results: PRECISION: 0.0581341, PRECISION_RECALL_MIN_DEN: 0.0632239, RECALL: 0.0266516, MAP: 0.0229665, MAP_MIN_DEN: 0.0245414, MRR: 0.1357936, NDCG: 0.0375545, F1: 0.0365479, HIT_RATE: 0.3587317, ARHR_ALL_HITS: 0.1748712, NOVELTY: 0.0283231, AVERAGE_POPULARITY: 0.2117261, DIVERSITY_MEAN_INTER_LIST: 0.8928519, DIVERSITY_HERFINDAHL: 0.9892704, COVERAGE_ITEM: 0.2956477, COVERAGE_ITEM_CORRECT: 0.0782900, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3577815, DIVERSITY_GINI: 0.0426198, SHANNON_ENTROPY: 7.5927960, RATIO_DIVERSITY_HERFINDAHL: 0.9900786, RATIO_DIVERSITY_GINI: 0.1218940, RATIO_SHANNON_ENTROPY: 0.7021998, RATIO_AVERAGE_POPULARITY: 0.7665241, RATIO_NOVELTY: 0.0959696, \n",
      "\n",
      "Iteration No: 9 ended. Search finished for the next optimal point.\n",
      "Time taken: 7.8298\n",
      "Function value obtained: -0.0224\n",
      "Current minimum: -0.0248\n",
      "Iteration No: 10 started. Searching for the next optimal point.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SearchBayesianSkopt: Testing config: {'topK': 5, 'shrink': 1000, 'similarity': 'cosine', 'normalize': True, 'feature_weighting': 'BM25'}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Iteration No: 9 ended. Search finished for the next optimal point.\n",
      "Time taken: 7.1110\n",
      "Function value obtained: -0.0230\n",
      "Current minimum: -0.0256\n",
      "Iteration No: 10 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 32, 'shrink': 0, 'similarity': 'dice', 'normalize': True}\n",
      "ItemKNNCBFRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "Similarity column 3883 (100.0%), 16843.04 column/sec. Elapsed time 0.23 sec\n",
      "Similarity column 3883 (100.0%), 12634.34 column/sec. Elapsed time 0.31 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 6.95 sec. Users per second: 867\n",
      "SearchBayesianSkopt: Config 9 is suboptimal. Config: {'topK': 763, 'shrink': 885, 'similarity': 'jaccard', 'normalize': True} - results: PRECISION: 0.0548473, PRECISION_RECALL_MIN_DEN: 0.0592178, RECALL: 0.0247383, MAP: 0.0216232, MAP_MIN_DEN: 0.0231221, MRR: 0.1270333, NDCG: 0.0354770, F1: 0.0340973, HIT_RATE: 0.3376494, ARHR_ALL_HITS: 0.1640062, NOVELTY: 0.0291814, AVERAGE_POPULARITY: 0.1976583, DIVERSITY_MEAN_INTER_LIST: 0.9073002, DIVERSITY_HERFINDAHL: 0.9907150, COVERAGE_ITEM: 0.2418233, COVERAGE_ITEM_CORRECT: 0.0762297, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3367550, DIVERSITY_GINI: 0.0401952, SHANNON_ENTROPY: 7.6047323, RATIO_DIVERSITY_HERFINDAHL: 0.9915243, RATIO_DIVERSITY_GINI: 0.1149595, RATIO_SHANNON_ENTROPY: 0.7033037, RATIO_AVERAGE_POPULARITY: 0.7155937, RATIO_NOVELTY: 0.0988778, \n",
      "\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 5.80 sec. Users per second: 1041\n",
      "SearchBayesianSkopt: Best config evaluated with evaluator_test with constructor data for final test. Config: {'topK': 210, 'shrink': 622, 'similarity': 'tversky', 'normalize': True, 'tversky_alpha': 1.974842434156474, 'tversky_beta': 0.023003328412736227} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.0804506, PRECISION_RECALL_MIN_DEN: 0.0836479, RECALL: 0.0266249, MAP: 0.0377567, MAP_MIN_DEN: 0.0389119, MRR: 0.1830461, NDCG: 0.0487640, F1: 0.0400090, HIT_RATE: 0.4160040, ARHR_ALL_HITS: 0.2556297, NOVELTY: 0.0292351, AVERAGE_POPULARITY: 0.1879684, DIVERSITY_MEAN_INTER_LIST: 0.9345670, DIVERSITY_HERFINDAHL: 0.9934412, COVERAGE_ITEM: 0.3525625, COVERAGE_ITEM_CORRECT: 0.1146021, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.4157285, DIVERSITY_GINI: 0.0761172, SHANNON_ENTROPY: 8.4245956, RATIO_DIVERSITY_HERFINDAHL: 0.9942537, RATIO_DIVERSITY_GINI: 0.2176833, RATIO_SHANNON_ENTROPY: 0.7791298, RATIO_AVERAGE_POPULARITY: 0.6842673, RATIO_NOVELTY: 0.0792480, \n",
      "\n",
      "\n",
      "SearchBayesianSkopt: Saving model in result_experiments/ItemKNNCBFRecommender_ICM_genres_tversky\n",
      "\n",
      "ItemKNNCBFRecommender: Saving model in file 'result_experiments/ItemKNNCBFRecommender_ICM_genres_tversky_best_model_last'\n",
      "Iteration No: 10 ended. Search finished for the next optimal point.\n",
      "Time taken: 8.4978\n",
      "Function value obtained: -0.0216\n",
      "Current minimum: -0.0220\n",
      "SearchBayesianSkopt: Search complete. Best config is 4: {'topK': 735, 'shrink': 92, 'similarity': 'jaccard', 'normalize': False}\n",
      "\n",
      "ItemKNNCBFRecommender: URM Detected 212 ( 5.5%) items with no interactions.\n",
      "SearchBayesianSkopt: Evaluation with constructor data for final test. Using best config: {'topK': 735, 'shrink': 92, 'similarity': 'jaccard', 'normalize': False}\n",
      "ItemKNNCBFRecommender: Saving complete\n",
      "Similarity column 3883 (100.0%), 16043.66 column/sec. Elapsed time 0.24 sec\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.78 sec. Users per second: 1259\n",
      "SearchBayesianSkopt: Config 9 is suboptimal. Config: {'topK': 5, 'shrink': 1000, 'similarity': 'cosine', 'normalize': True, 'feature_weighting': 'BM25'} - results: PRECISION: 0.0416501, PRECISION_RECALL_MIN_DEN: 0.0452913, RECALL: 0.0194183, MAP: 0.0160837, MAP_MIN_DEN: 0.0173225, MRR: 0.1082426, NDCG: 0.0296133, F1: 0.0264875, HIT_RATE: 0.2883466, ARHR_ALL_HITS: 0.1310877, NOVELTY: 0.0309839, AVERAGE_POPULARITY: 0.1328330, DIVERSITY_MEAN_INTER_LIST: 0.9663972, DIVERSITY_HERFINDAHL: 0.9966237, COVERAGE_ITEM: 0.4202936, COVERAGE_ITEM_CORRECT: 0.1228432, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.2875828, DIVERSITY_GINI: 0.1111290, SHANNON_ENTROPY: 9.0990507, RATIO_DIVERSITY_HERFINDAHL: 0.9974379, RATIO_DIVERSITY_GINI: 0.3178328, RATIO_SHANNON_ENTROPY: 0.8415018, RATIO_AVERAGE_POPULARITY: 0.4809028, RATIO_NOVELTY: 0.1049851, \n",
      "\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.66 sec. Users per second: 1292\n",
      "SearchBayesianSkopt: Config 9 is suboptimal. Config: {'topK': 32, 'shrink': 0, 'similarity': 'dice', 'normalize': True} - results: PRECISION: 0.0450365, PRECISION_RECALL_MIN_DEN: 0.0484482, RECALL: 0.0200291, MAP: 0.0177534, MAP_MIN_DEN: 0.0187372, MRR: 0.1145069, NDCG: 0.0289691, F1: 0.0277271, HIT_RATE: 0.2946547, ARHR_ALL_HITS: 0.1415684, NOVELTY: 0.0309639, AVERAGE_POPULARITY: 0.1376634, DIVERSITY_MEAN_INTER_LIST: 0.9763518, DIVERSITY_HERFINDAHL: 0.9976190, COVERAGE_ITEM: 0.6590265, COVERAGE_ITEM_CORRECT: 0.1560649, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.2938742, DIVERSITY_GINI: 0.1861630, SHANNON_ENTROPY: 9.7686363, RATIO_DIVERSITY_HERFINDAHL: 0.9984340, RATIO_DIVERSITY_GINI: 0.5324325, RATIO_SHANNON_ENTROPY: 0.9034267, RATIO_AVERAGE_POPULARITY: 0.4983906, RATIO_NOVELTY: 0.1049177, \n",
      "\n",
      "Iteration No: 10 ended. Search finished for the next optimal point.\n",
      "Time taken: 6.0000Iteration No: 10 ended. Search finished for the next optimal point.\n",
      "\n",
      "Function value obtained: -0.0161\n",
      "Time taken: 5.8916Current minimum: -0.0248\n",
      "Function value obtained: -0.0178\n",
      "\n",
      "Current minimum: -0.0256SearchBayesianSkopt: Search complete. Best config is 2: {'topK': 355, 'shrink': 176, 'similarity': 'cosine', 'normalize': False, 'feature_weighting': 'TF-IDF'}\n",
      "\n",
      "\n",
      "SearchBayesianSkopt: Search complete. Best config is 5: {'topK': 287, 'shrink': 52, 'similarity': 'dice', 'normalize': False}\n",
      "\n",
      "ItemKNNCBFRecommender: URM Detected 212 ( 5.5%) items with no interactions.\n",
      "SearchBayesianSkopt: Evaluation with constructor data for final test. Using best config: {'topK': 355, 'shrink': 176, 'similarity': 'cosine', 'normalize': False, 'feature_weighting': 'TF-IDF'}\n",
      "ItemKNNCBFRecommender: URM Detected 212 ( 5.5%) items with no interactions.\n",
      "SearchBayesianSkopt: Evaluation with constructor data for final test. Using best config: {'topK': 287, 'shrink': 52, 'similarity': 'dice', 'normalize': False}\n",
      "Similarity column 3883 (100.0%), 13959.07 column/sec. Elapsed time 0.28 sec\n",
      "Similarity column 3883 (100.0%), 11573.66 column/sec. Elapsed time 0.34 sec\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 6.60 sec. Users per second: 915\n",
      "SearchBayesianSkopt: Best config evaluated with evaluator_test with constructor data for final test. Config: {'topK': 735, 'shrink': 92, 'similarity': 'jaccard', 'normalize': False} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.0741385, PRECISION_RECALL_MIN_DEN: 0.0770749, RECALL: 0.0256674, MAP: 0.0341612, MAP_MIN_DEN: 0.0350863, MRR: 0.1674981, NDCG: 0.0460851, F1: 0.0381328, HIT_RATE: 0.3901590, ARHR_ALL_HITS: 0.2330428, NOVELTY: 0.0296800, AVERAGE_POPULARITY: 0.1816876, DIVERSITY_MEAN_INTER_LIST: 0.9081550, DIVERSITY_HERFINDAHL: 0.9908005, COVERAGE_ITEM: 0.2328097, COVERAGE_ITEM_CORRECT: 0.0883338, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.3899007, DIVERSITY_GINI: 0.0397487, SHANNON_ENTROPY: 7.5968933, RATIO_DIVERSITY_HERFINDAHL: 0.9916108, RATIO_DIVERSITY_GINI: 0.1136752, RATIO_SHANNON_ENTROPY: 0.7025816, RATIO_AVERAGE_POPULARITY: 0.6614030, RATIO_NOVELTY: 0.0804542, \n",
      "\n",
      "\n",
      "SearchBayesianSkopt: Saving model in result_experiments/ItemKNNCBFRecommender_ICM_genres_jaccard\n",
      "\n",
      "ItemKNNCBFRecommender: Saving model in file 'result_experiments/ItemKNNCBFRecommender_ICM_genres_jaccard_best_model_last'\n",
      "ItemKNNCBFRecommender: Saving complete\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 5.42 sec. Users per second: 1115\n",
      "SearchBayesianSkopt: Best config evaluated with evaluator_test with constructor data for final test. Config: {'topK': 287, 'shrink': 52, 'similarity': 'dice', 'normalize': False} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.0843274, PRECISION_RECALL_MIN_DEN: 0.0876359, RECALL: 0.0283476, MAP: 0.0417804, MAP_MIN_DEN: 0.0429283, MRR: 0.2005748, NDCG: 0.0532374, F1: 0.0424314, HIT_RATE: 0.4267727, ARHR_ALL_HITS: 0.2821362, NOVELTY: 0.0288814, AVERAGE_POPULARITY: 0.1934433, DIVERSITY_MEAN_INTER_LIST: 0.9264759, DIVERSITY_HERFINDAHL: 0.9926322, COVERAGE_ITEM: 0.2987381, COVERAGE_ITEM_CORRECT: 0.0968323, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.4264901, DIVERSITY_GINI: 0.0552374, SHANNON_ENTROPY: 8.0452929, RATIO_DIVERSITY_HERFINDAHL: 0.9934441, RATIO_DIVERSITY_GINI: 0.1579704, RATIO_SHANNON_ENTROPY: 0.7440508, RATIO_AVERAGE_POPULARITY: 0.7041977, RATIO_NOVELTY: 0.0782894, \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "SearchBayesianSkopt: Saving model in result_experiments/ItemKNNCBFRecommender_ICM_genres_dice\n",
      "\n",
      "ItemKNNCBFRecommender: Saving model in file 'result_experiments/ItemKNNCBFRecommender_ICM_genres_dice_best_model_last'\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 5.92 sec. Users per second: 1019\n",
      "SearchBayesianSkopt: Best config evaluated with evaluator_test with constructor data for final test. Config: {'topK': 355, 'shrink': 176, 'similarity': 'cosine', 'normalize': False, 'feature_weighting': 'TF-IDF'} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.0870278, PRECISION_RECALL_MIN_DEN: 0.0900327, RECALL: 0.0293889, MAP: 0.0410426, MAP_MIN_DEN: 0.0421953, MRR: 0.2014210, NDCG: 0.0550319, F1: 0.0439396, HIT_RATE: 0.4541087, ARHR_ALL_HITS: 0.2806119, NOVELTY: 0.0286268, AVERAGE_POPULARITY: 0.2092813, DIVERSITY_MEAN_INTER_LIST: 0.8945452, DIVERSITY_HERFINDAHL: 0.9894397, COVERAGE_ITEM: 0.1820757, COVERAGE_ITEM_CORRECT: 0.0875612, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.4538079, DIVERSITY_GINI: 0.0354275, SHANNON_ENTROPY: 7.4194349, RATIO_DIVERSITY_HERFINDAHL: 0.9902489, RATIO_DIVERSITY_GINI: 0.1013170, RATIO_SHANNON_ENTROPY: 0.6861698, RATIO_AVERAGE_POPULARITY: 0.7618530, RATIO_NOVELTY: 0.0775993, \n",
      "\n",
      "\n",
      "ItemKNNCBFRecommender: Saving complete\n",
      "SearchBayesianSkopt: Saving model in result_experiments/ItemKNNCBFRecommender_ICM_genres_cosine\n",
      "\n",
      "ItemKNNCBFRecommender: Saving model in file 'result_experiments/ItemKNNCBFRecommender_ICM_genres_cosine_best_model_last'\n",
      "ItemKNNCBFRecommender: Saving complete\n"
     ]
    }
   ],
   "source": [
    "from Recommenders.KNN.ItemKNNCBFRecommender import ItemKNNCBFRecommender\n",
    "\n",
    "recommender_class = ItemKNNCBFRecommender\n",
    "\n",
    "runHyperparameterSearch_Content(recommender_class,\n",
    "       URM_train = URM_train,\n",
    "       ICM_object = ICM_all,\n",
    "       ICM_name = \"ICM_genres\",\n",
    "       URM_train_last_test = URM_train_validation,\n",
    "       metric_to_optimize = metric_to_optimize,\n",
    "       cutoff_to_optimize = cutoff_to_optimize,\n",
    "       evaluator_validation = evaluator_validation,\n",
    "       evaluator_test = evaluator_test,\n",
    "       output_folder_path = output_folder_path,\n",
    "       parallelizeKNN = True,\n",
    "       allow_weighting = True,\n",
    "       resume_from_saved = True,\n",
    "       save_model = \"best\",\n",
    "       similarity_type_list = ['cosine', 'jaccard', \"dice\", \"tversky\"],\n",
    "       n_cases = n_cases,\n",
    "       n_random_starts = n_random_starts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### You can use the \"partial\" function to pre-set all attributes that are needed and use it to loop on the recommender_classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "from Recommenders.NonPersonalizedRecommender import TopPop, Random\n",
    "from Recommenders.GraphBased.P3alphaRecommender import P3alphaRecommender\n",
    "import os, multiprocessing\n",
    "from functools import partial\n",
    "\n",
    "\n",
    "runHyperparameterSearch_Collaborative_partial = partial(runHyperparameterSearch_Collaborative,\n",
    "       URM_train = URM_train,\n",
    "       URM_train_last_test = URM_train_validation,\n",
    "       metric_to_optimize = metric_to_optimize,\n",
    "       cutoff_to_optimize = cutoff_to_optimize,\n",
    "       evaluator_validation_earlystopping = evaluator_validation,\n",
    "       evaluator_validation = evaluator_validation,\n",
    "       evaluator_test = evaluator_test,\n",
    "       output_folder_path = output_folder_path,\n",
    "       parallelizeKNN = True,\n",
    "       allow_weighting = True,\n",
    "       resume_from_saved = True,\n",
    "       save_model = \"best\",\n",
    "       similarity_type_list = ['cosine', 'jaccard', \"asymmetric\", \"dice\", \"tversky\"],\n",
    "       n_cases = n_cases,\n",
    "       n_random_starts = n_random_starts)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "collaborative_algorithm_list = [\n",
    "    Random,\n",
    "    TopPop,\n",
    "    P3alphaRecommender,\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### You can call it as part of a parallel pool, hence parallizing the optimization of the various models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SearchSingleCase: Resuming 'RandomRecommender' Failed, no such file exists.\n",
      "\n",
      "SearchSingleCase: Testing config: {}\n",
      "RandomRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "SearchBayesianSkopt: Resuming 'P3alphaRecommender' Failed, no such file exists.\n",
      "\n",
      "Iteration No: 1 started. Evaluating function at random point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 593, 'alpha': 1.5162625884190866, 'normalize_similarity': False}\n",
      "P3alphaRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "SearchSingleCase: Resuming 'TopPopRecommender'... Loaded 1 configurations.\n",
      "TopPopRecommender: URM Detected 212 ( 5.5%) items with no interactions.\n",
      "SearchSingleCase: Resuming 'TopPopRecommender'... Result on last already available.\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.76 sec. Users per second: 1266\n",
      "SearchSingleCase: New best config found. Config 0: {} - results: PRECISION: 0.0072709, PRECISION_RECALL_MIN_DEN: 0.0077562, RECALL: 0.0028381, MAP: 0.0022469, MAP_MIN_DEN: 0.0024028, MRR: 0.0205622, NDCG: 0.0043718, F1: 0.0040826, HIT_RATE: 0.0667331, ARHR_ALL_HITS: 0.0214819, NOVELTY: 0.0322452, AVERAGE_POPULARITY: 0.0699540, DIVERSITY_MEAN_INTER_LIST: 0.9974217, DIVERSITY_HERFINDAHL: 0.9997256, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.0955447, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.0665563, DIVERSITY_GINI: 0.8560043, SHANNON_ENTROPY: 11.8749975, RATIO_DIVERSITY_HERFINDAHL: 1.0005424, RATIO_DIVERSITY_GINI: 2.4482009, RATIO_SHANNON_ENTROPY: 1.0982279, RATIO_AVERAGE_POPULARITY: 0.2532584, RATIO_NOVELTY: 0.1092592, \n",
      "\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 4.47 sec. Users per second: 1351\n",
      "SearchSingleCase: Config evaluated with evaluator_test. Config: {} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.0096090, PRECISION_RECALL_MIN_DEN: 0.0099116, RECALL: 0.0026474, MAP: 0.0033105, MAP_MIN_DEN: 0.0033909, MRR: 0.0291325, NDCG: 0.0056186, F1: 0.0041511, HIT_RATE: 0.0854871, ARHR_ALL_HITS: 0.0309822, NOVELTY: 0.0322926, AVERAGE_POPULARITY: 0.0698736, DIVERSITY_MEAN_INTER_LIST: 0.9974176, DIVERSITY_HERFINDAHL: 0.9997252, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.1205254, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.0854305, DIVERSITY_GINI: 0.8549826, SHANNON_ENTROPY: 11.8743501, RATIO_DIVERSITY_HERFINDAHL: 1.0005420, RATIO_DIVERSITY_GINI: 2.4452788, RATIO_SHANNON_ENTROPY: 1.0981681, RATIO_AVERAGE_POPULARITY: 0.2529673, RATIO_NOVELTY: 0.1094196, \n",
      "\n",
      "\n",
      "SearchSingleCase: Saving model in result_experiments/RandomRecommender\n",
      "\n",
      "RandomRecommender: Saving model in file 'result_experiments/RandomRecommender_best_model'\n",
      "RandomRecommender: Saving complete\n",
      "SearchSingleCase: Search complete. Best config is 0: {}\n",
      "\n",
      "RandomRecommender: URM Detected 212 ( 5.5%) items with no interactions.\n",
      "SearchSingleCase: Evaluation with constructor data for final test. Using best config: {}\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.92 sec. Users per second: 1225\n",
      "SearchBayesianSkopt: New best config found. Config 0: {'topK': 593, 'alpha': 1.5162625884190866, 'normalize_similarity': False} - results: PRECISION: 0.1405544, PRECISION_RECALL_MIN_DEN: 0.1607526, RECALL: 0.0858479, MAP: 0.0687499, MAP_MIN_DEN: 0.0772356, MRR: 0.3307953, NDCG: 0.1300448, F1: 0.1065917, HIT_RATE: 0.6743028, ARHR_ALL_HITS: 0.4709899, NOVELTY: 0.0256368, AVERAGE_POPULARITY: 0.3563809, DIVERSITY_MEAN_INTER_LIST: 0.9475963, DIVERSITY_HERFINDAHL: 0.9947439, COVERAGE_ITEM: 0.3590008, COVERAGE_ITEM_CORRECT: 0.1712593, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.6725166, DIVERSITY_GINI: 0.0796534, SHANNON_ENTROPY: 8.5545218, RATIO_DIVERSITY_HERFINDAHL: 0.9955566, RATIO_DIVERSITY_GINI: 0.2278113, RATIO_SHANNON_ENTROPY: 0.7911425, RATIO_AVERAGE_POPULARITY: 1.2902263, RATIO_NOVELTY: 0.0868673, \n",
      "\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 4.68 sec. Users per second: 1291\n",
      "SearchSingleCase: Best config evaluated with evaluator_test with constructor data for final test. Config: {} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.0095096, PRECISION_RECALL_MIN_DEN: 0.0097665, RECALL: 0.0026532, MAP: 0.0028943, MAP_MIN_DEN: 0.0029421, MRR: 0.0249688, NDCG: 0.0050534, F1: 0.0041488, HIT_RATE: 0.0844930, ARHR_ALL_HITS: 0.0269000, NOVELTY: 0.0326184, AVERAGE_POPULARITY: 0.0683136, DIVERSITY_MEAN_INTER_LIST: 0.9974199, DIVERSITY_HERFINDAHL: 0.9997255, COVERAGE_ITEM: 1.0000000, COVERAGE_ITEM_CORRECT: 0.1192377, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.0844371, DIVERSITY_GINI: 0.8555209, SHANNON_ENTROPY: 11.8746572, RATIO_DIVERSITY_HERFINDAHL: 1.0005431, RATIO_DIVERSITY_GINI: 2.4466570, RATIO_SHANNON_ENTROPY: 1.0982010, RATIO_AVERAGE_POPULARITY: 0.2486842, RATIO_NOVELTY: 0.0884192, \n",
      "\n",
      "\n",
      "SearchSingleCase: Saving model in result_experiments/RandomRecommender\n",
      "\n",
      "RandomRecommender: Saving model in file 'result_experiments/RandomRecommender_best_model_last'\n",
      "RandomRecommender: Saving complete\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 5.42 sec. Users per second: 1114\n",
      "SearchBayesianSkopt: Config evaluated with evaluator_test. Config: {'topK': 593, 'alpha': 1.5162625884190866, 'normalize_similarity': False} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.1741219, PRECISION_RECALL_MIN_DEN: 0.1885619, RECALL: 0.0849432, MAP: 0.0905345, MAP_MIN_DEN: 0.0966489, MRR: 0.3772665, NDCG: 0.1513188, F1: 0.1141834, HIT_RATE: 0.7321074, ARHR_ALL_HITS: 0.5741307, NOVELTY: 0.0256397, AVERAGE_POPULARITY: 0.3561936, DIVERSITY_MEAN_INTER_LIST: 0.9476603, DIVERSITY_HERFINDAHL: 0.9947503, COVERAGE_ITEM: 0.3597734, COVERAGE_ITEM_CORRECT: 0.1818182, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.7316225, DIVERSITY_GINI: 0.0797742, SHANNON_ENTROPY: 8.5566247, RATIO_DIVERSITY_HERFINDAHL: 0.9955630, RATIO_DIVERSITY_GINI: 0.2281570, RATIO_SHANNON_ENTROPY: 0.7913370, RATIO_AVERAGE_POPULARITY: 1.2895482, RATIO_NOVELTY: 0.0868772, \n",
      "\n",
      "\n",
      "SearchBayesianSkopt: Saving model in result_experiments/P3alphaRecommender\n",
      "\n",
      "P3alphaRecommender: Saving model in file 'result_experiments/P3alphaRecommender_best_model'\n",
      "P3alphaRecommender: Saving complete\n",
      "Iteration No: 1 ended. Evaluation done at random point.\n",
      "Time taken: 16.8209\n",
      "Function value obtained: -0.0687\n",
      "Current minimum: -0.0687\n",
      "Iteration No: 2 started. Evaluating function at random point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 476, 'alpha': 1.2018070319485148, 'normalize_similarity': False}\n",
      "P3alphaRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.97 sec. Users per second: 1212\n",
      "SearchBayesianSkopt: New best config found. Config 1: {'topK': 476, 'alpha': 1.2018070319485148, 'normalize_similarity': False} - results: PRECISION: 0.1977092, PRECISION_RECALL_MIN_DEN: 0.2259836, RECALL: 0.1228170, MAP: 0.1040237, MAP_MIN_DEN: 0.1166287, MRR: 0.4323520, NDCG: 0.1936222, F1: 0.1515136, HIT_RATE: 0.8061089, ARHR_ALL_HITS: 0.6600079, NOVELTY: 0.0242144, AVERAGE_POPULARITY: 0.4746429, DIVERSITY_MEAN_INTER_LIST: 0.9212821, DIVERSITY_HERFINDAHL: 0.9921129, COVERAGE_ITEM: 0.2049961, COVERAGE_ITEM_CORRECT: 0.1220706, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.8039735, DIVERSITY_GINI: 0.0407125, SHANNON_ENTROPY: 7.6867576, RATIO_DIVERSITY_HERFINDAHL: 0.9929234, RATIO_DIVERSITY_GINI: 0.1164391, RATIO_SHANNON_ENTROPY: 0.7108896, RATIO_AVERAGE_POPULARITY: 1.7183768, RATIO_NOVELTY: 0.0820475, \n",
      "\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 5.06 sec. Users per second: 1193\n",
      "SearchBayesianSkopt: Config evaluated with evaluator_test. Config: {'topK': 476, 'alpha': 1.2018070319485148, 'normalize_similarity': False} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.2455931, PRECISION_RECALL_MIN_DEN: 0.2674266, RECALL: 0.1250908, MAP: 0.1394013, MAP_MIN_DEN: 0.1491043, MRR: 0.4988728, NDCG: 0.2273781, F1: 0.1657555, HIT_RATE: 0.8638171, ARHR_ALL_HITS: 0.8148443, NOVELTY: 0.0242145, AVERAGE_POPULARITY: 0.4746544, DIVERSITY_MEAN_INTER_LIST: 0.9212764, DIVERSITY_HERFINDAHL: 0.9921124, COVERAGE_ITEM: 0.2049961, COVERAGE_ITEM_CORRECT: 0.1321143, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.8632450, DIVERSITY_GINI: 0.0407119, SHANNON_ENTROPY: 7.6867391, RATIO_DIVERSITY_HERFINDAHL: 0.9929229, RATIO_DIVERSITY_GINI: 0.1164375, RATIO_SHANNON_ENTROPY: 0.7108879, RATIO_AVERAGE_POPULARITY: 1.7184184, RATIO_NOVELTY: 0.0820481, \n",
      "\n",
      "\n",
      "SearchBayesianSkopt: Saving model in result_experiments/P3alphaRecommender\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "P3alphaRecommender: Saving model in file 'result_experiments/P3alphaRecommender_best_model'\n",
      "P3alphaRecommender: Saving complete\n",
      "Iteration No: 2 ended. Evaluation done at random point.\n",
      "Time taken: 15.3414\n",
      "Function value obtained: -0.1040\n",
      "Current minimum: -0.1040\n",
      "Iteration No: 3 started. Evaluating function at random point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 18, 'alpha': 1.7608333614795886, 'normalize_similarity': False}\n",
      "P3alphaRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.06 sec. Users per second: 1484\n",
      "SearchBayesianSkopt: Config 2 is suboptimal. Config: {'topK': 18, 'alpha': 1.7608333614795886, 'normalize_similarity': False} - results: PRECISION: 0.0425299, PRECISION_RECALL_MIN_DEN: 0.0468866, RECALL: 0.0216756, MAP: 0.0173111, MAP_MIN_DEN: 0.0192689, MRR: 0.1231091, NDCG: 0.0325182, F1: 0.0287160, HIT_RATE: 0.3014608, ARHR_ALL_HITS: 0.1451413, NOVELTY: 0.0303234, AVERAGE_POPULARITY: 0.1446706, DIVERSITY_MEAN_INTER_LIST: 0.9760597, DIVERSITY_HERFINDAHL: 0.9975898, COVERAGE_ITEM: 0.4944630, COVERAGE_ITEM_CORRECT: 0.2191604, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.3006623, DIVERSITY_GINI: 0.1985310, SHANNON_ENTROPY: 9.8566121, RATIO_DIVERSITY_HERFINDAHL: 0.9984048, RATIO_DIVERSITY_GINI: 0.5678052, RATIO_SHANNON_ENTROPY: 0.9115629, RATIO_AVERAGE_POPULARITY: 0.5237593, RATIO_NOVELTY: 0.1027472, \n",
      "\n",
      "Iteration No: 3 ended. Evaluation done at random point.\n",
      "Time taken: 6.7550\n",
      "Function value obtained: -0.0173\n",
      "Current minimum: -0.1040\n",
      "Iteration No: 4 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 489, 'alpha': 1.8928458446812502, 'normalize_similarity': False}\n",
      "P3alphaRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.75 sec. Users per second: 1267\n",
      "SearchBayesianSkopt: Config 3 is suboptimal. Config: {'topK': 489, 'alpha': 1.8928458446812502, 'normalize_similarity': False} - results: PRECISION: 0.0911355, PRECISION_RECALL_MIN_DEN: 0.1045679, RECALL: 0.0556234, MAP: 0.0413421, MAP_MIN_DEN: 0.0470895, MRR: 0.2399903, NDCG: 0.0833731, F1: 0.0690829, HIT_RATE: 0.5290505, ARHR_ALL_HITS: 0.3116104, NOVELTY: 0.0267022, AVERAGE_POPULARITY: 0.3039776, DIVERSITY_MEAN_INTER_LIST: 0.9696491, DIVERSITY_HERFINDAHL: 0.9969488, COVERAGE_ITEM: 0.4478496, COVERAGE_ITEM_CORRECT: 0.1861962, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.5276490, DIVERSITY_GINI: 0.1262308, SHANNON_ENTROPY: 9.2566314, RATIO_DIVERSITY_HERFINDAHL: 0.9977633, RATIO_DIVERSITY_GINI: 0.3610244, RATIO_SHANNON_ENTROPY: 0.8560752, RATIO_AVERAGE_POPULARITY: 1.1005077, RATIO_NOVELTY: 0.0904772, \n",
      "\n",
      "Iteration No: 4 ended. Search finished for the next optimal point.\n",
      "Time taken: 10.5375\n",
      "Function value obtained: -0.0413\n",
      "Current minimum: -0.1040\n",
      "Iteration No: 5 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 480, 'alpha': 1.210441266407592, 'normalize_similarity': False}\n",
      "P3alphaRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.81 sec. Users per second: 1251\n",
      "SearchBayesianSkopt: Config 4 is suboptimal. Config: {'topK': 480, 'alpha': 1.210441266407592, 'normalize_similarity': False} - results: PRECISION: 0.1957503, PRECISION_RECALL_MIN_DEN: 0.2242562, RECALL: 0.1223336, MAP: 0.1028706, MAP_MIN_DEN: 0.1155130, MRR: 0.4297558, NDCG: 0.1912672, F1: 0.1505693, HIT_RATE: 0.8051129, ARHR_ALL_HITS: 0.6543141, NOVELTY: 0.0242955, AVERAGE_POPULARITY: 0.4651164, DIVERSITY_MEAN_INTER_LIST: 0.9244981, DIVERSITY_HERFINDAHL: 0.9924345, COVERAGE_ITEM: 0.2127221, COVERAGE_ITEM_CORRECT: 0.1241308, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.8029801, DIVERSITY_GINI: 0.0422666, SHANNON_ENTROPY: 7.7425778, RATIO_DIVERSITY_HERFINDAHL: 0.9932452, RATIO_DIVERSITY_GINI: 0.1208840, RATIO_SHANNON_ENTROPY: 0.7160520, RATIO_AVERAGE_POPULARITY: 1.6838876, RATIO_NOVELTY: 0.0823225, \n",
      "\n",
      "Iteration No: 5 ended. Search finished for the next optimal point.\n",
      "Time taken: 10.3827\n",
      "Function value obtained: -0.1029\n",
      "Current minimum: -0.1040\n",
      "Iteration No: 6 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 1000, 'alpha': 0.09916364805583118, 'normalize_similarity': False}\n",
      "P3alphaRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 6.53 sec. Users per second: 922\n",
      "SearchBayesianSkopt: Config 5 is suboptimal. Config: {'topK': 1000, 'alpha': 0.09916364805583118, 'normalize_similarity': False} - results: PRECISION: 0.1472112, PRECISION_RECALL_MIN_DEN: 0.1652371, RECALL: 0.0813660, MAP: 0.0768775, MAP_MIN_DEN: 0.0845839, MRR: 0.3600970, NDCG: 0.1443967, F1: 0.1048047, HIT_RATE: 0.6774568, ARHR_ALL_HITS: 0.5170398, NOVELTY: 0.0221075, AVERAGE_POPULARITY: 0.7625312, DIVERSITY_MEAN_INTER_LIST: 0.5256416, DIVERSITY_HERFINDAHL: 0.9525554, COVERAGE_ITEM: 0.0187999, COVERAGE_ITEM_CORRECT: 0.0141643, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.6756623, DIVERSITY_GINI: 0.0051964, SHANNON_ENTROPY: 4.6356830, RATIO_DIVERSITY_HERFINDAHL: 0.9533336, RATIO_DIVERSITY_GINI: 0.0148618, RATIO_SHANNON_ENTROPY: 0.4287190, RATIO_AVERAGE_POPULARITY: 2.7606355, RATIO_NOVELTY: 0.0749088, \n",
      "\n",
      "Iteration No: 6 ended. Search finished for the next optimal point.\n",
      "Time taken: 14.4761\n",
      "Function value obtained: -0.0769\n",
      "Current minimum: -0.1040\n",
      "Iteration No: 7 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 216, 'alpha': 0.665732798559821, 'normalize_similarity': True}\n",
      "P3alphaRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.60 sec. Users per second: 1308\n",
      "SearchBayesianSkopt: New best config found. Config 6: {'topK': 216, 'alpha': 0.665732798559821, 'normalize_similarity': True} - results: PRECISION: 0.2035359, PRECISION_RECALL_MIN_DEN: 0.2316243, RECALL: 0.1240103, MAP: 0.1077651, MAP_MIN_DEN: 0.1201115, MRR: 0.4397373, NDCG: 0.2023980, F1: 0.1541190, HIT_RATE: 0.8114210, ARHR_ALL_HITS: 0.6760757, NOVELTY: 0.0230905, AVERAGE_POPULARITY: 0.6144502, DIVERSITY_MEAN_INTER_LIST: 0.8349281, DIVERSITY_HERFINDAHL: 0.9834789, COVERAGE_ITEM: 0.1200103, COVERAGE_ITEM_CORRECT: 0.0880762, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.8092715, DIVERSITY_GINI: 0.0207965, SHANNON_ENTROPY: 6.6733157, RATIO_DIVERSITY_HERFINDAHL: 0.9842824, RATIO_DIVERSITY_GINI: 0.0594787, RATIO_SHANNON_ENTROPY: 0.6171641, RATIO_AVERAGE_POPULARITY: 2.2245294, RATIO_NOVELTY: 0.0782395, \n",
      "\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 4.58 sec. Users per second: 1318\n",
      "SearchBayesianSkopt: Config evaluated with evaluator_test. Config: {'topK': 216, 'alpha': 0.665732798559821, 'normalize_similarity': True} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.2529821, PRECISION_RECALL_MIN_DEN: 0.2745950, RECALL: 0.1257672, MAP: 0.1451665, MAP_MIN_DEN: 0.1547935, MRR: 0.5064905, NDCG: 0.2379730, F1: 0.1680101, HIT_RATE: 0.8644798, ARHR_ALL_HITS: 0.8368393, NOVELTY: 0.0230908, AVERAGE_POPULARITY: 0.6144327, DIVERSITY_MEAN_INTER_LIST: 0.8349283, DIVERSITY_HERFINDAHL: 0.9834790, COVERAGE_ITEM: 0.1200103, COVERAGE_ITEM_CORRECT: 0.0947721, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.8639073, DIVERSITY_GINI: 0.0208021, SHANNON_ENTROPY: 6.6735418, RATIO_DIVERSITY_HERFINDAHL: 0.9842825, RATIO_DIVERSITY_GINI: 0.0594946, RATIO_SHANNON_ENTROPY: 0.6171850, RATIO_AVERAGE_POPULARITY: 2.2244661, RATIO_NOVELTY: 0.0782405, \n",
      "\n",
      "\n",
      "SearchBayesianSkopt: Saving model in result_experiments/P3alphaRecommender\n",
      "\n",
      "P3alphaRecommender: Saving model in file 'result_experiments/P3alphaRecommender_best_model'\n",
      "P3alphaRecommender: Saving complete\n",
      "Iteration No: 7 ended. Search finished for the next optimal point.\n",
      "Time taken: 13.3946\n",
      "Function value obtained: -0.1078\n",
      "Current minimum: -0.1078\n",
      "Iteration No: 8 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 434, 'alpha': 0.9714511150925541, 'normalize_similarity': False}\n",
      "P3alphaRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 5.22 sec. Users per second: 1154\n",
      "SearchBayesianSkopt: Config 7 is suboptimal. Config: {'topK': 434, 'alpha': 0.9714511150925541, 'normalize_similarity': False} - results: PRECISION: 0.1907039, PRECISION_RECALL_MIN_DEN: 0.2173385, RECALL: 0.1156303, MAP: 0.1003995, MAP_MIN_DEN: 0.1122824, MRR: 0.4245669, NDCG: 0.1934568, F1: 0.1439679, HIT_RATE: 0.7919987, ARHR_ALL_HITS: 0.6420352, NOVELTY: 0.0227281, AVERAGE_POPULARITY: 0.6635303, DIVERSITY_MEAN_INTER_LIST: 0.7763849, DIVERSITY_HERFINDAHL: 0.9776256, COVERAGE_ITEM: 0.0865310, COVERAGE_ITEM_CORRECT: 0.0638681, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.7899007, DIVERSITY_GINI: 0.0137055, SHANNON_ENTROPY: 6.1016875, RATIO_DIVERSITY_HERFINDAHL: 0.9784243, RATIO_DIVERSITY_GINI: 0.0391981, RATIO_SHANNON_ENTROPY: 0.5642985, RATIO_AVERAGE_POPULARITY: 2.4022167, RATIO_NOVELTY: 0.0770113, \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Iteration No: 8 ended. Search finished for the next optimal point.\n",
      "Time taken: 10.3573\n",
      "Function value obtained: -0.1004\n",
      "Current minimum: -0.1078\n",
      "Iteration No: 9 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 5, 'alpha': 1.0204916543650424, 'normalize_similarity': True}\n",
      "P3alphaRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 4.43 sec. Users per second: 1361\n",
      "SearchBayesianSkopt: Config 8 is suboptimal. Config: {'topK': 5, 'alpha': 1.0204916543650424, 'normalize_similarity': True} - results: PRECISION: 0.0985060, PRECISION_RECALL_MIN_DEN: 0.1026572, RECALL: 0.0372631, MAP: 0.0555751, MAP_MIN_DEN: 0.0586253, MRR: 0.2952565, NDCG: 0.0967650, F1: 0.0540718, HIT_RATE: 0.4973440, ARHR_ALL_HITS: 0.3977612, NOVELTY: 0.0313592, AVERAGE_POPULARITY: 0.2009188, DIVERSITY_MEAN_INTER_LIST: 0.8220619, DIVERSITY_HERFINDAHL: 0.9821925, COVERAGE_ITEM: 0.2106619, COVERAGE_ITEM_CORRECT: 0.1197528, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.4960265, DIVERSITY_GINI: 0.0399903, SHANNON_ENTROPY: 7.2515997, RATIO_DIVERSITY_HERFINDAHL: 0.9829950, RATIO_DIVERSITY_GINI: 0.1143737, RATIO_SHANNON_ENTROPY: 0.6706451, RATIO_AVERAGE_POPULARITY: 0.7273977, RATIO_NOVELTY: 0.1062571, \n",
      "\n",
      "Iteration No: 9 ended. Search finished for the next optimal point.\n",
      "Time taken: 7.3011\n",
      "Function value obtained: -0.0556\n",
      "Current minimum: -0.1078\n",
      "Iteration No: 10 started. Searching for the next optimal point.\n",
      "SearchBayesianSkopt: Testing config: {'topK': 231, 'alpha': 0.28805877651931483, 'normalize_similarity': True}\n",
      "P3alphaRecommender: URM Detected 238 ( 6.1%) items with no interactions.\n",
      "EvaluatorHoldout: Processed 6024 (100.0%) in 3.29 sec. Users per second: 1833\n",
      "SearchBayesianSkopt: Config 9 is suboptimal. Config: {'topK': 231, 'alpha': 0.28805877651931483, 'normalize_similarity': True} - results: PRECISION: 0.1983898, PRECISION_RECALL_MIN_DEN: 0.2246016, RECALL: 0.1182638, MAP: 0.1045122, MAP_MIN_DEN: 0.1160762, MRR: 0.4283005, NDCG: 0.1943353, F1: 0.1481892, HIT_RATE: 0.7984728, ARHR_ALL_HITS: 0.6568678, NOVELTY: 0.0229269, AVERAGE_POPULARITY: 0.6371885, DIVERSITY_MEAN_INTER_LIST: 0.8031965, DIVERSITY_HERFINDAHL: 0.9803063, COVERAGE_ITEM: 0.1112542, COVERAGE_ITEM_CORRECT: 0.0824105, COVERAGE_USER: 0.9973510, COVERAGE_USER_CORRECT: 0.7963576, DIVERSITY_GINI: 0.0179169, SHANNON_ENTROPY: 6.4357108, RATIO_DIVERSITY_HERFINDAHL: 0.9811072, RATIO_DIVERSITY_GINI: 0.0512430, RATIO_SHANNON_ENTROPY: 0.5951898, RATIO_AVERAGE_POPULARITY: 2.3068499, RATIO_NOVELTY: 0.0776852, \n",
      "\n",
      "Iteration No: 10 ended. Search finished for the next optimal point.\n",
      "Time taken: 7.3555\n",
      "Function value obtained: -0.1045\n",
      "Current minimum: -0.1078\n",
      "SearchBayesianSkopt: Search complete. Best config is 6: {'topK': 216, 'alpha': 0.665732798559821, 'normalize_similarity': True}\n",
      "\n",
      "P3alphaRecommender: URM Detected 212 ( 5.5%) items with no interactions.\n",
      "SearchBayesianSkopt: Evaluation with constructor data for final test. Using best config: {'topK': 216, 'alpha': 0.665732798559821, 'normalize_similarity': True}\n",
      "EvaluatorHoldout: Processed 6036 (100.0%) in 4.60 sec. Users per second: 1312\n",
      "SearchBayesianSkopt: Best config evaluated with evaluator_test with constructor data for final test. Config: {'topK': 216, 'alpha': 0.665732798559821, 'normalize_similarity': True} - results:\n",
      "CUTOFF: 10 - PRECISION: 0.3290590, PRECISION_RECALL_MIN_DEN: 0.3525748, RECALL: 0.1478919, MAP: 0.2348700, MAP_MIN_DEN: 0.2463609, MRR: 0.6197545, NDCG: 0.3046532, F1: 0.2040678, HIT_RATE: 0.8860172, ARHR_ALL_HITS: 1.1426408, NOVELTY: 0.0231784, AVERAGE_POPULARITY: 0.5975006, DIVERSITY_MEAN_INTER_LIST: 0.8500048, DIVERSITY_HERFINDAHL: 0.9849864, COVERAGE_ITEM: 0.1228432, COVERAGE_ITEM_CORRECT: 0.1055885, COVERAGE_USER: 0.9993377, COVERAGE_USER_CORRECT: 0.8854305, DIVERSITY_GINI: 0.0231268, SHANNON_ENTROPY: 6.8272596, RATIO_DIVERSITY_HERFINDAHL: 0.9857920, RATIO_DIVERSITY_GINI: 0.0661390, RATIO_SHANNON_ENTROPY: 0.6314038, RATIO_AVERAGE_POPULARITY: 2.1750998, RATIO_NOVELTY: 0.0628301, \n",
      "\n",
      "\n",
      "SearchBayesianSkopt: Saving model in result_experiments/P3alphaRecommender\n",
      "\n",
      "P3alphaRecommender: Saving model in file 'result_experiments/P3alphaRecommender_best_model_last'\n",
      "P3alphaRecommender: Saving complete\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[None, None, None]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pool = multiprocessing.Pool(processes=int(multiprocessing.cpu_count()), maxtasksperchild=1)\n",
    "pool.map(runHyperparameterSearch_Collaborative_partial, collaborative_algorithm_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Or sequentially, note that since resume_from_saved = True this new call to the function will just load the optimized values and not re-run it "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SearchSingleCase: Resuming 'RandomRecommender'... Loaded 1 configurations.\n",
      "RandomRecommender: URM Detected 212 ( 5.5%) items with no interactions.\n",
      "SearchSingleCase: Resuming 'RandomRecommender'... Result on last already available.\n",
      "SearchSingleCase: Resuming 'TopPopRecommender'... Loaded 1 configurations.\n",
      "TopPopRecommender: URM Detected 212 ( 5.5%) items with no interactions.\n",
      "SearchSingleCase: Resuming 'TopPopRecommender'... Result on last already available.\n",
      "SearchBayesianSkopt: Resuming 'P3alphaRecommender'... Loaded 10 configurations.\n",
      "P3alphaRecommender: URM Detected 212 ( 5.5%) items with no interactions.\n",
      "SearchBayesianSkopt: Resuming 'P3alphaRecommender'... Result on last already available.\n"
     ]
    }
   ],
   "source": [
    "for recommender_class in collaborative_algorithm_list:\n",
    "\n",
    "    try:\n",
    "\n",
    "        runHyperparameterSearch_Collaborative_partial(recommender_class)\n",
    "\n",
    "    except Exception as e:\n",
    "\n",
    "        print(\"On recommender {} Exception {}\".format(recommender_class, str(e)))\n",
    "        traceback.print_exc()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to load and save a model\n",
    "### The models are saved in two files:\n",
    "* RECOMMENDER_NAME + \"_best_model.zip\": contains the model trained on the URM_train\n",
    "* RECOMMENDER_NAME + \"_best_model_last.zip\": contains the model trained on the URM_train_validation\n",
    "\n",
    "#### You can load them using the \"load_model\" function of the Recommender class, to do so:\n",
    "* Create an instance of the recommender you want to load and provide the needed URM data and ICM data, if needed. The URM you pass is the one that will be used as user profiles to generate the recommendations.\n",
    "* Given the instance, call the load_model function\n",
    "* You can now use the recommender to generate recommendations.\n",
    "\n",
    "#### If you want to save your own models use the \"save_model\" function, it takes thesame parameters as \"load_model\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "P3alphaRecommender: URM Detected 177 ( 4.6%) items with no interactions.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<Recommenders.GraphBased.P3alphaRecommender.P3alphaRecommender at 0x7f9eeb9412e0>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from Recommenders.GraphBased.P3alphaRecommender import P3alphaRecommender\n",
    "\n",
    "recommender_object = P3alphaRecommender(URM_all)\n",
    "recommender_object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "P3alphaRecommender: Loading model from file 'result_experiments/P3alphaRecommender_best_model_last.zip'\n",
      "P3alphaRecommender: Loading complete\n"
     ]
    }
   ],
   "source": [
    "recommender_object.load_model(output_folder_path, \n",
    "                              file_name = recommender_object.RECOMMENDER_NAME + \"_best_model_last.zip\" )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<3883x3883 sparse matrix of type '<class 'numpy.float32'>'\n",
       "\twith 221647 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recommender_object.W_sparse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[48, 244, 23, 22, 40, 335, 246, 537, 350, 515]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "user_id = 10\n",
    "recommender_object.recommend(user_id, cutoff = 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "P3alphaRecommender: Saving model in file 'result_experiments/P3alphaRecommender_my_own_save.zip'\n",
      "P3alphaRecommender: Saving complete\n"
     ]
    }
   ],
   "source": [
    "recommender_object.save_model(output_folder_path, \n",
    "                              file_name = recommender_object.RECOMMENDER_NAME + \"_my_own_save.zip\" )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}