examples/04_model_select_and_optimize/nni_surprise_svd.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<i>Copyright (c) Microsoft Corporation. All rights reserved.<br>\n",
    "Licensed under the MIT License.</i>\n",
    "<br>\n",
    "# Hyperparameter Tuning for Matrix Factorization Using the Neural Network Intelligence Toolkit"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook shows how to use the **[Neural Network Intelligence](https://nni.readthedocs.io/en/latest/) toolkit (NNI)** for tuning hyperparameters of a matrix factorization model. In particular, we optimize the hyperparameters of [Surprise SVD](https://surprise.readthedocs.io/en/stable/matrix_factorization.html).\n",
    "\n",
    "NNI is a toolkit to help users design and tune machine learning models (e.g., hyperparameters), neural network architectures, or complex system’s parameters, in an efficient and automatic way. NNI has several appealing properties: ease of use, scalability, flexibility and efficiency. NNI comes with [several tuning algorithms](https://nni.readthedocs.io/en/latest/Builtin_Tuner.html) built in. It also allows users to [define their own general purpose tuners](https://nni.readthedocs.io/en/latest/Customize_Tuner.html). NNI can be executed in a distributed way on a local machine, a remote server, or a large scale training platform such as OpenPAI or Kubernetes. \n",
    "\n",
    "In this notebook we execute several NNI _experiments_ on the same data sets obtained from Movielens with a training-validation-test split. Each experiment corresponds to one of the built-in tuning algorithms. It consists of many parallel _trials_, each of which corresponds to a choice of hyperparameters sampled by the tuning algorithm. All the experiments require a call to the same [python script](../../recommenders/nni/svd_training.py) for training the SVD model and evaluating rating and ranking metrics on the test data. This script has been adapted from the [Surprise SVD notebook](../02_model/surprise_svd_deep_dive.ipynb) with only a few changes. In all experiments, we maximize precision@10. \n",
    "\n",
    "For this notebook we use a _local machine_ as the training platform (this can be any machine running the `reco_base` conda environment). In this case, NNI uses the available processors of the machine to parallelize the trials, subject to the value of `trialConcurrency` we specify in the configuration. Our runs and the results we report were obtained on a [Standard_D16_v3 virtual machine](https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sizes-general#dv3-series-1) with 16 vcpus and 64 GB memory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Global Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "System version: 3.6.10 |Anaconda, Inc.| (default, Mar 25 2020, 23:51:54) \n",
      "[GCC 7.3.0]\n",
      "Surprise version: 1.1.0\n",
      "NNI version: 1.5\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    
    "import json\n",
    "import os\n",
    "import surprise\n",
    "import papermill as pm\n",
    "import pandas as pd\n",
    "import shutil\n",
    "import subprocess\n",
    "import yaml\n",
    "import pkg_resources\n",
    "from tempfile import TemporaryDirectory\n",
    "\n",
    "import recommenders\n",
    "from recommenders.utils.timer import Timer\n",
    "from recommenders.datasets import movielens\n",
    "from recommenders.datasets.python_splitters import python_random_split\n",
    "from recommenders.evaluation.python_evaluation import rmse, precision_at_k, ndcg_at_k\n",
    "from recommenders.tuning.nni.nni_utils import (check_experiment_status, check_stopped, check_metrics_written, get_trials,\n",
    "                                      stop_nni, start_nni)\n",
    "from recommenders.models.surprise.surprise_utils import predict, compute_ranking_predictions\n",
    "\n",
    "print(\"System version: {}\".format(sys.version))\n",
    "print(\"Surprise version: {}\".format(surprise.__version__))\n",
    "print(\"NNI version: {}\".format(pkg_resources.get_distribution(\"nni\").version))\n",
    "\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Prepare Dataset\n",
    "1. Download data and split into training, validation and test sets\n",
    "2. Store the data sets to a local directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": [
     "parameters"
    ]
   },
   "outputs": [],
   "source": [
    "# Parameters used by papermill\n",
    "\n",
    "# Select Movielens data size: 100k, 1m\n",
    "MOVIELENS_DATA_SIZE = '100k'\n",
    "SURPRISE_READER = 'ml-100k'\n",
    "tmp_dir = TemporaryDirectory()\n",
    "TMP_DIR = tmp_dir.name\n",
    "NUM_EPOCHS = 30\n",
    "MAX_TRIAL_NUM = 10\n",
    "\n",
    "# time (in seconds) to wait for each tuning experiment to complete\n",
    "WAITING_TIME = 20\n",
    "MAX_RETRIES = 40 # it is recommended to have MAX_RETRIES>=4*MAX_TRIAL_NUM\n",
    "\n",
    "tmp_dir = TemporaryDirectory()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 4.81k/4.81k [00:00<00:00, 10.3kKB/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userID</th>\n",
       "      <th>itemID</th>\n",
       "      <th>rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>196</td>\n",
       "      <td>242</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>186</td>\n",
       "      <td>302</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>22</td>\n",
       "      <td>377</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>244</td>\n",
       "      <td>51</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>166</td>\n",
       "      <td>346</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userID  itemID  rating\n",
       "0     196     242     3.0\n",
       "1     186     302     3.0\n",
       "2      22     377     1.0\n",
       "3     244      51     2.0\n",
       "4     166     346     1.0"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = movielens.load_pandas_df(\n",
    "    size=MOVIELENS_DATA_SIZE,\n",
    "    header=[\"userID\", \"itemID\", \"rating\"]\n",
    ")\n",
    "\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "train, validation, test = python_random_split(data, [0.7, 0.15, 0.15])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "LOG_DIR = os.path.join(TMP_DIR, \"experiments\")\n",
    "os.makedirs(LOG_DIR, exist_ok=True)\n",
    "\n",
    "DATA_DIR = os.path.join(TMP_DIR, \"data\") \n",
    "os.makedirs(DATA_DIR, exist_ok=True)\n",
    "\n",
    "TRAIN_FILE_NAME = \"movielens_\" + MOVIELENS_DATA_SIZE + \"_train.pkl\"\n",
    "train.to_pickle(os.path.join(DATA_DIR, TRAIN_FILE_NAME))\n",
    "\n",
    "VAL_FILE_NAME = \"movielens_\" + MOVIELENS_DATA_SIZE + \"_val.pkl\"\n",
    "validation.to_pickle(os.path.join(DATA_DIR, VAL_FILE_NAME))\n",
    "\n",
    "TEST_FILE_NAME = \"movielens_\" + MOVIELENS_DATA_SIZE + \"_test.pkl\"\n",
    "test.to_pickle(os.path.join(DATA_DIR, TEST_FILE_NAME))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Prepare Hyperparameter Tuning "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now prepare a training script [svd_training_nni.py](../../recommenders/nni/svd_training.py) for the hyperparameter tuning, which will log our target metrics such as precision, NDCG, RMSE.\n",
    "We define the arguments of the script and the search space for the hyperparameters. All the parameter values will be passed to our training script.<br>\n",
    "Note that we specify _precision@10_ as the primary metric. We will also instruct NNI (in the configuration file) to _maximize_ the primary metric. This is passed as an argument in the training script and the evaluated metric is returned through the NNI python library. In addition, we also evaluate RMSE and NDCG@10.   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `script_params` below are the parameters of the training script that are fixed (unlike `hyper_params` which are tuned). In particular, `VERBOSE, BIASED, RANDOM_STATE, NUM_EPOCHS` are parameters used in the [SVD method](../02_model/surprise_svd_deep_dive.ipynb) and `REMOVE_SEEN` removes the training data from the recommended items. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "EXP_NAME = \"movielens_\" + MOVIELENS_DATA_SIZE + \"_svd_model\"\n",
    "PRIMARY_METRIC = \"precision_at_k\"\n",
    "RATING_METRICS = [\"rmse\"]\n",
    "RANKING_METRICS = [\"precision_at_k\", \"ndcg_at_k\"]  \n",
    "USERCOL = \"userID\"\n",
    "ITEMCOL = \"itemID\"\n",
    "REMOVE_SEEN = True\n",
    "K = 10\n",
    "RANDOM_STATE = 42\n",
    "VERBOSE = True\n",
    "BIASED = True\n",
    "\n",
    "script_params = \" \".join([\n",
    "    \"--datastore\", DATA_DIR,\n",
    "    \"--train-datapath\", TRAIN_FILE_NAME,\n",
    "    \"--validation-datapath\", VAL_FILE_NAME,\n",
    "    \"--surprise-reader\", SURPRISE_READER,\n",
    "    \"--rating-metrics\", \" \".join(RATING_METRICS),\n",
    "    \"--ranking-metrics\", \" \".join(RANKING_METRICS),\n",
    "    \"--usercol\", USERCOL,\n",
    "    \"--itemcol\", ITEMCOL,\n",
    "    \"--k\", str(K),\n",
    "    \"--random-state\", str(RANDOM_STATE),\n",
    "    \"--epochs\", str(NUM_EPOCHS),\n",
    "    \"--primary-metric\", PRIMARY_METRIC\n",
    "])\n",
    "\n",
    "if BIASED:\n",
    "    script_params += \" --biased\"\n",
    "if VERBOSE:\n",
    "    script_params += \" --verbose\"\n",
    "if REMOVE_SEEN:\n",
    "    script_params += \" --remove-seen\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# hyperparameters search space\n",
    "# We do not set 'lr_all' and 'reg_all' because they will be overriden by the other lr_ and reg_ parameters\n",
    "\n",
    "hyper_params = {\n",
    "    'n_factors': {\"_type\": \"choice\", \"_value\": [10, 50, 100, 150, 200]},\n",
    "    'init_mean': {\"_type\": \"uniform\", \"_value\": [-0.5, 0.5]},\n",
    "    'init_std_dev': {\"_type\": \"uniform\", \"_value\": [0.01, 0.2]},\n",
    "    'lr_bu': {\"_type\": \"uniform\", \"_value\": [1e-6, 0.1]}, \n",
    "    'lr_bi': {\"_type\": \"uniform\", \"_value\": [1e-6, 0.1]}, \n",
    "    'lr_pu': {\"_type\": \"uniform\", \"_value\": [1e-6, 0.1]}, \n",
    "    'lr_qi': {\"_type\": \"uniform\", \"_value\": [1e-6, 0.1]}, \n",
    "    'reg_bu': {\"_type\": \"uniform\", \"_value\": [1e-6, 1]},\n",
    "    'reg_bi': {\"_type\": \"uniform\", \"_value\": [1e-6, 1]}, \n",
    "    'reg_pu': {\"_type\": \"uniform\", \"_value\": [1e-6, 1]}, \n",
    "    'reg_qi': {\"_type\": \"uniform\", \"_value\": [1e-6, 1]}\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(os.path.join(TMP_DIR, 'search_space_svd.json'), 'w') as fp:\n",
    "    json.dump(hyper_params, fp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also create a yaml file for the configuration of the trials and the tuning algorithm to be used (in this experiment we use the [TPE tuner](https://nni.readthedocs.io/en/latest/hyperoptTuner.html)). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "config = {\n",
    "    \"authorName\": \"default\",\n",
    "    \"experimentName\": \"surprise_svd\",\n",
    "    \"trialConcurrency\": 8,\n",
    "    \"maxExecDuration\": \"1h\",\n",
    "    \"maxTrialNum\": MAX_TRIAL_NUM,\n",
    "    \"trainingServicePlatform\": \"local\",\n",
    "    # The path to Search Space\n",
    "    \"searchSpacePath\": \"search_space_svd.json\",\n",
    "    \"useAnnotation\": False,\n",
    "    \"logDir\": LOG_DIR,\n",
    "    \"tuner\": {\n",
    "        \"builtinTunerName\": \"TPE\",\n",
    "        \"classArgs\": {\n",
    "            #choice: maximize, minimize\n",
    "            \"optimize_mode\": \"maximize\"\n",
    "        }\n",
    "    },\n",
    "    # The path and the running command of trial\n",
    "    \"trial\":  {\n",
    "      \"command\": sys.prefix + \"/bin/python svd_training.py\" + \" \" + script_params,\n",
    "      \"codeDir\": os.path.join(os.path.split(os.path.abspath(recommenders.__file__))[0], \"tuning\", \"nni\"),\n",
    "      \"gpuNum\": 0\n",
    "    }\n",
    "}\n",
    " \n",
    "with open(os.path.join(TMP_DIR, \"config_svd.yml\"), \"w\") as fp:\n",
    "    fp.write(yaml.dump(config, default_flow_style=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Execute NNI Trials\n",
    "\n",
    "The conda environment comes with NNI installed, which includes the command line tool `nnictl` for controlling and getting information about NNI experiments. <br>\n",
    "To start the NNI tuning trials from the command line, execute the following command: <br>\n",
    "`nnictl create --config <path of config_svd.yml>` <br>\n",
    "In the cell below, we call this command programmatically. <br>\n",
    "You can see the progress of the experiment by using the URL links output by the above command.\n",
    "\n",
    "![](https://recodatasets.z20.web.core.windows.net/images/nn1.png)\n",
    "\n",
    "![](https://recodatasets.z20.web.core.windows.net/images/nn2.png)\n",
    "\n",
    "![](https://recodatasets.z20.web.core.windows.net/images/nn3.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make sure that there is no experiment running\n",
    "stop_nni()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "config_path = os.path.join(TMP_DIR, 'config_svd.yml')\n",
    "nni_env = os.environ.copy()\n",
    "nni_env['PATH'] = sys.prefix + '/bin:' + nni_env['PATH']\n",
    "proc = subprocess.run([sys.prefix + '/bin/nnictl', 'create', '--config', config_path], env=nni_env)\n",
    "if proc.returncode != 0:\n",
    "    raise RuntimeError(\"'nnictl create' failed with code %d\" % proc.returncode)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "with Timer() as time_tpe:\n",
    "    check_experiment_status(wait=WAITING_TIME, max_retries=MAX_RETRIES)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Show Results\n",
    "\n",
    "The trial with the best metric and the corresponding metrics and hyperparameters can also be read from the Web UI\n",
    "\n",
    "![](https://recodatasets.z20.web.core.windows.net/images/nni4.png)\n",
    "\n",
    "or from the JSON file created by the training script. Below, we do this programmatically using [nni_utils.py](../../recommenders/nni/nni_utils.py) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "trials, best_metrics, best_params, best_trial_path = get_trials('maximize')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'rmse': 0.9918669108638116,\n",
       " 'ndcg_at_k': 0.05341690918927691,\n",
       " 'precision_at_k': 0.051437699680511186}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'parameter_id': 1,\n",
       " 'parameter_source': 'algorithm',\n",
       " 'parameters': {'n_factors': 50,\n",
       "  'init_mean': -0.32199095465587735,\n",
       "  'init_std_dev': 0.11455022391401633,\n",
       "  'lr_bu': 0.07562149308631293,\n",
       "  'lr_bi': 0.0002925566403137119,\n",
       "  'lr_pu': 0.04052551251769183,\n",
       "  'lr_qi': 0.005161900020236067,\n",
       "  'reg_bu': 0.8623789452897435,\n",
       "  'reg_bi': 0.5692944661931444,\n",
       "  'reg_pu': 0.2579260065274244,\n",
       "  'reg_qi': 0.5207340528183084},\n",
       " 'parameter_index': 0}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/tmp/tmp21mjabn1/experiments/nR67pfst/trials/esAqs'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_trial_path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This directory path is where info about the trial can be found, including logs, parameters and the model that was learned. To evaluate the metrics on the test data, we get the SVD model that was saved as `model.dump` in the training script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "svd = surprise.dump.load(os.path.join(best_trial_path, \"model.dump\"))[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following function computes all the metrics given an SVD model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_test_results(svd):\n",
    "    test_results = {}\n",
    "    predictions = predict(svd, test, usercol=\"userID\", itemcol=\"itemID\")\n",
    "    for metric in RATING_METRICS:\n",
    "        test_results[metric] = eval(metric)(test, predictions)\n",
    "\n",
    "    all_predictions = compute_ranking_predictions(svd, train, usercol=\"userID\", itemcol=\"itemID\", remove_seen=REMOVE_SEEN)\n",
    "    for metric in RANKING_METRICS:\n",
    "        test_results[metric] = eval(metric)(test, all_predictions, col_prediction='prediction', k=K)\n",
    "    return test_results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'rmse': 0.9897488773596576, 'precision_at_k': 0.0769722814498934, 'ndcg_at_k': 0.0850595462545088}\n"
     ]
    }
   ],
   "source": [
    "test_results_tpe = compute_test_results(svd)\n",
    "print(test_results_tpe)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. More Tuning Algorithms\n",
    "We now apply other tuning algorithms supported by NNI to the same problem. For details about these tuners, see the [NNI docs.](https://nni.readthedocs.io/en/latest/tuners.html#)\n",
    "The only change needed is in the relevant entry in the configuration file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In summary, the tuners used in this notebook are the following:\n",
    "- Tree-structured Parzen Estimator (TPE), within the Sequential Model-Based Optimization (SMBO) framework,\n",
    "- SMAC, also an instance of SMBO,\n",
    "- Hyperband\n",
    "- Metis, an implementation of Bayesian optimization with Gaussian Processes\n",
    "- a Naive Evolutionary algorithm\n",
    "- an Annealing method for sampling, and  \n",
    "- plain Random Search as a baseline.  \n",
    "\n",
    "For more details and references to the relevant literature, see the [NNI github](https://github.com/Microsoft/nni/blob/master/docs/en_US/Builtin_Tuner.md)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Random search\n",
    "config['tuner']['builtinTunerName'] = 'Random'\n",
    "if 'classArgs' in config['tuner']:\n",
    "    config['tuner'].pop('classArgs')\n",
    "    \n",
    "with open(config_path, 'w') as fp:\n",
    "    fp.write(yaml.dump(config, default_flow_style=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "stop_nni()\n",
    "with Timer() as time_random:\n",
    "    start_nni(config_path, wait=WAITING_TIME, max_retries=MAX_RETRIES)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_metrics_written(wait=WAITING_TIME, max_retries=MAX_RETRIES)\n",
    "svd = surprise.dump.load(os.path.join(get_trials('maximize')[3], \"model.dump\"))[1]\n",
    "test_results_random = compute_test_results(svd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Annealing\n",
    "config['tuner']['builtinTunerName'] = 'Anneal'\n",
    "if 'classArgs' not in config['tuner']:\n",
    "    config['tuner']['classArgs'] = {'optimize_mode': 'maximize'}\n",
    "else:\n",
    "    config['tuner']['classArgs']['optimize_mode'] = 'maximize'\n",
    "    \n",
    "with open(config_path, 'w') as fp:\n",
    "    fp.write(yaml.dump(config, default_flow_style=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "stop_nni()\n",
    "with Timer() as time_anneal:\n",
    "    start_nni(config_path, wait=WAITING_TIME, max_retries=MAX_RETRIES)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_metrics_written(wait=WAITING_TIME, max_retries=MAX_RETRIES)\n",
    "svd = surprise.dump.load(os.path.join(get_trials('maximize')[3], \"model.dump\"))[1]\n",
    "test_results_anneal = compute_test_results(svd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Naive evolutionary search\n",
    "config['tuner']['builtinTunerName'] = 'Evolution'\n",
    "with open(config_path, 'w') as fp:\n",
    "    fp.write(yaml.dump(config, default_flow_style=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "stop_nni()\n",
    "with Timer() as time_evolution:\n",
    "    start_nni(config_path, wait=WAITING_TIME, max_retries=MAX_RETRIES)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_metrics_written(wait=WAITING_TIME, max_retries=MAX_RETRIES)\n",
    "svd = surprise.dump.load(os.path.join(get_trials('maximize')[3], \"model.dump\"))[1]\n",
    "test_results_evolution = compute_test_results(svd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The SMAC tuner requires to have been installed with the following command <br>\n",
    "`nnictl package install --name=SMAC`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# SMAC\n",
    "config['tuner']['builtinTunerName'] = 'SMAC'\n",
    "with open(config_path, 'w') as fp:\n",
    "    fp.write(yaml.dump(config, default_flow_style=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check if installed\n",
    "proc = subprocess.run([sys.prefix + '/bin/nnictl', 'package', 'show'], stdout=subprocess.PIPE)\n",
    "if proc.returncode != 0:\n",
    "    raise RuntimeError(\"'nnictl package show' failed with code %d\" % proc.returncode)\n",
    "if 'SMAC' not in proc.stdout.decode().strip().split():\n",
    "    proc = subprocess.run([sys.prefix + '/bin/nnictl', 'package', 'install', '--name=SMAC'])\n",
    "    if proc.returncode != 0:\n",
    "        raise RuntimeError(\"'nnictl package install' failed with code %d\" % proc.returncode)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Skipping SMAC optimization for now\n",
    "# stop_nni()\n",
    "with Timer() as time_smac:\n",
    "#    start_nni(config_path, wait=WAITING_TIME, max_retries=MAX_RETRIES)\n",
    "    pass\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "#check_metrics_written()\n",
    "#svd = surprise.dump.load(os.path.join(get_trials('maximize')[3], \"model.dump\"))[1]\n",
    "#test_results_smac = compute_test_results(svd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Metis\n",
    "config['tuner']['builtinTunerName'] = 'MetisTuner'\n",
    "with open(config_path, 'w') as fp:\n",
    "    fp.write(yaml.dump(config, default_flow_style=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "stop_nni()\n",
    "with Timer() as time_metis:\n",
    "    start_nni(config_path, wait=WAITING_TIME, max_retries=MAX_RETRIES)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_metrics_written()\n",
    "svd = surprise.dump.load(os.path.join(get_trials('maximize')[3], \"model.dump\"))[1]\n",
    "test_results_metis = compute_test_results(svd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hyperband follows a different style of configuration from other tuners. See [the NNI documentation](https://nni.readthedocs.io/en/latest/hyperbandAdvisor.html). Note that the [training script](../../recommenders/nni/svd_training.py) needs to be adjusted as well, since each Hyperband trial receives an additional parameter `STEPS`, which corresponds to the resource allocation _r<sub>i</sub>_ in the [Hyperband algorithm](https://arxiv.org/pdf/1603.06560.pdf). In this example, we used `STEPS` in combination with `R` to determine the number of epochs that SVD will run for in every trial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Hyperband\n",
    "config['advisor'] = {\n",
    "  'builtinAdvisorName': 'Hyperband',\n",
    "  'classArgs': {\n",
    "    'R': NUM_EPOCHS,\n",
    "    'eta': 3,\n",
    "    'optimize_mode': 'maximize'\n",
    "  }\n",
    "}\n",
    "config.pop('tuner')\n",
    "with open(config_path, 'w') as fp:\n",
    "    fp.write(yaml.dump(config, default_flow_style=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "stop_nni()\n",
    "with Timer() as time_hyperband:\n",
    "    start_nni(config_path, wait=WAITING_TIME, max_retries=MAX_RETRIES)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "check_metrics_written()\n",
    "svd = surprise.dump.load(os.path.join(get_trials('maximize')[3], \"model.dump\"))[1]\n",
    "test_results_hyperband = compute_test_results(svd)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_results_tpe.update({'time': time_tpe.interval})\n",
    "test_results_random.update({'time': time_random.interval})\n",
    "test_results_anneal.update({'time': time_anneal.interval})\n",
    "test_results_evolution.update({'time': time_evolution.interval})\n",
    "#test_results_smac.update({'time': time_smac.interval})\n",
    "test_results_metis.update({'time': time_metis.interval})\n",
    "test_results_hyperband.update({'time': time_hyperband.interval})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "algos = [\"TPE\", \n",
    "         \"Random Search\", \n",
    "         \"Annealing\", \n",
    "         \"Evolution\", \n",
    "         #\"SMAC\", \n",
    "         \"Metis\", \n",
    "         \"Hyperband\"]\n",
    "res_df = pd.DataFrame(index=algos,\n",
    "                      data=[res for res in [test_results_tpe, \n",
    "                                            test_results_random, \n",
    "                                            test_results_anneal, \n",
    "                                            test_results_evolution, \n",
    "                                            #test_results_smac, \n",
    "                                            test_results_metis, \n",
    "                                            test_results_hyperband]] \n",
    "                     )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>rmse</th>\n",
       "      <th>precision_at_k</th>\n",
       "      <th>ndcg_at_k</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>TPE</th>\n",
       "      <td>0.990</td>\n",
       "      <td>0.077</td>\n",
       "      <td>0.085</td>\n",
       "      <td>80.076</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Metis</th>\n",
       "      <td>0.944</td>\n",
       "      <td>0.076</td>\n",
       "      <td>0.085</td>\n",
       "      <td>105.617</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Random Search</th>\n",
       "      <td>0.974</td>\n",
       "      <td>0.052</td>\n",
       "      <td>0.051</td>\n",
       "      <td>84.977</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Hyperband</th>\n",
       "      <td>1.863</td>\n",
       "      <td>0.033</td>\n",
       "      <td>0.032</td>\n",
       "      <td>104.792</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Evolution</th>\n",
       "      <td>0.932</td>\n",
       "      <td>0.026</td>\n",
       "      <td>0.025</td>\n",
       "      <td>104.768</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Annealing</th>\n",
       "      <td>0.961</td>\n",
       "      <td>0.015</td>\n",
       "      <td>0.014</td>\n",
       "      <td>105.013</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                rmse  precision_at_k  ndcg_at_k     time\n",
       "TPE            0.990           0.077      0.085   80.076\n",
       "Metis          0.944           0.076      0.085  105.617\n",
       "Random Search  0.974           0.052      0.051   84.977\n",
       "Hyperband      1.863           0.033      0.032  104.792\n",
       "Evolution      0.932           0.026      0.025  104.768\n",
       "Annealing      0.961           0.015      0.014  105.013"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res_df.sort_values(by=\"precision_at_k\", ascending=False).round(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we see in the table above, _TPE_ performs best with respect to the primary metric (precision@10) that all the tuners optimized. Also the best NDCG@10 is obtained for TPE and correlates well with precision@10. RMSE on the other hand does not correlate well and is not optimized for TPE, since finding the top k recommendations in the right order is a different task from predicting ratings (high and low) accurately.     \n",
    "We have also observed that the above ranking of the tuners is not consistent and may change when trying these experiments multiple times. Since some of these tuners rely heavily on randomized sampling, a larger number of trials is required to get more consistent metrics.\n",
    "In addition, some of the tuning algorithms themselves come with parameters, which can affect their performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Stop the NNI experiment \n",
    "stop_nni()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "tmp_dir.cleanup()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Concluding Remarks\n",
    "\n",
    "We showed how to tune **all** the hyperparameters accepted by Surprise SVD simultaneously, by utilizing the NNI toolkit. \n",
    "For example, training and evaluation of a single SVD model takes about 50 seconds on the 100k MovieLens data on a Standard D2_V2 VM. Searching through 100 different combinations of hyperparameters sequentially would take about 80 minutes whereas each of the above experiments took about 10 minutes by exploiting parallelization on a single D16_v3 VM. With NNI, one can take advantage of concurrency and multiple processors on a virtual machine and can use a variety of tuning methods to navigate efficiently through a large space of hyperparameters.<br>\n",
    "For examples of scaling larger tuning workloads on clusters of machines, see [the notebooks](./README.md) that employ the [Azure Machine Learning service](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-tune-hyperparameters).  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References\n",
    "\n",
    "* [Matrix factorization algorithms in Surprise](https://surprise.readthedocs.io/en/stable/matrix_factorization.html) \n",
    "* [Surprise SVD deep-dive notebook](../02_model/surprise_svd_deep_dive.ipynb)\n",
    "* [Neural Network Intelligence toolkit](https://github.com/Microsoft/nni)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python (reco_base)",
   "language": "python",
   "name": "reco_base"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}