inf-sele-hyperparam-tuning-for-umap-k-means.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "37ec4c7a",
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
    "papermill": {
     "duration": 0.031301,
     "end_time": "2022-04-16T12:38:48.497309",
     "exception": false,
     "start_time": "2022-04-16T12:38:48.466008",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Introduction\n",
    "\n",
    "### UMAP + K-Means Clustering\n",
    "\n",
    "* `n_neighbors`: This determines the number of neighboring points used in local approximations of manifold structure. Larger values will result in more global structure being preserved at the loss of detailed local structure. In general this parameter should often be in the **range 5 to 50**, with a choice of 10 to 15 being a sensible default.\n",
    "\n",
    "* `min_dist`: This controls how tightly the embedding is allowed compress points together. **Larger values** ensure embedded points are **more evenly distributed**, while smaller values allow the algorithm to optimise more accurately with regard to local structure. Sensible values are in the range 0.001 to 0.5, with 0.1 being a reasonable default.\n",
    "\n",
    "* `metric`: This determines the choice of metric used to measure distance in the input space. A wide variety of metrics are already coded, and a user defined function can be passed as long as it has been JITd by numba.\n",
    "\n",
    "`['euclidean','manhattan','chebyshev','minkowski','canberra','braycurtis','mahalanobis','wminkowski',,'cosine'，'correlation','haversine', 'hamming','jaccard','dice','russelrao', 'kulsinski', 'll_dirichlet', 'hellinger' ]`\n",
    "\n",
    "euclidean\n",
    "manhattan\n",
    "chebyshev\n",
    "minkowski\n",
    "canberra\n",
    "braycurtis\n",
    "~~mahalanobis~~\n",
    "wminkowski\n",
    "~~seuclidean~~\n",
    "cosine\n",
    "correlation\n",
    "haversine\n",
    "hamming\n",
    "jaccard\n",
    "dice\n",
    "~~russelrao~~\n",
    "kulsinski\n",
    "ll_dirichlet\n",
    "hellinger\n",
    "rogerstanimoto\n",
    "sokalmichener\n",
    "sokalsneath\n",
    "yule\n",
    "\n",
    "[UMAP API Guide](https://umap.scikit-tda.org/api.html)\n",
    "\n",
    "* `n_components`: 2-3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "485483b9",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-16T12:38:48.564776Z",
     "iopub.status.busy": "2022-04-16T12:38:48.562980Z",
     "iopub.status.idle": "2022-04-16T12:40:38.088160Z",
     "shell.execute_reply": "2022-04-16T12:40:38.087421Z",
     "shell.execute_reply.started": "2022-04-16T09:33:48.888630Z"
    },
    "papermill": {
     "duration": 109.559333,
     "end_time": "2022-04-16T12:40:38.088353",
     "exception": false,
     "start_time": "2022-04-16T12:38:48.529020",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting hdbscan\r\n",
      "  Downloading hdbscan-0.8.28.tar.gz (5.2 MB)\r\n",
      "     |████████████████████████████████| 5.2 MB 2.0 MB/s            \r\n",
      "\u001b[?25h  Installing build dependencies ... \u001b[?25l-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \bdone\r\n",
      "\u001b[?25h  Getting requirements to build wheel ... \u001b[?25l-\b \b\\\b \b|\b \b/\b \bdone\r\n",
      "\u001b[?25h  Preparing metadata (pyproject.toml) ... \u001b[?25l-\b \b\\\b \b|\b \b/\b \bdone\r\n",
      "\u001b[?25hRequirement already satisfied: joblib>=1.0 in /opt/conda/lib/python3.7/site-packages (from hdbscan) (1.1.0)\r\n",
      "Requirement already satisfied: cython>=0.27 in /opt/conda/lib/python3.7/site-packages (from hdbscan) (0.29.28)\r\n",
      "Requirement already satisfied: scipy>=1.0 in /opt/conda/lib/python3.7/site-packages (from hdbscan) (1.7.3)\r\n",
      "Requirement already satisfied: numpy>=1.20 in /opt/conda/lib/python3.7/site-packages (from hdbscan) (1.20.3)\r\n",
      "Requirement already satisfied: scikit-learn>=0.20 in /opt/conda/lib/python3.7/site-packages (from hdbscan) (1.0.1)\r\n",
      "Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from scikit-learn>=0.20->hdbscan) (3.0.0)\r\n",
      "Building wheels for collected packages: hdbscan\r\n",
      "  Building wheel for hdbscan (pyproject.toml) ... \u001b[?25l-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \b|\b \b/\b \b-\b \b\\\b \bdone\r\n",
      "\u001b[?25h  Created wheel for hdbscan: filename=hdbscan-0.8.28-cp37-cp37m-linux_x86_64.whl size=3413272 sha256=b8fd4b13b83b8b6eb269d0e06031735e5ed20425618afad9ca554f238fbe2b15\r\n",
      "  Stored in directory: /root/.cache/pip/wheels/6e/7a/5e/259ccc841c085fc41b99ef4a71e896b62f5161f2bc8a14c97a\r\n",
      "Successfully built hdbscan\r\n",
      "Installing collected packages: hdbscan\r\n",
      "Successfully installed hdbscan-0.8.28\r\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\r\n",
      "CPU times: user 2.31 s, sys: 506 ms, total: 2.82 s\n",
      "Wall time: 1min 49s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "! pip install hdbscan\n",
    "# ! conda install -y gdown\n",
    "# ! gdown --id 1fskNgcHG8kxHIOr6MlZ61OCnDE4DdhgU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0f1ad806",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-16T12:40:38.216788Z",
     "iopub.status.busy": "2022-04-16T12:40:38.215757Z",
     "iopub.status.idle": "2022-04-16T12:41:00.739362Z",
     "shell.execute_reply": "2022-04-16T12:41:00.738589Z",
     "shell.execute_reply.started": "2022-04-16T09:34:35.396021Z"
    },
    "papermill": {
     "duration": 22.598051,
     "end_time": "2022-04-16T12:41:00.739504",
     "exception": false,
     "start_time": "2022-04-16T12:40:38.141453",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# for everything else\n",
    "import os\n",
    "import random\n",
    "# from random import randint\n",
    "from functools import partial\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from pandas_profiling import ProfileReport\n",
    "\n",
    "# for loading/processing the images  \n",
    "# import tensorflow\n",
    "# from imageio import imread\n",
    "# import cv2\n",
    "from tensorflow import keras\n",
    "from keras.models import load_model\n",
    "from keras.preprocessing.image import load_img \n",
    "from keras.preprocessing.image import img_to_array \n",
    "from keras.applications.vgg16 import preprocess_input \n",
    "\n",
    "# models \n",
    "from keras.applications.vgg16 import VGG16 \n",
    "from keras.models import Model\n",
    "import pickle\n",
    "# from keras.layers import Activation, Dense, Dropout\n",
    "\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.manifold import TSNE\n",
    "import umap\n",
    "\n",
    "# clustering and dimension reduction\n",
    "from sklearn.cluster import AgglomerativeClustering\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.cluster import MiniBatchKMeans\n",
    "from sklearn.cluster import SpectralClustering\n",
    "import hdbscan\n",
    "\n",
    "from sklearn import metrics\n",
    "from sklearn.metrics import roc_curve, auc\n",
    "from sklearn.metrics import silhouette_score\n",
    "from sklearn.metrics import calinski_harabasz_score\n",
    "from sklearn.metrics import precision_score, recall_score, f1_score, accuracy_score\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.metrics import precision_recall_fscore_support\n",
    "# from sklearn.preprocessing import LabelEncoder\n",
    "# from keras.utils.np_utils import to_categorical\n",
    "from tqdm import trange\n",
    "from hyperopt import fmin, tpe, hp, STATUS_OK, space_eval, Trials\n",
    "\n",
    "import plotly.graph_objs as go\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "\n",
    "%matplotlib inline\n",
    "sns.set(style='white', context='notebook', rc={'figure.figsize':(14,10)})\n",
    "\n",
    "from warnings import filterwarnings\n",
    "filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "3acf4ecd",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-16T12:41:00.852619Z",
     "iopub.status.busy": "2022-04-16T12:41:00.851975Z",
     "iopub.status.idle": "2022-04-16T12:41:00.854499Z",
     "shell.execute_reply": "2022-04-16T12:41:00.853970Z",
     "shell.execute_reply.started": "2022-04-16T09:34:45.760263Z"
    },
    "papermill": {
     "duration": 0.059439,
     "end_time": "2022-04-16T12:41:00.854648",
     "exception": false,
     "start_time": "2022-04-16T12:41:00.795209",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# dr_methods = ['pca', 't-sne', 'umap']\n",
    "# clustering_methods = ['agglomerative',\n",
    "#                       'vanilla kmeans',\n",
    "#                       'MinibatchKMeans',\n",
    "#                       'spectral',\n",
    "#                       'hdbscan ']\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "496a650c",
   "metadata": {
    "papermill": {
     "duration": 0.052398,
     "end_time": "2022-04-16T12:41:00.962557",
     "exception": false,
     "start_time": "2022-04-16T12:41:00.910159",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Hyperparameters\n",
    "* agglomerative: `n_clusters` `affinity`\n",
    "* k-means: `n_initint`, `max_iter`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b321c9fb",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-16T12:41:01.079058Z",
     "iopub.status.busy": "2022-04-16T12:41:01.078176Z",
     "iopub.status.idle": "2022-04-16T12:41:01.079991Z",
     "shell.execute_reply": "2022-04-16T12:41:01.080503Z",
     "shell.execute_reply.started": "2022-04-16T09:34:45.766023Z"
    },
    "papermill": {
     "duration": 0.064542,
     "end_time": "2022-04-16T12:41:01.080697",
     "exception": false,
     "start_time": "2022-04-16T12:41:01.016155",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "config = {\n",
    "    'method': 'MinibatchKMeans',  #['agglomerative','kmeans','MinibatchKMeans','spectral','hdbscan '],\n",
    "    'output_features': 512,\n",
    "#     'dimension_redcution': dr_methods,\n",
    "#     'clustering': clustering_methods,\n",
    "    'n_neighbors': range(20,500,20),\n",
    "    'min_dist': np.arange(0.0, 1.0, 0.05),\n",
    "    'n_components': 2,\n",
    "    \"umap_metric\": ['manhattan','canberra','hellinger','correlation','jaccard','cosine','dice','ll_dirichlet','minkowski'],\n",
    "    'n_clusters': 4,\n",
    "    'n_init': 10,\n",
    "    'batch_size': 512, # 256 * number of cores\n",
    "    'min_cluster_size': 15,\n",
    "    'num_evals': 0,\n",
    "    'max_evals': 1000,\n",
    "    'random_state': 42\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "cd9a9764",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-16T12:41:01.196875Z",
     "iopub.status.busy": "2022-04-16T12:41:01.196141Z",
     "iopub.status.idle": "2022-04-16T12:41:01.204028Z",
     "shell.execute_reply": "2022-04-16T12:41:01.204543Z",
     "shell.execute_reply.started": "2022-04-16T09:34:45.790390Z"
    },
    "papermill": {
     "duration": 0.067654,
     "end_time": "2022-04-16T12:41:01.204726",
     "exception": false,
     "start_time": "2022-04-16T12:41:01.137072",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('space:', 4320)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "'space:', len(config['n_neighbors']) * len(config['min_dist']) * len(config['umap_metric'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82292202",
   "metadata": {
    "papermill": {
     "duration": 0.052896,
     "end_time": "2022-04-16T12:41:01.311351",
     "exception": false,
     "start_time": "2022-04-16T12:41:01.258455",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Load Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "024ffb1a",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-16T12:41:01.421885Z",
     "iopub.status.busy": "2022-04-16T12:41:01.421129Z",
     "iopub.status.idle": "2022-04-16T12:41:01.599176Z",
     "shell.execute_reply": "2022-04-16T12:41:01.598607Z",
     "shell.execute_reply.started": "2022-04-16T09:42:46.873537Z"
    },
    "papermill": {
     "duration": 0.234458,
     "end_time": "2022-04-16T12:41:01.599320",
     "exception": false,
     "start_time": "2022-04-16T12:41:01.364862",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "base_dir = '../input/nbiinfframes/FRAMES'\n",
    "Fold1 = '../input/nbiinfframes/FRAMES/Fold1'\n",
    "Fold2 = '../input/nbiinfframes/FRAMES/Fold2'\n",
    "Fold3 = '../input/nbiinfframes/FRAMES/Fold3'\n",
    "\n",
    "model_path = r'vgg16_weights_tf_dim_ordering_tf_kernels.h5'\n",
    "pkl_file = r\"../input/feature-embedding/files_feature_embeddings_vgg16.pkl\"\n",
    "\n",
    "class_ = ['B', 'I', 'S', 'U']\n",
    "Folds = [Fold1, Fold2, Fold3]\n",
    "all_data = []\n",
    "\n",
    "# Define path to the data directory\n",
    "for fold in Folds:\n",
    "    for case in class_:\n",
    "        path = os.path.join(fold, case)\n",
    "        if os.path.isdir(path):\n",
    "            for img in os.listdir(path):\n",
    "                all_data.append([os.path.join(path, img), case, os.path.basename(fold)])\n",
    "        \n",
    "all_df = pd.DataFrame(all_data, columns=['image_path', 'class', 'folder'],index=None)\n",
    "all_df.dropna(inplace = True)\n",
    "# all_df.sample(10)\n",
    "\n",
    "#mapping the image name with the image path\n",
    "all_df['name'] = all_df['image_path'].map(lambda x: os.path.basename(x))\n",
    "img_name = all_df['name'] .to_list()\n",
    "files = all_df['image_path'].to_list()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "8169105c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-16T12:41:01.709905Z",
     "iopub.status.busy": "2022-04-16T12:41:01.709231Z",
     "iopub.status.idle": "2022-04-16T12:41:01.710839Z",
     "shell.execute_reply": "2022-04-16T12:41:01.711385Z",
     "shell.execute_reply.started": "2022-04-16T09:34:45.854576Z"
    },
    "papermill": {
     "duration": 0.059106,
     "end_time": "2022-04-16T12:41:01.711563",
     "exception": false,
     "start_time": "2022-04-16T12:41:01.652457",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# all_df['target'] = all_df['class'].astype('category').cat.codes\n",
    "# all_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c56d8b3d",
   "metadata": {
    "papermill": {
     "duration": 0.056693,
     "end_time": "2022-04-16T12:41:01.821184",
     "exception": false,
     "start_time": "2022-04-16T12:41:01.764491",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Methods: Building Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "423e5358",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-16T12:41:01.931770Z",
     "iopub.status.busy": "2022-04-16T12:41:01.931153Z",
     "iopub.status.idle": "2022-04-16T12:41:06.946338Z",
     "shell.execute_reply": "2022-04-16T12:41:06.947108Z",
     "shell.execute_reply.started": "2022-04-16T09:43:19.553912Z"
    },
    "papermill": {
     "duration": 5.072952,
     "end_time": "2022-04-16T12:41:06.947443",
     "exception": false,
     "start_time": "2022-04-16T12:41:01.874491",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-04-16 12:41:01.993737: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/vgg16/vgg16_weights_tf_dim_ordering_tf_kernels.h5\n",
      "553467904/553467096 [==============================] - 3s 0us/step\n",
      "553476096/553467096 [==============================] - 3s 0us/step\n",
      "4096\n",
      "CPU times: user 4.03 s, sys: 3.61 s, total: 7.63 s\n",
      "Wall time: 5.01 s\n"
     ]
    }
   ],
   "source": [
    "%%time \n",
    "\n",
    "# if os.path.exists(model_path):\n",
    "#     model = load_model(model_path)\n",
    "# else:\n",
    "#     print('model path do not exsit, downloading the model')\n",
    "model = VGG16()\n",
    "model = Model(inputs = model.inputs, outputs = model.layers[-2].output)\n",
    "    \n",
    "out_features = model.output.shape[1] # to obtain the number of the output feature\n",
    "# model.summary()\n",
    "print(out_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19496eed",
   "metadata": {
    "papermill": {
     "duration": 0.081969,
     "end_time": "2022-04-16T12:41:07.113331",
     "exception": false,
     "start_time": "2022-04-16T12:41:07.031362",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Feature Extraction by CNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "61410b7f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-16T12:41:07.290679Z",
     "iopub.status.busy": "2022-04-16T12:41:07.289615Z",
     "iopub.status.idle": "2022-04-16T12:41:07.292462Z",
     "shell.execute_reply": "2022-04-16T12:41:07.291878Z"
    },
    "papermill": {
     "duration": 0.097129,
     "end_time": "2022-04-16T12:41:07.292615",
     "exception": false,
     "start_time": "2022-04-16T12:41:07.195486",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def extract_features(files, model, pkl=\"\", out_feature=out_features):\n",
    "    \n",
    "    data = {}\n",
    "    if os.path.exists(pkl):\n",
    "        with open(pkl, 'rb') as f:\n",
    "            data = pickle.load(f)\n",
    "    else:\n",
    "        print(\"File data pkl file doesn't exist..., extracting by model\")\n",
    "        for file in files:\n",
    "            # load the image as a 224x224 array\n",
    "            resized_img = load_img(file, target_size=(224,224)) ##reshaped\n",
    "            # convert from 'PIL.Image.Image' to numpy array\n",
    "            img_arr = np.array(resized_img) \n",
    "            # reshape the data for the model reshape(num_of_samples, dim 1, dim 2, channels)\n",
    "            reshaped_img = img_arr.reshape(1,224,224,3)  ##reshape\n",
    "            # prepare image for model\n",
    "            imgx = preprocess_input(reshaped_img)\n",
    "            ## define the method to extract the feature vector\n",
    "            embeddings = model.predict(imgx, use_multiprocessing=True)\n",
    "            data[os.path.basename(file)] = embeddings\n",
    "            \n",
    "        pickle.dump(data, open(pkl, \"wb\"))\n",
    " \n",
    "    # get a list of the filenames\n",
    "    filenames = np.array(list(data.keys()))\n",
    "    # get a list of just the features\n",
    "    feat = np.array(list(data.values()))\n",
    "    # reshape so that there are 720 samples of 512 vectors\n",
    "    feat = feat.reshape(-1, out_features)\n",
    "        \n",
    "    return filenames, feat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d88d5462",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-16T12:41:07.462315Z",
     "iopub.status.busy": "2022-04-16T12:41:07.461662Z",
     "iopub.status.idle": "2022-04-16T12:41:07.680656Z",
     "shell.execute_reply": "2022-04-16T12:41:07.681123Z"
    },
    "papermill": {
     "duration": 0.306491,
     "end_time": "2022-04-16T12:41:07.681298",
     "exception": false,
     "start_time": "2022-04-16T12:41:07.374807",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "extracted feature shape: (720, 4096)\n",
      "CPU times: user 10.6 ms, sys: 14.8 ms, total: 25.4 ms\n",
      "Wall time: 216 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "filenames, feat = extract_features(files, model, pkl_file, out_features)    \n",
    "print('extracted feature shape: {}'.format(feat.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33c979f8",
   "metadata": {
    "papermill": {
     "duration": 0.082291,
     "end_time": "2022-04-16T12:41:07.846633",
     "exception": false,
     "start_time": "2022-04-16T12:41:07.764342",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Different embeddings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "840e8947",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-16T12:41:08.022172Z",
     "iopub.status.busy": "2022-04-16T12:41:08.020544Z",
     "iopub.status.idle": "2022-04-16T12:41:08.025480Z",
     "shell.execute_reply": "2022-04-16T12:41:08.024919Z"
    },
    "papermill": {
     "duration": 0.094013,
     "end_time": "2022-04-16T12:41:08.025643",
     "exception": false,
     "start_time": "2022-04-16T12:41:07.931630",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# def embed(model, model_type, sentences):\n",
    "#     if model_type == 'use':\n",
    "#         embeddings = model(sentences)\n",
    "#     elif model_type == 'sentence transformer':\n",
    "#         embeddings = model.encode(sentences, show_progress_bar=True)\n",
    "    \n",
    "#     return embeddings"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77112069",
   "metadata": {
    "papermill": {
     "duration": 0.086001,
     "end_time": "2022-04-16T12:41:08.199469",
     "exception": false,
     "start_time": "2022-04-16T12:41:08.113468",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Different Clusters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "63c109b1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-16T12:41:08.379086Z",
     "iopub.status.busy": "2022-04-16T12:41:08.378420Z",
     "iopub.status.idle": "2022-04-16T12:41:08.387742Z",
     "shell.execute_reply": "2022-04-16T12:41:08.388282Z"
    },
    "papermill": {
     "duration": 0.10224,
     "end_time": "2022-04-16T12:41:08.388459",
     "exception": false,
     "start_time": "2022-04-16T12:41:08.286219",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def generate_kmclusters(feature_embeddings,\n",
    "                       n_neighbors,\n",
    "                       min_dist,\n",
    "                       metric='euclidean',\n",
    "                       random_state = config['random_state'],\n",
    "                       n_components = config['n_components'],\n",
    "                       method = config['method'],\n",
    "                       n_clusters = config['n_clusters'],\n",
    "                       n_init = config['n_init'],batch_size = config['batch_size']):\n",
    "    \n",
    "    umap_embeddings = umap.UMAP(n_neighbors = n_neighbors,\n",
    "                                min_dist = min_dist,\n",
    "                                n_components = n_components,\n",
    "                                metric = metric,\n",
    "                                random_state=random_state).fit_transform(feature_embeddings)\n",
    "    \n",
    "    if method == 'kmeans':\n",
    "        clusters = KMeans(init='k-means++',\n",
    "                    n_clusters = n_clusters,\n",
    "                    max_iter=400,\n",
    "                    n_init=n_init,\n",
    "                    random_state=random_state).fit(umap_embeddings)\n",
    "        \n",
    "    elif method == 'MinibatchKMeans':\n",
    "        clusters = MiniBatchKMeans(init=\"k-means++\",\n",
    "                          n_clusters = n_clusters,\n",
    "                          batch_size = batch_size,\n",
    "                          max_iter=400,\n",
    "                          n_init = n_init,\n",
    "                          max_no_improvement=10,\n",
    "                          verbose=0,random_state=random_state).fit(umap_embeddings)\n",
    "\n",
    "    return clusters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "e4f7e2b5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-16T12:41:08.567017Z",
     "iopub.status.busy": "2022-04-16T12:41:08.566277Z",
     "iopub.status.idle": "2022-04-16T12:41:08.568344Z",
     "shell.execute_reply": "2022-04-16T12:41:08.568927Z"
    },
    "papermill": {
     "duration": 0.096733,
     "end_time": "2022-04-16T12:41:08.569099",
     "exception": false,
     "start_time": "2022-04-16T12:41:08.472366",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def generate_scclusters( feature_embeddings,\n",
    "                       n_neighbors,\n",
    "                       min_dist,\n",
    "                       metric='euclidean',\n",
    "                       random_state = config['random_state'],\n",
    "                       n_components = config['n_components'],\n",
    "                       n_clusters = config['n_clusters'],\n",
    "                       min_cluster_size = config['min_cluster_size']):\n",
    "        \n",
    "    umap_embeddings = umap.UMAP(n_neighbors = n_neighbors,\n",
    "                                min_dist = min_dist,\n",
    "                                n_components = n_components,\n",
    "                                metric = metric,\n",
    "                                random_state = random_state).fit_transform(feature_embeddings)\n",
    "    \n",
    "    sc_umap_cluster = SpectralClustering(n_clusters = n_clusters,\n",
    "                                       n_neighbors = config['n_neighbors'],\n",
    "                                       affinity='precomputed',\n",
    "                                       n_init=100, \n",
    "                                       assign_labels='discretize',\n",
    "                                      random_state=random_state).fit(umap_embeddings)\n",
    "\n",
    "    clusters = sc_umap_cluster\n",
    "    return clusters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "d8a0c23b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-16T12:41:08.743136Z",
     "iopub.status.busy": "2022-04-16T12:41:08.742465Z",
     "iopub.status.idle": "2022-04-16T12:41:08.750009Z",
     "shell.execute_reply": "2022-04-16T12:41:08.750481Z"
    },
    "papermill": {
     "duration": 0.096533,
     "end_time": "2022-04-16T12:41:08.750677",
     "exception": false,
     "start_time": "2022-04-16T12:41:08.654144",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def generate_hdclusters( feature_embeddings,\n",
    "                       n_neighbors,\n",
    "                       min_dist,\n",
    "                       metric='euclidean',\n",
    "                       random_state = config['random_state'],\n",
    "                       n_components = config['n_components'],\n",
    "                       n_clusters = config['n_clusters'],\n",
    "                       min_cluster_size = config['min_cluster_size']):\n",
    "\n",
    "    print('the using metric {}'.format(metric))\n",
    "    \n",
    "    umap_embeddings = umap.UMAP(n_neighbors = n_neighbors,\n",
    "                                min_dist = min_dist,\n",
    "                                n_components = n_components,\n",
    "                                metric = metric,\n",
    "                                random_state=random_state).fit_transform(feature_embeddings)\n",
    "\n",
    "    hd = hdbscan.HDBSCAN(min_cluster_size = min_cluster_size ,\n",
    "                               metric='euclidean', \n",
    "                               cluster_selection_method='eom').fit(umap_embeddings)\n",
    "        \n",
    "    clusters = hd\n",
    "    return clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "486d3dcd",
   "metadata": {
    "papermill": {
     "duration": 0.086406,
     "end_time": "2022-04-16T12:41:08.922390",
     "exception": false,
     "start_time": "2022-04-16T12:41:08.835984",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "### Score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "0eb9d5d1",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-16T12:41:09.101125Z",
     "iopub.status.busy": "2022-04-16T12:41:09.100401Z",
     "iopub.status.idle": "2022-04-16T12:41:09.118185Z",
     "shell.execute_reply": "2022-04-16T12:41:09.118702Z"
    },
    "papermill": {
     "duration": 0.108335,
     "end_time": "2022-04-16T12:41:09.118892",
     "exception": false,
     "start_time": "2022-04-16T12:41:09.010557",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def metrics_score_(clusters, filenames, image_label_df):\n",
    "    \n",
    "    \"\"\"\n",
    "    Returns  objects after first performing dimensionality reduction using UMAP\n",
    "    \n",
    "    Arguments:\n",
    "        clusters: clustering object\n",
    "        filenames: image name list\n",
    "        image_label_df: dataframe object, the image name and the groundtruth label\n",
    "        \n",
    "    Returns:\n",
    "        cluster_labels: K-Means object of clusters\n",
    "    \"\"\"\n",
    "    mapped_label = {}\n",
    "    ### prepare the labels from clusters\n",
    "    clustered_labels = clusters.labels_ #clustered labels\n",
    "    clustered_groups = {}\n",
    "    for file, cluster in zip(filenames, clustered_labels):\n",
    "        if cluster not in clustered_groups.keys():\n",
    "            clustered_groups[cluster] = []\n",
    "            clustered_groups[cluster].append(file)\n",
    "        else:\n",
    "            clustered_groups[cluster].append(file)\n",
    "    \n",
    "    \n",
    "    ### prepare the ground truth labels, using group by \n",
    "    image_groups = image_label_df.groupby('class')['name'] ## only get the image name for matching the predicted one\n",
    "    ground_labels = [*image_groups.groups.keys()] \n",
    "    \n",
    "    ## mapping each group in datafame with the max intersection clustered groups\n",
    "    for label in ground_labels:\n",
    "        group_label = {} # allocate to the max intersection number group\n",
    "        lst = image_groups.get_group(label).tolist()\n",
    "        \n",
    "        for x in range(len(clustered_groups)):\n",
    "            num = len(set(clustered_groups[x]) & set(lst))\n",
    "#             print('clustered group: {} intersection with label {}, instersection number {}'.format(x, label, num))\n",
    "            group_label[x]=num\n",
    "            \n",
    "        max_key = max(group_label, key=group_label.get)\n",
    "#         print('the {} label is the group {}'.format(label, max_key))\n",
    "        mapped_label[max_key] = label #reverse the label\n",
    "        \n",
    "        ### connect the mapped labels to the image name\n",
    "        np_groups = np.array(list(clustered_groups.items()))\n",
    "        imgs = []\n",
    "        clustered_img_labels = []\n",
    "        for i in range(np_groups.shape[0]):\n",
    "            imgs.extend(np_groups[i,:][1])\n",
    "            clustered_img_labels.extend([np_groups[i,:][0]]*len(np_groups[i,:][1]))\n",
    "        \n",
    "        ## merge the mapped df with the dataframe\n",
    "        clustered_df = pd.DataFrame({'name': imgs, 'predict_target': clustered_img_labels})\n",
    "        clustered_df['predict_class'] = clustered_df['predict_target'].map(mapped_label)\n",
    "        eval_df = pd.merge(image_label_df, clustered_df, on='name', how='left')\n",
    "        \n",
    "        y_test = eval_df['class'].tolist()\n",
    "        y_pred = eval_df['predict_class'].tolist()\n",
    "        \n",
    "        report = classification_report(y_test, y_pred)\n",
    "\n",
    "        precision_score_ = precision_recall_fscore_support(y_test, y_pred)[0]\n",
    "        recall_score_ = precision_recall_fscore_support(y_test, y_pred)[1]\n",
    "        f1_score_ = precision_recall_fscore_support(y_test, y_pred)[2]\n",
    "\n",
    "        avg_precision = np.average(precision_score_)\n",
    "        avg_recall = np.average(recall_score_)\n",
    "        avg_f1 = np.average(f1_score_)\n",
    "        \n",
    "    return avg_precision, avg_recall, avg_f1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "6ffe7a56",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-16T12:41:09.299817Z",
     "iopub.status.busy": "2022-04-16T12:41:09.299132Z",
     "iopub.status.idle": "2022-04-16T12:41:09.301810Z",
     "shell.execute_reply": "2022-04-16T12:41:09.302367Z"
    },
    "papermill": {
     "duration": 0.093175,
     "end_time": "2022-04-16T12:41:09.302532",
     "exception": false,
     "start_time": "2022-04-16T12:41:09.209357",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# %time clusters = generate_kmclusters(feat, min_dist=0, metric='euclidean') #default: euclidean\n",
    "# %time cost = score_(clusters, filenames, all_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "b1b40cec",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-16T12:41:09.479082Z",
     "iopub.status.busy": "2022-04-16T12:41:09.478403Z",
     "iopub.status.idle": "2022-04-16T12:41:09.486189Z",
     "shell.execute_reply": "2022-04-16T12:41:09.486820Z"
    },
    "papermill": {
     "duration": 0.096251,
     "end_time": "2022-04-16T12:41:09.486990",
     "exception": false,
     "start_time": "2022-04-16T12:41:09.390739",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def objective(params, embeddings, filenames, df):\n",
    "    \"\"\"\n",
    "    Objective function for hyperopt to minimize, which incorporates constraints\n",
    "    on the number of clusters we want to identify\n",
    "    \"\"\"\n",
    "    \n",
    "    clusters = generate_kmclusters(embeddings,\n",
    "                                   n_neighbors = params['n_neighbors'], \n",
    "                                   min_dist = params['min_dist'], \n",
    "                                   metric = params['umap_metric'])\n",
    "#     print(params['umap_metric'])\n",
    "    \n",
    "    avg_precision, avg_recall, avg_f1 = metrics_score_(clusters, filenames, df)\n",
    "    cost = 1 - avg_recall \n",
    "    \n",
    "    if (np.abs(avg_precision - avg_recall) > 0.025):\n",
    "        penalty = 0.1 * np.abs(avg_precision - avg_recall)\n",
    "    else:\n",
    "        penalty = 0\n",
    "    loss = cost + penalty # penalty\n",
    "    return {'loss': loss, 'avg_precision': avg_precision,'avg_recall': avg_recall, 'avg_f1':avg_f1, 'status': STATUS_OK}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "c0315fc5",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-16T12:41:09.660508Z",
     "iopub.status.busy": "2022-04-16T12:41:09.659874Z",
     "iopub.status.idle": "2022-04-16T12:41:09.667000Z",
     "shell.execute_reply": "2022-04-16T12:41:09.667625Z"
    },
    "papermill": {
     "duration": 0.096527,
     "end_time": "2022-04-16T12:41:09.667801",
     "exception": false,
     "start_time": "2022-04-16T12:41:09.571274",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def bayesian_search(features, filenames, df, space, max_evals=100):\n",
    "\n",
    "    \"\"\"\n",
    "    Perform bayseian search on hyperopt hyperparameter space to minimize objective function\n",
    "    \"\"\"\n",
    "    \n",
    "    trials = Trials()\n",
    "    \n",
    "    fmin_objective = partial(objective, embeddings=features, filenames=filenames, df=df)\n",
    "    \n",
    "    best = fmin(fmin_objective, \n",
    "                space = space, \n",
    "                algo=tpe.suggest,\n",
    "                max_evals=max_evals, \n",
    "                trials=trials)\n",
    "\n",
    "    best_params = space_eval(space, best)\n",
    "    print ('best:')\n",
    "    print (best_params)\n",
    "    print (f\"the average recall: {trials.best_trial['result']['avg_recall']}\")\n",
    "\n",
    "    best_clusters = generate_kmclusters(features, \n",
    "                                      n_neighbors = best_params['n_neighbors'], \n",
    "                                      min_dist = best_params['min_dist'], \n",
    "                                      metric = best_params['umap_metric'])\n",
    "    \n",
    "\n",
    "    return best_params, best_clusters, trials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "4ba19516",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-16T12:41:09.923336Z",
     "iopub.status.busy": "2022-04-16T12:41:09.921402Z",
     "iopub.status.idle": "2022-04-16T12:41:09.926496Z",
     "shell.execute_reply": "2022-04-16T12:41:09.927157Z"
    },
    "papermill": {
     "duration": 0.164119,
     "end_time": "2022-04-16T12:41:09.927324",
     "exception": false,
     "start_time": "2022-04-16T12:41:09.763205",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "hspace = {\n",
    "    \"n_neighbors\": hp.choice('n_neighbors', config['n_neighbors']),\n",
    "    \"min_dist\": hp.choice('min_dist', config['min_dist']),\n",
    "    \"umap_metric\": hp.choice('umap_metric', config['umap_metric']),\n",
    "}\n",
    "\n",
    "max_evals=config['max_evals']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "e7993efd",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-16T12:41:10.109012Z",
     "iopub.status.busy": "2022-04-16T12:41:10.108040Z",
     "iopub.status.idle": "2022-04-16T16:17:02.870903Z",
     "shell.execute_reply": "2022-04-16T16:17:02.870300Z"
    },
    "papermill": {
     "duration": 12952.854347,
     "end_time": "2022-04-16T16:17:02.871111",
     "exception": false,
     "start_time": "2022-04-16T12:41:10.016764",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test the time-cost for the method MinibatchKMeans\n",
      "100%|██████████| 1000/1000 [3:35:38<00:00, 12.94s/trial, best loss: 0.07222222222222219]\n",
      "best:\n",
      "{'min_dist': 0.30000000000000004, 'n_neighbors': 360, 'umap_metric': 'canberra'}\n",
      "the average recall: 0.9277777777777778\n",
      "CPU times: user 3h 36min 9s, sys: 1min 48s, total: 3h 37min 58s\n",
      "Wall time: 3h 35min 52s\n"
     ]
    }
   ],
   "source": [
    "print('test the time-cost for the method {}'.format(config['method']))\n",
    "%time best_params_use, best_clusters_use, trials_use = bayesian_search(feat, filenames, all_df, hspace, max_evals)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "a989d263",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-16T16:17:03.820191Z",
     "iopub.status.busy": "2022-04-16T16:17:03.817214Z",
     "iopub.status.idle": "2022-04-16T16:17:03.824990Z",
     "shell.execute_reply": "2022-04-16T16:17:03.824411Z"
    },
    "papermill": {
     "duration": 0.477932,
     "end_time": "2022-04-16T16:17:03.825132",
     "exception": false,
     "start_time": "2022-04-16T16:17:03.347200",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'state': 2,\n",
       " 'tid': 57,\n",
       " 'spec': None,\n",
       " 'result': {'loss': 0.07222222222222219,\n",
       "  'avg_precision': 0.9310420307600001,\n",
       "  'avg_recall': 0.9277777777777778,\n",
       "  'avg_f1': 0.9262374103130735,\n",
       "  'status': 'ok'},\n",
       " 'misc': {'tid': 57,\n",
       "  'cmd': ('domain_attachment', 'FMinIter_Domain'),\n",
       "  'workdir': None,\n",
       "  'idxs': {'min_dist': [57], 'n_neighbors': [57], 'umap_metric': [57]},\n",
       "  'vals': {'min_dist': [6], 'n_neighbors': [17], 'umap_metric': [1]}},\n",
       " 'exp_key': None,\n",
       " 'owner': None,\n",
       " 'version': 0,\n",
       " 'book_time': datetime.datetime(2022, 4, 16, 12, 51, 43, 597000),\n",
       " 'refresh_time': datetime.datetime(2022, 4, 16, 12, 51, 56, 846000)}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trials_use.best_trial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "4a99ac92",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-16T16:17:04.746863Z",
     "iopub.status.busy": "2022-04-16T16:17:04.746162Z",
     "iopub.status.idle": "2022-04-16T16:17:04.750337Z",
     "shell.execute_reply": "2022-04-16T16:17:04.749807Z"
    },
    "papermill": {
     "duration": 0.465182,
     "end_time": "2022-04-16T16:17:04.750495",
     "exception": false,
     "start_time": "2022-04-16T16:17:04.285313",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'0.5.2'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    " umap.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "966f645f",
   "metadata": {
    "papermill": {
     "duration": 0.46161,
     "end_time": "2022-04-16T16:17:05.672072",
     "exception": false,
     "start_time": "2022-04-16T16:17:05.210462",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "b758e7e2",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-16T16:17:06.612448Z",
     "iopub.status.busy": "2022-04-16T16:17:06.611790Z",
     "iopub.status.idle": "2022-04-16T16:17:06.615094Z",
     "shell.execute_reply": "2022-04-16T16:17:06.614488Z"
    },
    "papermill": {
     "duration": 0.47047,
     "end_time": "2022-04-16T16:17:06.615253",
     "exception": false,
     "start_time": "2022-04-16T16:17:06.144783",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# random_use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "7d2956e6",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-16T16:17:07.553820Z",
     "iopub.status.busy": "2022-04-16T16:17:07.553026Z",
     "iopub.status.idle": "2022-04-16T16:17:07.557232Z",
     "shell.execute_reply": "2022-04-16T16:17:07.556709Z"
    },
    "papermill": {
     "duration": 0.471591,
     "end_time": "2022-04-16T16:17:07.557380",
     "exception": false,
     "start_time": "2022-04-16T16:17:07.085789",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'min_dist': 0.30000000000000004,\n",
       " 'n_neighbors': 360,\n",
       " 'umap_metric': 'canberra'}"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_params_use"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e64f7130",
   "metadata": {
    "papermill": {
     "duration": 0.480381,
     "end_time": "2022-04-16T16:17:08.557151",
     "exception": false,
     "start_time": "2022-04-16T16:17:08.076770",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "## Under Evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "efd76a4c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-16T16:17:09.512734Z",
     "iopub.status.busy": "2022-04-16T16:17:09.511948Z",
     "iopub.status.idle": "2022-04-16T16:17:09.514986Z",
     "shell.execute_reply": "2022-04-16T16:17:09.514422Z"
    },
    "papermill": {
     "duration": 0.482484,
     "end_time": "2022-04-16T16:17:09.515171",
     "exception": false,
     "start_time": "2022-04-16T16:17:09.032687",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def random_search(features, space, num_evals, filenames, all_df):\n",
    "    \"\"\"\n",
    "    Randomly search parameter space of clustering pipeline\n",
    "\n",
    "    Arguments:\n",
    "        features: embeddings to use\n",
    "        space: dict, contains keys for 'n_neighbors', 'n_components',\n",
    "               and 'metrics' and values with\n",
    "               corresponding lists or ranges of parameters to search\n",
    "        num_evals: int, number of random parameter combinations to try\n",
    "\n",
    "    Returns:\n",
    "        df_result: pandas dataframe containing info on each evaluation\n",
    "                   performed, including run_id, parameters used, label\n",
    "                   count, and cost\n",
    "    \"\"\"\n",
    "    \n",
    "    results = []\n",
    "    \n",
    "    for i in trange(num_evals):\n",
    "        n_neighbors = random.choice(space['n_neighbors'])\n",
    "        min_dist = random.choice(space['min_dist'])\n",
    "        metric = random.choice(space['metric'])\n",
    "        \n",
    "        clusters = generate_kmclusters(features,\n",
    "                                     n_neighbors=n_neighbors, \n",
    "                                     min_dist=min_dist,\n",
    "                                     metric=metric)\n",
    "    \n",
    "        avg_precision, avg_recall, avg_f1 = metrics_score_(clusters, filenames, all_df)\n",
    "        print('round: {}, current setting: n_neighbors: {}, min_dist: {}, metric: {}'.format(i,n_neighbors,\n",
    "                                                                                             min_dist,\n",
    "                                                                                             metric))\n",
    "        print('### precsion: {}, recall: {}, f1-measure: {} ###'.format(np.round(avg_precision,2),\n",
    "                                                                                  np.round(avg_recall,2),\n",
    "                                                                                 np.round(avg_f1,2)))\n",
    "        results.append([i, n_neighbors, min_dist, metric, avg_precision, avg_recall, avg_f1])\n",
    "        \n",
    "        \n",
    "    result_df = pd.DataFrame(results, columns=['run_id', 'n_neighbors', 'min_dist', \n",
    "                                               'metric', 'avg_precision', 'avg_recall', 'avg_f1'])\n",
    "    \n",
    "    return result_df.sort_values(by='avg_recall', ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "7b8ccd82",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-04-16T16:17:10.471211Z",
     "iopub.status.busy": "2022-04-16T16:17:10.470368Z",
     "iopub.status.idle": "2022-04-16T16:17:10.473128Z",
     "shell.execute_reply": "2022-04-16T16:17:10.472535Z"
    },
    "papermill": {
     "duration": 0.487359,
     "end_time": "2022-04-16T16:17:10.473267",
     "exception": false,
     "start_time": "2022-04-16T16:17:09.985908",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "space = {\n",
    "        \"n_neighbors\": config['n_neighbors'],\n",
    "        \"min_dist\": config['min_dist'],\n",
    "        \"metric\": config['umap_metric'] \n",
    "    }\n",
    "\n",
    "# n_neighbors, min_dist, metric\n",
    "# %time random_use = random_search(feat, space, config['num_evals'] , filenames, all_df)\n",
    "# random_use.to_csv('umap_hyperparameters_random_search_{}.csv'.format(num_evals))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 13115.984684,
   "end_time": "2022-04-16T16:17:14.154525",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2022-04-16T12:38:38.169841",
   "version": "2.3.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}