2-Creating_classifiers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5bab868e",
   "metadata": {},
   "source": [
    "# Training Classifier\n",
    "In this Notebook we will:\n",
    "- [Load the data for each of the different classes](#data_collection)\n",
    "- [Load the selected base model and hyperparams](#load_model)\n",
    "- [Train the models](#train_model)\n",
    "- [Evaluate the models](#eval)\n",
    "- [Compare to baselines](#score)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "d524c04a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Imports\n",
    "import pandas as pd\n",
    "from setfit import SetFitModel, SetFitTrainer\n",
    "from sentence_transformers.losses import CosineSimilarityLoss\n",
    "from setfit import SetFitTrainer\n",
    "from datasets import Dataset\n",
    "from setfit import sample_dataset\n",
    "from tqdm.auto import tqdm\n",
    "from sklearn.metrics import f1_score\n",
    "import joblib\n",
    "import pickle\n",
    "from sklearn.metrics import confusion_matrix\n",
    "## Workaround for dashes in name\n",
    "from importlib import import_module\n",
    "nlbse_statistics = import_module('code-comment-classification.nlbse_statistics') \n",
    "\n",
    "tqdm.pandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5082794a",
   "metadata": {},
   "source": [
    "<a id='data_collection'></a>\n",
    "## Data collection\n",
    "We first load the data. \n",
    "For each language we create a dataset for each of the seperate category."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "1cca0b73",
   "metadata": {},
   "outputs": [],
   "source": [
    "langs = ['java', 'python', 'pharo']\n",
    "lan_cats = []\n",
    "datasets = {}\n",
    "for lan in langs: # for each language\n",
    "    df = pd.read_csv(f'./code-comment-classification/{lan}/input/{lan}.csv')\n",
    "    df['combo'] = df[['comment_sentence', 'class']].agg(' | '.join, axis=1)\n",
    "    df['label'] = df.instance_type\n",
    "    cats = list(map(lambda x: lan + '_' + x, list(set(df.category))))\n",
    "    lan_cats = lan_cats + cats\n",
    "    for cat in list(set(df.category)): # for each category\n",
    "        filtered =  df[df.category == cat]\n",
    "        train_data = Dataset.from_pandas(filtered[filtered.partition == 0])\n",
    "        test_data = Dataset.from_pandas(filtered[filtered.partition == 1])\n",
    "        datasets[f'{lan}_{cat}'] = {'train_data': train_data, 'test_data' : test_data}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4c29938",
   "metadata": {},
   "source": [
    "<a id='load_model'></a>\n",
    "\n",
    "## Load model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1245d289",
   "metadata": {},
   "outputs": [],
   "source": [
    "hyperparameters = hyperparameters={'learning_rate': 1.7094555110821448e-05, 'num_epochs': 6, \n",
    "                                   'batch_size': 8, 'seed': 11, 'num_iterations': 10, \n",
    "                                   'max_iter': 241, 'solver': 'lbfgs'}\n",
    "    \n",
    "def model_init(params):\n",
    "    params = params or {}\n",
    "    max_iter = params.get(\"max_iter\", 100)\n",
    "    solver = params.get(\"solver\", \"liblinear\")\n",
    "    params = {\n",
    "        \"head_params\": {\n",
    "            \"max_iter\": max_iter,\n",
    "            \"solver\": solver,\n",
    "        }\n",
    "    }\n",
    "    return SetFitModel.from_pretrained(\"sentence-transformers/all-mpnet-base-v2\", **params)\n",
    "\n",
    "# Create a fresh trainer with hyperparams\n",
    "def load_trainer(train_data, test_data):\n",
    "    trainer = SetFitTrainer(\n",
    "        train_dataset=train_data,\n",
    "        eval_dataset=test_data,\n",
    "        loss_class=CosineSimilarityLoss,\n",
    "        model_init=model_init,\n",
    "        column_mapping={\"combo\": \"text\", \"label\": \"label\"},\n",
    "    )\n",
    "\n",
    "    trainer.apply_hyperparameters(hyperparameters, final_model=True)\n",
    "    \n",
    "    return trainer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "249b020d",
   "metadata": {},
   "source": [
    "<a id='train_model'></a>\n",
    "\n",
    "\n",
    "## Train Models\n",
    "Train and save a model for each of the categories\n",
    "\n",
    "This will take around 3h per category, so around 2 days in total."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a865dd9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train model for each cat\n",
    "for lan_cat in lan_cats:\n",
    "    print(f'training {lan_cat}')\n",
    "    train_data = datasets[lan_cat]['train_data']\n",
    "    test_data = datasets[lan_cat]['test_data']\n",
    "    trainer = load_trainer(train_data, test_data)\n",
    "    \n",
    "    trainer.train()\n",
    "    \n",
    "    joblib.dump(trainer, f'./models/{lan_cat}_all-mpnet-base-v2.joblib')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab3ba49a",
   "metadata": {},
   "source": [
    "<a id='eval'></a>\n",
    "\n",
    "## Evaluation\n",
    "Next we evaluate each of our trained models on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a3819f2",
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Score each classifier and write scores to CSV\n",
    "scores = []\n",
    "for lan_cat in lan_cats:\n",
    "    trainer = joblib.load(f'./models/{lan_cat}_all-mpnet-base-v2.joblib')\n",
    "    test_data = datasets[lan_cat]['test_data']\n",
    "    y_hat = trainer.model(test_data['combo'])\n",
    "    y = test_data['label']\n",
    "    _, fp, fn, tp = confusion_matrix(y_hat, y).ravel()\n",
    "    wf1 = f1_score(y, y_hat, average='weighted')\n",
    "    precision, recall, f1 = nlbse_statistics.get_precision_recall_f1(tp, fp, fn)\n",
    "    scores.append({'lan_cat': lan_cat.lower(),'precision': precision,'recall': recall,'f1': f1,'wf1': wf1})\n",
    "\n",
    "df = pd.DataFrame(scores)\n",
    "df.sort_values('lan_cat').to_excel('scores.xlsx')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0d45fd3",
   "metadata": {},
   "source": [
    "<a id='score'></a>\n",
    "\n",
    "## Comparision with baseline\n",
    "We compare the weighed f1 scores with the baseline\n",
    "\n",
    "|                               | Baseline  |        |      |             | Ours      |        |      |      |          |\n",
    "| ----------------------------- | --------- | ------ | ---- | ----------- | --------- | ------ | ---- | ---- | -------- |\n",
    "|                               | precision | recall | f1   | weighted_f1 | precision | recall | f1   | wf1  | delta f1 |\n",
    "| java_deprecation              | 0,00      | 0,00   | 0,00 | 0,92        | 0,78      | 0,95   | 0,86 | 0,98 | 0,86     |\n",
    "| java_expand                   | 0,35      | 0,27   | 0,30 | 0,66        | 0,71      | 0,80   | 0,75 | 0,88 | 0,45     |\n",
    "| java_ownership                | 1,00      | 0,68   | 0,81 | 0,98        | 1,00      | 1,00   | 1,00 | 1,00 | 0,19     |\n",
    "| java_pointer                  | 0,67      | 0,24   | 0,35 | 0,84        | 0,71      | 0,82   | 0,76 | 0,93 | 0,40     |\n",
    "| java_rational                 | 0,63      | 0,30   | 0,40 | 0,88        | 0,81      | 0,92   | 0,86 | 0,97 | 0,46     |\n",
    "| java_summary                  | 0,38      | 0,29   | 0,33 | 0,78        | 0,85      | 0,76   | 0,80 | 0,93 | 0,48     |\n",
    "| java_usage                    | 0,54      | 0,36   | 0,43 | 0,62        | 0,83      | 0,89   | 0,86 | 0,90 | 0,43     |\n",
    "| pharo_classreferences         | 0,33      | 0,06   | 0,10 | 0,93        | 0,47      | 0,57   | 0,52 | 0,96 | 0,42     |\n",
    "| pharo_collaborators           | 0,47      | 0,25   | 0,33 | 0,91        | 0,36      | 0,91   | 0,51 | 0,94 | 0,19     |\n",
    "| pharo_example                 | 0,77      | 0,43   | 0,55 | 0,68        | 0,93      | 0,89   | 0,91 | 0,92 | 0,35     |\n",
    "| pharo_intent                  | 0,58      | 0,33   | 0,42 | 0,87        | 0,87      | 0,89   | 0,88 | 0,97 | 0,45     |\n",
    "| pharo_keyimplementationpoints | 0,18      | 0,10   | 0,13 | 0,79        | 0,69      | 0,79   | 0,73 | 0,93 | 0,60     |\n",
    "| pharo_keymessages             | 0,31      | 0,16   | 0,21 | 0,76        | 0,79      | 0,91   | 0,85 | 0,95 | 0,64     |\n",
    "| pharo_responsibilities        | 0,59      | 0,33   | 0,43 | 0,81        | 0,67      | 0,63   | 0,65 | 0,86 | 0,22     |\n",
    "| python_developmentnotes       | 0,17      | 0,17   | 0,17 | 0,79        | 0,43      | 0,54   | 0,48 | 0,88 | 0,31     |\n",
    "| python_expand                 | 0,26      | 0,20   | 0,22 | 0,72        | 0,52      | 0,56   | 0,54 | 0,82 | 0,31     |\n",
    "| python_parameters             | 0,51      | 0,22   | 0,31 | 0,65        | 0,78      | 0,86   | 0,81 | 0,89 | 0,50     |\n",
    "| python_summary                | 0,12      | 0,08   | 0,09 | 0,71        | 0,62      | 0,64   | 0,63 | 0,87 | 0,54     |\n",
    "| python_usage                  | 0,47      | 0,18   | 0,26 | 0,63        | 0,69      | 0,77   | 0,73 | 0,84 | 0,46     |\n",
    "| ----------------------------- | --------- | ------ | ---- | ----------- | --------- | ------ | ---- | ---- | -------- |\n",
    "| average                       | 0,44      | 0,24   | 0,31 | 0,79        | 0,71      | 0,79   | 0,74 | 0,92 | 0,43     |\n",
    "\n",
    "\n",
    "Finally we calculate our final score: \n",
    "\n",
    "\\begin{align}\n",
    "submission\\_score(model) &= (avg. \\space F_1) \\times 0.75 + (\\% \\space of \\space outperformed \\space categories) \\times 0.25\n",
    "\\end{align}\n",
    "\n",
    "We take the average unweighed f1 (0.743594163), and the proportion of outperformed categories (all):\n",
    "\n",
    "\\begin{align}\n",
    "submission\\_score = 0.743594163 * 0.75 + 1 * 0.25 = 0,8076956223\n",
    "\\end{align}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df8c95c3",
   "metadata": {},
   "source": [
    "<a id='hub'></a>\n",
    "\n",
    "## Push to hub\n",
    "\n",
    "Finally we push all of our models to the Hugging Face Hub to make them publically avaliable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d2e37f4",
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Push to hub\n",
    "token = 'hf_XXXXXXXXXXX'\n",
    "repo = 'XXXXXXXXXXXXX'\n",
    "for lan_cat in lan_cats:\n",
    "    trainer = joblib.load(f'./models/{lan_cat}_all-mpnet-base-v2.joblib')\n",
    "    name = lan_cat.lower().replace('_','-') + '-classifier'\n",
    "    print(name)\n",
    "    trainer.push_to_hub(f'{repo}/{name}', use_auth_token=token, private=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4c84a00",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "In this Notebook we created our own classifiers which beat the baseline in every category. We uploaded the models to the Hugging Face Hub as well.\n",
    "\n",
    "![display image](https://github.com/snipe/animated-gifs/blob/master/retro-computers/old-school.gif?raw=true)\n",
    "\n",
    "Please join us in the [next and final Notebook](./3-Inference.ipynb) to see how we can load and use these models."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.10 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "vscode": {
   "interpreter": {
    "hash": "916dbcbb3f70747c44a77c7bcd40155683ae19c65e1c03b4aa3499c5328201f1"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}