validated/vision/body_analysis/arcface/dependencies/train_arcface.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training Notebook for ArcFace model on Refined MS1M dataset\n",
    "\n",
    "## Overview\n",
    "Use this notebook to train a ArcFace model from scratch. Make sure to have the Refined MS1M dataset prepared before proceeding.\n",
    "\n",
    "## Prerequisites\n",
    "The training notebooks and scripts are tested on python 2.7. The following additional packages need to be installed before proceeding:\n",
    "* MXNet - `pip install mxnet-cu90mkl` (tested on this version, can use other versions)\n",
    "* OpenCV - `pip install opencv-python`\n",
    "* Scikit-learn - `pip install scikit-learn`\n",
    "* Scikit-image - `pip install scikit-image`\n",
    "* EasyDict - `pip install easydict`\n",
    "* numpy - `pip install numpy`\n",
    "\n",
    "Also the following scripts (included in the repo) must be present in the same folder as this notebook:\n",
    "* `face_image.py` (prepares face images in the dataset for training)\n",
    "* `face_preprocess.py` (performs preprocessing on face images)\n",
    "* `fresnet.py` (contains model definition of ResNet100)\n",
    "* `image_iter.py` (helper script)\n",
    "* `symbol_utils.py` (helper script)\n",
    "* `verification.py` (performs verification on validation sets)\n",
    "\n",
    "In order to train the model with a python script:\n",
    "* Generate the script : In Jupyter Notebook browser, go to File -> Download as -> Python (.py)\n",
    "* Run the script: `python train_arcface.py`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import dependencies\n",
    "Verify that all dependencies are installed using the cell below. Continue if no errors encountered, warnings can be ignored."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import absolute_import\n",
    "from __future__ import division\n",
    "from __future__ import print_function\n",
    "\n",
    "import os\n",
    "import sys\n",
    "import math\n",
    "import numpy as np\n",
    "import random\n",
    "import logging\n",
    "import pickle\n",
    "import numpy as np\n",
    "from image_iter import FaceImageIter\n",
    "from image_iter import FaceImageIterList\n",
    "import mxnet as mx\n",
    "from mxnet import ndarray as nd\n",
    "import mxnet.optimizer as optimizer\n",
    "import fresnet\n",
    "import verification\n",
    "import sklearn\n",
    "from easydict import EasyDict as edict\n",
    "import multiprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Specify model, hyperparameters and paths\n",
    "The training was done on a p3.8xlarge ec2 instance on AWS. It has 4 Nvidia Tesla V100 GPUs (16GB each) and Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz with 32 threads.\n",
    "\n",
    "The batch_size set below is per device. For multiple GPUs there are different batches in each GPU of size batch_size simultaneously.\n",
    "\n",
    "The rest of the parameters can be tuned to fit the needs of a user. The values shown below were used to train the model in the model zoo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Path to dataset\n",
    "data_dir = '/home/ubuntu/faces_ms1m_112x112'\n",
    "# Path to directory where models will be saved\n",
    "prefix = '/home/ubuntu/resnet100'\n",
    "# Load pretrained model\n",
    "pretrained = ''\n",
    "# Checkpoint saving option. 0: discard saving. 1: save when necessary. 2: always save\n",
    "ckpt = 1\n",
    "# do verification testing and model saving every verbose batches\n",
    "verbose = 2000\n",
    "# max training batches\n",
    "max_steps = 0\n",
    "# number of training epochs\n",
    "end_epoch = 30\n",
    "# initial learning rate\n",
    "lr = 0.1\n",
    "# learning rate decay iterations\n",
    "lr_steps = [100000, 140000, 160000]\n",
    "# weight decay\n",
    "wd = 0.0005\n",
    "# weight decay multiplier for fc7\n",
    "fc7_wd_mult = 1.0\n",
    "# momentum\n",
    "mom = 0.9\n",
    "# embedding length\n",
    "emb_size = 512\n",
    "# batch size in each context\n",
    "per_batch_size = 64\n",
    "# margin for loss\n",
    "margin_m = 0.5\n",
    "# scale for feature\n",
    "margin_s = 64.0\n",
    "# verification targets\n",
    "target = 'lfw,cfp_fp,agedb_30'\n",
    "beta = 1000.0\n",
    "beta_min = 5.0\n",
    "beta_freeze = 0\n",
    "gamma = 0.12\n",
    "power = 1.0\n",
    "scale = 0.9993"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Helper code\n",
    "class `AccMetric` : used to define and update accuracy metrics\n",
    "\n",
    "class `LossValueMetric` : used to define and update loss metrics\n",
    "\n",
    "`load_property()` : Function for loading num_classes and image_size from datasets folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Helper class for accuracy metrics\n",
    "class AccMetric(mx.metric.EvalMetric):\n",
    "    def __init__(self):\n",
    "        self.axis = 1\n",
    "        super(AccMetric, self).__init__(\n",
    "            'acc', axis=self.axis,\n",
    "            output_names=None, label_names=None)\n",
    "        self.losses = []\n",
    "        self.count = 0\n",
    "\n",
    "    def update(self, labels, preds):\n",
    "        self.count+=1\n",
    "        preds = [preds[1]] #use softmax output\n",
    "        for label, pred_label in zip(labels, preds):\n",
    "            if pred_label.shape != label.shape:\n",
    "                pred_label = mx.ndarray.argmax(pred_label, axis=self.axis)\n",
    "            pred_label = pred_label.asnumpy().astype('int32').flatten()\n",
    "            label = label.asnumpy()\n",
    "            if label.ndim==2:\n",
    "                label = label[:,0]\n",
    "            label = label.astype('int32').flatten()\n",
    "            assert label.shape==pred_label.shape\n",
    "            self.sum_metric += (pred_label.flat == label.flat).sum()\n",
    "            self.num_inst += len(pred_label.flat)\n",
    "\n",
    "# Helper class for loss metrics\n",
    "class LossValueMetric(mx.metric.EvalMetric):\n",
    "    def __init__(self):\n",
    "        self.axis = 1\n",
    "        super(LossValueMetric, self).__init__(\n",
    "            'lossvalue', axis=self.axis,\n",
    "            output_names=None, label_names=None)\n",
    "        self.losses = []\n",
    "\n",
    "    def update(self, labels, preds):\n",
    "        loss = preds[-1].asnumpy()[0]\n",
    "        self.sum_metric += loss\n",
    "        self.num_inst += 1.0\n",
    "        gt_label = preds[-2].asnumpy()\n",
    "\n",
    "# Helper function for loading num_classes and input image sizes\n",
    "def load_property(data_dir):\n",
    "    prop = edict()\n",
    "    for line in open(os.path.join(data_dir, 'property')):\n",
    "        vec = line.strip().split(',')\n",
    "        assert len(vec)==3\n",
    "        prop.num_classes = int(vec[0])\n",
    "        prop.image_size = [int(vec[1]), int(vec[2])]\n",
    "    return prop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Prepare network and define loss\n",
    "`get_symbol()` : Loads the model from the model definition file, defines ArcFace loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_symbol(arg_params, aux_params, image_channel, image_h, image_w, num_layers, num_classes, data_dir,prefix,pretrained,ckpt,verbose,max_steps,end_epoch,lr,lr_steps,wd,fc7_wd_mult,\n",
    "              mom,emb_size,per_batch_size,margin_m,margin_s,target,beta,beta_min,beta_freeze,gamma,power,scale):\n",
    "    data_shape = (image_channel,image_h,image_w)\n",
    "    image_shape = \",\".join([str(x) for x in data_shape])\n",
    "    margin_symbols = []\n",
    "    print('init resnet', num_layers)\n",
    "    \n",
    "    # Load Resnet100 model - model definition is present in fresnet.py\n",
    "    embedding = fresnet.get_symbol(emb_size, num_layers, \n",
    "        version_se=0, version_input=1, \n",
    "        version_output='E', version_unit=3,\n",
    "        version_act='prelu')\n",
    "    all_label = mx.symbol.Variable('softmax_label')\n",
    "    gt_label = all_label\n",
    "    extra_loss = None\n",
    "    _weight = mx.symbol.Variable(\"fc7_weight\", shape=(num_classes, emb_size), lr_mult=1.0, wd_mult=fc7_wd_mult)\n",
    "    \n",
    "    # Define ArcFace loss\n",
    "    s = margin_s\n",
    "    m = margin_m\n",
    "    assert s>0.0\n",
    "    assert m>=0.0\n",
    "    assert m<(math.pi/2)\n",
    "    _weight = mx.symbol.L2Normalization(_weight, mode='instance')\n",
    "    nembedding = mx.symbol.L2Normalization(embedding, mode='instance', name='fc1n')*s\n",
    "    fc7 = mx.sym.FullyConnected(data=nembedding, weight = _weight, no_bias = True, num_hidden=num_classes, name='fc7')\n",
    "    zy = mx.sym.pick(fc7, gt_label, axis=1)\n",
    "    cos_t = zy/s\n",
    "    cos_m = math.cos(m)\n",
    "    sin_m = math.sin(m)\n",
    "    mm = math.sin(math.pi-m)*m\n",
    "    threshold = math.cos(math.pi-m)\n",
    "    cond_v = cos_t - threshold\n",
    "    cond = mx.symbol.Activation(data=cond_v, act_type='relu')\n",
    "    body = cos_t*cos_t\n",
    "    body = 1.0-body\n",
    "    sin_t = mx.sym.sqrt(body)\n",
    "    new_zy = cos_t*cos_m\n",
    "    b = sin_t*sin_m\n",
    "    new_zy = new_zy - b\n",
    "    new_zy = new_zy*s\n",
    "    zy_keep = zy - s*mm\n",
    "    new_zy = mx.sym.where(cond, new_zy, zy_keep)\n",
    "    diff = new_zy - zy\n",
    "    diff = mx.sym.expand_dims(diff, 1)\n",
    "    gt_one_hot = mx.sym.one_hot(gt_label, depth = num_classes, on_value = 1.0, off_value = 0.0)\n",
    "    body = mx.sym.broadcast_mul(gt_one_hot, diff)\n",
    "    fc7 = fc7+body\n",
    "    out_list = [mx.symbol.BlockGrad(embedding)]\n",
    "    softmax = mx.symbol.SoftmaxOutput(data=fc7, label = gt_label, name='softmax', normalization='valid')\n",
    "    out_list.append(softmax)\n",
    "    out = mx.symbol.Group(out_list)\n",
    "    return (out, arg_params, aux_params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define train function\n",
    "`train_net()` : Train model, log training progress, save periodic checkpoints, compute and display validation accuracies periodically"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_net(data_dir,prefix,pretrained,ckpt,verbose,max_steps,end_epoch,lr,lr_steps,wd,fc7_wd_mult,\n",
    "              mom,emb_size,per_batch_size,margin_m,margin_s,target,beta,beta_min,beta_freeze,gamma,power,scale):\n",
    "    # define context\n",
    "    ctx = []\n",
    "    num_gpus = max(mx.test_utils.list_gpus()) + 1\n",
    "    if num_gpus>0:\n",
    "        for i in range(num_gpus):\n",
    "            ctx.append(mx.gpu(i))\n",
    "    if len(ctx)==0:\n",
    "        ctx = [mx.cpu()]\n",
    "        print('use cpu')\n",
    "    logger = logging.getLogger()\n",
    "    logger.setLevel(logging.INFO)\n",
    "    prefix_dir = os.path.dirname(prefix)\n",
    "    if not os.path.exists(prefix_dir):\n",
    "        os.makedirs(prefix_dir)\n",
    "    ctx_num = len(ctx)\n",
    "    num_layers = 100\n",
    "    print('num_layers',num_layers)\n",
    "    batch_size = per_batch_size*ctx_num\n",
    "    rescale_threshold = 0\n",
    "    image_channel = 3\n",
    "\n",
    "    os.environ['BETA'] = str(beta)\n",
    "    data_dir_list = data_dir.split(',')\n",
    "    assert len(data_dir_list)==1\n",
    "    data_dir = data_dir_list[0]\n",
    "    path_imgrec = None\n",
    "    path_imglist = None\n",
    "    prop = load_property(data_dir)\n",
    "    num_classes = prop.num_classes\n",
    "    image_size = prop.image_size\n",
    "    image_h = image_size[0]\n",
    "    image_w = image_size[1]\n",
    "    print('image_size', image_size)\n",
    "    assert(num_classes>0)\n",
    "    print('num_classes', num_classes)\n",
    "    path_imgrec = os.path.join(data_dir, \"train.rec\")\n",
    "\n",
    "    data_shape = (image_channel,image_size[0],image_size[1])\n",
    "    mean = None\n",
    "\n",
    "    begin_epoch = 0\n",
    "    base_lr = lr\n",
    "    base_wd = wd\n",
    "    base_mom = mom\n",
    "    if len(pretrained)==0:\n",
    "        arg_params = None\n",
    "        aux_params = None\n",
    "        sym, arg_params, aux_params = get_symbol(arg_params, aux_params, image_channel, image_h, image_w, \n",
    "                                                 num_layers, num_classes, data_dir,prefix,pretrained,ckpt,\n",
    "                                                 verbose,max_steps,end_epoch,lr,lr_steps,wd,fc7_wd_mult,\n",
    "                                                 mom,emb_size,per_batch_size,margin_m,margin_s,target,beta,\n",
    "                                                 beta_min,beta_freeze,gamma,power,scale)\n",
    "    else:\n",
    "        vec = pretrained.split(',')\n",
    "        print('loading', vec)\n",
    "        _, arg_params, aux_params = mx.model.load_checkpoint(vec[0], int(vec[1]))\n",
    "        sym, arg_params, aux_params = get_symbol(arg_params, aux_params)\n",
    "\n",
    "\n",
    "    model = mx.mod.Module(\n",
    "        context       = ctx,\n",
    "        symbol        = sym,\n",
    "    )\n",
    "    val_dataiter = None\n",
    "\n",
    "    train_dataiter = FaceImageIter(\n",
    "        batch_size           = batch_size,\n",
    "        data_shape           = data_shape,\n",
    "        path_imgrec          = path_imgrec,\n",
    "        shuffle              = True,\n",
    "        rand_mirror          = 1,\n",
    "        mean                 = mean,\n",
    "        cutoff               = 0,\n",
    "    )\n",
    "\n",
    "    _metric = AccMetric()\n",
    "    eval_metrics = [mx.metric.create(_metric)]\n",
    "\n",
    "    initializer = mx.init.Xavier(rnd_type='gaussian', factor_type=\"out\", magnitude=2) #resnet style\n",
    "    _rescale = 1.0/ctx_num\n",
    "    opt = optimizer.SGD(learning_rate=base_lr, momentum=base_mom, wd=base_wd, rescale_grad=_rescale)\n",
    "    som = 20\n",
    "    _cb = mx.callback.Speedometer(batch_size, som)\n",
    "\n",
    "    ver_list = []\n",
    "    ver_name_list = []\n",
    "    for name in target.split(','):\n",
    "        path = os.path.join(data_dir,name+\".bin\")\n",
    "        if os.path.exists(path):\n",
    "            data_set = verification.load_bin(path, image_size)\n",
    "            ver_list.append(data_set)\n",
    "            ver_name_list.append(name)\n",
    "            print('ver', name)\n",
    "\n",
    "\n",
    "\n",
    "    def ver_test(nbatch):\n",
    "        results = []\n",
    "        for i in xrange(len(ver_list)):\n",
    "            acc1, std1, acc2, std2, xnorm, embeddings_list = verification.test(ver_list[i], model, batch_size, 10, None, None)\n",
    "            print('[%s][%d]XNorm: %f' % (ver_name_list[i], nbatch, xnorm))\n",
    "            print('[%s][%d]Accuracy-Flip: %1.5f+-%1.5f' % (ver_name_list[i], nbatch, acc2, std2))\n",
    "            results.append(acc2)\n",
    "        return results\n",
    "\n",
    "\n",
    "\n",
    "    highest_acc = [0.0, 0.0]  #lfw and target\n",
    "    global_step = [0]\n",
    "    save_step = [0]\n",
    "    \n",
    "    p = 512.0/batch_size\n",
    "    for l in xrange(len(lr_steps)):\n",
    "        lr_steps[l] = int(lr_steps[l]*p)\n",
    "    print('lr_steps', lr_steps)\n",
    "    def _batch_callback(param):\n",
    "        global_step[0]+=1\n",
    "        mbatch = global_step[0]\n",
    "        for _lr in lr_steps:\n",
    "            if mbatch==beta_freeze+_lr:\n",
    "                opt.lr *= 0.1\n",
    "                print('lr change to', opt.lr)\n",
    "                break\n",
    "\n",
    "        _cb(param)\n",
    "        if mbatch%1000==0:\n",
    "            print('lr-batch-epoch:',opt.lr,param.nbatch,param.epoch)\n",
    "\n",
    "        if mbatch>=0 and mbatch%verbose==0:\n",
    "            acc_list = ver_test(mbatch)\n",
    "            save_step[0]+=1\n",
    "            msave = save_step[0]\n",
    "            do_save = False\n",
    "            if len(acc_list)>0:\n",
    "                lfw_score = acc_list[0]\n",
    "                if lfw_score>highest_acc[0]:\n",
    "                    highest_acc[0] = lfw_score\n",
    "                    if lfw_score>=0.998:\n",
    "                        do_save = True\n",
    "                if acc_list[-1]>=highest_acc[-1]:\n",
    "                    highest_acc[-1] = acc_list[-1]\n",
    "                    if lfw_score>=0.99:\n",
    "                        do_save = True\n",
    "            if ckpt==0:\n",
    "                do_save = False\n",
    "            elif ckpt>1:\n",
    "                do_save = True\n",
    "            if do_save:\n",
    "                print('saving', msave)\n",
    "                arg, aux = model.get_params()\n",
    "                mx.model.save_checkpoint(prefix, msave, model.symbol, arg, aux)\n",
    "            print('[%d]Accuracy-Highest: %1.5f'%(mbatch, highest_acc[-1]))\n",
    "        if mbatch<=beta_freeze:\n",
    "            _beta = beta\n",
    "        else:\n",
    "            move = max(0, mbatch-beta_freeze)\n",
    "            _beta = max(beta_min, beta*math.pow(1+gamma*move, -1.0*power))\n",
    "        os.environ['BETA'] = str(_beta)\n",
    "        if max_steps>0 and mbatch>max_steps:\n",
    "            sys.exit(0)\n",
    "\n",
    "    epoch_cb = None\n",
    "\n",
    "    model.fit(train_dataiter,\n",
    "        begin_epoch        = begin_epoch,\n",
    "        num_epoch          = end_epoch,\n",
    "        eval_data          = val_dataiter,\n",
    "        eval_metric        = eval_metrics,\n",
    "        kvstore            = 'device',\n",
    "        optimizer          = opt,\n",
    "        initializer        = initializer,\n",
    "        arg_params         = arg_params,\n",
    "        aux_params         = aux_params,\n",
    "        allow_missing      = True,\n",
    "        batch_end_callback = _batch_callback,\n",
    "        epoch_end_callback = epoch_cb )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train model\n",
    "* Run the cell below to start training\n",
    "* Logs are displayed in the cell output\n",
    "* An example run of 2000 batches is shown here\n",
    "* Symbols and params files are saved periodically in the `prefix` folder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:loading recordio /home/ubuntu/insightface/datasets/faces_ms1m_112x112/train.rec...\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "num_layers 100\n",
      "image_size [112, 112]\n",
      "num_classes 85164\n",
      "init resnet 100\n",
      "0 1 E 3 prelu\n",
      "header0 label [3804847. 3890011.]\n",
      "id2range 85164\n",
      "3804846\n",
      "rand_mirror 1\n",
      "loading bin 0\n",
      "loading bin 1000\n",
      "loading bin 2000\n",
      "loading bin 3000\n",
      "loading bin 4000\n",
      "loading bin 5000\n",
      "loading bin 6000\n",
      "loading bin 7000\n",
      "loading bin 8000\n",
      "loading bin 9000\n",
      "loading bin 10000\n",
      "loading bin 11000\n",
      "(12000L, 3L, 112L, 112L)\n",
      "ver lfw\n",
      "loading bin 0\n",
      "loading bin 1000\n",
      "loading bin 2000\n",
      "loading bin 3000\n",
      "loading bin 4000\n",
      "loading bin 5000\n",
      "loading bin 6000\n",
      "loading bin 7000\n",
      "loading bin 8000\n",
      "loading bin 9000\n",
      "loading bin 10000\n",
      "loading bin 11000\n",
      "loading bin 12000\n",
      "loading bin 13000\n",
      "(14000L, 3L, 112L, 112L)\n",
      "ver cfp_fp\n",
      "loading bin 0\n",
      "loading bin 1000\n",
      "loading bin 2000\n",
      "loading bin 3000\n",
      "loading bin 4000\n",
      "loading bin 5000\n",
      "loading bin 6000\n",
      "loading bin 7000\n",
      "loading bin 8000\n",
      "loading bin 9000\n",
      "loading bin 10000\n",
      "loading bin 11000\n",
      "(12000L, 3L, 112L, 112L)\n",
      "ver agedb_30\n",
      "lr_steps [200000, 280000, 320000]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ubuntu/anaconda3/envs/face/lib/python2.7/site-packages/mxnet/module/base_module.py:494: UserWarning: Optimizer created manually outside Module but rescale_grad is not normalized to 1.0/batch_size/num_workers (0.25 vs. 0.00390625). Is this intended?\n",
      "  optimizer_params=optimizer_params)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "call reset()\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Epoch[0] Batch [20]\tSpeed: 754.96 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [40]\tSpeed: 737.85 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [60]\tSpeed: 761.90 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [80]\tSpeed: 762.10 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [100]\tSpeed: 764.84 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [120]\tSpeed: 768.22 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [140]\tSpeed: 766.86 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [160]\tSpeed: 761.52 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [180]\tSpeed: 769.05 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [200]\tSpeed: 765.20 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [220]\tSpeed: 761.28 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [240]\tSpeed: 764.20 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [260]\tSpeed: 767.56 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [280]\tSpeed: 764.45 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [300]\tSpeed: 737.81 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [320]\tSpeed: 762.99 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [340]\tSpeed: 767.97 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [360]\tSpeed: 764.10 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [380]\tSpeed: 764.69 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [400]\tSpeed: 770.05 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [420]\tSpeed: 753.64 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [440]\tSpeed: 755.47 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [460]\tSpeed: 758.02 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [480]\tSpeed: 760.34 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [500]\tSpeed: 766.50 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [520]\tSpeed: 769.40 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [540]\tSpeed: 761.44 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [560]\tSpeed: 758.27 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [580]\tSpeed: 768.03 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [600]\tSpeed: 752.81 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [620]\tSpeed: 754.51 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [640]\tSpeed: 756.34 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [660]\tSpeed: 755.88 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [680]\tSpeed: 752.31 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [700]\tSpeed: 751.87 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [720]\tSpeed: 759.27 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [740]\tSpeed: 752.61 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [760]\tSpeed: 734.68 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [780]\tSpeed: 758.02 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [800]\tSpeed: 757.39 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [820]\tSpeed: 762.61 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [840]\tSpeed: 761.41 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [860]\tSpeed: 757.71 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [880]\tSpeed: 766.82 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [900]\tSpeed: 771.76 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [920]\tSpeed: 755.59 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [940]\tSpeed: 759.01 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [960]\tSpeed: 754.64 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [980]\tSpeed: 762.02 samples/sec\tacc=0.000000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lr-batch-epoch: 0.1 999 0\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Epoch[0] Batch [1000]\tSpeed: 762.73 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1020]\tSpeed: 759.52 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1040]\tSpeed: 762.22 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1060]\tSpeed: 766.62 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1080]\tSpeed: 763.39 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1100]\tSpeed: 766.87 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1120]\tSpeed: 763.20 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1140]\tSpeed: 764.81 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1160]\tSpeed: 763.07 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1180]\tSpeed: 772.91 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1200]\tSpeed: 759.62 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1220]\tSpeed: 770.68 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1240]\tSpeed: 764.11 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1260]\tSpeed: 743.71 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1280]\tSpeed: 761.33 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1300]\tSpeed: 762.06 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1320]\tSpeed: 755.56 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1340]\tSpeed: 769.52 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1360]\tSpeed: 765.20 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1380]\tSpeed: 755.56 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1400]\tSpeed: 766.99 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1420]\tSpeed: 752.36 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1440]\tSpeed: 766.26 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1460]\tSpeed: 761.00 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1480]\tSpeed: 758.01 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1500]\tSpeed: 766.72 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1520]\tSpeed: 769.79 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1540]\tSpeed: 759.39 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1560]\tSpeed: 768.87 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1580]\tSpeed: 769.85 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1600]\tSpeed: 755.14 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1620]\tSpeed: 760.69 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1640]\tSpeed: 772.00 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1660]\tSpeed: 762.07 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1680]\tSpeed: 760.18 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1700]\tSpeed: 768.91 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1720]\tSpeed: 759.69 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1740]\tSpeed: 766.22 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1760]\tSpeed: 763.70 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1780]\tSpeed: 729.06 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1800]\tSpeed: 758.22 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1820]\tSpeed: 762.05 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1840]\tSpeed: 767.52 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1860]\tSpeed: 760.04 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1880]\tSpeed: 758.56 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1900]\tSpeed: 764.91 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1920]\tSpeed: 758.03 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1940]\tSpeed: 767.97 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1960]\tSpeed: 766.79 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [1980]\tSpeed: 765.21 samples/sec\tacc=0.000000\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "lr-batch-epoch: 0.1 1999 0\n",
      "testing verification..\n",
      "(12000, 512)\n",
      "infer time 9.902401\n",
      "[lfw][2000]XNorm: 16.390527\n",
      "[lfw][2000]Accuracy-Flip: 0.84483+-0.01571\n",
      "testing verification..\n",
      "(14000, 512)\n",
      "infer time 11.520645\n",
      "[cfp_fp][2000]XNorm: 15.576616\n",
      "[cfp_fp][2000]Accuracy-Flip: 0.63686+-0.01520\n",
      "testing verification..\n",
      "(12000, 512)\n",
      "infer time 9.812545\n",
      "[agedb_30][2000]XNorm: 15.416758\n",
      "[agedb_30][2000]Accuracy-Flip: 0.64450+-0.01593\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2000]Accuracy-Highest: 0.64450\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Epoch[0] Batch [2000]\tSpeed: 124.99 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [2020]\tSpeed: 767.93 samples/sec\tacc=0.000000\n",
      "INFO:root:Epoch[0] Batch [2040]\tSpeed: 762.95 samples/sec\tacc=0.000000\n"
     ]
    }
   ],
   "source": [
    "def main():\n",
    "    train_net(data_dir,prefix,pretrained,ckpt,verbose,max_steps,end_epoch,lr,lr_steps,wd,fc7_wd_mult,\n",
    "              mom,emb_size,per_batch_size,margin_m,margin_s,target,beta,beta_min,beta_freeze,gamma,power,scale)\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    main()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Slim model\n",
    "The last layer in the trained model is not required for inference and can be discarded (using cell below) which reduces the model size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Saved checkpoint to \"/home/ubuntu/resnet100-0000.params\"\n"
     ]
    }
   ],
   "source": [
    "# Choose model to slim (give path to syms and params)\n",
    "prefix = '/home/ubuntu/resnet100'\n",
    "epoch = 1\n",
    "\n",
    "# Load model\n",
    "sym, arg_params, aux_params = mx.model.load_checkpoint(prefix, epoch)\n",
    "# Populate list containing nodes to be removed\n",
    "all_layers = sym.get_internals()\n",
    "sym = all_layers['fc1_output']\n",
    "dellist = []\n",
    "for k,v in arg_params.iteritems():\n",
    "    if k.startswith('fc7'):\n",
    "        dellist.append(k)\n",
    "        \n",
    "# Remove nodes\n",
    "for d in dellist:\n",
    "    del arg_params[d]\n",
    "\n",
    "# Save slimed model\n",
    "mx.model.save_checkpoint(prefix, 0, sym, arg_params, aux_params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Environment (conda_face)",
   "language": "python",
   "name": "conda_face"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}