tutorial/implement_your_own_model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Implement your own model\n",
    "\n",
    "This article will illustrate how to implement a trained model to ESP32.\n",
    "\n",
    "## Prerequest\n",
    "\n",
    "1. A trained model with coefficients converted to numpy file (.npy)\n",
    "2. A development board integrated with ESP32.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Let's start\n",
    "\n",
    "First we need to import some packages to load the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import os\n",
    "import re\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- numpy is for loading the coefficients\n",
    "- re is for searching certain parts of filename as the name of variables\n",
    "- json is for loading the bias/bn offset exponents when converting in fixed point"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we start loading the weights, we need to generate some headers for header files :)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"output/cnn.h\", mode='w', encoding='utf-8') as fc:\n",
    "    hdr = '#pragma once\\n'\n",
    "    hdr += '#include \"dl_lib_matrix3d.h\"\\n'\n",
    "    hdr += '#include \"dl_lib_matrix3dq.h\"\\n\\n'\n",
    "    fc.writelines(hdr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 2 types of network, convolution and fully connection. Thus we need to realize the 2 different conversions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_3d_conv_c(data):\n",
    "    (H, W, C, N) = data.shape\n",
    "    c_data = data.copy().reshape(N, H, W, C)\n",
    "    for n in range(N):\n",
    "        c_data[n, :, :, :] = data[:, :, :, n]\n",
    "    return c_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_3d_fc_w(data):\n",
    "    (W, H) = data.shape\n",
    "    fc_data = data.copy()\n",
    "    return fc_data.T.reshape([1, H, W, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also need to decide which data precision we need, float point as it is or fixed point with quantization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_type = 'fptp_t'    # For float point, 'qtp_t' for fixed point"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get the quantized coefficients, we map the range to the entire 16 bits."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_3d_quantization(data):\n",
    "    shape = data.shape\n",
    "    q_data = data.flatten()\n",
    "    _min = min(q_data)\n",
    "    _max = max(q_data)\n",
    "    if _min > 0: _min = 0\n",
    "    if abs(_min) > abs(_max): _max = abs(_min)\n",
    "        \n",
    "    exponent = 0\n",
    "    qtp_range = 2**15 - 1\n",
    "    if _max != 0:\n",
    "        while _max > qtp_range:\n",
    "            exponent += 1\n",
    "            _max = _max / 2\n",
    "        while _max < (qtp_range / 2):\n",
    "            exponent -= 1\n",
    "            _max = _max * 2\n",
    "            \n",
    "    q_data = q_data * 2**(-exponent)\n",
    "    q_data = np.array(q_data).reshape(shape).astype('int16')\n",
    "    return q_data, exponent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But sometimes we have our own exponents that got from testing or debugging."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_3d_quant_exponent(data, exponent):\n",
    "    shape = data.shape\n",
    "    q_data = data.flatten()\n",
    "    q_data = q_data * 2**(-exponent)\n",
    "    \n",
    "    q_data = np.array(q_data).reshape(shape).astype('int16')\n",
    "    return q_data, exponent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we start loading the coefficients through their names. \n",
    "\n",
    "In this example, the coefficients are put in the `weights` directory, and are named with `.npy`. So we need to get all of them, and then process one by one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "files = os.popen(\"find weights -name '*.npy' | sort\")\n",
    "# pattern of coefficient's name\n",
    "coef_pat = re.compile(\"weights/(.*)\\.npy\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "for f in files:\n",
    "    f = f.rstrip()\n",
    "    coef = np.load(f)\n",
    "    coef_name = coef_pat.search(f).group(1)\n",
    "    \n",
    "    if len(coef.shape) == 2:\n",
    "        # Fully connection\n",
    "        coef = convert_3d_fc_w(coef)\n",
    "    else:\n",
    "        if len(coef.shape) == 1:\n",
    "            # Bias/BN\n",
    "            coef = coef.reshape([1, 1, -1 ,1])\n",
    "        coef = convert_3d_conv_c(coef)\n",
    "        \n",
    "    if data_type == 'qtp_t':\n",
    "        coef, expo = convert_3d_quantization(coef)\n",
    "        \n",
    "    # Generate files\n",
    "    item_f = \"const static \"\n",
    "    if data_type == 'fptp_t':\n",
    "        item_f += f\"fptp_t {coef_name}_item_array[] = \"\n",
    "        data_template = \"{:.6f}f, \"\n",
    "    else:\n",
    "        item_f += f\"qtp_t {coef_name}_item_array[] = \"\n",
    "        data_template = \"{:d}, \"\n",
    "    item_f += \"{\\n\\t\"\n",
    "    intend = 0\n",
    "    for d in coef.flat:\n",
    "        item_f += data_template.format(d)\n",
    "        intend += 1\n",
    "        if intend % 8 == 0:\n",
    "            item_f += \"\\n\\t\"\n",
    "    item_f += \"\\n};\\n\\n\"\n",
    "    \n",
    "    (N, H, W, C) = coef.shape\n",
    "    struct_f = \"const static dl_matrix3d\"\n",
    "    if data_type == 'fptp_t':\n",
    "        struct_f += f\"_t {coef_name} = {{\\n\"\n",
    "    else:\n",
    "        struct_f += f\"q_t {coef_name} = {{\\n\"\n",
    "    struct_f += f\"\\t.w = {W},\\n\"\n",
    "    struct_f += f\"\\t.h = {H},\\n\"\n",
    "    struct_f += f\"\\t.c = {C},\\n\"\n",
    "    struct_f += f\"\\t.n = {N},\\n\"\n",
    "    struct_f += f\"\\t.stride = {W * C},\\n\"\n",
    "    if data_type == 'qtp_t':\n",
    "        struct_f += f\"\\t.exponent = {expo},\\n\"\n",
    "    struct_f += f\"\\t.item = ({data_type} *)(&{coef_name}_item_array[0])\\n}};\\n\\n\"\n",
    "    with open(\"output/cnn.h\", mode='a', encoding='utf-8') as fc:\n",
    "        fc.writelines(item_f)\n",
    "        fc.writelines(struct_f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With `output/cnn.h` we can get the coefficients in C codes.\n",
    "\n",
    "To test the simple network, we use the data from mnist. And convert it to a loadable header file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "in_data = np.load(\"2.npy\")\n",
    "hdr = '#pragma once\\n'\n",
    "hdr += '#include \"dl_lib_matrix3d.h\"\\n'\n",
    "item_f = \"const static uc_t input_item_array[] = {\\n\\t\"\n",
    "data_template = \"{:d}, \"\n",
    "intend = 0\n",
    "for d in in_data.flat:\n",
    "    item_f += data_template.format(d)\n",
    "    intend += 1\n",
    "    if intend % 8 == 0:\n",
    "        item_f += \"\\n\\t\"\n",
    "item_f += \"\\n};\\n\\n\"\n",
    "with open(\"output/input.h\", mode='w', encoding='utf-8') as fc:\n",
    "    fc.writelines(hdr)\n",
    "    fc.writelines(item_f)\n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have the input and weights, go and test the network."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In `test` directory, run `idf.py build flash` as other esp32 examples and there we get the result!\n",
    "\n",
    "![image.png](result.png)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}