6_CNN/5_CIFAR10/3_Kernel_Regularization_84.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 2,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "3_Kernel_Regularization_84.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.8"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Kernel Regularization\n",
        "\n",
        "* Deeper layer might lead to overfitting.\n",
        "* Use **Kernel Regularization** to avoid one node affecting the result too much.\n",
        "\n",
        "### Ridge Regression (L2 Regression)\n",
        "\n",
        "![Regularization.png](Regularization.png)\n",
        "\n",
        "To increase only training accuracy, minimize only SSE($\\sum_{i} (y_i-{\\hat{y}}_i)^2 $).\n",
        "\n",
        "To get higher accuracy in general too, parameter($\\beta_j$) also need to be minimized.\n",
        "\n",
        "Thus implement the penalty term($\\lambda \\sum_{j} {{\\beta}_j}^2$) to suppress parameter$\\beta$ from increasing abnormal.\n",
        "\n",
        "### Lasso Regression (L1 Regression)\n",
        "\n",
        "\n",
        "$$L_1(\\beta) = \\sum_{i} (y_i-{\\hat{y}}_i)^2 + \\lambda \\sum_{j}|\\beta_{j}|$$\n",
        "\n",
        "\n",
        "Implement penalty term. ($\\lambda \\sum_{j} |\\beta_{j}|$: not square, but absolute.)\n",
        "\n",
        "## Kernel Regularization Analysis\n",
        "\n",
        "for $Y=f(x)+\\epsilon, \\, \\widehat{Y} = \\hat{f}(x)$,\n",
        "\n",
        "$E\\big[ (\\hat{Y}-Y)^2 \\big] = E(\\hat{Y}^2 + Y^2 -2\\hat{Y}Y)$<br>\n",
        "$\\qquad= E(\\hat{Y}^2) + E(Y^2) - 2E(\\hat{Y}\\cdot Y)$<br>\n",
        "$\\qquad=V(\\hat Y) + E(\\hat Y)^2 + V(Y) + E(Y)^2 - 2E(\\hat{Y} \\cdot Y) \\quad (\\because E(X^2)=E(X)^2+V(X))$<br>\n",
        "$\\qquad=V(\\hat Y)+V(Y)+E(\\hat Y)^2 + E(Y)^2-2f(x)E(\\hat Y)$<br>\n",
        "$\\qquad(\\because 2E(\\hat{Y}\\cdot Y)=2E(\\hat{Y}(f(x)+\\epsilon))=2E(\\hat{Y}\\cdot f(x))+2E(\\hat{Y}\\cdot\\epsilon)=2f(x)E(\\hat Y))$<br>\n",
        "$\\qquad=V(\\hat Y)+(E(\\hat Y)-f(x))^2+V(Y)$<br>\n",
        "$\\qquad=(model\\_estimate\\_variance)+{bias}^2+\\sigma^2$\n",
        "\n",
        "![Regularization_graph.png](Regularization_graph.png)\n",
        "\n",
        "When overfitted,\n",
        "* Model Estimate Variance: Increase\n",
        "* Bias: Decrease\n",
        "\n",
        "suppress parameter$\\beta$ in order to prevent overfitting.\n",
        "\n",
        "## Kernel Regularization using ```tensorflow.keras```\n",
        "```python\n",
        "tensorflow.keras.regularizers.l1_l2(l1=0.01, l2=0.01)\n",
        "tensorflow.keras.regularizers.l1(l1=0.01)\n",
        "tensorflow.keras.regularizers.l2(l2=0.01)\n",
        "```\n",
        "\n",
        "eg)\n",
        "```python\n",
        "h = keras.layers.Conv2D(32, kernel_size=(3, 3), activation='relu', kernel_regularizer=keras.regularizers.l2(1.E-04), name='conv1')(h)\n",
        "```\n",
        "\n",
        "Note that parameter ```l1``` and ```l2``` means following:\n",
        "* ```l1```: L1 regularization factor which corresponds to $\\lambda$ of equation for L1 above.\n",
        "* ```l2```: L2 regularization factor which corresponds to $\\lambda$ of equation for L2 above."
      ],
      "metadata": {
        "id": "zzlBJ-CPUn0H"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow.keras import layers, models\n",
        "from tensorflow.keras import datasets \n",
        "from tensorflow.keras.utils import to_categorical\n",
        "\n",
        "# load CIFAR10 Dataset\n",
        "(x_train, y_train), (x_test, y_test) = datasets.cifar10.load_data()\n",
        "Y_train = to_categorical(y_train)\n",
        "Y_test = to_categorical(y_test)\n",
        "\n",
        "print(\"Length of train set:\", len(Y_train))\n",
        "print(\"Shape of x_train:\", x_train.shape[1:])"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading data from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz\n",
            "170500096/170498071 [==============================] - 4s 0us/step\n",
            "Length of train set: 50000\n",
            "Shape of x_train: (32, 32, 3)\n"
          ]
        }
      ],
      "metadata": {
        "id": "VCkOge6SnTKx",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5fb58258-17f9-42b5-983e-fd98b07fc323"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "source": [
        "img_rows, img_cols, channel = x_train.shape[1:]\n",
        "\n",
        "# Unifying image size (reshape X_train, X_test)\n",
        "X_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, channel)\n",
        "X_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, channel)\n",
        "input_shape = (img_rows, img_cols, channel)\n",
        "\n",
        "# Normalize pixel value in image\n",
        "X_train = X_train.astype('float32')\n",
        "X_test = X_test.astype('float32')\n",
        "X_train /= 255\n",
        "X_test /= 255\n",
        "\n",
        "# Label is already one-hot encoded\n",
        "print(Y_train[0])\n",
        "num_classes = 10\n",
        "batch_size = 32\n",
        "print(input_shape)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "[0. 0. 0. 0. 0. 0. 1. 0. 0. 0.]\n",
            "(32, 32, 3)\n"
          ]
        }
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "SU1ARsYfn5kD",
        "outputId": "4757d4eb-63ff-4b17-96aa-57bb11629c9d"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "source": [
        "x = layers.Input(shape=input_shape,  name='input')\n",
        "h = layers.BatchNormalization()(x)\n",
        "h = layers.Conv2D(32, kernel_size=(3, 3), activation='relu',\n",
        "                  kernel_regularizer=tf.keras.regularizers.l2(1.E-04), name='conv1')(h)\n",
        "h = layers.Conv2D(32, kernel_size=(3, 3), activation='relu',\n",
        "                  kernel_regularizer=tf.keras.regularizers.l2(1.E-04), name='conv2')(h)\n",
        "h = layers.BatchNormalization()(h)\n",
        "h = layers.MaxPooling2D(pool_size=(2, 2), name='pool1')(h)\n",
        "h = layers.Dropout(0.2)(h)\n",
        "\n",
        "h = layers.Conv2D(64, kernel_size=(3, 3), activation='relu', padding='same',\n",
        "                  kernel_regularizer=tf.keras.regularizers.l2(1.E-04), name='conv3')(h)\n",
        "h = layers.BatchNormalization()(h)\n",
        "h = layers.Conv2D(64, kernel_size=(3, 3), activation='relu', padding='same',\n",
        "                  kernel_regularizer=tf.keras.regularizers.l2(1.E-04), name='conv4')(h)\n",
        "h = layers.BatchNormalization()(h)\n",
        "h = layers.MaxPooling2D(pool_size=(2, 2), name='pool2')(h)\n",
        "h = layers.Dropout(0.3)(h)\n",
        "h = layers.Conv2D(128, kernel_size=(3, 3), activation='relu', padding='same',\n",
        "                  kernel_regularizer=tf.keras.regularizers.l2(1.E-04), name='conv5')(h)\n",
        "h = layers.BatchNormalization()(h)\n",
        "h = layers.Conv2D(128, kernel_size=(3, 3), activation='relu', padding='same',\n",
        "                  kernel_regularizer=tf.keras.regularizers.l2(1.E-04), name='conv6')(h)\n",
        "h = layers.BatchNormalization()(h)\n",
        "h = layers.MaxPooling2D(pool_size=(2, 2), name='pool3')(h)\n",
        "h = layers.Dropout(0.4)(h)\n",
        "\n",
        "h = layers.Flatten()(h)\n",
        "y = layers.Dense(num_classes, activation='softmax', name='output')(h)\n",
        "\n",
        "\n",
        "model = models.Model(x, y)\n",
        "print(model.summary())"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Model: \"model\"\n",
            "_________________________________________________________________\n",
            "Layer (type)                 Output Shape              Param #   \n",
            "=================================================================\n",
            "input (InputLayer)           [(None, 32, 32, 3)]       0         \n",
            "_________________________________________________________________\n",
            "batch_normalization (BatchNo (None, 32, 32, 3)         12        \n",
            "_________________________________________________________________\n",
            "conv1 (Conv2D)               (None, 30, 30, 32)        896       \n",
            "_________________________________________________________________\n",
            "conv2 (Conv2D)               (None, 28, 28, 32)        9248      \n",
            "_________________________________________________________________\n",
            "batch_normalization_1 (Batch (None, 28, 28, 32)        128       \n",
            "_________________________________________________________________\n",
            "pool1 (MaxPooling2D)         (None, 14, 14, 32)        0         \n",
            "_________________________________________________________________\n",
            "dropout (Dropout)            (None, 14, 14, 32)        0         \n",
            "_________________________________________________________________\n",
            "conv3 (Conv2D)               (None, 14, 14, 64)        18496     \n",
            "_________________________________________________________________\n",
            "batch_normalization_2 (Batch (None, 14, 14, 64)        256       \n",
            "_________________________________________________________________\n",
            "conv4 (Conv2D)               (None, 14, 14, 64)        36928     \n",
            "_________________________________________________________________\n",
            "batch_normalization_3 (Batch (None, 14, 14, 64)        256       \n",
            "_________________________________________________________________\n",
            "pool2 (MaxPooling2D)         (None, 7, 7, 64)          0         \n",
            "_________________________________________________________________\n",
            "dropout_1 (Dropout)          (None, 7, 7, 64)          0         \n",
            "_________________________________________________________________\n",
            "conv5 (Conv2D)               (None, 7, 7, 128)         73856     \n",
            "_________________________________________________________________\n",
            "batch_normalization_4 (Batch (None, 7, 7, 128)         512       \n",
            "_________________________________________________________________\n",
            "conv6 (Conv2D)               (None, 7, 7, 128)         147584    \n",
            "_________________________________________________________________\n",
            "batch_normalization_5 (Batch (None, 7, 7, 128)         512       \n",
            "_________________________________________________________________\n",
            "pool3 (MaxPooling2D)         (None, 3, 3, 128)         0         \n",
            "_________________________________________________________________\n",
            "dropout_2 (Dropout)          (None, 3, 3, 128)         0         \n",
            "_________________________________________________________________\n",
            "flatten (Flatten)            (None, 1152)              0         \n",
            "_________________________________________________________________\n",
            "output (Dense)               (None, 10)                11530     \n",
            "=================================================================\n",
            "Total params: 300,214\n",
            "Trainable params: 299,376\n",
            "Non-trainable params: 838\n",
            "_________________________________________________________________\n",
            "None\n"
          ]
        }
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EfN0bXAkoFYn",
        "outputId": "7c31bf9d-0b26-4803-b42d-9c9617e8853f"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "source": [
        "epochs = 25\n",
        "model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
        "history = model.fit(X_train, Y_train, batch_size=batch_size,\n",
        "          epochs=epochs, validation_split=0.1, verbose=2)\n",
        "\n",
        "score = model.evaluate(X_test, Y_test)\n",
        "print()\n",
        "print('Test loss:', score[0])\n",
        "print('Test accuracy:', score[1])"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/25\n",
            "1407/1407 - 40s - loss: 1.7927 - accuracy: 0.4387 - val_loss: 1.1565 - val_accuracy: 0.6130\n",
            "Epoch 2/25\n",
            "1407/1407 - 7s - loss: 1.0964 - accuracy: 0.6312 - val_loss: 0.9966 - val_accuracy: 0.6844\n",
            "Epoch 3/25\n",
            "1407/1407 - 7s - loss: 0.9275 - accuracy: 0.6989 - val_loss: 0.8007 - val_accuracy: 0.7444\n",
            "Epoch 4/25\n",
            "1407/1407 - 7s - loss: 0.8517 - accuracy: 0.7301 - val_loss: 0.7570 - val_accuracy: 0.7618\n",
            "Epoch 5/25\n",
            "1407/1407 - 7s - loss: 0.7984 - accuracy: 0.7576 - val_loss: 0.7626 - val_accuracy: 0.7742\n",
            "Epoch 6/25\n",
            "1407/1407 - 7s - loss: 0.7652 - accuracy: 0.7748 - val_loss: 0.7829 - val_accuracy: 0.7794\n",
            "Epoch 7/25\n",
            "1407/1407 - 7s - loss: 0.7404 - accuracy: 0.7888 - val_loss: 0.7962 - val_accuracy: 0.7756\n",
            "Epoch 8/25\n",
            "1407/1407 - 7s - loss: 0.7188 - accuracy: 0.8028 - val_loss: 0.7970 - val_accuracy: 0.7802\n",
            "Epoch 9/25\n",
            "1407/1407 - 7s - loss: 0.7060 - accuracy: 0.8116 - val_loss: 0.7135 - val_accuracy: 0.8156\n",
            "Epoch 10/25\n",
            "1407/1407 - 7s - loss: 0.6917 - accuracy: 0.8196 - val_loss: 0.7211 - val_accuracy: 0.8136\n",
            "Epoch 11/25\n",
            "1407/1407 - 7s - loss: 0.6807 - accuracy: 0.8252 - val_loss: 0.7278 - val_accuracy: 0.8152\n",
            "Epoch 12/25\n",
            "1407/1407 - 7s - loss: 0.6710 - accuracy: 0.8334 - val_loss: 0.7367 - val_accuracy: 0.8212\n",
            "Epoch 13/25\n",
            "1407/1407 - 7s - loss: 0.6636 - accuracy: 0.8368 - val_loss: 0.7021 - val_accuracy: 0.8282\n",
            "Epoch 14/25\n",
            "1407/1407 - 7s - loss: 0.6558 - accuracy: 0.8423 - val_loss: 0.7189 - val_accuracy: 0.8222\n",
            "Epoch 15/25\n",
            "1407/1407 - 7s - loss: 0.6509 - accuracy: 0.8452 - val_loss: 0.6732 - val_accuracy: 0.8410\n",
            "Epoch 16/25\n",
            "1407/1407 - 7s - loss: 0.6396 - accuracy: 0.8508 - val_loss: 0.7584 - val_accuracy: 0.8192\n",
            "Epoch 17/25\n",
            "1407/1407 - 7s - loss: 0.6427 - accuracy: 0.8512 - val_loss: 0.7099 - val_accuracy: 0.8310\n",
            "Epoch 18/25\n",
            "1407/1407 - 7s - loss: 0.6341 - accuracy: 0.8559 - val_loss: 0.7440 - val_accuracy: 0.8296\n",
            "Epoch 19/25\n",
            "1407/1407 - 7s - loss: 0.6354 - accuracy: 0.8553 - val_loss: 0.6980 - val_accuracy: 0.8406\n",
            "Epoch 20/25\n",
            "1407/1407 - 7s - loss: 0.6276 - accuracy: 0.8580 - val_loss: 0.6913 - val_accuracy: 0.8476\n",
            "Epoch 21/25\n",
            "1407/1407 - 7s - loss: 0.6275 - accuracy: 0.8609 - val_loss: 0.6994 - val_accuracy: 0.8412\n",
            "Epoch 22/25\n",
            "1407/1407 - 7s - loss: 0.6235 - accuracy: 0.8601 - val_loss: 0.7571 - val_accuracy: 0.8272\n",
            "Epoch 23/25\n",
            "1407/1407 - 7s - loss: 0.6212 - accuracy: 0.8626 - val_loss: 0.7018 - val_accuracy: 0.8496\n",
            "Epoch 24/25\n",
            "1407/1407 - 7s - loss: 0.6143 - accuracy: 0.8673 - val_loss: 0.7161 - val_accuracy: 0.8396\n",
            "Epoch 25/25\n",
            "1407/1407 - 7s - loss: 0.6150 - accuracy: 0.8672 - val_loss: 0.6936 - val_accuracy: 0.8530\n",
            "313/313 [==============================] - 1s 4ms/step - loss: 0.7177 - accuracy: 0.8438\n",
            "\n",
            "Test loss: 0.7176816463470459\n",
            "Test accuracy: 0.8438000082969666\n"
          ]
        }
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BtXm0buJUn0j",
        "outputId": "e4b10a8e-96c0-4e56-d9fd-c1f6a355110a"
      }
    }
  ]
}