Chapter4.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Chapter4.ipynb",
      "provenance": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyPeswlk5zI2j/jvsrK/ilsB",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/hank199599/deep_learning_keras_log/blob/main/Chapter4.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_PifvHdct1C9"
      },
      "source": [
        "# 機器學習的四個分支\n",
        "* 監督式學習 Supervised learning：所有訓練資料都有相對應的標籤\n",
        "* 非監督式學習 Unsupervised learning：輸入資料沒有任何標籤找出其中有意義的表示法\n",
        "* 自監督式學習 Self-supervised learning：一種沒有人工標註標籤的監督式學習\n",
        "* 增強式學習 Reinforcement learning"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KJUko0a-uSG9"
      },
      "source": [
        "# 評估機器學習模型"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J2zYubAguYSt"
      },
      "source": [
        "## 1. 把資料拆分為訓練集、驗證集、測試集\n",
        "* 訓練資料集 training set\n",
        "* 驗證資料集 validation set\n",
        "* 測試資料集 testing set\n",
        "  \n",
        "在訓練過程中，使用驗證集進行測試亦可能發生**資訊洩漏(information leak)**，  \n",
        "進而導致模型對該資料集過度配適。"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fb4G03qkujJU"
      },
      "source": [
        "### 簡單拆分驗證\n",
        "使用驗證集來調整模型  \n",
        "缺陷：可用資料很少的情況下，驗證及和測試集的樣本會太少，統計代表性不足  \n",
        "![3-3](https://github.com/hank199599/deep_learning_keras_log/blob/main/pictures/3-3.png?raw=true)  \n",
        "  \n",
        "\n",
        "```python\n",
        "nav_validation_samples = 10000 #設定預設的驗證集資料筆數\n",
        "\n",
        "np.random.shuffle(data) # 打散資料，重新洗牌\n",
        "validation_data = data[:nav_validation_samples] # 定義前10000筆(0~9999)為驗證集\n",
        "\n",
        "training_data = date[:]\n",
        "\n",
        "model = get_model()\n",
        "model.train(training_data) # 以訓練集資料進行模型訓練\n",
        "validation_score = model.evaluate(validation_data) # 以驗證集資料進行模型評估\n",
        "\n",
        "#對模型不斷進行：調整、訓練與評估\n",
        "\n",
        "\n",
        "model = get_model()\n",
        "#完成超參數的調整後，通常會使用完整的資料(訓練集+驗證集)來訓練一次\n",
        "model.train(np.concatenate([training_data,validation_data]))\n",
        "test_score = model.evaluate(test_data)\n",
        "```\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aGNprKCDunXU"
      },
      "source": [
        "### K折驗證\n",
        "\n",
        "\n",
        "\n",
        "```python\n",
        "k=4 # 進行4折交叉驗證\n",
        "nav_validation_samples = len(data) #設定預設的驗證集資料筆數\n",
        "\n",
        "np_random.shuffle(data)\n",
        "validation_scores=[]\n",
        "for flod in range(k):\n",
        "  validation_data = data[nav_validation_samples*flod:nav_validation_samples*(flod+1)]\n",
        "  teaining_data=data[:nav_validation_samples*flod]+data[nav_validation_samples*(flod+1):]\n",
        "\n",
        "  model = get_model()\n",
        "  model.train(training_data)\n",
        "  validation_score = model.evaluate(validation_data)\n",
        "  validation_scores.appned(validation_score)\n",
        "\n",
        "validation_score=np.average(validation_scores)\n",
        "\n",
        "model=get_model()\n",
        "model.train(data)\n",
        "test_score = model.evaluate(test_data)\n",
        "  \n",
        "```\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PcO_WdqKuqLV"
      },
      "source": [
        "### 多次洗牌的K折驗證\n",
        "此方法適用於可用資料較少且須要盡可能精確地評估模型的情況"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x8hSFDatux_t"
      },
      "source": [
        "## 2. 注意事項\n",
        "* 資料代表性：需**隨機打散(randomly shuffle)**使資料及具有代表性\n",
        "* 時間方向性：避免**時間漏失(temporal leak)**\n",
        "* 資料中的重複現象：確保訓練集與驗證集沒有交集發生"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AYvJbIGpvBfE"
      },
      "source": [
        "# 資料預處理 (preprocssing)\n",
        "讓手邊的原始資料更符合神經網路使用\n",
        "* 向量化 (Vertorization)：將資料轉換為**浮點數的張量**\n",
        "* 數值正規化 (Normalization)：使輸入資料的標準差為1，平均值為0\n",
        " * 取小值\n",
        " * 同質性\n",
        " * 單獨正規化每個特徵\n",
        "* 處理缺失值 (Missing Values)：將缺失值補為0 (前提是0必須是個沒意義的值)\n",
        "* 特徵提取："
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FgGOskULvGDl"
      },
      "source": [
        "# 特徵工程\n",
        "透過資深知識對手邊資料與機器學習的理解，直接以人工方式去轉換資料"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2E_fc-3GvR5O"
      },
      "source": [
        "# 低度配適與過度配適\n",
        "* 最佳化(Optimization)：調整模型使得能在訓練資料上獲得最佳表現的過程\n",
        "* 普適性(Generalization)：已訓練的模型對從未見過得資料的預測能力"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K7UuTWMwtm3N"
      },
      "source": [
        "## 常規化 (regularization) 的幾種手段\n",
        "1. 取得更多訓練資料 \n",
        "2. 調配模型儲存的資訊量或限制儲存資訊的類型或數值"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nwy1s7z0vW6d"
      },
      "source": [
        "## 縮減神經網路的大小\n",
        "減少模型可用來學習的**參數(記憶能力)**數量"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A__RxzpX1kzc",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7efdf72c-efa9-4214-8d6e-e9407841bf62"
      },
      "source": [
        "from tensorflow.keras.datasets import imdb\n",
        "import numpy as np\n",
        "\n",
        "(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=10000)\n",
        "\n",
        "def vectorize_sequences(sequences, dimension=10000):\n",
        "    \n",
        "    results = np.zeros((len(sequences), dimension))\n",
        "    for i, sequence in enumerate(sequences):\n",
        "        results[i, sequence] = 1.\n",
        "    return results\n",
        "\n",
        "x_train = vectorize_sequences(train_data)\n",
        "\n",
        "x_test = vectorize_sequences(test_data)\n",
        "\n",
        "y_train = np.asarray(train_labels).astype('float32')\n",
        "y_test = np.asarray(test_labels).astype('float32')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb.npz\n",
            "17465344/17464789 [==============================] - 0s 0us/step\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "<string>:6: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n",
            "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/datasets/imdb.py:159: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n",
            "  x_train, y_train = np.array(xs[:idx]), np.array(labels[:idx])\n",
            "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/datasets/imdb.py:160: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray\n",
            "  x_test, y_test = np.array(xs[idx:]), np.array(labels[idx:])\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "o2lgJKoWIIZT"
      },
      "source": [
        "from tensorflow.keras import models\n",
        "from tensorflow.keras import layers\n",
        "\n",
        "original_model = models.Sequential()\n",
        "original_model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))  #原始的為 16 個單元\n",
        "original_model.add(layers.Dense(16, activation='relu'))\n",
        "original_model.add(layers.Dense(1, activation='sigmoid'))\n",
        "\n",
        "original_model.compile(optimizer='rmsprop',\n",
        "                       loss='binary_crossentropy',\n",
        "                       metrics=['acc'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rWxTQOVGkTsl"
      },
      "source": [
        "### 程式 4.4 容量較低的模型版本"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QBL69--pIbWP"
      },
      "source": [
        "smaller_model = models.Sequential()\n",
        "smaller_model.add(layers.Dense(4, activation='relu', input_shape=(10000,)))  #改成容量較低的 4 個單元\n",
        "smaller_model.add(layers.Dense(4, activation='relu'))\n",
        "smaller_model.add(layers.Dense(1, activation='sigmoid'))\n",
        "\n",
        "smaller_model.compile(optimizer='rmsprop',\n",
        "                      loss='binary_crossentropy',\n",
        "                      metrics=['acc'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "83zKzmCG2crm",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "8bc3369d-4fa3-4c5e-9a8b-18703c481b71"
      },
      "source": [
        "original_hist = original_model.fit(x_train, y_train,\n",
        "                                   epochs=20,\n",
        "                                   batch_size=512,\n",
        "                                   validation_data=(x_test, y_test))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/20\n",
            "49/49 [==============================] - 4s 57ms/step - loss: 0.5445 - acc: 0.7658 - val_loss: 0.3379 - val_acc: 0.8814\n",
            "Epoch 2/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.2722 - acc: 0.9049 - val_loss: 0.2897 - val_acc: 0.8858\n",
            "Epoch 3/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.2019 - acc: 0.9307 - val_loss: 0.3067 - val_acc: 0.8752\n",
            "Epoch 4/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.1687 - acc: 0.9426 - val_loss: 0.2943 - val_acc: 0.8833\n",
            "Epoch 5/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.1398 - acc: 0.9525 - val_loss: 0.3203 - val_acc: 0.8776\n",
            "Epoch 6/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.1221 - acc: 0.9587 - val_loss: 0.3371 - val_acc: 0.8750\n",
            "Epoch 7/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.1036 - acc: 0.9661 - val_loss: 0.3616 - val_acc: 0.8722\n",
            "Epoch 8/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.0949 - acc: 0.9709 - val_loss: 0.3912 - val_acc: 0.8664\n",
            "Epoch 9/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.0813 - acc: 0.9752 - val_loss: 0.4097 - val_acc: 0.8650\n",
            "Epoch 10/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.0710 - acc: 0.9793 - val_loss: 0.4331 - val_acc: 0.8646\n",
            "Epoch 11/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.0639 - acc: 0.9808 - val_loss: 0.4643 - val_acc: 0.8615\n",
            "Epoch 12/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.0526 - acc: 0.9851 - val_loss: 0.5150 - val_acc: 0.8545\n",
            "Epoch 13/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.0481 - acc: 0.9867 - val_loss: 0.5236 - val_acc: 0.8587\n",
            "Epoch 14/20\n",
            "49/49 [==============================] - 2s 37ms/step - loss: 0.0391 - acc: 0.9898 - val_loss: 0.5644 - val_acc: 0.8565\n",
            "Epoch 15/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.0368 - acc: 0.9909 - val_loss: 0.5945 - val_acc: 0.8536\n",
            "Epoch 16/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.0289 - acc: 0.9928 - val_loss: 0.6337 - val_acc: 0.8523\n",
            "Epoch 17/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.0226 - acc: 0.9949 - val_loss: 0.6780 - val_acc: 0.8497\n",
            "Epoch 18/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.0180 - acc: 0.9963 - val_loss: 0.7110 - val_acc: 0.8494\n",
            "Epoch 19/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.0181 - acc: 0.9954 - val_loss: 0.7550 - val_acc: 0.8474\n",
            "Epoch 20/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.0144 - acc: 0.9968 - val_loss: 0.7968 - val_acc: 0.8477\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_w3YYwLq2ckm",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "bef95999-60a9-441f-e49f-7000323b8ffa"
      },
      "source": [
        "smaller_model_hist = smaller_model.fit(x_train, y_train,\n",
        "                                       epochs=20,\n",
        "                                       batch_size=512,\n",
        "                                       validation_data=(x_test, y_test))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/20\n",
            "49/49 [==============================] - 3s 48ms/step - loss: 0.5812 - acc: 0.7245 - val_loss: 0.4165 - val_acc: 0.8604\n",
            "Epoch 2/20\n",
            "49/49 [==============================] - 2s 34ms/step - loss: 0.3494 - acc: 0.8929 - val_loss: 0.3295 - val_acc: 0.8822\n",
            "Epoch 3/20\n",
            "49/49 [==============================] - 2s 34ms/step - loss: 0.2604 - acc: 0.9174 - val_loss: 0.2943 - val_acc: 0.8878\n",
            "Epoch 4/20\n",
            "49/49 [==============================] - 2s 36ms/step - loss: 0.2162 - acc: 0.9299 - val_loss: 0.2837 - val_acc: 0.8883\n",
            "Epoch 5/20\n",
            "49/49 [==============================] - 2s 34ms/step - loss: 0.1839 - acc: 0.9403 - val_loss: 0.2808 - val_acc: 0.8886\n",
            "Epoch 6/20\n",
            "49/49 [==============================] - 2s 34ms/step - loss: 0.1669 - acc: 0.9436 - val_loss: 0.2857 - val_acc: 0.8864\n",
            "Epoch 7/20\n",
            "49/49 [==============================] - 2s 34ms/step - loss: 0.1482 - acc: 0.9524 - val_loss: 0.2908 - val_acc: 0.8856\n",
            "Epoch 8/20\n",
            "49/49 [==============================] - 2s 34ms/step - loss: 0.1377 - acc: 0.9560 - val_loss: 0.3002 - val_acc: 0.8835\n",
            "Epoch 9/20\n",
            "49/49 [==============================] - 2s 34ms/step - loss: 0.1259 - acc: 0.9600 - val_loss: 0.3112 - val_acc: 0.8795\n",
            "Epoch 10/20\n",
            "49/49 [==============================] - 2s 34ms/step - loss: 0.1147 - acc: 0.9649 - val_loss: 0.3227 - val_acc: 0.8777\n",
            "Epoch 11/20\n",
            "49/49 [==============================] - 2s 34ms/step - loss: 0.1095 - acc: 0.9664 - val_loss: 0.3369 - val_acc: 0.8758\n",
            "Epoch 12/20\n",
            "49/49 [==============================] - 2s 34ms/step - loss: 0.0939 - acc: 0.9722 - val_loss: 0.3495 - val_acc: 0.8737\n",
            "Epoch 13/20\n",
            "49/49 [==============================] - 2s 34ms/step - loss: 0.0867 - acc: 0.9759 - val_loss: 0.3667 - val_acc: 0.8712\n",
            "Epoch 14/20\n",
            "49/49 [==============================] - 2s 34ms/step - loss: 0.0802 - acc: 0.9782 - val_loss: 0.3845 - val_acc: 0.8690\n",
            "Epoch 15/20\n",
            "49/49 [==============================] - 2s 34ms/step - loss: 0.0737 - acc: 0.9802 - val_loss: 0.3974 - val_acc: 0.8681\n",
            "Epoch 16/20\n",
            "49/49 [==============================] - 2s 34ms/step - loss: 0.0724 - acc: 0.9818 - val_loss: 0.4212 - val_acc: 0.8628\n",
            "Epoch 17/20\n",
            "49/49 [==============================] - 2s 34ms/step - loss: 0.0649 - acc: 0.9845 - val_loss: 0.4329 - val_acc: 0.8650\n",
            "Epoch 18/20\n",
            "49/49 [==============================] - 2s 34ms/step - loss: 0.0564 - acc: 0.9868 - val_loss: 0.4608 - val_acc: 0.8602\n",
            "Epoch 19/20\n",
            "49/49 [==============================] - 2s 34ms/step - loss: 0.0503 - acc: 0.9889 - val_loss: 0.4704 - val_acc: 0.8615\n",
            "Epoch 20/20\n",
            "49/49 [==============================] - 2s 34ms/step - loss: 0.0458 - acc: 0.9905 - val_loss: 0.4925 - val_acc: 0.8600\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BLdF6cYn2cbF"
      },
      "source": [
        "epochs = range(1, 21)\n",
        "original_val_loss = original_hist.history['val_loss']\n",
        "smaller_model_val_loss = smaller_model_hist.history['val_loss']"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ECwoTpWq2hRs",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "outputId": "72c0a0dc-2592-4909-ae6a-27cf2e8dc5c8"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "plt.plot(epochs, original_val_loss, 'b-', label='Original model')\n",
        "\n",
        "plt.plot(epochs, smaller_model_val_loss, 'bo', label='Smaller model')\n",
        "plt.xlabel('Epochs')\n",
        "plt.ylabel('Validation loss')\n",
        "plt.legend()\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3hUZfbA8e8hlKxIUYEVaQEBBQwIBJBFlCJFVkCwgbgrqKgoWNbFyv5EFJdde1dUioqgWJBVFEQBlQUhYESKVBHCokakhVCScH5/vJMYwiSZlJk75XyeZ57M3LkzczKZ3DP3LecVVcUYY0zsKud1AMYYY7xlicAYY2KcJQJjjIlxlgiMMSbGWSIwxpgYV97rAIqrRo0ampCQ4HUYxhgTUVasWPGrqtb0d1/EJYKEhASSk5O9DsMYYyKKiPxY0H3WNGSMMTHOEoExxsQ4SwTGGBPjIq6PwJ/MzExSU1M5dOiQ16GYAMXHx1O3bl0qVKjgdSjGxLygJgIR6Q08BcQBr6jqhHz31wemAtV9+9ytqnOK+zqpqalUqVKFhIQERKQMIjfBpKrs2rWL1NRUGjZs6HU4xsS8oDUNiUgc8BxwIdAcGCwizfPtNgZ4W1VbA4OA50vyWocOHeKUU06xJBAhRIRTTjnFzuCMCRPB7CNoD2xS1S2qegSYAfTPt48CVX3XqwH/K+mLWRKILPb3MiZ8BDMR1AG257md6tuW11jgKhFJBeYAo/w9kYhcLyLJIpKclpYWjFiNMSZsZWXB6NGwfXvR+5aE16OGBgNTVLUu0Ad4XUSOi0lVJ6pqkqom1azpd2Kc51JTU+nfvz9NmjTh9NNP59Zbb+XIkSN+9/3f//7HpZdeWuRz9unThz179pQonrFjx/Loo4+W6LGBmjJlCiNHjiz1PsaYgmVlwV//Co8+Ch99FJzXCGYi2AHUy3O7rm9bXtcCbwOo6hIgHqgRxJiCQlUZOHAgF198MRs3bmTDhg2kp6dz3333HbdvVlYWp512Gu+8806RzztnzhyqV68ejJCNMREgJwlMnw4TJsCNNwbndYKZCJYDTUSkoYhUxHUGz863zzagO4CINMMlgohr+/n888+Jj49n2LBhAMTFxfHEE08wadIkMjIymDJlCv369aNbt250796drVu3ctZZZwGQkZHB5ZdfTvPmzRkwYAAdOnTILaGRkJDAr7/+ytatW2nWrBnDhw+nRYsW9OzZk4MHDwLw8ssv065dO1q1asUll1xCRkZGobEOHTqUESNGcM4559CoUSMWLlzINddcQ7NmzRg6dGjuftOnTycxMZGzzjqLu+66K3f75MmTadq0Ke3bt2fx4sW529PS0rjkkkto164d7dq1O+Y+Y0zx5U8Cef4Ny1zQho+qapaIjATm4oaGTlLVNSIyDkhW1dnAHcDLInI7ruN4qJZy7czbboOUlNJGf6yzz4Ynnyz4/jVr1tC2bdtjtlWtWpX69euzadMmAFauXMmqVas4+eST2bp1a+5+zz//PCeddBJr165l9erVnH322X5fY+PGjUyfPp2XX36Zyy+/nHfffZerrrqKgQMHMnz4cADGjBnDq6++yqhRfrtacu3evZslS5Ywe/Zs+vXrx+LFi3nllVdo164dKSkp1KpVi7vuuosVK1Zw0kkn0bNnT2bNmkWHDh24//77WbFiBdWqVaNr1660bt0agFtvvZXbb7+dc889l23bttGrVy/WrVtX1FtrjPEjlEkAgjyPwDcnYE6+bf+X5/paoFMwYwgXPXr04OSTTz5u+1dffcWtt94KwFlnnUXLli39Pr5hw4a5SaJt27a5yWT16tWMGTOGPXv2kJ6eTq9evYqMpW/fvogIiYmJ/PGPfyQxMRGAFi1asHXrVn788Ue6dOlCTn/MkCFD+OKLLwCO2X7FFVewYcMGAObPn8/atWtzX2Pfvn2kp6cXGYsx5lihTgIQJTOL8yrsm3uwNG/e/Lg2/3379rFt2zYaN27MypUrqVy5cqleo1KlSrnX4+LicpuGhg4dyqxZs2jVqhVTpkxh4cKFAT9XuXLljnnecuXKkZWVVaLZvkePHmXp0qXEx8cX+7HGGMeLJADejxqKCt27dycjI4PXXnsNgOzsbO644w6GDh3KCSecUOhjO3XqxNtvvw3A2rVr+e6774r12vv376d27dpkZmYybdq0kv0C+bRv355Fixbx66+/kp2dzfTp0zn//PPp0KEDixYtYteuXWRmZjJz5szcx/Ts2ZNnnnkm93ZKWbfPGRPlvEoCYImgTIgI77//PjNnzqRJkyY0bdqU+Ph4Hn744SIfe9NNN5GWlkbz5s0ZM2YMLVq0oFq1agG/9oMPPkiHDh3o1KkTZ555Zml+jVy1a9dmwoQJdO3alVatWtG2bVv69+9P7dq1GTt2LB07dqRTp040a9Ys9zFPP/00ycnJtGzZkubNm/Piiy+WSSzGxAIvkwCAlLJvNuSSkpI0/8I069atO+agFEmys7PJzMwkPj6ezZs3c8EFF7B+/XoqVqzodWhBF8l/N2PKSqiSgIisUNUkf/dFXR9BpMnIyKBr165kZmaiqjz//PMxkQSMMd6fCeSwROCxKlWq2NKbxsSgcEkCYH0ExhgTcuGUBMASgTHGhFS4JQGwRGCMMSETjkkALBEYY0xIhGsSAEsEZWb8+PG0aNGCli1bcvbZZ/P111+XyfOeeOKJAMcUqgsHXbp0KbKTO5B9jIkF2dnhmwQgRhPBtGmQkADlyrmfpZ2Qu2TJEj788MPcwnLz58+nXr16RT8wiLKysjx9fWOMc/QoDB8evkkAYjARTJsG118PP/4Iqu7n9deXLhns3LmTGjVq5NbtqVGjBqeddhrgSknfc889nH322SQlJbFy5Up69erF6aefnjv7Nj09ne7du9OmTRsSExP54IMPCn297OxsRo8eTbt27WjZsiUvvfQSAAsXLqRz587069eP5s3zLw/tzi5Gjx5NixYtuOCCC1i2bBldunShUaNGzJ7tKoQfOnSIYcOGkZiYSOvWrVmwYAEABw8eZNCgQTRr1owBAwbk1joCmDdvHh07dqRNmzZcdtllVmzOGB9VVxF58mS4//7wTAKAW1Qlki5t27bV/NauXXvctoI0aKDq/jzHXho0CPgpjrN//35t1aqVNmnSREeMGKELFy7M83oN9Pnnn1dV1dtuu00TExN13759+ssvv2itWrVUVTUzM1P37t2rqqppaWl6+umn69GjR1VVtXLlyqqq+sMPP2iLFi1UVfWll17SBx98UFVVDx06pG3bttUtW7boggUL9IQTTtAtW7b4jRPQOXPmqKrqxRdfrD169NAjR45oSkqKtmrVSlVVH330UR02bJiqqq5bt07r1aunBw8e1Mceeyx3+7fffqtxcXG6fPlyTUtL086dO2t6erqqqk6YMEEfeOABVVU9//zzdfny5QW+b8X5uxkTie691x1f/vY3Vd+/tGdw5f/9HldjbkLZtm3F2x6IE088kRUrVvDll1+yYMECrrjiCiZMmJC70Eu/fv0ASExMJD09nSpVqlClShUqVarEnj17qFy5Mvfeey9ffPEF5cqVY8eOHfz888+ceuqpfl9v3rx5rFq1Krfi6d69e9m4cSMVK1akffv2NGzY0O/jKlasSO/evXNjqVSpEhUqVCAxMTG3rPVXX32Vu57BmWeeSYMGDdiwYQNffPEFt9xyCwAtW7bMLZe9dOlS1q5dS6dOrpr4kSNH6NixY8nfTGOixD//CQ8/7FocHn0URLyOqGAxlwjq13fNQf62l0ZcXBxdunShS5cuJCYmMnXq1NxEUFTZ52nTppGWlsaKFSuoUKECCQkJHDp0qMDXUlWeeeaZ49YeWLhwYaHlritUqID4Po15Y8mJoyRUlR49ejB9+vQSPd6YaPTMM3DvvXDllfD88+GdBCAG+wjGj4f8laFPOMFtL6n169ezcePG3NspKSk0aNAg4Mfv3buXWrVqUaFCBRYsWMCP/jJVHr169eKFF14gMzMTgA0bNnDgwIGSBZ9P586dc8tZb9iwgW3btnHGGWdw3nnn8eabbwJuMZxVq1YBcM4557B48eLcldgOHDiQu1iNMbFo8mS45Rbo3x+mTIG4OK8jKlrMnREMGeJ+3nefaw6qX98lgZztJZGens6oUaPYs2cP5cuXp3HjxkycOLEYMQ2hb9++JCYmkpSUVGQ56euuu46tW7fSpk0bVJWaNWsya9askv8Cedx0002MGDGCxMREypcvz5QpU6hUqRIjRoxg2LBhNGvWjGbNmuUuzVmzZk2mTJnC4MGDOXz4MAAPPfQQTZs2LZN4jIkkM2fCdddBjx7w1ltQgjWePGFlqI1n7O9moslHH8HFF8M558Ann0ApFyUsc4WVoY65piFjjClrCxbAJZdAq1bw4YfhlwSKYonAGGNKYelS6NsXTj/dnQkUY4HBsBE1iSDSmrhinf29TDRISYELL4RTT4X586FGDa8jKpmoSATx8fHs2rXLDi4RQlXZtWsX8fHxXodiTIl9/z307AlVqsBnn0Ht2l5HVHJRMWqobt26pKamkpaW5nUoJkDx8fHUrVvX6zCMKZEffoALLnD1yubPh2KMFg9LUZEIKlSoUOBsWmOMKUs7dkD37pCRAYsWQTSMlI6KRGCMMaGQlubOBNLSXHNQYqLXEZUNSwTGGBOAPXtcn8DWrW50UPv2XkdUdqKis9gYY4Jp1y7o0wfWrIH334fzz/c6orJlicAYYwqxciUkJcGKFTBjBvgK+EaVoCYCEektIutFZJOI3O3n/idEJMV32SAie4IZjzHGFMfUqdCpk1tv+MsvYeBAryMKjqD1EYhIHPAc0ANIBZaLyGxVXZuzj6renmf/UUDrYMVjjDGBOnIEbr/dlZDu2tWdCdSq5XVUwRPMM4L2wCZV3aKqR4AZQP9C9h8MWFF7Y4ynduyALl1cEhg9GubNi+4kAMEdNVQH2J7ndirQwd+OItIAaAh8XsD91wPXA9Qv7QoyxhhTgC++gMsvh/R0ePttuOwyryMKjXDpLB4EvKOq2f7uVNWJqpqkqkk1a9YMcWjGmGinCk89Bd26uaJxy5bFThKA4CaCHUC9PLfr+rb5MwhrFjLGeODAAbjqKrjtNrjoIpcEmjf3OqrQCmYiWA40EZGGIlIRd7CfnX8nETkTOAlYEsRYjDHmOJs2QceOMH26W6nwvfcis4x0aQWtj0BVs0RkJDAXiAMmqeoaERkHJKtqTlIYBMxQKx1qjAmhjz5yS9TGxbmZwj17eh2Rd4JaYkJV5wBz8m37v3y3xwYzBmOMyevoUXjwQRg7Flq3dmcBCQleR+UtqzVkjIkZu3fDX/7izgauvhpeeAH+8Aevo/KeJQJjTExYtQoGDIDt290cgRtvBBGvowoPlgiMMVFv0SJXNK56dXe9Y0evIwovlgiMMVFt+XK3uHyDBvD55259YXOscJlQZowxZW71alcttEYN+PRTSwIFsURgjIlKmzZBjx4QH+/WFa5Tx+uIwpc1DRljok5qqltSMjPT1Q9q1MjriMKbJQJjTFT55ReXBHbvdn0CsVYuoiQsERhjosaePdCrF2zb5spHt23rdUSRwRKBMSYqHDgAf/6zW1f4P/+Bc8/1OqLIYYnAGBPxDh2Ciy+GpUvdOgK9enkdUWSxRGCMiWhZWTB4sBsZNGUKXHKJ1xFFHhs+aoyJWEePwrBhMGsWPPOMqx9kis8SgTEmIqnCyJHwxhtuLYGRI72OKHJZIjDGRKR773XVQ++8E+65x+toIpslAmNMxPnnP2HCBFdBdMIEqyJaWpYIjDER5dln3dnAkCHw3HOWBMqCJQJjTMSYOhVGjYL+/WHyZChnR7AyYW+jMSYivPceXHONKx8xYwZUqOB1RNHDEoExJuy9/z4MGgQdOrihovHxXkcUXSwRGGPC2muvwWWXubpBc+ZA5cpeRxR9LBEYY8JWziSxLl3cwjLVq3sdUXQqMhGISCcRqey7fpWIPC4iDYIfmjEmVqnCQw/BLbe4GkIffggnnuh1VNErkDOCF4AMEWkF3AFsBl4LalTGmJilCqNHwz/+AX/9K8ycaX0CwRZIIshSVQX6A8+q6nNAleCGZYyJRdnZMHw4PPaYGyY6eTKUt9KYQRfIW7xfRO4BrgLOE5FygA3cMsaUqSNH4Kqr3BnAP/4BDzxgk8VCJZAzgiuAw8C1qvoTUBd4JKhRGWNiSkaGmyQ2cyY8+iiMG2dJIJQCOiMAnlLVbBFpCpwJTA9uWMaYWLF3L1x0ESxeDBMnuqYhE1qBnBF8AVQSkTrAPOAvwJRgBmWMiQ1padC1q1tZbPp0SwJeCSQRiKpmAAOB51X1MuCsQJ5cRHqLyHoR2SQidxewz+UislZE1ojIm4GHboyJZKmpcN55sG4dfPABXHGF1xHFrkCahkREOgJDgGt92wKZfxAHPAf0AFKB5SIyW1XX5tmnCXAP0ElVd4tIreL+AsaYyLNxI/ToAb/9BnPnuoRgvBPIGcFtuIP1+6q6RkQaAQsCeFx7YJOqblHVI8AM3BDUvIYDz6nqbgBV/SXw0I0xkWjVKujcGQ4cgAULLAmEgyLPCFR1EbBIRE4UkRNVdQtwSwDPXQfYnud2KtAh3z5NAURkMRAHjFXVTwKK3BgTcZYuhQsvdPWCFiyAZs28jshAYE08iSLyDbAGWCsiK0SkRRm9fnmgCdAFGAy8LCLHVRMRketFJFlEktPS0sropY0xoTR/vishfcop8NVXlgTCSSBNQy8Bf1PVBqpaH1dm4uUAHrcDqJfndl3ftrxSgdmqmqmqPwAbcInhGKo6UVWTVDWpZs2aAby0MSZcqLqVxP78Z2jYEL78EhISvI7K5BVIIqisqrl9Aqq6EAikEOxyoImINBSRisAgYHa+fWbhzgYQkRq4pqItATy3MSYCpKVBv34wciR07w6LFkHt2l5HZfILJBFsEZF/iEiC7zKGAA7WqpoFjATmAuuAt32dzeNEpJ9vt7nALhFZi+uAHq2qu0r2qxhjwsn8+dCqFcybB089BR99BCef7HVUxh9x9eQK2UHkJOAB4Fzfpi9xnbq7gxybX0lJSZqcnOzFSxtjAnDkCIwZA4884voBpk93CcF4S0RWqGqSv/sCGTW0m8BGCRljYtzGjTB4MKxYATfcAI8/Diec4HVUpigFJgIR+Q9Q4OmCqvYr6D5jTGxRhalTXV9ApUpuofkBA7yOygSqsDOCR0MWhTEmYu3ZAyNGwIwZbknJ11+HunW9jsoUR4GJwDeRzBhjCvTf/8KVV7q6QePHw113QVyc11GZ4rLF640xxZaV5dYM6NwZypVzJaTvvdeSQKSyReCMMcWybRsMGeJmBw8ZAs8/D1Wreh2VKQ1LBMaYgL3zjlszIDvb9QVcdZXXEZmyUGQi8K1KNhpokHd/Ve0WxLiMMWHk0CE3IujVV6F9e3jzTTj9dK+jMmUlkDOCmcCLuPpC2cENxxgTbg4edENB586Fe+5xi8pXqOB1VKYsBZIIslT1haBHYowJOwcOuFpBCxa4s4FrrvE6IhMMgYwa+o+I3CQitUXk5JxL0CMzxnhq/363dsDChfDaa5YEolkgZwRX+36OzrNNgUZlH44xJhzs3euSwLJlrj/A1hOOboHUGmoYikCMMeFh927o1Qu++QbefhsGDvQ6IhNsgYwaqgCMAHJWFl0IvKSqmUGMyxjjgV273KLya9a4ekF9+3odkQmFQJqGXgAqAM/7bv/Ft+26YAVljAm9X35xS0lu2AAffAC9e3sdkQmVQBJBO1XNW038cxH5NlgBGWNCb+dOt4LY1q3w4YcuIZjYEciooWwRyZ06IiKNsPkExkSNHTtc1dBt2+Djjy0JxKJAzghGAwtEZAsguBnGw4IalTEmJH78Ebp1c2sLz50LnTp5HZHxQiCjhj4TkSbAGb5N61X1cHDDMsYE2w8/QNeubj2BTz+FDh28jsh4pbAVyrqp6ucikn/wWGMRQVXfC3Jsxpgg2bjRnQlkZMDnn0ObNl5HZLxU2BnB+cDngL8BZApYIjAmAn3/vUsCmZkuCdjC8qawFcru910dp6o/5L1PRGySmTERaPVqNzpIxJWOaNHC64hMOAhk1NC7fra9U9aBGGOCKyXFjQ4qXx4WLbIkYH5XWB/BmUALoFq+foKqQHywAzPGlM6BA65MRHKyu3z4oVtJ7PPPoXFjr6Mz4aSwPoIzgIuA6hzbT7AfGB7MoIwxxXPoEKxa5Q74y5e7n2vXwtGj7v46ddz8gEcegYbWsGvyKayP4APgAxHpqKpLQhiTMaYQmZmurT/nm/7y5fDdd25BeYCaNaFdO1csLinJXWrX9jZmE94CmVD2jYjcjGsmym0SUlWrTm5MiOzdC+PGweLFrq3/sG8mT/Xq7kD/97+7g39SEtSr5zqDjQlUIIngdeB7oBcwDhgCrAtmUMaY32Vnw6BBbtJXp05w882/H/RPP90O+qb0AkkEjVX1MhHpr6pTReRN4MtgB2aMce68Ez75BF56Ca6/3utoTDQKZPhozroDe0TkLKAaUCt4IRljckyZAo8/DiNHWhIwwRNIIpgoIicB/wBmA2uBfwfy5CLSW0TWi8gmEbnbz/1DRSRNRFJ8F1vjwBif//4XbrjBjfZ54gmvozHRLJCic6/4ri6iGOsUi0gc8BzQA0gFlovIbFVdm2/Xt1R1ZKDPa0ws2LYNBgyA+vXhrbfcJDBjgqWwCWV/K+yBqvp4Ec/dHtikqlt8zzcD6I87ozDGFODAAejXz80NWLQITj7Z64hMtCusaaiK75KEW7O4ju9yIxBIrcI6wPY8t1N92/K7RERWicg7IlLP3xOJyPUikiwiyWlpaQG8tDGR6ehRuPpqNy9gxgw480yvIzLhYNo0SEiAcuXcz2nTyvb5C5tQ9gCAiHwBtFHV/b7bY4GPyuj1/wNMV9XDInIDMBXo5ieWicBEgKSkJC2j1zYm7IwbB+++C489Bhde6HU0JhxMm+YGCmRkuNs//vj7wIEhQ8rmNQLpLP4jcCTP7SO+bUXZAeT9hl/Xty2Xqu7Ks8jNK0DbAJ7XmKg0cyY88AAMHQq33+51NCZc3Hff70kgR0aG215WAumCeg1YJiLv+25fDEwJ4HHLgSa+ktU7gEHAlXl3EJHaqrrTd7MfNlHNxKhvvnFNQn/6E7z4ok0SM7/btq1420sikFFD40XkY6Czb9MwVf0mgMdlichIYC4QB0xS1TUiMg5IVtXZwC0i0g/IAn4Dhpbw9zAmYv30k+scrlED3nsPKlXyOiITTurXd81B/raXFVH13+QuIlVVdZ+I+B2zoKq/lV0YgUtKStLk5GQvXtqYMnf4sFs3+Ntv4auvoHVrryMy4SZ/HwHACSfAxInF6yMQkRWqmuTvvsLOCN7ElaFegVuaMvf5fLcDnlNgjDmeqpswtmSJ6x+wJGD8yTnY33efaw6qXx/Gjy+7jmIofNTQRb6fVr3cmCB4/HGYOhXGjoVLL/U6GhPOhgwp2wN/fgWOGhKRNoVdgheSMdFvzhxXTO7SS+Ef//A6GhNswZ4HUFqFNQ09Vsh9ip/x/saYoq1bB4MHQ8uWrqhcuUAGcZuIFYp5AKVVYGdxuLLOYhPJfvsN2reH9HRYtqxsR36Y8JSQ4H/UT4MGsHVr6OIoaWdx3ic4C2jOsSuUvVY24RkTGzIz4fLLYft2WLjQkkCsCMU8gNIq8qRURO4HnvFduuJKUPcLclzGRJ2//Q0++8wN++vY0etoTHGUpo2/oIQfTl8EAmmdvBToDvykqsOAVrjFaYwxATh4EG68EZ59Fu64w80gNpEjp43/xx/dkN+cNv5Ak8H48W7cf14nnOC2h4tAEsFBVT0KZIlIVeAXjq0hZIwpwPr1cM45bpnJO++Ef/3L64hMcZW21s+QIe4ssEEDVzqkQYPiTwYLtkD6CJJFpDrwMm5yWTqwJKhRGRMFpk1zE8bi4+Gjj6BPH68jMiVRFm38wZ4HUFqFLUzzHPCmqt7k2/SiiHwCVFXVVSGJzpgIlJEBt94Kr7wC554L06dD3bpeR2VKKhS1frxWWNPQBuBREdkqIv8WkdaqutWSgDEF+/576NDBJYF77oEFCywJRLpIaOMvrQITgao+paodgfOBXcAkEfleRO4XkaYhi9CYCPHaa9C2rasm+skn8PDDttZwNIiENv7SKtaEMhFpDUwCWqpqXNCiKoRNKDPh5sABGDUKJk+G886DN9+EOv4WZTXGQ4VNKAtkHkF5EekrItOAj4H1wMAyjtGYiLR2rZspPGUKjBnj5glYEjCRprDO4h7AYKAPsAyYAVyvqgdCFJsxYW3KFLjpJqhSBebNgwsu8DoiY0qmsBbMe3BrEtyhqrtDFI8xYS89HW6+2fUJdO3qhonWru11VMaUXGGdxd1U9RVLAsb8bvVqaNcOXn8d7r8fPv3UkkAkCPcy0F6zMQ3GBEAVJk1yncJVq8L8+dDNCrFHhEgoA+01q4RuTBFWrnQH/euugz/9CVJSLAlEktKWiIgFlgiMKUBqqisQl5TkmoSefRbmzoVTT/U6MlMckVAG2muWCIzJJz3dLR/ZtCnMmAGjR8OmTa6DOM6T2TMm2stAe80SgTE+2dmuNESTJvDQQ9CvnysZ8a9/QTUrvO6ZWCgD7bWYSAQ2YsAUZd48aN0ahg+Hhg1hyRJ3NtCwodeRmVgoA+21qF+zOP+IAXDfBuyDYADWrIG//93VBmrY0H37v/RSd8Aw4aFcOXcmkJ8IHD0a+ngiValKTEQ6GzFg/Pn5Z7dqWMuW7tv/o4/CunVw2WWWBMKNtfEHX9QnAhsxYPI6eNBVBW3cGF59FUaOhM2b3RKSlSp5HZ3xx9r4gy/qE4F9mzDgOoLfeAPOOMOdDV5wgWsWeuopOOUUr6MzhbE2/uCL+kRg3yZi2/798PTTbijoX/4CtWrBwoXw/vtum4kMQ4bA1q2uT2DrVksCZS2oiUBEeovIehHZJCJ3F7LfJSKiIuK3I6M07NtEbNq2zY3/r1fPLRt56qkwcyYsWwbnn+91dMaEl6AlAhGJA54DLgSaA4NFpLmf/aoAtwJfBysW+zYRO77+GgYNgqkQuGAAABDOSURBVEaN4IknoHdvWLoUFi92o4HKRf05cHiyIdzhLZj/Fu2BTaq6RVWP4NYz6O9nvweBfwGHghiLiWJZWfDOO9CpE5xzjhsKevvtsGWLmwvQoYPXEca20k4IM8EXzERQB9ie53aqb1suEWkD1FPVjwp7IhG5XkSSRSQ5LS2t7CM1EWnfPvetv3FjN+zzp59c5+/27fDIIzYgIFzYEO7w51kZahEpBzwODC1qX1WdCEwEN6EsuJGZcPfDD64D+NVXXWfweefBk09C375WCygc2RDu8BfMRLADqJfndl3fthxVgLOAheJm8JwKzBaRfqpqq9Ob4/z3v/D4427ET7lycMUVrgmobVuvIzOFqV/fNQf5227CQzCbhpYDTUSkoYhUBAYBs3PuVNW9qlpDVRNUNQFYClgSMMfZu9cN/ezUCT7/HO6803X6v/GGJYFQKU1nrw3hDn9BOyNQ1SwRGQnMBeKASaq6RkTGAcmqOrvwZzAGvvzSJYHUVLc05OjRULmy11HFltKu8JWzz333ueag+vVdErDRe+Ej6ovOmciUmQljx8KECa4Y3BtvuBFBJvQSEvw37TRo4M7MTGQorOicrVlsws6GDe7bYnIyXHON6wiuUsXrqGKXdfZGP5teY8KGqpv13bq1mwPw7rtuZJAlAW9Zva7oZ4nAhIW0NBgwAG64wS0Qv2oVDBzodVQGrLM3FlgiMJ775BO3LsDHH7vhoXPnQp06RT/OhIbV64p+lgiMZw4ehFGj4MILoUYNWL7czQuwekBlr7S1fqxeV3SzzmLjiZQUdzBZu9ZVB50wAeLjvY4qOpV2+KeJfvbdy4TU0aNuWcgOHeC331yz0JNPWhIIJqv1Y4piZwQmZFJT4eqr3ezgiy+Gl192TUImuGz4pymKnRGYoMvOhldecR3CS5e6BPDee5YEQsWGf5qiWCIwQbVkiWsGGj4cWrRwfQPXXedGn5jQsOGfpiiWCExQ7NwJf/2rmxOwc6frsPziC2jSxOvIYo8N/zRFsURgytSRI25RmKZN4a234J57YP16uPJKOwsoDRv+aYLJOotNmfn4Y7jtNlcrqG9fNzmscWOvo4p8NvzTBJudEZhS27TJHfj79HG358yB2bMtCZQVG/5pgs0SgSmx9HS4917XCbxwIfz73/Ddd26msCk7NvzTBJslAlNsqvDmm3DGGfDPf8KgQa45aPRoqFjR6+jCT2nb9234pwm2mEoEEbYGT1hKSXGLxQ8ZArVru3WEp051183xctr3f/zRff5y2vdtqUcTTmImEXz4IXTvDgcOeB1JZNq2DUaMcGsEr1/vJogtWwYdO3odWXgri/Z9G/5pgi1mEoEqLFrk/nmys4v32NKe2keqrCz44AP485/d7/3yy65a6IYNcO21ViU0EGXVvm/DP00wxcy/ct++8NRT7sB2xx2BP64sTu0jzbZtbqH4hARXEyglBcaMgc2bXYG46tW9jjByWPu+iQQxkwgARo509e6fespdAhErQ/eystyQz4sucovFP/igqw30/vsu+Y0b55okTPFY+76JBDGVCMCVQB440CWEWbOK3j/ah+5t3/77t//+/WHlSjckdMsWNx/g4ouhfIxPOyxN06C175tIIBphQ2mSkpI0OTm5VM+RkQHdurl1cRcuhPbtC943IcF9I86vQQPXVhuJsrLcLOCJE93BXhV69XJNXhddBBUqeB1h+Mg/qxfcN3o7mJtIIyIrVDXJ330xd0YA7h959mw49VTXd/DDDwXvG02n9tu3w9ixrumnXz9ITna1gDZvdolhwABLAvnFStOgiW0xmQgAatVyB7/MTFcaYfdu//tFw6n93r1w883u7GbcODcT+N13XfPWQw+5xGD8i/amQWMghhMBuJmxs2a59vABA+DwYf/7RerQPVVXAfTMM+HFF908gM2b3fKQAwfGzrf/0rTx26gfEwtiOhGAmyU7ebKbY3DttdEz+3jLFnemM2gQnHYafP01PPts7H37L+3w32hqGjSmIDGfCMDVyh8/3h0c7r/f62hK58gRePhh1/zz1Vdu3P/XX0OS3y6i6FfaNv5oaBo0pigxOWrIH1W3nOKrr7rLNdeU+UsE3VdfwQ03wNq1runnqaegbl2vo/JWuXL+z/JEXFOfMbHCs1FDItJbRNaLyCYRudvP/TeKyHcikiIiX4lI82DGUxgReOEF6NnTHUw//bTsnvunn9y39AED3Df0jRvL7rkBfvvNrQPcubMrDf2f/7jO4GhJAtbGb0yQqWpQLkAcsBloBFQEvgWa59unap7r/YBPinretm3bajDt3auamKhatarqqlUlf57sbNX581UvvVS1fHlVUK1f3/0E1aZNVW+/XfWzz1QPHy7Zaxw9qjp1qmqNGqpxcap//7tqenrJYw5Hb7yhesIJv79v4G6/8UZoHm9MtACStaDjdUF3lPYCdATm5rl9D3BPIfsPBj4u6nmDnQhUVbdtUz3tNNV69VR37CjeY9PSVB95RLVxY/funnyy6h13qK5f7+7fskX1mWdUe/dWrVTJ7VOliksYkyer/vxzYK/z/feq3bq5x59zjmpKSvHijBQNGhx7EM+5NGgQ+HO88YbbX8T9tCRgYpFXieBS4JU8t/8CPOtnv5t9Zw7bgSZFPW8oEoGq6jffqJ54omrr1qr79xe+79GjqosWqV55pWrFiu5dPfdc1ddfVz14sOADUXq66gcfqA4f7hIPuH3at1cdN051xQr33HkdPKh6//3udapVU33hBXf2Ea1E/CcCEa8jMyayhHUiyHP/lcDUAu67HkgGkuvXrx+0Nyq/OXNck0ufPqqZmcff/9tvqk8+qdqsmXsnq1VTHTVKdfXq3/cJtGni6FHVlStdAujQ4fcD4GmnqV53neqsWaoff6zapInbPniw6s6dwf39y0ppvpGXxRmBMca7RFDcpqFywN6injdUZwQ5XnrJvUs33ugO1kePqi5Zonr11arx8e6+9u1VJ01SPXDg+MeX9ED288+qU6a4JqOqVX9/XKNGqnPnBuEXDRJr4zcmPHiVCMoDW4CGeTqLW+Tbp0me630LCzTnEupEoKp6113unbr6atVWrdz1E09UveEG9y2+MGXRtHH4sOtUnjRJNSOjVL9KiXj9jd7a+I0pPU8SgXtd+gAbfH0A9/m2jQP6+a4/BawBUoAF+ROFv4sXiSA7W/WKK9y7dfbZqi++qLpvX2CPjfQDYWm/kVsbvzHhwbNEEIyLF4lA1fURrF9/fOdtUcKhacTLb/TWxm9MeLBE4DEvD8Ref6O3Nn5jwkNhicBKTIS50pZIKO3COmWxMM+0aa62z7Ztbkbv+PFWq8eYULOFaSJYaUsklLaefllU34zUMt7GxApLBGGutAfi0iYSq75pTPSzRBDmSnsgtm/0xpiilPc6AFO0IUNKfvDNeZy10RtjCmKJIAaUJpEYY6KfNQ0ZY0yMs0RgjDExzhKBMcbEOEsExhgT4ywRGGNMjIu4EhMikgb4KXoQFmoAv3odRCEsvtIJ9/gg/GO0+EqnNPE1UNWa/u6IuEQQzkQkuaBaHuHA4iudcI8Pwj9Gi690ghWfNQ0ZY0yMs0RgjDExzhJB2ZrodQBFsPhKJ9zjg/CP0eIrnaDEZ30ExhgT4+yMwBhjYpwlAmOMiXGWCIpJROqJyAIRWSsia0TkVj/7dBGRvSKS4rv8X4hj3Coi3/le+7h1PcV5WkQ2icgqEWkTwtjOyPO+pIjIPhG5Ld8+IX//RGSSiPwiIqvzbDtZRD4VkY2+nycV8NirfftsFJGrQxTbIyLyve/v976IVC/gsYV+FoIc41gR2ZHn79ingMf2FpH1vs/j3SGM7608sW0VkZQCHhvU97CgY0pIP38FLWZsF/8XoDbQxne9CrABaJ5vny7Ahx7GuBWoUcj9fYCPAQHOAb72KM444CfcRBdP3z/gPKANsDrPtn8Dd/uu3w38y8/jTga2+H6e5Lt+Ughi6wmU913/l7/YAvksBDnGscDfA/gMbAYaARWBb/P/PwUrvnz3Pwb8nxfvYUHHlFB+/uyMoJhUdaeqrvRd3w+sA+p4G1Wx9QdeU2cpUF1EansQR3dgs6p6PlNcVb8Afsu3uT8w1Xd9KnCxn4f2Aj5V1d9UdTfwKdA72LGp6jxVzfLdXArULcvXLK4C3r9AtAc2qeoWVT0CzMC972WqsPhERIDLgell/bqBKOSYErLPnyWCUhCRBKA18LWfuzuKyLci8rGItAhpYKDAPBFZISLX+7m/DrA9z+1UvElmgyj4n8/L9y/HH1V1p+/6T8Af/ewTDu/lNbgzPH+K+iwE20hf89WkApo2wuH96wz8rKobC7g/ZO9hvmNKyD5/lghKSEROBN4FblPVffnuXolr7mgFPAPMCnF456pqG+BC4GYROS/Er18kEakI9ANm+rnb6/fvOOrOw8NurLWI3AdkAdMK2MXLz8ILwOnA2cBOXPNLOBpM4WcDIXkPCzumBPvzZ4mgBESkAu4PNk1V38t/v6ruU9V03/U5QAURqRGq+FR1h+/nL8D7uNPvvHYA9fLcruvbFkoXAitV9ef8d3j9/uXxc06Tme/nL3728ey9FJGhwEXAEN+B4jgBfBaCRlV/VtVsVT0KvFzAa3v6WRSR8sBA4K2C9gnFe1jAMSVknz9LBMXka098FVinqo8XsM+pvv0Qkfa493lXiOKrLCJVcq7jOhVX59ttNvBX3+ihc4C9eU5BQ6XAb2Fevn/5zAZyRmFcDXzgZ5+5QE8ROcnX9NHTty2oRKQ3cCfQT1UzCtgnkM9CMGPM2+80oIDXXg40EZGGvrPEQbj3PVQuAL5X1VR/d4biPSzkmBK6z1+wesKj9QKciztFWwWk+C59gBuBG337jATW4EZALAX+FML4Gvle91tfDPf5tueNT4DncKM1vgOSQvweVsYd2Kvl2ebp+4dLSjuBTFw767XAKcBnwEZgPnCyb98k4JU8j70G2OS7DAtRbJtwbcM5n8EXffueBswp7LMQwvfvdd/naxXuoFY7f4y+231wI2U2BytGf/H5tk/J+dzl2Tek72Ehx5SQff6sxIQxxsQ4axoyxpgYZ4nAGGNinCUCY4yJcZYIjDEmxlkiMMaYGGeJwBgfEcmWYyujllklTBFJyFv50phwUt7rAIwJIwdV9WyvgzAm1OyMwJgi+OrR/9tXk36ZiDT2bU8Qkc99RdU+E5H6vu1/FLdGwLe+y598TxUnIi/7as7PE5E/+Pa/xVeLfpWIzPDo1zQxzBKBMb/7Q76moSvy3LdXVROBZ4EnfdueAaaqaktc0benfdufBhapK5rXBjcjFaAJ8JyqtgD2AJf4tt8NtPY9z43B+uWMKYjNLDbGR0TSVfVEP9u3At1UdYuvONhPqnqKiPyKK5uQ6du+U1VriEgaUFdVD+d5jgRc3fgmvtt3ARVU9SER+QRIx1VZnaW+gnvGhIqdERgTGC3genEcznM9m9/76P6Mq/3UBljuq4hpTMhYIjAmMFfk+bnEd/2/uGqZAEOAL33XPwNGAIhInIhUK+hJRaQcUE9VFwB3AdWA485KjAkm++ZhzO/+IMcuYP6JquYMIT1JRFbhvtUP9m0bBUwWkdFAGjDMt/1WYKKIXIv75j8CV/nSnzjgDV+yEOBpVd1TZr+RMQGwPgJjiuDrI0hS1V+9jsWYYLCmIWOMiXF2RmCMMTHOzgiMMSbGWSIwxpgYZ4nAGGNinCUCY4yJcZYIjDEmxv0/PVnwoYOI6pUAAAAASUVORK5CYII=\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4VBL10ZwIQog"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RCbKKv_v5PBW"
      },
      "source": [
        "### 程式 4.5 具有更高容量的模型版本"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xTnvguDD6lLj"
      },
      "source": [
        "bigger_model = models.Sequential()\n",
        "bigger_model.add(layers.Dense(512, activation='relu', input_shape=(10000,)))  #改以更高容量的 512 個輸出單位\n",
        "bigger_model.add(layers.Dense(512, activation='relu'))\n",
        "bigger_model.add(layers.Dense(1, activation='sigmoid'))\n",
        "\n",
        "bigger_model.compile(optimizer='rmsprop',\n",
        "                     loss='binary_crossentropy',\n",
        "                     metrics=['acc'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "anZK1H6x6lHp",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "902b18bb-8545-4e44-f36b-e844a895583b"
      },
      "source": [
        "bigger_model_hist = bigger_model.fit(x_train, y_train,\n",
        "                                     epochs=20,\n",
        "                                     batch_size=512,\n",
        "                                     validation_data=(x_test, y_test))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/20\n",
            "49/49 [==============================] - 16s 327ms/step - loss: 0.6349 - acc: 0.7219 - val_loss: 0.2981 - val_acc: 0.8775\n",
            "Epoch 2/20\n",
            "49/49 [==============================] - 16s 323ms/step - loss: 0.2156 - acc: 0.9143 - val_loss: 0.2817 - val_acc: 0.8880\n",
            "Epoch 3/20\n",
            "49/49 [==============================] - 16s 323ms/step - loss: 0.1239 - acc: 0.9538 - val_loss: 0.2965 - val_acc: 0.8853\n",
            "Epoch 4/20\n",
            "49/49 [==============================] - 16s 322ms/step - loss: 0.0484 - acc: 0.9868 - val_loss: 0.4588 - val_acc: 0.8788\n",
            "Epoch 5/20\n",
            "49/49 [==============================] - 16s 324ms/step - loss: 0.0481 - acc: 0.9918 - val_loss: 0.4974 - val_acc: 0.8818\n",
            "Epoch 6/20\n",
            "49/49 [==============================] - 16s 323ms/step - loss: 0.0084 - acc: 0.9984 - val_loss: 0.5105 - val_acc: 0.8640\n",
            "Epoch 7/20\n",
            "49/49 [==============================] - 16s 322ms/step - loss: 0.0026 - acc: 1.0000 - val_loss: 0.7098 - val_acc: 0.8812\n",
            "Epoch 8/20\n",
            "49/49 [==============================] - 16s 324ms/step - loss: 1.7567e-04 - acc: 1.0000 - val_loss: 0.9184 - val_acc: 0.8811\n",
            "Epoch 9/20\n",
            "49/49 [==============================] - 16s 323ms/step - loss: 0.0156 - acc: 0.9988 - val_loss: 1.0332 - val_acc: 0.8466\n",
            "Epoch 10/20\n",
            "49/49 [==============================] - 16s 323ms/step - loss: 8.2416e-04 - acc: 1.0000 - val_loss: 0.8308 - val_acc: 0.8688\n",
            "Epoch 11/20\n",
            "49/49 [==============================] - 16s 323ms/step - loss: 0.0290 - acc: 0.9951 - val_loss: 0.6796 - val_acc: 0.8773\n",
            "Epoch 12/20\n",
            "49/49 [==============================] - 16s 324ms/step - loss: 1.5238e-04 - acc: 1.0000 - val_loss: 0.8230 - val_acc: 0.8800\n",
            "Epoch 13/20\n",
            "49/49 [==============================] - 16s 323ms/step - loss: 1.9204e-05 - acc: 1.0000 - val_loss: 0.9851 - val_acc: 0.8810\n",
            "Epoch 14/20\n",
            "49/49 [==============================] - 16s 324ms/step - loss: 2.4746e-06 - acc: 1.0000 - val_loss: 1.1420 - val_acc: 0.8806\n",
            "Epoch 15/20\n",
            "49/49 [==============================] - 16s 338ms/step - loss: 3.9427e-07 - acc: 1.0000 - val_loss: 1.2785 - val_acc: 0.8808\n",
            "Epoch 16/20\n",
            "49/49 [==============================] - 16s 335ms/step - loss: 8.4296e-08 - acc: 1.0000 - val_loss: 1.4109 - val_acc: 0.8799\n",
            "Epoch 17/20\n",
            "49/49 [==============================] - 16s 326ms/step - loss: 2.7296e-08 - acc: 1.0000 - val_loss: 1.4693 - val_acc: 0.8807\n",
            "Epoch 18/20\n",
            "49/49 [==============================] - 16s 326ms/step - loss: 1.5794e-08 - acc: 1.0000 - val_loss: 1.5049 - val_acc: 0.8809\n",
            "Epoch 19/20\n",
            "49/49 [==============================] - 16s 325ms/step - loss: 1.0297e-08 - acc: 1.0000 - val_loss: 1.5333 - val_acc: 0.8804\n",
            "Epoch 20/20\n",
            "49/49 [==============================] - 16s 324ms/step - loss: 7.7881e-09 - acc: 1.0000 - val_loss: 1.5527 - val_acc: 0.8803\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XkdYQYBj6lDf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 281
        },
        "outputId": "17f0daff-e236-4db4-f65d-f2edd15768ba"
      },
      "source": [
        "bigger_model_val_loss = bigger_model_hist.history['val_loss']\n",
        "\n",
        "plt.plot(epochs, original_val_loss, 'b+', label='Original model')\n",
        "plt.plot(epochs, bigger_model_val_loss, 'bo', label='Bigger model')\n",
        "plt.xlabel('Epochs')\n",
        "plt.ylabel('Validation loss')\n",
        "plt.legend()\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEICAYAAABS0fM3AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZhU5Zn38e9NiyIu7HFQ7G4wGHaUzRASRUkQjYJLFrGHiEZJoswQx8mrBkfa+JIZo280buNgRDTp6MQ1zsREx6gYXBLAgAguoHaTRhOWaJBpiTTc7x/ndFPdVFVXd9Wpqq7z+1xXXVXnqaeq7i6Kuus8q7k7IiISX10KHYCIiBSWEoGISMwpEYiIxJwSgYhIzCkRiIjEnBKBiEjMRZYIzGyxmW02s1fT1JlsZqvMbK2ZLY0qFhERSc2imkdgZscDO4B73X1Ekvt7Ai8A09x9o5l9wt03t/W8ffv29crKypzHKyJSylauXLnV3fslu2+/qF7U3Z8zs8o0Vc4FHnb3jWH9NpMAQGVlJStWrMg+QBGRGDGzulT3FbKP4Gigl5k9a2YrzexrBYxFRCS2IjsjyPC1xwJTgAOBF83sJXd/s3VFM5sDzAEoLy/Pa5AiIqWukGcE9cAT7v6/7r4VeA4Ynayiuy9y93HuPq5fv6RNXCIi0kGFPCP4BXCrme0H7A8cB9zYkSfatWsX9fX17Ny5M5fxSYS6devGgAED6Nq1a6FDEYm9yBKBmd0HTAb6mlk9sADoCuDud7j7a2b2a+AVYA/wY3dPOdQ0nfr6eg455BAqKysxs9z8ARIZd2fbtm3U19czcODAQocjEnuRNQ25+0x37+/uXd19gLvfFSaAOxLqXO/uw9x9hLvf1NHX2rlzJ3369FES6CTMjD59+ugMTiRDNTVQWQldugTXNTW5ff5CNg3llJJA56J/L5HM1NTAnDnQ0BAc19UFxwBVVbl5DS0xISJSxObP35sEmjQ0BOW5okSQI/X19cyYMYPBgwdz1FFHMW/ePD7++OOkdd99912+9KUvtfmcp556Kh988EGH4qmuruaGG27o0GMztWTJEubOnZt1HZFSl03TzsaN7SvviFgngurq3DyPu3PWWWdxxhlnsH79et5880127NjB/CQpu7GxkcMPP5wHH3ywzed9/PHH6dmzZ26CFJGCaGraqasD971NO5kmg1RTp3I5pSrWieCaa3LzPE8//TTdunXj/PPPB6CsrIwbb7yRxYsX09DQwJIlS5g+fTonnXQSU6ZMoba2lhEjguWXGhoa+MpXvsKwYcM488wzOe6445qX0KisrGTr1q3U1tYydOhQLrroIoYPH87UqVP56KOPALjzzjsZP348o0eP5uyzz6ah9TlkK7Nnz+Zb3/oWn/70pxk0aBDPPvssF1xwAUOHDmX27NnN9e677z5GjhzJiBEjuPzyy5vL7777bo4++mgmTJjA888/31y+ZcsWzj77bMaPH8/48eNb3CcSZ9k27SxcCN27tyzr3j0oz5VYJ4JcWbt2LWPHjm1Rduihh1JeXs6GDRsAePnll3nwwQdZurTlIqu33347vXr1Yt26dVx77bWsXLky6WusX7+eSy65hLVr19KzZ08eeughAM466yyWL1/O6tWrGTp0KHfddVeb8b7//vu8+OKL3HjjjUyfPp1LL72UtWvXsmbNGlatWsW7777L5ZdfztNPP82qVatYvnw5jz76KO+99x4LFizg+eefZ9myZaxbt675OefNm8ell17K8uXLeeihh7jwwgvb9R6KlKpsm3aqqmDRIqioALPgetGi3HUUQwmNGspUdXXLM4GmwSsLFuSuqSiZL3zhC/Tu3Xuf8mXLljFv3jwARowYwahRo5I+fuDAgRxzzDEAjB07ltraWgBeffVVrrrqKj744AN27NjBySef3GYsp59+OmbGyJEjOeywwxg5ciQAw4cPp7a2lrq6OiZPnkzTLO6qqiqee+45gBblX/3qV3nzzWBFkKeeeqpFYti+fTs7duxoMxaRUldeHjQHJSvPVFVVbr/4W4vdGUF1ddBO17T6dtPtbJLAsGHD9vklv337djZu3MgnP/lJAA466KCOvwBwwAEHNN8uKyujsbERCJp6br31VtasWcOCBQsyGpvf9FxdunRp8bxdunRpft722rNnDy+99BKrVq1i1apVbNq0iYMPPrhDzyVSbLLp7M1H0062YpcIojBlyhQaGhq49957Adi9ezeXXXYZs2fPpnvrT0ArkyZN4uc//zkA69atY82aNe167Q8//JD+/fuza9cuanI0y2TChAksXbqUrVu3snv3bu677z5OOOEEjjvuOJYuXcq2bdvYtWsXDzzwQPNjpk6dyi233NJ8vGrVqpzEIlJo2Xb25qNpJ1uxTgQLFuTmecyMRx55hAceeIDBgwdz9NFH061bN77//e+3+diLL76YLVu2MGzYMK666iqGDx9Ojx49Mn7ta6+9luOOO45JkyYxZMiQbP6MZv379+ff/u3fOPHEExk9ejRjx45lxowZ9O/fn+rqaiZOnMikSZMYOnRo82NuvvlmVqxYwahRoxg2bBh33HFHmlcQ6TxyMY6/qgpqa2HPnuC6mJIARLhDWVTGjRvnrTemee2111p8KXUmu3fvZteuXXTr1o233nqLz3/+87zxxhvsv//+hQ4tcp35303io0uXvU3JicyCL/bOwsxWuvu4ZPfFrrO42DQ0NHDiiSeya9cu3J3bb789FklApLPIRWdvsYt101AxOOSQQ1ixYgWrV6/mlVde4ZRTTil0SCIlp9Q7e7OlRCAiJS0Onb3ZUiIQkZIWh87ebCkRiEhJy8eibZ2dEoGIlLR8LNrW2SkR5EhZWRnHHHMMo0ePZsyYMbzwwgtA5ktOF7NMZghrFrEUqzh09mYrlokgim3fDjzwQFatWsXq1av513/9V6688kqAjJeczkZHl4UQiYM4dPZmK3aJINsRBJnYvn07vXr1Ash4yem77rqreXnniy66qHkzl1TLO1dXVzNr1iwmTZrErFmzWrz+s88+ywknnMCMGTMYNGgQV1xxBTU1NUyYMIGRI0fy1ltvNcd20kknMWrUKKZMmcLGsNH0nXfeYeLEiYwcOZKrrrqqxXNff/31jB8/nlGjRrEgV1OzRSJW6p29WXP3TnUZO3ast7Zu3bp9ylKpqGhaZq7lpaIi46dIqkuXLj569Gj/1Kc+5YceeqivWLHC3d3feecdHz58uLu7X3/99T5nzhx3d1+zZo2XlZX58uXLfdOmTV5RUeHbtm3zjz/+2D/72c/6JZdc4u7uM2fO9N/+9rfu7l5XV+dDhgxxd/cFCxb4mDFjvKGhYZ9YnnnmGe/Ro4e/++67vnPnTj/88MP96quvdnf3m266yefNm+fu7qeddpovWbLE3d3vuusunzFjhru7n3766X7PPfe4u/utt97qBx10kLu7P/HEE37RRRf5nj17fPfu3f7FL37Rly5d6u7eXKc92vPvJiLZAVZ4iu/V2J0RRDWCoKlp6PXXX+fXv/41X/va1/BW89KXLVvGOeecA7Rccvr3v/89J5xwAr1796Zr1658+ctfbn7MU089xdy5cznmmGOYPn16i+Wdp0+fzoEHHpg0nvHjx9O/f38OOOAAjjrqKKZOnQrAyJEjm5ewfvHFFzn33HMBmDVrFsuWLQPg+eefZ+bMmc3lTZ588kmefPJJjj32WMaMGcPrr7/O+vXrs3rfRDIRRXOu7BXZEhNmthg4Ddjs7iPS1BsPvAic4+7RNqaTn+niEydOZOvWrWzZsiXr52pa3rlbt2773JduaevWy0snLj2dSZ+CNW3UkMDdufLKK/nGN76RSegiOdHUnNs0F6CpORfUxJMrUZ4RLAGmpatgZmXAdcCTEcbRQj5GELz++uvs3r2bPn36tChPteT0+PHjWbp0Ke+//z6NjY3Nu49BtMs7f+Yzn+H+++8HoKamhs997nPNcSaWNzn55JNZvHhx8xnJpk2b2Lx5c87iEUkmFxPCJL3Izgjc/Tkzq2yj2j8ADwHjo4qjtaZfEPPnB81B5eVBEsj2l8VHH33UvIOYu3PPPfdQVlbWos7FF1/Meeedx7BhwxgyZEjzktNHHHEE3/3ud5kwYQK9e/dmyJAhzUtR33zzzVxyySWMGjWKxsZGjj/++Jwt8XzLLbdw/vnnc/3119OvXz/uvvtuAH70ox9x7rnnct111zFjxozm+lOnTuW1115j4sSJQDBk9Kc//Smf+MQnchKPSDKaEBa9SJehDhPBfydrGjKzI4CfAScCi8N6bTYNdeZlqNMtOb1jxw4OPvhgGhsbOfPMM7ngggs488wzCx1ypDrLv5sUVmVl8ubciopgBJBkpliXob4JuNzd9yRrj05kZnOAOQDlnXg6YLolp6urq3nqqafYuXMnU6dO5YwzzihwtCLFYeHCln0EoAlhuVbIRDAOuD9MAn2BU82s0d0fbV3R3RcBiyA4I8hrlDnUtOR0MjfccEOeoxHpHKJqzpW9CpYI3H1g020zW0LQNLRPEmjH8yUd6SLFKcomSSk9VVX64o9SlMNH7wMmA33NrB5YAHQFcPecbmjbrVs3tm3bRp8+fZQMOgF3Z9u2bUmHxIpI/kU5amhmO+rOzua1BgwYQH19fU7G7Ut+dOvWjQEDBhQ6DBGhRPYs7tq1KwMHDmy7ooiI7CN2S0yIiEhLSgQiIjGnRCAiEnNKBCIiMadEICISc0oEIiIxp0QgIhJzSgQiIjGnRCAiEnNKBCIiMadEICKR0+bzxa0k1hoSkeKlzeeLn84IRCRS2ny++CkRiEiktPl88VMiEJFIpdpmvBNvP15ylAhEJFILFwabzSfS5vPFRYlARCJVVQWLFkFFBZgF14sWqaO4mGjUkIhETpvPFzedEYiIxJwSgYhIzCkRiIjEXGSJwMwWm9lmM3s1xf1VZvaKma0xsxfMbHRUsYiISGpRnhEsAaaluf8d4AR3HwlcCyyKMBYREUkhslFD7v6cmVWmuf+FhMOXgAFRxSIiIqkVSx/B14FfFToIEZE4Kvg8AjM7kSARfDZNnTnAHIByzUsXEcmpgp4RmNko4MfADHfflqqeuy9y93HuPq5fv375C1BEJAYKlgjMrBx4GJjl7m8WKg4RkbiLrGnIzO4DJgN9zaweWAB0BXD3O4CrgT7A7WYG0Oju46KKR0REkoty1NDMNu6/ELgwqtcXEZHMFMuoIRERKRAlAhFpkzafL20FHz4qIsVNm8+XPp0RiEha2ny+9LWZCMxskpkdFN7+ezP7oZlVRB+aiBQDbT5f+jI5I/h3oCFcHfQy4C3g3kijEpGioc3nS18miaDR3R2YAdzq7rcBh0QblogUC20+X/oySQQfmtmVwN8DvzSzLoQTw0Sk9Gnz+dKXyaihrwLnAl939z+FS0NcH21YIlJMtPl8acskEXwI/Mjdd5vZ0cAQ4L5owxIRkXzJpGnoOeAAMzsCeBKYRbD7mIiIlIBMEoG5ewNwFnC7u38ZGBFtWCIiki8ZJQIzmwhUAb9sx+NERKQTyOQL/dvAlcAj7r7WzAYBz0QbloiI5EubncXuvhRYamYHm9nB7v428I/RhyYiIvmQyRITI83sD8BaYJ2ZrTSz4dGHJiIi+ZBJ09B/AP/k7hXuXk6wzMSd0YYlIiL5kkkiOMjdm/sE3P1Z4KDIIhIRkbzKJBG8bWb/YmaV4eUq4O2oAxPJJW2sIpJaJongAqAf8HB46ReWiXQKTRur1NWB+96NVZQMRAIWLCzaeYwbN85XrFhR6DCkE6msDL78W6uogNrafEcjUhhmttLdxyW7L+XwUTP7LyBllnD36TmITSRy2lhFJL108whuyFsUIhEqL09+RqCNVUQCKRNBOJGsw8xsMXAasNnd91mbyMwM+BFwKtAAzHb3l7N5TZFkFi5sufk6aGMVkURRrhm0BJiW5v5TgMHhZQ7BlpgiOaeNVUTSy2Q/gg5x9+fMrDJNlRnAveE2mC+ZWU8z6+/u70UVk8SXNlYRSa2Qq4geAfwx4bg+LNuHmc0xsxVmtmLLli15CU6klGgehaTT5hlBuCvZd4CKxPruflKEcbXg7ouARRAMH83X64qUgqZ5FE19JE3zKEBnSRLIpGnoAeAOgvWFdufwtTcBRyYcDwjLRCSH5s9v2VEOwfH8+UoEEsgkETS6exQduY8Bc83sfuA44K/qHxDJPc2jkLZkkgj+y8wuBh4B/tZU6O5/SfcgM7sPmAz0NbN6YAHQNXzsHcDjBENHNxAMHz2/A/GLSBs0j0LakkkiOC+8/k5CmQOD0j3I3We2cb8Dl2Tw+iKSBc2jkLZkskPZwHwEIiLRaOoHmD8/aA4qLw+SgPoHpEkmo4a6At8Cjg+LngX+w913RRiXiOSQ5lFIOpk0Df07Qdv+7eHxrLDswqiCEhGR/MkkEYx399EJx0+b2eqoAhIRkfzKZGbxbjM7qunAzAaR2/kEIm3SzFiR6GRyRvAd4BkzexswghnGGuopeaOZsSLRymiHMjM7APhUePiGu/8tXf0oaYey+NEOYyLZ6+gOZSe5+9Nmdlaruz5pZrj7wzmNUiQFzYwViVa6pqETgKeB05Pc5wQb2YtETjNjRaKVboeyBeHN77n7O4n3mZkmmUneaGasSLQyGTX0UJKyB3MdiEgq2mFMJFrp+giGAMOBHq36CQ4FukUdmEgizYwViU66PoJPEWw+35OW/QQfAhdFGZSIiORPuj6CXwC/MLOJ7v5iHmMSEZE8ymRC2R/M7BKCZqLmJiF3vyCyqEREJG8y6Sz+CfB3wMnAUoItJT+MMigREcmfTBLBJ939X4D/dfd7gC8SbC0pIiIlIJNE0LTvwAdmNgLoAXwiupBERCSfMkkEi8ysF/AvBBvOrwN+EGlUIkVGq59KKWszEbj7j939fXdf6u6D3P0T4ebzIrHQtPppXR247139NJ/JQIlIopRy9VEz+6d0D3T3H0YSURu0+qjkW6FXP229DDcES2xodrW0R7rVR9OdERwSXsYR7Fl8RHj5JjAm10GKFKtCr346f37LJADB8fz5+Xl9KX0pE4G7X+Pu1xAMFx3j7pe5+2XAWCCjdR/NbJqZvWFmG8zsiiT3l5vZM2b2BzN7xcxO7egfIhKVVKuc5mv100InIil9mXQWHwZ8nHD8cViWlpmVAbcBpwDDgJlmNqxVtauAn7v7scA5wO2ZBC2STwsXBk0xifK5+mmhE5EUj+rqaJ43k0RwL/B7M6s2s2rgd8CSDB43Adjg7m+7+8fA/cCMVnWcYBE7CIalvptJ0CL5VOjVTwudiKR4XHNNNM+b6VaVY4DPhYfPufsfMnjMl4Bp7n5heDwLOM7d5ybU6Q88CfQCDgI+7+4rkzzXHGAOQHl5+di6ZD13IiWspiboE9i4MTgTWLhQHcVxZBaMXOvYYzvQWWxmh4bXvYFagqUmfgLUhWW5MBNY4u4DgFOBn5jZPjG5+yJ3H+fu4/r165ejlxbpPKqqghFKe/YE10oC8VFdHSQAs+C46XYum4nSLTr3M4JlqFcSNOE0sfB4UBvPvQk4MuF4QFiW6OvANAB3f9HMugF9gc1tRi4iEgPV1Xu/9LM5I0gn3TLUp4XXHd2WcjkwONzWchNBZ/C5repsBKYAS8xsKMHqpls6+HoiItIB6XYoSztXwN1fbuP+RjObCzwBlAGL3X2tmX0PWOHujwGXAXea2aUEZxmzPZNOCxGRTijx131HLFjQdp2OSDez+Jk0j3N3PymakNLTzGIR6ayiatrJ7LVTdxanaxo6MbqQRESkWGQyjwAzG2FmXzGzrzVdog5MRKQU5GPUT7banEdgZguAyQSzgx8nmCm8zN2/FHl0SahpSEQ6q2JtGsrkjOBLBCN7/uTu5wOjCWYBi4hICcgkEXzk7nuAxnCS2WZazg8QEZEMRDXqJ1uZJIIVZtYTuJNgctnLwIuRRiUlRxurSCnItl2/mPoFEqUbPnob8DN3fz6hrBI41N1fyUt0SaiPoPPRxipSKgrZxp+tjvYRvAncYGa1ZvYDMzvW3WsLmQSkc9LGKiLFLd3GND9y94nACcA2YLGZvW5mC8zs6LxFKJ2eNlaRzqwzDP/MVkbLUDdXNjsWWAyMcveyyKJKQ01DnU+h9/wVyZU4Ng01PXg/MzvdzGqAXwFvAGflOEYpYdpYRZ3lxaKUfsXnUrr9CL5gZouBeuAi4JfAUe5+jrv/Il8BSudX6B2+Cq2ps7yuLvg1WVcXHCsZ5F+2O3wV6/DPbKUbNfQ0wZ4ED7n7+3mNKg01DUlno6ax4tGZm3ay1aGmIXc/yd1/XExJQKQzUmd5YcWhszdbGS06JyIdV17evnLJrerq4Cyg6Uyg6bYSwV5KBCIRU2e5FDslApGIxb2zvJiUamdvtto1j6AYqLNYJL6y3eoxzrJdhlpEpChkO/xTklMiEBGJOSUCESlqGv4ZPfURiEinEecJYdkqWB+BmU0zszfMbIOZXZGizlfMbJ2ZrTWzn0UZj4iI7CuyRGBmZcBtBJvdDwNmmtmwVnUGA1cCk9x9OPDtqOIRkcLLtjlHwz+jEeUZwQRgg7u/7e4fA/cDM1rVuQi4rWkZC3ffHGE8IlJg2Y76Ub9ANKJMBEcAf0w4rg/LEh0NHG1mz5vZS2Y2LdkTmdkcM1thZiu2bNkSUbgiIvFU6FFD+wGDgcnATOBOM+vZupK7L3L3ce4+rl+/fnkOUUSyoVE/xS/KRLAJODLheEBYlqgeeMzdd7n7OwT7JA+OMCYRyTMt+lb8okwEy4HBZjbQzPYHzgEea1XnUYKzAcysL0FT0dsRxiQiWdCXd2mKLBG4eyMwF3gCeA34ubuvNbPvmdn0sNoTwDYzWwc8A3zH3bdFFZOIZEc7fJUmTSgTkYxpQlfnpUXnRKTD1Nlb+nRGICIZ0xlB56UzAhERSUmJQCRGtMSDJKOmIZEYUdNOfKlpSLJWUwOVldClS3BdU1PoiOJJHbQSBSUCaVNNDcyZA3V1wa/JurrgWMkg/zoyjl+jfqQtSgQxkO2v+fnzoaGhZVlDQ1AuxU9LPEhblAhKXC5+zW/c2L5yyS39opeoKRGUuFz8mi8vb1+55FYuf9Fr1I8ko0RQ4nLxa37hQujevWVZ9+5BuXQuOouQZJQISlwufs1XVcGiRVBRETRJVFQEx1VVuYlRMqdf9BIFzSMocU19BInNQ92764tcJG40j6CTy2bUj37NFxc1zUgx0hlBkdMv+tKimb1SKDoj6MQ0hl9EoqZEUOQ0hr/z0zwAKXZKBEVOY/iLS0e+vDWzV4qdEkGR0xj+4pLtnr0ixUiJoMhp1E9p0TwAKUZKBJ1AVRXU1sKePcG1kkB+5bKNX81BUoyUCCRW1MYvsq9IE4GZTTOzN8xsg5ldkabe2WbmZpZ0jKtIrqiNX2RfkSUCMysDbgNOAYYBM81sWJJ6hwDzgN9FFYtIrqiNX0pRlGcEE4AN7v62u38M3A/MSFLvWuA6YGeEsUiJ6GjTjtr4RVKLMhEcAfwx4bg+LGtmZmOAI939lxHGISWko1s1qo1fJLWCdRabWRfgh8BlGdSdY2YrzGzFli1bog9ORCRGokwEm4AjE44HhGVNDgFGAM+aWS3waeCxZB3G7r7I3ce5+7h+/fpFGLIUo1w27aiNX2Rfka0+amb7AW8CUwgSwHLgXHdfm6L+s8A/u3vapUXjtvpoqamuzq5JRqt3inRMQVYfdfdGYC7wBPAa8HN3X2tm3zOz6VG9rhQ3Dd8UKT77Rfnk7v448HirsqtT1J0cZSxSGtS0I5J7mlkskdPwTZHiFqtEoC+R7GmJBpHSE6utKtXRmL1s30P9G4gUhraqlKKhNn6R4lPyiUDbBO7V0b9ZbfwipU1NQzGSi78/7u+hSGelpqEioV/DIlKMYpEIamqgsjK4XVkZHBdCISZT5bppTG38IqWn5BNBTQ3MmQN1dcFxXV1wnM9kkKtEVAxDN3VWI1J6Sr6PoLJybxJIVFER7P/bHh1ZJ+fss+Hhh/ctP+sseOih9j2Xhm6KSEfFuo9g48b2lSfT9Iv+mmva/4t+5cr2lUdJzToikkzJJ4Ly8vaVt5Zt01K2iUhDN0UkaiXfNNT0Rd7QsLese3dYtAiqqtp+fM+e8Ne/7lveowd88EHbj89l05SadkSko2LdNFRVFXzpV1QEX6QVFZknAYDt29tX3trChUHiSdS9e1AuIlIMSj4RQPClX1sLe/YE15kmAci+aSnbRJRIbfwiEoWSbxrKVrZNSyIixSDWTUPZyuUvehGRYhTpDmWloqpKX/wiUrp0RiAiEnNKBCIiMadEICISc0oEIiIxp0QgIhJznW4egZltAZIs2lAU+gJbCx1EGsUeHxR/jIovO4ovO9nEV+Hu/ZLd0ekSQTEzsxWpJmwUg2KPD4o/RsWXHcWXnajiU9OQiEjMKRGIiMScEkFuLSp0AG0o9vig+GNUfNlRfNmJJD71EYiIxJzOCEREYk6JoJ3M7Egze8bM1pnZWjObl6TOZDP7q5mtCi9X5znGWjNbE772Pmt2W+BmM9tgZq+Y2Zg8xvaphPdllZltN7Nvt6qT9/fPzBab2WYzezWhrLeZ/Y+ZrQ+ve6V47HlhnfVmdl4e47vezF4P/w0fMbOeKR6b9vMQYXzVZrYp4d/x1BSPnWZmb4SfxyvyGN9/JsRWa2arUjw20vcv1XdKXj9/7q5LOy5Af2BMePsQ4E1gWKs6k4H/LmCMtUDfNPefCvwKMODTwO8KFGcZ8CeC8c0Fff+A44ExwKsJZT8ArghvXwFcl+RxvYG3w+te4e1eeYpvKrBfePu6ZPFl8nmIML5q4J8z+Ay8BQwC9gdWt/7/FFV8re7/f8DVhXj/Un2n5PPzpzOCdnL399z95fD2h8BrwBGFjardZgD3euAloKeZ9S9AHFOAt9y94BME3f054C+timcA94S37wHOSPLQk4H/cfe/uPv7wP8A0/IRn7s/6e6N4eFLwIBcv26mUrx/mZgAbHD3t939Y+B+gvc9p9LFZ2YGfAW4L9evm4k03yl5+/wpEWTBzCqBY4HfJbl7opmtNrNfmdnwvAYGDjxpZivNbE6S+zBGH98AAARRSURBVI8A/phwXE9hktk5pP7PV8j3r8lh7v5eePtPwGFJ6hTLe3kBwVleMm19HqI0N2y6WpyiaaMY3r/PAX929/Up7s/b+9fqOyVvnz8lgg4ys4OBh4Bvu3vrrexfJmjuGA3cAjya5/A+6+5jgFOAS8zs+Dy/fpvMbH9gOvBAkrsL/f7tw4Pz8KIcYmdm84FGoCZFlUJ9Hv4dOAo4BniPoPmlGM0k/dlAXt6/dN8pUX/+lAg6wMy6EvyD1bj7w63vd/ft7r4jvP040NXM+uYrPnffFF5vBh4hOP1OtAk4MuF4QFiWT6cAL7v7n1vfUej3L8Gfm5rMwuvNSeoU9L00s9nAaUBV+GWxjww+D5Fw9z+7+2533wPcmeJ1C/3+7QecBfxnqjr5eP9SfKfk7fOnRNBOYXviXcBr7v7DFHX+LqyHmU0geJ+35Sm+g8zskKbbBB2Kr7aq9hjwtXD00KeBvyacguZLyl9hhXz/WnkMaBqFcR7wiyR1ngCmmlmvsOljalgWOTObBvwfYLq7N6Sok8nnIar4EvudzkzxusuBwWY2MDxLPIfgfc+XzwOvu3t9sjvz8f6l+U7J3+cvqp7wUr0AnyU4RXsFWBVeTgW+CXwzrDMXWEswAuIl4DN5jG9Q+Lqrwxjmh+WJ8RlwG8FojTXAuDy/hwcRfLH3SCgr6PtHkJTeA3YRtLN+HegD/AZYDzwF9A7rjgN+nPDYC4AN4eX8PMa3gaB9uOlzeEdY93Dg8XSfhzzF95Pw8/UKwZda/9bxhcenEoyUeSuf8YXlS5o+dwl18/r+pflOydvnTzOLRURiTk1DIiIxp0QgIhJzSgQiIjGnRCAiEnNKBCIiMadEIBIys93WcmXUnK2EaWaViStfihST/QodgEgR+cjdjyl0ECL5pjMCkTaE69H/IFyT/vdm9smwvNLMng4XVfuNmZWH5YdZsD/A6vDymfCpyszsznDN+SfN7MCw/j+Ga9G/Ymb3F+jPlBhTIhDZ68BWTUNfTbjvr+4+ErgVuCksuwW4x91HESz4dnNYfjOw1INF88YQzEgFGAzc5u7DgQ+As8PyK4Bjw+f5ZlR/nEgqmlksEjKzHe5+cJLyWuAkd387XBzsT+7ex8y2EiybsCssf8/d+5rZFmCAu/8t4TkqCdaNHxweXw50dff/a2a/BnYQrLL6qIcL7onki84IRDLjKW63x98Sbu9mbx/dFwnWfhoDLA9XxBTJGyUCkcx8NeH6xfD2CwSrZQJUAb8Nb/8G+BaAmZWZWY9UT2pmXYAj3f0Z4HKgB7DPWYlIlPTLQ2SvA63lBua/dvemIaS9zOwVgl/1M8OyfwDuNrPvAFuA88PyecAiM/s6wS//bxGsfJlMGfDTMFkYcLO7f5Czv0gkA+ojEGlD2Ecwzt23FjoWkSioaUhEJOZ0RiAiEnM6IxARiTklAhGRmFMiEBGJOSUCEZGYUyIQEYk5JQIRkZj7/99SPSAF4VxxAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1lEo-wwH6k_2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "outputId": "5bca88ce-3d00-496d-8973-7d765453f21a"
      },
      "source": [
        "original_train_loss = original_hist.history['loss']\n",
        "bigger_model_train_loss = bigger_model_hist.history['loss']\n",
        "\n",
        "plt.plot(epochs, original_train_loss, 'b+', label='Original model')\n",
        "plt.plot(epochs, bigger_model_train_loss, 'bo', label='Bigger model')\n",
        "plt.xlabel('Epochs')\n",
        "plt.ylabel('Training loss')\n",
        "plt.legend()\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de3wV9bnv8c+TiEYuXkBqUUyCFgtBLkLAIla8Fa1W8FLrJbWirbQVdqntOUfdeCTqoa3Fo9ZbW9xea7bWS+2hu7Zaq9JitRJtFMELoIQGbQWqRU6kEnj2HzMrrMS1kpVkzbpkvu/Xa73WzG9+M/MwGeZZ85uZ35i7IyIi8VWS7wBERCS/lAhERGJOiUBEJOaUCEREYk6JQEQk5nbJdwBdtc8++3hlZWW+wxARKSovvPDCRncfnGpa0SWCyspK6uvr8x2GiEhRMbPGdNPUNCQiEnNKBCIiMadEICISc0V3jUBEcmfbtm00NTWxdevWfIciGSorK2Po0KH06dMn43mUCEQkraamJgYMGEBlZSVmlu9wpBPuzqZNm2hqamLYsGEZzxeLpqG6OqishJKS4LuuLt8RiRSHrVu3MmjQICWBImFmDBo0qMtncL3+jKCuDmbNgubmYLyxMRgHqKnJX1wixUJJoLh05+/V688I5s3bmQQSmpuDchERiUEiWLeua+UiUliampqYMWMGw4cP56CDDmLu3Ll89NFHKeu+/fbbfPGLX+x0mSeeeCLvv/9+t+Kpra3l2muv7da8mbrrrruYM2dOj+tkqtcngvLyrpWLSM/V1mZnOe7OaaedximnnMKqVat444032LJlC/NSnNK3tLSw33778dBDD3W63EcffZS99torO0H2Ar0+ESxYAH37ti3r2zcoF5FoXHlldpbz5JNPUlZWxvnnnw9AaWkp119/PXfccQfNzc3cddddTJ8+nWOOOYZjjz2WtWvXcsghhwDQ3NzMl770Jaqqqjj11FM57LDDWrunqaysZOPGjaxdu5aRI0dy4YUXMmrUKKZNm8aHH34IwG233cbEiRMZO3Ysp59+Os3t25jbmTlzJt/85jf5zGc+w4EHHsjTTz/NBRdcwMiRI5k5c2Zrvfvuu4/Ro0dzyCGHcMkll7SW33nnnRx88MFMmjSJZ555prV8w4YNnH766UycOJGJEye2mZYtvT4R1NTAokVQURGMV1QE47pQLFL4VqxYwYQJE9qU7bHHHpSXl7N69WoAXnzxRR566CGWLFnSpt6tt97K3nvvzcqVK7n66qt54YUXUq5j1apVzJ49mxUrVrDXXnvx8MMPA3DaaaexbNkyXnrpJUaOHMntt9/eabzvvfcezz77LNdffz3Tp0/n4osvZsWKFSxfvpyGhgbefvttLrnkEp588kkaGhpYtmwZv/zlL3nnnXeYP38+zzzzDEuXLmXlypWty5w7dy4XX3wxy5Yt4+GHH+ZrX/tal7ZhJnr9XUMQHPRrasAM1q7NdzQivVNtbdszgcTNK/PnZ6+pKJXPfe5zDBw48GPlS5cuZe7cuQAccsghjBkzJuX8w4YNY9y4cQBMmDCBteFB4pVXXuHyyy/n/fffZ8uWLRx//PGdxnLyySdjZowePZp9992X0aNHAzBq1CjWrl1LY2MjRx11FIMHB52A1tTU8Ic//AGgTfmZZ57JG2+8AcATTzzRJjFs3ryZLVu2dBpLV/T6M4La2mCHTOyUieEod0yROKqtBffgAzuHe/J/raqq6mO/5Ddv3sy6dev41Kc+BUC/fv26vwJgt912ax0uLS2lpaUFCJp6br75ZpYvX878+fMzujc/saySkpI2yy0pKWldblft2LGD5557joaGBhoaGli/fj39+/fv1rLSiUUiyPbOKSK5ceyxx9Lc3Mw999wDwPbt2/nud7/LzJkz6dv+4l87U6ZM4YEHHgBg5cqVLF++vEvr/uCDDxgyZAjbtm2jLktPoU6aNIklS5awceNGtm/fzn333cfUqVM57LDDWLJkCZs2bWLbtm08+OCDrfNMmzaNm266qXW8oaEhK7Ek6/WJQERyb/787CzHzHjkkUd48MEHGT58OAcffDBlZWV873vf63Teiy66iA0bNlBVVcXll1/OqFGj2HPPPTNe99VXX81hhx3GlClTGDFiRE/+Ga2GDBnCD37wA44++mjGjh3LhAkTmDFjBkOGDKG2tpbJkyczZcoURo4c2TrPjTfeSH19PWPGjKGqqoqf/OQnWYklmXnip3KRqK6u9u6+mKa2VmcCIl3x6quvtjkoFZPt27ezbds2ysrKWLNmDccddxyvv/46u+66a75Di1yqv5uZveDu1anqx+JicYKSgEh8NDc3c/TRR7Nt2zbcnVtvvTUWSaA7YpUIRCQ+BgwYoNfaZkjXCEREYk6JQEQk5pQIRERiTolARCTmlAhEpKCVlpYybtw4xo4dy/jx4/nTn/4EZN7ldCHL5AnhbD9FnIoSgYhkTRSvhd19991paGjgpZde4vvf/z6XXXYZQMZdTvdEd7uFKDZKBCKSFYnXwjY2Bt24JF4Lm813hG/evJm9994bIOMup2+//fbW7p0vvPDC1pe5pOveuba2lnPPPZcpU6Zw7rnntln/008/zdSpU5kxYwYHHnggl156KXV1dUyaNInRo0ezZs2a1tiOOeYYxowZw7HHHsu68E1Yb731FpMnT2b06NFcfvnlbZa9cOFCJk6cyJgxY5ifrUezM+XuRfWZMGGCi0hurFy5MuO6FRWJnrzafioqehZDSUmJjx071j/96U/7Hnvs4fX19e7u/tZbb/moUaPc3X3hwoU+a9Ysd3dfvny5l5aW+rJly3z9+vVeUVHhmzZt8o8++siPOOIInz17tru7n3322f7HP/7R3d0bGxt9xIgR7u4+f/58Hz9+vDc3N38slqeeesr33HNPf/vtt33r1q2+3377+RVXXOHu7jfccIPPnTvX3d2/8IUv+F133eXu7rfffrvPmDHD3d1PPvlkv/vuu93d/eabb/Z+/fq5u/tjjz3mF154oe/YscO3b9/uJ510ki9ZssTdvbVOV6T6uwH1nua4qjMCEcmKqF4Lm2gaeu211/jtb3/LV77yFbxd1zhLly7lrLPOAtp2Of38888zdepUBg4cSJ8+fTjjjDNa53niiSeYM2cO48aNY/r06W26d54+fTq77757yngmTpzIkCFD2G233TjooIOYNm0aAKNHj27twvrZZ5/lnHPOAeDcc89l6dKlADzzzDOcffbZreUJjz/+OI8//jiHHnoo48eP57XXXmPVqlU92m5doSeLRSQrysuD5qBU5dkyefJkNm7cyIYNG3q8rET3zmVlZR+b1lHX1u27l07uejqTawqW6BM/ibtz2WWX8fWvfz2T0LNOZwQikhW5eC3sa6+9xvbt2xk0aFCb8nRdTk+cOJElS5bw3nvv0dLS0vr2MYi2e+fDDz+c+++/H4C6ujo++9nPtsaZXJ5w/PHHc8cdd7Sekaxfv5533303a/F0RmcEIpIVide/zpsXNAeVlwdJoKevhf3www9b3yDm7tx9992Ulpa2qXPRRRdx3nnnUVVVxYgRI1q7nN5///3593//dyZNmsTAgQMZMWJEa1fUN954I7Nnz2bMmDG0tLRw5JFHZq2L55tuuonzzz+fhQsXMnjwYO68804AfvSjH3HOOedwzTXXMGPGjNb606ZN49VXX2Xy5MlAcMvovffeyyc+8YmsxNOZWHVDLSJdUyzdUHfU5fSWLVvo378/LS0tnHrqqVxwwQWceuqp+Q45UgXVDbWZnQD8CCgF/sPdf5Cm3unAQ8BEd9dRXkS6pKMup2tra3niiSfYunUr06ZN45RTTslztIUnskRgZqXALcDngCZgmZktdveV7eoNAOYCf44qFhHp3Trqcvraa6/NcTTFJ8qLxZOA1e7+prt/BNwPzEhR72rgGqDzN0OLSM4VW/Nx3HXn7xVlItgf+GvSeFNY1srMxgMHuPuvO1qQmc0ys3ozq8/GbWMikpmysjI2bdqkZFAk3J1NmzalvCW2I3m7a8jMSoDrgJmd1XX3RcAiCC4WRxuZiCQMHTqUpqamrNy3L7lRVlbG0KFDuzRPlIlgPXBA0vjQsCxhAHAI8HT4gMUngcVmNl0XjEUKQ58+fRg2bFi+w5CIRdk0tAwYbmbDzGxX4CxgcWKiu//T3fdx90p3rwSeA5QERERyLLJE4O4twBzgMeBV4AF3X2FmV5nZ9KjWKyIiXRPpNQJ3fxR4tF3ZFWnqHhVlLCIikpr6GhIRiTklAhGRmFMiEBGJOSUCEZGYUyIQEYk5JQIRkZhTIhARiTklAhGRmFMiEBGJOSUCEZGYUyIQEYk5JQIRkZhTIhARiTklAhGRmFMiEBGJOSUCEZGYUyIQEYk5JQIRkZhTIhARiTklAhGRmFMiEBGJOSUCEZGYUyIQEYk5JQIRkZhTIhARiTklAhGRmFMiEBGJOSUCEZGYUyIQEYk5JQIRkZhTIhARiTklAhGRmFMiEBGJOSUCEZGYizQRmNkJZva6ma02s0tTTP+GmS03swYzW2pmVVHGIyIiHxdZIjCzUuAW4PNAFXB2igP9f7r7aHcfB/wQuC6qeEREJLUozwgmAavd/U13/wi4H5iRXMHdNyeN9gM8wnhERCSFThOBmU0xs37h8JfN7Dozq8hg2fsDf00abwrL2i9/tpmtITgj+FaaGGaZWb2Z1W/YsCGDVYuISKYyOSP4MdBsZmOB7wJrgHuyFYC73+LuBwGXAJenqbPI3avdvXrw4MHZWrWIiJBZImhxdydo1rnZ3W8BBmQw33rggKTxoWFZOvcDp2SwXBERyaJMEsEHZnYZ8GXg12ZWAvTJYL5lwHAzG2ZmuwJnAYuTK5jZ8KTRk4BVmYUtIiLZsksGdc4EzgG+6u5/M7NyYGFnM7l7i5nNAR4DSoE73H2FmV0F1Lv7YmCOmR0HbAPeA87r7j9ERES6x4JWnw4qBBeKt7r7djM7GBgB/Mbdt+UiwPaqq6u9vr4+H6sWESlaZvaCu1enmpZJ09AfgN3MbH/gceBc4K7shSciIvmUSSIwd28GTgNudfczgEOiDUtERHIlo0RgZpOBGuDXXZhPRESKQCYH9G8DlwGPhBd7DwSeijYsERHJlU7vGnL3JcASM+tvZv3d/U3SPAEsIiLFJ5MuJkab2V+AFcBKM3vBzEZFH5qIiORCJk1DPwW+4+4V7l5O0M3EbdGGJSIiuZJJIujn7q3XBNz9aYKeQmOntjbfEYiIZF8mieBNM/vfZlYZfi4H3ow6sEJ05ZX5jkBEJPsySQQXAIOBX4SfwWGZiIj0Ap0mAnd/z92/5e7jw89cd38vF8EVgtpaMAs+sHNYzUQi0luk7WvIzH5FB28Mc/fpUQXVkXz2NWQGnXTNJCJSkDrqa6ij5wiujSgeEREpIGkTQfggmSSZPz/fEYiIZJ/6DOoCXRcQkd5IiUBEJOaUCEREYq7TTufS3D30T6Ae+Km7b40iMBERyY2MniwGthD0L3QbsBn4ADgY9TkkIlL0Mnl5/eHuPjFp/FdmtszdJ5rZiqgCExGR3MjkjKC/mZUnRsLh/uHoR5FEJSIiOZPJGcF3gaVmtgYwYBhwkZn1A+6OMjgREYleJm8oe9TMhgMjwqLXky4Q3xBZZCIikhOZnBEATAAqw/pjzQx3vyeyqEREJGcyuX30Z8BBQAOwPSx2QIlARKQXyOSMoBqo8nTdlIqISFHL5K6hV4BPRh2IiIjkRyZnBPsAK83seeBficJ8vY9ARESyK5NEUBt1EIWurg7mzYN166C8HBYsgJqafEclIpIdmdw+Guv3EtTVwaxZ0NwcjDc2BuOgZCAivUPaawRmtjT8/sDMNid9PjCzzbkLMb/mzduZBBKam4NyEZHeoKM3lB0Rfg/IXTiFZ926rpWLiBSbjB4oM7NSYN/k+u4ei0NheXnQHJSqXESkN+j09lEz+zfg78DvgF+Hn//KZOFmdoKZvW5mq83s0hTTv2NmK83sZTP7vZlVdDH+yC1YAH37ti3r2zco7yq96lJEClEmzxHMBT7t7qPcfXT4GdPZTOFZxC3A54Eq4Gwzq2pX7S9Adbi8h4Afdi386NXUwKJFUFEBZsH3okXdu1B85ZXZj09EpKcyaRr6K8EbybpqErDa3d8EMLP7gRnAykQFd38qqf5zwJe7sZ7I1dToDiER6b0yfUPZ02Z2WdiU8x0z+04G8+1PkEQSmsKydL4K/CaD5RaV2trgTMIsGE8Mq5lIRApFJmcE68LPruEn68zsywR9Gk1NM30WMAugvMiu0tbW7jzom4F6bBKRQpPJA2XdbdleDxyQND40LGvDzI4D5gFT3f1f7aeHMSwCFgFUV1frUCoikkVpE4GZ3eDu3zazXxF0O91GBn0NLQOGm9kwggRwFnBOu3UcCvwUOMHd3+1q8MVm/vx8RyAi8nEdnRH8LPy+tjsLdvcWM5sDPAaUAne4+wozuwqod/fFwEKC9x8/aEEj+rre3JmdrguISCGyYnvNQHV1tdfX1+c7DBGRomJmL7h7dappmbyhbDjwfYJnAcoS5e5+YNYiFBGRvMnk9tE7gR8DLcDRBK+ovDfKoESyqa4OKiuhpCT4rqvLd0QihSWTRLC7u/+eoBmp0d1rgZOiDUskOxLdiDc2BrfuJroRVzIQ2SmTRPAvMysBVpnZHDM7leACr0jBUzfiIp3LtK+hvsC3gAkE3UCcF2VQItmibsRFOtdhIgg7jjvT3be4e5O7n+/up7v7czmKT5Lo9tOuS/cgepE9oC4SqY7eULaLu28HjshhPNIB9V7addnsRlykt+ro9tHngfHAX8xsMfAg8P8TE939FxHHJtJjiV5j580LmoPKy4MkoN5kRXbK5BpBGbAJOAb4AnBy+C05oN5Le66mBtauhR07gm8lAZG20j5ZbGZNwHWAEfQ1ZEmT3d2viz68j4vzk8XqvVREuqu7TxaXEtwmaimm6XAkItJLdJQI3nH3q3IWiXRKvZeKSBQ6ukaQ6kxA8kjXBUQkCh0lgmNzFoWIiORN2kTg7v/IZSAiIpIfmdw+Kr2EmpZEJBUlghjRk8kikooSgYhIzCkR9HJ6MllEOqN3FseInkwWia+OnizWGYGISMwpEcSInkwWkVSUCGJE1wVEJBUlghyoq4PKSigpCb6L9cXpSiQivZMSQcTq6mDWLGhsDC7UNjYG48WYDPQcgkjvpEQQsXnzoLm5bVlzc1AuIlIIlAgitm5d18oLjZ5DEOn9lAgiVl7etfJCU1sbNGklnj9IDCsRiPQeSgQRW7AA+vZtW9a3b1AuIlIIlAgiVlMDixZBRUXQpFJREYwX4wvUe/ocgs4iRAqTupiQnFEXFyL5oy4mREQkLSUCiZTuOhIpfGoakpxR05BI/uStacjMTjCz181stZldmmL6kWb2opm1mNkXo4xFRERSiywRmFkpcAvweaAKONvMqtpVWwfMBP4zqjikcOiuI5HCFOUZwSRgtbu/6e4fAfcDM5IruPtad38Z2BFhHFIgenogV19HItGIMhHsD/w1abwpLOsyM5tlZvVmVr9hw4asBCciIoGiuGvI3Re5e7W7Vw8ePDjf4UgO6a4jkehFmQjWAwckjQ8Ny0Qyls2+jpQ8RFKLMhEsA4ab2TAz2xU4C1gc4fpEOqRrDCKpRZYI3L0FmAM8BrwKPODuK8zsKjObDmBmE82sCTgD+KmZrYgqHil+eueySDQivUbg7o+6+8HufpC7LwjLrnD3xeHwMncf6u793H2Qu4+KMh4pbt1tDtI1BpGO6cliiQ092Sxxpk7nREQkLSUCiQ092SySmpqGRDKkpiUpZmoairm6OqishJKS4LuuLt8RiUghUSLo5erqYNYsaGwMfs02NgbjSgaZyeZdR2pakkKlpqFerrIyOPi3V1EBa9fmOpri1tOmITUtST6paSjG1q3rWnkqaloS6d2UCHq58vKulbenpqWdunPXUbYfaFPzkkRBTUO9XOJA3ty8s6xvX1i0CGpqOp9fTUvZk42mITUvSXepaSjGamqCg35FRXAQqajIPAlAdpqWRKSwKREUgZ620dfUBL/ed+wIvjNNAtDzpiXZqbsPtOnOJYmamoYKXE+bdop9/dKW7lyS7lLTUBGbN6/tQRiC8XnzcrP+njYtiUjhUyIocIXQRt+TpiXJ7u23hXDnkvQ+ahoqcLprp7gVWtOamobiS01DRWzBguDAkaxv36BcCl++m/ayTWcRvZMSQYFTG31xK4SmvWQ97Yq7p+99ViIpTGoaEolQb2va011LxUtNQyJ50hua9nSxufdTIhCJUG9o2qutDX7FJ37JJ4YzTQR6IK7wqWlIRDKW76YhNS11n5qGRCQrenqxOd90RpGaEoGIZKynB9J8PxDX07ueeis1DYlI0VDTUvepaUhEYquQLlYXatOUzghEpGjU1vbsYJrvM4p8npHojEBEeoVC/UWdK1H9+5UIRCQ28nGxuhgudqtpSEQkQ/luGurJ/GoaEhEpUrno4mOX7C1KRKR36+kDdd1tmkoc9KO62KymIRGRIqGmISla2XxVo0icRdXFR6SJwMxOMLPXzWy1mV2aYvpuZvbzcPqfzawyyngk9xKvamxsDH7JNDYG411JBvlOJPlefzZi0Py9Y/6rropoH3T3SD5AKbAGOBDYFXgJqGpX5yLgJ+HwWcDPO1vuhAkTXIpHRUWi0+K2n4qKzOa/9173vn3bztu3b1CeC/lefzZi0Pzxnj8BqPd0x+t0E3r6ASYDjyWNXwZc1q7OY8DkcHgXYCPhdYt0HyWC4mKWOhGYZTZ/TxNJT+V7/dmIQfPHe/6EjhJBZBeLzeyLwAnu/rVw/FzgMHefk1TnlbBOUzi+Jqyzsd2yZgGzAMrLyyc0pnr3nxSknr6qsaQk9cUxM9ixo6fRFf76sxGD5o/3/DvrF/nFYndf5O7V7l49ePDgfIcjXdDTVzWWl3etPNvyvf5sxKD54z1/JqJMBOuBA5LGh4ZlKeuY2S7AnsCmCGOSHOvpqxrz/c7ffK8/GzFo/njPn5F0bUY9/RC0+b8JDGPnxeJR7erMpu3F4gc6W66uEcTPvfcG7aFmwXcuL9QWwvqzEYPmj/f87nm6RgBgZicCNxDcQXSHuy8ws6vCgBabWRnwM+BQ4B/AWe7+ZkfL1ANlIiJd19E1gki7mHD3R4FH25VdkTS8FTgjyhhERKRjRXGxWEREoqNEICISc0oEIiIxp0QgIhJzRdcNtZltAAr10eJ9CLrJKFSKr2cKPT4o/BgVX8/0JL4Kd0/5RG7RJYJCZmb16W7PKgSKr2cKPT4o/BgVX89EFZ+ahkREYk6JQEQk5pQIsmtRvgPohOLrmUKPDwo/RsXXM5HEp2sEIiIxpzMCEZGYUyIQEYk5JYIuMrMDzOwpM1tpZivMbG6KOkeZ2T/NrCH8XJFqWRHGuNbMlofr/lhXrRa40cxWm9nLZjY+h7F9Omm7NJjZZjP7drs6Od9+ZnaHmb0bvjUvUTbQzH5nZqvC773TzHteWGeVmZ2Xo9gWmtlr4d/vETPbK828He4LEcdYa2brk/6OJ6aZ9wQzez3cHy/NYXw/T4ptrZk1pJk30m2Y7piS0/0vXf/U+qR9z8IQYHw4PAB4A6hqV+co4L/yGONaYJ8Opp8I/AYw4DPAn/MUZynwN4IHXfK6/YAjgfHAK0llPwQuDYcvBa5JMd9AgvduDAT2Dof3zkFs04BdwuFrUsWWyb4QcYy1wP/IYB9YAxzIzveWVOUivnbT/y9wRT62YbpjSi73P50RdJG7v+PuL4bDHwCvAvvnN6oumwHc44HngL3MbEge4jgWWOPueX9S3N3/QPBOjGQzgLvD4buBU1LMejzwO3f/h7u/B/wOOCHq2Nz9cXdvCUefI3gDYN6k2X6ZmASsdvc33f0j4H6C7Z5VHcVnZgZ8Cbgv2+vNRAfHlJztf0oEPWBmlQQv1flzismTzewlM/uNmY3KaWDgwONm9oKZzUoxfX/gr0njTeQnmZ1F+v98+dx+Cfu6+zvh8N+AfVPUKYRteQHBGV4qne0LUZsTNl/dkaZpoxC232eBv7v7qjTTc7YN2x1Tcrb/KRF0k5n1Bx4Gvu3um9tNfpGguWMscBPwyxyHd4S7jwc+D8w2syNzvP5OmdmuwHTgwRST8739PsaD8/CCu9fazOYBLUBdmir53Bd+DBwEjAPeIWh+KURn0/HZQE62YUfHlKj3PyWCbjCzPgR/sDp3/0X76e6+2d23hMOPAn3MbJ9cxefu68Pvd4FHCE6/k60HDkgaHxqW5dLngRfd/e/tJ+R7+yX5e6LJLPx+N0WdvG1LM5sJfAGoCQ8UH5PBvhAZd/+7u2939x3AbWnWndd90cx2AU4Dfp6uTi62YZpjSs72PyWCLgrbE28HXnX369LU+WRYDzObRLCdN+Uovn5mNiAxTHBR8ZV21RYDXwnvHvoM8M+kU9BcSfsrLJ/br53FQOIujPOA/5eizmPANDPbO2z6mBaWRcrMTgD+FzDd3ZvT1MlkX4gyxuTrTqemWfcyYLiZDQvPEs8i2O65chzwmrs3pZqYi23YwTEld/tfVFfCe+sHOILgFO1loCH8nAh8A/hGWGcOsILgDojngMNzGN+B4XpfCmOYF5Ynx2fALQR3aywHqnO8DfsRHNj3TCrL6/YjSErvANsI2lm/CgwCfg+sAp4ABoZ1q4H/SJr3AmB1+Dk/R7GtJmgbTuyDPwnr7gc82tG+kMPt97Nw/3qZ4KA2pH2M4fiJBHfKrIkqxlTxheV3Jfa7pLo53YYdHFNytv+piwkRkZhT05CISMwpEYiIxJwSgYhIzCkRiIjEnBKBiEjMKRGIhMxsu7XtGTVrPWGaWWVyz5cihWSXfAcgUkA+dPdx+Q5CJNd0RiDSibA/+h+GfdI/b2afCssrzezJsFO135tZeVi+rwXvCHgp/BweLqrUzG4L+5x/3Mx2D+t/K+yL/mUzuz9P/0yJMSUCkZ12b9c0dGbStH+6+2jgZuCGsOwm4G53H0PQ6duNYfmNwBIPOs0bT/BEKsBw4BZ3HwW8D5well8KHBou5xtR/eNE0v1bZEQAAAEhSURBVNGTxSIhM9vi7v1TlK8FjnH3N8POwf7m7oPMbCNBtwnbwvJ33H0fM9sADHX3fyUto5Kg3/jh4fglQB93/z9m9ltgC0Evq7/0sMM9kVzRGYFIZjzNcFf8K2l4Ozuv0Z1E0PfTeGBZ2COmSM4oEYhk5syk72fD4T8R9JYJUAP8MRz+PfBNADMrNbM90y3UzEqAA9z9KeASYE/gY2clIlHSLw+RnXa3ti8w/627J24h3dvMXib4VX92WPZvwJ1m9j+BDcD5YflcYJGZfZXgl/83CXq+TKUUuDdMFgbc6O7vZ+1fJJIBXSMQ6UR4jaDa3TfmOxaRKKhpSEQk5nRGICISczojEBGJOSUCEZGYUyIQEYk5JQIRkZhTIhARibn/Bots9HP0oKAOAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hEhoqthkvfP1"
      },
      "source": [
        "## 加入權重常規化\n",
        "採用較小的權重值以抑制神經網路的複雜性  \n",
        "* L1常規化：增加一**代價項**，與權重係數的絕對值成正比\n",
        "* L2常規化/權重衰減：增加一**代價項**，與權重係數的平方成正比\n",
        "於訓練時才會使用到L1、L2正規化，在驗證與測試時會自動拿掉\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NTojcr29KofU"
      },
      "source": [
        "from keras import regularizers\n",
        "\n",
        "l2_model = models.Sequential()\n",
        "l2_model.add(layers.Dense(16, kernel_regularizer=regularizers.l2(0.001),  #加入 L2 權重常規化並將學習率設為 0.001 \n",
        "                          activation='relu', input_shape=(10000,)))\n",
        "l2_model.add(layers.Dense(16, kernel_regularizer=regularizers.l2(0.001),\n",
        "                          activation='relu'))\n",
        "l2_model.add(layers.Dense(1, activation='sigmoid'))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lMjbF9vnTVfk"
      },
      "source": [
        "l2(0.001)代表該層權重矩陣的每個係數都會加上(0.001*權重值)到神經網路的總損失函數上。"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Hbgm4Y3D8Jr4"
      },
      "source": [
        "l2_model.compile(optimizer='rmsprop',\n",
        "                 loss='binary_crossentropy',\n",
        "                 metrics=['acc'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wa5Y-yOQ8MDL",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7c62f94d-4fe4-4741-c1d9-e8ff0a41315e"
      },
      "source": [
        "l2_model_hist = l2_model.fit(x_train, y_train,\n",
        "                             epochs=20,\n",
        "                             batch_size=512,\n",
        "                             validation_data=(x_test, y_test))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/20\n",
            "49/49 [==============================] - 3s 54ms/step - loss: 0.5904 - acc: 0.7363 - val_loss: 0.3968 - val_acc: 0.8596\n",
            "Epoch 2/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.3220 - acc: 0.9034 - val_loss: 0.3276 - val_acc: 0.8894\n",
            "Epoch 3/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.2596 - acc: 0.9245 - val_loss: 0.3250 - val_acc: 0.8875\n",
            "Epoch 4/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.2404 - acc: 0.9319 - val_loss: 0.3294 - val_acc: 0.8862\n",
            "Epoch 5/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.2205 - acc: 0.9402 - val_loss: 0.3403 - val_acc: 0.8829\n",
            "Epoch 6/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.2125 - acc: 0.9438 - val_loss: 0.3446 - val_acc: 0.8806\n",
            "Epoch 7/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.2082 - acc: 0.9470 - val_loss: 0.3659 - val_acc: 0.8762\n",
            "Epoch 8/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.2010 - acc: 0.9481 - val_loss: 0.3564 - val_acc: 0.8794\n",
            "Epoch 9/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.1936 - acc: 0.9518 - val_loss: 0.3745 - val_acc: 0.8745\n",
            "Epoch 10/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.1882 - acc: 0.9543 - val_loss: 0.4253 - val_acc: 0.8596\n",
            "Epoch 11/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.1885 - acc: 0.9510 - val_loss: 0.3786 - val_acc: 0.8742\n",
            "Epoch 12/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.1808 - acc: 0.9568 - val_loss: 0.4205 - val_acc: 0.8616\n",
            "Epoch 13/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.1880 - acc: 0.9500 - val_loss: 0.4021 - val_acc: 0.8696\n",
            "Epoch 14/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.1746 - acc: 0.9589 - val_loss: 0.3976 - val_acc: 0.8714\n",
            "Epoch 15/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.1718 - acc: 0.9585 - val_loss: 0.4031 - val_acc: 0.8702\n",
            "Epoch 16/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.1723 - acc: 0.9592 - val_loss: 0.4311 - val_acc: 0.8631\n",
            "Epoch 17/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.1686 - acc: 0.9611 - val_loss: 0.4213 - val_acc: 0.8669\n",
            "Epoch 18/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.1667 - acc: 0.9616 - val_loss: 0.4209 - val_acc: 0.8669\n",
            "Epoch 19/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.1687 - acc: 0.9594 - val_loss: 0.4264 - val_acc: 0.8668\n",
            "Epoch 20/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.1631 - acc: 0.9619 - val_loss: 0.4464 - val_acc: 0.8630\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "o46DWX_N8NkR",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "outputId": "85a66c0f-3fe9-4f43-f63e-01e7b5f421ea"
      },
      "source": [
        "l2_model_val_loss = l2_model_hist.history['val_loss']\n",
        "\n",
        "plt.plot(epochs, original_val_loss, 'b+', label='Original model')\n",
        "plt.plot(epochs, l2_model_val_loss, 'bo', label='L2-regularized model')\n",
        "plt.xlabel('Epochs')\n",
        "plt.ylabel('Validation loss')\n",
        "plt.legend()\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de5gU9ZX/8fdh1BBQBJX4I4IzmECUu9xd1guihIiC8a4kP8dEjUYS1sSo++jqqCEbs65GEzSLUdEFIYIGSaKrAQV+Gk1mMIAy3kBBB4mOKCHsqDBwfn9UzdAMPT09011d3dOf1/P0M13VdTldNHWq6nszd0dERIpXh7gDEBGReCkRiIgUOSUCEZEip0QgIlLklAhERIrcPnEH0FqHHHKIl5WVxR2GiEhBWbFixYfu3j3ZZwWXCMrKyqiqqoo7DBGRgmJmG5r7TI+GRESKnBKBiEiRUyIQESlyBVdGkMyOHTuoqanh008/jTsUKRAdO3akZ8+e7LvvvnGHIhK7SBOBmU0A7gRKgF+7+0+bfH448CDQNVzmWnd/orX7qamp4YADDqCsrAwzy0Lk0p65O5s3b6ampobevXvHHY5I7CJ7NGRmJcAM4GtAP+B8M+vXZLHrgUfc/WjgPODutuzr008/5eCDD1YSkLSYGQcffLDuIKXgVFREs90oywhGAmvd/S133w7MAyY3WcaBLuH7A4H32rozJQFpDf1epBDddFM0240yERwGvJswXRPOS1QBfMPMaoAngO8l25CZXWpmVWZWVVtbG0WsIiJFK+5aQ+cDs9y9J3AK8N9mtldM7j7T3Ye7+/Du3ZM2jItdTU0NkydPpk+fPnzpS19i2rRpbN++Pemy7733HmeddVaL2zzllFPYsmVLm+KpqKjgtttua9O66Zo1axZTp07NeBkRaV5FBZgFL9j9PpuPiaJMBBuBXgnTPcN5ib4NPALg7i8AHYFDIoxpD9k6kO7OGWecwemnn86bb77JG2+8wbZt27juuuv2Wra+vp4vfvGLLFiwoMXtPvHEE3Tt2jU7QYpIQaqoAPfgBbvfF0oiqAT6mFlvM9uPoDB4UZNl3gHGAZjZUQSJIGfPfrL1vO2ZZ56hY8eOXHTRRQCUlJRwxx13cP/991NXV8esWbOYNGkSJ554IuPGjWP9+vUMGDAAgLq6Os455xz69evH17/+dUaNGtXYhUZZWRkffvgh69ev56ijjuKSSy6hf//+jB8/nk8++QSAe++9lxEjRjB48GDOPPNM6urqUsZaXl7O5ZdfzujRozniiCNYunQp3/rWtzjqqKMoLy9vXG7u3LkMHDiQAQMGcM011zTOf+CBB+jbty8jR47k+eefb5xfW1vLmWeeyYgRIxgxYsQen4lIfossEbh7PTAVeAp4laB20Bozu9nMJoWL/RC4xMxWAXOBci/AsTPXrFnDsGHD9pjXpUsXDj/8cNauXQvASy+9xIIFC1i2bNkey919991069aN6upqbrnlFlasWJF0H2+++SZXXHEFa9asoWvXrjz66KMAnHHGGVRWVrJq1SqOOuoo7rvvvhbj/fjjj3nhhRe44447mDRpEldeeSVr1qzh5ZdfZuXKlbz33ntcc801PPPMM6xcuZLKykoWLlzIpk2buPHGG3n++ed57rnnqK6ubtzmtGnTuPLKK6msrOTRRx/l4osvbtUxFJGW3XhjNNuNtB1B2CbgiSbzbkh4Xw2MiTKGpioq9rwTaHjuduON0VXNAjj55JM56KCD9pr/3HPPMW3aNAAGDBjAoEGDkq7fu3dvhgwZAsCwYcNYv349AK+88grXX389W7ZsYdu2bXz1q19tMZbTTjsNM2PgwIEceuihDBw4EID+/fuzfv16NmzYwAknnEBDecyUKVNYvnw5wB7zzz33XN544w0AFi9evEdi2Lp1K9u2bWsxFhFJX1TnqHbRsrg1Kip2H0yz3c/dMtGvX7+9nvlv3bqVd955hy9/+cu89NJLdO7cOaN9fO5zn2t8X1JS0vhoqLy8nIULFzJ48GBmzZrF0qVL095Whw4d9thuhw4dqK+vb1Nr2127dvHiiy/SsWPHVq8rIvGKu9ZQuzBu3Djq6up46KGHANi5cyc//OEPKS8vp1OnTinXHTNmDI888ggA1dXVvPzyy63a9z/+8Q969OjBjh07mDNnTtu+QBMjR45k2bJlfPjhh+zcuZO5c+dy/PHHM2rUKJYtW8bmzZvZsWMH8+fPb1xn/Pjx/OIXv2icXrlyZVZiEZHoFXUiyNbzNjPjt7/9LfPnz6dPnz707duXjh078pOf/KTFdb/73e9SW1tLv379uP766+nfvz8HHnhg2vu+5ZZbGDVqFGPGjOHII4/M5Gs06tGjBz/96U8ZO3YsgwcPZtiwYUyePJkePXpQUVHBMcccw5gxYzjqqKMa17nrrruoqqpi0KBB9OvXj1/96ldZiUVEomeFVjY7fPhwbzowzauvvrrHSamQ7Ny5kx07dtCxY0fWrVvHSSedxOuvv85+++0Xd2jtXiH/bkRay8xWuPvwZJ8VXRlBvqmrq2Ps2LHs2LEDd+fuu+9WEhCRnFIiiNkBBxygoTdFJFZFXUYgIiJKBCIiRU+JQESkyCkRiIjkSJS9F2RCiSBL9t9//73m3X777fTr149BgwYxbtw4NmzYkPO4TjjhhFYXRt9www0sXrw4430nOybZ1tAxX6bLiORCVAPLZKooE8GcOVBWBh06BH+z1CB3L0cffTRVVVWsXr2as846i6uvvrrFderr66MJJk07d+7k5ptv5qSTToo1DhHJnaJLBHPmwKWXwoYNQT9DGzYE01Ekg7FjxzZ2MTF69GhqamqSLldeXs5ll13GqFGjuPrqq1m3bh0TJkxg2LBhHHvssbz22msArFu3jtGjRzNw4ECuv/76xivupUuXcuqppzZub+rUqcyaNWuv/Vx++eUMHz6c/v37c2NCs+qysjKuueYahg4dyvz58ykvL2fBggVUVVUxZMgQhgwZwsCBAxuHd2wuvrfffptjjjmmMb5k1q9fz5FHHkl5eTl9+/ZlypQpLF68mDFjxtCnTx/+8pe/APDRRx9x+umnM2jQIEaPHs3q1asB2Lx5M+PHj6d///5cfPHFJDaInD17NiNHjmTIkCF85zvfYefOnS3/I4lELBcDy2TM3QvqNWzYMG+qurp6r3nNKS1tGNZhz1dpadqbSKpz584pP7/iiiv8lltuSfrZhRde6BMnTvT6+np3dz/xxBP9jTfecHf3F1980ceOHevu7hMnTvSHH37Y3d3vueeexn0+++yzPnHixD329cADD7i7+/HHH++VlZXu7r5582Z3d6+vr/fjjz/eV61a5e7upaWlfuutt+4Rz/z58/eI8aqrrvKrrroqZXynnXaaP/jgg+7u/stf/jLpMXn77be9pKTEV69e7Tt37vShQ4f6RRdd5Lt27fKFCxf65MmT3d196tSpXlFR4e7uS5Ys8cGDB7u7+/e+9z2/6aab3N3997//vQNeW1vr1dXVfuqpp/r27dvd3f3yyy9vjKW0tNRra2v3iqU1vxuRbIA4902VN3NeLboGZe+807r52TB79myqqqr2Gosg0dlnn01JSQnbtm3jT3/6E2effXbjZ5999hkAL7zwAgsXLgTgggsu4KqrrmpVHI888ggzZ86kvr6eTZs2UV1d3djt9bnnntvser/5zW946aWXePrpp1PG9/zzzzeOk/DNb35zjwFtEvXu3XuPrq/HjRvX2C12Q/fazz33XOO2TjzxRDZv3szWrVtZvnw5jz32GAATJ06kW7duACxZsoQVK1YwYsQIAD755BO+8IUvtOr4iBSroksEhx8ePA5KNj8KixcvZvr06Sxbtqyxy+frrruOP/zhD8DuXjobuqnetWsXXbt2bVXvnfvssw+7du1qnP7000/3Wubtt9/mtttuo7Kykm7dulFeXr7Hcs11k/3KK69QUVHB8uXLKSkpaTG+hsdHqTTt+jqxW+y2lpG4OxdeeCH//u//3qb1RXIhqoFlMlV0ZQTTp0PTnqE7dQrmZ9tf//pXvvOd77Bo0aI9rk6nT5/OypUrk55Mu3TpQu/evRu7eHZ3Vq1aBQTlDA1XyfPmzWtcp7S0lOrqaj777DO2bNnCkiVL9tru1q1b6dy5MwceeCDvv/8+Tz75ZIvxb9myhfPPP5+HHnqocTCaVPGNGTOmMa5Mu8Q+9thjG7exdOlSDjnkELp06cJxxx3Hww8/DMCTTz7Jxx9/DARdgS9YsIAPPvgACMoY4qilJZJKXpULJCi6RDBlCsycCaWlQYFNaWkwPWVKZtutq6ujZ8+eja/bb7+dH/3oR2zbto2zzz6bIUOGMGnSpJY3RHASve+++xg8eDD9+/fn8ccfB+DnP/85t99+O4MGDWLt2rWN3VX36tWLc845hwEDBnDOOedw9NFH77XNwYMHc/TRR3PkkUdywQUXMGZMywPDPf7442zYsIFLLrmksdA4VXx33nknM2bMYODAgWzcuDGt79qciooKVqxYwaBBg7j22mt58MEHAbjxxhtZvnw5/fv357HHHuPw8FauX79+/PjHP2b8+PEMGjSIk08+mU2bNmUUg0ixUDfUBaSuro7Pf/7zmBnz5s1j7ty5jSdhab1i+d2IgLqhbjdWrFjB1KlTcXe6du3K/fffH3dIItIOKBEUkGOPPbbxebyISLa0mzKCQnvEJfHS70Vkt3aRCDp27MjmzZv1n1vS4u5s3ryZjh07xh2KSF5oF4+GevbsSU1NDbW1tXGHIgWiY8eO9OzZM+4wRPJCu0gE++67L7179447DBGRgtQuHg2JiEjbKRGIiKQpX1sGZ0qJQEQkTfk6sEymlAhERIqcEoGISAoFMbBMhiLta8jMJgB3AiXAr939p00+vwMYG052Ar7g7l1TbTNZX0MiIrlgFgxlVYhi6WvIzEqAGcDJQA1QaWaL3L26YRl3vzJh+e8Be3ebKSIikYry0dBIYK27v+Xu24F5wOQUy58PzI0wHhGRjOTrwDKZijIRHAa8mzBdE87bi5mVAr2BZ5r5/FIzqzKzKrUeFpG4tKdygUT5Ulh8HrDA3Xcm+9DdZ7r7cHcf3jBSloiIZEeUiWAj0Cthumc4L5nz0GMhEZFYRJkIKoE+ZtbbzPYjONkvarqQmR0JdANeiDAWERFpRmSJwN3rganAU8CrwCPuvsbMbjazxMF7zwPmufqQFhGJRaS9j7r7E8ATTebd0GS6IsoYREQktXwpLBYRiVx7rfWTKSUCESka7bXTuEwpEYiIFDklAhFp14qh07hMRdrpXBTU6ZyItFUhdxqXqVSdzumOQESkyCkRiEjRaK+dxmVKiUBEiobKBZJTIhARKXJKBCIiRU6JQESkyCkRiIgUOSUCEZEip0QgIlLklAhERIqcEoGISJFTIhARKXJKBCJSMNQyOBpKBCJSMDSwTDSUCEREipwSgYjkNQ0sE70WE4GZjTGzzuH7b5jZ7WZWGn1oIiLBCd9994AyDe+VCLInnTuCe4A6MxsM/BBYBzwUaVQiIpIz6SSCeg/Gs5wM/NLdZwAHRBuWiMjeNLBMNPZJY5l/mNm/At8AjjOzDsC+0YYlIrI3PQ6KRjp3BOcCnwHfdve/AT2B/4g0KhERyZm07giAO919p5n1BY4E5kYbloiI5Eo6dwTLgc+Z2WHA08A3gVlRBiUiIrmTTiIwd68DzgDudvezgQHRhiUiIrmSViIws2OAKcAfWrGeiIgUgHRO6P8C/CvwW3dfY2ZHAM+ms3Ezm2Bmr5vZWjO7tpllzjGzajNbY2YPpx+6iIhkQ4uFxe6+DFhmZvub2f7u/hbw/ZbWM7MSYAZwMlADVJrZInevTlimD0GSGePuH5vZF9r6RUREpG3S6WJioJn9FVgDVJvZCjPrn8a2RwJr3f0td98OzCNolJboEmCGu38M4O4ftC58ESkkageQn9J5NPRfwA/cvdTdDyfoZuLeNNY7DHg3YbomnJeoL9DXzJ43sxfNbEKyDZnZpWZWZWZVtbW1aexaRPKRupHOT+kkgs7u3lgm4O5Lgc5Z2v8+QB/gBOB84F4z69p0IXef6e7D3X149+7ds7RrERGB9BLBW2b2b2ZWFr6uB95KY72NQK+E6Z7hvEQ1wCJ33+HubwNvECQGEWkn1I10/ksnEXwL6A48Fr66h/NaUgn0MbPeZrYfcB6wqMkyCwnuBjCzQwgeFaWTZESkQKgb6fyXTq2hj0mjllCS9erNbCrwFFAC3B9WP70ZqHL3ReFn482sGtgJ/MjdN7d2XyIi0nbNJgIz+x3gzX3u7pNa2ri7PwE80WTeDQnvHfhB+BKRdk7dSOenVHcEt+UsChEpCnoclJ+aTQRhQzIREWnn1GeQiEiRUyIQESlySgQiIkUunb6G+prZvWb2tJk90/DKRXAikl9U2Ns+mXuzNUSDBcxWAb8CVhDU9QfA3VdEG1pyw4cP96qqqjh2LVL0zHY3DJPCYmYr3H14ss/SGbO43t3vyXJMIiKSJ9IpI/idmX3XzHqY2UENr8gjE5G8oL6C2r90Hg29nWS2u/sR0YSUmh4NicRHj4YKV0aPhty9d/ZDEhGRfNFiIjCzfYHLgePCWUuB/3L3HRHGJSJ5SH0FtU/pFBbfA+wL3B1OfzOcd3FUQYlIflK5QPuUTiIY4e6DE6afCauUiohIO5BOraGdZvalhgkzO4KE9gQiIlLY0rkj+BHwrJm9BRhQClwUaVQiIpIz6dQaWmJmfYCvhLNed/fPog1LRERyJdUIZSe6+zNmdkaTj75sZrj7YxHHJiIiOZDqjuB44BngtCSfOcFA9iIiUuBSjVDWUGP4Znffo3WxmamRmYhIO5FOraFHk8xbkO1AREQkHqnKCI4E+gMHNikn6AJ0jDowEcm+igo1CpO9pboj+ApwKtCVoJyg4TUUuCT60EQk2266Ke4IJB+lKiN4HHjczI5x9xdyGJOIiORQOmUEfzWzK8zsbjO7v+EVeWQispe2PNbReALSknTGI5gPvAZcANwMTAFedfdp0Ye3N41HIMUs0/EANJ5A8Uo1HkE6dwRfdvd/A/7X3R8EJgKjshmgiIjEJ51E0DDuwBYzGwAcCHwhupBEJFE2H+1oPAFJJp1HQxcTtCUYBDwA7A/c4O6/ij68venRkBQzPdqRtsp0qMpfh2+XAbGMUywiItFJ1aDsB6lWdPfbW9q4mU0A7gRKgF+7+0+bfF4O/AewMZz1y4TEIyJN6NGORCHVHcEB4d+vACOAReH0acBfWtqwmZUAM4CTgRqg0swWuXt1k0V/4+5TWxW1SJFSlU+JQqoGZTcBmNlyYKi7/yOcrgD+kMa2RwJr3f2tcL15wGSgaSIQEZEYpVNr6FBge8L09nBeSw4D3k2YrgnnNXWmma02swVm1ivZhszsUjOrMrOq2traNHYtIiLpSicRPAT8xcwqwruBPwOzsrT/3wFl7j4I+CPwYLKF3H2muw939+Hdu3fP0q5FRATSqzU03cyeBI4NZ13k7n9NY9sbgcQr/J7sLhRu2PbmhMlfAz9LY7siIpJFqWoNdXH3rWZ2ELA+fDV8dpC7f9TCtiuBPuEgNhuB8wi6qUjcRw933xROTgJebfU3EBGRjKS6I3iYoBvqFQRDUzawcDplmwJ3rzezqcBTBNVH73f3NWZ2M1Dl7ouA75vZJKAe+Agob+sXERGRtmmxZXG+UctiEZHWa1PLYjMbmmqj7v5SpoGJiEj8Uj0a+s8UnzlwYpZjERGRGKRqUDY2l4GIiEg8Wqw+ChB2P92PhEHr3f2hqIISEZHcaTERmNmNwAkEieAJ4GvAcwQNzUREpMCl07L4LGAc8Dd3vwgYTDA4jYi0gjqMk3yVTiL4xN13AfVm1gX4gD1bDItIGm66Ke4IRJJLp4ygysy6AvcSNC7bBrwQaVQiIpIzzd4RmNkMMxvj7t919y3h0JQnAxeGj4hEpAXZHG9YJCrNtiw2s2kE/QP1AB4B5qbZ2Vyk1LJYCpXGG5Y4pWpZ3Owdgbvf6e7HAMcDm4H7zew1M7vRzPpGFKuIiORYi4XF7r7B3W9196OB84HTUS+hIq2m8YYlX7WYCMxsHzM7zczmAE8CrwNnRB6ZSDujcgFpqzlzoKwMOnQI/s6Zk93tp+p07mSCO4BTCAarnwdc6u7/m90QRESkOXPmwKWXQl1dML1hQzANMGVKdvaRqrD4GYIxCR5194+zs7vMqbBYRIpJWVlw8m+qtBTWr09/O23qhtrd1buoiEjM3nmndfPbIp2WxSIiEpPDD2/d/LZQIhBJkwp7JQ7Tp0OnTnvO69QpmJ8tSgQiaVJfQfGJutZMPpsyBWbODMoEzIK/M2dmr6AY0hyPQEQkLrmoNZPvpkyJ9rvqjkAkBfUVFL/rrtudBBrU1QXzC0W+39HojkAkhT59guexiSeiTp2C+ZIbuag1E6VCuKPRHYFICvlyNZrvV5RRykatmUyPXybr58tvKCV3L6jXsGHDXCRXzNyDPkP3fJnlLobZs907ddpz/506BfOLQabfP+718+E35O4OVHkz59VmWxbnK7UsllzKVqvOQo8hbnPmBFfQ77wT3AlMn57+Y5VMj1/c62dLm7qhFpHc1OFuSaE/I8+GKVOCk+auXcHf1jxbz/T4Zbp+PvyGWqJEIEWjLTV9clGHuyW5aFnanmV6/DJdPx9+Qy1RIpCi0dYGYZlcjWZDIVxR5rNMj182jn/cv6GWKBGI5LlCuKLMZ5kev2I4/koE0q61lwZhmV5Rxl39NO79Z3r88v2KPlORJgIzm2Bmr5vZWjO7NsVyZ5qZm1nSEm2Rtqqo2F1hD3a/L7REkImGBk0bNgTfvaFBU65OxnHvX1oWWSIwsxJgBvA1oB9wvpn1S7LcAcA04M9RxRL31YhInLLRoKndN6gqclHeEYwE1rr7W+6+nWCoy8lJlrsFuBX4NIogdDUiDYp18PhMqz9m+n9I1V/zX5SJ4DDg3YTpmnBeIzMbCvRy9z+k2pCZXWpmVWZWVVtb26ogdDXSfmT6OKeYHgclyrT6Y6b/h1T9Nf/FVlhsZh2A24EftrSsu8909+HuPrx79+6t2o+uRtoPjQfQNplWfyyGBlXFLspEsBHolTDdM5zX4ABgALDUzNYDo4FF2S4w1tWIFLtMqz8WQ4OqYhdlIqgE+phZbzPbDzgPWNTwobv/3d0Pcfcydy8DXgQmuXtWOxLS1Uhhay/VP+OWSfXHYmhQVewiSwTuXg9MBZ4CXgUecfc1ZnazmU2Kar9N6WqksKn6Z/z0f6j9U++jUjDMdieEQpJJz5ki2ZKq91GNUCYFoxCrfxbC6FQi6mJCCkZbHwfF2aBQ1ZelEOiOQNq1uK/IVX1ZCoHuCKRdi/uKXNWXpRAoEUjk4nw0E/cVuaovSyFQIpBIxd3XU9xX5Kp6KYVA1UclUnEP3N20jACCK3KdjKXYaPB6iU3cj2Z0RS7SMiUCiVQ2Hs1kWsag7g1EUlMikEgNG9a6+U3FXcYgUgyUCCRSjz4Ks2cHj2Qg+Dt7djA/HXFX/xQpBioslpxpS19BHTokX8cseNQjIulRYbHkhbb0FRR39U+RYqBEIC3KVoOwtvQVpAZZItFTIigCmZzI4y6sVfVPkegVVRlBRUXxDWiSaYOquBuEiUh2pCojKKpEUKgDm2Qi0xO5CmtF2gcVFhexTFv2qrBWpP1r94mgPQx+nskz/kxP5Jk2CBOR/FcUiSDTwc/j7EY508LaTGvdZNogTEQKgLsX1GvYsGHeVtD6dWbPdu/UqSF9BK9OnYL5rdlGaam7WfC3NeuWlu6574ZXaWlu9p+oLcdPRPIDUOXNnFeLqrC4LbWGMi1szbTWTj4V1hZjrSuR9kK1hjKQ6Yk400SSzeqbOpGLFC/VGspApoWtmdbayWbL2ptuav06ItL+KRG0INMTcaaJRC1rRSRqSgQtyPREnI0r+kwGVmkP1WdFJFoqI8iBOXOC/vPfeSe4E5g+PZ4r+mJsWS0iAZURxKzhiv6GGzRUoojkHyWCHIq7sLYt4wGISPunRFBEVC4gIslEmgjMbIKZvW5ma83s2iSfX2ZmL5vZSjN7zsz6RRlPHFRYKyL5LrLCYjMrAd4ATgZqgErgfHevTlimi7tvDd9PAr7r7hNSbbcQC4sbqLBWROISV2HxSGCtu7/l7tuBecDkxAUakkCoM6DTZAq6ixCRKESZCA4D3k2Yrgnn7cHMrjCzdcDPgO8n25CZXWpmVWZWVVtbG0mwuZBpYW3chc0i0j7FXljs7jPc/UvANcD1zSwz092Hu/vw7t275zbABJlekeuKXkTyUZSJYCPQK2G6ZzivOfOA0yOMJ2NxXJGrsFlEohZlIqgE+phZbzPbDzgPWJS4gJn1SZicCLwZYTwFKRsD64iIpBJZInD3emAq8BTwKvCIu68xs5vDGkIAU81sjZmtBH4AXBhVPG2lK3IRae/U11ArxF39U+MJiEhbqa+hdkJJQESioETQCuqrR0TaIyWCVtAVuYi0R0oEIiJFTolARKTIKRGIiBQ5JQIRkSKnRCAiUuQKrkGZmdUCG+KOoxmHAB/GHUQKii8z+R4f5H+Mii8zmcRX6u5Je+0suESQz8ysqrmWe/lA8WUm3+OD/I9R8WUmqvj0aEhEpMgpEYiIFDklguyaGXcALVB8mcn3+CD/Y1R8mYkkPpURiIgUOd0RiIgUOSUCEZEip0TQSmbWy8yeNbPqcHS1aUmWOcHM/m5mK8PXDTmOcb2ZvRzue69RfCxwl5mtNbPVZjY0h7F9JeG4rDSzrWb2L02WyfnxM7P7zewDM3slYd5BZvZHM3sz/NutmXUvDJd508yyPspeM7H9h5m9Fv77/dbMujazbsrfQsQxVpjZxoR/x1OaWXeCmb0e/h6vzWF8v0mIbX04UmKydSM9hs2dU3L6+3N3vVrxAnoAQ8P3BwBvAP2aLHMC8PsYY1wPHJLi81OAJwEDRgN/jinOEuBvBA1dYj1+wHHAUOCVhHk/A64N318L3JpkvYOAt8K/3cL33XIQ23hgn/D9rcliS+e3EHGMFcBVafwG1gFHAPsBq5r+f4oqviaf/ydwQ6/ntv0AAAT8SURBVBzHsLlzSi5/f7ojaCV33+TuL4Xv/0EwHvNh8UbVapOBhzzwItDVzHrEEMc4YJ27x95S3N2XAx81mT0ZeDB8/yBwepJVvwr80d0/cvePgT8CE6KOzd2f9mBccIAXgZ7Z3GdrNXP80jESWOvub7n7dmAewXHPqlTxmZkB5wBzs73fdKQ4p+Ts96dEkAEzKwOOBv6c5ONjzGyVmT1pZv1zGhg48LSZrTCzS5N8fhjwbsJ0DfEks/No/j9fnMevwaHuvil8/zfg0CTL5MOx/BbBHV4yLf0WojY1fHx1fzOPNvLh+B0LvO/ubzbzec6OYZNzSs5+f0oEbWRm+wOPAv/i7lubfPwSweOOwcAvgIU5Du+f3X0o8DXgCjM7Lsf7b5GZ7QdMAuYn+Tju47cXD+7D866utZldB9QDc5pZJM7fwj3Al4AhwCaCxy/56HxS3w3k5BimOqdE/ftTImgDM9uX4B9sjrs/1vRzd9/q7tvC908A+5rZIbmKz903hn8/AH5LcPudaCPQK2G6Zzgvl74GvOTu7zf9IO7jl+D9hkdm4d8PkiwT27E0s3LgVGBKeKLYSxq/hci4+/vuvtPddwH3NrPvWH+LZrYPcAbwm+aWycUxbOackrPfnxJBK4XPE+8DXnX325tZ5v+Ey2FmIwmO8+YcxdfZzA5oeE9QqPhKk8UWAf83rD00Gvh7wi1orjR7FRbn8WtiEdBQC+NC4PEkyzwFjDezbuGjj/HhvEiZ2QTgamCSu9c1s0w6v4UoY0wsd/p6M/uuBPqYWe/wLvE8guOeKycBr7l7TbIPc3EMU5xTcvf7i6okvL2+gH8muEVbDawMX6cAlwGXhctMBdYQ1IB4EfinHMZ3RLjfVWEM14XzE+MzYAZBbY2XgeE5PoadCU7sBybMi/X4ESSlTcAOgues3wYOBpYAbwKLgYPCZYcDv05Y91vA2vB1UY5iW0vwbLjhN/ircNkvAk+k+i3k8Pj9d/j7Wk1wUuvRNMZw+hSCmjLroooxWXzh/FkNv7uEZXN6DFOcU3L2+1MXEyIiRU6PhkREipwSgYhIkVMiEBEpckoEIiJFTolARKTIKRGIhMxsp+3ZM2rWesI0s7LEni9F8sk+cQcgkkc+cfchcQchkmu6IxBpQdgf/c/CPun/YmZfDueXmdkzYadqS8zs8HD+oRaMEbAqfP1TuKkSM7s37HP+aTP7fLj898O+6Feb2byYvqYUMSUCkd0+3+TR0LkJn/3d3QcCvwR+Hs77BfCguw8i6PTtrnD+XcAyDzrNG0rQIhWgDzDD3fsDW4Azw/nXAkeH27ksqi8n0hy1LBYJmdk2d98/yfz1wInu/lbYOdjf3P1gM/uQoNuEHeH8Te5+iJnVAj3d/bOEbZQR9BvfJ5y+BtjX3X9sZv8DbCPoZXWhhx3uieSK7ghE0uPNvG+NzxLe72R3Gd1Egr6fhgKVYY+YIjmjRCCSnnMT/r4Qvv8TQW+ZAFOA/xe+XwJcDmBmJWZ2YHMbNbMOQC93fxa4BjgQ2OuuRCRKuvIQ2e3ztucA5v/j7g1VSLuZ2WqCq/rzw3nfAx4wsx8BtcBF4fxpwEwz+zbBlf/lBD1fJlMCzA6ThQF3ufuWrH0jkTSojECkBWEZwXB3/zDuWESioEdDIiJFTncEIiJFTncEIiJFTolARKTIKRGIiBQ5JQIRkSKnRCAiUuT+P2NHQwoYtxqkAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XPJ5Q5W8Rxbk",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "24d87846-1fa5-4bdd-b0c0-a72123f6d6d1"
      },
      "source": [
        "from tensorflow.keras import regularizers\n",
        "\n",
        "regularizers.l1(0.001)  #L1 常規化\n",
        "\n",
        "regularizers.l1_l2(l1=0.001, l2=0.001)  #同時使用 L1 及 L2 常規化"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<tensorflow.python.keras.regularizers.L1L2 at 0x7f9d608081d0>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6XSkpdsOvf5d"
      },
      "source": [
        "## 丟棄法(Dropout)\n",
        "想像一個Numpy矩陣包含了某一層的輸出，名稱為layer_output,shape是(batch_size,features)  \n",
        "在訓練時，我們隨機將矩陣中的一小部分值歸零。\n",
        "```python\n",
        "layer_output *= np.random.randint(0, high=2, size=layer_output.shape)\n",
        "```  \n",
        "\n",
        "在測試時，依照丟棄率縮減(scale dowm)輸出  \n",
        "```python\n",
        "layer_output *= 0.5\n",
        "```\n",
        "\n",
        "實務上，可以在訓練時執行dropout馬上將輸出值同比率放大，如此一來在測試時就不需對layer_output做任何變動了\n",
        "```python\n",
        "layer_output *= np.random.randint(0, high=2, size=layer_output.shape)\n",
        "layer_output /= 0.5\n",
        "```  \n",
        "  \n",
        "在keras中，可以透過add()一個Dropout層並指定丟棄率  \n",
        "```python\n",
        "model.add(layers.Dropout(0.5))\n",
        "```\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "miSyZBx_kmRF"
      },
      "source": [
        "### 程式 4.8 將 Dropout 層添加到 IMDB 神經網路"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6iErARnHTEY0"
      },
      "source": [
        "dpt_model = models.Sequential()\n",
        "dpt_model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))\n",
        "dpt_model.add(layers.Dropout(0.5))\n",
        "dpt_model.add(layers.Dense(16, activation='relu'))\n",
        "dpt_model.add(layers.Dropout(0.5))\n",
        "dpt_model.add(layers.Dense(1, activation='sigmoid'))\n",
        "\n",
        "dpt_model.compile(optimizer='rmsprop',\n",
        "                  loss='binary_crossentropy',\n",
        "                  metrics=['acc'])"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1snHDYIj9_Yt",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4c367b12-0189-4dea-b287-84775702f07b"
      },
      "source": [
        "dpt_model_hist = dpt_model.fit(x_train, y_train,\n",
        "                               epochs=20,\n",
        "                               batch_size=512,\n",
        "                               validation_data=(x_test, y_test))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/20\n",
            "49/49 [==============================] - 7s 131ms/step - loss: 0.6254 - acc: 0.6407 - val_loss: 0.4134 - val_acc: 0.8598\n",
            "Epoch 2/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.4332 - acc: 0.8189 - val_loss: 0.3204 - val_acc: 0.8777\n",
            "Epoch 3/20\n",
            "49/49 [==============================] - 2s 39ms/step - loss: 0.3337 - acc: 0.8776 - val_loss: 0.2888 - val_acc: 0.8846\n",
            "Epoch 4/20\n",
            "49/49 [==============================] - 2s 39ms/step - loss: 0.2799 - acc: 0.9026 - val_loss: 0.2774 - val_acc: 0.8871\n",
            "Epoch 5/20\n",
            "49/49 [==============================] - 2s 39ms/step - loss: 0.2461 - acc: 0.9148 - val_loss: 0.2828 - val_acc: 0.8885\n",
            "Epoch 6/20\n",
            "49/49 [==============================] - 2s 39ms/step - loss: 0.2188 - acc: 0.9310 - val_loss: 0.2919 - val_acc: 0.8873\n",
            "Epoch 7/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.1886 - acc: 0.9397 - val_loss: 0.3087 - val_acc: 0.8858\n",
            "Epoch 8/20\n",
            "49/49 [==============================] - 2s 39ms/step - loss: 0.1804 - acc: 0.9442 - val_loss: 0.3544 - val_acc: 0.8789\n",
            "Epoch 9/20\n",
            "49/49 [==============================] - 2s 39ms/step - loss: 0.1588 - acc: 0.9498 - val_loss: 0.3649 - val_acc: 0.8793\n",
            "Epoch 10/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.1469 - acc: 0.9533 - val_loss: 0.4229 - val_acc: 0.8746\n",
            "Epoch 11/20\n",
            "49/49 [==============================] - 2s 39ms/step - loss: 0.1388 - acc: 0.9547 - val_loss: 0.4112 - val_acc: 0.8806\n",
            "Epoch 12/20\n",
            "49/49 [==============================] - 2s 39ms/step - loss: 0.1321 - acc: 0.9581 - val_loss: 0.4192 - val_acc: 0.8778\n",
            "Epoch 13/20\n",
            "49/49 [==============================] - 2s 39ms/step - loss: 0.1267 - acc: 0.9597 - val_loss: 0.4692 - val_acc: 0.8781\n",
            "Epoch 14/20\n",
            "49/49 [==============================] - 2s 39ms/step - loss: 0.1233 - acc: 0.9617 - val_loss: 0.4780 - val_acc: 0.8773\n",
            "Epoch 15/20\n",
            "49/49 [==============================] - 2s 39ms/step - loss: 0.1144 - acc: 0.9645 - val_loss: 0.5022 - val_acc: 0.8778\n",
            "Epoch 16/20\n",
            "49/49 [==============================] - 2s 39ms/step - loss: 0.1101 - acc: 0.9648 - val_loss: 0.5276 - val_acc: 0.8770\n",
            "Epoch 17/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.1081 - acc: 0.9642 - val_loss: 0.5639 - val_acc: 0.8756\n",
            "Epoch 18/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.1021 - acc: 0.9679 - val_loss: 0.5912 - val_acc: 0.8742\n",
            "Epoch 19/20\n",
            "49/49 [==============================] - 2s 39ms/step - loss: 0.0990 - acc: 0.9667 - val_loss: 0.6021 - val_acc: 0.8733\n",
            "Epoch 20/20\n",
            "49/49 [==============================] - 2s 38ms/step - loss: 0.0929 - acc: 0.9719 - val_loss: 0.6088 - val_acc: 0.8688\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xpKe_Bk6-BLV",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 279
        },
        "outputId": "23d449ea-845a-4299-8b1e-bae438195a24"
      },
      "source": [
        "dpt_model_val_loss = dpt_model_hist.history['val_loss']\n",
        "\n",
        "plt.plot(epochs, original_val_loss, 'b+', label='Original model')\n",
        "plt.plot(epochs, dpt_model_val_loss, 'bo', label='Dropout-regularized model')\n",
        "plt.xlabel('Epochs')\n",
        "plt.ylabel('Validation loss')\n",
        "plt.legend()\n",
        "\n",
        "plt.show()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de5gU1Z3/8feXAQUUr6BBEQYNKsMdB9C4ouANjQHBxMjOGpFEHlS8JkazaBz1Zx7jJppoNAkoQTd4xagka1aNCKwGN1wcUSBcZAcdJTqQoOJIHOD7+6Nrxmbo20x39fXzep5+uut0ddV3iqa/VefUOcfcHRERKV3tch2AiIjklhKBiEiJUyIQESlxSgQiIiVOiUBEpMS1z3UArdW1a1cvLy/PdRgiIgVl2bJlm929W6z3Ci4RlJeXs3Tp0lyHISJSUMxsY7z3VDUkIlLilAhEREqcEoGISIkruDaCWBobG6mrq2P79u25DkUkpo4dO9KjRw86dOiQ61BE9hBqIjCzMcDPgTLgAXe/o8X7PYGHgAOCdW5w9+dau5+6ujq6dOlCeXk5ZpaByEUyx93ZsmULdXV19O7dO9fhiOwhtKohMysD7gPOAiqAiWZW0WK1G4En3H0IcAFwf1v2tX37dg4++GAlAclLZsbBBx+sK1ZJW3V1ONsNs41gOLDe3Te4++fAY8C4Fus4sF/wen/g/bbuTElA8pm+n5IJt9wSznbDTASHA+9GLdcFZdGqgX8zszrgOeCKWBsysylmttTMltbX14cRq4hIycr1XUMTgdnu3gM4G/hPM9sjJnef4e6V7l7ZrVvMjnE5V1dXx7hx4+jTpw9HHXUUV111FZ9//nnMdd9//32+/vWvJ93m2WefzdatW9sUT3V1NT/5yU/a9NlUzZ49m2nTpqW9jojEV10NZpEHfPE6k9VEYSaC94AjopZ7BGXRvg08AeDui4GOQNcQY9pNpg6kuzNhwgTOPfdc1q1bx9q1a9m2bRvTp0/fY90dO3Zw2GGHMXfu3KTbfe655zjggAMyE6SIFKTqanCPPOCL14WSCJYAfcyst5ntRaQxeF6Ldd4BTgUws75EEkHW6n4yVd82f/58OnbsyMUXXwxAWVkZd999N7NmzaKhoYHZs2czduxYRo8ezamnnkptbS39+/cHoKGhgfPPP5+KigrGjx/PiBEjmofQKC8vZ/PmzdTW1tK3b18uueQS+vXrxxlnnMFnn30GwMyZMxk2bBiDBg3ivPPOo6GhIWGskyZN4tJLL+X444/nyCOPZMGCBUyePJm+ffsyadKk5vUeffRRBgwYQP/+/bn++uuby3/zm99w9NFHM3z4cF599dXm8vr6es477zyGDRvGsGHDdntPRPJbaInA3XcA04DngdVE7g5aaWa3mtnYYLXvApeY2RvAo8AkL8C5M1euXMlxxx23W9l+++1Hz549Wb9+PQDLly9n7ty5LFy4cLf17r//fg488EBWrVrFbbfdxrJly2LuY926dVx++eWsXLmSAw44gKeeegqACRMmsGTJEt544w369u3Lgw8+mDTef/zjHyxevJi7776bsWPHcs0117By5UrefPNNampqeP/997n++uuZP38+NTU1LFmyhGeeeYZNmzZx88038+qrr/LKK6+watWq5m1eddVVXHPNNSxZsoSnnnqK73znO606hiKS3M03h7PdUPsRBH0CnmtR9sOo16uAE8OMoaXq6t2vBJrq3W6+ObxbswBOP/10DjrooD3KX3nlFa666ioA+vfvz8CBA2N+vnfv3gwePBiA4447jtraWgDeeustbrzxRrZu3cq2bds488wzk8byta99DTNjwIABHHrooQwYMACAfv36UVtby8aNGznllFNoao+pqqpi0aJFALuVf/Ob32Tt2rUA/OlPf9otMXz88cds27YtaSwikrqwfqOKomdxa1RXf3Ewzb6od0tHRUXFHnX+H3/8Me+88w5f/vKXWb58Ofvss09a+9h7772bX5eVlTVXDU2aNIlnnnmGQYMGMXv2bBYsWJDyttq1a7fbdtu1a8eOHTva1Pt1165dvPbaa3Ts2LHVnxWR3Mr1XUNF4dRTT6WhoYGHH34YgJ07d/Ld736XSZMm0blz54SfPfHEE3niiScAWLVqFW+++War9v3JJ5/QvXt3GhsbmTNnTtv+gBaGDx/OwoUL2bx5Mzt37uTRRx/l5JNPZsSIESxcuJAtW7bQ2NjIk08+2fyZM844g3vvvbd5uaamJiOxiEj4SjoRZKq+zcx4+umnefLJJ+nTpw9HH300HTt25Ec/+lHSz1522WXU19dTUVHBjTfeSL9+/dh///1T3vdtt93GiBEjOPHEEzn22GPT+TOade/enTvuuINRo0YxaNAgjjvuOMaNG0f37t2prq7mhBNO4MQTT6Rv377Nn7nnnntYunQpAwcOpKKigl/96lcZiUVEwmeF1jZbWVnpLSemWb169W4/SoVk586dNDY20rFjR95++21OO+001qxZw1577ZXr0CTDCvl7KoXPzJa5e2Ws90qujSDfNDQ0MGrUKBobG3F37r//fiUBEckqJYIc69Kli6beFJGcKuk2AhERUSIQESl5SgQiIlkSZqfVdCgRiIhkSVjzCaRLiSBDysrKGDx4MP369WPQoEH89Kc/ZdeuXTmL52c/+1nSAejC0Jbhr+fNm8cdd9yRfMUkTjnllNAb3idNmpR05NhU1hHJJyWZCObMgfJyaNcu8pyJDrmdOnWipqaGlStX8uKLL/LHP/6RW2Kk/x07dqS/sxSkmgiyFU+i/Y8dO5Ybbrghp3GIhCUb8wmkq+QSwZw5MGUKbNwYGWdo48bIcoZGZwDgkEMOYcaMGfziF7/A3fcYhvrvf/875557LgMHDuT4449nxYoVQORs+sILL+SEE06gT58+zJw5E4jMd3DdddfRv39/BgwYwOOPPw7AggULOOecc5r3O23aNGbPns0999zD+++/z6hRoxg1atQe8bWM59NPP2Xy5MkMHz6cIUOG8OyzzwKJh8jed999m7c3d+7c3YawbhJviOxJkyYxdepURowYwfe///3dJq8ZPHhw86NTp04sXLgwbnyfffYZF1xwAX379mX8+PHN4y+1VF5ezg9+8AMGDx5MZWUly5cv58wzz+Soo45q7gEd7xi7O9OmTeOYY47htNNO48MPP2ze7rJlyzj55JM57rjjOPPMM9m0aVPC74WUpmzMJ5CukutHMH06tDxRbmiIlFdVZW4/Rx55JDt37mz+4Vi+fDkrVqzgoIMO4oorrmDIkCE888wzzJ8/n29961vNY/OsWLGC1157jU8//ZQhQ4bw1a9+lcWLF1NTU8Mbb7zB5s2bGTZsGCNHjoy77yuvvJK77rqLl19+ma5dY8/zEx3Pv//7vzN69GhmzZrF1q1bGT58OKeddhq//OUvm4fIfuutt5pHP03VhAkTuOSSSwC48cYbefDBB7niishspHV1dfz5z3+mrKyM2bNnN3+m6Tj8/ve/58477+QrX/kKN998c8z4fv3rX9O5c2dWr17NihUrGDp0aNxYevbsSU1NDddccw2TJk3i1VdfZfv27fTv35+pU6fyu9/9LuYxXrx4MWvWrGHVqlV88MEHVFRUMHnyZBobG7niiit49tln6datG48//jjTp09n1qxZrTpGIvmg5BLBO++0rjxTooehfuWVV5rnExg9ejRbtmzh448/BmDcuHF06tSJTp06MWrUKP7yl7/wyiuvMHHiRMrKyjj00EM5+eSTWbJkCfvtt19G4nnhhReYN29ec93+9u3beeedd1IeIjueRENkf+Mb36CsrCzm59atW8d1113Hyy+/TIcOHeLGt2jRIq688koABg4cmDC+sWMjU2AMGDCAbdu20aVLF7p06cLee+/N1q1b4x7jRYsWNZcfdthhjB49GoA1a9bw1ltvcfrppwORoUK6d+/equMjpSes+QTSVXKJoGfPSHVQrPJM2rBhA2VlZRxyyCEAKQ9DbU0ViXGWo7Vv3363Bunt27fHXO/pp59ubq944IEH9ojH3Xnqqac45phjUoqxZVzx9ptoiOx4x2Pbtm2cf/75zJw5s/mHtS3xtZRs6O3Wcnf69evH4sWL2xyTlJ58qg6KVnJtBLffDi1Hhu7cOVKeKfX19UydOpVp06bF/CE/6aSTmoeMXrBgAV27dm0+u3/22WfZvn07W7ZsYcGCBQwbNoyTTjqJxx9/nJ07d1JfX8+iRYsYPnw4vXr1YtWqVfzzn/9k69atvPTSS8376NKlC5988gkA48ePp6amhpqaGior9xxz6swzz+Tee++laQDC119/HUg8RPahhx7K6tWr2bVrF08//XTM49CWIbInT57MxRdfzEknnZQ0vpEjR/LII48AkauPpraWtoh3jEeOHNlcvmnTJl5++WUAjjnmGOrr65sTQWNjIytXrmzz/kVyqeSuCJraAaZPj1QH9ewZSQLptg989tlnDB48mMbGRtq3b8+FF17ItddeG3Pd6upqJk+ezMCBA+ncuTMPPfRQ83sDBw5k1KhRbN68mZtuuonDDjuM8ePHs3jxYgYNGoSZceedd/KlL30JgPPPP5/+/fvTu3dvhgwZ0rydKVOmMGbMGA477LDmH694brrpJq6++moGDhzIrl276N27N3/4wx+47LLLuOiii6ioqODYY4/dbYjsO+64g3POOYdu3bpRWVkZczaypiGyu3XrxogRI5oTUzwbN25k7ty5rF27trmu/YEHHogb36WXXsrFF19M37596du37x7ThbZGvGM8fvx45s+fT0VFBT179uSEE04AYK+99mLu3LlceeWVfPTRR+zYsYOrr76afv36tTkGkVzRMNR5pLq6mn333Zfvfe97uQ4F0BDZmVYs31MpTBqGWtpEQ2SLlAYlgjxSnWctSRoiW6Q0FE1jcaFVcUlp0fdT8llRJIKOHTuyZcsW/WeTvOTubNmyhY4dO+Y6FJGYiqJqqEePHtTV1VFfX5/rUERi6tixIz169Mh1GCIxFUUi6NChA7179851GCIiBakoqoZERLIhz+7nyBglAhGRFOXrxDLpUiIQESlxSgQiIgkUwsQy6Qp1iAkzGwP8HCgDHnD3O1q8fzfQNHNKZ+AQdz8g0TZjDTEhIpINZl9MMFNocjLEhJmVAfcBpwN1wBIzm+fuq5rWcfdrota/Ahiyx4ZERCRUYVYNDQfWu/sGd/8ceAwYl2D9icCjIcYjIpKWfJ1YJl1hJoLDgXejluuCsj2YWS+gNzA/zvtTzGypmS1VpzERyZViaheIli+NxRcAc919Z6w33X2Gu1e6e2W3bt2yHJqISHELMxG8BxwRtdwjKIvlAlQtJCKSE2EmgiVAHzPrbWZ7Efmxn9dyJTM7FjgQ0OSvIiI5EFoicPcdwDTgeWA18IS7rzSzW81sbNSqFwCPuYYOFRHJiVAHnXP354DnWpT9sMVydZgxiIg0qa4u3gbfdORLY7GISOiKdaygdCkRiIiUOCUCESlqpTBWULpCHWsoDBprSETaqpDHCkpXorGGdEUgIlLilAhEpGQU61hB6VIiEJGSoXaB2JQIRERKnBKBiEiJUyIQESlxSgQiIiVOiUBEpMQpEYiIlDglAhGREqdEICJS4pQIRERKnBKBiBQM9QwOhxKBiBQMTSwTDiUCEZESp0QgInlNE8uETxPTiEjBKOWJZdKliWlERCSupInAzE40s32C1/9mZneZWa/wQxMR2Z0mlglHKlcEvwQazGwQ8F3gbeDhUKMSEYlB7QLhSCUR7PBIQ8I44Bfufh/QJdywREQkW9qnsM4nZvYD4N+AkWbWDugQblgiIpItqVwRfBP4J/Btd/8b0AP4j1CjEhGRZnPmQHk5tGsXeZ4zJ7PbT+mKAPi5u+80s6OBY4FHMxuGiIjEMmcOTJkCDQ2R5Y0bI8sAVVWZ2UcqVwSLgL3N7HDgBeBCYHZmdi8iUvzSOaOfPv2LJNCkoSFSnimpJAJz9wZgAnC/u38D6J/Kxs1sjJmtMbP1ZnZDnHXON7NVZrbSzB5JPXQRkfzXdEa/cWOkM1zTGX2qyeCdd1pX3hYpJQIzOwGoAv4r1c+ZWRlwH3AWUAFMNLOKFuv0AX4AnOju/YCrWxG7iEjeS/eMvmfP1pW3RSqJ4GoiP9ZPu/tKMzsSeDmFzw0H1rv7Bnf/HHiMyC2o0S4B7nP3fwC4+4ephy4ihaYU+wGke0Z/++3QufPuZZ07R8ozJWkicPeF7j4WuM/M9g1+2K9MYduHA+9GLdcFZdGOBo42s1fN7DUzGxNrQ2Y2xcyWmtnS+vr6FHYtIvmoUIeRTqeOP90z+qoqmDEDevWKjLXUq1dkOVMNxZBaFc8AM3sdWAmsMrNlZtYvQ/tvD/QBTgEmAjPN7ICWK7n7DHevdPfKbt26ZWjXIiLJpVvHn4kz+qoqqK2FXbsiz5lMApBa1dCvgWvdvZe79yQyzMTMFD73HnBE1HKPoCxaHTDP3Rvd/f+AtUQSg4gUiUIfRjrdOv5snNGnK+kw1Gb2hrsPSlYW43Ptifywn0okASwB/tXdV0atMwaY6O4XmVlX4HVgsLtvibddDUMtUrgKcRjpdu1ix2wWOUMvFOkOQ73BzG4ys/LgcSOwIdmH3H0HMA14HlgNPBE0Nt9qZmOD1Z4HtpjZKiIN0NclSgIiItmWjbt2ci2VnsWTgVuA3wXL/xOUJeXuzwHPtSj7YdRrB64NHiJS5ApxGOnbb9+9Zy9k/q6dXEuaCIJbO1O5S0hEJKFCaReI1lSXP3165JbPnj0jSSCf6vjTFTcRmNnvgbi1ecEtpSIiRa+qqrh++FtKdEXwk6xFISIiORM3Ebj7wmwGIiIiuaHJ60VESpwSgYikrBAbeyH8iV0KnRKBiKSsEMcKSneIiFKQSs/io4HrgF5EtSm4++hwQ4tNPYtFcqcQewaXl0d+/Fvq1Ssybk+pSLdn8ZPAcuBGIgmh6SEiJaDQxwrKxsQuhS6VK4Jl7n5cluJJSlcEIrmjK4LCle4Vwe/N7DIz625mBzU9MhyjiEgosjGxS6FLZayhi4Ln6OogB47MfDgiks8KcaygUhgiIl1Jq4byjaqGRERaL1HVUNIrAjPrAFwKjAyKFgC/dvfGjEUoIiI5k0rV0C+BDsD9wfKFQdl3wgpKRESyJ5VEMKzFbGTzzeyNsAISEZHsSuWuoZ1mdlTTgpkdCewMLyQREcmmVK4IrgNeNrMNgBHpYXxxqFGJiEjWpDJD2Utm1gc4Jiha4+7/DDcsERHJlrhVQ2Y2OnieAHwV+HLw+GpQJiKSFRo9NFyJrghOBuYDX4vxnvPFZPYiUiCqqwtnjKAmTaOHNk0e3zR6KKhTWKakMtZQb3f/v2Rl2aIOZSJtp7GCSle6Yw09FaNsbnohiYikRqOHhi9RG8GxZnYesL+ZTYh6TAI6Zi1CEWnWlmqdQh9GumfP1pVL68WtGjKzccC5wFhgXtRbnwCPufufww9vT6oaklKWbtVOIVYNtWwjgMjooTNmqI2gNdo01pC7Pws8a2YnuPvi0KITEUlAo4eGL5U2gtfN7HIzu9/MZjU9Qo9MRIDMVu0U4jDSEPnRr62FXbsiz0oCmZXKXUNPAn8F/hW4FagCVrv7VeGHtydVDUkpK8SqHckP6d419GV3vwn41N0fItK5bEQmAxSR4qYOYfktlUTQNO/AVjPrD+wPHJLKxs1sjJmtMbP1ZnZDjPcnmVm9mdUEDw1tLZJAIVbtNDX2btwYuZpp6hCmZJA/Uqka+g6RvgQDgd8A+wI/dPdfJflcGbAWOB2oA5YAE919VdQ6k4BKd5+WasCqGhIpLOoQlh/SmqHM3R8IXi6kdfMUDwfWu/uGIIjHgHHAqoSfEpGiog5h+S9uIjCzaxN90N3vSrLtw4F3o5briN22cJ6ZjSRy9XCNu7/bcgUzmwJMAeipXiQiBaVnz9hXBPqvnD8StRF0CR6VROYsPjx4TAWGZmj/vwfK3X0g8CLwUKyV3H2Gu1e6e2W3bt0ytGsRyYbbb490AIvWuXOkXPJDog5ltwCY2SJgqLt/EixXA/+VwrbfA46IWu4RlEXvY0vU4gPAnSlFLSIFQx3C8l8qM5QdCnwetfx5UJbMEqCPmfUmkgAuINIXoZmZdXf3TcHiWGB1CtsVkQJTVaUf/nyWSiJ4GPiLmT0dLJ8LzE72IXffYWbTgOeBMmCWu680s1uBpe4+D7jSzMYCO4C/A5Na/yeIiEg6kt4+CmBmQ4GTgsVF7v56qFEloNtHRURar009i81sv+D5IKAW+M/gsTEoE5ESoZ7BxS1R1dAjwDnAMiJTUzaxYLk1fQpEpEBpqsjil1LVUD5R1ZBIdqlncHFoa9XQ0ESP8MIVkUxLp2pHPYOLX6KqoZ8meM+B0RmORURCkG7VjnoGF7+4VwTuPirBQ0lApJVyNUfw9Om7T/MIkeXp01P7vHoGF79Ubx/tD1QQNWm9uz8cYlxxqY1AClWuJpVp1y72fs0iM36lYs4c9QwudGmNPmpmNwOnEEkEzwFnAa8Q6WgmInkuE1U76hlc3FKZmObrwKnA39z9YmAQkclpRCSJTM433Faq2pFkUkkEn7n7LmBH0MnsQ3YfTE5E4qiujlTLNFXNNL3OZiKoqoIZMyK3e5pFnmfM0Bm+fCGVsYaWmtkBwEwincu2AYtDjUpEMkpVO5JIoolp7gMecffLgqJfmdl/A/u5+4qsRCdSRApxvmEpDYmuCNYCPzGz7sATwKO5HGxOpNDl6vZRkWQS9SP4ubufAJwMbAFmmdlfzexmMzs6axGKiEiokjYWu/tGd/+xuw8BJhKZj0ATyIiIFImkicDM2pvZ18xsDvBHYA0wIfTIREQkKxI1Fp9O5ArgbOAvwGPAFHf/NEuxiYhIFiS6IvgB8Gegr7uPdfdHlASklKmxV4qV5iMQSVGuxgoSyYQ2zUcgIiKlQYlAJIF8GCtIJGxKBCIJVFfDb38bGZ8HIs+//a0SgRQXJQKRBJpm92oaxrlpdq/WTPWYqTjaOtWkSDJKBCIJpDu7VyZEJyP33CUjKV4lkQh0NiVtlQ8Tt+dDMpLiVvSJQGdTko54s3i1ZnavdE9E8iEZSXEr+kSgsylp0pYG3nRn98rEiUgmkpFIIkWfCHQ2JU1uuaX1n0l3dq9MnIhoqkkJW9EnAp1NSbqqqqC2Fnbtijy3ZqavTJyIaKpJCVuoicDMxpjZGjNbb2Y3JFjvPDNzM4vZ/TkdOpsqbbnuEJapE5F0kpFIMqElAjMrA+4DzgIqgIlmVhFjvS7AVcD/hhGHzqZKW64nj9eJiBSCMK8IhgPr3X2Du39OZBjrcTHWuw34MbA9rEB0NiW5ohMRKQRhJoLDgXejluuCsmZmNhQ4wt3/K9GGzGyKmS01s6X19fWZj1RKQq4mj9eJiOS7nDUWm1k74C7gu8nWdfcZ7l7p7pXdunULPzjJS+lW52h8IJHYwkwE7wFHRC33CMqadAH6AwvMrBY4HpgXRoOxFIe23P4pIsmFmQiWAH3MrLeZ7QVcAMxretPdP3L3ru5e7u7lwGvAWHfXrDMiIlkUWiJw9x3ANOB5YDXwhLuvNLNbzWxsWPuV4pLr2z9FSoGmqpSCoakiRdpOU1WKiEhcSgRSMHJ1+6dIsVMikIKhdgGRcCgRiIiUOCUCEZESp0QgRU9TlYokpkQgWZOLOn5NVSqSnPoRSNbkoh9AeXnkx7+lXr0iA8CJlAr1I5CSpalKRZJTIpBQZWKIiHTq+DVVqUhySgQSqnRnCEu3jl8zhIkkp0QgeW36dGho2L2soSFSngrNECaSXPtcByCloy1DRGSijr+qSj/8IonoikCypi23j6qOXyR8SgSS11THLxK+kkoEGrQsPbk4fqrjFwlfSXUo08Qm6dHxEylc6lAmIiJxFX0i0Jy36dHxEyl+qhqSlOn4iRQuVQ2JiEhcJZUINOdtenT8RIpTSSWCttZrF8vEJunW66tdQKQ4lVQiaItimtjklltyHYGI5CMlgiTSHfRMRCTfKREkUegTm+j2TxFJRokgiUIf9Czd+QAyoVjaWESKlRJBEpkc9KwUz8KLqY1FpFgpESSRyUHPct1Ym4vbP9XGIpL/Qu1ZbGZjgJ8DZcAD7n5Hi/enApcDO4FtwBR3X5Vom+n0LM61UuyZ265d7L/ZDHbtyn48IqUqJz2LzawMuA84C6gAJppZRYvVHnH3Ae4+GLgTuCuseHKl1BtrC72NRaQUhFk1NBxY7+4b3P1z4DFgXPQK7v5x1OI+QNGdL2eysbYQk4cmlhHJf2EmgsOBd6OW64Ky3ZjZ5Wb2NpErgitjbcjMppjZUjNbWl9fH0qwhSBXbQzp3PWjiWVE8l/OG4vd/T53Pwq4Hrgxzjoz3L3S3Su7deuW3QCjpHtGXohj9WTirp+qKqitjbQJ1NYqCYjkmzATwXvAEVHLPYKyeB4Dzg0xnrS19Yy86Yz61ltbf0ad6zYG3fUjUvxCu2vIzNoDa4FTiSSAJcC/uvvKqHX6uPu64PXXgJvjtWo3yeVdQ22566fpjDr6x7Rz57ZVj+TiriPd9SNSHHJy15C77wCmAc8Dq4En3H2lmd1qZmOD1aaZ2UozqwGuBS4KK562SveMPB/OqNOp49ddPyLFr6RmKEtXW87IM3lGXV3d+iqhdK9IMnlFIyK5oxnKcigTZ9TptDGke0Wiu35Eip8SQSu05a6fdO+jT/eunUyMnqq7fkSKmxJBK7TlTp10z6jTPaNXHb+IJKNEkAXpnFGne0avnr0ikowSQZ5L94xedfwikowSQZ7LxBm96vhFJBElgjynM3oRCVv7XAcgyVVV6YdfRMKjKwIRkRKnRCAiUuKUCERESpwSgYhIiVMiEBEpcQU3+qiZ1QMbcx1HHF2BzbkOIgHFl558jw/yP0bFl5504uvl7jGneCy4RJDPzGxpsol1cknxpSff44P8j1HxpSes+FQ1JCJS4pQIRERKnBJBZs3IdQBJKL705Ht8kP8xKr70hBKf2ghEREqcrghEREqcEoGISIlTImglMzvCzF42s1VmttLMroqxzilm9pGZ1QSPH2Y5xlozezPY942MjOQAAAXvSURBVNIY75uZ3WNm681shZkNzWJsx0Qdlxoz+9jMrm6xTtaPn5nNMrMPzeytqLKDzOxFM1sXPB8Y57MXBeusM7OLshTbf5jZX4N/v6fN7IA4n034XQg5xmozey/q3/HsOJ8dY2Zrgu/jDVmM7/Go2GrNrCbOZ0M9hvF+U7L6/XN3PVrxALoDQ4PXXYC1QEWLdU4B/pDDGGuBrgnePxv4I2DA8cD/5ijOMuBvRDq65PT4ASOBocBbUWV3AjcEr28AfhzjcwcBG4LnA4PXB2YhtjOA9sHrH8eKLZXvQsgxVgPfS+E78DZwJLAX8EbL/09hxdfi/Z8CP8zFMYz3m5LN75+uCFrJ3Te5+/Lg9SfAauDw3EbVauOAhz3iNeAAM+uegzhOBd5295z3FHf3RcDfWxSPAx4KXj8EnBvjo2cCL7r73939H8CLwJiwY3P3F9x9R7D4GtAjk/tsrTjHLxXDgfXuvsHdPwceI3LcMypRfGZmwPnAo5nebyoS/KZk7funRJAGMysHhgD/G+PtE8zsDTP7o5n1y2pg4MALZrbMzKbEeP9w4N2o5Tpyk8wuIP5/vlwevyaHuvum4PXfgENjrJMPx3IykSu8WJJ9F8I2Lai+mhWnaiMfjt9JwAfuvi7O+1k7hi1+U7L2/VMiaCMz2xd4Crja3T9u8fZyItUdg4B7gWeyHN6/uPtQ4CzgcjMbmeX9J2VmewFjgSdjvJ3r47cHj1yH59291mY2HdgBzImzSi6/C78EjgIGA5uIVL/ko4kkvhrIyjFM9JsS9vdPiaANzKwDkX+wOe7+u5bvu/vH7r4teP0c0MHMumYrPnd/L3j+EHiayOV3tPeAI6KWewRl2XQWsNzdP2j5Rq6PX5QPmqrMgucPY6yTs2NpZpOAc4Cq4IdiDyl8F0Lj7h+4+0533wXMjLPvnH4Xzaw9MAF4PN462TiGcX5Tsvb9UyJopaA+8UFgtbvfFWedLwXrYWbDiRznLVmKbx8z69L0mkij4lstVpsHfCu4e+h44KOoS9BsiXsWlsvj18I8oOkujIuAZ2Os8zxwhpkdGFR9nBGUhcrMxgDfB8a6e0OcdVL5LoQZY3S70/g4+14C9DGz3sFV4gVEjnu2nAb81d3rYr2ZjWOY4Dcle9+/sFrCi/UB/AuRS7QVQE3wOBuYCkwN1pkGrCRyB8RrwFeyGN+RwX7fCGKYHpRHx2fAfUTu1ngTqMzyMdyHyA/7/lFlOT1+RJLSJqCRSD3rt4GDgZeAdcCfgIOCdSuBB6I+OxlYHzwuzlJs64nUDTd9B38VrHsY8Fyi70IWj99/Bt+vFUR+1Lq3jDFYPpvInTJvhxVjrPiC8tlN37uodbN6DBP8pmTt+6chJkRESpyqhkRESpwSgYhIiVMiEBEpcUoEIiIlTolARKTEKRGIBMxsp+0+MmrGRsI0s/LokS9F8kn7XAcgkkc+c/fBuQ5CJNt0RSCSRDAe/Z3BmPR/MbMvB+XlZjY/GFTtJTPrGZQfapE5At4IHl8JNlVmZjODMedfMLNOwfpXBmPRrzCzx3L0Z0oJUyIQ+UKnFlVD34x67yN3HwD8AvhZUHYv8JC7DyQy6Ns9Qfk9wEKPDJo3lEiPVIA+wH3u3g/YCpwXlN8ADAm2MzWsP04kHvUsFgmY2TZ33zdGeS0w2t03BIOD/c3dDzazzUSGTWgMyje5e1czqwd6uPs/o7ZRTmTc+D7B8vVAB3f/f2b238A2IqOsPuPBgHsi2aIrApHUeJzXrfHPqNc7+aKN7qtExn4aCiwJRsQUyRolApHUfDPqeXHw+s9ERssEqAL+J3j9EnApgJmVmdn+8TZqZu2AI9z9ZeB6YH9gj6sSkTDpzEPkC51s9wnM/9vdm24hPdDMVhA5q58YlF0B/MbMrgPqgYuD8quAGWb2bSJn/pcSGfkyljLgt0GyMOAed9+asb9IJAVqIxBJImgjqHT3zbmORSQMqhoSESlxuiIQESlxuiIQESlxSgQiIiVOiUBEpMQpEYiIlDglAhGREvf/Acfshie/ljDiAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "idHAfkxDvn6m"
      },
      "source": [
        "# 機器學習的通用流程\n",
        "1. 定義問題並建立資料集\n",
        "2. 選擇一種評量成功的準則\n",
        "3. 決定驗證程序\n",
        "4. 準備資料\n",
        "5. 開發出優於基準(baseline)的模型\n",
        "6. 開發一個過度適配的模型\n",
        "7. 常規化模型並調整參數"
      ]
    }
  ]
}