notebook/DL_image_hyperparameter_tuning/03_pretrained_weights.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 使用ImageNet预训练模型权重\n",
    "\n",
    "Keras框架提供了预置好的VGG16模型，并提供使用ImageNet进行大规模训练的权重。我们使用预置的VGG16模型，加载预训练的权重，保留VGG16的卷积网络结构，只在最后的全连接层进行调整，使用GlobalAveragePooling2D将7x7x512的卷积结果进行全局池化，减少训练参数，并加入softmax激活、output shape为2的全连接层进行二分类。\n",
    "\n",
    "首先，我们准备数据："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: http://mirrors.aliyun.com/pypi/simple/\n",
      "Requirement already up-to-date: keras_applications==1.0.6 in /opt/anaconda/envs/mask-rcnn/lib/python3.6/site-packages (1.0.6)\n",
      "Requirement already up-to-date: keras==2.2.4 in /opt/anaconda/envs/mask-rcnn/lib/python3.6/site-packages (2.2.4)\n",
      "Requirement already satisfied, skipping upgrade: h5py in /opt/anaconda/envs/mask-rcnn/lib/python3.6/site-packages (from keras_applications==1.0.6) (2.9.0)\n",
      "Requirement already satisfied, skipping upgrade: numpy>=1.9.1 in /opt/anaconda/envs/mask-rcnn/lib/python3.6/site-packages (from keras_applications==1.0.6) (1.15.4)\n",
      "Requirement already satisfied, skipping upgrade: six>=1.9.0 in /opt/anaconda/envs/mask-rcnn/lib/python3.6/site-packages (from keras==2.2.4) (1.12.0)\n",
      "Requirement already satisfied, skipping upgrade: keras-preprocessing>=1.0.5 in /opt/anaconda/envs/mask-rcnn/lib/python3.6/site-packages (from keras==2.2.4) (1.0.5)\n",
      "Requirement already satisfied, skipping upgrade: scipy>=0.14 in /opt/anaconda/envs/mask-rcnn/lib/python3.6/site-packages (from keras==2.2.4) (1.2.0)\n",
      "Requirement already satisfied, skipping upgrade: pyyaml in /opt/anaconda/envs/mask-rcnn/lib/python3.6/site-packages (from keras==2.2.4) (3.13)\n"
     ]
    }
   ],
   "source": [
    "!pip install --upgrade keras_applications==1.0.6 keras==2.2.4\n",
    "\n",
    "import os\n",
    "if os.path.exists('./data') == False:\n",
    "    from modelarts.session import Session\n",
    "    session = Session()\n",
    "    \n",
    "    if session.region_name == 'cn-north-1':\n",
    "        bucket_path=\"modelarts-labs/notebook/DL_image_recognition/image_recognition.tar.gz\"\n",
    "    elif session.region_name == 'cn-north-4':\n",
    "        bucket_path=\"modelarts-labs-bj4/notebook/DL_image_recognition/image_recognition.tar.gz\"\n",
    "    else:\n",
    "        print(\"请更换地区到北京一或北京四\")\n",
    "\n",
    "    session.download_data(\n",
    "    bucket_path=bucket_path,\n",
    "    path=\"./image_recognition.tar.gz\")\n",
    "\n",
    "    # 使用tar命令解压资源包\n",
    "    !tar xf ./image_recognition.tar.gz\n",
    "\n",
    "    # 清理压缩包\n",
    "    !rm -f ./image_recognition.tar.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 导入相关库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.applications.vgg16 import VGG16\n",
    "from keras.preprocessing import image\n",
    "import numpy as np\n",
    "\n",
    "from keras.preprocessing import image\n",
    "from keras.models import Model\n",
    "from keras.layers import Dense, GlobalAveragePooling2D\n",
    "from keras import backend as K\n",
    "from keras.models import load_model\n",
    "\n",
    "from keras.preprocessing.image import ImageDataGenerator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 准备数据集"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from PIL import Image\n",
    "def load_data():\n",
    "    dirname = \"./data\"\n",
    "    path = \"./data\"\n",
    "\n",
    "    num_train_samples = 25000\n",
    "\n",
    "    x_train = np.empty((num_train_samples, 224,224,3), dtype='uint8')\n",
    "    y_train = np.empty((num_train_samples,1), dtype='uint8')\n",
    "    index = 0\n",
    "    for file in os.listdir(\"./data\"):\n",
    "        image = Image.open(os.path.join(dirname,file)).resize((224,224))\n",
    "        image = np.array(image)\n",
    "        x_train[index,:,:,:] = image\n",
    "\n",
    "        if \"cat\" in file:\n",
    "            y_train[index,0] =1\n",
    "        elif \"dog\" in file:\n",
    "            y_train[index,0] =0\n",
    "\n",
    "        index += 1\n",
    "    return (x_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "(x_train, y_train) = load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(25000, 224, 224, 3)\n",
      "(25000, 1)\n"
     ]
    }
   ],
   "source": [
    "print(x_train.shape)\n",
    "print(y_train.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.utils import np_utils\n",
    "def process_data(x_train,y_train):\n",
    "    x_train = x_train.astype(np.float32)\n",
    "    x_train /= 255\n",
    "    n_classes = 2\n",
    "    y_train = np_utils.to_categorical(y_train, n_classes)\n",
    "    return x_train,y_train"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_model(base_model):\n",
    "    x = base_model.output\n",
    "    x = GlobalAveragePooling2D()(x)\n",
    "    predictions = Dense(2, activation='softmax')(x)\n",
    "    model = Model(inputs=base_model.input, outputs=predictions)\n",
    "    print(type(model))\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(25000, 224, 224, 3)\n",
      "(25000, 2)\n"
     ]
    }
   ],
   "source": [
    "x_train,y_train= process_data(x_train,y_train)\n",
    "print(x_train.shape)\n",
    "print(y_train.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 准备模型\n",
    "\n",
    "我们定义输入数据的维度，并构建VGG16模型，加载ImageNet预训练的权重（如果预训练权重不存在，则从网络下载，并保存到到$HOME目录的.keras/models/路径下）。include_top=False表示只取卷积网络结构中的参数，不包含全连接层和softmax分类层（ImageNet有1000个分类）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/ma-user/anaconda3/envs/TensorFlow-1.13.1/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n"
     ]
    }
   ],
   "source": [
    "base_model = VGG16(weights='imagenet', include_top=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们把所有卷基层的trainable设置为False，不进行训练。然后将池化层和全连接的二分类层添加到模型中，输入层不变"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'keras.engine.training.Model'>\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_1 (InputLayer)         (None, None, None, 3)     0         \n",
      "_________________________________________________________________\n",
      "block1_conv1 (Conv2D)        (None, None, None, 64)    1792      \n",
      "_________________________________________________________________\n",
      "block1_conv2 (Conv2D)        (None, None, None, 64)    36928     \n",
      "_________________________________________________________________\n",
      "block1_pool (MaxPooling2D)   (None, None, None, 64)    0         \n",
      "_________________________________________________________________\n",
      "block2_conv1 (Conv2D)        (None, None, None, 128)   73856     \n",
      "_________________________________________________________________\n",
      "block2_conv2 (Conv2D)        (None, None, None, 128)   147584    \n",
      "_________________________________________________________________\n",
      "block2_pool (MaxPooling2D)   (None, None, None, 128)   0         \n",
      "_________________________________________________________________\n",
      "block3_conv1 (Conv2D)        (None, None, None, 256)   295168    \n",
      "_________________________________________________________________\n",
      "block3_conv2 (Conv2D)        (None, None, None, 256)   590080    \n",
      "_________________________________________________________________\n",
      "block3_conv3 (Conv2D)        (None, None, None, 256)   590080    \n",
      "_________________________________________________________________\n",
      "block3_pool (MaxPooling2D)   (None, None, None, 256)   0         \n",
      "_________________________________________________________________\n",
      "block4_conv1 (Conv2D)        (None, None, None, 512)   1180160   \n",
      "_________________________________________________________________\n",
      "block4_conv2 (Conv2D)        (None, None, None, 512)   2359808   \n",
      "_________________________________________________________________\n",
      "block4_conv3 (Conv2D)        (None, None, None, 512)   2359808   \n",
      "_________________________________________________________________\n",
      "block4_pool (MaxPooling2D)   (None, None, None, 512)   0         \n",
      "_________________________________________________________________\n",
      "block5_conv1 (Conv2D)        (None, None, None, 512)   2359808   \n",
      "_________________________________________________________________\n",
      "block5_conv2 (Conv2D)        (None, None, None, 512)   2359808   \n",
      "_________________________________________________________________\n",
      "block5_conv3 (Conv2D)        (None, None, None, 512)   2359808   \n",
      "_________________________________________________________________\n",
      "block5_pool (MaxPooling2D)   (None, None, None, 512)   0         \n",
      "_________________________________________________________________\n",
      "global_average_pooling2d_1 ( (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 2)                 1026      \n",
      "=================================================================\n",
      "Total params: 14,715,714\n",
      "Trainable params: 1,026\n",
      "Non-trainable params: 14,714,688\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "for layer in base_model.layers:\n",
    "    layer.trainable = False\n",
    "\n",
    "model = build_model(base_model)\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到，训练的参数个数为1026，我们仅训练分类部分。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 设置模型的损失函数和优化器"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import keras \n",
    "opt = keras.optimizers.rmsprop(lr=0.0001, decay=1e-6)\n",
    "model.compile(loss='binary_crossentropy',\n",
    "              optimizer=opt,\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 设置callback并训练模型\n",
    "\n",
    "使用预训练的权重，可以较快提高模型精度。在高性能GPU环境下，每一轮需要几分钟左右，请实践者保持耐心，笔者的实践中训练到第11轮就达到了约90%的精度，耗时约8分钟。\n",
    "\n",
    "后续训练精度提升开始变慢，在41轮达到约92%的精度，耗时约30分钟。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.callbacks import ModelCheckpoint, EarlyStopping, ReduceLROnPlateau\n",
    "es = EarlyStopping(monitor='val_acc', baseline=0.9, patience=15, verbose=1, mode='auto')\n",
    "cp = ModelCheckpoint(filepath=\"./ckpt_vgg16_dog_and_cat.h5\", monitor=\"val_acc\", verbose=1, save_best_only=True, mode=\"auto\", period=1)\n",
    "lr = ReduceLROnPlateau(monitor=\"val_acc\", factor=0.1, patience=10, verbose=1, mode=\"auto\", min_lr=0)\n",
    "callbacks = [es,cp,lr]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /home/ma-user/anaconda3/envs/TensorFlow-1.13.1/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "Train on 18750 samples, validate on 6250 samples\n",
      "Epoch 1/100\n",
      "18750/18750 [==============================] - 48s 3ms/step - loss: 0.6284 - acc: 0.6794 - val_loss: 0.5503 - val_acc: 0.7995\n",
      "\n",
      "Epoch 00001: val_acc improved from -inf to 0.79952, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 2/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.4959 - acc: 0.8314 - val_loss: 0.4565 - val_acc: 0.8410\n",
      "\n",
      "Epoch 00002: val_acc improved from 0.79952 to 0.84096, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 3/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.4233 - acc: 0.8564 - val_loss: 0.4012 - val_acc: 0.8630\n",
      "\n",
      "Epoch 00003: val_acc improved from 0.84096 to 0.86304, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 4/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.3789 - acc: 0.8707 - val_loss: 0.3669 - val_acc: 0.8710\n",
      "\n",
      "Epoch 00004: val_acc improved from 0.86304 to 0.87104, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 5/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.3493 - acc: 0.8793 - val_loss: 0.3417 - val_acc: 0.8824\n",
      "\n",
      "Epoch 00005: val_acc improved from 0.87104 to 0.88240, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 6/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.3274 - acc: 0.8848 - val_loss: 0.3233 - val_acc: 0.8867\n",
      "\n",
      "Epoch 00006: val_acc improved from 0.88240 to 0.88672, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 7/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.3105 - acc: 0.8903 - val_loss: 0.3089 - val_acc: 0.8864\n",
      "\n",
      "Epoch 00007: val_acc did not improve from 0.88672\n",
      "Epoch 8/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2975 - acc: 0.8933 - val_loss: 0.2966 - val_acc: 0.8909\n",
      "\n",
      "Epoch 00008: val_acc improved from 0.88672 to 0.89088, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 9/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2867 - acc: 0.8960 - val_loss: 0.2869 - val_acc: 0.8979\n",
      "\n",
      "Epoch 00009: val_acc improved from 0.89088 to 0.89792, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 10/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2778 - acc: 0.8975 - val_loss: 0.2797 - val_acc: 0.8990\n",
      "\n",
      "Epoch 00010: val_acc improved from 0.89792 to 0.89904, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 11/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2702 - acc: 0.9004 - val_loss: 0.2721 - val_acc: 0.9016\n",
      "\n",
      "Epoch 00011: val_acc improved from 0.89904 to 0.90160, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 12/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2637 - acc: 0.9020 - val_loss: 0.2658 - val_acc: 0.9011\n",
      "\n",
      "Epoch 00012: val_acc did not improve from 0.90160\n",
      "Epoch 13/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2581 - acc: 0.9029 - val_loss: 0.2605 - val_acc: 0.9034\n",
      "\n",
      "Epoch 00013: val_acc improved from 0.90160 to 0.90336, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 14/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2530 - acc: 0.9045 - val_loss: 0.2559 - val_acc: 0.9045\n",
      "\n",
      "Epoch 00014: val_acc improved from 0.90336 to 0.90448, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 15/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2486 - acc: 0.9055 - val_loss: 0.2518 - val_acc: 0.9064\n",
      "\n",
      "Epoch 00015: val_acc improved from 0.90448 to 0.90640, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 16/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2445 - acc: 0.9074 - val_loss: 0.2480 - val_acc: 0.9070\n",
      "\n",
      "Epoch 00016: val_acc improved from 0.90640 to 0.90704, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 17/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2410 - acc: 0.9084 - val_loss: 0.2447 - val_acc: 0.9083\n",
      "\n",
      "Epoch 00017: val_acc improved from 0.90704 to 0.90832, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 18/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2377 - acc: 0.9090 - val_loss: 0.2417 - val_acc: 0.9083\n",
      "\n",
      "Epoch 00018: val_acc did not improve from 0.90832\n",
      "Epoch 19/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2346 - acc: 0.9106 - val_loss: 0.2398 - val_acc: 0.9078\n",
      "\n",
      "Epoch 00019: val_acc did not improve from 0.90832\n",
      "Epoch 20/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2319 - acc: 0.9115 - val_loss: 0.2362 - val_acc: 0.9107\n",
      "\n",
      "Epoch 00020: val_acc improved from 0.90832 to 0.91072, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 21/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2293 - acc: 0.9118 - val_loss: 0.2339 - val_acc: 0.9107\n",
      "\n",
      "Epoch 00021: val_acc did not improve from 0.91072\n",
      "Epoch 22/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2270 - acc: 0.9122 - val_loss: 0.2318 - val_acc: 0.9114\n",
      "\n",
      "Epoch 00022: val_acc improved from 0.91072 to 0.91136, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 23/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2247 - acc: 0.9132 - val_loss: 0.2299 - val_acc: 0.9114\n",
      "\n",
      "Epoch 00023: val_acc did not improve from 0.91136\n",
      "Epoch 24/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2228 - acc: 0.9135 - val_loss: 0.2278 - val_acc: 0.9125\n",
      "\n",
      "Epoch 00024: val_acc improved from 0.91136 to 0.91248, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 25/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2209 - acc: 0.9147 - val_loss: 0.2265 - val_acc: 0.9122\n",
      "\n",
      "Epoch 00025: val_acc did not improve from 0.91248\n",
      "Epoch 26/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2190 - acc: 0.9151 - val_loss: 0.2243 - val_acc: 0.9120\n",
      "\n",
      "Epoch 00026: val_acc did not improve from 0.91248\n",
      "Epoch 27/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2173 - acc: 0.9155 - val_loss: 0.2227 - val_acc: 0.9141\n",
      "\n",
      "Epoch 00027: val_acc improved from 0.91248 to 0.91408, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 28/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2158 - acc: 0.9163 - val_loss: 0.2213 - val_acc: 0.9142\n",
      "\n",
      "Epoch 00028: val_acc improved from 0.91408 to 0.91424, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 29/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2142 - acc: 0.9166 - val_loss: 0.2199 - val_acc: 0.9139\n",
      "\n",
      "Epoch 00029: val_acc did not improve from 0.91424\n",
      "Epoch 30/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2129 - acc: 0.9174 - val_loss: 0.2187 - val_acc: 0.9144\n",
      "\n",
      "Epoch 00030: val_acc improved from 0.91424 to 0.91440, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 31/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2114 - acc: 0.9179 - val_loss: 0.2173 - val_acc: 0.9149\n",
      "\n",
      "Epoch 00031: val_acc improved from 0.91440 to 0.91488, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 32/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2101 - acc: 0.9169 - val_loss: 0.2170 - val_acc: 0.9154\n",
      "\n",
      "Epoch 00032: val_acc improved from 0.91488 to 0.91536, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 33/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2090 - acc: 0.9179 - val_loss: 0.2151 - val_acc: 0.9170\n",
      "\n",
      "Epoch 00033: val_acc improved from 0.91536 to 0.91696, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 34/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2079 - acc: 0.9193 - val_loss: 0.2145 - val_acc: 0.9173\n",
      "\n",
      "Epoch 00034: val_acc improved from 0.91696 to 0.91728, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 35/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2068 - acc: 0.9182 - val_loss: 0.2130 - val_acc: 0.9171\n",
      "\n",
      "Epoch 00035: val_acc did not improve from 0.91728\n",
      "Epoch 36/100\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2057 - acc: 0.9198 - val_loss: 0.2122 - val_acc: 0.9176\n",
      "\n",
      "Epoch 00036: val_acc improved from 0.91728 to 0.91760, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 37/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2047 - acc: 0.9196 - val_loss: 0.2114 - val_acc: 0.9187\n",
      "\n",
      "Epoch 00037: val_acc improved from 0.91760 to 0.91872, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 38/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2038 - acc: 0.9199 - val_loss: 0.2103 - val_acc: 0.9182\n",
      "\n",
      "Epoch 00038: val_acc did not improve from 0.91872\n",
      "Epoch 39/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2027 - acc: 0.9206 - val_loss: 0.2096 - val_acc: 0.9189\n",
      "\n",
      "Epoch 00039: val_acc improved from 0.91872 to 0.91888, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 40/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2019 - acc: 0.9204 - val_loss: 0.2090 - val_acc: 0.9178\n",
      "\n",
      "Epoch 00040: val_acc did not improve from 0.91888\n",
      "Epoch 41/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2011 - acc: 0.9205 - val_loss: 0.2082 - val_acc: 0.9202\n",
      "\n",
      "Epoch 00041: val_acc improved from 0.91888 to 0.92016, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 42/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.2003 - acc: 0.9218 - val_loss: 0.2072 - val_acc: 0.9195\n",
      "\n",
      "Epoch 00042: val_acc did not improve from 0.92016\n",
      "Epoch 43/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1994 - acc: 0.9220 - val_loss: 0.2068 - val_acc: 0.9202\n",
      "\n",
      "Epoch 00043: val_acc did not improve from 0.92016\n",
      "Epoch 44/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1987 - acc: 0.9216 - val_loss: 0.2059 - val_acc: 0.9184\n",
      "\n",
      "Epoch 00044: val_acc did not improve from 0.92016\n",
      "Epoch 45/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1979 - acc: 0.9214 - val_loss: 0.2065 - val_acc: 0.9189\n",
      "\n",
      "Epoch 00045: val_acc did not improve from 0.92016\n",
      "Epoch 46/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1960 - acc: 0.9229 - val_loss: 0.2033 - val_acc: 0.9194\n",
      "\n",
      "Epoch 00048: val_acc did not improve from 0.92048\n",
      "Epoch 49/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1953 - acc: 0.9226 - val_loss: 0.2037 - val_acc: 0.9192\n",
      "\n",
      "Epoch 00049: val_acc did not improve from 0.92048\n",
      "Epoch 50/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1946 - acc: 0.9226 - val_loss: 0.2029 - val_acc: 0.9189\n",
      "\n",
      "Epoch 00050: val_acc did not improve from 0.92048\n",
      "Epoch 51/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1942 - acc: 0.9232 - val_loss: 0.2021 - val_acc: 0.9211\n",
      "\n",
      "Epoch 00051: val_acc improved from 0.92048 to 0.92112, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 52/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1934 - acc: 0.9231 - val_loss: 0.2015 - val_acc: 0.9211\n",
      "\n",
      "Epoch 00052: val_acc did not improve from 0.92112\n",
      "Epoch 53/100\n",
      "13264/18750 [====================>.........] - ETA: 9s - loss: 0.1921 - acc: 0.9226"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1882 - acc: 0.9254 - val_loss: 0.1966 - val_acc: 0.9210\n",
      "\n",
      "Epoch 00063: val_acc did not improve from 0.92208\n",
      "Epoch 64/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1876 - acc: 0.9256 - val_loss: 0.1963 - val_acc: 0.9210\n",
      "\n",
      "Epoch 00064: val_acc did not improve from 0.92208\n",
      "Epoch 65/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1873 - acc: 0.9251 - val_loss: 0.1968 - val_acc: 0.9230\n",
      "\n",
      "Epoch 00065: val_acc improved from 0.92208 to 0.92304, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 66/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1870 - acc: 0.9254 - val_loss: 0.1958 - val_acc: 0.9205\n",
      "\n",
      "Epoch 00066: val_acc did not improve from 0.92304\n",
      "Epoch 67/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1866 - acc: 0.9256 - val_loss: 0.1953 - val_acc: 0.9214\n",
      "\n",
      "Epoch 00067: val_acc did not improve from 0.92304\n",
      "Epoch 68/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1862 - acc: 0.9265 - val_loss: 0.1956 - val_acc: 0.9230\n",
      "\n",
      "Epoch 00068: val_acc did not improve from 0.92304\n",
      "Epoch 69/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1858 - acc: 0.9265 - val_loss: 0.1947 - val_acc: 0.9213\n",
      "\n",
      "Epoch 00069: val_acc did not improve from 0.92304\n",
      "Epoch 70/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1854 - acc: 0.9255 - val_loss: 0.1950 - val_acc: 0.9229\n",
      "\n",
      "Epoch 00070: val_acc did not improve from 0.92304\n",
      "Epoch 71/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1852 - acc: 0.9268 - val_loss: 0.1941 - val_acc: 0.9216\n",
      "\n",
      "Epoch 00071: val_acc did not improve from 0.92304\n",
      "Epoch 72/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1848 - acc: 0.9268 - val_loss: 0.1940 - val_acc: 0.9232\n",
      "\n",
      "Epoch 00072: val_acc improved from 0.92304 to 0.92320, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 73/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1845 - acc: 0.9274 - val_loss: 0.1937 - val_acc: 0.9230\n",
      "\n",
      "Epoch 00073: val_acc did not improve from 0.92320\n",
      "Epoch 74/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1841 - acc: 0.9266 - val_loss: 0.1933 - val_acc: 0.9206\n",
      "\n",
      "Epoch 00074: val_acc did not improve from 0.92320\n",
      "Epoch 75/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1838 - acc: 0.9268 - val_loss: 0.1930 - val_acc: 0.9218\n",
      "\n",
      "Epoch 00075: val_acc did not improve from 0.92320\n",
      "Epoch 76/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1834 - acc: 0.9266 - val_loss: 0.1928 - val_acc: 0.9227\n",
      "\n",
      "Epoch 00076: val_acc did not improve from 0.92320\n",
      "Epoch 77/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1832 - acc: 0.9273 - val_loss: 0.1926 - val_acc: 0.9224\n",
      "\n",
      "Epoch 00077: val_acc did not improve from 0.92320\n",
      "Epoch 78/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1826 - acc: 0.9273 - val_loss: 0.1924 - val_acc: 0.9237\n",
      "\n",
      "Epoch 00078: val_acc improved from 0.92320 to 0.92368, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 79/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1824 - acc: 0.9277 - val_loss: 0.1921 - val_acc: 0.9229\n",
      "\n",
      "Epoch 00079: val_acc did not improve from 0.92368\n",
      "Epoch 80/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1821 - acc: 0.9285 - val_loss: 0.1920 - val_acc: 0.9234\n",
      "\n",
      "Epoch 00080: val_acc did not improve from 0.92368\n",
      "Epoch 81/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1820 - acc: 0.9279 - val_loss: 0.1916 - val_acc: 0.9232\n",
      "\n",
      "Epoch 00081: val_acc did not improve from 0.92368\n",
      "Epoch 82/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1816 - acc: 0.9272 - val_loss: 0.1924 - val_acc: 0.9238\n",
      "\n",
      "Epoch 00082: val_acc improved from 0.92368 to 0.92384, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 83/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1814 - acc: 0.9282 - val_loss: 0.1911 - val_acc: 0.9226\n",
      "\n",
      "Epoch 00083: val_acc did not improve from 0.92384\n",
      "Epoch 84/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1811 - acc: 0.9273 - val_loss: 0.1909 - val_acc: 0.9222\n",
      "\n",
      "Epoch 00084: val_acc did not improve from 0.92384\n",
      "Epoch 85/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1808 - acc: 0.9278 - val_loss: 0.1908 - val_acc: 0.9219\n",
      "\n",
      "Epoch 00085: val_acc did not improve from 0.92384\n",
      "Epoch 86/100\n",
      "18750/18750 [==============================] - 45s 2ms/step - loss: 0.1804 - acc: 0.9276 - val_loss: 0.1903 - val_acc: 0.9227\n",
      "\n",
      "Epoch 00087: val_acc did not improve from 0.92384\n",
      "Epoch 88/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1801 - acc: 0.9286 - val_loss: 0.1901 - val_acc: 0.9227\n",
      "\n",
      "Epoch 00088: val_acc did not improve from 0.92384\n",
      "Epoch 89/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1797 - acc: 0.9284 - val_loss: 0.1913 - val_acc: 0.9234\n",
      "\n",
      "Epoch 00089: val_acc did not improve from 0.92384\n",
      "Epoch 90/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1795 - acc: 0.9279 - val_loss: 0.1897 - val_acc: 0.9230\n",
      "\n",
      "Epoch 00090: val_acc did not improve from 0.92384\n",
      "Epoch 91/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1795 - acc: 0.9284 - val_loss: 0.1896 - val_acc: 0.9224\n",
      "\n",
      "Epoch 00091: val_acc did not improve from 0.92384\n",
      "Epoch 92/100\n",
      "10608/18750 [===============>..............] - ETA: 14s - loss: 0.1796 - acc: 0.9285"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "IOPub message rate exceeded.\n",
      "The notebook server will temporarily stop sending output\n",
      "to the client in order to avoid crashing it.\n",
      "To change this limit, set the config variable\n",
      "`--NotebookApp.iopub_msg_rate_limit`.\n",
      "\n",
      "Current values:\n",
      "NotebookApp.iopub_msg_rate_limit=1000.0 (msgs/sec)\n",
      "NotebookApp.rate_limit_window=3.0 (secs)\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1779 - acc: 0.9288 - val_loss: 0.1884 - val_acc: 0.9232\n",
      "\n",
      "Epoch 00098: val_acc did not improve from 0.92464\n",
      "Epoch 99/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1776 - acc: 0.9291 - val_loss: 0.1882 - val_acc: 0.9237\n",
      "\n",
      "Epoch 00099: val_acc did not improve from 0.92464\n",
      "Epoch 100/100\n",
      "18750/18750 [==============================] - 44s 2ms/step - loss: 0.1774 - acc: 0.9292 - val_loss: 0.1880 - val_acc: 0.9243\n",
      "\n",
      "Epoch 00100: val_acc did not improve from 0.92464\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(x=x_train, \n",
    "                    y=y_train, \n",
    "                    batch_size=16, \n",
    "                    epochs=100, \n",
    "                    verbose=1, \n",
    "                    callbacks=callbacks, \n",
    "                    validation_split=0.25, \n",
    "                    shuffle=True, \n",
    "                    initial_epoch=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7995200000190735\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XmcXHWZ7/HPU1Vdva/pztrZCIEk7BBEAZVNNhccB1kcRkEdxjsjOip60auAqHNxruMKL+ei5ApuiBvGGRxkUVTWAIlAEkIWCOksne5OL+m1tuf+cU4nlU53qhNSXUn39/161St1tqrndOX1POf3+53F3B0REZF9iRQ6ABEROfSpWIiISE4qFiIikpOKhYiI5KRiISIiOalYiIhITioWMuGZ2RwzczOLjWLdq83sL2MRl8ihRMVCDitm9qqZJcysfsj85WHCn1OYyETGNxULORy9Alw5OGFmxwFlhQvn0DCalpHIgVKxkMPRD4H3Z01/ALg7ewUzqzazu82sxcw2mtnnzSwSLoua2dfMrNXMNgBvH2bbO81sq5ltNrMvm1l0NIGZ2c/NbJuZdZrZn8zsmKxlpWb272E8nWb2FzMrDZedaWaPm1mHmW0ys6vD+X80sw9nfcYe3WBha+qfzWwtsDac963wM7rM7Fkze3PW+lEz+5yZrTezneHymWZ2u5n9+5B9WWpmnxjNfsv4p2Ihh6MngSozWxgm8SuAHw1Z5ztANXAE8FaC4nJNuOwfgHcAJwGLgUuHbPsDIAUcGa5zPvBhRud3wHxgMvAc8OOsZV8DTgFOB+qAzwAZM5sdbvcdoAE4EVgxyu8DeDdwGrAonF4WfkYd8BPg52ZWEi77JEGr7GKgCvgg0AvcBVyZVVDrgfPC7UXA3fXS67B5Aa8SJLHPA/8buBB4EIgBDswBokACWJS13T8CfwzfPwJ8JGvZ+eG2MWAKMACUZi2/EvhD+P5q4C+jjLUm/NxqggOzPuCEYdb7LPDrET7jj8CHs6b3+P7w88/JEUf74PcCa4BLRlhvNfC28P1HgfsL/Xvrdei81Mcph6sfAn8C5jKkCwqoB4qAjVnzNgIzwvfTgU1Dlg2aHW671cwG50WGrD+ssJXzFeC9BC2ETFY8xUAJsH6YTWeOMH+09ojNzK4HPkSwn07Qghg8IWBf33UXcBVB8b0K+NbriEnGGXVDyWHJ3TcSDHRfDPxqyOJWIEmQ+AfNAjaH77cSJM3sZYM2EbQs6t29JnxVufsx5PY+4BKClk81QSsHwMKY+oF5w2y3aYT5AD3sOXg/dZh1dt06Ohyf+AxwGVDr7jVAZxhDru/6EXCJmZ0ALATuG2E9mYBULORw9iGCLpie7JnungbuBb5iZpXhmMAn2T2ucS/wMTNrNLNa4IasbbcCvwf+3cyqzCxiZvPM7K2jiKeSoNC0EST4f8363AywBPi6mU0PB5rfZGbFBOMa55nZZWYWM7NJZnZiuOkK4D1mVmZmR4b7nCuGFNACxMzsRoKWxaDvA18ys/kWON7MJoUxNhGMd/wQ+KW7941in2WCULGQw5a7r3f3Z0ZYfB3BUfkG4C8EA7VLwmXfAx4A/kowCD20ZfJ+IA6sIujv/wUwbRQh3U3QpbU53PbJIcuvB14gSMg7gK8CEXd/jaCF9Klw/grghHCbbxCMvzQTdBP9mH17APhv4OUwln727Kb6OkGx/D3QBdwJlGYtvws4jqBgiOxi7nr4kYgEzOwtBC2w2a7kIFnUshARAMysCPg48H0VChlKxUJEMLOFQAdBd9s3CxyOHILUDSUiIjmpZSEiIjmNm4vy6uvrfc6cOYUOQ0TksPLss8+2untDrvXGTbGYM2cOzzwz0lmUIiIyHDPbmHstdUOJiMgoqFiIiEhOKhYiIpLTuBmzGE4ymaSpqYn+/v5ChzJmSkpKaGxspKioqNChiMg4Mq6LRVNTE5WVlcyZM4es202PW+5OW1sbTU1NzJ07t9DhiMg4Mq67ofr7+5k0adKEKBQAZsakSZMmVEtKRMbGuC4WwIQpFIMm2v6KyNgY191QIiIHUzrj9CRSVMRjRCL5PTBrau/lL2tbSaYz1JUXU1cex93p6EvS2ZekZyBFIp0hkcowubKE9502K/eHvg4qFnnU1tbGueeeC8C2bduIRqM0NAQXSj799NPE4/Gcn3HNNddwww03cPTRR+c1VpGxlEwHT5wtih5Y50YilWFbZz9N7b1s3NHLqi1dvLilk1dae5hSWcLc+nKOaCjn6KmVLJhaxdz6cl5p7eGpV9p4dmM7tWVxTp5dy0kza4hEjKYdvTS197GxrYf1rT280tJDUSzC8TOqOb6xmnTG+fPaVv6yrpXOviQRg6rSIurK48yoKaWxtpSqkiK2dPazub2Xtp4ExbEIpUVR4rEIPQNpugdS9AykGEgFCT6RzuyxT1OqipldV870mhJWbuli7fbuUf89Tp5Vk/diMW5uJLh48WIfegX36tWrWbhwYYEi2tPNN99MRUUF119//R7zBx+GHokcvB7BQ2m/ZfxxdzZ39PHi5k66B9JkMk7GnSMnV3B8Yw3xWIRMxnl+cycPrWqmtXsg3A5auwfY0NrDazt6iUaMhdOqOKGxmilVJXT1JenoTZLMZKguLaKmNE7anZe37eSlbV1sau9jMF9lhqStiuIYi6ZVMW9yOS07w+9o6yU1dEVgcmUxXf1J+pOZvZZFDGbWlXFEfTl9yTQvbu6ieyAFBMn8LfMbOGpKJTv7k3T0JWnZOcDmjj42t/fR1Z9kek0pM2pKqa8oJpHK0JdMk0hlKC+OUlEco7w4RnEsKCDxqO3qNs64s7Wzn9faemlq7+WIhgrOOrqBs46eTFVpjB09CXZ0JzAzasqKqC4tCj8rQlE0QvR1tHLM7Fl3X5xrPbUsCmDdunW8613v4qSTTmL58uU8+OCDfPGLX+S5556jr6+Pyy+/nBtvvBGAM888k9tuu41jjz2W+vp6PvKRj/C73/2OsrIyfvOb3zB58uQC740cKtIZpz+Z3jXdk0ixvWuAlu4BEqkMdeVx6srjGPBKaw+vtPawuaOPvkSavmSadMZ3JbuGymK6+pK09SRo70nQk0jTn0zT2Zfkhc2dtOwcGDaGkqIIJzTWsLGtl21d/UQjRn3F7hZ0bVmcRdOqePtx00ikM/x1Uwe/fLaJnkSaeCxCTWkRRdEIXX1Jdg6kMIO5k8pZNL2Ki46bRixMitGIMb06OKKfWVfGjJrSvbqFEqkMG1q7WbNtJ+u2dzOzrow3HTGJxtpSUhlnzbadrNjUQTRiu1oHM2pLKY5Fd31GJuNsaA2O8Oc1VBRsTHByZUlBvjfbhCkWX/ztSlZt6Tqon7loehU3vfOYA9r2pZde4u6772bx4qCg33rrrdTV1ZFKpTj77LO59NJLWbRo0R7bdHZ28ta3vpVbb72VT37ykyxZsoQbbrhhuI+Xw4S709WX2nUEmsxkqCyJUVcWJxaNkEpn2L5zgG1d/ezsT9GXCNaNmFEWj1FaFGXd9p08tr6NJze0sbM/tV/fXx6PUl4coyweJMgHVzUzkNrziLssHqUsHgv/jXLmkfWcPKuG4xtrqC2LE4kErYaVW7p4ckMbz73Wzgkzq/nMMUdzzoLJ1JTtu7s1nXGS6QwlRdE95qfSGdLueyRvADIZ2LkFKqfDPlrk8ViEBVMqWVDWAzWt0LAAyssAKIoax86o5tgZ1fuMLRIxjpxcuc91RpTohce/Dc/dDRaBolIoKgte8cF/y3fPt3BfPAP9ndDbBr07oOEoWHQJzH0rRIsgk4adW6FtHbSsgZaXoKQGzrvpwOIcpQlTLA418+bN21UoAH76059y5513kkql2LJlC6tWrdqrWJSWlnLRRRcBcMopp/DnP/95TGOWQFd/kgdXNrOuZXefclHEqC6LU1NaREVJjKgZ0ajRn0izvqWbtdu7aWrvwwgSkLuzfecAWzv7SaT27g4BqCyJ0TOQ2qvLZTgz60p5+3HTmFtfzuDBb0lRlMmVxTRUllAci+xqJaQzzpz6co6oL6e2fM9E7u50rnqY1MsPkT7pA1TPOGqvJI47bHwcXvwuNJ4Ki94N0Rgz68q4cG4MXnwMyuth/gIojofrPwbP/gA6NwfJMV4GM06BN/4T0Vgx0ciQ70gNEGt6hlj7q1BWB2X1kOyF1b8NXt3bIF4J008MikBvK3S8Bju3QSQKReVBYu3YGCReAIvC3DcHiXfKcVA+Kfjc4koYrsWQTsGODUEy7m3bPb+vPZjX8hL0tAXxlddDxdQgsTcsCL7z4S9BVxPMvwDKJgXxJ3uDItK7A5JNu6eTfUGRGFRSFcRWWgMv/iooOCU1wXRnE2SyDgpKqmHeubn/k7xOE6ZYHGgLIF/Ky8t3vV+7di3f+ta3ePrpp6mpqeGqq64a9lqJ7AHxaDRKKrV/R5ES6Euk+eOa7Ty0ejvbdwZH7D0DKSpKYkyrLmFqVSnVpUWUxaOUxKMYQZfGQCrD8tfa+eOaFhLpDLGIEQmTTDKTYV/DfzOqSzixpociS5NOOSmLMXVGIxccM5XJlcWUxqPEo0H/c1d/krbuBJ09/Uwu6mNWcS/T4j3E62YRrZ1NSTxKxp2+RIbeRIrpNUFXzF4yGdi6AtY+CM0vBImttxUiMTjibJj/Npj1JoiG/6+2/RV76IvUbPhDML3yB3DujXDaPwZHvW3rYd1DQdJvWR0k36f+Ax76Ipz6QWheBavug3Qi2D5WEiSxHet3H/1OPQ76O6BzU5D0l/8I3v51mPsW2PZ8EOuGP8KmpyE9TFdXrCSIe/YZ0LoWtjwHf70HKiZDzUyoPzpIusleSA0ExWzyQqiZDZuehJX3wX9+Ys/PjMahvCFI6JFomLx7g8KTSQ7/g1bNgIajYfKiIPH3tsL21fDXn+xeZ+rx8Lffg9mnj/wfYzSS/bD+EXjpP4N9OuY9UDML6uZCw8Jg38ege2zCFItDWVdXF5WVlVRVVbF161YeeOABLrzwwkKHdUhyd7oHUpQWRYmFZ9L0J9O0dgdH6eu2d7O2uZvXdvQSixhl8SjFRVHSmQzJZJJZ7U+yfGuCF5LToGwSR06KszDazJyyLaxhDn/dVsuja1roSaT3+N56Onlv9FHeHm/hQ9PrmD2tnikzjsDmnAENC8lg7OxP0dmXZOdAkkwGUpkMRdEIR2Q2UvbI/4JX/rTnzvROhaLTof50qJ8P1TODo9QNj0LTffDy7yHZs+c2ZZNg+klQXLX7SNUiu7s38OAoNdED21dBTwtgweeXTw4SZ38nLPsePHn73n/g0lo4/yuw4GL43Q3wwGeDo9q+9uBoHmDaifCu2+CYvwn26fHvwEM3BzGdcg2ccnWw/qr7YM3vgmR2ye1BkotnFbV1D8F/fQruflewX4NH71OPh1M/DHPOCI/SO4JC5xmYcyYUVxzYf56jzodzvhB03XRshJ7WIMn3tAbf3dMafEdNWdAyqWgIknHD0VA5Nfg7QvD9xSN0TfV3QsvLMNAFR5wVFJ/Xq6gk+D0WXPz6P+t1ULE4BJx88sksWrSIBQsWMHv2bM4444xCh1RQG1q6+a/nt9LWk+D0eZN407zgKvxfL9/Mj57YyJrmnQBMKkpyZHQrG/oraaUaJ0IZ/RxbtJXjKrtZGTmKF1I1DKTSHMMr3JD5vyzMrIMoEAUvrsVau8AHC4PBwnfCZdeRqj2SgbZXSbW+QvHLSyle+19YJomXT8V29kFbLzwfHnWW1hGZehzV8Qqqi0qDZFJWH3RNtKyB5+4KugrOvQkqpwXbJLph01Pw6l9g5a/2/iOU1cPx7w2SVXnYHdG+MTiS3rIC2l/ds0D0tIWFxXb3ic99K8w/H448N/iMbImeINFve4FdTaKSajjxyuBfgPf9DJ6/Nygqc84MkvfsM4OulkGDSaxtPVRM2TORzzkDLv4/I//QR54H//QkPPZtaFsL884JWiKVU3L/JzlQZjB5QfDKh5JqmHlqfj67wHTq7Dh0qO23u9PU3sf6luCI/7W2Xuoq4pw+r57jZlSTcef5pg4eW9fG71dt48XNnZQxQHVRio5kjFSkmNIo1KS2c2ZDLxfXtzJrx2PM6HyOqAddcelInFS8muL+lj2/fMpxUH8krPpNkIDP/9LuJN66NjiinbwQaufAmvth2Z3BkWy2kmo48SpY/MHgswa1bwz64l99DFrXBEf0yV7o74K+HcFRqkWDo+SzbghaDXv/cYIumfaNQZ97d3PQdTL79INzVCqSg06dlTHX1j3A6q07ae7qZ0dPgtbuAVZv28kLTR209+7u+y2ORcIzbtZQWRLDM2lOSD3PVdGHuLLoVWrLuohlwv7qMF9mMCJRhy6CV8NCeNM/BYOkPS1EO14j2rsj7MddEBzlvvZ40Ae+9sEg0Z/zheAIHYKj2qEaF8OZn4QXfh4cedfMCl4NRweDskPVzg5eJ75v72WZzO6iM1yRGGS2+3tEDmEqFjJqvYkUm9v7aGrvY3NHH9u7+mkJxwpe2rqTbV17DsoXRY15DRW8bdEUjm+s4agplcypMhoyLXRuW8/G9Wto3/wyx3Y+Sn2iiUzpJCLzzwv6uMsbgsHM8Gg9YhGobgyS6qQjoWp67oBnngpnfHz/drK4AhZfs3/bDCcS2XeREDnMqFjIsNyd/mSGjTt6eHj1dh5e3czyTR17nPFjBovKdvL3RY9wbWkPk2vT1MbTxMtriFc1EK+sx/o7gu6VFzbBn1+Dnu0A1IQvIjGYsRgW30hk0SXBYJ6IHHJULIRkOsPa5m4eX9/K4+vbeHFzJx29SSrSHSyKbKTLy2iYMoNPvuUIZk2tp7GunMaSARpW3E7k6TsgkYZYPSRKIVMCnatgXSuk+oPTEqtnBqc1Hn0hVM8K3g92vVROU9+8yGFAxWICyGScdS3dLH+tnVdae9nRM8COngRtXb145ybq+zYwhXYAjquIc0VNP8fGn2Zaz2qMsCnRATwVfmCsNDiDKJ2EE66Asz+3d5+7ezDYGyvd51W2InJ4ULEYp9ydx9e38eOnNvLnl1thoIubi37AByMvEjGImFFBD3FPQPYTWAeAFgsGe9/wOZh5WpD0e1qDM3wSPcG0ezCwO2WEix3NglsZiMi4oGKRRwfjFuUAS5Ys4eKLL2bq1KkjrpNxD64yTqbp6kty7tcfZUNLDzVlRfzj/C6u3vJFyvu2wLF/iw1eGBWvCM70aVgQdBUN3pumqDS43YCISEjFIo8mTZrEihUrgJFvUT4aS5Ys4eSTT96rWPQnUvT3dBIZ6KQs3U2JpSkBqga289vuq0lObqSyYRbRDY8Gp5Jecz/MeuPB2DURmWBULArkrrvu4vbbbyeRSHD66adz2223kclkuOaaa1ixYgXuzrXXXsuUKVNYsWIFl19+OaWlpTz9yH9hniST6KMoM0CJORmM/lgFyVgJsWgELxmg/NS/D85C6ngtuHHaxf9Hp3KKyAGbOMXidzcEtzY4mKYeBxfdut+bvfjii/z617/m8ccfJxaLce2113LPPfcwb948WltbeeGFIM6Ojg5qamr4zre/zTduvYlT5k0l0r+dpEdJWpx0vJZ4WRWx0mrKsgaRraQTLv63g7abIiITp1gcQh566CGWLVu26xblfX19zJw5kwsuuIA1a9bwsY99jLdfeAFnnXkqfc3r8GQvRX076PbZdETrKC2voq48/rqejiUisj/yWizM7ELgWwQ3bfi+u986ZPlsYAnQAOwArnL3pnDZB4DPh6t+2d3vel3BHEALIF/cnQ9+8IN86Utf2mN+JuM88dTTPPibn3H7N/+NX/5kErd/9WYyFqWnfAZlU+dTdYDPLBYReT3yVizMLArcDrwNaAKWmdlSd1+VtdrXgLvd/S4zOwf438Dfm1kdcBOwGHDg2XDb9nzFO5bOO+88Lr30Uj7+kQ8xqQya27po6erHYjFmFPfzdxedzuzZs/nYZ76ATT2WmroGMhl23ZJbRGSs5bNl8QZgnbtvADCze4BLgOxisQj4ZPj+D8B94fsLgAfdfUe47YPAhcBP8xjvmDnumEXc9OmPct75F5BxpygW4z9u/RzRaISzr/8yRGJYJMpXv/pVYtEI11xzDR/+8IeDAe79OOVWRORgyWexmAFsyppuAk4bss5fgfcQdFX9DVBpZpNG2HbG0C8ws2uBawFmzTq079p58003Bbe/6G7Ge1q58uIzOf+ii9kRrWdSRQk1JRGipFnx/JV7PfXqsssu47LLLitQ5CIihR/gvh64zcyuBv4EbAbS+9wii7vfAdwBwfMs8hHgQTGwMziFNXzcZIJimjLTiJdVMa+mNGugumjkzxARKaB8FovNwMys6cZw3i7uvoWgZYGZVQB/6+4dZrYZOGvItn/MY6z54xm84zUyGac10sCOVAkZK2J6XSm1ZepOEpHDQz5HTJcB881srpnFgSuApdkrmFm92eA9JvgswZlRAA8A55tZrZnVAueH8/ZboZ8EmOjajqUTvJaeRKdVM7mmkgXTKvNWKAq9vyIyPuWtWLh7CvgoQZJfDdzr7ivN7BYze1e42lnAGjN7GZgCfCXcdgfwJYKCswy4ZXCwe3+UlJTQ1tZWkASacae5o4dIdzM9lFJfX8/8yRVMqigmmqe7sLo7bW1tlJTomRAicnCN62dwJ5NJmpqa6O/vH2Gr/EhlMuzoSVKa3kkFvXjFVCKxselyKikpobGxkaIijX+ISG56BjdQVFTE3Llzx/Q7/7hmO//y8xVMTm/nd9FPED3uUvib745pDCIiB5uu8jqI7n7iVa75wTIaK6P8pvEnQXfTOZ/PuZ2IyKFOxeIgeXFzJ7f8dhXnHNXAfbPupXTzY/DOb0P1XpeHiIgcdlQsDoK+RJqP37Oc+opibmt8kNgL98DZ/wtOuLzQoYmIHBTjeswi7zIZuP96+p+/n6/3lzJ7+lRKH3scTvw7eMunCx2diMhBo5bFgXKHBz4Lz9zJ8r4plNdMpsZ64YT3wTu+udctO0REDmdqWRyox74FT/0HP468g7uq/oGl170ZiqKFjkpEJC9ULA7E8/fCQzfxdPnZ3NLxPu77h5MpUaEQkXFM3VD7K5OG33+BltoTuartaj594UIWTqsqdFQiInmlYrG/Nj4O3dv4SutZnHrkVD54xthe9CciUgjqhtpfL/6Sfivhiehi7nvvCUT0HGwRmQDUstgf6SSpF+/jgdTJXHvesUyrLi10RCIiY0LFYn+88iixgXYejp7B5afOzL2+iMg4oW6o/dD97M/IeBmzTnsXFcX604nIxKGWxWilBoi+fD8P+WL+/syjCh2NiMiYUrEYpe6VD1Ca7qZtzjuZUqWHC4nIxKJiMUpb/vIjdngFb7ng0kKHIiIy5lQsRsG7tjK75RGeqziLo2fUFTocEZExp2IxCh0Pf52op+lb/E+FDkVEpCBULHLpaaXihbv5TeZ0TjzhpEJHIyJSECoWuTxxO9HMAEsrr2BmXVmhoxERKQgVi33pa8ef/h6/99OYdZRaFSIycalY7MtTd2CJnXwz8W7ePL++0NGIiBSMisVI3GHZ91hfeyZrbTZvnDep0BGJiBSMisVIurZATwsPJY7jpJk1VJUUFToiEZGCUbEYyfZVADzSXs+Z6oISkQlOxWIkzSsBWJ2ZyZvnNxQ4GBGRwlKxGMn2VXTGGvCSGk5orC50NCIiBaViMZLmVazONHL6vEnEovozicjEpiw4nHQSb13DisQMTphZU+hoREQKTsViOG3rsXSCNZmZzGuoKHQ0IiIFl9diYWYXmtkaM1tnZjcMs3yWmf3BzJab2fNmdnE4f46Z9ZnZivD1H/mMcy/bg8HtNT6TeQ3lY/rVIiKHorw9G9TMosDtwNuAJmCZmS1191VZq30euNfdv2tmi4D7gTnhsvXufmK+4tun5lVkiPKKNTKrTsVCRCSfLYs3AOvcfYO7J4B7gEuGrONAVfi+GtiSx3hGb/sqmosamVZXTTymnjoRkXxmwhnApqzppnBetpuBq8ysiaBVcV3Wsrlh99SjZvbm4b7AzK41s2fM7JmWlpaDF3nzStYwkyPUBSUiAhR+gPtK4Afu3ghcDPzQzCLAVmCWu58EfBL4iZlVDd3Y3e9w98Xuvrih4SBdODewEzo2srx/uga3RURC+SwWm4GZWdON4bxsHwLuBXD3J4ASoN7dB9y9LZz/LLAeOCqPse62/SUAVqUbVSxEREL5LBbLgPlmNtfM4sAVwNIh67wGnAtgZgsJikWLmTWEA+SY2RHAfGBDHmPdLTwT6iVXN5SIyKC8nQ3l7ikz+yjwABAFlrj7SjO7BXjG3ZcCnwK+Z2afIBjsvtrd3czeAtxiZkkgA3zE3XfkK9Y9NK8iGS2lyRvUshARCeWtWAC4+/0EA9fZ827Mer8KOGOY7X4J/DKfsY1o+yq2Fs+llhJqy+MFCUFE5FBT6AHuQ4s7NK9krc/iiHp1QYmIDMpry+Kwk+iGvh28GFEXlIhINrUssnVvB+CV/koNbouIZFGxyNbdDEAL1WpZiIhkUbHINlgsvIZ5k1UsREQGqVhkC7uhOiI1zKwtLXAwIiKHDhWLbN3NpIhSXTdFT8cTEcmijJitu5l2q+GIyZWFjkRE5JCiYpHFd25nW7qKufUarxARyaZikcW7m9nuNVSW6PITEZFsOYuFmV1nZrVjEUzBdW+jxasp1gOPRET2MJqsOIXgkaj3hs/UtnwHVRCZNNbTSgs1FGlwW0RkDzmzort/nuAW4XcCVwNrzexfzWxenmMbW707ME/T4nqUqojIUKPKiu7uwLbwlQJqgV+Y2b/lMbaxlXVBnloWIiJ7yjmSa2YfB94PtALfBz7t7snw8adrgc/kN8QxsqtYqGUhIjLUaE77qQPe4+4bs2e6e8bM3pGfsAogvHq7hRrialmIiOxhNFnxd8Cup9SZWZWZnQbg7qvzFdiYy+qGisfG5xi+iMiBGk2x+C7QnTXdHc4bX7q3k46V0UsJ8Wi00NGIiBxSRlMsLBzgBoLuJ8bjQ5O6m0mU1ANQFFXLQkQk22iKxQYz+5iZFYWvjwMb8h3YmOtuZqCkAUAD3CIiQ4wmK34EOB3YDDQBpwHX5jOogujeTn/xYMuwu/8WAAAQg0lEQVRCxUJEJFvO7iR33w5cMQaxFFZ3M31VpwDodh8iIkOM5jqLEuBDwDFAyeB8d/9gHuMaW6kB6O+gNz4JUMtCRGSo0WTFHwJTgQuAR4FGYGc+gxpz4TUWPUVBsdCYhYjInkaTFY909y8APe5+F/B2gnGL8SMsFjtjdYCKhYjIUKPJisnw3w4zOxaoBibnL6QCCC/IGywW6oYSEdnTaK6XuCN8nsXngaVABfCFvEY11sJi0RmtAzo0wC0iMsQ+i0V4s8Aud28H/gQcMSZRjbWwG6orUgt0qGUhIjLEPrNieLX2+Lir7L50N0NpHf0eIRoxohFdwS0ikm00h9APmdn1ZjbTzOoGX3mPbCx1N0PFFJJp160+RESGMZpicTnwzwTdUM+Gr2dG8+HhY1jXmNk6M7thmOWzzOwPZrbczJ43s4uzln023G6NmV0wut05QN3boWIyiVRGtycXERnGaK7gnnsgH2xmUeB24G0EtwlZZmZL3X1V1mqfB+519++a2SLgfmBO+P4KggsBpxO0bo5y9/SBxJJTdzPMPI1EOqPTZkVEhjGaK7jfP9x8d787x6ZvANa5+4bwc+4BLgGyi4UDVeH7amBL+P4S4B53HwBeMbN14ec9kSve/ea+u2XRpZaFiMhwRnPq7KlZ70uAc4HngFzFYgawKWt68CaE2W4Gfm9m1wHlwHlZ2z45ZNsZQ7/AzK4lvKnhrFmzcoQzgoGdkOoLxizaMxSpZSEispfRdENdlz1tZjXAPQfp+68EfuDu/25mbwJ+GF74NyrufgdwB8DixYs9x+rDy6TglGtg+kkkXlHLQkRkOAeSGXuA0YxjbAZmZk03hvOyfQi4F8DdnyBoudSPctuDo6wO3vlNmPvmYIBbLQsRkb2MZszitwRjCxAUl0WECT6HZcB8M5tLkOivAN43ZJ3XCLq1fmBmCwmKRQvBleI/MbOvEwxwzweeHsV3vi6JdEYX5ImIDGM0YxZfy3qfAja6e1Oujdw9ZWYfBR4AosASd19pZrcAz7j7UuBTwPfM7BMEBenq8BGuK83sXoLB8BTwz3k7EyqLWhYiIsMbTbF4Ddjq7v0AZlZqZnPc/dVcG7r7/QSnw2bPuzHr/SrgjBG2/QrwlVHEd9Ak0xnK4uPv8eIiIq/XaA6jfw5ksqbT4bxxR9dZiIgMbzSZMebuicGJ8H08fyEVTjKl232IiAxnNMWixczeNThhZpcArfkLqXCClkW00GGIiBxyRtNB/xHgx2Z2WzjdBAx7VffhLpHKqGUhIjKM0VyUtx54o5lVhNPdeY+qQBLpjB58JCIyjJyZ0cz+1cxq3L3b3bvNrNbMvjwWwY21pK6zEBEZ1mgy40Xu3jE4ET417+J9rH/Y0i3KRUSGN5rMGDWz4sEJMysFivex/mFLF+WJiAxvNAPcPwYeNrP/BxhwNXBXPoMqhEzGSWVc3VAiIsMYzQD3V83srwS3D3eC23fMzndgYy2RDq47VMtCRGRvo82MzQSF4r3AOcDqvEVUIMnBYqGWhYjIXkZsWZjZUQTPm7iS4CK8nwHm7mePUWxjKpFSy0JEZCT76oZ6Cfgz8A53XwcQ3h12XEqmg7uwa8xCRGRv+8qM7wG2An8ws++Z2bkEA9zjkloWIiIjGzEzuvt97n4FsAD4A/AvwGQz+66ZnT9WAY6VwQFu3e5DRGRvOQ+j3b3H3X/i7u8keLzpcuB/5j2yMTbYstDtPkRE9rZfmdHd2939Dnc/N18BFUpyV8tCxUJEZChlxpCusxARGZkyY2jXALdaFiIie1FmDO0a4FbLQkRkL8qMIbUsRERGpswYSmrMQkRkRMqMIbUsRERGpswYSmrMQkRkRMqMIbUsRERGpswYSoQ3ElSxEBHZmzJjSDcSFBEZmTJjSGdDiYiMTJkxlEhliBhEI7rrrIjIUCoWoUQ6o1aFiMgIlB1DiVRGd5wVERlBXrOjmV1oZmvMbJ2Z3TDM8m+Y2Yrw9bKZdWQtS2ctW5rPOCFoWehZFiIiw9vXM7hfFzOLArcDbwOagGVmttTdVw2u4+6fyFr/OuCkrI/oc/cT8xXfUEm1LERERpTP7PgGYJ27b3D3BHAPcMk+1r8S+Gke49knjVmIiIwsn9lxBrApa7opnLcXM5sNzAUeyZpdYmbPmNmTZvbuEba7NlznmZaWltcVbDKtloWIyEgOlex4BfALd09nzZvt7ouB9wHfNLN5QzcKH/G62N0XNzQ0vK4AEqmMrt4WERlBPrPjZmBm1nRjOG84VzCkC8rdN4f/bgD+yJ7jGQddIu26iaCIyAjymR2XAfPNbK6ZxQkKwl5nNZnZAqAWeCJrXq2ZFYfv64EzgFVDtz2YEqk0xWpZiIgMK29nQ7l7ysw+CjwARIEl7r7SzG4BnnH3wcJxBXCPu3vW5guB/2tmGYKCdmv2WVT5kEw7pUXRfH6FiMhhK2/FAsDd7wfuHzLvxiHTNw+z3ePAcfmMbahEKkNVSV7/HCIihy31u4QSKZ06KyIyEmXHkE6dFREZmbJjaEAtCxGRESk7hpJpXWchIjISZceQbvchIjIyZceQbiQoIjIyZceQWhYiIiNTdgTcnWTa1bIQERmBsiNBqwLQw49EREag7Ehwqw9AZ0OJiIxA2ZHg6m2AoqgVOBIRkUOTigW7i0U8phsJiogMR8WC4II8UMtCRGQkKhYEt/oAdOqsiMgIlB3Z3bLQALeIyPCUHckes9CfQ0RkOMqOZI9Z6M8hIjIcZUfUshARyUXZkd1XcKtlISIyPGVHdrcsdLsPEZHhKTuSdbsPFQsRkWEpOwKJdBpQN5SIyEiUHdEAt4hILsqOQCLshtLtPkREhqdiQdYAd1Q3EhQRGY6KBVkX5cXUshARGY6KBVljFhrgFhEZlrIjQcvCDKIRtSxERIajYkHQsohHI5ipWIiIDEfFguB2HzptVkRkZHnNkGZ2oZmtMbN1ZnbDMMu/YWYrwtfLZtaRtewDZrY2fH0gn3EOtixERGR4sXx9sJlFgduBtwFNwDIzW+ruqwbXcfdPZK1/HXBS+L4OuAlYDDjwbLhtez5iTaplISKyT/nMkG8A1rn7BndPAPcAl+xj/SuBn4bvLwAedPcdYYF4ELgwX4EmUhnd6kNEZB/ymSFnAJuyppvCeXsxs9nAXOCR/d32YNCYhYjIvh0qGfIK4Bfunt6fjczsWjN7xsyeaWlpOeAvT6RcLQsRkX3IZ4bcDMzMmm4M5w3nCnZ3QY16W3e/w90Xu/vihoaGAw5ULQsRkX3LZ4ZcBsw3s7lmFicoCEuHrmRmC4Ba4Ims2Q8A55tZrZnVAueH8/IimcoQ100ERURGlLezodw9ZWYfJUjyUWCJu680s1uAZ9x9sHBcAdzj7p617Q4z+xJBwQG4xd135CvWRDpDSZFaFiIiI8lbsQBw9/uB+4fMu3HI9M0jbLsEWJK34LIk0xkqS/L6pxAROazpcBpdlCcikosyJBrgFhHJRRkStSxERHJRhkS3+xARyUUZEt3uQ0QkF2VIwm4otSxEREakDAkk07rdh4jIvkz4DOnuOhtKRCSHCZ8hk+ngwnHd7kNEZGQTvlgk0hkAtSxERPZhwmfIZCooFhqzEBEZ2YTPkJGI8fbjp3FEQ0WhQxEROWRN+LvnVZcWcfv7Ti50GCIih7QJ37IQEZHcVCxERCQnFQsREclJxUJERHJSsRARkZxULEREJCcVCxERyUnFQkREcjJ3L3QMB4WZtQAbX8dH1AOtBymcw8VE3GeYmPs9EfcZJuZ+7+8+z3b3hlwrjZti8XqZ2TPuvrjQcYylibjPMDH3eyLuM0zM/c7XPqsbSkREclKxEBGRnFQsdruj0AEUwETcZ5iY+z0R9xkm5n7nZZ81ZiEiIjmpZSEiIjmpWIiISE4TvliY2YVmtsbM1pnZDYWOJ1/MbKaZ/cHMVpnZSjP7eDi/zsweNLO14b+1hY71YDOzqJktN7P/DKfnmtlT4W/+MzOLFzrGg83MaszsF2b2kpmtNrM3jfff2sw+Ef7fftHMfmpmJePxtzazJWa23cxezJo37G9rgW+H+/+8mR3wk94mdLEwsyhwO3ARsAi40swWFTaqvEkBn3L3RcAbgX8O9/UG4GF3nw88HE6PNx8HVmdNfxX4hrsfCbQDHypIVPn1LeC/3X0BcALB/o/b39rMZgAfAxa7+7FAFLiC8flb/wC4cMi8kX7bi4D54eta4LsH+qUTulgAbwDWufsGd08A9wCXFDimvHD3re7+XPh+J0HymEGwv3eFq90FvLswEeaHmTUCbwe+H04bcA7wi3CV8bjP1cBbgDsB3D3h7h2M89+a4DHRpWYWA8qArYzD39rd/wTsGDJ7pN/2EuBuDzwJ1JjZtAP53oleLGYAm7Kmm8J545qZzQFOAp4Cprj71nDRNmBKgcLKl28CnwEy4fQkoMPdU+H0ePzN5wItwP8Lu9++b2bljOPf2t03A18DXiMoEp3As4z/33rQSL/tQctxE71YTDhmVgH8EvgXd+/KXubBedTj5lxqM3sHsN3dny10LGMsBpwMfNfdTwJ6GNLlNA5/61qCo+i5wHSgnL27aiaEfP22E71YbAZmZk03hvPGJTMrIigUP3b3X4WzmwebpeG/2wsVXx6cAbzLzF4l6GI8h6AvvybsqoDx+Zs3AU3u/lQ4/QuC4jGef+vzgFfcvcXdk8CvCH7/8f5bDxrptz1oOW6iF4tlwPzwjIk4wYDY0gLHlBdhX/2dwGp3/3rWoqXAB8L3HwB+M9ax5Yu7f9bdG919DsFv+4i7/x3wB+DScLVxtc8A7r4N2GRmR4ezzgVWMY5/a4LupzeaWVn4f31wn8f1b51lpN92KfD+8KyoNwKdWd1V+2XCX8FtZhcT9GtHgSXu/pUCh5QXZnYm8GfgBXb333+OYNziXmAWwS3eL3P3oYNnhz0zOwu43t3fYWZHELQ06oDlwFXuPlDI+A42MzuRYFA/DmwAriE4OBy3v7WZfRG4nODMv+XAhwn658fVb21mPwXOIrgVeTNwE3Afw/y2YeG8jaBLrhe4xt2fOaDvnejFQkREcpvo3VAiIjIKKhYiIpKTioWIiOSkYiEiIjmpWIiISE4qFiL7wczSZrYi63XQbsZnZnOy7yQqciiJ5V5FRLL0ufuJhQ5CZKypZSFyEJjZq2b2b2b2gpk9bWZHhvPnmNkj4bMEHjazWeH8KWb2azP7a/g6PfyoqJl9L3wuw+/NrLRgOyWSRcVCZP+UDumGujxrWae7H0dwxew3w3nfAe5y9+OBHwPfDud/G3jU3U8guG/TynD+fOB2dz8G6AD+Ns/7IzIquoJbZD+YWbe7Vwwz/1XgHHffEN6wcZu7TzKzVmCauyfD+Vvdvd7MWoDG7FtPhLeOfzB8gA1m9j+BInf/cv73TGTf1LIQOXh8hPf7I/u+RWk0riiHCBULkYPn8qx/nwjfP05wx1uAvyO4mSMEj778H7DrGeHVYxWkyIHQUYvI/ik1sxVZ0//t7oOnz9aa2fMErYMrw3nXETyx7tMET6+7Jpz/ceAOM/sQQQvifxA84U3kkKQxC5GDIByzWOzurYWORSQf1A0lIiI5qWUhIiI5qWUhIiI5qViIiEhOKhYiIpKTioWIiOSkYiEiIjn9f14+meF4CyMzAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "print(history.history['val_acc'][0])\n",
    "# 绘制训练 & 验证的准确率值\n",
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.title('Model accuracy')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.legend(['Train', 'Test'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3XmYXHWd7/H3t7au3ju9ZF9JwpIIhNiDCsgmCAiDMyMjoCgCyujo4FxmEWfmcWGcK8wdURSuDjpxUGdALoiioigOjgKDJECELASykKSz0J1Oel9q+94/zkmnknSSDunqSro+r+epp6tOnar6Hir0p3/L+R1zd0RERAAixS5ARESOHgoFEREZolAQEZEhCgURERmiUBARkSEKBRERGaJQEBkBM5ttZm5msRHs+yEze/JI30ekGBQKMu6Y2WtmljKzxn22vxD+Qp5dnMpEjn4KBRmvNgBX735gZicDFcUrR+TYoFCQ8eq7wAfzHl8LfCd/BzOrNbPvmFmbmW00s38ws0j4XNTM/sXMdpjZeuDSYV77b2a2zcy2mNkXzCx6uEWa2VQze8TMdprZWjP7SN5zp5vZMjPrMrPXzeyOcHvSzL5nZu1m1mFmS81s0uF+tshwFAoyXj0D1JjZSeEv66uA7+2zz9eAWuA44ByCELkufO4jwGXAaUAzcMU+r/13IAPMC/d5J/DhN1Dn/UALMDX8jP9tZueHz90J3OnuNcBc4IFw+7Vh3TOABuCjQP8b+GyR/SgUZDzb3Vq4EFgNbNn9RF5QfNrdu939NeBLwAfCXd4LfMXdN7v7TuCLea+dBLwL+Et373X3VuDL4fuNmJnNAM4EPuXuA+6+HPgWe1o4aWCemTW6e4+7P5O3vQGY5+5Zd3/O3bsO57NFDkShIOPZd4H3AR9in64joBGIAxvztm0EpoX3pwKb93lut1nha7eF3TcdwL8CEw+zvqnATnfvPkANNwDHAy+HXUSX5R3XY8D9ZrbVzP7ZzOKH+dkiw1IoyLjl7hsJBpzfBfxgn6d3EPzFPStv20z2tCa2EXTP5D+322ZgEGh097rwVuPuCw+zxK1AvZlVD1eDu7/q7lcThM3twINmVunuaXf/vLsvAM4g6Ob6ICKjQKEg490NwPnu3pu/0d2zBH30/2Rm1WY2C7iZPeMODwA3mdl0M5sA3JL32m3AL4AvmVmNmUXMbK6ZnXM4hbn7ZuBp4Ivh4PEpYb3fAzCza8ysyd1zQEf4spyZnWdmJ4ddYF0E4ZY7nM8WORCFgoxr7r7O3Zcd4Om/AHqB9cCTwH8CS8LnvknQRfN74Hn2b2l8EEgAq4BdwIPAlDdQ4tXAbIJWw8PAZ9398fC5i4GVZtZDMOh8lbv3A5PDz+siGCv5b4IuJZEjZrrIjoiI7KaWgoiIDFEoiIjIEIWCiIgMUSiIiMiQY2753sbGRp89e3axyxAROaY899xzO9y96VD7HXOhMHv2bJYtO9AMQxERGY6ZbTz0Xuo+EhGRPAoFEREZolAQEZEhx9yYwnDS6TQtLS0MDAwUu5Qxk0wmmT59OvG4FscUkdEzLkKhpaWF6upqZs+ejZkVu5yCc3fa29tpaWlhzpw5xS5HRMaRcdF9NDAwQENDQ0kEAoCZ0dDQUFItIxEZG+MiFICSCYTdSu14RWRsjJtQOJTewQzbO/vRqrAiIgdWMqHQl8rS2j1IrgCh0N7ezqJFi1i0aBGTJ09m2rRpQ49TqdSI3uO6665jzZo1o16biMjhGBcDzSMRCeMvm4PoKEdhQ0MDy5cvB+Bzn/scVVVV/PVf//Ve+7g77k4kMvyHf/vb3x7dokRE3oCSaSlEwz74QrQUDmTt2rUsWLCA97///SxcuJBt27Zx44030tzczMKFC7n11luH9j3rrLNYvnw5mUyGuro6brnlFk499VTe9ra30draOmY1i0hpG3cthc//eCWrtnbttz2bcwbSWcoTUSKHOUi7YGoNn/3Dw70me+Dll1/mO9/5Ds3NzQDcdttt1NfXk8lkOO+887jiiitYsGDBXq/p7OzknHPO4bbbbuPmm29myZIl3HLLLcO9vYjIqCqZlsLuHBjrcea5c+cOBQLAfffdx+LFi1m8eDGrV69m1apV+72mvLycSy65BIA3v/nNvPbaa2NVroiUuHHXUjjQX/T9qSyvtnYzq76C2orEmNVTWVk5dP/VV1/lzjvv5Nlnn6Wuro5rrrlm2HMNEok99UWjUTKZzJjUKiJSMi2F3YPL2SLOSO3q6qK6upqamhq2bdvGY489VrxiRESGMe5aCgcSKcJA874WL17MggULOPHEE5k1axZnnnlm0WoRERmOHWsnczU3N/u+F9lZvXo1J5100kFfl3NnxZZOJtckmViTLGSJY2Ykxy0iAmBmz7l786H2K5nuo4gZZkb2GAtBEZGxVNBQMLOLzWyNma01s2HnVJrZe81slZmtNLP/LGQ9UTNyOYWCiMiBFGxMwcyiwN3AhUALsNTMHnH3VXn7zAc+DZzp7rvMbGKh6oHgrOZiDjSLiBztCtlSOB1Y6+7r3T0F3A+8e599PgLc7e67ANy9oKfuRtRSEBE5qEKGwjRgc97jlnBbvuOB483sKTN7xswuHu6NzOxGM1tmZsva2trecEFRjSmIiBxUsQeaY8B84FzgauCbZla3707ufo+7N7t7c1NT0xv+sEhELQURkYMpZChsAWbkPZ4ebsvXAjzi7ml33wC8QhASBRE1oxCZMBpLZwMsWbKE7du3j36BIiIjVMiT15YC881sDkEYXAW8b599fkjQQvi2mTUSdCetL1RBkUiwMN5oG8nS2SOxZMkSFi9ezOTJk0e7RBGRESlYKLh7xsw+ATwGRIEl7r7SzG4Flrn7I+Fz7zSzVUAW+Bt3by9UTUFLYWy7j+69917uvvtuUqkUZ5xxBnfddRe5XI7rrruO5cuX4+7ceOONTJo0ieXLl3PllVdSXl7Os88+u9caSCIiY6Ggy1y4+6PAo/ts+0zefQduDm+j42e3wPaXhn2qIZujOpPDy6IYh7F89uST4ZLbDruUFStW8PDDD/P0008Ti8W48cYbuf/++5k7dy47duzgpZeCOjs6Oqirq+NrX/sad911F4sWLTrszxIRGQ0ls/ZRMTz++OMsXbp0aOns/v5+ZsyYwUUXXcSaNWu46aabuPTSS3nnO99Z5EpFRALjLxQO8hd9T+8gLbv6OXFyDYlY4SdeuTvXX389//iP/7jfcy+++CI/+9nPuPvuu3nooYe45557Cl6PiMihFHtK6pga65VSL7jgAh544AF27NgBBLOUNm3aRFtbG+7On/7pn3Lrrbfy/PPPA1BdXU13d/eY1CYiMpzx11I4iGgkCIVCzEAazsknn8xnP/tZLrjgAnK5HPF4nG984xtEo1FuuOEG3B0z4/bbbwfguuuu48Mf/rAGmkWkaEpm6WyA3sEM69p6mNNYSXUyXqgSx4yWzhaRkdLS2cOIhC0FndUsIjK8kgqFaDgLVSuliogMb9yEwki6wYYGmsdBS+FY6/YTkWPDuAiFZDJJe3v7IX9R7u4+OtZXSnV32tvbSSbHx2VFReToMS5mH02fPp2WlhZGsqx2a0c/fWUxdpYf2wPNyWSS6dOnF7sMERlnxkUoxONx5syZM6J9P/CFX/LOhZP533+sWTsiIvsaF91Hh6OyLEbvYKbYZYiIHJVKLhSqymL0DCgURESGU3KhUFkWo1stBRGRYY2LMYUR2bkBXl9BdWIS27sHi12NiMhRqXRaCqt+BN+/hgmJND1qKYiIDKt0WgoV9QBMjPXRq4aCiMiwSqelUB6EQmOkh24NNIuIDKt0QqGiAYB662EwkyOdzRW5IBGRo08JhULQUqgjuIiNzlUQEdlf6YRC2H1U40EoaLBZRGR/JRQKEwCoynUBCgURkeGUTihEY1BWS0U2DAUNNouI7Kd0QgGgYgLlmU5ALQURkeGUViiU15NMdwAKBRGR4ZRWKFQ0EB/cBWj2kYjIcEosFOqJDQYtBZ3AJiKyv9IKhfJ6rH93SyFb5GJERI4+pRUKFfVYqpuauNMzmC52NSIiR53SCoXwXIUpiX561FIQEdlPaYVCuNTF1ES/Zh+JiAyjtEIhXOpiYrxXs49ERIZRWqEQrpQ6MdqjM5pFRIZRYqGw+5oKfbpOs4jIMAoaCmZ2sZmtMbO1ZnbLMM9/yMzazGx5ePtwIevZ3X1UH+lW95GIyDAKdjlOM4sCdwMXAi3AUjN7xN1X7bPr9939E4WqYy+JCoglqaNHA80iIsMoZEvhdGCtu6939xRwP/DuAn7eyJTXU0O3QkFEZBiFDIVpwOa8xy3htn29x8xeNLMHzWxGAesJVNRTnesilcmRyuiSnCIi+Yo90PxjYLa7nwL8Erh3uJ3M7EYzW2Zmy9ra2o7sE8snUBleU0HjCiIieytkKGwB8v/ynx5uG+Lu7e4+GD78FvDm4d7I3e9x92Z3b25qajqyqioaKM8EodDRr6UuRETyFTIUlgLzzWyOmSWAq4BH8ncwsyl5Dy8HVhewnkBFPcl0sCjejp7BQ+wsIlJaCjb7yN0zZvYJ4DEgCixx95VmdiuwzN0fAW4ys8uBDLAT+FCh6hlSXk881YmRo7VLoSAikq9goQDg7o8Cj+6z7TN59z8NfLqQNeynoh7zHNX00do9MKYfLSJytCv2QPPYC09ga4r00tqtloKISL7SC4VwqYvZlYO0KRRERPZSeqEQthRmlQ+opSAiso/SC4W8ayq0dmlMQUQkX8mGwuR4n7qPRET2UXqhUFYLFqEp2svOvhTprJa6EBHZrfRCIRKB8glMsG7cob0nVeyKRESOGqUXChCslOrdADpXQUQkT2mGQkX90KJ4OqtZRGSP0gyF8nqS6U4A2rT+kYjIkNIMhYoGYoPBonhqKYiI7FGioTAB69/JhIq4xhRERPKUZiiU10NmgBlV6KxmEZE8pRkKFQ0AHFc5oBPYRETylGYo1E4HYG5il0JBRCRPaYZC3SwAZkfbaesexN2LXJCIyNGhREMhuHT0VFpJZXN09OlazSIiUKqhECuD6ik0ZV4HdK6CiMhupRkKAHWzqBvcCuhcBRGR3Uo4FGZS3rcF0PpHIiK7lW4oTJhFrGcrUbI6V0FEJFS6oVA3E/MscxIdmpYqIhIq4VAIpqW+qaJDLQURkVAJh8JMAOYndupazSIiodINhdrpYBHmxHao+0hEJFS6oRCNQ800ptGmUBARCZVuKADUzaIpu53uwQz9qWyxqxERKboSD4WZ1A1uA3SugogIlHooTJhF+UArCdK07OovdjUiIkU3olAws7lmVhbeP9fMbjKzusKWNgbqZmI4U6yddW09xa5GRKToRtpSeAjImtk84B5gBvCfBatqrITnKpyQ2MnaVoWCiMhIQyHn7hngj4GvufvfAFMKV9YYCc9VOKW6Sy0FERFGHgppM7sauBb4SbgtXpiSxlDNVIjEOLGsnXWtvcWuRkSk6EYaCtcBbwP+yd03mNkc4LuFK2uMRKJQO52Z0R1s7xqge0AX2xGR0jaiUHD3Ve5+k7vfZ2YTgGp3v73AtY2NullMDC+2s75NrQURKW0jnX30azOrMbN64Hngm2Z2xwhed7GZrTGztWZ2y0H2e4+ZuZk1j7z0UVI3k6qB4GI7GmwWkVI30u6jWnfvAv4E+I67vwW44GAvMLMocDdwCbAAuNrMFgyzXzXwSeB3h1P4qJkwi1hfK5WRtAabRaTkjTQUYmY2BXgvewaaD+V0YK27r3f3FHA/8O5h9vtH4HagOKcUh9NS31KnGUgiIiMNhVuBx4B17r7UzI4DXj3Ea6YBm/Met4TbhpjZYmCGu//0YG9kZjea2TIzW9bW1jbCkkdo4kkAnFG1Xd1HIlLyRjrQ/P/c/RR3/1j4eL27v+dIPtjMIsAdwF+N4PPvcfdmd29uamo6ko/dX9OJEEtyanQDG9v7SGdzo/v+IiLHkJEONE83s4fNrDW8PWRm0w/xsi0EZz7vNj3ctls18Cbg12b2GvBW4JExH2yOxmHyycxOvUIm52xs7xvTjxcROZqMtPvo28AjwNTw9uNw28EsBeab2RwzSwBXhe8BgLt3unuju89299nAM8Dl7r7sMI/hyE09jfrOVUTIaVxBREraSEOhyd2/7e6Z8PbvwEH7ccJlMT5BMBaxGnjA3Vea2a1mdvkRVT3api4mmunjONuqcQURKWmxEe7XbmbXAPeFj68G2g/1Ind/FHh0n22fOcC+546wltE39TQA3l6xWS0FESlpI20pXE8wHXU7sA24AvhQgWoae43zIV7JW8s3sU5nNYtICRvp7KON7n65uze5+0R3/yPgiGYfHVUiUZhyKgtya1nX2oO7F7siEZGiOJIrr908alUcDaYtZsrAWgYGB2jtHix2NSIiRXEkoWCjVsXRYOppxHKDzLctrNneXexqRESK4khCYXz1sYSDzadG1vPcxl1FLkZEpDgOOvvIzLoZ/pe/AeUFqahYJsyBslrOLtvM917bWexqRESK4qCh4O7VY1VI0UUiMHURi15/jb/a1EE6myMePZKGlIjIsUe/9fJNPY3J/evIpgdYubWr2NWIiIw5hUK+qacR8TQn2UaWqQtJREqQQiHfzLcBcGnVKzy7QaEgIqVHoZCvehJMOpnz4ytYtnGXTmITkZKjUNjXvHdwXP8KBns7teSFiJQchcK+5r2DiGc4I7JS4woiUnIUCvua8VY8Xsk7y1byrEJBREqMQmFfsQQ252zOjb7I0g2HXB1cRGRcUSgMZ947aMpsI9qxge2dA8WuRkRkzCgUhjPvHQCcE3mRZ9artSAipUOhMJz64/AJc7ggsYJfrNpe7GpERMaMQuEAbN4FvMVW8uTLWxlIZ4tdjojImFAoHMi8d5DIDXBKdiW/eaWt2NWIiIwJhcKBzDkHL6vmvYmn+flKdSGJSGlQKBxIogJ703u4yH7H/6zaQDqbK3ZFIiIFp1A4mEXXUOYDnJ1+iv9Zp1lIIjL+KRQOZnozucbjuSr23+pCEpGSoFA4GDMip32A0+wV1qx4jmxOq6aKyPimUDiUU64kZ1EuHHyc5zbuKnY1IiIFpVA4lOpJ5OZdyHuiv+XhZa8VuxoRkYJSKIxAbPEHaLIOOl78KZ196WKXIyJSMAqFkTj+IlKVU7nBHuH/LdtU7GpERApGoTAS0TiJc26mOfIKK5/+KTkNOIvIOKVQGKnTPsBAWSNX9N7PU+t2FLsaEZGCUCiMVDxJ7O2f5MzoSp564tFiVyMiUhAKhcMQO/0G+mK1vGXzErZ09Be7HBGRUadQOByJStJ/8DHOiy7n0cd+VuxqRERGnULhMNWe83H6otUsWHkHLTt7i12OiMioKmgomNnFZrbGzNaa2S3DPP9RM3vJzJab2ZNmtqCQ9YyKZA3psz7FmZGX+PnD9xa7GhGRUVWwUDCzKHA3cAmwALh6mF/6/+nuJ7v7IuCfgTsKVc9oqj37o7QnZ3H+xq+yqkWrp4rI+FHIlsLpwFp3X+/uKeB+4N35O7h7V97DSuDYOAEgGid56Rc5LrKNFx76P8WuRkRk1BQyFKYBm/Met4Tb9mJmHzezdQQthZuGeyMzu9HMlpnZsra2o+PSmJVvehebJ7yVy3bey7OrXi12OSIio6LoA83ufre7zwU+BfzDAfa5x92b3b25qalpbAs8EDMmXvEvVNkA7Q/fwmAmW+yKRESOWCFDYQswI+/x9HDbgdwP/FEB6xl1ZdNOpmXBjVySfpzHHvjXYpcjInLEChkKS4H5ZjbHzBLAVcAj+TuY2fy8h5cCx1w/zKz3fIGNyZM4Z80XeOWV1cUuR0TkiBQsFNw9A3wCeAxYDTzg7ivN7FYzuzzc7RNmttLMlgM3A9cWqp6Cicapu+ZeopYj/cANZNJaWltEjl3mfmxM+NmtubnZly1bVuwy9rP8J//KomV/y3MzruXNN3y12OWIiOzFzJ5z9+ZD7Vf0gebx4tRLb+TJ2st48+Z72fjDW4tdjojIG6JQGCVmxqKPLuHx+LnMWv4lOn91TJyHJyKyF4XCKKoqL2POh+/l5/42an/7edJPf6PYJYmIHBaFwiibO6mO6BXf5LFsM/FffApffl+xSxIRGTGFQgFcePIM1p59J09lF+I//Dis0TLbInJsUCgUyJ9fsJDHT/0yL+Zmkfn+B2HDb4tdkojIISkUCsTM+Ic/OZ3/mPcl1meayHz3Cljz82KXJSJyUAqFAopGjC+87xy+PO0OVmamkrv/ffD8d4tdlojIASkUCqwsFuXL11/IPXPu5LeZhfDIJ/Bf3wa5XLFLExHZj0JhDCTjUb7ywbP4ycI7eCj7duzXXyR335XQt7PYpYmI7EWhMEbi0Qi3v7eZl99yO/+Qvo7sq0+Q/cbbYfPSYpcmIjJEoTCGIhHj7y9byKl//Fdcmfkcr3en8CUXweOfg/RAscsTEVEoFMOfNs/gs392DdfGv8SD2bfDk1/Gv3EWbHqm2KWJSIlTKBTJqTPquO+mi3h0zt9zTerTtHd2wZKL4Ad/Bl3bil2eiJQohUIRNVaVseRDf8D577qSC/pv59uRPyG74iH8rmb47R2Q7i92iSJSYhQKRWZmXH/WHP7j4+fzYN31nNf/zyyPnQK/+jx87c3wwvcgp+s/i8jYUCgcJRZOreVHHz+TD1xyLu/r/iTX5j7D614HP/o4fP1MeOlBhYOIFJxC4SgSi0b4yNnH8Yv/dTaJeefwlra/4zNlf0vPYAoeugHuaobnv6OZSiJSMLoc51HsN6+08fkfr2R9WzcfnbiaP4/+kOpdK6GiARZ/EJqvh7qZxS5TRI4BI70cp0LhKJfO5vj+0s383yfWsrWznw9O2sjHq55g4tZfYe4w5VSYex7MPR9mnQmRaLFLFpGjkEJhnEllcjz4XAt3P7GWLR39nNXUz99N+z0n9j5LZMsyyGWg6SQ47+/gpD8Es2KXLCJHEYXCOJXO5vjx77fy9V+v49XWHqbUJrnhDxq5unYllb/7Mux4BSafAqd/BE64FCobil2yiBwFFArjXC7nPLGmlSVPbeCpte0k4xEuP3kSH2t4ntmrvo61rwWLwuyz4MTL4ISLNf4gUsIUCiXk5e1d3Pv0azyyfCu9qSzzmir5sxN6uSTyLFXrfgrtrwY7TnoTzL8wGH+Y8RaIlRW3cBEZMwqFEtQ7mOGnL27j/qWbeH5TB2Zw5txG3jc3xXn2HOUbfgmbnwnGH+IVMOfsICTmXwR1M4pdvogUkEKhxG3Y0cvDL2zhhy9sYdPOPmIR44x5jfzhidVcVPEKNVt+A6/+Ejo2Bi+YMDsYi5hyCkw/HWadAdF4UY9BREaPQkEAcHdWbOnipy9t49GXtrFpZx8Rg9Pn1HPBiRN5x8QuZu98Etv8LGx/CXZtCF6YrIMTLoHjLwq6mmqmFvdAROSIKBRkP+7Oqm1dPLZiOz9bsZ1XW3sAmFRTxlnzmjj7+EbOmpGgofUZWP0TeOXnMNARvLhmOkxvhmmLYeppwfkRydoiHo2IHA6FghzSlo5+nny1jd+8soOn1u2goy8NwMKpNZw5r5Gzjqvl9ORmktufh83PQssy6Ny05w0a5gUBMfU0mLgAJi2EqolFOhoRORiFghyWbM5ZsaWT37zSxlPrdvD8xg5S2RyxiHHK9FreelwDp8+p57SGLLUdK2HLC7A1vHVv3fNGFY0w+eTgNmkhlNdDohLKqqHpRIglineQIiVMoSBHpD+V5dnXdvLM+naeWd/OSy2dZHLBv5XjGitZNLOOxTMnsHjmBE6o6ie6YzW0robXV8D2FdC6CrKpvd80Xglz3h5Mia2fG5xYV9kE1VMhorUZRQpJoSCjqi+VYfmmDl7Y3MHyzR28sGkXO3qCX/oViSgnTalhwZQaFkyt4dTpdRzfWEasaxMMdEGqB/ra4bUnYd1/7RnM3i1RDVMXBd1QTSdA3azgRLuaqZoBJTJKFApSUO7O5p39PL9pF8s3d7BqaxertnXRM5gBoDwe5eRptSyYWsOJk6s5cUoN8yZWUVUWg47N0LUV+nZAT2vQutjyfPBzr9aFQWUjVE8Opsw2nRjcJsyB6klQOVHdUSIjpFCQMZfLOZt29vH7lqA18WJLJy9v66I3tefiQJNqypjbVMX8iVXMn1TN8ZOqmdtUSX1lAsumoasFOjYFt66t0L0tuGb1zvXBzfe50FD5hCAcKpuCQe7aacFMqboZQRfVhNkQT47tfwiRo5BCQY4KuZzTsquf1du7WNfWw7rWXta29bCutWeoVQFQnYxxXGMlxzVVMW9icJvbVMmM+grKYuFy4JlBaF8bBEbP69D9evCztw16d0DPdujcAtnBvAoMqqdAsiYY8E7WBkFRf1zQPTXQGbRW+jv2LENePXlM/xuJjIWRhkKswEVcDNwJRIFvuftt+zx/M/BhIAO0Ade7+8ZC1iRjKxIxZjZUMLOhYq/t7s7WzgFeeb2bDW29bNjRy/odPfzPunYefmHLntcbTK0rZ3ZDEBAz6yuY1bCIOVMrmd1QSXlin+tHuAfjFx0boX19ECKdm2GwG1K90L8Ltv4Q+nfu/bp4BaT7gvsTF0DDXKiaFN4mBq2RqongufC9eoLXVE8JwqV8gpYrl3GhYC0FM4sCrwAXAi3AUuBqd1+Vt895wO/cvc/MPgac6+5XHux91VIY/7oH0qxr62XDjh427OjjtR29bNzZx+adfezs3XtG06SaMqbVlTN9QgVT68qZXFPGpJokk2qTTK8rp7GqjEhkmF/W/buC7qnyCUHXk0WDMY11/wUbfgNdW4JWSP+ukRUdr4DaGeEA+ZTg6ngVDcEU3aqmIFwqJwbbouHfYqle2PVaMBg/aYFOBpSCKnr3kZm9Dficu18UPv40gLt/8QD7nwbc5e5nHux9FQqlrXsgzcb2Pl5r72VDWxAWW3b1s6Wjn22d/aSze/97TkQjTKlLhsFRzrS6CibXBsExpbacybVJapIx7EB/5WcGg+6pntagi8oiUFYFiargl3r31mDMo2tL0Drp2BR0a/W1Qy49zBsaVNRDJBaETr7G44MurLpZ4djItGC5kWRt0P1VVh1M69X0XXkDjoafnVgkAAANeUlEQVTuo2nA5rzHLcBbDrL/DcDPhnvCzG4EbgSYOVPXBChl1ck4b5pWy5um7f9XdS7n7OpLsb1rgO2dA2zt6Kelo38oNJ5Y00Zb9+B+ryuPR5lSm2RybRAUU2qTNFQlqK9M0FBZRn1lDY01jdRPThCLjvAXsnvQzdTbFobK63uCpbctmGU1YVYwkypRBdtfDGZgbXoGVvxg/wH1IRbsvzuYyqogmghCJhKFWBLi5UHLpaIBaqcHt4qGYFuiKlglt39XcIsmoH5O0MLR9F+hwGMKI2Vm1wDNwDnDPe/u9wD3QNBSGMPS5BgSiRgNVWU0VJWxcOrwXTGDmSytXYO83jXAts6BoZ/bOvvZ3jnA0+t28HrXALlh/pWZQUNlIuieqkkysbqMidVlNFWXMaEyQU0yTk15nIbKBBNryihL1gR/4TfMPXTxJ1y85342Ew6kbwvWnhroCgbEB7v33FLdMNgTjG1k08FYRyYVPJfuh1Qf9LZCZmBk//EsGoyNVNTvCZB0f3DDg1ZL7fSgqy0Tvn8uE3SZ1c8JfsbKwnCKBffj5UFIaazlmFLIUNgC5C/SPz3cthczuwD4e+Acd9//zziRUVQWizKjvoIZ9RUH3Cebczr70+zsTbGzN0V7zyA7elPs6B6ktTsIlO2dA7y0pZP2nsFhAwSgsSpBU3WShsoEEyoT1JXHqSyLUVUWpaY8TmNVECgNlQnqKhLUJGNBSyQaC7qPaqcd2cG6Q9/OYKB9oCPo7kr1Bl1g5ROCW2YAdm4Ipvt2tgQD8H3t0L19T4vDHbY8B6t+tKdLLBILgiR7qP9lLej2StYGt0RVMAssUQGx8qClEo0HIRIrC0IkGodIPNyeDPevDPeP79k/UR20lLA93XjpvqAbrulETUV+gwoZCkuB+WY2hyAMrgLel79DOI7wr8DF7t5awFpERiwaMeorg+6jQ8lkc+zsTdHRn6arP01nf5r2nhTbOgfY3jVAW/cAO3tTbOnop6MvRe9gllQ2d8D3qy6LUVkWo7IsSlUyTn1FnPrKMhqqEtRVxKkrD37WJONUJWNUJ2PUJOPUlsdJxPbp2jILlxI5xHW6Z50xkv8skMvBYFcQFLFEEBY9rcEZ6p0tQZdYLhsER2YwbGn0Ba2Xgc7gluoJAqprS7BPNh0ESyYV/hxhy+ZQLBp0iZVVhUFTFozFWCRsySTzutrKw0AqzwumRBCmbS8H1z0vr4fpfwAzTg9aRdFYsA8WtNI8F7zv7nW+ElXBe5nt+e/UsTGY+lw1MWyVNR6V40MFPU/BzN4FfIVgSuoSd/8nM7sVWObuj5jZ48DJwLbwJZvc/fKDvacGmuVYl8rk6OxPs6NnkLbuQXb0DNIZBkpnf5rewQw9gxm6BzJ7Wiu9KVKZA4cJBMuNTKhIDI2HVCfjVMSjlCeiVCdj1FUkmBAGSkVZlIpEjMpENAyhGFVlMaLDzdQaS+5BUOTSQchkBve0cNL94fZ0EB6p3iBwPLdnanAsGfwif30FtK8L9ssMBO+Tywb75jLB40zYPZYZgPRA8DhftCxodTTOD36pb31+z7TlkbBIEKC57P7vHewQBEc07GqrbAy65yobg23RsCsuWhaEcLQsuL7JtMVv6D9t0WcfFYpCQUpVfypLR3+KXb1pegYz9Aym6R7IBGHSl6ajP82uMEDaewfpHczSl8rQl8rSM5hhJP+rVySiVIUBMdRiKYtRnYxTHbZMdt+vKotRHo+SDIOnMhFsq0rGqEhEKYtFDjyr62i0O5AyA0EgJev2TB+GYKyndVUwVpPN7FmSxcIWSC69Z5wn1bNnbMcsmFE2YTaU1wXjRV3bggkHuwMr3RdOQmgNuu+y6SC8snnhmEvDZV+G5uvf0OEdDbOPRGQUlSeilCfKmVJbftivzeWcroFgnKRnMDMUGL2pLL2DGXrDlslQK2UwQ1+439aOAboHu+keyNDVnz7gGMq+ohHbK2SqkntCJBmPUJkIAqYqGbRYkvEo5fE9rZqgtRMb2j8Ziw5/zsloMQv+Ij/QelrRWHC52mLJ5YDC/xGvUBApAZGIUVcRDGgfCXenP52leyAIkYF0loF0lv50NgyULD0DaXpTYegMZsMQCsKmP5WlayDNQDo3FET5y50cSnk86O5KxiNEI0bEjHjUwtCJU5mIEo9GiEcjJGKRvO6x6F6tmt0zxWqSe3eZJWIRysN9EtGjrKUzRuMPCgURGTEzoyIRoyIRY1LN6LxnNudDwTIQBs6u3hQ7+1L0DmYYSOcYSGfpC4OmZzDLYDpLzp2sQzqTozcVtGK2dfSTzuZIZ53BTG6o++yNiEaMyrClU56IUhYLWizxaISIGWYQj0aCVkw8SjJ8viwMn6qyIJAqwtcmwqCKRYxYNEIsapTFIkPvmwjvl8UiJKKRwraKDkKhICJFFY3Y0GB3IeRyTt/uFk0qCJfugWBQv3sgQy4cbHGHVDZHfyoIqN0tnfwW0UAmSybrZHNO1n1PaGXC58MAGzzEpICRCEIiCJyy8P5fXnA8f3jq1CN+74NRKIjIuBaJ2NC4xljJ5nyoldI7mCGVzZHKBLdMLgiVVDbHYDrHYCYb/Az3GUhng5+7t2fCfTI56ioKf9a5QkFEZJRFIxbO0jr2lg45+s6cEBGRolEoiIjIEIWCiIgMUSiIiMgQhYKIiAxRKIiIyBCFgoiIDFEoiIjIkGNu6WwzawM2vsGXNwI7RrGcY0UpHncpHjOU5nGX4jHD4R/3LHdvOtROx1woHAkzWzaS9cTHm1I87lI8ZijN4y7FY4bCHbe6j0REZIhCQUREhpRaKNxT7AKKpBSPuxSPGUrzuEvxmKFAx11SYwoiInJwpdZSEBGRg1AoiIjIkJIJBTO72MzWmNlaM7ul2PUUgpnNMLMnzGyVma00s0+G2+vN7Jdm9mr4c0Kxax1tZhY1sxfM7Cfh4zlm9rvw+/6+mR3ZFeuPQmZWZ2YPmtnLZrbazN5WIt/1/wr/fa8ws/vMLDnevm8zW2JmrWa2Im/bsN+tBb4aHvuLZrb4SD67JELBzKLA3cAlwALgajNbUNyqCiID/JW7LwDeCnw8PM5bgF+5+3zgV+Hj8eaTwOq8x7cDX3b3ecAu4IaiVFVYdwI/d/cTgVMJjn9cf9dmNg24CWh29zcBUeAqxt/3/e/AxftsO9B3ewkwP7zdCHz9SD64JEIBOB1Y6+7r3T0F3A+8u8g1jTp33+buz4f3uwl+SUwjONZ7w93uBf6oOBUWhplNBy4FvhU+NuB84MFwl/F4zLXA2cC/Abh7yt07GOffdSgGlJtZDKgAtjHOvm93/w2wc5/NB/pu3w18xwPPAHVmNuWNfnaphMI0YHPe45Zw27hlZrOB04DfAZPcfVv41HZgUpHKKpSvAH8L5MLHDUCHu2fCx+Px+54DtAHfDrvNvmVmlYzz79rdtwD/AmwiCINO4DnG//cNB/5uR/X3W6mEQkkxsyrgIeAv3b0r/zkP5iCPm3nIZnYZ0OruzxW7ljEWAxYDX3f304Be9ukqGm/fNUDYj/5uglCcClSyfzfLuFfI77ZUQmELMCPv8fRw27hjZnGCQPgPd/9BuPn13c3J8GdrseorgDOBy83sNYJuwfMJ+trrwu4FGJ/fdwvQ4u6/Cx8/SBAS4/m7BrgA2ODube6eBn5A8G9gvH/fcODvdlR/v5VKKCwF5oczFBIEA1OPFLmmURf2pf8bsNrd78h76hHg2vD+tcCPxrq2QnH3T7v7dHefTfC9/pe7vx94Argi3G1cHTOAu28HNpvZCeGmdwCrGMffdWgT8FYzqwj/ve8+7nH9fYcO9N0+AnwwnIX0VqAzr5vpsJXMGc1m9i6CvucosMTd/6nIJY06MzsL+C3wEnv61/+OYFzhAWAmwbLj73X3fQexjnlmdi7w1+5+mZkdR9ByqAdeAK5x98Fi1jfazGwRweB6AlgPXEfwh964/q7N7PPAlQSz7V4APkzQhz5uvm8zuw84l2B57NeBzwI/ZJjvNgzHuwi60fqA69x92Rv+7FIJBRERObRS6T4SEZERUCiIiMgQhYKIiAxRKIiIyBCFgoiIDFEoiOzDzLJmtjzvNmqLypnZ7PyVL0WONrFD7yJScvrdfVGxixApBrUUREbIzF4zs382s5fM7Fkzmxdun21m/xWuZf8rM5sZbp9kZg+b2e/D2xnhW0XN7JvhNQF+YWblRTsokX0oFET2V75P99GVec91uvvJBGeQfiXc9jXgXnc/BfgP4Kvh9q8C/+3upxKsS7Qy3D4fuNvdFwIdwHsKfDwiI6YzmkX2YWY97l41zPbXgPPdfX248OB2d28wsx3AFHdPh9u3uXujmbUB0/OXWwiXNP9leKEUzOxTQNzdv1D4IxM5NLUURA6PH+D+4chfkyeLxvbkKKJQEDk8V+b9/J/w/tMEK7QCvJ9gUUIILpn4MRi6hnTtWBUp8kbpLxSR/ZWb2fK8xz93993TUieY2YsEf+1fHW77C4IroP0NwdXQrgu3fxK4x8xuIGgRfIzgamEiRy2NKYiMUDim0OzuO4pdi0ihqPtIRESGqKUgIiJD1FIQEZEhCgURERmiUBARkSEKBRERGaJQEBGRIf8fWIUij6YFxi4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 绘制训练 & 验证的损失值\n",
    "plt.plot(history.history['loss'])\n",
    "plt.plot(history.history['val_loss'])\n",
    "plt.title('Model loss')\n",
    "plt.ylabel('Loss')\n",
    "plt.xlabel('Epoch')\n",
    "plt.legend(['Train', 'Test'], loc='upper left')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在上面的训练中，我们使用预训练模型的权重，并保留了卷积网络结构中的参数权重，只训练分类层的参数。可以看到，由于预训练模型已经具备了比较好的特征提取能力，初始时就能获得比较高的准确率。\n",
    "\n",
    "接下来，我们将卷积网络中的参数也一起加入训练。在预训练模型的特征提取能力的基础上开始训练，找到更适合猫狗二分类任务的参数权重："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'keras.engine.training.Model'>\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_5 (InputLayer)         (None, None, None, 3)     0         \n",
      "_________________________________________________________________\n",
      "block1_conv1 (Conv2D)        (None, None, None, 64)    1792      \n",
      "_________________________________________________________________\n",
      "block1_conv2 (Conv2D)        (None, None, None, 64)    36928     \n",
      "_________________________________________________________________\n",
      "block1_pool (MaxPooling2D)   (None, None, None, 64)    0         \n",
      "_________________________________________________________________\n",
      "block2_conv1 (Conv2D)        (None, None, None, 128)   73856     \n",
      "_________________________________________________________________\n",
      "block2_conv2 (Conv2D)        (None, None, None, 128)   147584    \n",
      "_________________________________________________________________\n",
      "block2_pool (MaxPooling2D)   (None, None, None, 128)   0         \n",
      "_________________________________________________________________\n",
      "block3_conv1 (Conv2D)        (None, None, None, 256)   295168    \n",
      "_________________________________________________________________\n",
      "block3_conv2 (Conv2D)        (None, None, None, 256)   590080    \n",
      "_________________________________________________________________\n",
      "block3_conv3 (Conv2D)        (None, None, None, 256)   590080    \n",
      "_________________________________________________________________\n",
      "block3_pool (MaxPooling2D)   (None, None, None, 256)   0         \n",
      "_________________________________________________________________\n",
      "block4_conv1 (Conv2D)        (None, None, None, 512)   1180160   \n",
      "_________________________________________________________________\n",
      "block4_conv2 (Conv2D)        (None, None, None, 512)   2359808   \n",
      "_________________________________________________________________\n",
      "block4_conv3 (Conv2D)        (None, None, None, 512)   2359808   \n",
      "_________________________________________________________________\n",
      "block4_pool (MaxPooling2D)   (None, None, None, 512)   0         \n",
      "_________________________________________________________________\n",
      "block5_conv1 (Conv2D)        (None, None, None, 512)   2359808   \n",
      "_________________________________________________________________\n",
      "block5_conv2 (Conv2D)        (None, None, None, 512)   2359808   \n",
      "_________________________________________________________________\n",
      "block5_conv3 (Conv2D)        (None, None, None, 512)   2359808   \n",
      "_________________________________________________________________\n",
      "block5_pool (MaxPooling2D)   (None, None, None, 512)   0         \n",
      "_________________________________________________________________\n",
      "global_average_pooling2d_5 ( (None, 512)               0         \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 2)                 1026      \n",
      "=================================================================\n",
      "Total params: 14,715,714\n",
      "Trainable params: 14,715,714\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "base_model = VGG16(weights='imagenet', include_top=False)\n",
    "model = build_model(base_model)\n",
    "model.compile(loss='binary_crossentropy',\n",
    "              optimizer=opt,\n",
    "              metrics=['accuracy'])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到，这次模型的Non-trainable params个数为0，所有参数都参与训练，相当于我们使用预训练的参数权重初始化了卷积部分。\n",
    "\n",
    "### 训练所有参数\n",
    "\n",
    "直接在预训练权重的基础上进行训练，可以更快的让泛化性较好的权重收敛到适合猫狗二分类的权重，我们仅训练3轮就达到了约95%的精度，耗时约5分钟。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 18750 samples, validate on 6250 samples\n",
      "Epoch 1/5\n",
      "18750/18750 [==============================] - 103s 5ms/step - loss: 0.4086 - acc: 0.8029 - val_loss: 0.2317 - val_acc: 0.8989\n",
      "\n",
      "Epoch 00001: val_acc did not improve from 0.95120\n",
      "Epoch 2/5\n",
      "18750/18750 [==============================] - 102s 5ms/step - loss: 0.2157 - acc: 0.9247 - val_loss: 0.2462 - val_acc: 0.9019\n",
      "\n",
      "Epoch 00002: val_acc did not improve from 0.95120\n",
      "Epoch 3/5\n",
      "18750/18750 [==============================] - 101s 5ms/step - loss: 0.1959 - acc: 0.9379 - val_loss: 0.1228 - val_acc: 0.9555\n",
      "\n",
      "Epoch 00003: val_acc improved from 0.95120 to 0.95552, saving model to ./ckpt_vgg16_dog_and_cat.h5\n",
      "Epoch 4/5\n",
      "18750/18750 [==============================] - 101s 5ms/step - loss: 0.1849 - acc: 0.9386 - val_loss: 0.3406 - val_acc: 0.9373\n",
      "\n",
      "Epoch 00004: val_acc did not improve from 0.95552\n",
      "Epoch 5/5\n",
      "18750/18750 [==============================] - 101s 5ms/step - loss: 0.2295 - acc: 0.9324 - val_loss: 0.1599 - val_acc: 0.9496\n",
      "\n",
      "Epoch 00005: val_acc did not improve from 0.95552\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(x=x_train, \n",
    "                    y=y_train, \n",
    "                    batch_size=16, \n",
    "                    epochs=5, \n",
    "                    verbose=1, \n",
    "                    callbacks=callbacks, \n",
    "                    validation_split=0.25, \n",
    "                    shuffle=True, \n",
    "                    initial_epoch=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8988799999618531\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8VdW1wPHfyhzIBAlzSIKAMogIxHlgckSrdaqiWFEsdWxfW19r3/NZq52n11ZQi5WqaEXr9KhiaRVwtsyggGiYwximhAAZ73p/7BO4hEBuQs49Gdb38zmfnHvGdS/cs+7e+5y9RVUxxhhjjiUm6ACMMcY0f5YsjDHG1MuShTHGmHpZsjDGGFMvSxbGGGPqZcnCGGNMvSxZmDZPRPJEREUkLoJtx4vIB9GIy5jmxJKFaVFEZJ2IVIhIVq3li70Lfl4wkRnTulmyMC3RWmBszQsRGQS0Cy6c5iGSkpExjWXJwrRE04Cvh72+BXg2fAMRSReRZ0WkSETWi8gDIhLjrYsVkd+IyA4RWQNcVse+T4nIFhHZJCI/EZHYSAITkb+JyFYRKRaR90RkYNi6ZBH5rRdPsYh8ICLJ3rpzReQjEdkjIhtFZLy3fK6I3B52jMOqwbzS1N0i8iXwpbfsD94xSkRkoYicF7Z9rIj8l4isFpG93vqeIjJZRH5b673MEJHvRPK+TetnycK0RJ8AaSLS37uI3wA8V2ubR4F04ARgOC653Oqt+wZwOTAEyAeurbXv00AV0Mfb5iLgdiLzFtAX6AwsAp4PW/cbYBhwNtAR+D4QEpFcb79HgU7AqcCSCM8H8FXgDGCA93q+d4yOwF+Bv4lIkrfuu7hS2RggDbgN2A88A4wNS6hZwAXe/saAqtpkU4uZgHW4i9gDwM+BS4B/AXGAAnlALFABDAjb75vAXG9+NnBH2LqLvH3jgC5AOZActn4sMMebHw98EGGsGd5x03E/zA4Ag+vY7ofAa0c5xlzg9rDXh53fO/6oeuLYXXNeYBVw5VG2Wwlc6M3fA8wM+t/bpuYzWR2naammAe8BvahVBQVkAfHA+rBl64Ee3nx3YGOtdTVyvX23iEjNspha29fJK+X8FLgOV0IIhcWTCCQBq+vYtedRlkfqsNhE5D5gAu59Kq4EUXNDwLHO9QwwDpd8xwF/OI6YTCtj1VCmRVLV9biG7jHAq7VW7wAqcRf+GjnAJm9+C+6iGb6uxkZcySJLVTO8KU1VB1K/G4ErcSWfdFwpB0C8mMqA3nXst/EoywH2cXjjfdc6tjnYdbTXPvF94GtAB1XNAIq9GOo713PAlSIyGOgPvH6U7UwbZMnCtGQTcFUw+8IXqmo18BLwUxFJ9doEvsuhdo2XgG+JSLaIdADuD9t3C/BP4LcikiYiMSLSW0SGRxBPKi7R7MRd4H8WdtwQMBX4nYh09xqazxKRRFy7xgUi8jURiRORTBE51dt1CXC1iLQTkT7ee64vhiqgCIgTkQdxJYsafwYeEZG+4pwiIplejIW49o5pwCuqeiCC92zaCEsWpsVS1dWquuAoq+/F/SpfA3yAa6id6q17EpgFLMU1QtcumXwdSABW4Or7Xwa6RRDSs7gqrU3evp/UWn8f8CnugrwL+CUQo6obcCWk73nLlwCDvX3+F9f+sg1XTfQ8xzYL+AfwhRdLGYdXU/0Olyz/CZQATwHJYeufAQbhEoYxB4mqDX5kjHFE5HxcCSxX7eJgwljJwhgDgIjEA98G/myJwtRmycIYg4j0B/bgqtt+H3A4phmyaihjjDH1spKFMcaYerWah/KysrI0Ly8v6DCMMaZFWbhw4Q5V7VTfdr4mCxG5BPcUaCyu0ewXtdbn4m5n7IS7ZXCcd683IpKDuye8J+6hozGquu5o58rLy2PBgqPdRWmMMaYuIrK+/q18rIbyuj6YDFyK6+BsrIgMqLXZb4BnVfUU4GFcXz81ngV+rar9gdOB7X7Faowx5tj8bLM4HShQ1TWqWgFMx3WFEG4ArlM3gDk1672kEqeq/wJQ1VJV3e9jrMYYY47Bz2TRg8OfHC3kUEduNZYCV3vzVwGpXtcDJwJ7RORVbwS0X9c1noCITBSRBSKyoKioyIe3YIwxBoJv4L4PmOQN9PIerpuEalxc5+HGEtgAvIjrmvmp8J1VdQowBSA/P/+Ie4ArKyspLCykrKzMv3fQzCQlJZGdnU18fHzQoRhjWhE/k8UmDu/ZM5tDvX4CoKqb8UoWIpICXKOqe0SkEFiiqmu8da8DZ1IrWdSnsLCQ1NRU8vLyCOtuutVSVXbu3ElhYSG9evUKOhxjTCviZzXUfKCviPQSkQTcaGYzwjcQkayakblwA8BMDds3Q0RqbucaheuYrUHKysrIzMxsE4kCQETIzMxsUyUpY0x0+JYsVLUKN9rWLNwIXC+p6nIReVhErvA2GwGsEpEvcCOU/dTbtxpXRfWOiHyK64v/ycbE0VYSRY229n6NMdHha5uFqs4EZtZa9mDY/Mu47p/r2vdfwCl+xmdMVHw+Eyr3Q98LISk96GiMaZSgG7hbtZ07dzJ69GgAtm7dSmxsLJ06uZq1efPmkZCQUO8xbr31Vu6//35OOukkX2M1PgiF4O0H4aNH3euYeOh1HvS7DE4aA2ndg43PmAawZOGjzMxMlixZAsBDDz1ESkoK991332Hb1AyGHhNTd43gX/7yF9/jND6oroQZ34Klf4XTvgGDroXP33TTm99zU/eh0G8M9LscOvUDq0I0zZh1JBiAgoICBgwYwE033cTAgQPZsmULEydOJD8/n4EDB/Lwww8f3Pbcc89lyZIlVFVVkZGRwf3338/gwYM566yz2L7dHmpvlir2w/SbXKIY+d8w5teQcyZc9AjcuxDungejHwSJgdk/gcfOhD8OgVn/Des/glB10O/AmCO0mZLFj/++nBWbS5r0mAO6p/Gjrwxs1L6ff/45zz77LPn5+QD84he/oGPHjlRVVTFy5EiuvfZaBgw4vHeU4uJihg8fzi9+8Qu++93vMnXqVO6///66Dm+CcmA3/PV62DgPLvsdnFZryGwR6HSSm877HpRsgS/eciWOeVPg40nQLgtOvMRVV50wAhLaBfFOjDlMm0kWzU3v3r0PJgqAF154gaeeeoqqqio2b97MihUrjkgWycnJXHrppQAMGzaM999/P6oxm3qUbIZpV8Ou1XDd0zDwq/Xvk9YN8m9zU1kJrH7HJY6Vf4clz0FcMvQZ7do4TrwE2mf6/jaMqUubSRaNLQH4pX379gfnv/zyS/7whz8wb948MjIyGDduXJ3PSoQ3iMfGxlJVVRWVWE0EdnwJ066CA3vgppfhhOENP0ZSGgy8yk1VFbD+Q5c4Vs2Ez99w1VY5Zx1qIO9oD16a6LE2i2agpKSE1NRU0tLS2LJlC7NmzQo6JNMQmxbC1IuhqgzGv9G4RFFbXAL0HgmX/Qa+sxwmzoXz7nPJaNZ/wR9PhcfOdm0emxeDjXhpfNZmShbN2dChQxkwYAD9+vUjNzeXc845J+iQTKRWz4bp46B9Ftz8GmT2bvpziED3IW4a9d+wa61X2pgJ7/8W3vs1pPVwpY1+YyD3XJdsTOukCsWFsHmR+6GyaZH7/3fd076ettWMwZ2fn6+1Bz9auXIl/fv3Dyii4LTV9x11n70Cr37TNVaPewVSu0Y/hn074ctZrrqq4B2oOgCJ6e4BwH6XQZ8LXPWWabn27/ISQ820EPZ5d0LGJkCXk6H3KBj9P406vIgsVNX8+razkoUxjTHvSZj5n64NYewLkJwRTBztM+HUG91UsR/WzIVVb8Kqt+Czl92DgCcMd6WOk8a4BnXTfFXsh63LDiWFTQth91pvpUDWie4HQI+hbupyMsQlRiU0SxbGNIQqzP05vPtLOOkyuPYpiE8OOionoZ33kN8Y96zGxnmuYfzzN+HN77qpxzCvgfwyVyKyBwGDU10FRZ8fSgqbF8G2FaDeczZpPVxCGHaLe4Cz+6mBdhdjycKYSIWqYeZ9sGAqDBkHl/8BYpvpVygmFnLPctNFP3EXpZonyN952E0dex96gjz7NLeP8Ycq7F4XVp20ELYsdX2GgUsCPYbBud9xf3sMDaZa8xia6f90Y5qZqnJ49Ruw4v/cF3r0j1rOr3IR6NzfTeff554HqWkg/+QJ13dVuyw46dJDDwI2l9JSS1VadHgD9KaFcGCXWxebCN0Gw9BbvOqkYdDxhGb//8mShTH1KSuB6TfCuvfh4p/BWXcHHdHxSesOp93uprJiKHjblThW/B8sngbx7VyDab/L4cSLoV3HoCNu3spLYcuSsHaGRVC8wa2TGOjU35XgegxzU+cBENvyRrK0ZGHMsZRuh+euge0r4KopMPj6oCNqWknpcPI1bqqqcAmxptTx+RsgsZB79qHbcjvkBR1xsKorYdvyQ20Mmxa5Kj4NufUZOZA9DM6Y6NoZug2GxJRgY24ilix81BRdlANMnTqVMWPG0LVr86rDbPV2r3NPZZdsgbHT3e2orVlcgutapM9oGPMb97BfzRPks37opi4nH3qCvNvgZl91clxCIdi1Jqw6aSFs/dQ9fAnQLtMlhP5XHGpnaJ8VbMw+smTho0i6KI/E1KlTGTp0qCWLaNr6GTx3tWuruGUG9Dw96IiiS+TQ7Zmj/8ddND+f6ZLHe792d4OlZXsN5JdB7jktsmrlMHu3Ht7GsHmRq6YDVzXX7VRXdVfTzpCR27qTZS2+JgsRuQT4AxAL/FlVf1FrfS5u3O1OwC5gnKoWhq1Pw429/bqq3uNnrNH2zDPPMHnyZCoqKjj77LOZNGkSoVCIW2+9lSVLlqCqTJw4kS5durBkyRKuv/56kpOTG1QiMY207kN4YayrPrhtBnTuF3REwet4Apx9j5v27YAv/uGSx6JprrfcpHToe7FLHn0ugMTUoCM+trJi2Lwk7LbVxVCyya2TWOgywPXRVdPOkHVS873zLUp8e/ciEgtMBi4ECoH5IjJDVVeEbfYb4FlVfUZERgE/B24OW/8I8F6TBPTW/a4I2ZS6DoJLf1H/drV89tlnvPbaa3z00UfExcUxceJEpk+fTu/evdmxYweffuri3LNnDxkZGTz66KNMmjSJU089tWnjN0f6/E34263QIRfGvQoZPYOOqPlpn+VuHR4yznsQcI5XXfUWfPqSe6q413CvuurS4G8BrSp3JcXw6qQdXxxa3/EE1y7T3SsxdB1k3cLXwc9UeTpQoKprAERkOnAlrqRQYwDwXW9+DvB6zQoRGQZ0Af4B1Psoekvy9ttvM3/+/INdlB84cICePXty8cUXs2rVKr71rW9x2WWXcdFFFwUcaRuzaBr8/VuuD6Yb/2bdgUcioZ1LCv0ucw+Zbfz3oV5y3/gPN2Wf5jWQXw6dTvQ3nlAIdn55KClsWuR+JIYq3fr2nV1CGPQ1V53UfYjd7RUhP5NFD2Bj2OtC4Ixa2ywFrsZVVV0FpIpIJrAb+C0wDrjgaCcQkYnARICcnJxjR9OIEoBfVJXbbruNRx555Ih1y5Yt46233mLy5Mm88sorTJkyJYAI2xhV+PD38PZD0Hs0fO3ZVnMHS1TFxkHeOW666CewfaX3IOAb8M6P3ZTZ59AT5NmnwVGGE46Iqqs6Cu8aY/MSqNjr1iekuqeez7rrUHVSWo821c7QlIKuhLsPmCQi43HVTZuAauAuYKaqFsox/mFVdQowBVxHgr5H20QuuOACrr32Wr797W+TlZXFzp072bdvH8nJySQlJXHdddfRt29fbr/9dgBSU1PZu3dvwFG3UqEQ/PMB+GQynHwtfPXxVtVjq6pSHVKqVQmFoNp7HTq4TMOWha2v2S9s3v3l8PU1xwhff3BZGtXtbyA05HoST9pC921zyN42h24fTSbmwz+wPz6TtZnnsbrjcNaln0alJBw8ZkpCHF3Sk+ialkTX9CS6pCWRpnuRzYtdcqipUird5t5oTDx0Pdnd2txjmKtSyuprT6U3IT+TxSYgvMI321t2kKpuxpUsEJEU4BpV3SMiZwHnichdQAqQICKlqtoqxhAdNGgQP/rRj7jgggsIhULEx8fzxBNPEBsby4QJE1BVRIRf/vKXANx6663cfvvt1sDd1Kor4f/uhmUvwhl3wMU/P75fumFUlcLdB1i4fjdLNu5h176KOi6sSrVycFn4xfvQRZyjXNgPra8OhbxjcsRxmlen0oOAQaSxjxExS7mwegEjtsxi4NbX2aeJvBc6hXf0NN5lKCVVcQyUdQyOWc3gmNXEy2rSY7YdPNK2hBy2pw1lX4/BhLoPJTlnMF06pNMpNZH4WBumxw++dVEuInHAF8BoXJKYD9yoqsvDtskCdqlqSER+ClSr6oO1jjMeyK/vbijrovyQtvq+G6RiH7x0CxT8C0b9jxsP+ziqJ8oqq/l0UzGL1u9m4frdLNqwhx2l5QC0S4ilc2oiMTFCrAixMUJMzd8YIVY4bNlh60WIjam1XuTgsWJivPVhy2qOG1fnManzPOHHqus8h2IVYrzzhS87PH5qxR82HTyPNx8qJ279B8R88RayaiaUbgWJRUWQkBsJ8kBSF7amDmRNwkksl94sKM9jTWks20vKqagOHfbvIAJZKYl0SUuka5orkXRNSzqylJIUx7FqLdqSwLsoV9UqEbkHmIW7dXaqqi4XkYeBBao6AxgB/FxEFFcN1cL7UTAtwv5d8NevuWqMr/zR9erZQJv3HGDRhkOJYcXmYiqr3Q+vvMx2nN83iyG5HRiW04GTuqYSG2MXprq1g5MuctNlv3W3sH7xFqKhg9VJyWnd6AX0wv3yrKGq7NpXwdaSMraVlLG1uNzNF5extaTsYMlu9/7KI86aHB/rJY7Ew5NJ2LyVUg5ngx+1Qm31fUekeJN72G7XWte9eP+v1LtLRVWI5ZuLWbh+N4s37GHRht1sKXZP8SbFx3BKdgZDczowLLcDQ3IyyEqJzvgCJjJlldVsL3GJJDyZhM8fq5RysISSHj5/KLGkJrbsUkrgJYvmoqb+v61oLcnfF0VfuO47ykvg5lch79w6N9teUsaiDa7EsGj9bpZtKqaiyl1IemQkk5/XkWE5GQzN7UD/bmn267OZS4qPJSezHTmZR392ov5Syn4WrN/FnjpKKe0SYumalkTnY5RSOqcmEtfC/5+06mSRlJTEzp07yczMbBMJQ1XZuXMnSUlJQYfS/BQugOevg5g4GP8mdDsFgMrqEJ9v2eslB1etVLj7AAAJsTGc3CONW87KZWhOB4bmdqBLmn22rZGIkJmSSGZKIgO7H32AobLKai+ZlB2WWLZ5JZX563azfW/ZwSrJQ8dv+aWUVl0NVVlZSWFhIWVlZQFFFX1JSUlkZ2cTH9/C++lpSgVvw4s3Q0pn9lzzEgtKMg4mhmWFxRyodCOTdUlLDKtO6sDJPdJIjLNbL03DhELKrv0VbC12yWRbyeGllJrEcqxSSpewxviuaYmH5tOT6JTStKWUSKuhWnWyMG1bdUjZ+uE0us3+DpsT8/hWzAMs2uVuO46LEQZ0TztYYhiW24Hu6UnN9ledaX3qK6VsLS6rs5QSU1NKST90t9dJXVMZd2Zuo+KwNgvT5hTvr2Txxt0s8u5QGrjxr/xQnuaTUH9+UHE/J+Zl84PTXWIY1COd5AQrNZjgJMXHkpvZntzM9kfdpnYp5fAG+nI27NzPvLW7OGlr45NFpCxZmBYpFFLW7Ch1t66ud3cofbm9FIAYUX6WPoMb5EU2d7uAbl+dwtzOHa3UYFqcmBghKyWRrJRETu5x9LaUqlp3cvnBkoVpEUrLq1ji3ba6aIMrPZSUuYe20pPjGZqTwRWDuzOsZyqnrfgp8UtehKG30P3y/7UuH0yrF407rSxZmGZHVVm3c79XneQaor/YtpeQurtK+nZOYcygbgzN7cDQnA6ckNWemBiByjJ4ZYLruO68+2DUA9ZpnDFNxJKFCdyBimqWFu45WGJYtMH1pQSQmhjHqTkZXDywK0NzO3BqzwzSk+u406usGF64EdZ/AJf8Es68I8rvwpjWzZKFiaqaDvbCE8OKLSVUh9wdHydktWdUv84Hb2Ht0zml/q4y9m6D566BopVw9Z/hlOui8E6MaVssWRhflVVWH+wqY9H6PSzcsJuiva6DveT4WE7tmcEdw09gaI57tqFj+wb2qLtrjXsqu3Q73PiiG9LTGNPkLFmYJrW1uMzrXM+1NSwP62CvZ8dkzumdebCtoV/X1ONrmNuyzJUoQlVwy98hu1UNqGhMs2LJwjRaRVWIFVtKDjZEL1q/m81eB3sJcTEMzk7ntnN6MdTrYK9zahN2lbH2fZh+IySmwfg3oNNJTXdsY8wRLFmYBpv6wVre+mwLywqLKfc62OuensSQ3A7c7j0RPaBbGglxPt3Ot/Lv8PIE6JDnOgRMz/bnPMaYgyxZmAaZv24XD7+xgv7d0rjpjFyG5XZgaG4G3dKToxPAwmfgjf9wYx3c+BK06xid8xrTxlmyMA0yaXYBme0TeOXOs2iXEMX/Pqrw/m9h9iPQ50L42jOQcPRuEowxTatld7BuompZ4R7e/aKI287tFd1EEQrBP+53ieKU62HsC5YojIkyX5OFiFwiIqtEpEBE7q9jfa6IvCMiy0Rkrohke8tPFZGPRWS5t+56P+M0kZk8p4C0pDi+fpa/HZYdpqoCXv0G/PsJOPNu+OoTEGvdrxsTbb4lCxGJBSYDlwIDgLEiMqDWZr8BnlXVU4CHgZ97y/cDX1fVgcAlwO9FJMOvWE39Vm3dy6zl2xh/dh6pSVG6WJeXwgvXw2cvwwUPwcU/hRgrDBsTBD+/eacDBaq6RlUrgOnAlbW2GQDM9ubn1KxX1S9U9UtvfjOwHejkY6ymHo/NLaBdQiy3ntMrOifctxOevQLWzIUrJsG537F+nowJkJ/JogewMex1obcs3FLgam/+KiBVRDLDNxCR04EEYHXtE4jIRBFZICILioqKmixwc7h1O/bx96WbGXdmLh0a+oR1Y+zZCFMvhm3L4frnYOjN/p/TGHNMQZfp7wOGi8hiYDiwCaiuWSki3YBpwK2qekSH7ao6RVXzVTW/UycrePjl8bmriYuN4fbzolCq2P65SxSl22Hcq9DvMv/PaYypl5+3tGwCeoa9zvaWHeRVMV0NICIpwDWqusd7nQa8Cfy3qn7iY5zmGDbtOcAriwq58Yycpn0Cuy4b58Hz10FcItw6E7qe7O/5jDER87NkMR/oKyK9RCQBuAGYEb6BiGSJSE0MPwSmessTgNdwjd8v+xijqceUd13t3zeH9/b3RF/+C565wj1kN+GfliiMaWZ8SxaqWgXcA8wCVgIvqepyEXlYRK7wNhsBrBKRL4AuwE+95V8DzgfGi8gSbzrVr1hN3bbvLeOF+Ru5emgPemT4+IT20hfhhRsgqy/cNst142GMaVZ8fbJKVWcCM2stezBs/mXgiJKDqj4HPOdnbKZ+T72/lqrqEHeO6OPfST6eDLP+C/LOgxv+Cklp/p3LGNNo1t2HqdPufRVM+2Q9l5/SnV5ZPjwtrQrv/Bg++F/ofwVc/STE+9wmYoxpNEsWpk5/+Wgd+yuquXukD6WK6ip449uw+DkYditc9luIiW368xhjmowlC3OEvWWVPP3hWi4a0IWTuqY27cErD8DLt8GqmTD8BzDih/awnTEtgCULc4Rpn6ynpKyKe0Y1caniwB54YSxs+Bgu/TWcMbFpj2+M8Y0lC3OYAxXVPPX+Ws4/sROnZDdhd1x7t7ohUItWwbVPwcnXNN2xjTG+s2RhDvPCvA3s3FfBvU1Zqti5GqZdBft2wE0vQe9RTXdsY0xUWLIwB5VXVfOn91Zzeq+OnJbXRCPQbV7iShQojP+7G+HOGNPiBN03lGlGXlm4iW0l5U1XqljzLjx9OcQnu4ftLFEY02JZsjAAVFWHePzdAgZnp3Nun6zjP+Dy1+H5ayE923XfkdX3+I9pjAmMJQsDwIylm9m46wD3jOqLHO+trPOfgr+Nh+5DXIeAad2bJEZjTHCszcIQCimT5xTQr2sqo/t1bvyBVOHdX8Hcn0Hfi+G6pyGhXZPFaYwJjpUsDP9YvpXVRfu4e2QfYmIaWaoIhWDmf7pEMfhGuOF5SxTGtCJWsmjjVJVHZxdwQlZ7xgzq1riDVJXDa3fA8lfh7HvhwkfsqWxjWhlLFm3cnFXbWbmlhF9dewqxjSlVlO+FF8e5sbIvfBjO+XaTx2iMCZ4lizasplTRIyOZq4bUHh49Avt2uDuetiyDKx+DITc1fZDGmGbB2izasI9X72Txhj3cMfwE4mMb+F9hzwY3Vvb2la59whKFMa2alSzasElzCuiUmsh1+T3r37hGqBq2LYe/fg0q98PNr0PuWf4FaYxpFnxNFiJyCfAHIBb4s6r+otb6XNy4252AXcA4VS301t0CPOBt+hNVfcbPWFu1qgrXtlCxF8pLoaKULzdupsPaxfz2lAySFnxxxHrKa+b3hs2XugQBkNoNbn0LugwM9r0ZY6JCVNWfA4vEAl8AFwKFwHxgrKquCNvmb8AbqvqMiIwCblXVm0WkI7AAyAcUWAgMU9XdRztffn6+LliwwJf3EnWq7qJ82IV7rzdfxwX8iPWlUF5yaL66IrLzxiVDYgokpLi/iWmH5hNSIDHVm9JgwJWQ3oh2DmNMsyIiC1U1v77t/CxZnA4UqOoaL6DpwJXAirBtBgDf9ebnAK978xcD/1LVXd6+/wIuAV7wMd7jE6qudbGudQGv85d77fVhyUBDEZxU6riYp0C7XO9inxq2PvXg+nV7Y/jO66u59qx+3HT+wEPrY61W0hhTNz+vDj2AjWGvC4Ezam2zFLgaV1V1FZAqIplH2feIn7EiMhGYCJCTk9O4KEPVUFxYR9VL+AV8b631dVzsK/dFdr6YOO8CnnboIp+U5n6lJ6QeeeGvWXbwwh+WAOLbQ0zD71H41fMLKUhI4PILR0FyfIP3N8a0PUH/lLwPmCQi44H3gE1AdaQ7q+oUYAq4aqhGRbBvB/zhlGNvE5d05K/0lC6Q2fvw6pnDfuWn1V2lE5cY6ANrBdv38tZnW7lrRG/SLVEYYyLkZ7LYBITfZpPtLTtIVTfjShaISApwjaruEZFNwIiTNYfFAAAaM0lEQVRa+871JcrkDnDl5Dp+uYf9yo9tPRfVx+asJikultvO6RV0KMaYFsTPZDEf6CsivXBJ4gbgxvANRCQL2KWqIeCHuDujAGYBPxORDt7ri7z1TS8uAYaM8+XQzc2Gnfv5v6WbGX92HpkpiUGHY4xpQXx7KE9Vq4B7cBf+lcBLqrpcRB4WkSu8zUYAq0TkC6AL8FNv313AI7iEMx94uKax2zTe4++uJlaEieefEHQoxpgWxtc2C1WdCcystezBsPmXgZePsu9UDpU0zHHaUnyAlxdu5Gv5PemSlhR0OMaYFsa6+2gjpry3hpDCHcN7Bx2KMaYFqjdZiMi9YW0HpgXaUVrOC/M28NVTe9Czo40xYYxpuEhKFl2A+SLykohcIsc95qaJtqc+WEt5VYi7RlqpwhjTOPUmC1V9AOgLPAWMB74UkZ+JiF15WoDi/ZVM+3g9YwZ1o3enlKDDMca0UBG1WajrQGqrN1UBHYCXReRXPsZmmsDTH62jtLyKe0b2CToUY0wLVu/dUCLybeDrwA7gz8B/qmqliMQAXwLf9zdE01il5VVM/XAtF/TvTP9uaUGHY4xpwSK5dbYjcLWqrg9fqKohEbncn7BMU3j+k/UUH6jkbitVGGOOUyTVUG/hxpoAQETSROQMAFVd6Vdg5viUVVbz5PtrObdPFkNy7GY2Y8zxiSRZPA6Uhr0u9ZaZZuzF+RvZUVrOPaOsVGGMOX6RJAvRsBGSvH6cgu6t1hxDRVWIJ95dTX5uB87o1THocIwxrUAkyWKNiHxLROK96dvAGr8DM4332uJCthSXcc+oPthjMcaYphBJsrgDOBvXc2zNAEYT/QzKNF5VdYjH5q5mUI90hp/YKehwjDGtRL3VSaq6Hde9uGkB3vx0C+t37ueJccOsVGGMaTKRPGeRBEwABgIHuytV1dt8jMs0QiikTJpdwIldUrhoQJegwzHGtCKRVENNA7oCFwPv4kat2+tnUKZx/rliG19uL+XukX2IibFShTGm6USSLPqo6v8A+1T1GeAyXLuFaUZUlUlzviQ3sx2XDeoWdDjGmFYmkmRR6f3dIyInA+lAZ/9CMo3x7hdFfLaphLtG9CYu1oYpMcY0rUiuKlO88SweAGYAK4BfRnJwr0vzVSJSICL317E+R0TmiMhiEVkmImO85fEi8oyIfCoiK0XEn/G3WwlV11bRPT2Jq4ZkBx2OMaYVOmYDt9dZYImq7gbeAyIevFlEYoHJwIW4W27ni8gMVV0RttkDuLG5HxeRAbghWPOA64BEVR0kIu2AFSLygqqui/yttR3/XruLBet38+MrBpIQZ6UKY0zTO+aVxXtau7G9yp4OFKjqGlWtAKYDV9Y+BVDTHWo6sDlseXsRiQOSgQqgpJFxtHqTZheQlZLI9af1DDoUY0wrFcnP0LdF5D4R6SkiHWumCPbrAWwMe13oLQv3EDBORApxpYp7veUvA/uALcAG4DequqvWvojIRBFZICILioqKIgip9Vm8YTcfFOzgG+f1Iik+NuhwjDGtVCTJ4nrgblw11EJvWtBE5x8LPK2q2cAYYJpX9XU6UA10B3oB3xORI6rAVHWKquaran6nTm3zaeXJcwpIT47npjNzgw7FGNOKRfIEd69GHnsTEF4vku0tCzcBuMQ7z8feA4BZwI3AP1S1EtguIh8C+VifVIdZsbmEt1du5zsXnEhKovXtaIzxTyRPcH+9ruWq+mw9u84H+opIL1ySuAGXBMJtAEYDT4tIf9wT4kXe8lG4kkZ74Ezg9/XF2tZMnltASmIc48/OCzoUY0wrF8nP0dPC5pNwF/dFwDGThapWicg9wCwgFpiqqstF5GFggarOAL4HPCki38E1ao9XVRWRycBfRGQ5IMBfVHVZQ99ca7a6qJSZn27hjuG9SW8XH3Q4xphWLpJqqHvDX4tIBu7Opnqp6kxcw3X4sgfD5lcA59SxXynu9llzFI/PXU1iXAwTzm1sLaExxkSuMTfl78M1OpuAbNy1n9cWb2Ls6TlkpSQGHY4xpg2IpM3i77gqInDJZQDwkp9BmWP703uriRGYeH7Ez0gaY8xxiaTN4jdh81XAelUt9CkeU49tJWW8NL+Qa4f1pFt6ctDhGGPaiEiSxQZgi6qWAYhIsojkWdcbwXjyvTVUq3Ln8N5Bh2KMaUMiabP4GxAKe13tLTNRtmtfBc//ewNXDu5OTma7oMMxxrQhkSSLOK9vJwC8+QT/QjJHM/WDtZRVVXPXSCtVGGOiK5JkUSQiV9S8EJErgR3+hWTqUnygkmc+WselJ3elT+fUoMMxxrQxkbRZ3AE8LyKTvNeFQJ1PdRv/TPt4HXvLq7hrRJ+gQzHGtEGRPJS3GjhTRFK816W+R2UOs6+8iqc+WMuofp05uUd60OEYY9qgequhRORnIpKhqqWqWioiHUTkJ9EIzjgvzNvA7v2V3D3SShXGmGBE0mZxqaruqXnhjZo3xr+QTLiyymr+9N4azjohk2G5HYIOxxjTRkWSLGJF5GCfEiKSDFgfE1Hyt4WFFO0t595RVqowxgQnkgbu54F3ROQvuB5gxwPP+BmUcSqrQzwxdzVDcjI4q3dm0OEYY9qwSBq4fykiS4ELcH1EzQJsWLYoeH3xJjbtOcAjXx2IiAQdjjGmDYu019ltuERxHW5QopW+RWQAqA4pj81dzYBuaYw8qXPQ4Rhj2rijlixE5ETcGNljcQ/hvQiIqo6MUmxt2sxPt7B2xz4eu2molSqMMYE7VjXU58D7wOWqWgDgjWhnfBYKKZPnFNC7U3suGdg16HCMMeaY1VBXA1uAOSLypIiMxjVwR0xELhGRVSJSICL317E+R0TmiMhiEVkmImPC1p0iIh+LyHIR+VREkhpy7pbsnc+38/nWvdw9sg8xMVaqMMYE76jJQlVfV9UbgH7AHOA/gM4i8riIXFTfgUUkFpgMXIobMGmsiAyotdkDwEuqOgS4AXjM2zcOeA64Q1UHAiOAyga+txZJVZk0+0t6dkzmisHdgw7HGGOACBq4VXWfqv5VVb8CZAOLgR9EcOzTgQJVXeP1VDsduLL24YE0bz4d2OzNXwQsU9WlXgw7VbU6gnO2eB8U7GBpYTF3Du9DXGxjRr01xpim16CrkaruVtUpqjo6gs17ABvDXhd6y8I9BIwTkUJgJnCvt/xEQEVklogsEpHv13UCEZkoIgtEZEFRUVFD3kqz9ejsArqmJXHNsNoflTHGBCfon65jgadVNRvXhcg0EYnBNbyfC9zk/b3KazM5jJe48lU1v1OnTtGM2xfz1u5i3tpdTDz/BBLjYoMOxxhjDvIzWWwCeoa9zvaWhZsAvASgqh8DSUAWrhTynqruUNX9uFLHUB9jbRYmzSkgs30CY0/PCToUY4w5jJ/JYj7QV0R6iUgCrgF7Rq1tNgCjAUSkPy5ZFOGeEh8kIu28xu7hwAofYw3cssI9vPdFERPO60VygpUqjDHNSyR9QzWKqlaJyD24C38sMFVVl4vIw8ACVZ0BfA940nt+Q4HxqqrAbhH5HS7hKDBTVd/0K9bmYNLsAtKS4rj5TOtJxRjT/PiWLABUdSauCil82YNh8yuAc46y73O422dbvVVb9/LPFdv41ui+pCbFBx2OMcYcIegGbgNMnlNAu4RYbj07L+hQjDGmTpYsArZ2xz7eWLaZm8/MpUP7hKDDMcaYOlmyCNjjcwuIi41hwnm9gg7FGGOOypJFgDbtOcCrizYx9rSedE5tM11fGWNaIEsWAfrTu6sBmDi8d8CRGGPMsVmyCMj2vWVMn7+Ra4Zm0yMjOehwjDHmmCxZBOTP76+lqjrEnSOsVGGMaf4sWQRg974KnvtkPV8Z3J28rPZBh2OMMfWyZBGAv3y4lv0V1dw1ok/QoRhjTEQsWUTZ3rJKnv5oHRcP7MJJXVODDscYYyJiySLKpn2ynpKyKu4Z2TfoUIwxJmKWLKLoQEU1T72/luEndmJQdnrQ4RhjTMQsWUTRC/M2sHNfBfeMsrYKY0zLYskiSsqrqvnTe6s5o1dHTsvrGHQ4xhjTIJYsouSVhZvYVlJupQpjTItkySIKqqpDPP5uAYN7ZnBun6ygwzHGmAazZBEFM5ZuZuOuA9wzsg8iEnQ4xhjTYL4mCxG5RERWiUiBiNxfx/ocEZkjIotFZJmIjKljfamI3OdnnH4KhZTJcwro1zWV0f06Bx2OMcY0im/JQkRigcnApcAAYKyIDKi12QPAS6o6BLgBeKzW+t8Bb/kVYzT8Y/lWVhft4+6RfYiJsVKFMaZl8rNkcTpQoKprVLUCmA5cWWsbBdK8+XRgc80KEfkqsBZY7mOMvlJVHp1dwAlZ7RkzqFvQ4RhjTKP5mSx6ABvDXhd6y8I9BIwTkUJgJnAvgIikAD8AfnysE4jIRBFZICILioqKmiruJjNn1XZWbinhzhG9ibVShTGmBQu6gXss8LSqZgNjgGkiEoNLIv+rqqXH2llVp6hqvqrmd+rUyf9oG6CmVNEjI5mvDqmdI40xpmWJ8/HYm4CeYa+zvWXhJgCXAKjqxyKSBGQBZwDXisivgAwgJCJlqjrJx3ib1Merd7J4wx4e+erJxMcGnZONMeb4+Jks5gN9RaQXLkncANxYa5sNwGjgaRHpDyQBRap6Xs0GIvIQUNqSEgXAo7ML6JyayHXDsoMOxRhjjptvP3lVtQq4B5gFrMTd9bRcRB4WkSu8zb4HfENElgIvAONVVf2KKVoWrt/Fx2t2MvH8E0iKjw06HGOMOW5+lixQ1Zm4huvwZQ+Gza8AzqnnGA/5EpyPJs0uoEO7eG48IyfoUIwxpklYZXoT+2xTMXNWFTHh3F60S/A1FxtjTNRYsmhik+cUkJoUx9fPzgs6FGOMaTKWLJrQl9v28o/lW7nlrDzSkuKDDscYY5qMJYsm9Njc1STFxXLbub2CDsUYY5qUJYsmsn7nPmYs3cxNZ+TQsX1C0OEYY0yTsmTRRJ54dzWxMcI3zj8h6FCMMabJWbJoAluKD/DywkK+lp9Nl7SkoMMxxpgmZ8miCfzp3TWEFL55fu+gQzHGGF9YsjhORXvLmT5/A1cN6UHPju2CDscYY3xhyeI4PfXBWsqrQtw5wkoVxpjWy5LFcdizv4JpH6/jskHd6N0pJehwjDHGN5YsjsPTH61jX0U1d4/sE3QoxhjjK0sWjVRaXsVfPlzHBf270L9bWv07GGNMC2bJopGe+2Q9xQcquWeUlSqMMa2fJYtGKKus5s/vr+G8vlmc2jMj6HCMMcZ3liwaYfq8DeworbC2CmNMm2HJooEqqkL86b01nJbXgTN6dQw6HGOMiQpfk4WIXCIiq0SkQETur2N9jojMEZHFIrJMRMZ4yy8UkYUi8qn3d5SfcTbEq4sK2VJcxt0j+yAiQYdjjDFR4dtQbiISC0wGLgQKgfkiMsMbSrXGA7ixuR8XkQG4IVjzgB3AV1R1s4icjBvHu4dfsUaqqjrE4++uZlCPdIaf2CnocIwxJmr8LFmcDhSo6hpVrQCmA1fW2kaBmvtO04HNAKq6WFU3e8uXA8kikuhjrBF5Y9kW1u/cb6UKY0yb42ey6AFsDHtdyJGlg4eAcSJSiCtV3FvHca4BFqlqee0VIjJRRBaIyIKioqKmifooQiFl8pwCTuySwkUDuvh6LmOMaW6CbuAeCzytqtnAGGCaiByMSUQGAr8EvlnXzqo6RVXzVTW/Uyd/q4X+uWIbX24v5e6RfYiJsVKFMaZt8TNZbAJ6hr3O9paFmwC8BKCqHwNJQBaAiGQDrwFfV9XVPsZZL1Vl0pwvyctsx2WDugUZijHGBMLPZDEf6CsivUQkAbgBmFFrmw3AaAAR6Y9LFkUikgG8Cdyvqh/6GGNE3v2iiM82lXDniN7ExQZdGDPGmOjz7cqnqlXAPbg7mVbi7npaLiIPi8gV3mbfA74hIkuBF4Dxqqrefn2AB0VkiTd19ivWY1FVJs0uoHt6ElcNyQ4iBGOMCZxvt84CqOpMXMN1+LIHw+ZXAOfUsd9PgJ/4GVuk/r12FwvW7+bHVwwkIc5KFcaYtsmufvWYNLuArJRErj+tZ/0bG2NMK2XJ4hgWb9jNBwU7+MZ5vUiKjw06HGOMCYwli2OYPKeAjHbx3HRmbtChGGNMoCxZHMWKzSW8vXI7t57di5REX5t2jDGm2bNkcRST5xaQkhjH+LPzgg7FGGMCZ8miDquLSpn56RZuPiuX9HbxQYdjjDGBs2RRh8fmrCYxLoYJ5/YKOhRjjGkWLFnUsnHXfl5fsomxp+eQlRJ4R7fGGNMsWLKo5Yl3VxMrwsTzTwg6FGOMaTYsWYTZVlLG3xYUcs2wbLqlJwcdjjHGNBuWLMJMeW8N1arcObx30KEYY0yzYsnCs7O0nL/+ewNXDu5OTma7oMMxxphmxZKFZ+qHaymrquaukVaqMMaY2ixZAMUHKnn2o/VcenJX+nRODTocY4xpdixZAM9+tI695VXcPbJP0KEYY0yz1OaTxb7yKqZ+uJZR/TozsHt60OEYY0yz1OZ7yCstr+Ks3plMONeeqzDGmKPxtWQhIpeIyCoRKRCR++tYnyMic0RksYgsE5ExYet+6O23SkQu9ivGLmlJPHbTMIbldvDrFMYY0+L5VrIQkVhgMnAhUAjMF5EZ3lCqNR7Ajc39uIgMwA3BmufN3wAMBLoDb4vIiapa7Ve8xhhjjs7PksXpQIGqrlHVCmA6cGWtbRRI8+bTgc3e/JXAdFUtV9W1QIF3PGOMMQHwM1n0ADaGvS70loV7CBgnIoW4UsW9DdgXEZkoIgtEZEFRUVFTxW2MMaaWoO+GGgs8rarZwBhgmohEHJOqTlHVfFXN79Spk29BGmNMW+fn3VCbgJ5hr7O9ZeEmAJcAqOrHIpIEZEW4rzHGmCjxs2QxH+grIr1EJAHXYD2j1jYbgNEAItIfSAKKvO1uEJFEEekF9AXm+RirMcaYY/CtZKGqVSJyDzALiAWmqupyEXkYWKCqM4DvAU+KyHdwjd3jVVWB5SLyErACqALutjuhjDEmOOKuzS1ffn6+LliwIOgwjDGmRRGRhaqaX+92rSVZiEgRsP44DpEF7GiicJqSxdUwFlfDWFwN0xrjylXVeu8QajXJ4niJyIJIsmu0WVwNY3E1jMXVMG05rqBvnTXGGNMCWLIwxhhTL0sWh0wJOoCjsLgaxuJqGIurYdpsXNZmYYwxpl5WsjDGGFMvSxbGGGPq1aaSRQSDMSWKyIve+n+LSF4ziWu8iBSJyBJvuj1KcU0Vke0i8tlR1ouI/NGLe5mIDG0mcY0QkeKwz+vBKMXV0xvMa4WILBeRb9exTdQ/swjjivpnJiJJIjJPRJZ6cf24jm2i/p2MMK5AvpPeuWO9AePeqGOdf5+XqraJCdflyGrgBCABWAoMqLXNXcAT3vwNwIvNJK7xwKQAPrPzgaHAZ0dZPwZ4CxDgTODfzSSuEcAbAXxe3YCh3nwq8EUd/5ZR/8wijCvqn5n3GaR48/HAv4Eza20TxHcykrgC+U565/4u8Ne6/r38/LzaUskiksGYrgSe8eZfBkaLiDSDuAKhqu8Bu46xyZXAs+p8AmSISLdmEFcgVHWLqi7y5vcCKzlyHJaof2YRxhV13mdQ6r2M96bad9xE/TsZYVyBEJFs4DLgz0fZxLfPqy0li0gGVDq4japWAcVAZjOIC+Aar9riZRHpWcf6IEQaexDO8qoR3hKRgdE+uVf8H4L7VRou0M/sGHFBAJ+ZV6WyBNgO/EtVj/p5RfE7GUlcEMx38vfA94HQUdb79nm1pWTRkv0dyFPVU4B/ceiXg6nbIlx/N4OBR4HXo3lyEUkBXgH+Q1VLonnuY6knrkA+M1WtVtVTcWPWnC4iJ0fjvPWJIK6ofydF5HJgu6ou9PtcdWlLySKSAZUObiMicbhxwXcGHZeq7lTVcu/ln4FhPscUqWY5SJWqltRUI6jqTCBeRLKicW4RicddkJ9X1Vfr2CSQz6y+uIL8zLxz7gHm4A2GFiaI72S9cQX0nTwHuEJE1uGqq0eJyHO1tvHt82pLySKSwZhmALd489cCs9VrKQoyrlp12lfg6pybgxnA1707fM4EilV1S9BBiUjXmnpaETkd9//c9wuMd86ngJWq+rujbBb1zyySuIL4zESkk4hkePPJwIXA57U2i/p3MpK4gvhOquoPVTVbVfNw14nZqjqu1ma+fV5+DqvarGhkgzE9hRsHvADXgHpDM4nrWyJyBW4gqF24OzF8JyIv4O6SyRKRQuBHuMY+VPUJYCbu7p4CYD9wazOJ61rgThGpAg4AN0Qh6YP75Xcz8KlX3w3wX0BOWGxBfGaRxBXEZ9YNeEZEYnHJ6SVVfSPo72SEcQXynaxLtD4v6+7DGGNMvdpSNZQxxphGsmRhjDGmXpYsjDHG1MuShTHGmHpZsjDGGFMvSxbGNICIVIf1NLpE6ugl+DiOnSdH6UnXmKC1mecsjGkiB7xuIIxpU6xkYUwTEJF1IvIrEfnUGwuhj7c8T0Rmex3OvSMiOd7yLiLymtdx31IROds7VKyIPCluHIV/ek8QGxM4SxbGNExyrWqo68PWFavqIGASrndQcJ3yPeN1OPc88Edv+R+Bd72O+4YCy73lfYHJqjoQ2ANc4/P7MSYi9gS3MQ0gIqWqmlLH8nXAKFVd43Xat1VVM0VkB9BNVSu95VtUNUtEioDssM7oaroP/5eq9vVe/wCIV9Wf+P/OjDk2K1kY03T0KPMNUR42X421K5pmwpKFMU3n+rC/H3vzH3GoM7ebgPe9+XeAO+HgQDvp0QrSmMawXy3GNExyWM+tAP9Q1ZrbZzuIyDJc6WCst+xe4C8i8p9AEYd6mf02MEVEJuBKEHcCgXfvbszRWJuFMU3Aa7PIV9UdQcdijB+sGsoYY0y9rGRhjDGmXlayMMYYUy9LFsYYY+plycIYY0y9LFkYY4yplyULY4wx9fp/C479fnTeYjwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "print(history.history['val_acc'][0])\n",
    "# 绘制训练 & 验证的准确率值\n",
    "plt.plot(history.history['acc'])\n",
    "plt.plot(history.history['val_acc'])\n",
    "plt.title('Model accuracy')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.legend(['Train', 'Test'], loc='upper left')\n",
    "plt.show();"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到，模型在开始训练后就得到了较高的精度。在VGG论文中，作者也提到了使用预训练好的浅层VGG模型去初始化深层VGG模型的一部分层，再继续训练的技巧。在上面的实践中，我们使用了泛化性更强的参数权重初始化模型，并针对我们特定的任务进行训练，可以更快的得到较好的精度。\n",
    "\n",
    "### 扩展\n",
    "\n",
    "尝试使用更深层的网络，如VGG19, ResNet在ImageNet数据集预训练的权重，并针对猫狗二分类任务进行迁移学习，查看模型训练的效果。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}