【官方示例】手写藏文MNIST数据集的图像分类.ipynb

{"cells":[{"source":"# 手写藏文数字数据集分类\n\n## 藏文数字\n\n\n![Image Name](https://cdn.kesci.com/upload/image/piy7a8zo9h.jpg?imageView2/0/w/320/h/320)\n\n\n藏文数字与阿拉伯数字一一对应，但写法不同[[1]](https://zh.wikipedia.org/wiki/%E8%97%8F%E6%96%87)。\n\n|藏文数字|阿拉伯数字|\n|---|---|\n|༠ | 0|\n|༡ | 1|\n|༢ | 2|\n|༣ | 3|\n|༤ | 4|\n|༥ | 5|\n|༦ | 6|\n|༧ | 7|\n|༨ | 8|\n|༩ | 9|\n\n\n## 关于Input数据的再处理\n由于**.npz** 数据是**有序**的图像及类别构成的，这对于图像分类训练来说是不利的，所以为了要保证**平衡样**本和**随机分布**的条件，我们需要做一些数据再处理。这在解决现实场景中图像相关问题的数据准备也是有帮助的。\n\n不过通常的做法是在读取图片列表的时候就完成这些步骤，不过如果当你发现你已经完成了图片到数组数据转化，而忘记了保证训练集与测试集的样本类别的分布情况，那么你可以参考下面👇的数据处理方法，相信会给你一些启发与帮助。\n\n\n## 使用两种模型来进行图像分类\n在这篇notebook中，我们会尝试两种模型来进行图像分类。在这里，FCNN模型的训练时间在2分钟以内，CNN模型的训练时间在20分钟以内，所以不用担心会等很久。\n- Fully Connected Neural Network 全连接神经网络\n- Convolutional Neural Network 卷积神经网络\n\n**Tips**\n\nK-Lab提供了查看项目目录的功能，更简洁明了的向用户展示项目的框架结构，点击目录章节即可跳转到对应的章节。（点击左侧的目录列表图标即可）\n\n![Image Name](https://cdn.kesci.com/upload/image/piy6i27du8.png?imageView2/0/w/1280/h/1280)\n","cell_type":"markdown","metadata":{"collapsed":false,"id":"4C95A42F650147858F372E6B62883E72","mdEditEnable":false}},{"metadata":{"id":"876C2FF1324C4BDBB56598D7B08C9064","collapsed":false,"scrolled":false},"cell_type":"code","outputs":[{"output_type":"stream","text":"TibetanMnist7208\r\n","name":"stdout"}],"source":"# 查看当前挂载的数据集目录\n!ls /home/kesci/input/","execution_count":1},{"metadata":{"id":"3C4B74F2CF074EC485ABA1AAFA602920","collapsed":false,"scrolled":false},"cell_type":"code","outputs":[],"source":"# 显示cell运行时长\n%load_ext klab-autotime","execution_count":2},{"metadata":{"id":"ECBE68426F154207911D7BD23CAEE04B","mdEditEnable":false},"cell_type":"markdown","source":"## 1.读取数据\n### 1.1.从.NPZ文件中读取"},{"metadata":{"id":"03B097D382944F7C8F04AD2E89E76EB8","collapsed":false,"scrolled":false},"cell_type":"code","outputs":[{"output_type":"stream","text":"time: 1.47 s\n","name":"stdout"},{"output_type":"stream","text":"Using TensorFlow backend.\n","name":"stderr"}],"source":"from tensorflow import keras\nfrom keras import utils\nimport matplotlib.pyplot as plt","execution_count":3},{"metadata":{"id":"10267287FC3B41338EB03EDDC99C845D","collapsed":false,"scrolled":false},"cell_type":"code","outputs":[{"output_type":"stream","text":"image数据： (17768, 28, 28)\nLabel数据： (17768,)\n","name":"stdout"}],"source":"import numpy as np\n\nimg_rows,img_cols = 28,28\n\ndata = np.load('/home/kesci/input/TibetanMnist7208/TibetanMNIST.npz')\n\nimage_data, label_data= data['image'], data['label']\n\nprint('image数据：',image_data.shape)\nprint('Label数据：',label_data.shape)","execution_count":2},{"metadata":{"id":"A9DF39CDBA1544069EE4ECFD8574608B","mdEditEnable":false},"cell_type":"markdown","source":"### 1.2. 查看图片数组的类别分布\n通过查看每个类别的所对应的下标，可以看到每个类别的图片是有序分布的，这对于我们后续拆分训练集与测试集训练模型是有弊端的。所以，我们需要让图像及对应类别样本**平衡**的**随机**分布，保证在训练集与测试集中涵盖了所有的类别数据，且每个类别的**数量相当**。"},{"metadata":{"id":"F49D7C156E364234BE84FD471D2C0358","collapsed":false,"scrolled":false,"hide_input":false},"cell_type":"code","outputs":[{"output_type":"stream","text":"类别： 0 数量： 1996 \n对应index: (array([   0,    1,    2, ..., 1993, 1994, 1995]),)\n类别： 1 数量： 2134 \n对应index: (array([1996, 1997, 1998, ..., 4127, 4128, 4129]),)\n类别： 2 数量： 1872 \n对应index: (array([4130, 4131, 4132, ..., 5999, 6000, 6001]),)\n类别： 3 数量： 1816 \n对应index: (array([6002, 6003, 6004, ..., 7815, 7816, 7817]),)\n类别： 4 数量： 1967 \n对应index: (array([7818, 7819, 7820, ..., 9782, 9783, 9784]),)\n类别： 5 数量： 1350 \n对应index: (array([ 9785,  9786,  9787, ..., 11132, 11133, 11134]),)\n类别： 6 数量： 1618 \n对应index: (array([11135, 11136, 11137, ..., 12750, 12751, 12752]),)\n类别： 7 数量： 1302 \n对应index: (array([12753, 12754, 12755, ..., 14052, 14053, 14054]),)\n类别： 8 数量： 1642 \n对应index: (array([14055, 14056, 14057, ..., 15694, 15695, 15696]),)\n类别： 9 数量： 2071 \n对应index: (array([15697, 15698, 15699, ..., 17765, 17766, 17767]),)\ntime: 10.9 ms\n","name":"stdout"}],"source":"idx_list = {}\nfor i in range(10):\n    idx_list[i] = np.where(label_data == i) # return tuple dtype (rows indices, column indices)\n\nfor label,idxs in idx_list.items():\n    print('类别：',label,'数量：',len(idxs[0]), '\\n对应index:',idxs)","execution_count":5},{"metadata":{"id":"585427C673A64F9D8EB12178D8FF64C4","mdEditEnable":false},"cell_type":"markdown","source":"### 1.3. 平衡样本 + 随机排序\n注意：选取的是原样本列表的下标index\n- **Step1：每个类别随机选出20%的样本作为测试集样本，其余的用于训练集样本。 `测试集:训练集 = 8 ：2`**\n- **Step2：将训练集与测试集分别shuffle(X与y以同样的order进行shuffle)**"},{"metadata":{"id":"8E16C2D34BB24204A47C93E53FA7D1AF","mdEditEnable":false},"cell_type":"markdown","source":"#### 1.3.1. Step1 每个类别随机选出20%的样本作为测试集样本，其余的用于训练集样本\n**测试集的下标选取(随机选取20%)**"},{"metadata":{"id":"11FD55A99D3948D68C9EA879C87B106A","collapsed":false,"scrolled":false,"hide_input":false},"cell_type":"code","outputs":[{"output_type":"stream","text":"time: 7.18 ms\n","name":"stdout"}],"source":"import random\n\nselected_test_sample_indices = {}\nfor label in range(10):\n    selected_test_sample_indices[label] = random.sample(set(idx_list[label][0]), int(len(idx_list[label][0])*0.2))\n    ","execution_count":6},{"metadata":{"id":"F67F8BF7D0A0481596901716EF6B76B7","collapsed":false,"scrolled":false},"cell_type":"code","outputs":[{"output_type":"stream","text":"类别 0 选取测试集样本量： 399 原本样本量： 1996\n类别 1 选取测试集样本量： 426 原本样本量： 2134\n类别 2 选取测试集样本量： 374 原本样本量： 1872\n类别 3 选取测试集样本量： 363 原本样本量： 1816\n类别 4 选取测试集样本量： 393 原本样本量： 1967\n类别 5 选取测试集样本量： 270 原本样本量： 1350\n类别 6 选取测试集样本量： 323 原本样本量： 1618\n类别 7 选取测试集样本量： 260 原本样本量： 1302\n类别 8 选取测试集样本量： 328 原本样本量： 1642\n类别 9 选取测试集样本量： 414 原本样本量： 2071\ntime: 7.77 ms\n","name":"stdout"}],"source":"for label in range(10):\n    # print(selected_test_sample_indices[label])\n    print('类别',label,'选取测试集样本量：',len(selected_test_sample_indices[label]),'原本样本量：',len(idx_list[label][0]))\n    ","execution_count":7},{"metadata":{"id":"FF85D06366A0438A8A11AF768F36CD7E","mdEditEnable":false},"cell_type":"markdown","source":"**训练集的下标选取（剩余80%）**"},{"metadata":{"id":"EB094D5E50784CE59736D6DD5EC2A9D3","collapsed":false,"scrolled":false},"cell_type":"code","outputs":[{"output_type":"stream","text":"time: 3.87 ms\n","name":"stdout"}],"source":"selected_train_sample_indicies = {}\n\nfor label in range(10):\n    selected_train_sample_indicies[label] = list(set(idx_list[label][0])- set(selected_test_sample_indices[label]))\n    ","execution_count":8},{"metadata":{"id":"5041DBBD7CBF4D11ABE8F2255C1ABEF9","collapsed":false,"scrolled":false},"cell_type":"code","outputs":[{"output_type":"stream","text":"类别 0 选训练集集样本量： 1597 原本样本量： 1996\n类别 1 选训练集集样本量： 1708 原本样本量： 2134\n类别 2 选训练集集样本量： 1498 原本样本量： 1872\n类别 3 选训练集集样本量： 1453 原本样本量： 1816\n类别 4 选训练集集样本量： 1574 原本样本量： 1967\n类别 5 选训练集集样本量： 1080 原本样本量： 1350\n类别 6 选训练集集样本量： 1295 原本样本量： 1618\n类别 7 选训练集集样本量： 1042 原本样本量： 1302\n类别 8 选训练集集样本量： 1314 原本样本量： 1642\n类别 9 选训练集集样本量： 1657 原本样本量： 2071\ntime: 7.13 ms\n","name":"stdout"}],"source":"for label in range(10):\n    # print(selected_train_sample_indicies[label])\n    print('类别',label,'选训练集集样本量：',len(selected_train_sample_indicies[label]),'原本样本量：',len(idx_list[label][0]))\n    ","execution_count":9},{"metadata":{"id":"DAF2BE01ABB54EF5868C8D88714E57CA","mdEditEnable":false},"cell_type":"markdown","source":"#### Step 2: 将训练集与测试集分别shuffle\n**对于训练集与测试集下标字典，将每个类别选取出的下标整合成一个列表**\n"},{"metadata":{"id":"DA0F2F133AE147F897E4CA16F87A0CAA","collapsed":false,"scrolled":false},"cell_type":"code","outputs":[{"output_type":"stream","text":"训练集样本量： 14218 测试集样本量： 3550\ntime: 19 ms\n","name":"stdout"}],"source":"train_data_indicies, test_data_indicies = [],[]\n\n# 训练集的所有类别下标\nfor label,indicies in selected_train_sample_indicies.items():\n    train_data_indicies = train_data_indicies + indicies # merge 2 list\n# 测试集的所有类别下标\nfor label,indicies in selected_test_sample_indices.items():\n    test_data_indicies = test_data_indicies + indicies\n\n# 打乱顺序(随机排列)\nrandom.shuffle(train_data_indicies)\nrandom.shuffle(test_data_indicies)\n\nprint('训练集样本量：',len(train_data_indicies), '测试集样本量：',len(test_data_indicies))","execution_count":10},{"metadata":{"id":"822ED0C537F3403282C9D6A6083DCEF0","collapsed":false,"scrolled":false,"mdEditEnable":false},"cell_type":"markdown","source":"### 1.4. 构建测试集与训练集\n**根据1.3准备好顺序的下标列表，依次应用到图像及类别数据上。**\n- 让训练集中的图像与label以同样的顺序构成新的列表\n- 让测试集中的图像与label以同样的顺序构成新的列表"},{"metadata":{"id":"53CBA8FBEF9F44838D656E29B505E55F","collapsed":false,"scrolled":false},"cell_type":"code","outputs":[{"output_type":"stream","text":"time: 86.7 ms\n","name":"stdout"}],"source":"y_train_data = np.array([label_data[idx] for idx in train_data_indicies])\nX_train_data = np.array([image_data[idx] for idx in train_data_indicies])\n\ny_test_data = np.array([label_data[idx] for idx in test_data_indicies])\nX_test_data = np.array([image_data[idx] for idx in test_data_indicies])","execution_count":11},{"metadata":{"id":"5D12593C91054CBE8EC7EE29757A5BDE","mdEditEnable":false},"cell_type":"markdown","source":"### 1.4. 查看样例图\n当前已经实现了平衡分布与随机排列的两个条件，现在随机抽查两个样本来看下图片，及对应类别label是否正确。"},{"metadata":{"id":"BAD6F89BE3C14499902FDC11144BFF4B","collapsed":false,"scrolled":false},"cell_type":"code","outputs":[{"output_type":"display_data","metadata":{"needs_background":"light"},"data":{"text/plain":"<Figure size 432x288 with 1 Axes>","text/html":"<img src=\"https://cdn.kesci.com/rt_upload/BAD6F89BE3C14499902FDC11144BFF4B/piy2lbgvpz.png\">"}},{"output_type":"stream","text":"time: 179 ms\n","name":"stdout"}],"source":"plt.imshow(X_test_data[2541], cmap = 'gray')\nplt.title('Class '+ str(y_test_data[2541]))\nplt.show()","execution_count":13},{"metadata":{"id":"6756AE501DFD480E8F657DC9B92870EF","collapsed":false,"scrolled":false},"cell_type":"code","outputs":[{"output_type":"display_data","metadata":{"needs_background":"light"},"data":{"text/plain":"<Figure size 432x288 with 1 Axes>","text/html":"<img src=\"https://cdn.kesci.com/rt_upload/6756AE501DFD480E8F657DC9B92870EF/piy2nw4035.png\">"}},{"output_type":"stream","text":"time: 228 ms\n","name":"stdout"}],"source":"plt.imshow(X_train_data[2], cmap = 'gray')\nplt.title('Class '+ str(y_train_data[2]))\nplt.show()","execution_count":15},{"metadata":{"id":"4DEE884453344193AC20ED90478B0F7D","collapsed":false,"scrolled":false,"hide_input":true},"cell_type":"code","outputs":[],"source":"# split =int(0.8*len(image_data))\n# print(\"切分值：\",split)\n\n# X_train_data = image_data[:split]\n# X_test_data = image_data[split:]\n# y_train_data = label_data[:split]\n# y_test_data = label_data[split:]\n\n# print('original训练集X：',X_train_data.shape,'original训练集y：',y_train_data.shape)\n# print('original测试集X',X_test_data.shape,'original测试集y：',y_test_data.shape)","execution_count":null},{"metadata":{"id":"D775E1F88B044C0985586D307BD366B1","mdEditEnable":false},"cell_type":"markdown","source":"## 2.数据预处理\n### 2.1. label数据进行**one-hot encoding**处理"},{"metadata":{"id":"D0797487CE814C6587123B1BD2F04EE2","mdEditEnable":false,"collapsed":false,"scrolled":false},"cell_type":"code","source":"number_of_classes = 10\n\ny_train = utils.to_categorical(y_train_data, number_of_classes) # one-hot encoding\ny_test = utils.to_categorical(y_test_data, number_of_classes) # one-hot encoding","execution_count":16,"outputs":[{"output_type":"stream","text":"time: 2.64 ms\n","name":"stdout"}]},{"metadata":{"id":"EA1C2804B36249F988FC9DF3F193F457","collapsed":false,"scrolled":false},"cell_type":"code","outputs":[{"output_type":"stream","text":"对应类别为7\n [0. 0. 0. 0. 0. 0. 0. 1. 0. 0.]\ntime: 2.19 ms\n","name":"stdout"}],"source":"# 查看一个类别样本\nprint('对应类别为7\\n',y_train[12112])","execution_count":23},{"metadata":{"id":"79D6B9F35FDB4E9F8E29F5E85E20CAC5","mdEditEnable":false},"cell_type":"markdown","source":"### 2.2. reshape图片特征array\n这是用于模型1全连接神经网络的输入所用"},{"metadata":{"id":"965A41626C464737BF55E04AA4BC5D27","collapsed":false,"scrolled":false},"cell_type":"code","outputs":[{"output_type":"stream","text":"reshaped 训练集X： reshaped 训练集y： (14218, 784) (14218, 10)\nreshaped 测试集X reshaped 测试集y： (3550, 784) (3550, 10)\ntime: 82.1 ms\n","name":"stdout"}],"source":"# Reshape 特征样本\nX_train = X_train_data.reshape(X_train_data.shape[0], img_rows*img_cols)\nX_test = X_test_data.reshape(X_test_data.shape[0], img_rows*img_cols)\n\nprint('reshaped 训练集X：','reshaped 训练集y：',X_train.shape,y_train.shape)\nprint('reshaped 测试集X','reshaped 测试集y：',X_test.shape,y_test.shape)","execution_count":24},{"metadata":{"id":"17BF76FC0E374D65A76028758A0F238E","mdEditEnable":false},"cell_type":"markdown","source":"### 2.2. 归一化图像数据\n- **rescale 取值范围 `[0,255]`成`[0,1]`**"},{"metadata":{"id":"F47E2E0E6F6946A9AE4E754B4EE59007","collapsed":false,"scrolled":false},"cell_type":"code","outputs":[{"output_type":"stream","text":"time: 33.6 ms\n","name":"stdout"}],"source":"X_train = X_train / 255\nX_test = X_test / 255","execution_count":25},{"metadata":{"id":"86098D8275C847C79535B5D28D217DA9","mdEditEnable":false},"cell_type":"markdown","source":"## 3.训练模型\n## 3.1. 模型1：全连接神经网络\n- epochs = 30\n\t- 运行时间1min29s\n\t- 测试集Accuracy: 82.06%\n\t- 测试集loss:2.16；\n- epochs = 40\n\t- 运行时间1min58秒\n\t- 测试集loss:1.60\n\t- 测试集Accuracy:85.83%"},{"metadata":{"id":"3E45C4E678B246F284E313BEFFB6E053","collapsed":false,"scrolled":false},"cell_type":"code","outputs":[{"output_type":"stream","text":"time: 3.46 ms\n","name":"stdout"}],"source":"def FCNN(epochs):\n    # create model\n    model = keras.Sequential()\n    model.add(keras.layers.Dense(784, input_dim=784, kernel_initializer='normal', activation= 'tanh'))\n    model.add(keras.layers.Dense(512, kernel_initializer='normal', activation= 'tanh'))\n    model.add(keras.layers.Dense(512, kernel_initializer='normal', activation= 'tanh'))\n    model.add(keras.layers.Dense(10, kernel_initializer='normal', activation= 'softmax'))\n    \n    # Compile model \n    # 损失函：数交叉函数熵\n    # 优化器：adam\n    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n    \n    # Fit the model\n    model.fit(X_train, \n            y_train, \n            epochs=epochs, \n            batch_size=200, \n            verbose=1, \n            validation_data=(X_test,y_test)\n            )\n    return model","execution_count":31},{"metadata":{"id":"467F8EC12BC740E297CDB6252F9C4247","mdEditEnable":false},"cell_type":"markdown","source":"**Epochs = 30**"},{"metadata":{"id":"929633CAB91C417FBC71201AA6DD57F4","collapsed":false,"scrolled":true},"cell_type":"code","outputs":[{"output_type":"stream","text":"Train on 14218 samples, validate on 3550 samples\nEpoch 1/30\n14218/14218 [==============================] - 3s 237us/step - loss: 11.8595 - acc: 0.1814 - val_loss: 10.0464 - val_acc: 0.2851\nEpoch 2/30\n14218/14218 [==============================] - 3s 210us/step - loss: 8.4464 - acc: 0.3913 - val_loss: 7.6192 - val_acc: 0.4420\nEpoch 3/30\n14218/14218 [==============================] - 3s 207us/step - loss: 6.6349 - acc: 0.5067 - val_loss: 6.2335 - val_acc: 0.5344\nEpoch 4/30\n14218/14218 [==============================] - 3s 206us/step - loss: 5.5854 - acc: 0.5722 - val_loss: 5.5333 - val_acc: 0.5845\nEpoch 5/30\n14218/14218 [==============================] - 3s 209us/step - loss: 4.6863 - acc: 0.6307 - val_loss: 4.7545 - val_acc: 0.6223\nEpoch 6/30\n14218/14218 [==============================] - 3s 208us/step - loss: 4.0149 - acc: 0.6729 - val_loss: 4.2106 - val_acc: 0.6566\nEpoch 7/30\n14218/14218 [==============================] - 3s 208us/step - loss: 3.5094 - acc: 0.7097 - val_loss: 3.8513 - val_acc: 0.6831\nEpoch 8/30\n14218/14218 [==============================] - 3s 205us/step - loss: 3.1287 - acc: 0.7356 - val_loss: 3.6089 - val_acc: 0.7082\nEpoch 9/30\n14218/14218 [==============================] - 3s 206us/step - loss: 2.9027 - acc: 0.7541 - val_loss: 3.4444 - val_acc: 0.7237\nEpoch 10/30\n14218/14218 [==============================] - 3s 206us/step - loss: 2.6606 - acc: 0.7734 - val_loss: 3.2876 - val_acc: 0.7310\nEpoch 11/30\n14218/14218 [==============================] - 3s 205us/step - loss: 2.4626 - acc: 0.7910 - val_loss: 3.2157 - val_acc: 0.7414\nEpoch 12/30\n14218/14218 [==============================] - 3s 209us/step - loss: 2.3095 - acc: 0.8043 - val_loss: 3.1658 - val_acc: 0.7417\nEpoch 13/30\n14218/14218 [==============================] - 3s 209us/step - loss: 2.2725 - acc: 0.8067 - val_loss: 2.9554 - val_acc: 0.7577\nEpoch 14/30\n14218/14218 [==============================] - 3s 206us/step - loss: 2.1731 - acc: 0.8130 - val_loss: 2.9087 - val_acc: 0.7634\nEpoch 15/30\n14218/14218 [==============================] - 3s 207us/step - loss: 2.0684 - acc: 0.8202 - val_loss: 2.6595 - val_acc: 0.7792\nEpoch 16/30\n14218/14218 [==============================] - 3s 207us/step - loss: 1.9813 - acc: 0.8248 - val_loss: 2.5978 - val_acc: 0.7789\nEpoch 17/30\n14218/14218 [==============================] - 3s 211us/step - loss: 1.8774 - acc: 0.8383 - val_loss: 2.6325 - val_acc: 0.7845\nEpoch 18/30\n14218/14218 [==============================] - 3s 205us/step - loss: 1.7792 - acc: 0.8412 - val_loss: 2.5107 - val_acc: 0.7913\nEpoch 19/30\n14218/14218 [==============================] - 3s 210us/step - loss: 1.7474 - acc: 0.8469 - val_loss: 2.4956 - val_acc: 0.7944\nEpoch 20/30\n14218/14218 [==============================] - 3s 205us/step - loss: 1.6487 - acc: 0.8538 - val_loss: 2.3637 - val_acc: 0.8011\nEpoch 21/30\n14218/14218 [==============================] - 3s 207us/step - loss: 1.5835 - acc: 0.8594 - val_loss: 2.4087 - val_acc: 0.8008\nEpoch 22/30\n14218/14218 [==============================] - 3s 205us/step - loss: 1.5696 - acc: 0.8598 - val_loss: 2.3261 - val_acc: 0.8034\nEpoch 23/30\n14218/14218 [==============================] - 3s 202us/step - loss: 1.5664 - acc: 0.8598 - val_loss: 2.3286 - val_acc: 0.8070\nEpoch 24/30\n14218/14218 [==============================] - 3s 202us/step - loss: 1.5602 - acc: 0.8606 - val_loss: 2.3357 - val_acc: 0.8068\nEpoch 25/30\n14218/14218 [==============================] - 3s 206us/step - loss: 1.5049 - acc: 0.8640 - val_loss: 2.2634 - val_acc: 0.8065\nEpoch 26/30\n14218/14218 [==============================] - 3s 205us/step - loss: 1.4552 - acc: 0.8661 - val_loss: 2.1214 - val_acc: 0.8169\nEpoch 27/30\n14218/14218 [==============================] - 3s 203us/step - loss: 1.4107 - acc: 0.8728 - val_loss: 2.1411 - val_acc: 0.8152\nEpoch 28/30\n14218/14218 [==============================] - 3s 204us/step - loss: 1.3851 - acc: 0.8768 - val_loss: 2.1382 - val_acc: 0.8183\nEpoch 29/30\n14218/14218 [==============================] - 3s 205us/step - loss: 1.3834 - acc: 0.8763 - val_loss: 2.0805 - val_acc: 0.8239\nEpoch 30/30\n14218/14218 [==============================] - 3s 204us/step - loss: 1.3450 - acc: 0.8762 - val_loss: 2.1597 - val_acc: 0.8206\ntime: 1min 29s\n","name":"stdout"}],"source":"epochs = 30\nmodel1 = FCNN(epochs=epochs)","execution_count":32},{"metadata":{"id":"8C51F7A430A74F9C91F4D0FFE930D8AF","collapsed":false,"scrolled":false},"cell_type":"code","outputs":[{"output_type":"stream","text":"Epochs:30, 测试集loss:2.16, 测试集Accuracy:82.06%\ntime: 411 ms\n","name":"stdout"}],"source":"score1 = model1.evaluate(X_test,y_test, verbose=0)\nprint('Epochs:{0}, 测试集loss:{1:.02f}, 测试集Accuracy:{2:.2%}'.format(epochs,score1[0],score1[1]))","execution_count":42},{"metadata":{"id":"A8C788DE0F1C4B05AA8BA112E5FDD366","mdEditEnable":false},"cell_type":"markdown","source":"**Epochs = 40**"},{"metadata":{"id":"DFF6A399BC8A4CF8BB7D0F0129CDC02C","collapsed":false,"scrolled":true},"cell_type":"code","outputs":[{"output_type":"stream","text":"Train on 14218 samples, validate on 3550 samples\nEpoch 1/40\n14218/14218 [==============================] - 3s 240us/step - loss: 11.9571 - acc: 0.1811 - val_loss: 10.1538 - val_acc: 0.2893\nEpoch 2/40\n14218/14218 [==============================] - 3s 206us/step - loss: 8.6996 - acc: 0.3744 - val_loss: 7.4362 - val_acc: 0.4482\nEpoch 3/40\n14218/14218 [==============================] - 3s 207us/step - loss: 6.6314 - acc: 0.5025 - val_loss: 5.9358 - val_acc: 0.5558\nEpoch 4/40\n14218/14218 [==============================] - 3s 205us/step - loss: 5.4291 - acc: 0.5857 - val_loss: 5.0473 - val_acc: 0.6042\nEpoch 5/40\n14218/14218 [==============================] - 3s 207us/step - loss: 4.6773 - acc: 0.6366 - val_loss: 4.5936 - val_acc: 0.6470\nEpoch 6/40\n14218/14218 [==============================] - 3s 206us/step - loss: 4.2236 - acc: 0.6678 - val_loss: 4.2520 - val_acc: 0.6623\nEpoch 7/40\n14218/14218 [==============================] - 3s 205us/step - loss: 3.7653 - acc: 0.6987 - val_loss: 3.9279 - val_acc: 0.6845\nEpoch 8/40\n14218/14218 [==============================] - 3s 207us/step - loss: 3.3819 - acc: 0.7203 - val_loss: 3.4516 - val_acc: 0.7130\nEpoch 9/40\n14218/14218 [==============================] - 3s 211us/step - loss: 3.0868 - acc: 0.7449 - val_loss: 3.3089 - val_acc: 0.7318\nEpoch 10/40\n14218/14218 [==============================] - 3s 205us/step - loss: 2.8364 - acc: 0.7586 - val_loss: 3.1146 - val_acc: 0.7470\nEpoch 11/40\n14218/14218 [==============================] - 3s 205us/step - loss: 2.6447 - acc: 0.7755 - val_loss: 2.9783 - val_acc: 0.7524\nEpoch 12/40\n14218/14218 [==============================] - 3s 209us/step - loss: 2.5152 - acc: 0.7872 - val_loss: 2.8797 - val_acc: 0.7606\nEpoch 13/40\n14218/14218 [==============================] - 3s 208us/step - loss: 2.3171 - acc: 0.8048 - val_loss: 2.6465 - val_acc: 0.7755\nEpoch 14/40\n14218/14218 [==============================] - 3s 211us/step - loss: 2.1760 - acc: 0.8125 - val_loss: 2.6764 - val_acc: 0.7775\nEpoch 15/40\n14218/14218 [==============================] - 3s 207us/step - loss: 2.0496 - acc: 0.8235 - val_loss: 2.6030 - val_acc: 0.7814\nEpoch 16/40\n14218/14218 [==============================] - 3s 205us/step - loss: 1.9351 - acc: 0.8303 - val_loss: 2.4683 - val_acc: 0.7885\nEpoch 17/40\n14218/14218 [==============================] - 3s 206us/step - loss: 1.8451 - acc: 0.8392 - val_loss: 2.5317 - val_acc: 0.7859\nEpoch 18/40\n14218/14218 [==============================] - 3s 206us/step - loss: 1.7428 - acc: 0.8451 - val_loss: 2.3859 - val_acc: 0.7997\nEpoch 19/40\n14218/14218 [==============================] - 3s 207us/step - loss: 1.7534 - acc: 0.8430 - val_loss: 2.3371 - val_acc: 0.8003\nEpoch 20/40\n14218/14218 [==============================] - 3s 205us/step - loss: 1.6725 - acc: 0.8517 - val_loss: 2.3015 - val_acc: 0.8017\nEpoch 21/40\n14218/14218 [==============================] - 3s 206us/step - loss: 1.6187 - acc: 0.8590 - val_loss: 2.2633 - val_acc: 0.8124\nEpoch 22/40\n14218/14218 [==============================] - 3s 208us/step - loss: 1.5830 - acc: 0.8593 - val_loss: 2.2822 - val_acc: 0.8079\nEpoch 23/40\n14218/14218 [==============================] - 3s 208us/step - loss: 1.5938 - acc: 0.8576 - val_loss: 2.1409 - val_acc: 0.8169\nEpoch 24/40\n14218/14218 [==============================] - 3s 205us/step - loss: 1.5137 - acc: 0.8643 - val_loss: 2.0565 - val_acc: 0.8245\nEpoch 25/40\n14218/14218 [==============================] - 3s 207us/step - loss: 1.4624 - acc: 0.8707 - val_loss: 2.0349 - val_acc: 0.8214\nEpoch 26/40\n14218/14218 [==============================] - 3s 206us/step - loss: 1.4164 - acc: 0.8740 - val_loss: 2.0285 - val_acc: 0.8245\nEpoch 27/40\n14218/14218 [==============================] - 3s 203us/step - loss: 1.3681 - acc: 0.8738 - val_loss: 1.9707 - val_acc: 0.8276\nEpoch 28/40\n14218/14218 [==============================] - 3s 206us/step - loss: 1.3212 - acc: 0.8784 - val_loss: 1.9962 - val_acc: 0.8279\nEpoch 29/40\n14218/14218 [==============================] - 3s 205us/step - loss: 1.3266 - acc: 0.8784 - val_loss: 1.9321 - val_acc: 0.8276\nEpoch 30/40\n14218/14218 [==============================] - 3s 208us/step - loss: 1.3260 - acc: 0.8794 - val_loss: 1.9941 - val_acc: 0.8248\nEpoch 31/40\n14218/14218 [==============================] - 3s 205us/step - loss: 1.2547 - acc: 0.8826 - val_loss: 1.9604 - val_acc: 0.8349\nEpoch 32/40\n14218/14218 [==============================] - 3s 207us/step - loss: 1.2681 - acc: 0.8823 - val_loss: 1.8829 - val_acc: 0.8338\nEpoch 33/40\n14218/14218 [==============================] - 3s 204us/step - loss: 1.2315 - acc: 0.8867 - val_loss: 1.9290 - val_acc: 0.8315\nEpoch 34/40\n14218/14218 [==============================] - 3s 206us/step - loss: 1.2034 - acc: 0.8877 - val_loss: 1.8596 - val_acc: 0.8380\nEpoch 35/40\n14218/14218 [==============================] - 3s 204us/step - loss: 1.1886 - acc: 0.8885 - val_loss: 1.8834 - val_acc: 0.8377\nEpoch 36/40\n14218/14218 [==============================] - 3s 207us/step - loss: 1.1983 - acc: 0.8899 - val_loss: 1.8341 - val_acc: 0.8448\nEpoch 37/40\n14218/14218 [==============================] - 3s 202us/step - loss: 1.1904 - acc: 0.8868 - val_loss: 1.8228 - val_acc: 0.8389\nEpoch 38/40\n14218/14218 [==============================] - 3s 201us/step - loss: 1.1631 - acc: 0.8915 - val_loss: 1.7688 - val_acc: 0.8414\nEpoch 39/40\n14218/14218 [==============================] - 3s 202us/step - loss: 1.0857 - acc: 0.8970 - val_loss: 1.7688 - val_acc: 0.8462\nEpoch 40/40\n14218/14218 [==============================] - 3s 205us/step - loss: 1.0932 - acc: 0.8944 - val_loss: 1.5959 - val_acc: 0.8583\ntime: 1min 58s\n","name":"stdout"}],"source":"epochs = 40\nmodel2 = FCNN(epochs=epochs)","execution_count":44},{"outputs":[{"output_type":"stream","text":"Epochs:40, 测试集loss:1.60, 测试集Accuracy:85.83%\ntime: 408 ms\n","name":"stdout"}],"execution_count":46,"source":"score2 = model2.evaluate(X_test,y_test, verbose=0)\nprint('Epochs:{0}, 测试集loss:{1:.02f}, 测试集Accuracy:{2:.2%}'.format(epochs,score2[0],score2[1]))","cell_type":"code","metadata":{"collapsed":false,"id":"02F3EB30587A4F608BDEFD92FEF16271","scrolled":false}},{"metadata":{"id":"3DFD9FA5912648629BEA7B7EE8709CF5","collapsed":false,"scrolled":false,"mdEditEnable":false},"cell_type":"markdown","source":"在一定程度下，iterate的次数的增加可以提升准确度，但这并不代表无限的增加循环次数，准确率就一定会更高，很有可能会出现过拟合。\n全连接神经网络的能力是有限的，是**不适合做图像识别任务**的。因为通过梯度下降训练它很难，它的梯度很难传递超过三层，我们是得不到一个很深的全连接神经网络。"},{"metadata":{"id":"DE1C5859B1D349EA8B3AD92EBE476FCA","mdEditEnable":false},"cell_type":"markdown","source":"## 3.2. 模型2：卷积神经网络（CNN）\nCNN是很适合做图像识别的，它可以尽可能的保留图像中重要的参数，舍去大量不必要的参数，从而达到更好的学习效果。接下来我们尝试使用CNN来进行图片分类\n- epochs = 5, batchsize = 64*3 = 192\n- 运行时间15分43秒\n- 测试集loss: 0.08\n- 测试集accuracy: 97.55%"},{"metadata":{"id":"D4AF518C9AC84FBD80C16C9B8BBE2FFD","collapsed":false,"scrolled":false},"cell_type":"code","outputs":[],"source":"from keras.models import Sequential\nfrom keras.utils import np_utils\nfrom keras.layers import Conv2D, MaxPooling2D\nfrom keras.layers import Dense, Dropout, Activation, Flatten\nfrom keras.optimizers import Adam\nfrom keras.layers.normalization import BatchNormalization\nfrom keras.layers.advanced_activations import LeakyReLU\n\ninput_shape = (img_rows, img_cols, 1)","execution_count":3},{"metadata":{"id":"6AE275FE461E4F8D86C06B171C68CDA9","mdEditEnable":false},"cell_type":"markdown","source":"### 3.2.1. 数据准备\n#### 3.2.2. 图像数据\n- 将图像数据reshape成（X_train_data.shape[0],28,28,1）\n- 归一化图像数据"},{"metadata":{"id":"BA35102F80394B078B6296C1F2A756B6","collapsed":false,"scrolled":false},"cell_type":"code","outputs":[{"output_type":"stream","text":"训练集X: (14218, 28, 28, 1) \n测试集X: (3550, 28, 28, 1)\ntime: 45 ms\n","name":"stdout"}],"source":"X_train = X_train_data.reshape(X_train_data.shape[0], img_rows,img_cols,1)\nX_test = X_test_data.reshape(X_test_data.shape[0], img_rows,img_cols,1)\n\nX_train = X_train.astype('float32')\nX_test = X_test.astype('float32')\n\nX_train /= 255\nX_test /= 255\n\nprint('训练集X:',X_train.shape, '\\n测试集X:',X_test.shape)","execution_count":48},{"metadata":{"id":"826F048107494FEB89789C558E1236EC","collapsed":false,"mdEditEnable":false},"cell_type":"markdown","source":"#### 3.2.3. label数据\n- one-hot encoding处理，这个在之前的过程中已经完成，所以这里就不重复了。\n- 比如：y_train[0] = array([1., 0., 0., 0., 0., 0., 0., 0., 0., 0.])， 表示label为1。\n"},{"metadata":{"id":"44D35286577E426B87FD2A2900F633ED","mdEditEnable":false},"cell_type":"markdown","source":"### 3.2.2. 构建模型\n#### CNN网络构建的3个步骤：\n1. **卷积 Convolution**\n\t- **关于卷积层输出值的计算动图如下所示**\n\t假设图像是5x5，使用一个3x3的filter进行卷积，得到一个3x3的feature map。\n\t![Image Name](https://cdn.kesci.com/upload/image/piy82itx4i.gif?imageView2/0/w/320/h/320)\n2. **激活 Activation**\n3. **池化 Pooling**\n**重复1-3步骤来添加更多的隐藏层（hidden layers**）\n4. **然后构建一个全连接网络，让CNN能进行样本分类**\n\n\n\n\n"},{"metadata":{"id":"F2EFAF3A537A4979BDABC4C19211CB4A","collapsed":false,"scrolled":false},"cell_type":"code","outputs":[{"output_type":"stream","text":"_________________________________________________________________\nLayer (type)                 Output Shape              Param #   \n=================================================================\nconv2d_1 (Conv2D)            (None, 26, 26, 32)        320       \n_________________________________________________________________\nbatch_normalization_1 (Batch (None, 26, 26, 32)        128       \n_________________________________________________________________\nactivation_1 (Activation)    (None, 26, 26, 32)        0         \n_________________________________________________________________\nconv2d_2 (Conv2D)            (None, 24, 24, 32)        9248      \n_________________________________________________________________\nbatch_normalization_2 (Batch (None, 24, 24, 32)        128       \n_________________________________________________________________\nactivation_2 (Activation)    (None, 24, 24, 32)        0         \n_________________________________________________________________\nconv2d_3 (Conv2D)            (None, 22, 22, 64)        18496     \n_________________________________________________________________\nbatch_normalization_3 (Batch (None, 22, 22, 64)        256       \n_________________________________________________________________\nactivation_3 (Activation)    (None, 22, 22, 64)        0         \n_________________________________________________________________\nconv2d_4 (Conv2D)            (None, 20, 20, 64)        36928     \n_________________________________________________________________\nbatch_normalization_4 (Batch (None, 20, 20, 64)        256       \n_________________________________________________________________\nactivation_4 (Activation)    (None, 20, 20, 64)        0         \n_________________________________________________________________\nmax_pooling2d_1 (MaxPooling2 (None, 10, 10, 64)        0         \n_________________________________________________________________\nflatten_1 (Flatten)          (None, 6400)              0         \n_________________________________________________________________\ndense_1 (Dense)              (None, 512)               3277312   \n_________________________________________________________________\nbatch_normalization_5 (Batch (None, 512)               2048      \n_________________________________________________________________\nactivation_5 (Activation)    (None, 512)               0         \n_________________________________________________________________\ndropout_1 (Dropout)          (None, 512)               0         \n_________________________________________________________________\ndense_2 (Dense)              (None, 10)                5130      \n_________________________________________________________________\nactivation_6 (Activation)    (None, 10)                0         \n=================================================================\nTotal params: 3,350,250\nTrainable params: 3,348,842\nNon-trainable params: 1,408\n_________________________________________________________________\n","name":"stdout"}],"source":"model = Sequential()\n\nmodel.add(Conv2D(32, kernel_size=(3, 3), input_shape=(28,28,1)))\nmodel.add(BatchNormalization(axis=-1)) # normalize\nmodel.add(Activation('relu'))\n\nmodel.add(Conv2D(32, kernel_size=(3, 3)))\nmodel.add(BatchNormalization(axis=-1))\nmodel.add(Activation('relu'))\n\nmodel.add(Conv2D(64, (3, 3)))\nmodel.add(BatchNormalization(axis=-1))\nmodel.add(Activation('relu'))\n\nmodel.add(Conv2D(64, (3, 3)))\nmodel.add(BatchNormalization(axis=-1))\nmodel.add(Activation('relu'))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\n\nmodel.add(Flatten()) # 展开成1-D的阵列\n\n# Fully Connected Layer\nmodel.add(Dense(512))\nmodel.add(BatchNormalization())\nmodel.add(Activation('relu'))\nmodel.add(Dropout(0.2))\nmodel.add(Dense(10)) # 10: num_classes\n\nmodel.add(Activation('softmax'))\n\nmodel.summary()","execution_count":4},{"metadata":{"id":"9D56A3914AFC467382A1DF1B553569AA","collapsed":false,"scrolled":false},"cell_type":"code","outputs":[{"output_type":"stream","text":"time: 44 ms\n","name":"stdout"}],"source":"model.compile(loss='categorical_crossentropy',\n              optimizer=Adam(),\n              metrics=['accuracy'])","execution_count":50},{"metadata":{"id":"17B0D8183D814E27B39E7E7FF2D6771D","mdEditEnable":false},"cell_type":"markdown","source":"#### 3.2.3.训练模型"},{"metadata":{"id":"F89E6B0402B74F2585F79A3E8B60671C","collapsed":false,"scrolled":false},"cell_type":"code","outputs":[{"output_type":"stream","text":"Train on 14218 samples, validate on 3550 samples\nEpoch 1/5\n14218/14218 [==============================] - 190s 13ms/step - loss: 0.1960 - acc: 0.9372 - val_loss: 0.0980 - val_acc: 0.9713\nEpoch 2/5\n14218/14218 [==============================] - 188s 13ms/step - loss: 0.0402 - acc: 0.9875 - val_loss: 0.0839 - val_acc: 0.9727\nEpoch 3/5\n14218/14218 [==============================] - 188s 13ms/step - loss: 0.0216 - acc: 0.9949 - val_loss: 0.0661 - val_acc: 0.9797\nEpoch 4/5\n14218/14218 [==============================] - 188s 13ms/step - loss: 0.0123 - acc: 0.9971 - val_loss: 0.0784 - val_acc: 0.9786\nEpoch 5/5\n14218/14218 [==============================] - 188s 13ms/step - loss: 0.0095 - acc: 0.9974 - val_loss: 0.0823 - val_acc: 0.9755\n","name":"stdout"},{"output_type":"execute_result","metadata":{},"data":{"text/plain":"<keras.callbacks.History at 0x7f397ca30978>"}},{"output_type":"stream","text":"time: 15min 43s\n","name":"stdout"}],"source":"epochs = 5\n\nmodel.fit(X_train, y_train,\n          batch_size=192,\n          epochs=epochs,\n          verbose=1,\n          validation_data=(X_test, y_test), )","execution_count":51},{"metadata":{"id":"4019209520254860BCF0425ED5302CFB","mdEditEnable":false},"cell_type":"markdown","source":"#### 3.2.4.模型效果"},{"metadata":{"id":"639E6AF5EFBC434F863218FC33982B73","collapsed":false,"scrolled":false},"cell_type":"code","outputs":[{"output_type":"stream","text":"Epochs:5, 测试集loss:0.08, 测试集Accuracy:97.55%\ntime: 15.2 s\n","name":"stdout"}],"source":"score = model.evaluate(X_test, y_test, verbose=0)\nprint('Epochs:{0}, 测试集loss:{1:.02f}, 测试集Accuracy:{2:.2%}'.format(epochs,score[0],score[1]))","execution_count":53},{"metadata":{"id":"36FCDCB3B0F245CC8CEAFB00F432D141","collapsed":false,"scrolled":false,"mdEditEnable":false},"cell_type":"markdown","source":"我们用这个简单的CNN模型得到了97.55%的准确率，想要提升准确率，可以尝试数据增量（data augmentation）,或者模型集成（ensembling）等。这些就等你来探索啦~"},{"metadata":{"id":"B23708F3F1BA463E81883311D117447A"},"cell_type":"code","outputs":[],"source":"","execution_count":null}],"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"version":"3.5.5","pygments_lexer":"ipython3","name":"python","mimetype":"text/x-python","nbconvert_exporter":"python","codemirror_mode":{"version":3,"name":"ipython"},"file_extension":".py"}},"nbformat":4,"nbformat_minor":0}