diff --git a/recognition/45307915/README.MD b/recognition/45307915/README.MD new file mode 100644 index 0000000000..ea326ff7cc --- /dev/null +++ b/recognition/45307915/README.MD @@ -0,0 +1,123 @@ +# Improved UNET - Segmentation of ISIC Dataset +The ISIC challenge is an international effort to improve melanoma diagnosis, one of the deadliest forms of skin cancer. Early detection greatly increases the chances of survival for melanoma victims. The ISIC have published datasets for the community to further computer vision research. + +By using a neural network a model can be trained to segment the melanoma within a skin image. This project shows the use of an Improved UNET to achieve this. + +## ISIC Dataset +The ISIC dataset contains a set of skin lesion images and their respective segmentation masks. The 2017 dataset used was already split into training, testing and validation sets. + +Due to all the images in the set being different sizes they were all resized. The skin lesion images were resized to (256, 256, 3), keeping their colour channels. The segmentation masks were resized to (256, 256, 1), reducing to a single colour channel. Thresholding was then completed on the segmentation masks, pixels with a value >0.5 were set to 1, and all other pixels set to 0. + +![Training Image](https://github.com/ryanjohnson08/45307915/blob/topic-recognition/recognition/45307915/images/image.png) +![Training Mask](https://github.com/ryanjohnson08/45307915/blob/topic-recognition/recognition/45307915/images/mask.png) + +Available at https://challenge.isic-archive.com/data/#2017 + +## Improved UNET Model Architecture + +The following UNET model was proposed in the Brain Tumor Segmentation and Radiomics paper. [1] + +![Improved UNET Model Architecture](https://github.com/ryanjohnson08/45307915/blob/topic-recognition/recognition/45307915/images/IUNET_Architecture.JPG) + +Like the original UNET the Improved UNET is made up of encoding and decoding layers. Components of the improved UNET Architecture are as follows. + +- Leaky ReLU - "leaky ReLU nonlinearities with a negative slope of $10^{−2}$ for all feature map computing convolutions" [1] +- Instance Normalisation - "replace the traditional batch with instance normalization" [1] +- Context Module - "Each context module is in fact a pre-activation residual block with two 3x3x3 convolutional layers and a dropout layer (pdrop = 0.3) in between." [1] +- Upsampling Module - "This is achieved by first upsampling the low resolution feature maps, which is done by means of a simple upscale that repeats the feature voxels twice in each spatial dimension, followed by a 3x3x3 convolution" [1] +- Localisation Module - "A localization module consists of a 3x3x3 convolution followed by a 1x1x1 convolution that halves the number of feature maps." [1] +- Skip Connection - Like the original UNET the improved UNET makes use of skip connections +- Segmentation Layer - These are 3x3 convolutions with a single output filter. These segments are summed to form the final network output. + +## Training + +### Optimiser and Loss Function +The optimiser used for training was "the adam optimizer with an initial learning rate $lr_{init} = 5 · 10^{−4}$, the following learning rate schedule: $lr_{init} · 0.985^{epoch}$ ". [1] + +For the loss function the dice coefficient for the true and predicted values was used. The dice score is defined as follows: +$Dice = 2 |A∩B| / (|A|+|B|)$ + +``` +def diceCoefficient(self, y_true, y_pred): + """ + Dice Coefficient + + Credit Karan Jakhar + https://karan-jakhar.medium.com/100-days-of-code-day-7-84e4918cb72c + + Parameters: + y_true (tf.Tensor): true output + y_true (tf.Tensor): output predicted by model + + Return: + tf.Tensor: Dice coefficient based on true output and prediction + + """ + y_true_f = K.flatten(y_true) + y_pred_f = K.flatten(y_pred) + intersection = K.sum(y_true_f * y_pred_f) + dice = (2. * intersection + 1.) / (K.sum(y_true_f) + K.sum(y_pred_f) + 1.) + return dice +``` + +### Training Dice Coefficient and Loss +The following graphs show the training Dice Coefficient Loss and Accuracy over 20 Epochs. +![Dice Coefficient Loss](https://github.com/ryanjohnson08/45307915/blob/topic-recognition/recognition/45307915/images/diceCoefficientLoss.png) +![Dice Coefficient](https://github.com/ryanjohnson08/45307915/blob/topic-recognition/recognition/45307915/images/diceCoefficient.png) + +### Adjustments +Initially, the structure layed out in the paper was followed for model training. However, the model test accuracy didn't have a Dice similarity coefficient of greater than 0.8. + +![Test data dice coefficient original](https://github.com/ryanjohnson08/45307915/blob/topic-recognition/recognition/45307915/images/TestDataDiceCoefficientOriginal.JPG) + +Through testing the following adjustments were found to make a difference. +First, the learning rate scheduler which was used as per the paper was removed. +``` +results = model.fit(train_batch, epochs=self.epochs, validation_data=validate_batch, + callbacks=[tf.keras.callbacks.LearningRateScheduler( + lambda epoch: self.learning_rate * 0.985 ** (epoch) + )]) +``` +Replaced with a constant learning rate. +``` +adamOptimizer = Adam(learning_rate=self.learning_rate) +model.compile(optimizer=adamOptimizer, loss=self.diceLoss, metrics=[self.diceCoefficient]) +``` +This resulted in a slight increase in the dice similarity coefficient. +![Test data dice coefficient adjust learning rate](https://github.com/ryanjohnson08/45307915/blob/topic-recognition/recognition/45307915/images/TestDataDiceCoefficientLearningRate.JPG) + +By decreasing the batch size the model was able to achieve a dice similarity coefficient greater than 0.8 for the test set. +![Test data dice coefficient adjust batch size](https://github.com/ryanjohnson08/45307915/blob/topic-recognition/recognition/45307915/images/TestDataDiceCoefficientDecreaseBatchSize.JPG) + +## Predictions +Using the model the following segmentation predictions were made from test set inputs. +![prediction1](https://github.com/ryanjohnson08/45307915/blob/topic-recognition/recognition/45307915/images/p1.png) +![prediction2](https://github.com/ryanjohnson08/45307915/blob/topic-recognition/recognition/45307915/images/p2.png) +![prediction3](https://github.com/ryanjohnson08/45307915/blob/topic-recognition/recognition/45307915/images/p3.png) +![prediction4](https://github.com/ryanjohnson08/45307915/blob/topic-recognition/recognition/45307915/images/p4.png) +![prediction5](https://github.com/ryanjohnson08/45307915/blob/topic-recognition/recognition/45307915/images/p5.png) +![prediction6](https://github.com/ryanjohnson08/45307915/blob/topic-recognition/recognition/45307915/images/p6.png) +![prediction7](https://github.com/ryanjohnson08/45307915/blob/topic-recognition/recognition/45307915/images/p7.png) +![prediction8](https://github.com/ryanjohnson08/45307915/blob/topic-recognition/recognition/45307915/images/p8.png) +![prediction9](https://github.com/ryanjohnson08/45307915/blob/topic-recognition/recognition/45307915/images/p9.png) +![prediction10](https://github.com/ryanjohnson08/45307915/blob/topic-recognition/recognition/45307915/images/p10.png) +![prediction11](https://github.com/ryanjohnson08/45307915/blob/topic-recognition/recognition/45307915/images/p11.png) + +## Usage +To use, first ensure all dependencies are downloaded. + +Download the 2017 ISIC dataset and place training, testing, and validation image and mask sets in the working directory. + +By running ```train.py``` an Improved UNET model will be trained on the training data, and validated on the validation data. The batch size, learning rate and number of epochs for the model can be adjusted within the ```train.py``` file. + +By running ```predict.py``` predictions will be made on a test dataset using the trained model. + +### Dependencies +- Python: 3.7.15 +- Numpy: 1.21.6 +- Matplotlib: 3.2.2 +- Tensorflow: 2.9.2 +- Tensorflow Addons: 0.18.0 + +## References +[1] F. Isensee, P. Kickingereder, W. Wick, M. Bendszus, and K. H. Maier-Hein, “Brain Tumor Segmentation and Radiomics Survival Prediction: Contribution to the BRATS 2017 Challenge,” Feb. 2018. [Online]. Available: https://arxiv.org/abs/1802.10508v1 diff --git a/recognition/45307915/dataset.py b/recognition/45307915/dataset.py new file mode 100644 index 0000000000..a3ff05fd69 --- /dev/null +++ b/recognition/45307915/dataset.py @@ -0,0 +1,131 @@ +import tensorflow as tf + +TRAIN_IMAGES_PATH = "./ISIC-2017_Training_Data/*.jpg" +TRAIN_MASKS_PATH = "./ISIC-2017_Training_Part1_GroundTruth/*.png" + +TEST_IMAGES_PATH = "./ISIC-2017_Test_v2_Data/*.jpg" +TEST_MASKS_PATH = "./ISIC-2017_Test_v2_Part1_GroundTruth/*.png" + +VALIDATE_IMAGES_PATH = "./ISIC-2017_Validation_Data/*.jpg" +VALIDATE_MASKS_PATH = "./ISIC-2017_Validation_Part1_GroundTruth/*.png" + +IMAGE_HEIGHT = 256 +IMAGE_WIDTH = 256 + +class DataLoader(): + + def __init__(self, train_images_path=TRAIN_IMAGES_PATH, train_masks_path=TRAIN_MASKS_PATH, + test_images_path=TEST_IMAGES_PATH, test_masks_path=TEST_MASKS_PATH, + validate_images_path=VALIDATE_IMAGES_PATH, validate_masks_path=VALIDATE_MASKS_PATH, + img_width=IMAGE_WIDTH, img_height=IMAGE_HEIGHT): + """ + Create a new Data Loader instance to load in the training, testing and validation data + + Parameters: + images_path (str): Path of the dataset images + masks_path (str): Path of the dataset masks + img_width (int): Image Width + img_height (int): Image Height + """ + self.train_images_path = train_images_path + self.train_masks_path = train_masks_path + + self.test_images_path = test_images_path + self.test_masks_path = test_masks_path + + self.validate_images_path = validate_images_path + self.validate_masks_path = validate_masks_path + + self.img_width = img_width + self.img_height = img_height + + def preprocessImages(self, filenames): + """ + Load and preprocess the image files. + + Image loading code from TensorFlow + https://www.tensorflow.org/api_docs/python/tf/io/read_file + + Parameters: + filenames (tf.string): names of all image files + + Return: + tf.Dataset: Dataset containing all the image file data + + """ + raw = tf.io.read_file(filenames) + images = tf.io.decode_jpeg(raw, channels=3) + + #resize images + images = tf.image.resize(images, [self.img_height, self.img_width]) + + #Normalise + images = images / 255. + + return images + + + def preprocessMasks(self, filenames): + """ + Load and preprocess the mask files. + + Image loading code from TensorFlow + https://www.tensorflow.org/api_docs/python/tf/io/read_file + + Parameters: + filenames (tf.string): names of all mask files + + Return: + tf.Dataset: Dataset containing all the mask file data + + """ + raw = tf.io.read_file(filenames) + images = tf.io.decode_png(raw, channels=1) + + #resize images + images = tf.image.resize(images, [self.img_height, self.img_width]) + + #Normalise + images = images / 255. + + #Threshold image to 0-1 + images = tf.where(images > 0.5, 1.0, 0.0) + + return images + + def loadData(self): + """ + Loads and prepocesses all the image and mask data, for the training, testing and validation datasets + + Return: + (tf.Dataset, tf.Dataset, tf.Dataset): 3 datasets containing all the image and mask data + """ + + #train + train_image_data = tf.data.Dataset.list_files(self.train_images_path, shuffle=False) + train_processed_images = train_image_data.map(self.preprocessImages) + + train_mask_data = tf.data.Dataset.list_files(self.train_masks_path, shuffle=False) + train_processed_masks = train_mask_data.map(self.preprocessMasks) + + train_dataset = tf.data.Dataset.zip((train_processed_images, train_processed_masks)) + + #test + test_image_data = tf.data.Dataset.list_files(self.test_images_path, shuffle=False) + test_processed_images = test_image_data.map(self.preprocessImages) + + test_mask_data = tf.data.Dataset.list_files(self.test_masks_path, shuffle=False) + test_processed_masks = test_mask_data.map(self.preprocessMasks) + + test_dataset = tf.data.Dataset.zip((test_processed_images, test_processed_masks)) + + #validate + validate_image_data = tf.data.Dataset.list_files(self.validate_images_path, shuffle=False) + validate_processed_images = validate_image_data.map(self.preprocessImages) + + validate_mask_data = tf.data.Dataset.list_files(self.validate_masks_path, shuffle=False) + validate_processed_masks = validate_mask_data.map(self.preprocessMasks) + + validate_dataset = tf.data.Dataset.zip((validate_processed_images, validate_processed_masks)) + + return train_dataset, test_dataset, validate_dataset \ No newline at end of file diff --git a/recognition/45307915/images/IUNET_Architecture.JPG b/recognition/45307915/images/IUNET_Architecture.JPG new file mode 100644 index 0000000000..0cfe221b80 Binary files /dev/null and b/recognition/45307915/images/IUNET_Architecture.JPG differ diff --git a/recognition/45307915/images/TestDataDiceCoefficientDecreaseBatchSize.JPG b/recognition/45307915/images/TestDataDiceCoefficientDecreaseBatchSize.JPG new file mode 100644 index 0000000000..695d8d85d0 Binary files /dev/null and b/recognition/45307915/images/TestDataDiceCoefficientDecreaseBatchSize.JPG differ diff --git a/recognition/45307915/images/TestDataDiceCoefficientLearningRate.JPG b/recognition/45307915/images/TestDataDiceCoefficientLearningRate.JPG new file mode 100644 index 0000000000..a8419a4f85 Binary files /dev/null and b/recognition/45307915/images/TestDataDiceCoefficientLearningRate.JPG differ diff --git a/recognition/45307915/images/TestDataDiceCoefficientOriginal.JPG b/recognition/45307915/images/TestDataDiceCoefficientOriginal.JPG new file mode 100644 index 0000000000..8e02b26d1c Binary files /dev/null and b/recognition/45307915/images/TestDataDiceCoefficientOriginal.JPG differ diff --git a/recognition/45307915/images/diceCoefficient.png b/recognition/45307915/images/diceCoefficient.png new file mode 100644 index 0000000000..9f38eff78b Binary files /dev/null and b/recognition/45307915/images/diceCoefficient.png differ diff --git a/recognition/45307915/images/diceCoefficientLoss.png b/recognition/45307915/images/diceCoefficientLoss.png new file mode 100644 index 0000000000..d61b3d47db Binary files /dev/null and b/recognition/45307915/images/diceCoefficientLoss.png differ diff --git a/recognition/45307915/images/image.png b/recognition/45307915/images/image.png new file mode 100644 index 0000000000..37cfd7aceb Binary files /dev/null and b/recognition/45307915/images/image.png differ diff --git a/recognition/45307915/images/mask.png b/recognition/45307915/images/mask.png new file mode 100644 index 0000000000..b604258052 Binary files /dev/null and b/recognition/45307915/images/mask.png differ diff --git a/recognition/45307915/images/p1.png b/recognition/45307915/images/p1.png new file mode 100644 index 0000000000..18ac9a3641 Binary files /dev/null and b/recognition/45307915/images/p1.png differ diff --git a/recognition/45307915/images/p10.png b/recognition/45307915/images/p10.png new file mode 100644 index 0000000000..7892c85490 Binary files /dev/null and b/recognition/45307915/images/p10.png differ diff --git a/recognition/45307915/images/p11.png b/recognition/45307915/images/p11.png new file mode 100644 index 0000000000..97f15db64f Binary files /dev/null and b/recognition/45307915/images/p11.png differ diff --git a/recognition/45307915/images/p2.png b/recognition/45307915/images/p2.png new file mode 100644 index 0000000000..d5bd520389 Binary files /dev/null and b/recognition/45307915/images/p2.png differ diff --git a/recognition/45307915/images/p3.png b/recognition/45307915/images/p3.png new file mode 100644 index 0000000000..92a087dae3 Binary files /dev/null and b/recognition/45307915/images/p3.png differ diff --git a/recognition/45307915/images/p4.png b/recognition/45307915/images/p4.png new file mode 100644 index 0000000000..9010bc0880 Binary files /dev/null and b/recognition/45307915/images/p4.png differ diff --git a/recognition/45307915/images/p5.png b/recognition/45307915/images/p5.png new file mode 100644 index 0000000000..22d687a58f Binary files /dev/null and b/recognition/45307915/images/p5.png differ diff --git a/recognition/45307915/images/p6.png b/recognition/45307915/images/p6.png new file mode 100644 index 0000000000..51b9eebf02 Binary files /dev/null and b/recognition/45307915/images/p6.png differ diff --git a/recognition/45307915/images/p7.png b/recognition/45307915/images/p7.png new file mode 100644 index 0000000000..2e9ac13cc1 Binary files /dev/null and b/recognition/45307915/images/p7.png differ diff --git a/recognition/45307915/images/p8.png b/recognition/45307915/images/p8.png new file mode 100644 index 0000000000..96603bbf8e Binary files /dev/null and b/recognition/45307915/images/p8.png differ diff --git a/recognition/45307915/images/p9.png b/recognition/45307915/images/p9.png new file mode 100644 index 0000000000..d23942731e Binary files /dev/null and b/recognition/45307915/images/p9.png differ diff --git a/recognition/45307915/modules.py b/recognition/45307915/modules.py new file mode 100644 index 0000000000..6a29258dd9 --- /dev/null +++ b/recognition/45307915/modules.py @@ -0,0 +1,224 @@ +import tensorflow as tf +from tensorflow.keras.layers import Input, Conv2D, LeakyReLU, Dropout, Add, UpSampling2D, Concatenate, Activation +from tensorflow.keras.models import Model + +import tensorflow_addons as tfa +from tensorflow_addons.layers import InstanceNormalization + +IMAGE_HEIGHT = 192 +IMAGE_WIDTH = 256 + +class ImprovedUNETModel(): + + def __init__(self, img_width=IMAGE_WIDTH, img_height=IMAGE_HEIGHT): + """ Create a new Improved UNET Model instance. + + Described in the paper Brain Tumor Segmentation and Radiomics + https://arxiv.org/pdf/1802.10508v1.pdf + + Parameters: + img_wdith (int): Image Width + img_height (int): Image Height + """ + self.img_width = img_width + self.img_height = img_height + + self.init_filters = 16 + self.padding = 'same' + self.leakyAlpha = 1e-2 + self.dropoutRate = 0.3 + + def contextModule(self, inputs, n_filters): + """ Improved UNET Context Block + + "Each context module is in fact a pre-activation residual block with two + 3x3x3 convolutional layers and a dropout layer (pdrop = 0.3) in between" + + Parameters: + inputs (tf.Tensor): Tensor inputted into the module + n_filters (int): Number of filters for this module + + Return: + tf.Tensor: Tensor output from this module + + """ + x = InstanceNormalization()(inputs) + x = LeakyReLU(alpha=self.leakyAlpha)(x) + x = Conv2D(n_filters, (3,3), padding=self.padding)(x) + + x = Dropout(self.dropoutRate)(x) + + x = InstanceNormalization()(x) + x = LeakyReLU(alpha=self.leakyAlpha)(x) + x = Conv2D(n_filters, (3,3), padding=self.padding)(x) + + return x + + def upsamplingModule(self, inputs, n_filters): + """ Improved UNET Upsampling Block + + "This is achieved by first upsampling the low resolution feature maps, + which is done by means of a simple upscale that repeats the feature voxels + twice in each spatial dimension, followed by a 3x3x3 convolution" + + Parameters: + inputs (tf.Tensor): Tensor inputted into the module + n_filters (int): Number of filters for this module + + Return: + tf.Tensor: Tensor output from this module + + """ + x = UpSampling2D(interpolation='bilinear')(inputs) + x = Conv2D(n_filters, (3,3), padding=self.padding)(x) + x = LeakyReLU(alpha=self.leakyAlpha)(x) + x = InstanceNormalization()(x) + + return x + + def localisationModule(self, inputs, n_filters): + """ Improved UNET Localisation Layer + + "A localization module consists of a 3x3x3 convolution followed + by a 1x1x1 convolution" + + Parameters: + inputs (tf.Tensor): Tensor inputted into the module + n_filters (int): Number of filters for this module + + Return: + tf.Tensor: Tensor output from this module + + """ + x = Conv2D(n_filters, (3,3), padding=self.padding)(inputs) + x = LeakyReLU(alpha=self.leakyAlpha)(x) + x = InstanceNormalization()(x) + + x = Conv2D(n_filters, (1,1), padding=self.padding)(x) + x = LeakyReLU(alpha=self.leakyAlpha)(x) + x = InstanceNormalization()(x) + + return x + + def segmentationLayer(self, inputs, n_filters): + """ Improved UNET Segmentation Layer + + A 1x1 convolution layer + + Parameters: + inputs (tf.Tensor): Tensor inputted into the layer + n_filters (int): Number of filters for this layer + + Return: + tf.Tensor: Tensor output from this layer + + """ + x = Conv2D(n_filters, (1,1), padding=self.padding)(inputs) + x = LeakyReLU(alpha=self.leakyAlpha)(x) + + return x + + def encoderBlock(self, inputs, n_filters, strides=(1,1)): + """ Improved UNET Encoder Block + + 1 convolution layer and 1 context module + + Parameters: + inputs (tf.Tensor): Tensor inputted into the block + n_filters (int): Number of filters for this block + strides ((int, int)): Strides for the convolution + + Return: + tf.Tensor: Tensor output from this block + + """ + + x = Conv2D(n_filters, (3,3), strides=strides, padding=self.padding)(inputs) + convOutput = LeakyReLU(alpha=self.leakyAlpha)(x) + contextOutput = self.contextModule(convOutput, n_filters) + x = Add()([convOutput, contextOutput]) + + return x + + def decoderBlock(self, prev_layer_input, skip_layer_input, n_filters): + """ Improved UNET Decoder Block + + 1 concatenation layer + 1 localisation layer + 1 upsampling layer + + Parameters: + prev_layer_input (tf.Tensor): Tensor input from the previous layer + skip_layer_input (tf.Tensor): Tensor from the skip connection + n_filters (int): Number of filters for this block + + Return: + (tf.Tensor,tf.Tensor): Tensor from the localisation layer + and a Tensor output from this module + + """ + x = Concatenate()([prev_layer_input, skip_layer_input]) + localisation = self.localisationModule(x, n_filters) + x = self.upsamplingModule(localisation, n_filters / 2) + + return localisation, x + + + def modelArchitecture(self): + """ + Defines Improved UNET model network + Described in the paper Brain Tumor Segmentation and Radiomics + + + Return: + tf.Model: Network defined by Improved UNET architecture + + """ + + inputs = Input(shape=(self.img_height,self.img_width,3)) + + # Stack 5 encoders, doubling number of filter in each block. + # Saving a skip connection for each + encoder = self.encoderBlock(inputs, self.init_filters) + skip1 = encoder + encoder = self.encoderBlock(encoder, self.init_filters * 2, strides=(2,2)) + skip2 = encoder + encoder = self.encoderBlock(encoder, self.init_filters * 4, strides=(2,2)) + skip3 = encoder + encoder = self.encoderBlock(encoder, self.init_filters * 8, strides=(2,2)) + skip4 = encoder + encoder = self.encoderBlock(encoder, self.init_filters * 16, strides=(2,2)) + + # Upsampling + upsample = self.upsamplingModule(encoder, self.init_filters * 8) + + # Stack 3 decoders, halving number of filter in each block. + # Saving localisation layers + x, decoder = self.decoderBlock(upsample, skip4, self.init_filters * 8) + localisation1, decoder = self.decoderBlock(decoder, skip3, self.init_filters * 4) + localisation2, decoder = self.decoderBlock(decoder, skip2, self.init_filters * 2) + + # Upsample the first segmentation layer, Add the segmentation layers + segmentation1 = self.segmentationLayer(localisation1, 1) + upScaleSegmentation1 = UpSampling2D(interpolation='bilinear')(segmentation1) + segmentation2 = self.segmentationLayer(localisation2, 1) + firstSum = Add()([upScaleSegmentation1, segmentation2]) + + # Final decoding block + # Concatenation, Convolution, Segmentation + decoder = Concatenate()([decoder, skip1]) + decoder = Conv2D(self.init_filters * 2, (3,3), padding=self.padding)(decoder) + decoder = LeakyReLU(alpha=self.leakyAlpha)(decoder) + segmentation3 = self.segmentationLayer(decoder, 1) + + # Upsample the first segmentation sum, Add the final segmentation layers + upScaleFirstSum = UpSampling2D(interpolation='bilinear')(firstSum) + secondSum = Add()([upScaleFirstSum, segmentation3]) + + outputs = Activation('sigmoid')(secondSum) + + model = Model(inputs, outputs) + +# model.summary() + + return model \ No newline at end of file diff --git a/recognition/45307915/predict.py b/recognition/45307915/predict.py new file mode 100644 index 0000000000..bc766ff713 --- /dev/null +++ b/recognition/45307915/predict.py @@ -0,0 +1,100 @@ +import numpy as np +import matplotlib +import matplotlib.pyplot as plt +import tensorflow as tf + +from modules import ImprovedUNETModel +from dataset import DataLoader +from train import ModelTrainer + +BATCH_SIZE = 16 + +class ModelPredictor(): + + def __init__(self, model, test_dataset, batch_size=BATCH_SIZE): + """ + Create a new model predictor instance to make predictions using a trained model + + Parameters: + model (tf.Model): Trained Improved UNET model + test_datset (tf.Dataset): A dataset containing all the image and mask test data + """ + + self.model = model + self.test_dataset = test_dataset + self.batch_size = batch_size + + + def evaluateModel(self): + """ + Evaluate the test dataset Dice Similarity of the model + + Parameters: + + Return: + """ + + test_batch = self.test_dataset.batch(self.batch_size) + loss, coefficient = self.model.evaluate(test_batch) + + print("Test Data Dice Coefficient: ", coefficient) + + def makePredictions(self): + """ + Using the trained model make predictions on the testing dataset + + Parameters: + + Return: + """ + + test_batch = self.test_dataset.batch(self.batch_size) + test_image, test_mask = next(iter(test_batch)) + predictions = self.model.predict(test_image) + + for i in range(self.batch_size): + + plt.figure(figsize=(10,10)) + + # Plot the test image + plt.subplot(1, 3, 1) + plt.imshow(test_image[i]) + plt.title("Input") + + # Plot the test mask + plt.subplot(1, 3, 2) + mask = test_mask[i] + plt.imshow(mask[:, :, 0], cmap='gray') + plt.title("Ground Truth Mask") + + # Plot the resultant mask + plt.subplot(1, 3, 3) + # Display 0 or 1 for classes + prediction = tf.where(predictions[i] > 0.5, 1.0, 0.0) + plt.imshow(prediction[:, :, 0], cmap='gray') + plt.title("Predicted Mask") + + plt.show() + + +def main(): + + # Data loading and preprocessing + dataLoader = DataLoader() + train_dataset, test_dataset, validate_dataset = dataLoader.loadData() + + # Generate the model + improvedUNETModel = ImprovedUNETModel() + model = improvedUNETModel.modelArchitecture() + + # Train the model + t = ModelTrainer() + model = t.trainModel(train_dataset, test_dataset, validate_dataset, model) + + # Evaluate the model and make predictions + p = ModelPredictor(model, test_dataset) + testEval = p.evaluateModel() + pred = p.makePredictions() + +if __name__ == "__main__": + main() \ No newline at end of file diff --git a/recognition/45307915/train.py b/recognition/45307915/train.py new file mode 100644 index 0000000000..bde35413f5 --- /dev/null +++ b/recognition/45307915/train.py @@ -0,0 +1,139 @@ +import numpy as np +import matplotlib +import matplotlib.pyplot as plt +import tensorflow as tf +from tensorflow.keras.optimizers import Adam +from tensorflow.keras import backend as K + +from modules import ImprovedUNETModel +from dataset import DataLoader + +BATCH_SIZE = 16 +INIT_LEARNING_RATE = 5e-4 +EPOCHS = 20 + +class ModelTrainer(): + + def __init__(self, batch_size=BATCH_SIZE, learning_rate=INIT_LEARNING_RATE, epochs=EPOCHS): + """ + Create a new model trainer instance to train the model + + Parameters: + batch_size (int): Batch size + learning_rate (int): Learning rate for training + epochs (int): Number of epochs for training + """ + self.batch_size = batch_size + self.learning_rate = learning_rate + self.epochs = epochs + + def diceCoefficient(self, y_true, y_pred): + """ + Dice Coefficient + + Credit Karan Jakhar + https://karan-jakhar.medium.com/100-days-of-code-day-7-84e4918cb72c + + Parameters: + y_true (tf.Tensor): true output + y_true (tf.Tensor): output predicted by model + + Return: + tf.Tensor: Dice coefficient based on true output and prediction + + """ + y_true_f = K.flatten(y_true) + y_pred_f = K.flatten(y_pred) + intersection = K.sum(y_true_f * y_pred_f) + dice = (2. * intersection + 1.) / (K.sum(y_true_f) + K.sum(y_pred_f) + 1.) + return dice + + def diceLoss(self, y_true, y_pred): + """ + Dice loss function + + Parameters: + y_true (tf.Tensor): true output + y_true (tf.Tensor): output predicted by model + + Return: + tf.Tensor: Dice coefficient to be used as a loss function + + """ + return 1 - self.diceCoefficient(y_true, y_pred) + + def plotResults(self, history): + """ + Plots Dice Coefficient and Dice Coefficient Loss vs Epoch. + For both training and validation. + + Parameters: + history (History): record of training and validation metrics + + """ + modelHistory = history.history + + #Loss plots + plt.plot(modelHistory['loss']) + plt.plot(modelHistory['val_loss']) + plt.title('Dice Coefficient Loss') + plt.ylabel('Loss (%)') + plt.xlabel('Epoch') + plt.legend(['Training', 'Validation'], loc='upper right') + plt.show() + + #Accuracy plots + plt.plot(modelHistory['diceCoefficient']) + plt.plot(modelHistory['val_diceCoefficient']) + plt.title('Dice Coefficient') + plt.ylabel('DSC') + plt.xlabel('Epoch') + plt.legend(['Training', 'Validation'], loc='upper left') + plt.show() + + def trainModel(self, train_dataset, test_dataset, validate_dataset, model): + """ + Train the model using the training and validation datasets. + Using Dice Coefficient as the loss function. + + Parameters: + train_dataset (tf.Dataset): Dataset containing all the training data + test_dataset (tf.Dataset): Dataset containing all the test data + validate_dataset (tf.Dataset): Dataset containing all the validation data + model (tf.model): Untrained improved UNET model + + Return: + tf.Model: trained improved UNET model + + """ + + # Batch the data + train_batch = train_dataset.batch(self.batch_size) + test_batch = test_dataset.batch(self.batch_size) + validate_batch = validate_dataset.batch(self.batch_size) + + adamOptimizer = Adam(learning_rate=self.learning_rate) + model.compile(optimizer=adamOptimizer, loss=self.diceLoss, metrics=[self.diceCoefficient]) + + results = model.fit(train_batch, epochs=self.epochs, validation_data=validate_batch) + + self.plotResults(results) + + return model + +def main(): + + # Data loading and preprocessing + dataLoader = DataLoader() + train_dataset, test_dataset, validate_dataset = dataLoader.loadData() + + # Generate the model + improvedUNETModel = ImprovedUNETModel() + model = improvedUNETModel.modelArchitecture() + + # Train the model + t = ModelTrainer() + model = t.trainModel(train_dataset, test_dataset, validate_dataset, model) + +if __name__ == "__main__": + main() \ No newline at end of file diff --git a/recognition/ISICs_Unet/README.md b/recognition/ISICs_Unet/README.md index f2c009212e..788ea17b79 100644 --- a/recognition/ISICs_Unet/README.md +++ b/recognition/ISICs_Unet/README.md @@ -1,52 +1,101 @@ -# Segmenting ISICs with U-Net +# Segment the ISICs data set with the U-net -COMP3710 Report recognition problem 3 (Segmenting ISICs data set with U-Net) solved in TensorFlow +## Project Overview +This project aim to solve the segmentation of skin lesian (ISIC2018 data set) using the U-net, with all labels having a minimum Dice similarity coefficient of 0.7 on the test set[Task 3]. -Created by Christopher Bailey (45576430) +## ISIC2018 +![ISIC example](imgs/example.jpg) -## The problem and algorithm -The problem solved by this program is binary segmentation of the ISICs skin lesion data set. Segmentation is a way to label pixels in an image according to some grouping, in this case lesion or non-lesion. This translates images of skin to masks representing areas of concern for skin lesions. +Skin Lesion Analysis towards Melanoma Detection -U-Net is a form of autoencoder where the downsampling path is expected to learn the features of the image and the upsampling path learns how to recreate the masks. Long skip connections between downpooling and upsampling layers are utilised to overcome the bottleneck in traditional autoencoders allowing feature representations to be recreated. +Task found in https://challenge2018.isic-archive.com/ -## How it works -A four layer padded U-Net is used, preserving skin features and mask resolution. The implementation utilises Adam as the optimizer and implements Dice distance as the loss function as this appeared to give quicker convergence than other methods (eg. binary cross-entropy). -The utilised metric is a Dice coefficient implementation. My initial implementation appeared faulty and was replaced with a 3rd party implementation which appears correct. 3 epochs was observed to be generally sufficient to observe Dice coefficients of 0.8+ on test datasets but occasional non-convergence was observed and could be curbed by increasing the number of epochs. Visualisation of predictions is also implemented and shows reasonable correspondence. Orange bandaids represent an interesting challenge for the implementation as presented. +## U-net +![UNet](imgs/uent.png) -### Training, validation and testing split -Training, validation and testing uses a respective 60:20:20 split, a commonly assumed starting point suggested by course staff. U-Net in particular was developed to work "with very few training images" (Ronneberger et al, 2015) The input data for this problem consists of 2594 images and masks. This split appears to provide satisfactory results. +U-net is one of the popular image segmentation architectures used mostly in biomedical purposes. The name UNet is because it’s architecture contains a compressive path and an expansive path which can be viewed as a U shape. This architecture is built in such a way that it could generate better results even for a less number of training data sets. -## Using the model -### Dependencies required -* Python3 (tested with 3.8) -* TensorFlow 2.x (tested with 2.3) -* glob (used to load filenames) -* matplotlib (used for visualisations, tested with 3.3) +## Data Set Structure -### Parameter tuning -The model was developed on a GTX 1660 TI (6GB VRAM) and certain values (notably batch size and image resolution) were set lower than might otherwise be ideal on more capable hardware. This is commented in the relevant code. +data set folder need to be stored in same directory with structure same as below +```bash +ISIC2018 + |_ ISIC2018_Task1-2_Training_Input_x2 + |_ ISIC_0000000 + |_ ISIC_0000001 + |_ ... + |_ ISIC2018_Task1_Training_GroundTruth_x2 + |_ ISIC_0000000_segmentation + |_ ISIC_0000001_segmentation + |_ ... +``` -### Running the model -The model is executed via the main.py script. +## Dice Coefficient -### Example output -Given a batch size of 1 and 3 epochs the following output was observed on a single run: -Era | Loss | Dice coefficient ---- | ---- | ---------------- -Epoch 1 | 0.7433 | 0.2567 -Epoch 2 | 0.3197 | 0.6803 -Epoch 3 | 0.2657 | 0.7343 -Testing | 0.1820 | 0.8180 +The Sørensen–Dice coefficient is a statistic used to gauge the similarity of two samples. +Further information in https://en.wikipedia.org/wiki/S%C3%B8rensen%E2%80%93Dice_coefficient -### Figure 1 - example visualisation plot -Skin images in left column, true mask middle, predicted mask right column -![Visualisation of predictions](visual.png) +## Dependencies -## References -Segments of code in this assignment were used from or based on the following sources: -1. COMP3710-demo-code.ipynb from Guest Lecture -1. https://www.tensorflow.org/tutorials/load_data/images -1. https://www.tensorflow.org/guide/gpu -1. Karan Jakhar (2019) https://medium.com/@karan_jakhar/100-days-of-code-day-7-84e4918cb72c +- python 3 +- tensorflow 2.1.0 +- pandas 1.1.4 +- numpy 1.19.2 +- matplotlib 3.3.2 +- scikit-learn 0.23.2 +- pillow 8.0.1 + + +## Usages + +- Run `train.py` for training the UNet on ISIC data. +- Run `evaluation.py` for evaluation and case present. + +## Advance + +- Modify `setting.py` for custom setting, such as different batch size. +- Modify `unet.py` for custom UNet, such as different kernel size. + +## Algorithm + +- data set: + - The data set we used is the training set of ISIC 2018 challenge data which has segmentation labels. + - Training: Validation: Test = 1660: 415: 519 = 0.64: 0.16 : 0.2 (Training: Test = 4: 1 and in Training, further split 4: 1 for Training: Validation) + - Training data augmentations: rescale, rotate, shift, zoom, grayscale +- model: + - Original UNet with padding which can keep the shape of input and output same. + - The first convolutional layers has 16 output channels. + - The activation function of all convolutional layers is ELU. + - Without batch normalization layers. + - The inputs is (384, 512, 1) + - The output is (384, 512, 1) after sigmoid activation. + - Optimizer: Adam, lr = 1e-4 + - Loss: dice coefficient loss + - Metrics: accuracy & dice coefficient + +## Results + +Evaluation dice coefficient is 0.805256724357605. + +plot of train/valid Dice coefficient: + +![img](imgs/train_and_valid_dice_coef.png) + +case present: + +![case](imgs/case%20present.png) + +## Reference +Manna, S. (2020). K-Fold Cross Validation for Deep Learning using Keras. [online] Medium. Available at: https://medium.com/the-owl/k-fold-cross-validation-in-keras-3ec4a3a00538 [Accessed 24 Nov. 2020]. + +zhixuhao (2020). zhixuhao/unet. [online] GitHub. Available at: https://github.com/zhixuhao/unet. + +GitHub. (n.d.). NifTK/NiftyNet. [online] Available at: https://github.com/NifTK/NiftyNet/blob/a383ba342e3e38a7ad7eed7538bfb34960f80c8d/niftynet/layer/loss_segmentation.py [Accessed 24 Nov. 2020]. + +Team, K. (n.d.). Keras documentation: Losses. [online] keras.io. Available at: https://keras.io/api/losses/#creating-custom-losses [Accessed 24 Nov. 2020]. + +262588213843476 (n.d.). unet.py. [online] Gist. Available at: https://gist.github.com/abhinavsagar/fe0c900133cafe93194c069fe655ef6e [Accessed 24 Nov. 2020]. + +Stack Overflow. (n.d.). python - Disable Tensorflow debugging information. [online] Available at: https://stackoverflow.com/questions/35911252/disable-tensorflow-debugging-information [Accessed 24 Nov. 2020]. diff --git a/recognition/XUE4645768/Readme.md b/recognition/XUE4645768/Readme.md index 94bc1848c0..36250adaa3 100644 --- a/recognition/XUE4645768/Readme.md +++ b/recognition/XUE4645768/Readme.md @@ -53,52 +53,6 @@ python gcn.py Warning: Please pay attention to whether the data path is correct when you run the gcn.py. -# Training - -Learning rate= 0.01 -Weight dacay =0.005 - -For 200 epoches: -```Epoch 000: Loss 0.2894, TrainAcc 0.9126, ValAcc 0.8954 -Epoch 001: Loss 0.2880, TrainAcc 0.9126, ValAcc 0.895 -Epoch 002: Loss 0.2866, TrainAcc 0.9126, ValAcc 0.8961 -Epoch 003: Loss 0.2853, TrainAcc 0.9132, ValAcc 0.8961 -Epoch 004: Loss 0.2839, TrainAcc 0.9137, ValAcc 0.8961 -Epoch 005: Loss 0.2826, TrainAcc 0.9141, ValAcc 0.8963 -Epoch 006: Loss 0.2813, TrainAcc 0.9146, ValAcc 0.8956 -Epoch 007: Loss 0.2800, TrainAcc 0.9146, ValAcc 0.8956 -Epoch 008: Loss 0.2788, TrainAcc 0.9146, ValAcc 0.8959 -Epoch 009: Loss 0.2775, TrainAcc 0.9146, ValAcc 0.8970 -Epoch 010: Loss 0.2763, TrainAcc 0.915, ValAcc 0.8974 -Epoch 011: Loss 0.2751, TrainAcc 0.915, ValAcc 0.8972 -Epoch 012: Loss 0.2739, TrainAcc 0.915, ValAcc 0.8976 -Epoch 013: Loss 0.2727, TrainAcc 0.9157, ValAcc 0.8979 -Epoch 014: Loss 0.2716, TrainAcc 0.9157, ValAcc 0.8983 -Epoch 015: Loss 0.2704, TrainAcc 0.9161, ValAcc 0.8990 -Epoch 016: Loss 0.2693, TrainAcc 0.9168, ValAcc 0.8988 -Epoch 017: Loss 0.2682, TrainAcc 0.9181, ValAcc 0.8990 -Epoch 018: Loss 0.2671, TrainAcc 0.9179, ValAcc 0.8990 -Epoch 019: Loss 0.2660, TrainAcc 0.9179, ValAcc 0.8992 -Epoch 020: Loss 0.2650, TrainAcc 0.9188, ValAcc 0.8996 -...... -Epoch 190: Loss 0.1623, TrainAcc 0.9553, ValAcc 0.9134 -Epoch 191: Loss 0.1619, TrainAcc 0.9555, ValAcc 0.9134 -Epoch 192: Loss 0.1615, TrainAcc 0.9555, ValAcc 0.9132 -Epoch 193: Loss 0.1611, TrainAcc 0.9557, ValAcc 0.9130 -Epoch 194: Loss 0.1607, TrainAcc 0.9562, ValAcc 0.9130 -Epoch 195: Loss 0.1603, TrainAcc 0.9559, ValAcc 0.9130 -Epoch 196: Loss 0.1599, TrainAcc 0.9562, ValAcc 0.9126 -Epoch 197: Loss 0.1595, TrainAcc 0.9562, ValAcc 0.9123 -Epoch 198: Loss 0.1591, TrainAcc 0.9562, ValAcc 0.9123 -Epoch 199: Loss 0.1587, TrainAcc 0.9562, ValAcc 0.9123``` - -For test accuracy:around 0.9 - -# TSNE -For the test:iteration=500, with lower dimension to 2 - - - ```python