Heavy Augmentations + More Epochs

[alexriedel1]

Hello!
I'm wondering if anyone has experimented with heavy augmentations and therefore increasing the number of training epochs.
If so, did you get any performance gains and/or information about conflicting augmentations?

Thanks!

[samet-akcay]

@alexriedel1, increasing number of epochs would help for cflow and stfpm. You might want to either remove early stopping, or increase the patience in early stopping.

Other algorithms don't require CNN training and only performs feature extraction. That's why the number of epochs are set to 1.

For the augmentations, I remember that @blakshma performed some experiments to see how augmentations impact the performance. Maybe he could comment. @blakshma ?

Also, @alexriedel1, would you be alright if we move this to Discussions that we recently enabled?

[alexriedel1]

@samet-akcay
Thanks for your insights! Yes you can totally move this to discussion.

For the feature extracting methods I think increasing the dataloader with augmented samples will have the same effect as multiple epoch learning

[blakshma]

@alexriedel1 we did experiment some experiments with augmentation. For the MVTec AD dataset, augmentation policies needs to be carefully crafted for each category. For example, rotation deteriorates performance for transistor category but improves it for hazel_nut category.

Overall, the performance improvement that we get heavy augmentation is very small for MVTec case. However, given a more complex dataset, augmentation is warranted to provide significant performance improvement.

[mblagu]

Slightly off topic so I apologize beforehand if I should have started another thread, but did anyone experience GPU memory issues when using augmentations? I am just trying to add some Gaussian noise to my images and CUDA runs out of memory (I dropped the batch size all the way down to 1 and this error still occurs). When not using any augmentations, I have no issues running training sessions with increased batch sizes.

[alexriedel1]

Hi, please show your config file that gets you the memory errors

[mblagu]

The thing is, I didn't modify any config file, rather I am trying to pass the transforms, root/good/bad directories and normalization methods to the MVTec datamodule constructor. Anyway, what I am trying to do is to see how little data an how much noise will significantly impact the results of the Patchcore model, as the dataset I am trying to make it work on is a bit noisier than MVTEC and there is more variation between individual non-anomalous samples. Running this code without passing the transforms to the MVTec datamodule works perfectly even with increased number of workers and increased batch size.

I also tried setting the environment variable PYTORCH_CUDA_ALLOC_CONF to `max_split_size_mb:' (value: 512, 256 or 128) in hopes of preventing RAM segmentation which also didn't work.

You can see the code below:

# Declare train and val trainsforms
train_transforms = []
val_transforms = []

# Add normalization
if normalization == InputNormalizationMethod.IMAGENET:
    train_transforms.append(A.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)))
    val_transforms.append(A.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)))
elif normalization == InputNormalizationMethod.CLIP:
    train_transforms.append(A.Normalize(mean=(0.48145466, 0.4578275, 0.40821073), std=(0.26862954, 0.26130258, 0.27577711)))
    val_transforms.append(A.Normalize(mean=(0.48145466, 0.4578275, 0.40821073), std=(0.26862954, 0.26130258, 0.27577711)))
elif normalization == InputNormalizationMethod.NONE:
    train_transforms.append(A.ToFloat(max_value=255))
    val_transforms.append(A.ToFloat(max_value=255))
else:
    msg = f"Unknown normalization method: {normalization}"
    raise ValueError(msg)

# Set noise for training and testing
if noise_combination[0]:
    train_transforms.append(A.GaussNoise(var_limit=variance_range, always_apply=False, p=1.0))
else:
    train_transforms.append(A.NoOp())

if noise_combination[1]:
    val_transforms.append(A.GaussNoise(var_limit=variance_range, always_apply=False, p=1.0))
else:
    val_transforms.append(A.NoOp())

# Add ToTensorV2 transform
train_transforms.append(ToTensorV2())
val_transforms.append(ToTensorV2())

# Convert to A.Compose
train_transform_config = A.Compose(train_transforms)
val_transform_config = A.Compose(val_transforms)

# prepare MVTEC datamodule
mvtec_datamodule = MVTec(
    root="./data_mount/mvtec",
    category="screw",
    image_size=224,
    train_batch_size=1,
    eval_batch_size=1,
    num_workers=1,
    task="segmentation",
    # transform_config_train=train_transform_config,
    # transform_config_eval=val_transform_config,
    normalization=normalization, # has no effect when transformations are provided
)
mvtec_datamodule.prepare_data()
mvtec_datamodule.setup()  

# Set random seed for reproductibility
random.seed(42)
# Calculate the number of samples to keep
num_samples_to_keep = int(len(mvtec_datamodule.train_data) * train_subset_ratio)
# Generate random indices from the training set
random_indices = random.sample(range(len(mvtec_datamodule.train_data)), num_samples_to_keep)
# Subsample the training_data
mvtec_datamodule.train_data = mvtec_datamodule.train_data.subsample(indices=random_indices)

# Initialize Model
pathcore_model = Patchcore()

# Initialize Engine callbacks and Engine
callbacks = [
    ModelCheckpoint(
        mode="max",
        monitor="image_AUROC",
    ),
    EarlyStopping(
        monitor="image_AUROC",
        mode="max",
        patience=3,
    )
]

engine = Engine(
    callbacks=callbacks,
    pixel_metrics=["AUROC", "AUPR", "F1Score"],
    image_metrics=["AUROC", "AUPR", "F1Score"], 
    accelerator="auto",  # \<"cpu", "gpu", "tpu", "ipu", "hpu", "auto">,
    devices=1,
    threshold="F1AdaptiveThreshold",
    # logger=wandb_logger, # passed through kwargs to trainer        
)

# Train and Test model
engine.train(model=pathcore_model, datamodule=mvtec_datamodule)

[alexriedel1]

If you pass tranforms to the MVTec Datamodule class the image size will be ignored. You then have to add the image size transforma manually like you add your custom transforms.

anomalib/src/anomalib/data/base/datamodule.py

Lines 255 to 266 in 07bf60b

	def train_transform(self) -> Transform:
	"""Get the transforms that will be passed to the train dataset.
	If the train_transform is not set, the engine will request the transform from the model.
	"""
	if self._train_transform:
	return self._train_transform
	if getattr(self, "trainer", None) and self.trainer.model and self.trainer.model.transform:
	return self.trainer.model.transform
	if self.image_size:
	return Resize(self.image_size, antialias=True)
	return None

[mblagu]

Yes, of course this was the issue, thanks for the quick responses!