Slow Training Speed in PatchCore Model

[silentway01]

Hi, I’m using the PatchCore model from anomalib on a custom dataset and noticed a performance issue during core-set subsampling.

Issue:

• With 1000 normal images, the training runs at 76it/s and completes in about 10 minutes.
• With 2000 normal images, the speed drops to 1it/s, with an estimated time of 50 hours.

The only difference is the number of normal images.
Any suggestions to speed up the core-set subsampling step?
Thanks for your help!

[blaz-r]

I'm not sure if this has an "easy" fix.
Patchcore is an unsupervised method, so it only uses normal images during training, so this is an important "parameter" of the method.
Since Patchcore is a memory bank method (it builds core-set memory bank), it's training time and memory-consumption is dependent on image count. So you can expect longer training time and higher memory usage with more images.
However, the slowdown you are seeing is a bit too extreme. I think that it might be a problem of your memory (RAM), since 1000 images can still fit in the memory, but with 2000, images are then offloaded to your disk (swap space) which then becomes very slow.

As I said, I'm not sure how to easily fix this. One suggestion would be that you find the largest number of images that can still fit in RAM (i.e. the it/s stays around 76).
Another thing is that Patchcore probably doesn't benefit much from many more images, due to the way it works (the core-set is a subset that best cover the whole set of patch features). Adding images probably yields diminishing improvements, since more and more elements in core-set become redundant.
That's why reducing the image count might be a totally viable solution. You can also verify the impact on performance by starting with a small train set and systematically increasing the image count while observing some performance metric on validation set (e.g. image AUROC).

Hope that helps 😄

[silentway01]

Thanks so much for your explanation and helpful suggestions!😄