Machine generated Heatmaps [ Discussion ]

[leoarc]

So I found this interesting paper and wanted to know your thoughts :

A Fast and Refined Cancer Regions Segmentation Framework in Whole-slide Breast Pathological Images

TL;DR version :

The aim is to automatically produce heatmaps of WSI and extract polygons of lesion regions.

The problem is WSI are huge in size and applying segmentation models(which are usually trained on small patches ) to them is difficult and time consuming. To overcome these problems the paper proposes :

• applying OTSU segmentation to remove most of the non-tissue background quickly.

• Use a classification model (Inception_v3) to get rough predicton of tumor areas .

• Use a semantic segmentation model(DCNN) on those preselected areas to get refined segmentation of tumor regions.

The claim is they produce dense heatmaps of 1/8 size in much lesser time for whole slides.

One we obtain the heatmaps we can work on coverting them into the format (json ?) that is used in caMicroscope and support functions that are available for normal heatmaps . May be if this works we can also add support for uploading user models for the task ( kind of like how segment app started with watershed algorithm only and was then extented ) .

I thought since applying ML models to whole slides is already in caMicroscope's plans ( through the ROI project ) this maybe one avenue where we can extend our features.I know this is a big project and it is complicated and I have oversimplified a lot a things above ( Also all my statements are assuming the claims of the paper are true and reproducible ). So only if you think it will be useful then I will look into it more.

As an initial step if you think this is a viable idea, I will start working on the segmentation model alone and add it to this repo ( which is why I was going through papers and found this ) .

@Insiyaa @birm would love to hear your thoughts on this .

PS : Just putting out a random idea . This can be unrelated/useless to caMicroscope. In that case feel free to close this :p .

[birm]

This is an interesting concept, though, as you mention, it's probably difficult to have a model work on a whole slide within the browser, at least at a high zoom level.
That said, you're absolutely welcome to make a proof of concept, and I don't think we have any reason to reject models here even if they end up being too big to actually be a tfjs model.
I'll also reach out to a colleague who has been working on something vaguely similar to this. (and also actually read the paper 😅 )