Leverage existing iterator features

[jcohenadad]

Possible inspiration:

• Case Iterator module
  • project was inactive for a while (last commit to .py file 3 years ago)
  • but recent development
  • limitation: missing feature (eg: range of intensity for manual segmentation)
• MONAILabelReviewer (part of the MONAI label extension)
  • project started approximately at the same time as slicerCART
  • need to investigate further
  • more developed towards active learning, with server pointing to the data
• SegmentationReview.
  • Allows to review a list of volumes/masks contained in a single folder.
  • Likert scale and Segment Editor Widget

[laurentletg]

SlicerCaseIterator: if you want to try this module, I made a small utility script to get the required CSV that can then be loaded into Slicer (drag and drop the CSV file, which will be converted to a table node). https://github.com/laurentletg/slicer_case_iterator_csv_utility

Also contains 2 examples in data (MR examples from Slicer).

...will report back with my findings later..

[laurentletg]

Slicer Case Iterator module review

feel free to edit

Video demonstrating how to use the simple_csv_iteration option using the csv generated by the script in the previous comment. This uses a pair of volume (T1) and masks. I did not try the option with other columns (see TODO).

Pros:

• Allows iteration through a batch of cases
• Save segmentation with the annotator name as a suffix (optional)
• Allows to control segmentation visualization (fill vs only margins)

Cons:

• Requires to create a csv file with paths for each image, mask (option to load more than one volume (e.g. MR weighting? not tested yet).
• No possibility to go directly to a specific case (unlike the list widget).
• Maintained ? (last commit to module code 3 years ago)
• Does not impose structure: Does not allow for the generation of specific labels or image-level classification widgets. The lack of constraints may lead to errors and difficult QC. I think it is better to start with clear annotation guidelines that are incorporated in the annotation tool to minimize tedious data cleaning after all annotations are completed. No ontology (SNOMED) for labels, no file structure (BIDS), etc.

TODO:

• : test functionality with multiple other columns (other series for same subject ?)
  
• Test mask_comparison mode (no documentation on the README
  

[laurentletg]

MONAI label & MONAIlabel reviewer modules

Feel free to edit

MONAI label

YouTube example

Pros:

• Allows for DL/GPU-driven active learning with time-saving potential. An alternative approach is to annotate a small batch and train a model then refine predictions.

Cons:

• Requires a GPU-computer: From the documentation: It is an open-source and easy-to-install ecosystem that can run locally on a machine with single or multiple GPU. The requirement for use on a GPU computer limits the accessibility of the module (e.g. trainees doing annotations on their own laptop computers or distribution annotation assignments to a large group of annotators, unless they can connect to a remote GPU server).
• Inference may be long, need to wait for it to complete before starting annotation. For example: vertebra segmentation inference runs during at least 30 seconds (not sure of the GPU used in the demo).
• Initial set-up not user friendly for users without minimal computer tool knowledge (minimally requires entry-level CLI and python skills).

MONAI label reviewer

• cannot find documentation! The slicer module points to the main MONAI label GitHub page but there is no specific instruction on how to use this module.
• How to set up the server ?

Issues:

Seems like this needs to be configured similarly to MONAI label (GPU needed ?)
Project-MONAI/MONAILabel#1290
Project-MONAI/MONAILabel#1300

Is the module actively maintained or stable enough ? from issue 1300: But we should rethink a complete re-write of the Reviewer which was initially committed just as a prototype

[AcastaPaloma]

There seems to be very few details and documentation about this extension, but here is
https://gitlab.com/flywheel-io/scientific-solutions/app/Slicerflywheelcaseiterator, another iterator.
I downloaded the extension, but an API key from Flywheel seems necessary. Should I dig more into it?

https://flywheel.io/

Maybe this could be an interesting implementation in SlicerCART? -maybe connecting SlicerCART with datasets already online? Could be really practical.

[jcohenadad]

@AcastaPaloma Flywheel is a lucrative company, offering 'packaged' solutions into proprietary platform, so this is not the kind of solutions we are looking into.

[laurentletg]

SegmentationReview

Repo
Paper

Note that the code includes all widget and methods in the Widget class similar to the current SlicerCART version, i.e. no methods in the Logic class and no tests - still the module is distributed through the Slicer Extension Manager.

Pros:

• Good documentation on their GitHub repo
• Simple
• Navigation through a list of volume/mask pairs
• Allows quick assessment of segmentation quality using a 5-category Likert scale (good for prospective real-time validation by clinicians)
• Embeds the SegmentEditor widget and saves a version of the edited segmentations.

Cons:

• Navigation is only forward, no possibility to go back of easily jump to a specific case
• Not configurable, what if you want to change the Likert scale ? This is hardcoded.
• I cannot find the edited segmentation file once saved (I think there is a bug related to the edited path construction)
• No support for multiple series.

[maxradx]

Hi,

I included your comments above in a review google doc that can be used for further comments/discussion about existing tools.

I also reviewed the different existing Slicer tools and other open source platforms aimed to facilitate manual segmentation, and listed them in the google doc mentioned above. I added comments related to your findings above mentioned (@jcohenadad @laurentletg @AcastaPaloma). Since leveraging existing solutions require different approaches, further research work has been done through literature overview, Github research by topics and through the Web. Librarian from Polytechnique Montreal has been consulted for methodology input.

Summary:
3D Slicer extensions/modules:

• Basic modules exist in the Slicer extension manager and code from other sources (e.g. Github). However, many limitations are found in each of them: too basic (not an all-in solution), loading multiple format not supported (e.g. only DICOM … or from a server), or are not still being developed/supported.
• SlicerRT (publication) is a good example of what would bring a module like SlicerCART to the Slicer community. The purpose of SlicerRT (radiotherapy oriented-tasks) is not the same as SlicerCART.

Other relevant open-source existing platform not mentioned above in this issue:

• “Medical Imaging Interaction Toolkit (MITK)”: this is a very good solution (from my perspective), enabling most of the main scoped SlicerCART features. However, some key features of SlicerCART (e.g. revision steps management, segmentation stats like time, and classification task GUI) are not available. Moreover, there is >100K lines of code to my understanding, which makes customization less likely. Slicer is an application that is widely used and actively developed for different goals, so a module like SlicerCART would help Slicer evolving. See more comments in the google doc.
• imageJ : this open-source software continues to be actively developed since the 1990s, and enables the use of macros (e.g. for batch processing). However, it does not explicitly offer classification or revision steps opportunities. This can be a good alternative for measures and angle measurement of a large dataset.

Any more suggestion?

Cheers,

Maxime B.