Crash reporting for MONAI

[zomglings]

Discussed this with @Nic-Ma over email and he suggested we create a discussion thread on the GitHub repo.

@SophiaAr and I are building a product called Bugout (https://bugout.dev) which helps library maintainers understand what their users are experiencing, especially when it comes to errors raised from the library or because there is some friction in the library interface.

We took the liberty of making a short demo of how this would work for MONAI using the MedNIST classification tutorial: https://www.loom.com/share/ca328b45221541b797ccf6bc00adfee9

Do you think this kind of crash reporting could be helpful for MONAI?

[zomglings]

PS: The reporting code lives on my fork of MONAI. Here is a diff between my branch and MONAI's master branch: master...zomglings:reporting-integration

[charliebudd]

From the look of things we could expand this to report usage analytics aswell. Might be worth double checking though so we dont end up with multiple reporting integrations? However, 1000 events a month (free tier for bugout) wouldnt be enough to support usage analytics, possibly not even unique installs.

[zomglings]

@charliebudd : You could certainly use Bugout for usage analytics as well. Do you know what kind of information you would like in your analytics?

Out of the box, you can get insight into:

1. Clients (unique installs)
2. Sessions per client
3. Operating systems (down to things like kernel version)
4. Python versions
5. MONAI versions
6. Errors encountered (exception type)

Is there any other kind of information you would like?

If MONAI would be willing to share (de-identified) reports publicly, we we would be happy to increase the free tier limit for MONAI.

[zomglings]

torch version and CUDA version would also be pretty straightforward.

[zomglings]

Just checking in @charliebudd , @wyli . Did you have a chance to think about this?

[zomglings]

Checking in again. :)

Actually, we just released a cool feature where you can view user errors (including stack traces) right from where they are raised in your code base. Makes debugging a lot more pleasant.

https://dev.to/zomglings/see-the-errors-your-users-are-experiencing-from-your-ide-live-19id