torchmetric optimizations

[czmrand]

add nan_strategy "disable" to disable nan checks
set default in update to None to prevent sync event
see https://medium.com/@chaimrand/efficient-metric-collection-in-pytorch-avoiding-the-performance-pitfalls-of-torchmetrics-0dea81413681 for motivation

What does this PR do?

Naive use of torch metrics creates multiple CPU-GPU sync events which slows down training.
This is demonstrated in https://medium.com/@chaimrand/efficient-metric-collection-in-pytorch-avoiding-the-performance-pitfalls-of-torchmetrics-0dea81413681

Before submitting

• [] Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
  No
• Did you read the contributor guideline, Pull Request section?
  Yes
• Did you make sure to update the docs?
  NR
• Did you write any new necessary tests?
  NR

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

---

📚 Documentation preview 📚: https://torchmetrics--2943.org.readthedocs.build/en/2943/

[Borda]

wow, thank you @czmrand for this work!
could you pls add some demo/benchmark case where we can see the improvement?

[williamFalcon]

@czmrand nice job! let's get these optimizations in @Borda @SkafteNicki :)

[codecov]

Codecov Report

Attention: Patch coverage is 50.00000% with 12 lines in your changes missing coverage. Please review.

Project coverage is 70%. Comparing base (a7c8ee0) to head (a8cf385).
Report is 5 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #2943    +/-   ##
=======================================
- Coverage      70%     70%    -0%     
=======================================
  Files         346     332    -14     
  Lines       19172   19016   -156     
=======================================
- Hits        13428   13258   -170     
- Misses       5744    5758    +14