ci/gpu: debug skipped cache

[Borda]

What does this PR do?

since cached is not restored we are missing the 40% speed-up and takes 5min without nay benefit of restoring

error:  cannot create tests/_cache-references/tests/_cache-references/.unittests.classification.test_auroc._reference_sklearn_auroc_multilabel_d5af40e0c0b49514985c2869dfaf092ce9b138b8e0ee41c260fb92a8c83c810a
        No space left on device

Before submitting

• Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

---

📚 Documentation preview 📚: https://torchmetrics--2709.org.readthedocs.build/en/2709/

[Borda]

checking the cache generated on the main is about 500MB

so archived 4.3GB is suspicious

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 37%. Comparing base (c8ebda5) to head (4b0f3d3).
Report is 1 commits behind head on master.

❗ There is a different number of reports uploaded between BASE (c8ebda5) and HEAD (4b0f3d3). Click for more details.

HEAD has 76 uploads less than BASE

Flag	BASE (c8ebda5)	HEAD (4b0f3d3)
Windows	6	3
python3.8	6	3
cpu	39	20
torch1.13.1+cpu	6	3
macOS	7	4
python3.10	9	5
torch2.0.1	3	2
torch2.4.0+cpu	2	1
python3.11	6	3
torch2.4.0	2	1
torch1.12.1+cpu	2	1
Linux	26	13
python3.9	18	9
torch1.11.0+cpu	2	1
torch1.10.2+cpu	2	1
torch2.0.1+cpu	6	3
torch1.13.1	2	1
torch2.3.1+cpu	4	2
torch2.1.2+cpu	2	1
torch2.2.2+cpu	4	2
torch2.4.0+cu121	2	1

Additional details and impacted files

@@           Coverage Diff            @@
##           master   #2709     +/-   ##
========================================
- Coverage      69%     37%    -32%     
========================================
  Files         316     329     +13     
  Lines       17900   18064    +164     
========================================
- Hits        12328    6613   -5715     
- Misses       5572   11451   +5879