test-ibmcloud-rhel7-x64-1: Mere 13gb of free storage may be causing repro test to fail

[adamfarley]

Please set the title to indicate the test name and machine name where known.

To make it easy for the infrastructure team to repeat and diagnose, please
answer the following questions:

• test suite/name (e.g, BUILD_LIST, TARGET, CUSTOM_TARGET)?
  Rebuild_Same_JDK_Reproducibility_Test_0
• a link into recent Test_ job on https://ci.adoptium.net which showed the failure
  https://ci.adoptium.net/job/Grinder/11811/consoleFull
• Hyperlink to re-run in Grinder:
  Grinder rerun link
• Is there an existing issue elsewhere covering this?
  It was first seen here.
• Which machine(s) does it work on?
  test-ibmcloud-ubuntu1604-x64-1 - This test passed here on November 6th. Link.
• Which machine(s) does it fail on?
  test-ibmcloud-rhel7-x64-1

Any other details:
The repro test completes the build and then fails with this error mid-sentence:

tee: write error

[adamfarley]

@steelhead31 - FYI

[steelhead31]

@adamfarley is there anyway to exclude the repro tests from running on this node? it has a small disk, and even with extreme levels of housekeeping, we'll struggle to get too much more.. if we cant, then we'll ask IBM if they can increase the size of the disk, as we don't have access to manage this machine.

[sxa]

There is a 100GiB disk that looks unused at the moment - that could be allocated to a separate docker or workspace file system.
/var/lib/docker is currently chewing about 47GiB so I suspect it just needs a bit of a prune - another case where we need to start being better about docker disk management I feel.

Having said that, since docker system df is not showing information anywhere near du on the docker directory, it might be a good use case for shutting down the service, wiping that directory and starting it again ... I've done a cache prune but that has only yielded another 3GiB. Running another test job at https://ci.adoptium.net/job/Grinder/11842/

[steelhead31]

this machine has now been housekept, and the problematic overlay2 folder recreated, test running here https://ci.adoptium.net/job/Grinder/11848/ .  some additional housekeeping is being created and put in place in this issue.. #3797 ,