Make 2-proc cuda distributed remap soft-fail

[charleskawczynski]

This PR makes the distributed remapping with CUDA (2 processes) job soft-fail for now, since it's failing pretty consistently.

[Sbozzolo]

I worry that this will just hide the problem and issue will not be addressed.

In the past, this job showed no signs of flakiness, so we must have introduced something new relatively recently.

Maybe before making this a soft fail, we can try spending two hours to see if we can reproduce the problem reliably. If in two hours of work, we cannot find come up with a reproducer, we can turn this into a soft fail.

[charleskawczynski]

I worry that this will just hide the problem and issue will not be addressed.

In the past, this job showed no signs of flakiness, so we must have introduced something new relatively recently.

I understand, but this has been an issue as far back as December 20, 2024 (#2108). This is why I opened the issue, to make sure that we don't lose track.

Maybe before making this a soft fail, we can try spending two hours to see if we can reproduce the problem reliably. If in two hours of work, we cannot find come up with a reproducer, we can turn this into a soft fail.

In this build: https://buildkite.com/clima/climacore-ci/builds/4912, I retried the same test multiple times, and it's clearly a race condition since the number of tests that passed were different on different attempts. So, it's not clear to me that we can make a reproducer aside from running the tests multiple times. I suppose we could try putting it inside a loop and call that a reproducer.

I'm happy to take a look over it, my thought was that we could merge this in the mean time, since it kind of clouds the overall CI status.