cleanup redirected stdout crash before re-redirecting

[sbailey]

This PR fixes #2269 by merging any per-rank log files leftover from a previous crash before proceeding with a new stdout/stderr log redirect per-rank. Previously these logfiles would get overwritten, losing the record of the crash.

I tested this with $CFS/desi/users/sjbailey/dev/stdout_redirect/mpi_test_redirect which purposefully crashes a requested rank. e.g.

# run without crashing any ranks; generates a blat.log file
$> srun -n 4 python mpi_test_redirect
Starting up; will crash rank -1
INFO:parallel.py:465:stdouterr_redirected: Begin log redirection to blat.log at Thu Jul 25 17:46:51 2024
INFO:parallel.py:524:stdouterr_redirected: End log redirection to blat.log at Thu Jul 25 17:46:51 2024
All done
$> ls
blat.log  mpi_test_redirect

# run while crashing rank 2, leaves behind backup of previous run and per-rank files from this run
$> srun -n 4 python mpi_test_redirect 2
Starting up; will crash rank 2
INFO:parallel.py:465:stdouterr_redirected: Begin log redirection to blat.log at Thu Jul 25 17:48:12 2024
srun: error: nid200053: task 2: Segmentation fault
srun: Terminating StepId=28590762.10
slurmstepd: error: *** STEP 28590762.10 ON nid200053 CANCELLED AT 2024-07-26T00:48:15 ***
srun: error: nid200053: tasks 0-1,3: Terminated
srun: Force Terminated StepId=28590762.10
$> ls
blat.log.0  blat.log-rank0  blat.log-rank1  blat.log-rank2  blat.log-rank3  mpi_test_redirect

# run again without crashing; will cleanup blat.log-rank* files, back them up to blat.log.1
# then generate blat.log
$> srun -n 4 python mpi_test_redirect
Starting up; will crash rank -1
INFO:parallel.py:465:stdouterr_redirected: Begin log redirection to blat.log at Thu Jul 25 17:50:24 2024
INFO:parallel.py:524:stdouterr_redirected: End log redirection to blat.log at Thu Jul 25 17:50:24 2024
All done
$> ls
blat.log  blat.log.0  blat.log.1  mpi_test_redirect

Detail: previously the per-rank files had names like output.log_0 which was confusingly similar to the backup names like output.log.0. I changed this to output.log-rank0 instead to make it clearer which files are leftover per-rank files and which are the backups.

Note: if a crash happens cleanly with an exception, stdouterr_redirected recovers and is able to merge the per-rank files. The leftover per-rank logfiles case comes from hard crashes like OOM or in this test case, from a purposeful segfault.

[akremin]

Thanks for this useful update to make the pipeline logging more robust. I have two very minor comments. Both should take two seconds to implement or they can both be ignored, as I don't feel strongly about either of them.