App=S2S does not compile with Cheyenne.gnu; failure is not reported in regression test

[DeniseWorthen]

Description

The coupled model does not compile on Cheyenne.gnu after the recent upgrade to a newer version. The compile failure is not reported so it appears that the RT for Cheyenne.gnu is successful.

To Reproduce:

Cheyenne.gnu
Check the compile_006/err file in the regression test run directory.
The RT log for PR #639 cheyenne.gnu shows only 5 compile jobs; the compile jobs for app=s2s are missing but the RT job is reported as being successful.

Additional context

Trying to compile the following in Cheyenne.gnu:

COMPILE | -DAPP=S2S -DCCPP_SUITES=FV3_GFS_2017_coupled,FV3_GFS_v16_coupled,FV3_GFS_v16_coupled_nsstNoahmpUGWPv1 -DDEBUG=ON        | - wcoss_cray                            | fv3 |

gives the attached err log file.

err.txt

[DusanJovic-NOAA]

Please run:

RT_COMPLIER=gnu ./rt.sh -n cpld_debug -l rt_gnu.conf -e

on Cheyenne and see what's the slurm status at the end of log_hera.gnu/compile_001.log

[DeniseWorthen]

TEST 001 compile is submitted
1 min. TEST 001 compile is waiting in a queue,  status: Q jobid 9550867
2 min. TEST 001 compile is waiting in a queue,  status: Q jobid 9550867
3 min. TEST 001 compile is waiting in a queue,  status: Q jobid 9550867
4 min. TEST 001 compile is waiting in a queue,  status: Q jobid 9550867
5 min. TEST 001 compile is waiting in a queue,  status: Q jobid 9550867
6 min. TEST 001 compile is running,  status: R jobid 9550867
qstat: 9550867.chadmin1.ib0.cheyenne.ucar.edu Job has finished, use -x or -H to obtain historical job information
qstat: 9550867.chadmin1.ib0.cheyenne.ucar.edu Job has finished, use -x or -H to obtain historical job information
7 min. TEST 001 compile is finished,  status: - jobid 9550867
+ cp '/glade/scratch/worthen/FV3_RT/rt_44866/compile_001/compile_*_time.log' /glade/work/worthen/ufs-weather-model-gnu/tests/log_cheyenne.gnu
cp: cannot stat '/glade/scratch/worthen/FV3_RT/rt_44866/compile_001/compile_*_time.log': No such file or directory

[DusanJovic-NOAA]

@DeniseWorthen Thanks. I see Cheyenne is not using slurm but pbs scheduler.
Can you please make this change in rt_utils.sh and rerun:

$ git diff rt_utils.sh
diff --git a/tests/rt_utils.sh b/tests/rt_utils.sh
index cfe5e7c..6aa80f0 100755
--- a/tests/rt_utils.sh
+++ b/tests/rt_utils.sh
@@ -126,7 +126,7 @@ submit_and_wait() {
         status_label='held in a queue'
       elif [[ $status = 'R' ]];  then
         status_label='running'
-      elif [[ $status = 'E' ]] || [[ $status = 'C' ]];  then
+      elif [[ $status = 'E' ]] || [[ $status = 'C' ]] || [[ $status = '-' ]];  then
         status_label='finished'
         test_status='DONE'
         exit_status=$( qstat ${jobid} -x -f | grep Exit_status | awk '{print $3}')

[DeniseWorthen]

This is the log_cheyenne.gnu/compile_001.log now:

TEST 001 compile is waiting to enter the queue
TEST 001 compile is submitted
1 min. TEST 001 compile is waiting in a queue,  status: Q jobid 9551700
2 min. TEST 001 compile is waiting in a queue,  status: Q jobid 9551700
3 min. TEST 001 compile is waiting in a queue,  status: Q jobid 9551700
4 min. TEST 001 compile is running,  status: R jobid 9551700
qstat: 9551700.chadmin1.ib0.cheyenne.ucar.edu Job has finished, use -x or -H to obtain historical job information
qstat: 9551700.chadmin1.ib0.cheyenne.ucar.edu Job has finished, use -x or -H to obtain historical job information
5 min. TEST 001 compile is finished,  status: - jobid 9551700


Test 001 compile FAIL

[DusanJovic-NOAA]

Now the status of the compile job is 'FAIL'. The rt.sh should now cancel all RUN jobs that depend on this COMPILE job, and the final status of the regression test should be FAILED. Can you confirm this?

[DeniseWorthen]

Yes, I will try the standard rt_gnu.conf at the current develop.

[DusanJovic-NOAA]

And there should be a fail_test file in tests directory.

[DeniseWorthen]

On cheyenne.gnu, I now see the compile failures reported (in fail_test)

compile 007 failed
compile 008 failed
compile 006 failed

The Regression test log also now correctly reports failure:

FAILED TESTS:
Test compile 007 failed failed
Test compile 008 failed failed
Test compile 006 failed failed

REGRESSION TEST FAILED
Tue Jul 20 20:35:40 MDT 2021
Elapsed time: 08h:43m:59s. Have a nice day!