Rocoto and scrontab on Gaea C6

[BijuThomas-NOAA]

crontab is not accessible on Gaea. Instead, users will need to use scrontab in order to submit cron jobs. When the rocoto command is executed using scrontab, rocoto fails to submit the job(it keep trying but failing):

+ rocotorun --verbose=5 -w hafs_workflow.xml -d hafs_workflow.db
08/15/24 10:42:14 EDT :: hafs_workflow.xml :: Submission status of Task1 for cycle 202108271200 could not be retrieved because the server process at druby://192.188.179.70:46157 died
08/15/24 10:42:14 EDT :: hafs_workflow.xml :: Submission of Task1 for cycle 202108271200 probably, but not necessarily, failed.  It will be resubmitted
08/15/24 10:42:14 EDT :: hafs_workflow.xml :: Submitting Task1 using sbatch < /tmp/sbatch.in20240815-276566-hq1ki2 with input

2024-08-15 10:38:14 -0400 :: gaea61 :: Submitting Task1
2024-08-15 10:38:14 -0400 :: gaea61 :: Submission status of Task1 is pending at druby://192.188.179.70:38415
2024-08-15 10:40:14 -0400 :: gaea61 :: Submission status of Task1 for cycle 202108271200 could not be retrieved because the server process at druby://192.188.179.70:38415 died
2024-08-15 10:40:14 -0400 :: gaea61 :: Submission of Task1 for cycle 202108271200 probably, but not necessarily, failed.  It will be resubmitted
2024-08-15 10:40:14 -0400 :: gaea61 :: Submitting Task1
2024-08-15 10:40:14 -0400 :: gaea61 :: Submission status of Task1 is pending at druby://192.188.179.70:46157
2024-08-15 10:42:14 -0400 :: gaea61 :: Submission status of Task1 for cycle 202108271200 could not be retrieved because the server process at druby://192.188.179.70:46157 died
2024-08-15 10:42:14 -0400 :: gaea61 :: Submission of Task1 for cycle 202108271200 probably, but not necessarily, failed.  It will be resubmitted

Any suggestions on this?. There is no issue when rocotorun is being executed repeatedly using a bash while loop instead of scron.

Here are the scrontab directives:

#SCRON --partition=cron_c6
#SCRON --time=0:10:00
#SCRON --account=windfall
#SCRON --job-name=scron_test
#SCRON --chdir=/gpfs/f6/hurr1/proj-shared/Biju.Thomas/rocoto_test
*/2 * * * *  /gpfs/f6/hurr1/proj-shared/Biju.Thomas/rocoto_test/launch.sh 1>> //gpfs/f6/hurr1/proj-shared/Biju.Thomas/rocoto_test/cron.log 2>&1

[christopherwharrop-noaa]

@BijuThomas-NOAA - I'm sorry I don't have any ideas at the moment. My guess is that there is something about the scron environment that is causing the job submission process to abort. Without knowing more about how scron works, it's hard for me to guess what might be happening.