Improve resilience of apptainer testing (chapel-lang#23543)

This PR takes two steps to improve nightly apptainer testing:
1. It checks for files before deleting them, so that a failure to `git
clone` in one config does not stop the testing for all configs.
 2. It adds a wait-and-retry loop to the `git clone` command.

Reviewed by @bhavanijayakumaran - thanks!

util/devel/test/apptainer/tryit.py

@@ -135,10 +135,12 @@ def main():
             status[name] = statusline
 
             if args.cleanup:
-                printAndLog(log, "Removing {}/image.sif".format(d));
-                os.remove("image.sif")
-                printAndLog(log, "Removing {}/chapel".format(d));
-                shutil.rmtree("chapel")
+                if os.path.exists("image.sif"):
+                    printAndLog(log, "Removing {}/image.sif".format(d));
+                    os.remove("image.sif")
+                if os.path.exists("chapel"):
+                    printAndLog(log, "Removing {}/chapel".format(d));
+                    shutil.rmtree("chapel")
 
             printAndLog(log, "")
             os.chdir(startDir)

util/devel/test/vagrant/provision-scripts/chapel-update.sh

@@ -1,10 +1,32 @@
 #!/usr/bin/env bash
 
-if [ -d chapel ]
-then
-  echo chapel directory already exists - updating
-  cd chapel && git checkout main && git pull --ff-only && cd .. && echo UPDATED
-else
-  echo cloning chapel
-  git clone https://github.com/chapel-lang/chapel && echo CLONED
-fi
+for try in 1 2 3 4 5
+do
+
+  if [ -d chapel ]
+  then
+    echo chapel directory already exists - updating
+    cd chapel && git checkout main && git pull --ff-only && cd .. && echo UPDATED
+  else
+    echo cloning chapel
+    git clone https://github.com/chapel-lang/chapel && echo CLONED
+  fi
+
+  if [ $? -eq 0 ]
+  then
+    # OK, all done, exit with success
+    exit 0
+  else
+    echo "Error with git. Pausing for 30s."
+    sleep 30
+    if [ -d chapel ]
+    then
+      echo "Deleting the chapel directory that failed update"
+      rm -rf chapel
+    fi
+  fi
+
+done
+
+# if we got here, there was an error
+exit 1