…lobal-workflow into develop

…lobal-workflow into feature/gfsv17_issue_2490

This provides a new pygfs task, archive.py, that provides all of the
tools necessary to archive data to the local (`ARCDIR`) and backup
(`ATARDIR`) archive directories. YAML-Jinja2 templates are provided to
define the file to be archived or tarred to replace the
`hpssarch_gen.sh`, `exglobal_earc`, and `exglobal_archive.sh` scripts
and make it easier to add new data and explicitly handle optional and
required files.

For `ATARDIR` archiving, a master jinja template is provided for each
`RUN` (i.e. master_gdas.yaml.j2, master_gfs.yaml.j2,
master_enkf.yaml.j2). The master_enkf.yaml.j2 template is used for both
`enkfgdas` and `enkfgfs` `RUN`s. These templates then include the
appropriate `RUN`-specific jinja templates (e.g. gdas.yaml.j2) based on
experiment, cycle, and coupled parameters. Each of these templates
corresponds to a single tarball to populate and are tabbed 4 spaces so
they are defined within the master `datasets` dictionary.

Future developers should not have to make modifications to archive.py
unless archiving is being enabled for a new `RUN` (e.g. `gefs`) and then
only a single `elif` needs to be added to the configure method to
specify the master `ATARDIR` template to archive (e.g.
`master_gefs.yaml.j2`). If a new component is coming online that needs
to be archived to `ATARDIR` (e.g. SNOW), then create a new template for
each `RUN` that it needs to be archived for (e.g. `gdassnow.yaml.j2`)
and reference the template in the appropriate master templates, e.g.

`master_gdas.yaml:`
```jinja
{% if DO_SNOW %}
{% include "gdassnow.yaml.j2" %}
{% endif %}
```

A few other issues were addressed along the way:
1. Aerosols have been reenabled. Aerosol forecasts should only be
performed during gdas cycles, but analyses can be performed for both gfs
and gdas cycles. This was accomplished by setting separate
`AERO_<job>_CDUMP` variables to parse on for both `ANL` and `FCST` jobs.
2. Fixed the name of the `cice6_rst_ok` variable in `forecast_det.sh`.
This prevented restarts from being used for cice-enable experiments.
This feature was not tested.
3. Create a temporary fix for the `wgrib` utility. For spack-stack
1.6.0, the `grib-util` module file does not declare `WGRIB`. An issue is
open (JCSDA/spack-stack#1097) to fix this in
existing installations. Once complete, this temporary fix should be
removed.
4. The number of `earc` jobs has been reduced for lower resolution
experiments. Both C48 and C96 experiments will now only have two earc
jobs (one for the non-member files to archive and another for the member
files). C192 will have up to 3 earc jobs (one non-member, one for
members 1-40 and another for members 41-80, if needed).

Resolves NOAA-EMC#2345
Resolves NOAA-EMC#2318 

---------

Co-authored-by: Walter Kolczynski - NOAA <[email protected]>

Allows cycling and restarting CICE with the sea-ice analysis if the
marine DA is switched on.

Resolves NOAA-EMC#2568 
Resolves NOAA-EMC/GDASApp#1103

…AA-EMC#2614)

- Updating STMP and PTMP settings in host file for Orion and Hercules
because they are cross mounted.
- Also took the opportunity to finally update **SLURM_ACCOUNT** to
**HPC_ACCOUT** in CI over rides.
- Added a refactor of the `rocotostat.py` tool that is more pythonic and
as a execute retry feature because the `rocotostat` utility on Orion has
been failing sometimes.

- Port global-workflow’s build and run capability to Gaea-C5 
- Building global-workflow on Gaea-C5
- Setting up experiments with global-workflow on Gaea-C5

---------

Co-authored-by: AnilKumar-NOAA <[email protected]>
Co-authored-by: DavidBurrows-NCO <[email protected]>

This PR creates the atmensanlfv3inc job, the ensemble version of
atmanlfv3inc, created in GW PR NOAA-EMC#2420. Its GDASApp companion PR is
#[1104](NOAA-EMC/GDASApp#1104), and its JCB-GDAS
companion PR is #[3](NOAA-EMC/jcb-gdas#3).

…OAA-EMC#2611)

Based on users' feedback, this PR do the following:
1. Change GRIB2 parameter names DLWRF -> NLWRF and DSWRF -> NSWRF
2. Change the vertical level of ocean 3D variables (WTMP, SALIN, UOGRD,
  and VOGRD) from "%g m below water surface" to "%g m below sea level"
3. Round up depth numbers to integer (e.g. 4481.0625 -> 4481 m)

Co-authored-by: Rahul Mahajan <[email protected]>

This "persistent" Java Agent launch script can be ran from a cron job:

- Uses Jenkins Remote API to check the status of the Node connection
using curl for a given machine.
- If it is not connected a new agent is launched for that node.

Resolves NOAA-EMC#2633

Adds the C384mx025_3DVarAOWCDA yaml files for one experiment into a new GFSv17 folder.

This PR adds options to use a global 1 deg grid, intended for testing
with the SFS application.

Requires new fix file changes NOAA-EMC#2618

NCO has requested that each COM variable specify whether it is an input
or an output. This completes that process for the forecast job.

Refs NOAA-EMC#2451

---------

Co-authored-by: David Huber <[email protected]>

Add the capability to use slurm reservation nodes
Add "ACCOUNT_SERVICE" for jobs to run in PARTITION_SERVICE

Resolves NOAA-EMC#2626

The workflow was updated to use modules built on Rocky 9, but the
external packages (like prepobs) were still pointing to the versions
built on CentOS (Orion). This transitions to packages built on Rocky 9.

Updating of the tracker package has been deferred until later. As such,
the tracker jobs have been disabled by returning immediately if they are
on Hercules. Since these jobs are small, resource-wise, it should not
meaningfully impact turnover time.

…40531