Change from manage_externals to git-fleximod

[jedwards4b]

Replace manage_externals with git-fleximod. This PR
Replaces #946

[cacraigucar]

I'm moving this from the old PR. I should point out that @jtruesdal also tripped across this on izumi, so I'm sure others will as well.

---

I'm pretty sure I won't be the only user to trip on this on izumi. I'm not sure why it is happening. I have two requests:

1. Trap and write out a meaningful message if python is not new enough to support git-fleximod - if that is what this error message means.
2. Figure out why when I set python in my .profile and then use git-fleximod it doesn't work. But when I load it manually, it does. This is on izumi, and I suspect other users on izumi may have the same problem. See the listing below:

izumi$ bin/git-fleximod -h
ImportError: No module named site
izumi$ module list
Currently Loaded Modulefiles:
 1)lang/python/3.11.5
izumi$ module load lang/python/3.11.5
module-info mode load in progress
    Unloading python module: lang/python/3.11.5
izumi$ bin/git-fleximod -h
usage: git-fleximod [-h] [-C PATH] [-g [GITMODULES]] [-x [EXCLUDE ...]] [-f] [-o] [-v] [-V] [--backtrace] [-d]
                    {update,status,test} [components ...]  *** The rest of the help message was written out 

[cacraigucar]

When a bin/git-fleximod update is run a second time, without changing anything so it should be a "do nothing" command, it indicates that it Successfully checked out a lot of externals along with Recursively checking out submodules. Should those messages only appear when a checkout really is occurring and only the XXX up to date and XXX at tag YYY appear?

[cacraigucar]

@jtruesdal reported that he needed to type in his password for ssh. Any idea on why this might need to be done?

Successfully checked out parallelio /project/amp/jet/collections/CAM_flexi/libraries/parallelio
          parallelio up to date.
Recursively checking out submodules of cime /project/amp/jet/collections/CAM_flexi/cime https://github.com/jedwards4b/cime
Enter passphrase for key '/home/jet/.ssh/id_ed25519': 
e    CIME/non_py/cprnc not checked out, out of sync at tag None, expected tag is v1.0.6
Successfully checked out CIME/non_py/cprnc /project/amp/jet/collections/CAM_flexi/cime/CIME/non_py/cprnc

[cacraigucar]

A user who tested git-fleximod ended up with an "M" and wasn't sure what to do. When I suggested an error message something along the lines of what manage_externals spits out, they agreed that that would be helpful.

What manage_externals spits out:

The external repositories labeled with 'M' above are not in a clean state.
The following are four options for how to proceed:
(1) Go into each external that is not in a clean state and issue either a 'git status' or
an 'svn status' command (depending on whether the external is managed by git or
svn). Either revert or commit your changes so that all externals are in a clean
state. (To revert changes in git, follow the instructions given when you run 'git
status'.) (Note, though, that it is okay to have untracked files in your working
directory.) Then rerun checkout_externals.
(2) Alternatively, you do not have to rely on checkout_externals. Instead, you can manually
update out-of-sync externals (labeled with 's' above) as described in the
configuration file Externals.cfg. (For example, run 'git fetch' and 'git checkout'
commands to checkout the appropriate tags for each external, as given in
Externals.cfg.)
(3) You can also use checkout_externals to manage most, but not all externals: You can specify
one or more externals to ignore using the '-x' or '--exclude' argument to
checkout_externals. Excluding externals labeled with 'M' will allow checkout_externals to
update the other, non-excluded externals.
(4) As a last resort, if you are confident that there is no work that needs to be saved
from a given external, you can remove that external (via "rm -rf [directory]") and
then rerun the checkout_externals tool. This option is mainly useful as a workaround for
issues with this tool (such as ESMCI/manage_externals#157).
The external repositories labeled with '?' above are not under version
control using the expected protocol. If you are sure you want to switch
protocols, and you don't have any work you need to save from this
directory, then run "rm -rf [directory]" before rerunning the
checkout_externals tool.

[cacraigucar]

Does git-fleximod do the shallow (I think that is the term) checkouts so that checkouts of libraries like RRTMGP are not huge?

[cacraigucar]

Major problem - My checkout of a hash from my personal repo is not happening:

In .gitmodules, I changed

[submodule "atmos_phys"]
path = src/atmos_phys
url = https://github.com/cacraigucar/atmospheric_physics
fxtag = 98712a
fxurl = https://github.com/ESCOMP/atmospheric_physics
fxrequired = AlwaysRequired

I see in the listing:
Successfully checked out atmos_phys /scratch/cluster/cacraig/cam4_remove_manage_externals/src/atmos_phys
atmos_phys updated to 98712a

When I cd to src/atmos_phys and do a git log, I see the original checkout:
commit bdbf4e08581f818ae12fb459fb6e017a9458c8fc (HEAD, tag: atmos_phys0_02_000)
Merge: 5dd0c55 859e059
Author: cacraigucar [email protected]
Date: Fri Jan 19 14:24:40 2024 -0700

[cacraigucar]

Running two bin/git-fleximod update commands without any changes, do not exhibit the following problem. This problem shows up when a change has been made to the .gitmodule file

In the listing see the error/warning:
Recursively checking out submodules of hemco /scratch/cluster/cacraig/cam4_remove_manage_externals/src/hemco https://github.com/jedwards4b/HEMCO_CESM.git
HEMCO at tag 3.6.3-cesm

Recursively checking out submodules of HEMCO /scratch/cluster/cacraig/cam4_remove_manage_externals/src/hemco/HEMCO https://github.com/geoschem/hemco.git
e docs/source/geos-chem-shared-docs has no fxtag defined in .gitmodules, module at 228288d

[cacraigucar]

When a bogus tag is provided, git-fleximod gives no indication that the tag doesn't exist or that anything went wrong:

Inside .gitmodules:
[submodule "atmos_phys"]
path = src/atmos_phys
url = https://github.com/cacraigucar/atmospheric_physics
fxtag = ZZZZZ
fxurl = https://github.com/ESCOMP/atmospheric_physics
fxrequired = AlwaysRequired

The listing has:

Successfully checked out atmos_phys /scratch/cluster/cacraig/cam4_remove_manage_externals/src/atmos_phys
atmos_phys updated to ZZZZZ

[cacraigucar]

Documentation suggestions for .lib/git-fleximod/README.md:

• fxurl should not be changed by the casual user, but because users need to change fxtag, they gravitate to fxurl for specifying their forks. I would suggest renaming this to be something a little more verbose (fxurl_for_commit_testing or fxurl_do_not_change or something like that)
• I don't think "checkout" is an option that is used any longer. If that is true, it should be removed from the options and example sections
• Installation instructions are not pertinent for CESM and component checkouts (should this be reflected in the comment)?

[jedwards4b]

It's really hard to track and respond to individual issues this way. I think that it would be easier if you open issues here.

The personal repo issue is fixed. The issue with specifying a non-existent branch is as well:

s           atmos_phys atmos_phys0_02_000-5-g98712a6 98712a6 is out of sync with .gitmodules fred

Successfully checked out atmos_phys /glade/work/jedwards/sandboxes/cam6_3_150.01/src/atmos_phys
          atmos_phys updated to fred
Cmd('git') failed due to: exit code(1)
  cmdline: git checkout fred
  stderr: 'error: pathspec 'fred' did not match any file(s) known to git'

I don't think it is appropriate to mention CESM in the git-fleximod README since the tool is meant for a wider audience. Also my preference would be not to distribute git-fleximod with cesm but simply to provide instructions for getting and installing the tool before checking out the model. I understand that I lost that argument, but I think that the install instructions should remain in place.

I see that there is still an issue with running update twice and will address that now.

[jedwards4b]

The update issue has also been addressed.

[cacraigucar]

See the issue I opened: https://github.com/ESMCI/git-fleximod/issues

Until this is fixed, I can not verify any of the changes which have been made

[cacraigucar]

@jedwards4b - is this up-to-date with the latest git-fleximod changes?

[jedwards4b]

This is using git-fleximod tag 0.7.3 and needs to be updated - you can do it using the instructions for git subtrees on the cime wiki or I will get to it soon.

[cacraigucar]

This is using git-fleximod tag 0.7.3 and needs to be updated - you can do it using the instructions for git subtrees on the cime wiki or I will get to it soon.

@jedwards4b - we'll just wait for you

[jedwards4b]

Superceeded by #1052