cosp2 svn access to github support is ending soon

[jedwards4b]

What happened?

IN cam_cesm2_2_rel_07

[cosp2]
local_path = src/physics/cosp2/src
protocol = svn
repo_url = https://github.com/CFMIP/COSPv2.0/tags/
tag = v2.1.4cesm/src
required = True

This is currently giving an error message:
svn co https://github.com/CFMIP/COSPv2.0/tags/v2.1.4cesm/src
svn: E175002: Unable to connect to a repository at URL 'https://github.com/CFMIP/COSPv2.0/tags/v2.1.4cesm/src'
svn: E175002: Unexpected HTTP status 422 'Unprocessable Content' on '/CFMIP/COSPv2.0/tags/v2.1.4cesm/src'

svn: E900002: Additional errors:
svn: E900002: Subversion support temporarily disabled.

GitHub has temporarily disabled Subversion support as part of a planned brownout
in order to give you advance notice of the upcoming permanent removal of
Subversion support on 2024-01-08.

Please see https://github.blog/2023-01-20-sunsetting-subversion-support/ for details.

Note that this is a github issue and not a manage_externals issue.

What are the steps to reproduce the bug?

svn co https://github.com/CFMIP/COSPv2.0/tags/v2.1.4cesm/src
svn: E175002: Unable to connect to a repository at URL 'https://github.com/CFMIP/COSPv2.0/tags/v2.1.4cesm/src'
svn: E175002: Unexpected HTTP status 422 'Unprocessable Content' on '/CFMIP/COSPv2.0/tags/v2.1.4cesm/src'

svn: E900002: Additional errors:
svn: E900002: Subversion support temporarily disabled.

GitHub has temporarily disabled Subversion support as part of a planned brownout
in order to give you advance notice of the upcoming permanent removal of
Subversion support on 2024-01-08.

Please see https://github.blog/2023-01-20-sunsetting-subversion-support/ for details.

What CAM tag were you using?

cam_cesm2_2_rel_07

What machine were you running CAM on?

CISL machine (e.g. cheyenne)

What compiler were you using?

Intel

Path to a case directory, if applicable

No response

Will you be addressing this bug yourself?

Yes

Extra info

machine and compiler are irrelevant

[jedwards4b]

NOTE that this is also a problem in the current tag (cam6_3_132)
With:
[cosp2]
local_path = src/physics/cosp2/src
protocol = svn
repo_url = https://github.com/CFMIP/COSPv2.0/tags/
tag = v2.1.4cesm/src
required = True

[clubb]
local_path = src/physics/clubb
protocol = svn
repo_url = https://github.com/larson-group/clubb_release/tags/
tag = clubb_4ncar_20221129_59cb19f_20230330_branchtag/src/CLUBB_core
required = True

[silhs]
local_path = src/physics/silhs
protocol = svn
repo_url = https://github.com/larson-group/clubb_release/tags/
tag = clubb_4ncar_20221129_59cb19f_20230330_branchtag/src/SILHS
required = True

[gold2718]

While manage_externals supports the git sparse checkout feature, it does not address the reason we have been using the svn bridge. COSP and CLUBB are huge repositories and git sparse checkout still begins with a clone of the entire repository.
I think we need to add support for the --filter option to git clone, probably in conjunction with a sparse checkout file if we are going to regain something like the current svn bridge functionality.
Note, I think this issue needs to be moved or duplicated to manage externals.

[gold2718]

If we fix it with manage externals, we can make sure the solution allows us to run it on at least some old tags (using a combination of a newer manage externals tag with a sparse checkout file we keep somewhere (probably at the top level of the CAM repo?).

[jedwards4b]

@gold2718 I don't think this will work with any unmodified old tags. Even if the user were to checkout a new manage externals they would also need the Externals.cfg file(s) unless we reinterpret the protocol=svn in the file - is that what you are thinking?

[jedwards4b]

So if protocol=svn and repo_url=github.com then do
git clone --filter? And for new tags we do
protocol = git filter ???

[gold2718]

Ooh, that's actually an interesting idea since the only use of svn left in CESM that I know of are repos using the GitHub svn bridge.
We could reinterpret a current entry as you describe, we have to create a sparse checkout file we can use with a filter.
Note, I'm still not sure this will work. See the description of --filter=sparse:oid=<blob-ish> under git help rev-list.

Note, we have lots of old tags that no longer checkout externals sucessfully because they specified real svn repos that became inaccssible.

[jedwards4b]

@nusbaume if you can identify outside of manage_externals the git commands that will work for your sparse checkout - I think that that would be a great start.

[nusbaume]

@jedwards4b @gold2718 I think I have found a method for doing a sparse checkout with git that doesn't blow up disk space. Using COSP as an example the following git workflow seems to work:

mkdir cosp2  #provided by "local_path" in cfg file
cd cosp2
git init 
git remote add origin https://github.com/CFMIP/COSPv2.0.git  #URL provided by cfg file
git fetch --tags --depth=1 --filter=blob:none  #grab the tree, commits, tags, etc. but not any of the actual binaries
git config core.sparseCheckout true
echo '/src/' > .git/info/sparse-checkout  #sparse checkout info provided by "sparse" in cfg file.  
git checkout v2.1.4cesm

I have tested this with both COSP and CLUBB as described in CAM and it appears to work (it actually has a smaller footprint than the current svn bridge method due to the removal of the .svn directory).

It also removes the need to specify a complicated filter when cloning the repo, as you simply filter all of the "blob" objects. However, it would probably make doing development or getting extra metadata (e.g. git log info) difficult within the checked-out external itself.

Thoughts? I have no idea how easy or hard it would be to implement this workflow in manage_externals, so I can try and find a different workflow if need be.

[jedwards4b]

Thank you @nusbaume I will explore implementing this in manage_externals tomorrow and will try to have some feedback by the time of the cseg meeting in the afternoon.

[jedwards4b]

@nusbaume
The cosp2 directory already exists in a cam checkout so the first two steps are not required.

This step:
echo '/src/' > .git/info/sparse-checkout #sparse checkout info provided by "sparse" in cfg file.

Should be replaced with:
git sparse-checkout set src

The full sequence of commands:

git clone [email protected]:ESCOMP/CAM.git -b cam_development
cd CAM/src/physics/cosp2

 git init
 git remote add origin https://github.com/CFMIP/COSPv2.0.git
 git fetch --tags --depth=1 --filter=blob:none
 git config core.sparseCheckout true
 git sparse-checkout set src
 git checkout v2.1.4cesm

[jedwards4b]

Currently we have:

[cosp2]
local_path = src/physics/cosp2/src
protocol = svn
repo_url = https://github.com/CFMIP/COSPv2.0/tags/
tag = v2.1.4cesm/src
required = True

And we are to infer that the sparce path to checkout is part of the tag name - so in this case we want to checkout the src directory for tag v2.1.4cesm. But v2.1.4cesm/src is also a perfectly valid tag name. I propose that we change this to be:

[cosp2]
local_path = src/physics/cosp2/src
protocol = git
sparse_path = src
repo_url = https://github.com/CFMIP/COSPv2.0/tags/
tag = v2.1.4cesm
required = True

This means that we will need changes in this file for every tag that we want to update but we will need to change manage_externals for that tag anyway so I don't think this extra step will be too burdensome. This has the added advantage that we can specify more than a single directory for sparse checkout - I don't think that we use that feature at this time.

[jedwards4b]

It seems that manage_externals already supports sparse checkout and cam already uses it in the mpas dycore. It was added in version 1.2.1 of manage_externals in August of 2019 by @gold2718 I will have a PR to cam_development soon. I would like the cam development team to back-port to the appropriate cam branch tags for 2.1.5 and 2.2.2 if they deem the proposed changes appropriate.

[gold2718]

I added sparse checkout (starting with a prototype implementation from Tom Clune) back when we thought Tom Clune & co. were joining us in using manage_externals (they needed it).

[jedwards4b]

@cacraigucar asked whether changes in manage_externals could allow us to maintain the current cam structure in the face of the upcoming removal of support for svn access to github.com. I have opened a PR in manage_externals to address that request. However that PR does not maintain the original paths - for example cosp2 was
cosp2/src and is now cosp2/src/src. So I think that this change with no changes in manage_externals is a better path forward. Thoughts?

[gold2718]

@cacraigucar asked whether changes in manage_externals could allow us to maintain the current cam structure in the face of the upcoming removal of support for svn access to github.com. I have opened a PR in manage_externals to address that request. However that PR does not maintain the original paths - for example cosp2 was cosp2/src and is now cosp2/src/src. So I think that this change with no changes in manage_externals is a better path forward. Thoughts?

I don't think it can work.

[jedwards4b]

@gold2718 I agree but wanted to make an attempt at meeting @cacraigucar 's request.

[cacraigucar]

@jedwards4b - Thank you SO much for attempting this and proving that manage_externals can not be a standalone change to fix this problem. We were in a meeting discussing this (along with a bunch of other CAM stuff) which is why we didn't get back with you immediately. We have agreed that since there does not appear to be a way to avoid changes to CAM code, we are going to implement your original fix. Again, thanks for verifying that we couldn't solve it with just changes to manage_externals.