1.5.3 release build issue

[tonycurtis]

There's no configure script, and I can't run autogen.sh to create one (

Or am I missing something?

[markbrown314]

Hi @tonycurtis are you running into errors when you run autogen.sh?

[tonycurtis]

Yes, but shouldn't the release already have a configure? Anyway, here's what happens:

$ wget https://github.com/Sandia-OpenSHMEM/SOS/archive/refs/tags/v1.5.3.tar.gz
...
...

$ tar xpf v1.5.3.tar.gz

$ cd SOS-1.5.3/

$ ./configure
-bash: ./configure: No such file or directory

$ ./autogen.sh
Error: test submodule (modules/tests-sos) contents are missing.
Please run the following command to download the SOS tests:
   git submodule update --init

$ git submodule update --init
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

[markbrown314]

Yes, the build assumes that git was used to pull the package which is why git submodule fails (this is an oversight that needs to be fixed). The tests are located in a separate tree so they need to be copied into the build.

Check if the following works:

(From SOS-1.5.3 directory)

$ wget -O - https://github.com/openshmem-org/tests-sos/archive/refs/tags/v1.5.2.tar.gz | tar -xvzf - -C modules/tests-sos/ --strip-components=1
$ ./autogen.sh

(Then configure)

[markbrown314]

Proposing the following fixes for future releases:

1. GitHub releases should be built from a release branch that contains the output of autoreconf (which includes configure).
2. The release branch will incorporate a compatible release of SOS tests to allow full reproducibility.
3. A script will be available to automate the release creation process.

[davidozog]

Thanks for spotting this @tonycurtis. Hopefully you have been ok using the v1.5.3 git tag for now? (or the main branch)

Proposing the following fixes for future releases:

1. GitHub releases should be built from a release branch that contains the output of autoreconf (which includes configure).
2. The release branch will incorporate a compatible release of SOS tests to allow full reproducibility.
3. A script will be available to automate the release creation process.

Thanks @markbrown314, I'm good with this approach. This is a new and unforeseen procedure change for SOS release, so I propose we do the following:

1. Document the procedure here: https://github.com/Sandia-OpenSHMEM/SOS/wiki/Release-Checklist
2. Align the tests-sos "package" with a release branch name identical to the SOS release branch name in the style release/v1.x.y (a tag won't work for v1.5.3 because of step 3:)
3. We should revert the post-v1.5.3 commits (e.g., the PR adding shmem_scan b/c the tests won't build against SOS v1.5.3) on the tests-sos release/v1.5.3 branch.
4. Also use the style release/v1.x.y for the SOS release branch name - I'm nervous about colliding with the existing tag names of SOS, we should ideally be careful about this to avoid any git confusion/problems.
5. Edit the v1.5.3 release with the updated tarball - it's not useable in it's current state.

I'm happy to take care of this if you'd like @markbrown314, it's a bit more of a burden than I had imagined and it's my mistake. Shall I assign this issue to me? I will also help to clean up the existing branches on SOS, looks like that could use some attention.

Another option (perhaps in the future?) might be to leverage a Github Action for release generation, for example via https://github.com/actions/create-release (or more likely, one of the maintained options). However, I think this will take relatively more time and testing to get it right, and I'm not sure how well it would automate. Thoughts?

[tonycurtis]

I just did a checkout on the 1.5.3 tag before. No real users here, just keeping track of things.

[davidozog]

Yes, we could also simply not release the tarballs since it's unlikely we have users of those anyway. A release would simply be a tag/checkpoint (of SOS and tests-sos) with an announcement of the changes.

If only Github supported releasing without the source archives, perhaps we could do that... it currently seems it does not:
https://github.com/orgs/community/discussions/6003

Although if there are any users of the source tarballs, please request this ticket be reopened.

[markbrown314]

Hi @davidozog, my original goal was not to change the build process for builds sourced from git clone (too much). It is fine to run autogen.sh from there. I think it might be better to make the tests optional and the configure script can discover if it is available or not. If you run make check and the tests are not available it could just give you an error message with details on how you can download it.

The major changes in the build flow would be for releases only.
Hopefully the build procedure for the release.
./configure
make install
(optionally)
make check

It is sad that the actions/create-release was deprecated.

It would be best to create a simple release automation script to stage the release branch (with the release/ prefix)

There are some updates around the margins I would like to make as well:
It seems the configure.ac needs some updating it is outputting a lot of obsolete warnings.
Let me work on updating the documentation, and if it works for you we could divvy up the work.

We can continue this work in a new GitHub task.

[davidozog]

Hi @davidozog, my original goal was not to change the build process for builds sourced from git clone (too much). It is fine to run autogen.sh from there

Yup, agreed. From a git clone nothing changes (except eventually the following feature...):

If you run make check and the tests are not available it could just give you an error message...

Definitely. Looks like this needs an issue, like you said.

It seems the configure.ac needs some updating it is outputting a lot of obsolete warnings.

I believe this should already be addressed in this PR:
#1130

It's a big blind upstream configury update, so I opted to wait until after v1.5.3 to make the change. I will bring that PR aligned with main and then ask for reviews.

Let me work on updating the documentation, and if it works for you we could divvy up the work.
We can continue this work in a new GitHub task.

Yup! Feel free to ping me anytime or setup a call to coordinate.