Add weekly build/test scripts

[AlexanderRichert-NOAA]

Summary

This PR will add scripting for weekly unit testing and build caching of the unified env through Jenkins, cron jobs, etc.

Also a few updates for Acorn, including a change to spack stack create env --compiler that allows compiler version to be specified (spack stack create env --compiler intel -> envs/myenv-intel; spack stack create env --compiler intel@2022 -> envs/myenv-intel-2022), which is needed to differentiate on platforms platform where more than one version of a compiler is installed.

Testing/implementation by platform

On each system:

• Configure ShellSetup.sh (including setting centralized build_cache dir under spack-stack/ dir)
• Set up centralized source cache
• Get it running (Jenkins, cron, GHA); test installing a few packages
• Enable notifications (email, Slack, whatever)

Platforms (check when all of the above items are complete):

• Acorn - @AlexanderRichert-NOAA - cron
• Orion - @rickgrubin-tomorrow test with Jenkins on Orion
• Nautilus - @climbfuji - either cron or Github actions

Applications affected

all

Systems affected

all (eventually)

Dependencies

none

Issue(s) addressed

#1184, #365

Checklist

• This PR addresses one issue/problem/enhancement, or has a very good reason for not doing so.
• These changes have been tested on the affected systems and applications.
• All dependency PRs/issues have been resolved and this PR can be merged.

[AlexanderRichert-NOAA]

closing temporarily so I don't spam the CI with jobs...

[AlexanderRichert-NOAA]

Cron entry for Acorn: 0 1 * * SUN /path/to/spack-stack-weekly-build/util/weekly_build/SpackStackBuildCache_AllSteps.sh $(date +\%y\%m\%d) /path/to/spack-stack/weekly_build acorn > /path/to/spack-stack/weekly_build/logs/cron.$(date +\%y\%m\%d).out 2>&1

[rickgrubin-tomorrow]

Within a Jenkins pipeline:

Re-running everything after the recent merge, now I'm seeing a dreaded Install tree root is too long error:

[...]
17:10:48 ==> No patches needed for gmake
17:10:48 ==> gmake: Executing phase: 'install'
17:11:11 ==> Error: SbangPathError: Install tree root is too long. Spack cannot patch shebang lines when script path length (128) exceeds limit (127).
17:11:11   /work/noaa/epic/role-epic/jenkins/workspace/spack-stack/ufswm-test/Orion-intel/weekly/spack/user_cache/bootstrap/store/bin/sbang
[...]
17:11:11 ==> Warning: Skipping build of clingo-bootstrap-spack-wdkesgo7ubjc5jn7l2ee5s7awdugr6nr since gmake-4.4.1-e4ee4hnuu22kdvldnoenit3gsjbjaldl failed
17:11:11 ==> Warning: Skipping build of re2c-2.2-i4mprqgx2bz2zmsz3e3imasiyv5aycpc since gmake-4.4.1-e4ee4hnuu22kdvldnoenit3gsjbjaldl failed
17:11:11 ==> Error: cannot bootstrap the "clingo" Python module from spec "clingo-bootstrap@spack+python %gcc target=x86_64" due to the following failures:
17:11:11 github-actions-v0.5 raised SbangPathError: Install tree root is too long. Spack cannot patch shebang lines when script path length (128) exceeds limit (127).
[...]

Ironically, the build of gmake-4.4.1 succeeded (per its build log).

Not clear on what I'm doing wrong / how to correct.

Will try taking this out of Jenkins and running in a shell (emulating a cron job).

[climbfuji]

@rickgrubin-tomorrow The error message is right there:

github-actions-v0.5 raised SbangPathError: Install tree root is too long. Spack cannot patch shebang lines when script path length (128) exceeds limit (127).

You need to use a shorter path.

[climbfuji]

I am fine with merging this into develop in its current stage, it works perfectly for my github action runs on nautilus. Maybe we should all just get to the step where the environments build and then populate the cache. We can make adjustments and add more capabilities in subsequent PRs, for example for downstream application testing and package unit testing.

[AlexanderRichert-NOAA]

If I can ever manage to log onto Acorn again I want to do another end-to-end test on there... But after that yes that's fine with me.

[rickgrubin-tomorrow]

@rickgrubin-tomorrow The error message is right there:

github-actions-v0.5 raised SbangPathError: Install tree root is too long. Spack cannot patch shebang lines when script path length (128) exceeds limit (127).

You need to use a shorter path.

I saw the message, and it's meaning was / is clear.

Curious as to why I didn't see this prior, and the only thing that changed was the merge. That said, I imagine that's not the culprit.

Thanks.

I'll have to get the pipleline to shorten:

SPACK_ENV=/work/noaa/epic/role-epic/jenkins/workspace/spack-stack/ufswm-test/Orion-intel/weekly/envs/build-unified-dev-intel

echo $SPACK_ENV | wc -c
     115

[AlexanderRichert-NOAA]

I re-tested on Acorn after the last batch of updates and it was smooth sailing, so @climbfuji if you're still down to merge this sooner and build it out further over time, I'm fine with that.

[climbfuji]

I re-tested on Acorn after the last batch of updates and it was smooth sailing, so @climbfuji if you're still down to merge this sooner and build it out further over time, I'm fine with that.

Let's merge it after the spack dev update if that's ok

[AlexanderRichert-NOAA]

[AlexanderRichert-NOAA]

@climbfuji can you check that the contents of .github/workflows/ look correct? Was deleting the ubuntu containers CI one of your modifications?

[climbfuji]

@climbfuji can you check that the contents of .github/workflows/ look correct? Was deleting the ubuntu containers CI one of your modifications?

Oh no - that must have slipped through. Can you restore the file .github/workflows/ubuntu-ci-containers-x86_64.yaml please?

[AlexanderRichert-NOAA]

I just noticed that on Acorn, it was creating a directory called "file:" in $SPACK_ENV because of the way it handles the mirror path. I'm tweaking it to drop the "file://" part when it looks up the directory; @rickgrubin-tomorrow @climbfuji please re-test with this change and make sure I didn't break anything...

[climbfuji]

I just noticed that on Acorn, it was creating a directory called "file:" in $SPACK_ENV because of the way it handles the mirror path. I'm tweaking it to drop the "file://" part when it looks up the directory; @rickgrubin-tomorrow @climbfuji please re-test with this change and make sure I didn't break anything...

Yes, that's fine. I never used file:// in the spack mirror create command.