Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add pre-install hook that raises an error in case keeppreviousinstall (used for rebuilds) is set but the installation directory still contains files #881

Open
wants to merge 2 commits into
base: 2023.06-software.eessi.io
Choose a base branch
from
Open

Add pre-install hook that raises an error in case keeppreviousinstall (used for rebuilds) is set but the installation directory still contains files #881

wants to merge 2 commits into from

Conversation

bedroge
Copy link
Collaborator

@bedroge bedroge commented Jan 23, 2025

This is an enhancement for #871 and implements the comment from @boegel in #871 (comment).

I've tested it locally with EESSI-extend for the following scenarios:

  • no keeppreviousinstall -> this always works, as expected
  • with --try-amend=keeppreviousinstall=True:
    • empty install dir -> works
    • only some empty subdirs in the install dir -> works
    • file in the root of install dir -> error
    • file in a subdir -> error

You can test it by loading EESSI-extend, overriding EASYBUILD_HOOKS and setting it to the hooks file of this PR, and doing a small build with or without --try-amend=keeppreviousinstall=True.

@bedroge bedroge added the enhancement New feature or request label Jan 23, 2025
@eessi-bot eessi-bot
Copy link

eessi-bot bot commented Jan 23, 2025

Instance eessi-bot-mc-aws is configured to build for:

  • architectures: x86_64/generic, x86_64/intel/haswell, x86_64/intel/sapphire_rapids, x86_64/intel/skylake_avx512, x86_64/amd/zen2, x86_64/amd/zen3, aarch64/generic, aarch64/neoverse_n1, aarch64/neoverse_v1
  • repositories: eessi.io-2023.06-software, eessi.io-2023.06-compat

@eessi-bot eessi-bot
Copy link

eessi-bot bot commented Jan 23, 2025

Instance eessi-bot-mc-azure is configured to build for:

  • architectures: x86_64/amd/zen4
  • repositories: eessi.io-2023.06-software, eessi.io-2023.06-compat

@bedroge bedroge mentioned this pull request Jan 23, 2025
Merged
@boegel boegel added the 2023.06-software.eessi.io 2023.06 version of software.eessi.io label Jan 24, 2025
@boegel
Copy link
Contributor

boegel commented Jan 24, 2025

bot: build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2

@eessi-bot eessi-bot
Copy link

eessi-bot bot commented Jan 24, 2025 •
edited
Loading

Updates by the bot instance eessi-bot-mc-aws (click for details)

@eessi-bot eessi-bot
Copy link

eessi-bot bot commented Jan 24, 2025 •
edited
Loading

Updates by the bot instance eessi-bot-mc-azure (click for details)

  • received bot command build repo:eessi.io-2023.06-software arch:x86_64/amd/zen2 from boegel

    • expanded format: build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2

  • handling command build repository:eessi.io-2023.06-software architecture:x86_64/amd/zen2 resulted in:

    • no jobs were submitted

@eessi-bot eessi-bot
Copy link

eessi-bot bot commented Jan 24, 2025 •
edited
Loading

New job on instance eessi-bot-mc-aws for CPU micro-architecture x86_64-amd-zen2 for repository eessi.io-2023.06-software in job dir /project/def-users/SHARED/jobs/2025.01/pr_881/42405

date job status comment
Jan 24 11:24:19 UTC 2025 submitted job id 42405 awaits release by job manager
Jan 24 11:24:53 UTC 2025 released job awaits launch by Slurm scheduler
Jan 24 11:25:55 UTC 2025 running job 42405 is running
Jan 24 11:33:02 UTC 2025 finished
😁 SUCCESS (click triangle for details)
Details
✅ job output file slurm-42405.out
✅ no message matching FATAL:
✅ no message matching ERROR:
✅ no message matching FAILED:
✅ no message matching required modules missing:
✅ found message(s) matching No missing installations
✅ found message matching .tar.gz created!
Artefacts
eessi-2023.06-software-linux-x86_64-amd-zen2-1737717953.tar.gzsize: 0 MiB (14895 bytes)
entries: 1
modules under 2023.06/software/linux/x86_64/amd/zen2/modules/all
no module files in tarball
software under 2023.06/software/linux/x86_64/amd/zen2/software
no software packages in tarball
other under 2023.06/software/linux/x86_64/amd/zen2
2023.06/init/easybuild/eb_hooks.py
Jan 24 11:33:02 UTC 2025 test result
😁 SUCCESS (click triangle for details)
ReFrame Summary
[ OK ] ( 1/10) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/29Aug2024-foss-2023b-kokkos %scale=1_node /aeb2d9df @BotBuildTests:x86-64-amd-zen2-node+default
P: perf: 441.115 timesteps/s (r:0, l:None, u:None)
[ OK ] ( 2/10) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/2Aug2023_update2-foss-2023a-kokkos %scale=1_node /04ff9ece @BotBuildTests:x86-64-amd-zen2-node+default
P: perf: 427.9 timesteps/s (r:0, l:None, u:None)
[ OK ] ( 3/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node %device_type=cpu /775175bf @BotBuildTests:x86-64-amd-zen2-node+default
P: latency: 1.92 us (r:0, l:None, u:None)
[ OK ] ( 4/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node %device_type=cpu /52707c40 @BotBuildTests:x86-64-amd-zen2-node+default
P: latency: 1.77 us (r:0, l:None, u:None)
[ OK ] ( 5/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node %device_type=cpu /b1aacda9 @BotBuildTests:x86-64-amd-zen2-node+default
P: latency: 3.84 us (r:0, l:None, u:None)
[ OK ] ( 6/10) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node %device_type=cpu /c6bad193 @BotBuildTests:x86-64-amd-zen2-node+default
P: latency: 4.09 us (r:0, l:None, u:None)
[ OK ] ( 7/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node /15cad6c4 @BotBuildTests:x86-64-amd-zen2-node+default
P: latency: 0.57 us (r:0, l:None, u:None)
[ OK ] ( 8/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node /6672deda @BotBuildTests:x86-64-amd-zen2-node+default
P: latency: 0.58 us (r:0, l:None, u:None)
[ OK ] ( 9/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.2-gompi-2023b %scale=1_node /2a9a47b1 @BotBuildTests:x86-64-amd-zen2-node+default
P: bandwidth: 7444.26 MB/s (r:0, l:None, u:None)
[ OK ] (10/10) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.1-1-gompi-2023a %scale=1_node /1b24ab8e @BotBuildTests:x86-64-amd-zen2-node+default
P: bandwidth: 7388.1 MB/s (r:0, l:None, u:None)
[ PASSED ] Ran 10/10 test case(s) from 10 check(s) (0 failure(s), 0 skipped, 0 aborted)
Details
✅ job output file slurm-42405.out
✅ no message matching ERROR:
✅ no message matching [\s*FAILED\s*].*Ran .* test case

boegel
boegel requested changes Jan 24, 2025
View reviewed changes
eb_hooks.py
@@ -450,6 +450,16 @@ def pre_fetch_hook_zen4_gcccore1220(self, *args, **kwargs):
print_msg("Updated build option 'force' to 'True'")


def pre_install_hook(self, *args, **kwargs):
"""Main pre install hook: trigger custom functions based on software name."""
if 'keeppreviousinstall' in self.cfg and self.cfg['keeppreviousinstall']:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bedroge I think we deserve some context here, where/how we're using keeppreviousinstall, and why we only want to use this only when empty directories are available.
Definitely point to #556 + #871

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: cc79fd7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@boegel boegel boegel requested changes

Requested changes must be addressed to merge this pull request.

Assignees
No one assigned
Labels
2023.06-software.eessi.io 2023.06 version of software.eessi.io enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@bedroge @boegel