Skip to content

Latest commit

 

History

History
395 lines (303 loc) · 19.9 KB

README.md

File metadata and controls

395 lines (303 loc) · 19.9 KB
Raw

Description

This module can be used to setup and install Spack on a VM. To actually run Spack commands to install other software use the spack-execute module.

This module generates a script that performs the following:

  1. Install system dependencies needed for Spack
  2. Clone Spack into a predefined directory
  3. Check out a specific version of Spack

There are several options on how to consume the outputs of this module:

Important

Breaking changes between after v1.21.0. spack-install module replaced by spack-setup and spack-execute modules. Details Below

Examples

use spack-setup with spack-execute

This will prepend the spack-setup script to the spack-execute commands.

  - id: spack-setup
    source: community/modules/scripts/spack-setup

  - id: spack-build
    source: community/modules/scripts/spack-execute
    use: [spack-setup]
    settings:
      commands: |
        spack install [email protected] target=x86_64

  - id: builder
    source: modules/compute/vm-instance
    use: [network1, spack-build]

use spack-setup with vm-instance or Slurm module

This will run spack-setup scripts on the downstream compute resource.

  - id: spack-setup
    source: community/modules/scripts/spack-setup

  - id: spack-installer
    source: modules/compute/vm-instance
    use: [network1, spack-setup]

OR

  - id: spack-setup
    source: community/modules/scripts/spack-setup

  - id: slurm_controller
    source: community/modules/scheduler/schedmd-slurm-gcp-v5-controller
    use: [network1, partition1, spack-setup]

Build starup-script with spack-runner output

This will use the generated spack-setup script as one step in startup-script.

  - id: spack-setup
    source: community/modules/scripts/spack-setup

  - id: startup-script
    source: modules/scripts/startup-script
    settings:
      runners:
      - $(spack-setup.spack-runner)
      - type: shell
        destination: "my-script.sh"
        content: echo 'hello world'

  - id: workstation
    source: modules/compute/vm-instance
    use: [network1, startup-script]

To see a full example of this module in use, see the hpc-slurm-gromacs.yaml example.

Environment Setup

Activating Spack

Spack installation produces a setup script that adds spack to your PATH as well as some other command-line integration tools. This script can be found at <install path>/share/spack/setup-env.sh. This script will be automatically added to bash startup by any machine that runs the spack_runner.

If you have multiple machines that all want to use the same shared Spack installation you can just have both machines run the spack_runner.

Managing Spack Python dependencies

Spack is configured with SPACK_PYTHON to ensure that Spack itself uses a Python virtual environment with a supported copy of Python with the package google-cloud-storage pre-installed. This enables Spack to use mirrors and build caches on Google Cloud Storage. It does not configure Python packages inside Spack virtual environments. If you need to add more Python dependencies for Spack itself, use the spack python command:

sudo -i spack python -m pip install package-name

Spack Permissions

System spack user is created - Default

By default this module will create a spack linux user and group with consistent UID and GID. This user and group will own the Spack installation. To allow a user to manually add Spack packages to the system Spack installation, you can add the user to the spack group:

sudo usermod -a -G spack <username>

Log out and back in so the group change will take effect, then <username> will be able to call spack install <package>.

Note

A background persistent SSH connections may prevent the group change from taking effect.

You can use the system_user_name, system_user_uid, and system_user_gid to customize the name and ids of the system user. While unlikely, it is possible that the default system_user_uid or system_user_gid could conflict with existing UIDs.

Use and existing user

Alternatively, if system_user_name is a user already on the system, then this existing user will be used for Spack installation.

OS Login User

If OS Login is enabled (default for most Cluster Toolkit modules) then you can provide an OS Login user name:

  - id: spack-setup
    source: community/modules/scripts/spack-setup
    settings:
      system_user_name: username_company_com

This will work even if the user has not yet logged onto the machine. When the specified user does log on to the machine they will be able to call spack install without any further configuration.

Pre-configured user

You can also use a startup script to configure a user:

  - id: spack-setup
    source: community/modules/scripts/spack-setup
    settings:
      system_user_name: special-user

  - id: startup
    source: modules/scripts/startup-script
    settings:
      runners:
        - type: shell
          destination: "create_user.sh"
          content: |
            #!/bin/bash
            sudo useradd -u 799 special-user
            sudo groupadd -g 922 org-group
            sudo usermod -g org-group special-user
        - $(spack-setup.spack_runner)

  - id: spack-vms
    source: modules/compute/vm-instance
    use: [network1, startup]
    settings:
      name_prefix: spack-vm
      machine_type: n2d-standard-2
      instance_count: 5

Chaining spack installations

If there is a need to have a non-root user to install spack packages it is recommended to create a separate installation for that user and chain Spack installations (Spack docs).

Steps to chain Spack installations:

  1. Get the version of the system Spack:

    $ spack --version

0.20.0 (e493ab31c6f81a9e415a4b0e0e2263374c61e758)
#       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
# Note commit hash and use in next step

  2. Clone a new spack installation:

    git clone -c feature.manyFiles=true https://github.com/spack/spack.git <new_location>/spack
git -C <new_location>/spack checkout <commit_hash>

  3. Point the new Spack installation to the system Spack installation. Create a file at <new_location>/spack/etc/spack/upstreams.yaml with the following contents:

    upstreams:
  spack-instance-1:
    install_tree: /sw/spack/opt/spack/

  4. Add the following line to your .bashrc to make sure the new spack is in your PATH.

    . <new_location>/spack/share/spack/setup-env.sh

Deprecations and Breaking Changes

The old spack-install module has been replaced by the spack-setup and spack-execute modules. Generally this change strives to allow for a more flexible definition of a Spack build by using native Spack commands.

For every deprecated variable from spack-install there is documentation on how to perform the equivalent action using commands and data_files. The documentation can be found on the inputs table below.

Below is a simple example of the same functionality shown before and after the breaking changes.

  # Before
  - id: spack-install
    source: community/modules/scripts/spack-install
    settings:
      install_dir: /sw/spack
      compilers:
      - [email protected] target=x86_64
      packages:
      - [email protected]%[email protected]

- id: spack-startup
    source: modules/scripts/startup-script
    settings:
      runners:
      - $(spack.install_spack_deps_runner)
      - $(spack.install_spack_runner)
  # After
  - id: spack-setup
    source: community/modules/scripts/spack-setup
    settings:
      install_dir: /sw/spack

  - id: spack-execute
    source: community/modules/scripts/spack-execute
    use: [spack-setup]
    settings:
      commands: |
        spack install [email protected] target=x86_64
        spack load [email protected] target=x86_64
        spack compiler find --scope site
        spack install [email protected]%[email protected]

- id: spack-startup
    source: modules/scripts/startup-script
    settings:
      runners:
      - $(spack-execute.spack-runner)

Although the old spack-install module will no longer be maintained, it is still possible to use the old module in a blueprint by referencing an old version from GitHub. Note the source line in the following example.

  - id: spack-install
    source: github.com/GoogleCloudPlatform/hpc-toolkit//community/modules/scripts/spack-install?ref=v1.22.1&depth=1

License

Copyright 2022 Google LLC

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

 http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Requirements

Name Version
terraform >= 1.0.0
google >= 4.42
local >= 2.0.0

Providers

Name Version
google >= 4.42
local >= 2.0.0

Modules

Name Source Version
startup_script github.com/GoogleCloudPlatform/hpc-toolkit//modules/scripts/startup-script v1.39.0&depth=1

Resources

Name Type
google_storage_bucket.bucket resource
local_file.debug_file_shell_install resource

Inputs

Name Description Type Default Required
caches_to_populate DEPRECATED

Use spack-execute module with the following commands can be used to populate a cache:
MIRROR_URL=gs://my-bucket
spack buildcache create --mirror-url $MIRROR_URL -af $(spack find --format /{hash});
spack gpg publish --mirror-url $MIRROR_URL;
spack buildcache update-index --mirror-url $MIRROR_URL --keys;
Defines caches which will be populated with the installed packages.

NOTE: GPG Keys should be installed before trying to populate a cache
with packages.

NOTE: The gpg_keys variable can be used to install existing GPG keys
and create new GPG keys, both of which are acceptable for populating a
cache.		 list(map(any)) null no
chgrp_group Deprecated: installation will be owned by group of system_user_name. If special group is needed, supply user with group assigned. string null no
chmod_mode chmod to apply to the Spack installation. Adds group write by default. Set to "" (empty string) to prevent modification.
For usage information see:
https://docs.ansible.com/ansible/latest/collections/ansible/builtin/file_module.html#parameter-mode		 string "g+w" no
chown_owner Deprecated: use system_user_name. string null no
compilers DEPRECATED

Use spack-execute module with the following commands can be used to install compilers:
spack install [email protected] target=x86_64
spack load [email protected] target=x86_64
spack compiler find --scope site
spack clean -s
spack unload [email protected]
Defines compilers for spack to install before installing packages.		 list(string) null no
concretize_flags DEPRECATED - spack concretize is now performed using the spack-execute module commands variable. string null no
configs DEPRECATED

Use spack-execute module with the following commands can be used to add a single config:
spack config --scope defaults add config:default:true
Alternatively, use data_files to transfer a config file and use the spack config add -f <file> command to add the config.

List of configuration options to set within spack.		 list(map(any)) null no
configure_for_google When true, the spack installation will be configured to pull from Google's Spack binary cache. bool true no
deployment_name Name of deployment, used to name bucket containing startup script. string n/a yes
environments DEPRECATED

Use spack-execute module with the following commands can be used to configure an environment:
if ! spack env list | grep -q my-env; then
  spack env create my-env
fi
spack env activate my-env
spack add [email protected] %[email protected]
spack concretize
spack install
Defines spack environments to configure.
For more information, see: https://spack.readthedocs.io/en/latest/environments.html.		 any null no
gpg_keys DEPRECATED

Use spack-execute module with the following commands can be used to create a new GPG key:
spack gpg init
  spack gpg create
Alternatively, data_files can be used to transfer an existing GPG key. Then use spack gpg trust <file> to add the key to the keyring.

GPG Keys to trust within spack.		 list(map(any)) null no
install_dir Directory to install spack into. string "/sw/spack" no
install_flags DEPRECATED - spack install is now performed using the spack-execute module commands variable. string null no
labels Key-value pairs of labels to be added to created resources. map(string) n/a yes
licenses DEPRECATED

Use spack-execute module with data_files variable to install license files:
data_files = [{
  source = "/abs/path/on/deployment/machine/license.lic"
  destination = "/sw/spack/etc/spack/licenses/license.lic"
}]
List of software licenses to install within spack.		 
list(object({
    source = string
    dest   = string
  }))
 null no
log_file DEPRECATED

All install logs are printed to stdout/stderr.
Execution log_file location can be set on spack-execute module.		 string null no
packages DEPRECATED

Use spack-execute module with the following commands can be used to install a package:
spack install [email protected] %[email protected]
Defines root packages for spack to install.		 list(string) null no
project_id Project in which the HPC deployment will be created. string n/a yes
region Region to place bucket containing startup script. string n/a yes
spack_cache_url DEPRECATED

Use spack-execute module with the following commands can be used to add a build cache:
spack mirror add --scope site  gs://my-build-cache
spack buildcache keys --install --trust
List of build caches for Spack.		 
list(object({
    mirror_name = string
    mirror_url  = string
  }))
 null no
spack_profile_script_path Path to the Spack profile.d script. Created by this module string "/etc/profile.d/spack.sh" no
spack_ref Git ref to checkout for spack. string "v0.20.0" no
spack_url URL to clone the spack repo from. string "https://github.com/spack/spack" no
spack_virtualenv_path Virtual environment path in which to install Spack Python interpreter and other dependencies string "/usr/local/spack-python" no
system_user_gid GID used when creating system user group. Ignored if system_user_name already exists on system. Default of 1104762903 is arbitrary. number 1104762903 no
system_user_name Name of system user that will perform installation of Spack. It will be created if it does not exist. string "spack" no
system_user_uid UID used when creating system user. Ignored if system_user_name already exists on system. Default of 1104762903 is arbitrary. number 1104762903 no

Outputs

Name Description
controller_startup_script Spack installation script, duplicate for SLURM controller.
gcs_bucket_path Bucket containing the startup scripts for spack, to be reused by spack-execute module.
spack_path Path to the root of the spack installation
spack_profile_script_path Path to the Spack profile.d script.
spack_runner Runner to be used with startup-script module or passed to spack-execute module.
- installs Spack dependencies
- installs Spack
- generates profile.d script to enable access to Spack
This is safe to run in parallel by multiple machines. Use in place of deprecated setup_spack_runner.
startup_script Spack installation script.
system_user_name The system user used to install Spack. It can be reused by spack-execute module to install spack packages.