From 35d318885cbf27714d9e635ca2ad57c4344bf692 Mon Sep 17 00:00:00 2001 From: Murat Keceli Date: Sun, 8 Dec 2024 01:39:04 -0600 Subject: [PATCH 1/4] 1. Corrected grammatical errors, such as ensuring consistent use of possessive pronouns and verb agreement. 2. Fixed formatting issues, including consistent use of capitalization for terms like "GitLab Project" and "GitLab Group/SubGroup." 3. Improved clarity by rephrasing some sentences for better readability. 4. Added YAML syntax highlighting for code blocks to improve readability. 5. Ensured consistent use of terminology, such as "log in" instead of "login" when used as a verb. --- docs/services/gitlab-ci.md | 149 ++++++++++++++++++------------------- 1 file changed, 71 insertions(+), 78 deletions(-) diff --git a/docs/services/gitlab-ci.md b/docs/services/gitlab-ci.md index 79cf612c2b..81f3d7c147 100644 --- a/docs/services/gitlab-ci.md +++ b/docs/services/gitlab-ci.md @@ -1,30 +1,29 @@ # Continuous Integration via GitLab-CI ## GitLab-CI - GitLab is an application that offers combined functionality as git repository, issue tracker, and CI/CD platform. The ALCF implementation of the GitLab-CI environment leverages upstream GitLab runners combined with the [ECP's Jacamar custom executor](https://gitlab.com/ecp-ci/jacamar-ci). As CI/CD is built directly into GitLab, it can allow for tighter devops processes. +GitLab is an application that offers combined functionality as a git repository, issue tracker, and CI/CD platform. The ALCF implementation of the GitLab-CI environment leverages upstream GitLab runners combined with the [ECP's Jacamar custom executor](https://gitlab.com/ecp-ci/jacamar-ci). As CI/CD is built directly into GitLab, it can allow for tighter DevOps processes. - GitLab-CI is meant to provide CI/CD services for projects using GitLab-CI to store your git repositories and executing code on our HPC clusters. ALCF does not allow users to join your own private runners to our existing GitLab CI/CD environment and provides dedicated runners for our supported systems. +GitLab-CI is meant to provide CI/CD services for projects using GitLab-CI to store your git repositories and execute code on our HPC clusters. ALCF does not allow users to join their own private runners to our existing GitLab CI/CD environment and provides dedicated runners for our supported systems. Additional information, technical and user documentation, and community support can be found on [GitLab's Runner website](https://docs.gitlab.com/runner/). Also see [GitLab's CI/CD YAML syntax reference](https://docs.gitlab.com/ee/ci/yaml) for the full list of keywords supported by GitLab CI. - ALCF's GitLab-CI environment can be accessed by logging into the [ALCF GitLab-CI web portal](https://gitlab-ci.alcf.anl.gov) using your ALCF credentials (ALCF username and cryptocard token password). ## Quickstart -* A user Emails [ALCF Support](mailto:support@alcf.anl.gov) requesting access for your ALCF Project for [gitlab-ci.alcf.anl.gov](https://gitlab-ci.alcf.anl.gov) . +* A user emails [ALCF Support](mailto:support@alcf.anl.gov) requesting access for their ALCF Project for [gitlab-ci.alcf.anl.gov](https://gitlab-ci.alcf.anl.gov). * ALCF Support will add the ALCF Project to the appropriate system(s) via the Account and Project management system. -* ALCF will create a `GitLab Group/SubGroup` for the ALCF Project and map it to the appropriate ldap group that maps to the ALCF Project +* ALCF will create a `GitLab Group/SubGroup` for the ALCF Project and map it to the appropriate LDAP group that maps to the ALCF Project. * ALCF Support will reply back to the user and inform them that the project is created. -* User(s) will need to login to [gitlab-ci.alcf.anl.gov](https://gitlab-ci.alcf.anl.gov) and configure their initial GitLab profile. Users will add an SSH key so they can pull/push code to the gitlab server. -* User will then need to create a `GitLab Project` in your assigned `GitLab Group/SubGroup`. +* User(s) will need to log in to [gitlab-ci.alcf.anl.gov](https://gitlab-ci.alcf.anl.gov) and configure their initial GitLab profile. Users will add an SSH key so they can pull/push code to the GitLab server. +* User will then need to create a `GitLab Project` in their assigned `GitLab Group/SubGroup`. * CI/CD needs to be enabled for the GitLab Project. -* When ready to run CI/CD jobs, users will add add a `.gitlab-ci.yml` file to your git repositories. +* When ready to run CI/CD jobs, users will add a `.gitlab-ci.yml` file to their git repositories. * They will need to set any [ALCF specific variable(s)](gitlab-ci.md#alcf-specific-variables). _Example: `.gitlab-ci.yml` file_ -``` +```yaml # this include allows us to reference defaults in anl/ci-resource/defaults include: - project: 'anl/ci-resources/defaults' @@ -45,21 +44,21 @@ submit_batch: # CI jobs may have any name - id - hostname - echo "Running on $(hostname) with setuid shell runner" - ``` + For a more complete example, see the [.gitlab-ci.yml](https://gitlab-ci.alcf.anl.gov/anl/ci-resources/examples/large-example/-/blob/master/.gitlab-ci.yml?ref_type=heads) file in the [large-example](https://gitlab-ci.alcf.anl.gov/anl/ci-resources/examples/large-example) project. ## Glossary -* **Group** - A collection of projects. Certain settings can be applied at the `Group` level and apply down to all child `SubGroups` and/or `Projects`. When a ALCF Project is allocated resources on the GitLab-CI environment we will create a GitLab `Group` that will map to your ALCF Project allocation. +* **Group** - A collection of projects. Certain settings can be applied at the `Group` level and apply down to all child `SubGroups` and/or `Projects`. When an ALCF Project is allocated resources on the GitLab-CI environment, we will create a GitLab `Group` that will map to your ALCF Project allocation. * **Jacamar-CI** - A Custom Executor we use that runs jobs as a given user on the shell and is capable of submitting jobs to schedulers like Cobalt and PBS. -* **Job** - An individual set of commands that are ran. This is the lowest unit of GitLab-CI abstraction. +* **Job** - An individual set of commands that are run. This is the lowest unit of GitLab-CI abstraction. * **Pipeline** - GitLab organizes your jobs for each run into a `pipeline`. -* **Project** - GitLab Projects can be thought of as an individual git repository plus all services and features GitLab layers on top. This term is unrelated to the ALCF Project concept. ALCF Projects often map to ldap groups and/or quotas and allocations. -* **Stage** - A collection of jobs in a pipeline. Jobs in the next stage will not start till the jobs in the current stage complete. If a job fails, the pipeline will not run the following stages by default. -* **Triggering User** - The user whose actions causes a CI/CD job to run and who the Jacamar-CI executor will run the jobs as. Examples include pushing commits up to the server, creating a merge request, and/or merging one branch into another branch. +* **Project** - GitLab Projects can be thought of as an individual git repository plus all services and features GitLab layers on top. This term is unrelated to the ALCF Project concept. ALCF Projects often map to LDAP groups and/or quotas and allocations. +* **Stage** - A collection of jobs in a pipeline. Jobs in the next stage will not start until the jobs in the current stage complete. If a job fails, the pipeline will not run the following stages by default. +* **Triggering User** - The user whose actions cause a CI/CD job to run and who the Jacamar-CI executor will run the jobs as. Examples include pushing commits up to the server, creating a merge request, and/or merging one branch into another branch. ## Projects Using CI/CD -Any project with a git repository on the GitLab-CI environment has access to the CI/CD environment by default. In order to launch a shell job on a system you must already have access to that system. +Any project with a git repository on the GitLab-CI environment has access to the CI/CD environment by default. In order to launch a shell job on a system, you must already have access to that system. ### On-Boarding with CI/CD To gain access to the GitLab-CI environment, send an email to [support@alcf.anl.gov](mailto:support@alcf.anl.gov) requesting access for your project(s). @@ -69,106 +68,105 @@ Include with the request: * The ALCF Project shortname * The PI’s name -GitLab-CI jobs run as the triggering user on relevant systems. The triggering user's home directory will be used by Jacamar-CI to copy the git repository and cache files into `~/.jacamar-ci`. This job will run out of their home directory and consume filesystem quota. If you need more space you should try to reference files in any ALCF Project allocations you have on shared filesystems. Unfortunately the initial git clone must run out of `~/.jacamar-ci` in your home directory. +GitLab-CI jobs run as the triggering user on relevant systems. The triggering user's home directory will be used by Jacamar-CI to copy the git repository and cache files into `~/.jacamar-ci`. This job will run out of their home directory and consume filesystem quota. If you need more space, you should try to reference files in any ALCF Project allocations you have on shared filesystems. Unfortunately, the initial git clone must run out of `~/.jacamar-ci` in your home directory. -The triggering user is defined as the user account who caused the CI/CD pipeline to execute. Via scheduling a re-occurring job, pushing commits up to the server, creating a merge request, and/or merging a branch. When the CI/CD jobs run they will run as that user on the relevant systems. For a job to succeed the `triggering user` must have appropriate permissions and access to all relevant systems and files. +The triggering user is defined as the user account who caused the CI/CD pipeline to execute, via scheduling a re-occurring job, pushing commits up to the server, creating a merge request, and/or merging a branch. When the CI/CD jobs run, they will run as that user on the relevant systems. For a job to succeed, the `triggering user` must have appropriate permissions and access to all relevant systems and files. -### Initial Login and Profile setup of GitLab-CI -* Login to [gitlab-ci.alcf.anl.gov](https://gitlab-ci.alcf.anl.gov) using your username and Cryptocard token. +### Initial Login and Profile Setup of GitLab-CI +* Log in to [gitlab-ci.alcf.anl.gov](https://gitlab-ci.alcf.anl.gov) using your username and Cryptocard token. * Once logged in, add your public key you already have or created earlier so that it can be associated with your account. - * Click Profile icon on the upper right hand corner, then click "Edit Profile" + * Click the Profile icon on the upper right-hand corner, then click "Edit Profile"
![GitLab Profile Dropdown](files/gitlab-ci/GitlabProfileDropDown.png){ width="200" }
GitLab Profile Dropdown screenshot
- * Click "SSH Keys" on the left hand menu. + * Click "SSH Keys" on the left-hand menu.
![GitLab Profile Add SSH Key](files/gitlab-ci/GitlabAddSSHKey.png){ width="700" }
GitLab Profile Add SSH Key screenshot
- * Copy/Paste in your SSH public key into the large text box under the word Key - * On Linux, Unix, and OSX based systems using OpenSSH your SSH public key is commonly found at `~/.ssh/id_rsa.pub`. If using windows you will need to consult your applications documentation on the location of your public key. - * Give it a descriptive title such as the where the key resides, by default it will extract the name from the end of the public key if possible. + * Copy/Paste your SSH public key into the large text box under the word Key. + * On Linux, Unix, and OSX-based systems using OpenSSH, your SSH public key is commonly found at `~/.ssh/id_rsa.pub`. If using Windows, you will need to consult your application's documentation on the location of your public key. + * Give it a descriptive title such as where the key resides; by default, it will extract the name from the end of the public key if possible. * Click the `Add Key` button. The button is disabled until you paste a key. ### GitLab Projects (repositories) - GitLab takes a git repository, adds additional functionality, and calls it a `GitLab Project`. This is the most common level you will be interacting with GitLab at. Please do not confuse ALCF Projects with `GitLab Projects` as they are two separate things. ALCF Projects more closely map to the `GitLab Group/SubGroup` concept; which we explain in the next section. - Once you are assigned access to a `GitLab Group/SubGroup` you will be able to create arbitrary `GitLab Projects` underneath. Configuring CI/CD jobs for each independently. +GitLab takes a git repository, adds additional functionality, and calls it a `GitLab Project`. This is the most common level you will be interacting with GitLab at. Please do not confuse ALCF Projects with `GitLab Projects` as they are two separate things. ALCF Projects more closely map to the `GitLab Group/SubGroup` concept, which we explain in the next section. +Once you are assigned access to a `GitLab Group/SubGroup`, you will be able to create arbitrary `GitLab Projects` underneath, configuring CI/CD jobs for each independently. To create a new `GitLab Project`: -* In the left pane, click "Groups", and then click "Explore groups" link on the right. +* In the left pane, click "Groups", and then click the "Explore groups" link on the right.
![GitLab Your Groups](files/gitlab-ci/GitlabSelectYourGroup.png){ width="700" }
GitLab Your Groups Page screenshot
-* From the list in the "Explore groups" page, click the group you were informed corresponds to your `ALCF Project` +* From the list in the "Explore groups" page, click the group you were informed corresponds to your `ALCF Project`.
![GitLab Your Groups](files/gitlab-ci/GitlabSelectFromExplore.png){ width="700" }
GitLab Explore Groups Page screenshot
- -* Click the `New project` button near the upper right. If this is the first project you are creating you will have two large square buttons near the middle of the screen to create `GitLab SubGroups` or `GitLab Projects` +* Click the `New project` button near the upper right. If this is the first project you are creating, you will have two large square buttons near the middle of the screen to create `GitLab SubGroups` or `GitLab Projects`.
![GitLab Empty Group](files/gitlab-ci/GitlabNewSG_or_Project.png){ width="700" }
GitLab Empty Group Page screenshot
-* On the `Create new project` page, click `Create blank project` +* On the `Create new project` page, click `Create blank project`.
![GitLab Create New Project](files/gitlab-ci/GitlabNewProject1.png){ width="700" }
GitLab Create New Project screenshot
-* Fill in the `Project Name` field. The `Project slug` field will auto populate based on the `Project Name`, do not change it. If you are pushing an existing repository, you *MUST* uncheck the default `Initialize repository with a README` option. Failure to uncheck this option will result in a merge conflict that you will need to resolve manually between your existing "local" git repository and the one you just created on the server. +* Fill in the `Project Name` field. The `Project slug` field will auto-populate based on the `Project Name`; do not change it. If you are pushing an existing repository, you *MUST* uncheck the default `Initialize repository with a README` option. Failure to uncheck this option will result in a merge conflict that you will need to resolve manually between your existing "local" git repository and the one you just created on the server.
![GitLab Create New Project](files/gitlab-ci/GitlabNewProject2.png){ width="700" }
GitLab Create New Project screenshot
-* Click `Create project` button near the bottom -* After creating the project, navigate to "Settings" > "General", Expand the "Visibility, project features, permissions" section and enable the "CI/CD" toggle. +* Click the `Create project` button near the bottom. +* After creating the project, navigate to "Settings" > "General", expand the "Visibility, project features, permissions" section, and enable the "CI/CD" toggle. ### GitLab Groups/SubGroups (Folders) - GitLab organizes `GitLab Projects` into "folders" called `Groups` or `SubGroups`. When an ALCF Project is granted access to GitLab-CI a GitLab `Group` will be created with access for all members of that ALCF Project. Users will then be able to create arbitrary GitLab `Projects`. +GitLab organizes `GitLab Projects` into "folders" called `Groups` or `SubGroups`. When an ALCF Project is granted access to GitLab-CI, a GitLab `Group` will be created with access for all members of that ALCF Project. Users will then be able to create arbitrary GitLab `Projects`. - Each ALCF Project will have a top-level `Group` or `SubGroup` created with the ALCF Project’s name. It is used for organization in the multi-project environment and is required for implementing the needed level of security. The `Group` folder is where all of the your `GitLab Projects` are to be stored, you can additionally create new `SubGroups`, `Projects`, group variables, etc within your designated `Group`, `SubGroups`, and/or `Projects`. +Each ALCF Project will have a top-level `Group` or `SubGroup` created with the ALCF Project’s name. It is used for organization in the multi-project environment and is required for implementing the needed level of security. The `Group` folder is where all of your `GitLab Projects` are to be stored. You can additionally create new `SubGroups`, `Projects`, group variables, etc., within your designated `Group`, `SubGroups`, and/or `Projects`. To create a new `GitLab SubGroup`: -* In the left pane, click "Groups", and then click "Explore groups" link on the right. +* In the left pane, click "Groups", and then click the "Explore groups" link on the right.
![GitLab Your Groups](files/gitlab-ci/GitlabSelectYourGroup.png){ width="700" }
GitLab Your Groups Page screenshot
-* From the list in the "Explore groups" page, click the group you were informed corresponds to your `ALCF Project` +* From the list in the "Explore groups" page, click the group you were informed corresponds to your `ALCF Project`.
![GitLab Your Groups](files/gitlab-ci/GitlabSelectFromExplore.png){ width="700" }
GitLab Explore Groups Page screenshot
-* Click the `New subgroup` button near the upper right. If this is the first project you are creating you will have two large square buttons near the middle of the screen to create `GitLab SubGroups` or `GitLab Projects` +* Click the `New subgroup` button near the upper right. If this is the first project you are creating, you will have two large square buttons near the middle of the screen to create `GitLab SubGroups` or `GitLab Projects`.
![GitLab Empty Group](files/gitlab-ci/GitlabNewSG_or_Project.png){ width="700" }
GitLab Empty Group Page screenshot
-* On the `Create subgroup` page, enter the `Subgroup name`. `Subgroup slug` will auto populate, do not change it. +* On the `Create subgroup` page, enter the `Subgroup name`. `Subgroup slug` will auto-populate; do not change it.
![GitLab Create New Group](files/gitlab-ci/GitlabNewSubGroup.png){ width="700" }
GitLab Create New SubGroup screenshot
-* Click `Create subgroup` button near the bottom +* Click the `Create subgroup` button near the bottom. ### GitLab Runner Nodes - Each system is assigned one or more GitLab runner node(s) that are shared by all users in GitLab-CI. Each runner is only capable of running one users pipeline at a time. While multiple jobs in that pipeline may run in parallel. +Each system is assigned one or more GitLab runner node(s) that are shared by all users in GitLab-CI. Each runner is only capable of running one user's pipeline at a time, while multiple jobs in that pipeline may run in parallel. - Each node will have two runners available, `shell` and `batch`. `shell` will run shell jobs directly on the runner node as the user. `batch` will submit the job to the HPC cluster's scheduler that is paired to that node. You will need to select the appropriate runner in your `.gitlab-ci.yml` file for the job to be executed properly. For more details on the `.gitlab-ci.yml` file, please see upstream [docs](https://docs.gitlab.com/ee/ci/yaml/gitlab_ci_yaml.html). +Each node will have two runners available, `shell` and `batch`. `shell` will run shell jobs directly on the runner node as the user. `batch` will submit the job to the HPC cluster's scheduler that is paired to that node. You will need to select the appropriate runner in your `.gitlab-ci.yml` file for the job to be executed properly. For more details on the `.gitlab-ci.yml` file, please see upstream [docs](https://docs.gitlab.com/ee/ci/yaml/gitlab_ci_yaml.html). ### `.gitlab-ci.yml` Configuration Sections - GitLab uses a per repository `.gitlab-ci.yml` file. On any commit, merge request, or merge gitlab will attempt to trigger a CI/CD pipeline based on the contents of this file. Within the `.gitlab-ci.yml` file you can limit jobs to only run under certain conditions. A common workflow is to have linting and validation happen on every commit to a non-master/non-main branch. Larger more complex tasks are then performed when that branch is merged back into master/main. All jobs launched on a given event are organized into a `Pipeline`. You can watch the progress of your pipeline via the CI/CD pipeline page for your `Project`. +GitLab uses a per repository `.gitlab-ci.yml` file. On any commit, merge request, or merge, GitLab will attempt to trigger a CI/CD pipeline based on the contents of this file. Within the `.gitlab-ci.yml` file, you can limit jobs to only run under certain conditions. A common workflow is to have linting and validation happen on every commit to a non-master/non-main branch. Larger, more complex tasks are then performed when that branch is merged back into master/main. All jobs launched on a given event are organized into a `Pipeline`. You can watch the progress of your pipeline via the CI/CD pipeline page for your `Project`.
![GitLab CI/CD Pipeline Overview](files/gitlab-ci/GitlabPipelineOverview.png){ width="700" } @@ -181,11 +179,10 @@ To create a new `GitLab SubGroup`:
#### Tags - Tags are used to select which runner a job will be sent to. Improper tags can prevent your job from running and result in a failed job. -Tags should be added by extending the defaults in the 'anl/ci-resources/defaults' runner.yml file. ALCF specific tags are described here in case overrides are needed. +Tags are used to select which runner a job will be sent to. Improper tags can prevent your job from running and result in a failed job. Tags should be added by extending the defaults in the 'anl/ci-resources/defaults' runner.yml file. ALCF specific tags are described here in case overrides are needed. ##### ALCF Specific tags - Two tags are necessary to run on our systems. One tag will select which cluster the jobs are sent to. The other will determine if the job is to be run locally on the gitlab runner host, or if it is to be submitted to a job scheduler on an HPC cluster. +Two tags are necessary to run on our systems. One tag will select which cluster the jobs are sent to. The other will determine if the job is to be run locally on the GitLab runner host, or if it is to be submitted to a job scheduler on an HPC cluster. _Cluster Tag(s)_ @@ -197,19 +194,18 @@ _Job Type Tag(s)_ | tag | Description | |:---------:|:------------:| -| shell | This tag will execute the job locally on the gitlab runner host. | +| shell | This tag will execute the job locally on the GitLab runner host. | | batch | This tag will submit the job to the HPC cluster's job scheduler. | - #### Variables - Variables can be stored two ways, inline in the `.gitlab-ci.yml` file or as a setting in the GitLab `Group` or `Project` itself. Variables are exported as environment variables by gitlab-runner for each job and can be used inside the `.gitlab-ci.yml` file. +Variables can be stored two ways: inline in the `.gitlab-ci.yml` file or as a setting in the GitLab `Group` or `Project` itself. Variables are exported as environment variables by gitlab-runner for each job and can be used inside the `.gitlab-ci.yml` file. GitLab also has a list of [predefined variables](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html) available in every GitLab CI/CD pipeline. To set a variable directly in the `.gitlab-ci.yml` file, declare a `variables:` section with each `VariableName: "VariableValue"` being on its own line. `variables:` can be declared globally or in individual jobs. _Example: Declaring variables_ -``` +```yaml variables: GlobalVariable1: "Global Value 1" GlobalVariable2: "Global Value 2" @@ -223,7 +219,7 @@ job: To store variables in the `Group` or `Project` settings, in the left side menu, click `Settings>CI/CD`. Expand the Variables option on the right side frame. You can then add variables by clicking `Add variable`. -For for more details please set please see the upstream [docs](https://docs.gitlab.com/ee/ci/variables/#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file) +For more details, please see the upstream [docs](https://docs.gitlab.com/ee/ci/variables/#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file).
![GitLab CI/CD Variable settings ](files/gitlab-ci/GitlabVariables.png){ width="700" } @@ -236,14 +232,14 @@ For for more details please set please see the upstream [docs](https://docs.gitl
##### ALCF Specific Variables -If you are planning to submit jobs to a scheduler then you will need to specify a per system variable `ANL_${CLUSTER}_SCHEDULER_PARAMETERS`; where `${CLUSTER}` is the name of the cluster. This variable will contain any command line flags you would need to submit jobs as if you were on the command line / scripting. Please consult the below table for more info. +If you are planning to submit jobs to a scheduler, then you will need to specify a per system variable `ANL_${CLUSTER}_SCHEDULER_PARAMETERS`; where `${CLUSTER}` is the name of the cluster. This variable will contain any command line flags you would need to submit jobs as if you were on the command line/scripting. Please consult the below table for more info. | Cluster | Scheduler | Variable Name | Support docs | |:--------|:---------:|:-------------:|:------------:| | Polaris | PBS | ANL_POLARIS_SCHEDULER_PARAMETERS | [Polaris Getting Started](../polaris/getting-started.md) | _Example: Running a batch job_ -``` +```yaml include: - project: 'anl/ci-resources/defaults' ref: main @@ -263,7 +259,7 @@ batch_test: ``` #### Stages -Jobs can be organized into `stages`. Jobs in the next stage will not start until all dependencies in the previous stage have completed. This is often used if there building and testing steps required before code may be ran or packaged. These stages are assembled in a `Pipeline`, a directed graph of `stages`. By default GitLab includes the following stages executed in the below order : +Jobs can be organized into `stages`. Jobs in the next stage will not start until all dependencies in the previous stage have completed. This is often used if there are building and testing steps required before code may be run or packaged. These stages are assembled in a `Pipeline`, a directed graph of `stages`. By default, GitLab includes the following stages executed in the below order: ``` .pre build @@ -275,7 +271,7 @@ deploy You may declare your own stages by first declaring a `stages:` array near the top of your `.gitlab-ci.yml` file. Stages will be processed in the order given in the array. _Example: Declaring Stages_ -``` +```yaml stages: - stage1 - stage2 @@ -283,7 +279,7 @@ stages: ``` _Example: Pipeline with custom stages_ -``` +```yaml # this include allows us to reference defaults in anl/ci-resource/defaults include: - project: 'anl/ci-resources/defaults' @@ -317,9 +313,9 @@ test2: ``` #### Rules - GitLab allows for CI/CD jobs to be launched only if certain conditions are met. GitLab sets a series of variables in addition to any the user explicitly sets when a job launches. A job can check these variables and choose to run or not based on the results. This is often used to ensure certain jobs only run on commits, merge requests, and/or merges. By default if any rule matches it will run. You can override this behavior with commands like `when: never` when a conditional matches. +GitLab allows for CI/CD jobs to be launched only if certain conditions are met. GitLab sets a series of variables in addition to any the user explicitly sets when a job launches. A job can check these variables and choose to run or not based on the results. This is often used to ensure certain jobs only run on commits, merge requests, and/or merges. By default, if any rule matches, it will run. You can override this behavior with commands like `when: never` when a conditional matches. -For more details please set please see the upstream [docs](https://docs.gitlab.com/ee/ci/yaml/index.html#rules) +For more details, please see the upstream [docs](https://docs.gitlab.com/ee/ci/yaml/index.html#rules). Rules can use the following conditional checks: ``` @@ -332,7 +328,7 @@ when ``` _Example: GitLab job designed to only run on merge requests_ -``` +```yaml test1: rules: - if: $CI_COMMIT_TAG # Do not execute jobs for tag context @@ -347,14 +343,13 @@ test1: ``` #### Job Scheduling - GitLab provides pipeline scheduling functionality to support recurring pipelines, specified using a Cron-like syntax. Pipeline scheduling is available in the project sidebar under "Build" > "Pipeline schedules". For more details, see the [upstream documentation](https://gitlab-ci.alcf.anl.gov/help/ci/pipelines/schedules). - +GitLab provides pipeline scheduling functionality to support recurring pipelines, specified using a Cron-like syntax. Pipeline scheduling is available in the project sidebar under "Build" > "Pipeline schedules". For more details, see the [upstream documentation](https://gitlab-ci.alcf.anl.gov/help/ci/pipelines/schedules). #### Template Jobs - GitLab allows you to create `template jobs`, these are pieces of job specifications which can be included in jobs. Each `template job` name must begin with a period (.) and follow the same syntax as normal jobs. To instantiate a job based on the `template job` use the keyword `extends`. If your specific job declares a key/value already in the template, the specific job will overwrite it. +GitLab allows you to create `template jobs`, these are pieces of job specifications which can be included in jobs. Each `template job` name must begin with a period (.) and follow the same syntax as normal jobs. To instantiate a job based on the `template job`, use the keyword `extends`. If your specific job declares a key/value already in the template, the specific job will overwrite it. -Example: Use a job template so two tests will only run on merge requests -``` +_Example: Use a job template so two tests will only run on merge requests_ +```yaml .MR_rules: rules: - if: $CI_COMMIT_TAG # Do not execute jobs for tag context @@ -379,31 +374,29 @@ test2: ``` ### Console Output - To see the output of a job click on it in the GUI and it will show the STDOUT and STDERR from the job run. If the job did not launch successfully it will have error messages from gitlab-runner or Jacamar-CI or both. - Please be aware of any sensitive data you do not want exported or saved to the output console, such as passwords. - Please do not output large amounts of data from your jobs to the stdout. If your CI/CD job outputs large amounts of text to STDOUT or STDERR, consider redirecting it into a job log. +To see the output of a job, click on it in the GUI, and it will show the STDOUT and STDERR from the job run. If the job did not launch successfully, it will have error messages from gitlab-runner or Jacamar-CI or both. Please be aware of any sensitive data you do not want exported or saved to the output console, such as passwords. Please do not output large amounts of data from your jobs to the stdout. If your CI/CD job outputs large amounts of text to STDOUT or STDERR, consider redirecting it into a job log.
![GitLab CI/CD Add Variable](files/gitlab-ci/GitlabJobConsole.png){ width="700" }
GitLab Group Job Console
- ## Storage Use and Policy ### GitLab Project Quota - Each repository has a default quota of 1GB. Quota increases may be requested by emailing [Support](mailto:support@alcf.anl.gov). This quota is separate from the storage quotas allocated to ALCF Projects and ALCF Users on the HPC clusters and shared filesystems. +Each repository has a default quota of 1GB. Quota increases may be requested by emailing [Support](mailto:support@alcf.anl.gov). This quota is separate from the storage quotas allocated to ALCF Projects and ALCF Users on the HPC clusters and shared filesystems. -### CI/CD Filesystem usage - CI/CD jobs will run out of your home directory by default. Each job will begin by cloning the repository into a path under `~/.jacamar-ci` and will continue to write there unless you reference other destinations in your CI/CD job. You will need to ensure that you have the minimum amount of space for this runner operation. If you do not, the job will fail to run. Each gitlab runner will create a new sub directory under `~/.jacamar-ci` for itself, however it will reuse space for subsequent pipelines launched for that project on that runner. +### CI/CD Filesystem Usage +CI/CD jobs will run out of your home directory by default. Each job will begin by cloning the repository into a path under `~/.jacamar-ci` and will continue to write there unless you reference other destinations in your CI/CD job. You will need to ensure that you have the minimum amount of space for this runner operation. If you do not, the job will fail to run. Each GitLab runner will create a new subdirectory under `~/.jacamar-ci` for itself; however, it will reuse space for subsequent pipelines launched for that project on that runner. - It is recommended that if you need more space then your home directory can provide, that you leverage any ALCF Project space you may have been allocated on a shared filesystem. +It is recommended that if you need more space than your home directory can provide, you leverage any ALCF Project space you may have been allocated on a shared filesystem. ## GitLab-CI Access Termination Policy - Projects that have been inactive for at least 6 months will have their access disabled and their repositories deleted. Notification will be sent to the PI 30 days prior to the day of the action. +Projects that have been inactive for at least 6 months will have their access disabled and their repositories deleted. Notification will be sent to the PI 30 days prior to the day of the action. Inactivity is defined as, but not limited to: - * No new projects created - * No new commits to an existing project - * Prolonged period of continuously failing CI/CD jobs (In the case of re-occurring scheduled jobs) +* No new projects created +* No new commits to an existing project +* Prolonged period of continuously failing CI/CD jobs (In the case of re-occurring scheduled jobs) +--- \ No newline at end of file From 721293b1a7b31e96670c15400854398c4acba7b5 Mon Sep 17 00:00:00 2001 From: Murat Keceli Date: Sun, 8 Dec 2024 01:39:11 -0600 Subject: [PATCH 2/4] 1. Corrected capitalization: Changed "Gitlab" to "GitLab" for consistency with the official branding. 2. Corrected capitalization: Changed "devops" to "DevOps" for consistency with standard terminology. 3. Improved formatting: Removed extra spaces before the link in the "GitLab-CI" section. --- docs/services/continuous-integration.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/services/continuous-integration.md b/docs/services/continuous-integration.md index d262831de6..85f260cc33 100644 --- a/docs/services/continuous-integration.md +++ b/docs/services/continuous-integration.md @@ -11,5 +11,5 @@ Today it is a common practice to use a CI tool for defining pipelines and execut ## CI Tools at ALCF -### Gitlab-CI - [Gitlab](gitlab-ci.md) is an application that offers combined functionality as git repository, issue tracker, and CI/CD platform. The ALCF implementation of the Gitlab-CI environment leverages upstream gitlab runners combined with the [ECP's Jacamar custom executor](https://gitlab.com/ecp-ci/jacamar-ci). As CI/CD is built directly into Gitlab, it can allow for tighter devops processes. [Gitlab-CI](https://gitlab-ci.alcf.anl.gov) is meant to provide CI/CD services for projects using [Gitlab-CI](https://gitlab-ci.alcf.anl.gov) to store their git repositories. ALCF does not allow users to join their own private runners to our existing gitlab ci environment and provides runners on our supported systems. +### GitLab-CI +[GitLab](gitlab-ci.md) is an application that offers combined functionality as a git repository, issue tracker, and CI/CD platform. The ALCF implementation of the GitLab-CI environment leverages upstream GitLab runners combined with the [ECP's Jacamar custom executor](https://gitlab.com/ecp-ci/jacamar-ci). As CI/CD is built directly into GitLab, it can allow for tighter DevOps processes. [GitLab-CI](https://gitlab-ci.alcf.anl.gov) is meant to provide CI/CD services for projects using [GitLab-CI](https://gitlab-ci.alcf.anl.gov) to store their git repositories. ALCF does not allow users to join their own private runners to our existing GitLab CI environment and provides runners on our supported systems. \ No newline at end of file From 4c6bb9eb497635cad22d99cee4b69e255992dd16 Mon Sep 17 00:00:00 2001 From: Murat Keceli Date: Sun, 8 Dec 2024 01:39:13 -0600 Subject: [PATCH 3/4] 1. Fixed grammar: Changed "An automated processes" to "Automated processes" for subject-verb agreement. 2. Improved formatting: Removed unnecessary line break in the first bullet point for consistency. --- docs/services/getting-started.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/services/getting-started.md b/docs/services/getting-started.md index 242505245c..983581d14d 100644 --- a/docs/services/getting-started.md +++ b/docs/services/getting-started.md @@ -2,7 +2,5 @@ Below is a list of some of the services ALCF offers. -- [JupyterHub](jupyter-hub.md): An interactive computing environment for - Python and other languages. -- [Continuous Integration](continuous-integration.md): An automated processes to - help build, test, package, and deploy on ALCF systems. +- [JupyterHub](jupyter-hub.md): An interactive computing environment for Python and other languages. +- [Continuous Integration](continuous-integration.md): Automated processes to help build, test, package, and deploy on ALCF systems. \ No newline at end of file From f697f0c014f68d7deb08359b41c0564017144305 Mon Sep 17 00:00:00 2001 From: Murat Keceli Date: Sun, 8 Dec 2024 01:39:35 -0600 Subject: [PATCH 4/4] 1. Fixed grammatical errors: Corrected "provide" to "provides" in the "Queue Name" section and added missing commas for clarity. 2. Improved clarity: Added "has" in "after the Notebook has started" for grammatical correctness. 3. Enhanced readability: Added backticks around `%pip` and `%conda` to indicate they are commands. 4. Corrected minor formatting issues: Ensured consistent use of punctuation and spacing throughout the document. --- docs/services/jupyter-hub.md | 131 +++++++++++------------------------ 1 file changed, 39 insertions(+), 92 deletions(-) diff --git a/docs/services/jupyter-hub.md b/docs/services/jupyter-hub.md index 09cfe9bbb3..f42eeef6eb 100644 --- a/docs/services/jupyter-hub.md +++ b/docs/services/jupyter-hub.md @@ -1,14 +1,6 @@ # JupyterHub -JupyterHub is an open-source service application that enables users to launch -separate Jupyter instances on a remote server. [ALCF -JupyterHub](https://jupyter.alcf.anl.gov) provides access to Polaris with the same [authentication -protocol](../account-project-management/accounts-and-access/alcf-passcode-tokens.md) -that is used to access these systems, but through a web interface rather than a -terminal. On the [ALCF JupyterHub home page](https://jupyter.alcf.anl.gov), -users can choose their desired system. Upon selection, they'll be directed to -the sign-in page to enter their ALCF username and [passcode -token](../account-project-management/accounts-and-access/alcf-passcode-tokens.md). +JupyterHub is an open-source service application that enables users to launch separate Jupyter instances on a remote server. [ALCF JupyterHub](https://jupyter.alcf.anl.gov) provides access to Polaris with the same [authentication protocol](../account-project-management/accounts-and-access/alcf-passcode-tokens.md) that is used to access these systems, but through a web interface rather than a terminal. On the [ALCF JupyterHub home page](https://jupyter.alcf.anl.gov), users can choose their desired system. Upon selection, they'll be directed to the sign-in page to enter their ALCF username and [passcode token](../account-project-management/accounts-and-access/alcf-passcode-tokens.md).
![JupyterHub](files/Jupyter-0-login.png){ width="700" } @@ -19,135 +11,90 @@ We describe below how to use JupyterHub on Polaris in more detail. ## Polaris -The Polaris JupyterHub server runs on a Polaris login node and launches individual -users' environments on the compute nodes through the PBS job scheduler. -After the authentication step, the user will be presented with the -menu of the available job options to start the Jupyter instance. - -- Select a job profile: This field lists the available profiles, which is - limited to “Polaris Compute Node” at this time. -- Queue Name: This field provide a list of available queues on the system. -- Project List: This field displays the active projects associated with the user - on Polaris. -- Number of Nodes: This field allows the user to select the number of compute - nodes to be allocated. -- Runtime (minutes:seconds): This field allows the user to set the runtime of - the job in minutes and seconds. The user should refer to the [Polaris queue - scheduling policy](../polaris/running-jobs.md) - for minimum and maximum runtime allowed for the selected queue. -- File Systems: This field allows the user to select the file systems to be - mounted. By default all the file systems are selected. +The Polaris JupyterHub server runs on a Polaris login node and launches individual users' environments on the compute nodes through the PBS job scheduler. After the authentication step, the user will be presented with the menu of the available job options to start the Jupyter instance. + +- Select a job profile: This field lists the available profiles, which is limited to “Polaris Compute Node” at this time. +- Queue Name: This field provides a list of available queues on the system. +- Project List: This field displays the active projects associated with the user on Polaris. +- Number of Nodes: This field allows the user to select the number of compute nodes to be allocated. +- Runtime (minutes:seconds): This field allows the user to set the runtime of the job in minutes and seconds. The user should refer to the [Polaris queue scheduling policy](../polaris/running-jobs.md) for minimum and maximum runtime allowed for the selected queue. +- File Systems: This field allows the user to select the file systems to be mounted. By default, all the file systems are selected.
![Add options](files/Jupyter-6-job-options.png){ width="700" }
Polaris Job options
-Once the appropriate information is provided the user will click the “Start” -button and wait for the job to spawn. If there's an extended wait time due to a -lengthy job queue, the interface might time out, leading to the job's removal -from the queue. If not, the job kicks off and it begins to use up the user's -allocation based on the chosen job options. It's crucial for users to shut down -the server when resources are no longer required. Failing to do so will result -in continued consumption of the allocated time until the predetermined runtime -concludes. +Once the appropriate information is provided, the user will click the “Start” button and wait for the job to spawn. If there's an extended wait time due to a lengthy job queue, the interface might time out, leading to the job's removal from the queue. If not, the job kicks off and it begins to use up the user's allocation based on the chosen job options. It's crucial for users to shut down the server when resources are no longer required. Failing to do so will result in continued consumption of the allocated time until the predetermined runtime concludes.
![Job queued](files/Jupyter-3-job-queued.png){ width="700" }
Job queued
-> **_NOTE:_** If you would like to change your selection about where to run the -> Jupyter instance after the Notebook started, you need to stop the server to be -> able to see the drop-down menu again. +> **_NOTE:_** If you would like to change your selection about where to run the Jupyter instance after the Notebook has started, you need to stop the server to be able to see the drop-down menu again. ## Known Issues ### Spawn Failed: Timeout -This happens when the queue is backed up. Since Jupyter is interactive, it expects an immediate connect. -Therefore, it waits 5 minutes for your job to begin before throwing this error. -You can monitor the queue usage with [Gronk](https://status.alcf.anl.gov/#/polaris) and submit when there isn't a wait. - +This happens when the queue is backed up. Since Jupyter is interactive, it expects an immediate connection. Therefore, it waits 5 minutes for your job to begin before throwing this error. You can monitor the queue usage with [Gronk](https://status.alcf.anl.gov/#/polaris) and submit when there isn't a wait. ## Additional Notes - + ### Custom IPython Kernels -ALCF JupyterHub provides a set of pre-configured IPython kernels for the users -to select. However, users may need custom kernels with additional packages -installed. This can be achieved by first creating custom Python environments -either through [venv](https://docs.python.org/3/library/venv.html) or -[conda](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html). -More information on creating custom Python environments can be found in our -documentation for [Polaris](../polaris/data-science/python.md). -After activating the custom environment, `ipykernel` package needs to be -installed with the following command: +ALCF JupyterHub provides a set of pre-configured IPython kernels for the users to select. However, users may need custom kernels with additional packages installed. This can be achieved by first creating custom Python environments either through [venv](https://docs.python.org/3/library/venv.html) or [conda](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html). More information on creating custom Python environments can be found in our documentation for [Polaris](../polaris/data-science/python.md). After activating the custom environment, the `ipykernel` package needs to be installed with the following command: + ```bash pip install ipykernel ``` -Once `ipykernel` is installed, the custom kernel can be added to the list of -available kernels with the following command: + +Once `ipykernel` is installed, the custom kernel can be added to the list of available kernels with the following command: + ```bash -python -m ipykernel install --user --name custom_kernel_name +python -m ipykernel install --user --name custom_kernel_name ``` -where `custom_kernel_name` is the name of the kernel that will appear in the -kernel list. This name does not have to match the name of the environment, but -should not contain spaces. If you want more flexibility in naming, you can add -the `--display-name` argument as shown below. + +where `custom_kernel_name` is the name of the kernel that will appear in the kernel list. This name does not have to match the name of the environment, but should not contain spaces. If you want more flexibility in naming, you can add the `--display-name` argument as shown below. + ```bash -python -m ipykernel install --user --name custom_kernel_name --display-name "Polaris Python 3.11 Tensorflow 2.4.1" +python -m ipykernel install --user --name custom_kernel_name --display-name "Polaris Python 3.11 Tensorflow 2.4.1" ``` -Note that, you still need to provide `--name` with a simple name that does not -contain spaces. Additionally, you can also set environment variables for the -kernel with the `--env` argument, i.e: + +Note that you still need to provide `--name` with a simple name that does not contain spaces. Additionally, you can also set environment variables for the kernel with the `--env` argument, i.e: + ```bash python -m ipykernel install --user --name custom_kernel_name --env http_proxy http://proxy.alcf.anl.gov:3128 --env https_proxy http://proxy.alcf.anl.gov:3128 ``` + You can see the list of available kernels with the following command: + ```bash jupyter kernelspec list ``` -By default, the kernels are installed in the user's home directory under -`~/.local/share/jupyter/kernels/`. All the configuration is specified in the -`kernel.json` file under the kernel directory. For the example above, the path -for the json file will be -`~/.local/share/jupyter/kernels/custom_kernel_name/kernel.json`. You can edit -this file to add additional environment variables or change the display name. - -Once you've followed the steps above, your new kernel will be visible on -JupyterHub. It's recommended to perform these steps in a terminal, ideally on -the login node of the system you're using. After setting up a custom kernel, you -can easily add more packages directly within JupyterHub. Simply create a new -notebook using your custom kernel and use the %pip or %conda magic commands to -install packages. If you're on a compute node, remember to enable internet -access by configuring the `http_proxy` and `https_proxy` environment variables as -previously mentioned. + +By default, the kernels are installed in the user's home directory under `~/.local/share/jupyter/kernels/`. All the configuration is specified in the `kernel.json` file under the kernel directory. For the example above, the path for the json file will be `~/.local/share/jupyter/kernels/custom_kernel_name/kernel.json`. You can edit this file to add additional environment variables or change the display name. + +Once you've followed the steps above, your new kernel will be visible on JupyterHub. It's recommended to perform these steps in a terminal, ideally on the login node of the system you're using. After setting up a custom kernel, you can easily add more packages directly within JupyterHub. Simply create a new notebook using your custom kernel and use the `%pip` or `%conda` magic commands to install packages. If you're on a compute node, remember to enable internet access by configuring the `http_proxy` and `https_proxy` environment variables as previously mentioned. ### Accessing Project Folders -Jupyter file browser limits the user to view files and directories within their -home directory. To access directories located outside of the user home directory -a symbolic link to the directory must be created within the user home directory. -An example of this is: +The Jupyter file browser limits the user to view files and directories within their home directory. To access directories located outside of the user home directory, a symbolic link to the directory must be created within the user home directory. An example of this is: ```bash ln -s /project/ABC ~/ABC_project_link ``` -Please note that one can run any shell command directly on a Jupyter notebook by -simply adding an exclamation mark, `!`, to the beginning of the command. For -example, the above command can be run from a notebook cell as follows: + +Please note that one can run any shell command directly on a Jupyter notebook by simply adding an exclamation mark, `!`, to the beginning of the command. For example, the above command can be run from a notebook cell as follows: ```bash !ln -s /project/ABC ~/ABC_project_link ``` ### Ending a Jupyter Notebook running on a compute node -Failing to correctly end a running Jupyter Notebook will continue to consume the -selected project's allocation on the resource in question. When a user has -completed their task in Jupyter the user should stop the Jupyter instance -running on the compute node before logging out. To stop the Notebook, click the -“Control Panel” button in the top right, then click “Stop My Server”. + +Failing to correctly end a running Jupyter Notebook will continue to consume the selected project's allocation on the resource in question. When a user has completed their task in Jupyter, the user should stop the Jupyter instance running on the compute node before logging out. To stop the Notebook, click the “Control Panel” button in the top right, then click “Stop My Server”.
![Stop panel](files/Jupyter-4-stop-panel.png){ width="700" } @@ -159,8 +106,8 @@ running on the compute node before logging out. To stop the Notebook, click the
Stop server
- ## Resources + * Jupyter Lab [documentation](https://jupyterlab.readthedocs.io/en/stable/). * ALCF Hands-on HPC Workshop presentation on Python and Jupyter on Polaris: [slides](https://www.alcf.anl.gov/support-center/training-assets/python-jupyter-notebook-and-containers) and [video](https://youtu.be/fhCe5eO1RSM). -* ALCF webinar on JupyterHub: [slides](https://github.com/keceli/ezHPC/blob/main/webinar/jupyterhub_webinar.pdf) and [video](https://youtu.be/X9g9eQcYseI?feature=shared). +* ALCF webinar on JupyterHub: [slides](https://github.com/keceli/ezHPC/blob/main/webinar/jupyterhub_webinar.pdf) and [video](https://youtu.be/X9g9eQcYseI?feature=shared). \ No newline at end of file