Skip to content

Commit

Permalink
Merge pull request #70 from jmduarte/patch-1
Browse files Browse the repository at this point in the history
Quick start with apptainer
  • Loading branch information
valsdav authored Dec 13, 2023
2 parents 68c1407 + 0611bb1 commit d3e84b6
Showing 1 changed file with 23 additions and 21 deletions.
44 changes: 23 additions & 21 deletions content/software_envs/containers.md
Original file line number Diff line number Diff line change
@@ -1,42 +1,47 @@
Containers are a great solution to isolate a software environment, especially in batch systems like lxplus.
At the moment two container solutations are supported **Apptainer** ( previously called Singularity), and **Docker**.
Currently, two container solutations are supported **Apptainer** (previously called Singularity), and **Docker**.

### Using Singularity
### Using Apptainer

The unpacked.cern.ch service mounts on CVMFS contains many singularity images, some of which are suitable for machine
#### Quickstart
One-line access to TensorFlow + PyTorch + Numpy + JupyterLab on lxplus:
```bash
apptainer shell -B /afs -B /eos --nv /cvmfs/unpacked.cern.ch/registry.hub.docker.com/cmsml/cmsml:latest
```

#### More information

The unpacked.cern.ch service mounts on CVMFS contains many apptainer images, some of which are suitable for machine
learning applications.
A description of each of the images is beyond the scope of this document.
However, if you find an image which is useful for your application, you can use if by running a Singularity container
A description of each image is beyond the scope of this document.
However, if you find an image useful for your application, you can use it by running an Apptainer container
with the appropriate options.
For example:
```bash
singularity run --nv --bind <bind_mount_path> /cvmfs/unpacked.cern.ch/<path_to_image>
apptainer run --nv --bind <bind_mount_path> /cvmfs/unpacked.cern.ch/<path_to_image>
```

#### Examples

After installing package, you can then use GPU based machine learning algorithms. Two examples are supplied as an
example.

1. The first example aims at using a CNN to perform handwritten digits classification with `MNIST` dataset. The whole notebook can be found at [pytorch_mnist](../resources/gpu_resources/cms_resources/notebooks/pytorch_mnist.md). This example is modified from [an official `pytorch` example](https://github.com/pytorch/examples/tree/master/mnist).

2. The second example is modified from the simple MLP example from [`weaver-benchmark`](https://github.com/colizz/weaver-benchmark). The whole notebook can be found at [toptagging_mlp](../resources/gpu_resources/cms_resources/notebooks/toptagging_mlp.md).
After installing the package, you can then use GPU-based machine learning algorithms. Two examples are supplied.

1. The first example aims at using a CNN to perform handwritten digits classification with `MNIST` dataset. The notebook can be found at [pytorch_mnist](../resources/gpu_resources/cms_resources/notebooks/pytorch_mnist.md). This example is modified from [an official `pytorch` example](https://github.com/pytorch/examples/tree/master/mnist).

2. The second example is modified from the simple MLP example from [`weaver-benchmark`](https://github.com/colizz/weaver-benchmark). The notebook can be found at [toptagging_mlp](../resources/gpu_resources/cms_resources/notebooks/toptagging_mlp.md).

### Using Docker
Docker is not supported at the moment in the interactive node of lxplus (like lxplus-gpu). However Docker is supported
on HTCondor for job submission.
Docker is currently supported on lxplus9 interactive nodes (through emulation of the CLI with Podman) and
on HTCondor for job submission.

This option can be very handy for users, as HTCondor can pull images from any public registry, like
[DockerHub](https://hub.docker.com/) or [GitLab registry](https://gitlab.cern.ch/).
The user can follow this workflow:
1. Define a custom image on top of a commonly available pytorch or tensorflow image
2. Add the desidered packages and configuration
2. Add the desired packages and configuration
3. Push the docker image on a registry
4. Use the image in a HTCondor job

The rest of the page is a step by step tutorial for this workflow.
The rest of the page is a step-by-step tutorial for this workflow.

#### Define the image

Expand Down Expand Up @@ -81,7 +86,7 @@ The rest of the page is a step by step tutorial for this workflow.
```bash
docker push username/pytorch-condor-gpu:tag
```

3. Setup the condor job with a submission file `submitfile` as:

```bash
Expand All @@ -98,12 +103,9 @@ The rest of the page is a step by step tutorial for this workflow.
+JobFlavour = espresso
queue 1
```



4. For testing purpose one can start a job interactively and debug

```bash
condor_submit -interactive submitfile
```


0 comments on commit d3e84b6

Please sign in to comment.