Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add singularity mini lesson #254

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 1 addition & 17 deletions episodes/docker-image-examples.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,23 +36,7 @@ Ask @mkuzak to make a PR to add extra for <https://github.com/escience-academy/d

## Using Containers on an HPC Cluster

It is possible to run containers on shared computing systems run by a university or national
computing center. As a researcher, you can build container images and test containers on your own
computer and then run your full-scale computing work on a shared computing
system like a high performance cluster or high throughput grid.

The catch? Most university and national computing centers do not support *running*
containers with Docker commands, and instead use a similar tool called Singularity or
Shifter. However, both of these programs can be used to run containers based on Docker container images,
so often people create their container image as a Docker container image, so they can
run it using either of Docker or Singularity.

There isn't yet a working example of how to use Docker container images on a shared
computing system, partially because each system is slightly different, but the
following resources show what it can look like:

- [Introduction to Singularity](https://carpentries-incubator.github.io/singularity-introduction/): See the episode titled "Running MPI parallel jobs using Singularity containers"
- [Container Workflows at Pawsey](https://pawseysc.github.io/container-workflows/): See the episode titled "Run containers on HPC with Shifter (and Singularity)"
[In this example](../instructors/singularity.md), you will learn about using running containers on HPC clusters.

## Seeking Examples

Expand Down
65 changes: 65 additions & 0 deletions instructors/singularity.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
---
title: Running Containers on HPC
teaching: 30
exercises: 0
---

::::::::::::::::::::::::::::::::::::::: objectives

- Learn how to convert Docker images to SIF
- Distinguish the `run` and `exec` subcommands

::::::::::::::::::::::::::::::::::::::::::::::::::

:::::::::::::::::::::::::::::::::::::::: questions

- How is singularity different to Docker?
- How do I use my Docker images on a shared HPC?

::::::::::::::::::::::::::::::::::::::::::::::::::

::::::::::::::::::::::::::::::::::::::::: callout

You can find more detail about using Singularity in the [singularity-introduction](https://carpentries-incubator.github.io/singularity-introduction/) Carpentries workshop.

::::::::::::::::::::::::::::::::::::::::::::::::::

Singularity is a container engine, like Docker.
However, unlike Docker, container images are stored as single files called `.sif` (Singularity Image Format).
For a number of reasons, Singularity suits shared High Performance Computing (HPC) environments much better than Docker, so is valuable to learn if you work in these environments.
A related tool called singularity is a fork of Singularity that generally has the same command line interface.

::: challenge
## Singularity Command Line Interface

Like we did with Docker, try to work out what commands Singularity has.
Which one do you think is the equivalent of `docker run`?
:::
::: solution
`singularity run` behaves similarly to `docker run`, but as we will see, the arguments are somewhat different.
:::

## Running Docker Containers

Since Singularity containers have their own file format, if we have a Docker image we want to run, it first has to be converted.
We can do this using `singularity pull`.
For example, we can pull the container we previously pushed to Docker Hub:

```bash
singularity pull docker://alice/alpine-python
```

This creates a file called `alpine_python.sif` in our working directory.
To run this container, we then use `singularity run`:

```bash
singularity run alpine_python.sif
```

## Singularity Exec

If we want to modify the command run in the container, we have to use `singularity exec`.
For example, to make Python add numbers like in our sum example, we could do:
```bash
apptainer exec alpine_python.sif python -c 'print(1 + 1)'
```
Loading