ch-run raises "No space left on device" error

[garykuok]

Hello,

I've been submitting some jobs on the MPI clusters with ch-run environment. Sometimes the job output will raise and error like the below, after several jobs are done without error (the image folder size will grow):

ch-run[18252]: error: No space left on device (ch_core.c:410 28)

I tried to bind the path to an arbitrary directory inside the image (e.g., /mnt/0) with write permission, it still raises the error. I'm sure that the directory I mount has more space than required (~100TB), so this error shouldn't happen.

My current solution is to remove the whole image and use ch-tar2dir to extract a new image again, then submit the jobs as usual, after some trials delete the image and extract it again. Is there a way to not limit the image folder size, so that it can grow as it is, and I don't need to remove and extract the image from time to time?

An example command is as this:

mpiexec -n $NTASKS ch-run -w -b $SCRATCH:/mnt/0 --cd /mnt/0 <image_name> -- <command>

Best,
Chon Kit

[heasterday]

Hello,

Looking through this, my first suspicion is that the filesystem where the image is residing (perhaps /tmp?) is being filled up by the application you are running. If this is the case I would recommend doing a couple runs until you get the same error and then see where the growth is. If the extra space is being consumed by information you want to keep you should mount a directory to that path so that it is retained across runs.

If possible, I would recommend working on getting to the point where you do not need the -w flag to run and all filespaces that are being written to have a mount.

Let me know if you have questions.

[garykuok]

Hello,

The image folder is at my scratch directory. When running the image, does charliecloud also puts some stuff in the /tmp folder? If not, then the directory does have enough space for new output.
I mounted the image with -b and without -w flag, copied the executable out from the image to the mount directory. The problem occur less now. Still, in some large node configuration on HPC it will still raise this error (e.g., using in total of 896 cores and 1568GB RAM).

A similar error happens on using the raw method (pure mpi command without ch-run) is, sometimes it raises the signal 9 error, which is caused by insufficient memory. I was thinking if the error in charliecloud is actually referring to insufficient memory instead of not having enough storage space?

One more add up to this is, the raw method will have the signal 9 error in higher configuration setting (e.g., double of the above), whereas charliecloud will happen in lower level setting as well (and for sure the same setting to the raw method also raises the error).

In performance comparison, I see charliecloud is much slower than the raw method in heavy computation, is charliecloud suitable for running heavy computation job? Or is charliecloud typically designed for lightweight jobs?

Best,
Chon Kit

[reidpr]

In performance comparison, I see charliecloud is much slower than the raw method in heavy computation, is charliecloud suitable for running heavy computation job? Or is charliecloud typically designed for lightweight jobs?

There are lots of studies showing that container overhead is minimal or perhaps even truly zero, across multiple container implementations including Charliecloud. However build differences and other environment differences can have an impact. That needs diagnosis on a case-by-case basis.

To answer your question directly, yes Charliecloud is suitable for compute-bound HPC jobs.