shub://* urls do not work

[karcaw]

Describe the bug
when i try to pull or run a shub://* image the image does not work:

To Reproduce

-➤ singularity pull shub://store.emsl.pnl.gov/ondemand/el7-hpc-prod:latest
FATAL:   While pulling shub image: error fetching image to cache: unable to check if  exists in cache: failed to create temporary file: open /home/efelix/.apptainer/cache/shub/tmp_1442884314: no such file or directory

-➤ singularity run shub://store.emsl.pnl.gov/ondemand/el7-hpc-prod:latest
FATAL:   Unable to handle shub://store.emsl.pnl.gov/ondemand/el7-hpc-prod:latest uri: unable to check if  exists in cache: failed to create temporary file: open /home/efelix/.apptainer/cache/shub/tmp_3050288960: no such file or directory

Expected behavior
The image would download, and run.

** Notes **
As far as i can tell by looking at the apptainer code (1.1.7) it downloads the manifest from sregistry, but it does not contain the Commit key needed: https://github.com/apptainer/apptainer/blob/744c94b5837d34d12d15cb87298b989b3c77f050/internal/pkg/client/shub/api.go#L51

and so fails, giving a very unhelpful message.

the pull should work per the sregistry docs: https://singularityhub.github.io/sregistry/docs/client

Can you tell me what I did wrong, or how to fix this?

i am running the quay.io/vanessa/sregistry:1.1.27 docker images currently.

[karcaw]

I am going to attempt to roll forward the version to 2.0.0 of quay.io/vanessa/sregistry to see if it is just a version issue.

[vsoch]

I think you probably want to use library:// - shub:// was mostly for Singularity Hub. I also am not developing for apptainer in mind (I am for SingularityCE) so I don't know what changes they have made, they likely no longer support shub.

[karcaw]

The library:// does work for pulls, but I am tryng to find a way to run a container without messing with a users environment, and the apptainer run command does not support it.

does the $ singularity pull shub://containers.page/collection/container:tag command on https://singularityhub.github.io/sregistry/docs/client work on your current version?

[vsoch]

shub:// was added at the early onset of development, and was intended for Singularity Hub. I also originally added support for Singularity Registry, but ultimately deprecated it in favor of the library API, which we created a community standard for. It likely hasn't been used (or tested) and should be removed, but most people use the library:// endpoint so it wasn't looked into.

The library:// does work for pulls, but I am tryng to find a way to run a container without messing with a users environment, and the apptainer run command does not support it.

Can you give me more detail on that? Perhaps you can pull the SIF and run the binary separately, and specifically what user environment mucking are you worried about / need to do?

[karcaw]

I am looking at deploying user runnable singularity containers for the OpenOndemand system for our clusters. I can easily set up a remote for my userid, but i need to write scripts for any user of the system to run a container, and so it would be nice to just run a remote container(that could be cached), without changing the users remote list. I can just pull, then run, delete... but i was looking for a one step command that would just work.

If you are not supporting the shub command, probably should take it out of the documentation online so others dont think it will work. If it works with SingularitCE, maybe its ok to leave it. with it in there it seems like it should work.

[vsoch]

I would hugely discourage an approach that requires a re-pull - that's going to tax the registry / networking quite a bit. And the cache just is saving to the user's home (or your custom Singularity cache) so is there any reason you can't prepare a custom shared cache and have people use it? Another approach that I'll suggest (and disclaimer I am the developer) is singularity-hpc, which allows you to install containers as modules - meaning your user would just load the module and use it. https://github.com/singularityhub/singularity-hpc

[karcaw]

I will look into singularity-hpc, but at the current time i am building a CI pipeline to build all our containers, and we were hoping to push them into the registry, and pull them directly from there. That process needs to happen, if we pull them into a common cache that is an option.

[vsoch]

All the containers in shpc (over 8k) are already pre built on BioContainers (sifs) or another actual registry.

Is there anything else you’d like help with or to chat about?

[karcaw]

Probably should address that the docs say that shub://* requests work, while they dont.. which was my original issue...

[karcaw]

also is there a better place to just chat about sregistry? slack, discord, etc?

[vsoch]

It’s an open source project that I accept contributions to, but don’t invest too much time in, so you are welcome to post a PR here for fixes and have discussion here. I am in a few community slacks if you want to chat but I think here is OK.