Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extended container support #1363

Open
wants to merge 4 commits into
base: master
Choose a base branch
from

Conversation

bernt-matthias
Copy link
Contributor

@bernt-matthias bernt-matthias commented Apr 11, 2023

  • use yaml based job config (simplifies future extensions)
  • allow singularity in addition to docker
    • → tool developers do not need to install docker
  • allow to specify container resolver config file OR allow to enable/disable/configure certain container resolvers

@mvdbeek what would be your ideas on the 3rd point? Just a single option allowing to set a custom config file / multiple options? Would it be a good idea to use cached containers from CVMFS in IWC workflows .. or is there no advantage with this?

@bernt-matthias bernt-matthias marked this pull request as draft April 11, 2023 19:07
@mvdbeek
Copy link
Member

mvdbeek commented Apr 11, 2023

  • weekly CI can run with custom config that does not use the building solver (so we will recognize more easily if a container was not added to biocontainers)

there's a linter for that, no need ro run through tool tests.

  • IWC could use singularity containers on CVMFS

I would not at all like to do that. Less cvmfs is good.

  • → tool developers do not need to install docker

singularity is hard to install on linux and impossible to install on OSX

@bernt-matthias
Copy link
Contributor Author

there's a linter for that, no need ro run through tool tests.

for IUC tool tests? Where is this defined? I never have seen this in action .. maybe its not working (galaxyproject/tools-iuc#5240).

singularity is hard to install on linux and impossible to install on OSX

Hah. My impression on Ubuntu was exactly the opposite. singularity worked out of the box while docker needed lots of google searches and manually editing config files. But maybe I remember wrong.

Also thought that it might be good to use (or at least support) singularity in too tests because it seems to be the default choice for many installations .. basically most that run on a HPC, or? I guess OSX does not play a role for production systems (but only for some developers).

@mvdbeek
Copy link
Member

mvdbeek commented Apr 11, 2023

Where is this defined?

planemo lint --biocontainers

@bernt-matthias
Copy link
Contributor Author

bernt-matthias commented Apr 11, 2023

planemo lint --biocontainers

So we should add this to the weekly CI? Obviously we can't ad it to the PR workflow.

Edit. But I guess planemo lint --biocontainers would trigger all the other linting which we decided that we don't it in weekly CI.

@mvdbeek
Copy link
Member

mvdbeek commented Apr 11, 2023

I think that's a good idea, yeah.

@bernt-matthias
Copy link
Contributor Author

Seems to work with planemo lint --biocontainers tools/circos -s tests,output,inputs,help,general,command,citations,tool_xsd .. thanks for the tip. I tried to implement this here: galaxyproject/tools-iuc@7bfc486

@bernt-matthias bernt-matthias force-pushed the topic/container-ext branch 3 times, most recently from 388732f to c3113da Compare April 12, 2023 14:19
@bgruening
Copy link
Member

We have new users that have local SIF files and would like to use Galaxy. It would be nice if we could have a good integration into planemo for that.

Singularity can be installed with conda on Linux.

@mvdbeek
Copy link
Member

mvdbeek commented Aug 22, 2023

You still gotta set up one of the 3 things here: https://rootlesscontaine.rs/getting-started/apptainer/ . user namespaces are not that widely available on HPCs, the second option isn't really any safer than docker.

I said this for tool developers, for which installing docker is so much easier than installing singularity. I don't mind adding more support in planemo, but it's not an option for IWC or IUC.

@bernt-matthias
Copy link
Contributor Author

Hi @bgruening I was able to run

planemo test --biocontainers --singularity --container_resolvers_config_file ./container_resolvers_dest.yml --no_dependency_resolution --no_conda_auto_init --galaxy_python_version 3.7 --no_cleanup TOOL_FILE

With ./container_resolvers_dest.yml:

- type: cached_explicit_singularity
  cache_directory: /tmp/container_cache/
- type: cached_mulled_singularity
  cache_directory: /tmp/container_cache/

Not sure yet if this enforces to use the containers in the cache or allows to fall back to conda.

Maybe your contacts could test this?

Image naming is a bit tricky. For instance for mulled containers the : must not be replaced by _ (which singularity pull does) and also the .sif extension should be removed.

@bernt-matthias
Copy link
Contributor Author

Besides this we should have a few tests here. Could need a bit of inspiration where to put them etc...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants