Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Request: increase memory allocation for mashmap #833

Open
jennaj opened this issue Sep 10, 2024 · 4 comments
Open

Request: increase memory allocation for mashmap #833

jennaj opened this issue Sep 10, 2024 · 4 comments
Labels
site/org usegalaxy.org tools/resources Memory allocation for tool(s)

Comments

@jennaj
Copy link
Member

jennaj commented Sep 10, 2024

toolshed.g2.bx.psu.edu/repos/iuc/mashmap/mashmap/3.1.3+galaxy0

Another example job from Ross @fubar2 that fails at ORG but runs Ok at EU (same inputs/params/tool version). Seems to have similar default mem/cores at both. Maybe rolled over to higher mem at EU?

  • ORG 11ac94870d0bb33a863b3f3d81b29b0a
  • EU 11ac94870d0bb33ad74c8b84f6b01321
@jennaj jennaj added site/org usegalaxy.org tools/resources Memory allocation for tool(s) labels Sep 10, 2024
@fubar2
Copy link
Member

fubar2 commented Sep 14, 2024

@natefoo @mvdbeek @jennaj: tl;dr Mashmap is tricky as RAM useage depends heavily on input and parameters - 4GB - 60GB so there may be a good argument for setting boundaries on the % identity and match length to keep the RAM usage sane

Have tried a couple of sets of parameters with VGP sized inputs. The size of all the fasta matters, but a low percentage identity and a short match length will result in much, much higher RAM use.

For example, this with about 6GB of fasta @ 90% identity for 1000nt which is a relatively easy criterion to match so takes up a lot of RAM and almost 10X time:
image

whereas with the same fasta @ a much more stringent match requirement of 95% identity over 5000nt uses much less:
image

@fubar2
Copy link
Member

fubar2 commented Sep 16, 2024

@mvdbeek: With the current

Cores Allocated 	8
Memory Allocated (MB) 	28672

seeing OOM failures for 1k/93% and 5k/85% (MashMap defaults) settings on typical VGP inputs. Four other combinations and inputs work fine. https://usegalaxy.org/u/fubar/h/treevalgalaugtwohaps-hg002

@natefoo
Copy link
Member

natefoo commented Oct 2, 2024

Bumped to 64GB for VGP.

@fubar2
Copy link
Member

fubar2 commented Oct 3, 2024

All 5 different sets of mashmap settings ran without drama just now - thanks @nate!
Issue resolved I think - thanks everyone!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
site/org usegalaxy.org tools/resources Memory allocation for tool(s)
Projects
Development

No branches or pull requests

3 participants