Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: more examples of dynamic resource calculations #5626

Open
nick-youngblut opened this issue Dec 27, 2024 · 0 comments
Open

docs: more examples of dynamic resource calculations #5626

nick-youngblut opened this issue Dec 27, 2024 · 0 comments

Comments

@nick-youngblut
Copy link
Contributor

New feature

https://www.nextflow.io/docs/latest/process.html#dynamic-task-resources provides just one example of dynamic resource calculations:

process foo {
    memory { 2.GB * task.attempt }
    time { 1.hour * task.attempt }

    errorStrategy { task.exitStatus in 137..140 ? 'retry' : 'terminate' }
    maxRetries 3

    script:
    """
    your_command --here
    """
}

It does not show (or link to) more complex examples, which are non-trivial to implement, such as:

memory { 16.GB + input_file.size() / 10.GB }

The above calculation actually does not work: ERROR ~ Unknown method invocation div on Long type

So, it would helpful to show examples of:

  1. using input file sizes in the calculations
  2. perform

Usage scenario

More complex dynamic resource allocation examples than just memory { 2.GB * task.attempt }.

Suggest implementation

More examples in the docs. I would create just a PR, but as you can see by my failed example above, I do not know how to actually perform some of the more complex resource allocation calculations.

General comment on docs

As a more general reason for including such docs: users and developers obtain their information from ChatGPT, Claude, etc. So, the more publicly available Nextflow docs, the more helpful these LLM services will be in regards to Nextflow.

For this reason, it is best to include more details than less.

If you are worried that the docs will be too cluttered for users to find the relevant docs, that issue is becoming less of a concern as users rely more on web search via ChatGPT, Claude (possible with MCP), Perplexity, etc. to find the specific relevant docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant