Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase job time for FastQC and Fastq screen #59

Merged
merged 2 commits into from
Mar 28, 2024

Conversation

matrulda
Copy link
Collaborator

Solves issue where slurm job for FastQC would be cancelled due to time limit. I increased the time for fastq screen as well, since these jobs also takes longer with larger data sizes.

It would be more optimal to have a solution like nf-core pipelines have, where processes failing with this error code are re-run with incrementally increasing compute resources. I do however see no reason to implement this in this pipeline, since the long-term plan is to use switch to the nf-core/seqinspector.

@matrulda matrulda requested a review from Aratz March 27, 2024 11:51
Copy link
Contributor

@Aratz Aratz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me 👍

Although I have a few comments/questions:

  • will 6h be enough or do we want to increase it even more? We don't really have any scheduling issues on Miarka atm to we could go even higher.
  • Would it maybe be better to have this config in miarka-provision and thus avoid having to do PRs in seqreports, sequencing-report-service and miarka-provision every time we want update the config?

@matrulda
Copy link
Collaborator Author

matrulda commented Mar 27, 2024

Thanks for the review. Good points!

  1. Yeah, we could increase it even more. I think this should work for quite a while though. 25B is the largest flowcell on the NovaSeq X and we have already processed other ones without issues (not that many though). One issue with having a too generous timelimit is that jobs can become blocked due to maintenance. However that's mainly an issue when the timelimit it set to > 24 hours. Perhaps I should change it to 12 hours?
  2. Yes, I agree that this is cumbersome. My vision is that we in the future should able to add/overwrite config variables via the service. So that the service basically will run nextflow run -c <config from request> -c <config deployed by miarka-provision> -c <default pipeline config> [...]
    EDIT: Perhaps this is already a thing with the version that supports arbitrary pipelines?

@Aratz
Copy link
Contributor

Aratz commented Mar 28, 2024

  1. Yeah 12h sounds like a good compromise 👍
  2. I see, yes as of now one can add extra arguments but not send an extra configuration file. So what one could do is place the new config file on Miarka and specify its path as an extra argument

@matrulda
Copy link
Collaborator Author

Nice, we should look into that. I can create a ticket.

Also, the timelimit is now increased to 12h.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants