Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge slurm deskew with mantis deskew CLI call #57

Open
ieivanov opened this issue Jul 17, 2023 · 1 comment
Open

Merge slurm deskew with mantis deskew CLI call #57

ieivanov opened this issue Jul 17, 2023 · 1 comment
Assignees
Labels
analysis Software development needed for data analysis enhancement New feature or request

Comments

@ieivanov
Copy link
Collaborator

Currently the mantis deskew CLI call processes positions consecutively. We can process positions in parallel using slurm by calling a python script: python slurm_deskew.py, after we modify the script with the right dataset parameters. It would be convenient to merge the two pipelines and add an optional --use_slurm flag to mantis deskew, e.g.

mantis deskew /input_path /deskew_params_path -o /output_path --use_slurm

with optional parameters such as --cpus_per_task or --mem_per_cpu.

Based on our discussion this morning, processing datasets with slurm is not fool-proof enough to be abstracted as a CLI call. At this point calling a python script is the more appropriate format. We decided to revisit that after more experience with running parallel slurm jobs.

@ieivanov ieivanov added enhancement New feature or request analysis Software development needed for data analysis labels Jul 17, 2023
@talonchandler
Copy link
Contributor

After discussion, @edyoshikun and I agree that this is not on the immediate critical path. We're planning to continue using slurmkit scripts for our analyses.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
analysis Software development needed for data analysis enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants