Allow importing on workers #5098
Open
+169
−64
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
DailyDreaming
requested changes
Sep 19, 2024
| @@ -3143,7 +3177,7 @@ def __init__( | |||
| self.cwlwf = cwlwf | |||
| self.cwljob = cwljob | |||
| self.runtime_context = runtime_context | |||
| self.cwlwf = remove_pickle_problems(self.cwlwf) | |||
| self.cwlwf = self.cwlwf | |||
| @@ -3111,7 +3141,11 @@ def hasChild(self, c: Job) -> Any: | |||
|
|
|||
|
|
|||
| def remove_pickle_problems(obj: ProcessType) -> ProcessType: | |||
| """Doc_loader does not pickle correctly, causing Toil errors, remove from objects.""" | |||
| """ | |||
There was a problem hiding this comment.
This function is only called in one place now, other than itself. Is it safe to remove and was the issue resolved?
| @@ -1,4 +1,5 @@ | |||
| """Implemented support for Common Workflow Language (CWL) for Toil.""" | |||
| import argparse | |||
| @@ -281,6 +282,21 @@ def add_cwl_options(parser: ArgumentParser, suppress: bool = True) -> None: | |||
| help=suppress_help or "Disable file streaming for files that have 'streamable' flag True", | |||
| dest="disable_streaming", | |||
| ) | |||
| parser.add_argument( | |||
| "--runImportsOnWorkers", "--run-imports-on-workers", | |||
There was a problem hiding this comment.
Can we add a test for this?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#5025
Changelog Entry
To be copied to the draft changelog by merger:
--runImportsOnWorkersto enable importing files on workers--importWorkersDiskto control how much disk space the import worker will useReviewer Checklist
issues/XXXX-fix-the-thingin the Toil repo, or from an external repo.camelCasethat want to be insnake_case.docs/running/{cliOptions,cwl,wdl}.rstMerger Checklist