Support for GlobusComputeExecutor #3607
Conversation
|
Aside from a lack of CI test, my main concern (from seeing people implement this before) is what goes on with different Parsl versions: there's one version in the endpoint (pinned to a specific version by Globus Compute) and another version on the submit side, and having different Parsl versions like that is out of scope for Parsl. This might be a case of documenting what users should expect to work or not work, or might be something deeper. At the very least we should be expecting them to be able to use the same combination of versions as used in CI. |
| config.addinivalue_line( | ||
| 'markers', | ||
| 'globus_compute: Marks tests that require a valid globus_compute target' | ||
| ) |
| endpoint_id = os.environ.get("GLOBUS_COMPUTE_ENDPOINT", | ||
| "4b116d3c-1703-4f8f-9f6f-39921e5864df") |
There was a problem hiding this comment.
While anyone could potentially run tests against our tutorial EP, I don't think we should suggest doing so.
| def __init__( | ||
| self, | ||
| endpoint_id: Optional[UUID_LIKE_T] = None, | ||
| task_group_id: Optional[UUID_LIKE_T] = None, | ||
| resource_specification: Optional[dict[str, Any]] = None, | ||
| user_endpoint_config: Optional[dict[str, Any]] = None, | ||
| label: str = "GlobusComputeExecutor", | ||
| batch_size: int = 128, | ||
| amqp_port: Optional[int] = None, | ||
| **kwargs, | ||
| ): |
There was a problem hiding this comment.
Consider adding the client parameter
yadudoc
force-pushed
the
globus_compute_executor.py
branch
from
8273cc6
to
3ed482e
Compare
yadudoc
force-pushed
the
globus_compute_executor.py
branch
from
3ed482e
to
495d009
Compare
Description
This PR adds a new
GlobusComputeExecutorthat wraps the Globus Compute SDK to allow Parsl to execute tasks via Globus Compute. This mechanism supports remote execution of tasks similar to the functionality thatparsl.channelsenabled and is a potential replacement.Since
GlobusComputeoften runs on remote machines that do not have a shared-filesystem with the parsl runtime, tests have been updated with a newshared_fsandstaging_requiredpytest markers. I have not added tests and CI actions to enable executing these tests against our CI system, but you can run tests locally with these steps:globus-compute-sdkwithpip install .[globus-compute]globus-compute-endpoint start <endpoint_name>export GLOBUS_COMPUTE_ENDPOINT=<endpoint_id>pytest -v -k "not shared_fs" --config parsl/tests/configs/globus_compute.py parsl/tests/Changed Behaviour
N/A
Fixes
Fixes # (issue)
Type of change
Choose which options apply, and delete the ones which do not apply.