DM-42579: Add call to allocateNodes inside bps #44
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In HPC centers that do not have HTCondor pools natively, the necessary resources to execute HTCondor jobs/workflows must be provisioned manually by the LSST users using ``allocateNodes.py`` that creates glideins using center's native batch system like Slurm. I added the class that should automate this process by adding a special service node to the HTCondor's DAGMan workflow that will be running the script ensuring that these glideins will be created as needed automatically during the workflow execution.
Added a mechanism allowing for storing and adding a service node specification to the class representing an HTCondor DAG.
Added a logic which will allow the plugin to enable automatic resource provisioning during workflow execution.
Added a module with the default settings related to provisioning compute resources automatically during a workflow execution.
Provisioning job will always run on the access point (aka submit node). As a result, there's no need for transferring any files. I modified the HTCondor submit description file accordingly.
Modified attributes of the provisioning job and HTCDag so it shows up when 'bps report' is run.
Attempting to cover different scenarios when handling the configuration file that may be required by the provisioning script made Provisioner.configure() overly complex. I decided to limit its scope so it just handles two basic use cases: 1. The configuration file does not exists and needs to be created from scratch based on the provided BPS configuration. 2. The configuration file already exists and will be used as is. If the user wishes to recreate the configuration file for the provisioning script based on new/updated settings, they will need to manually delete the existing one.
Changed ``bps report`` behavior so it will display the information related to the resource provisioning only when it was enabled in the BPS configuration.
Added a new section to the existing documenatation that describes how to enable automatic provisioning of the resources.
``ctrl_execute`` is a part of the **lsst_distrib** from some time. Removed the note which claimed otherwise.
The search options necessary to find settings for the provisioning job were defined outside of the Provisioner class. Made them part of the class with on option to override them in necessary.
The methods of the Provisioner class need to be called in certain order, but there were no safeguards to ensure it. Added them.
A wrong exception type was used in the safeguard that was supposed to catch any issues with writing the provisioning script config. Fixed it.
Fixed some typos, reworded selected comments, and renamed a variable to make code easier to read.
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #44 +/- ##
==========================================
+ Coverage 57.41% 60.86% +3.44%
==========================================
Files 8 10 +2
Lines 2078 2279 +201
Branches 366 400 +34
==========================================
+ Hits 1193 1387 +194
- Misses 843 845 +2
- Partials 42 47 +5 ☔ View full report in Codecov by Sentry. |
There was not an easy way to include custom command line options when calling the provisioning script. Added a new configuration option to address this limitation.
mxk62
force-pushed
the
tickets/DM-42579
branch
from
654656b
to
a5f603d
Compare
mxk62
force-pushed
the
tickets/DM-42579
branch
from
a5f603d
to
9f5f8a1
Compare
MichelleGower
approved these changes
Sep 23, 2024
When DAGMan deletes the service job on its exits this job's status is set to DELETED. For this reason to avoid confusing users the provisioning job status was shown by ``bps report `` only while the workflow was running. As a result there is no way to easily know if the run was using automatic provisioning of the resources after it is finished. Made changes to always report the provisioning job status, but if the provisioning job was deleted by DAGMan its final status is changed to SUCCEEDED and displayed as such.
mxk62
force-pushed
the
tickets/DM-42579
branch
2 times, most recently
from
934e213
to
252429e
Compare
Fixed a type descriptions in _get_state_counts_from_jobs() docstring and rename variable to follow the naming convention used in other functions.
mxk62
force-pushed
the
tickets/DM-42579
branch
2 times, most recently
from
976e511
to
058711b
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Checklist
doc/changes