diff --git a/doc/changes/DM-33497.api.md b/doc/changes/DM-33497.api.md deleted file mode 100644 index ba6cfd58..00000000 --- a/doc/changes/DM-33497.api.md +++ /dev/null @@ -1,5 +0,0 @@ -Several modification to multiple classes to support execution with Quantum-backed butler: -* `CmdLineFwk` class adds two new methods: `preExecInitQBB`, which only runs `PreExecInit` step of the execution to produce InitOutputs; and `runGraphQBB`, which executes `QuantumGraph` using Quantum-backed butler. -* Abstract classes `QuantumExecutor` and `QuantumGraphExecutor` do not accept `Butler` instance in their `execute()` methods. -* `MPGraphExecutor` and `SingleQuantumExecutor` methods updated to reflect above change and support execution with either full `Butler` or `LimitedButler`. -* New class `PreExecInitLimited` which performs pre-exec-init in case of Quantum-backed butler. The code that it shares with a regular `PreExecInit` class is now in their common base class `PreExecInitBase`. diff --git a/doc/changes/DM-33497.feature.md b/doc/changes/DM-33497.feature.md deleted file mode 100644 index b742d2b0..00000000 --- a/doc/changes/DM-33497.feature.md +++ /dev/null @@ -1,5 +0,0 @@ -Adds support for executing quantum graph using Quantum-backed butler. -`pipetask` adds two new commands which use new feature, this is mostly useful for BPS. -`pipetask` adds two new commands to support execution with Quantum-backed butler: -* `pre-exec-init-qbb` which runs `PreExecInit` step of the execution to produce InitOutputs. -* `run-qbb` which executes `QuantumGraph` (or individual quanta) using Quantum-backed butler. diff --git a/doc/changes/DM-34420.feature.md b/doc/changes/DM-34420.feature.md deleted file mode 100644 index 2f6a661b..00000000 --- a/doc/changes/DM-34420.feature.md +++ /dev/null @@ -1 +0,0 @@ -* Added --coverage and --cov-packages to allow for code coverage calculations when running diff --git a/doc/changes/DM-36162.feature.md b/doc/changes/DM-36162.feature.md deleted file mode 100644 index 33cacc66..00000000 --- a/doc/changes/DM-36162.feature.md +++ /dev/null @@ -1,3 +0,0 @@ -Added SeparablePipelineExecutor, a pipeline executor midway in capability between SimplePipelineExecutor and CmdLineFwk. -SeparablePipelineExecutor is designed to be run from Python, and lets the caller decide when each pipeline processing step is carried out. -It also allows certain pipeline steps to be customized by passing alternate implementations of execution strategies (e.g., custom graph builder). diff --git a/doc/changes/DM-37582.feature.rst b/doc/changes/DM-37582.feature.rst deleted file mode 100644 index 9f977bfb..00000000 --- a/doc/changes/DM-37582.feature.rst +++ /dev/null @@ -1 +0,0 @@ -`pipetask` will now produce QuantumGraph with resolved output references, even with execution butler option. diff --git a/doc/changes/DM-38601.bugfix.md b/doc/changes/DM-38601.bugfix.md deleted file mode 100644 index 5cabd74a..00000000 --- a/doc/changes/DM-38601.bugfix.md +++ /dev/null @@ -1,2 +0,0 @@ -Fixed `SingleQuantumExecutor` class to correctly handle the case with `clobberOutputs=True` and `skipExistingIn=None`. -Documentation says that complete quantum outputs should be removed in this case, but they were not removed. diff --git a/doc/changes/DM-38779.misc.rst b/doc/changes/DM-38779.misc.rst deleted file mode 100644 index 07ad842e..00000000 --- a/doc/changes/DM-38779.misc.rst +++ /dev/null @@ -1,5 +0,0 @@ -* ``SingleQuantumExecutor`` has been modified such that it no longer unresolves ``DatasetRef`` when putting the non-``PipelineTask`` datasets (such as packages and configs). - This has been done so that the refs in the quantum graph are preserved when they are written to a normal Butler. -* Fixed a race condition when ``pipetask run`` creates the graph with a timestamped output run and then executes it. - Previously the graph creation and run execution phases calculated their own timestamped output run and it would be possible for the execution output run to be one second later than the graph run. - Previously this did not matter (the graph run was being ignored) but with the change to always use the ``DatasetRef`` from the graph it becomes critical that they match. diff --git a/doc/changes/DM-38780.feature.md b/doc/changes/DM-38780.feature.md deleted file mode 100644 index 21a08ae0..00000000 --- a/doc/changes/DM-38780.feature.md +++ /dev/null @@ -1 +0,0 @@ -`pipetask` CLI adds new command `update-graph-run`. It updates existing quantum graph with new output run name and re-generates output dataset IDs. diff --git a/doc/changes/DM-38952.misc.md b/doc/changes/DM-38952.misc.md deleted file mode 100644 index ba90cec8..00000000 --- a/doc/changes/DM-38952.misc.md +++ /dev/null @@ -1,3 +0,0 @@ -Revive the previously-bitrotted pipeline mocking system. - -Most of the implementation has been moved to `pipe_base`, and the point at which mocking occurs has moved from execution to just before `QuantumGraph` generation, which changes which `pipetask` subcommands the `--mock` option is valid for. diff --git a/doc/changes/DM-39122.doc.md b/doc/changes/DM-39122.doc.md deleted file mode 100644 index 26e59db6..00000000 --- a/doc/changes/DM-39122.doc.md +++ /dev/null @@ -1 +0,0 @@ -Command line help for `pipetask run` is updated to reflect its correct clobbering behavior. diff --git a/doc/changes/DM-39294.misc.md b/doc/changes/DM-39294.misc.md deleted file mode 100644 index 720008fb..00000000 --- a/doc/changes/DM-39294.misc.md +++ /dev/null @@ -1,6 +0,0 @@ -The current pipeline directed graph nodes use dark gray as their background color. -It has been reported that it is difficult to read the black text on the dark gray background. -This ticket updates the directed graph color scheme with an aim towards making node text easier to read. -In the process of exploring what color schemes would be optimal to satisfy the aim of this ticket, it emerged that making use of the Rubin visual identity colors may be desirable. -This will help to make LSST pipeline graphs more instantly recognizable as Rubin-associated products. -Colors: https://rubin.canto.com/g/RubinVisualIdentity diff --git a/doc/changes/DM-39377.misc.md b/doc/changes/DM-39377.misc.md deleted file mode 100644 index 0868bb63..00000000 --- a/doc/changes/DM-39377.misc.md +++ /dev/null @@ -1 +0,0 @@ -The `saveMetadata` configuration field is now ignored by executors in this package, metadata is assumed to be saved for each task. diff --git a/doc/changes/DM-39661.api.rst b/doc/changes/DM-39661.api.rst deleted file mode 100644 index f00175c6..00000000 --- a/doc/changes/DM-39661.api.rst +++ /dev/null @@ -1,2 +0,0 @@ -* Added new ``resources`` parameter to ``SingleQuantumExecutor``, ``SimplePipelineExecutor``, and ``SeparablePipelineExecutor`` constructors. - This optional parameter is a `~lsst.pipe.base.ExecutionResources` object and allows the execution context to be passed into the `~lsst.pipe.base.PipelinesTask.runQuantum` method. diff --git a/doc/changes/DM-39661.feature.rst b/doc/changes/DM-39661.feature.rst deleted file mode 100644 index 14b88da9..00000000 --- a/doc/changes/DM-39661.feature.rst +++ /dev/null @@ -1,2 +0,0 @@ -* Added new command line options ``--cores-per-quantum`` and ``--memory-per-quantum``. - These can be used to pass some execution context into a quantum, allowing that quantum to change how it executes (maybe by using multiple threads). diff --git a/doc/changes/DM-39672.feature.md b/doc/changes/DM-39672.feature.md deleted file mode 100644 index 5bd21f32..00000000 --- a/doc/changes/DM-39672.feature.md +++ /dev/null @@ -1 +0,0 @@ -Make it possible to force failures in mocked pipelines from the command-line. diff --git a/doc/changes/DM-40254.feature.md b/doc/changes/DM-40254.feature.md deleted file mode 100644 index f746945b..00000000 --- a/doc/changes/DM-40254.feature.md +++ /dev/null @@ -1 +0,0 @@ -The output of the `pipetask ... --show=graph` now includes extended information about dataset references and their related datastore records. diff --git a/doc/changes/DM-40332.misc.md b/doc/changes/DM-40332.misc.md deleted file mode 100644 index edb239ed..00000000 --- a/doc/changes/DM-40332.misc.md +++ /dev/null @@ -1 +0,0 @@ -Improve logging and remove some obsolete code paths in SingleQuantumExecutor. diff --git a/doc/lsst.ctrl.mpexec/CHANGES.rst b/doc/lsst.ctrl.mpexec/CHANGES.rst index f851f436..7ceba9b2 100644 --- a/doc/lsst.ctrl.mpexec/CHANGES.rst +++ b/doc/lsst.ctrl.mpexec/CHANGES.rst @@ -1,3 +1,70 @@ +lsst-ctrl-mpexec v26.0.0 (2023-09-23) +===================================== + +New Features +------------ + +- Added support for executing quantum graph using Quantum-backed butler. + ``pipetask`` adds two new commands to support execution with Quantum-backed butler, mostly useful for BPS: + + * ``pre-exec-init-qbb`` which runs ``PreExecInit`` step of the execution to produce InitOutputs. + * ``run-qbb`` which executes ``QuantumGraph`` (or individual quanta) using Quantum-backed butler. (`DM-33497 `_) +- Added ``--coverage`` and ``--cov-packages`` to ``pipetask`` commands to allow for code coverage calculations when running (`DM-34420 `_) +- Added ``SeparablePipelineExecutor``, a pipeline executor midway in capability between ``SimplePipelineExecutor`` and ``CmdLineFwk``. + ``SeparablePipelineExecutor`` is designed to be run from Python, and lets the caller decide when each pipeline processing step is carried out. + It also allows certain pipeline steps to be customized by passing alternate implementations of execution strategies (e.g., custom graph builder). (`DM-36162 `_) +- ``pipetask`` will now produce ``QuantumGraph`` with resolved output references, even with execution butler option. (`DM-37582 `_) +- Added new command ``update-graph-run`` to ``pipetask``. + It updates existing quantum graph with new output run name and re-generates output dataset IDs. (`DM-38780 `_) +- Added new command line options ``--cores-per-quantum`` and ``--memory-per-quantum``. + These can be used to pass some execution context into a quantum, allowing that quantum to change how it executes (maybe by using multiple threads). (`DM-39661 `_) +- Made it possible to force failures in mocked pipelines from the command-line. (`DM-39672 `_) +- The output of the ``pipetask ... --show=graph`` now includes extended information about dataset references and their related datastore records. (`DM-40254 `_) + + +API Changes +----------- + +- Several modification to multiple classes to support execution with Quantum-backed butler: + + * ``CmdLineFwk`` class adds two new methods: ``preExecInitQBB``, which only runs ``PreExecInit`` step of the execution to produce InitOutputs; and ``runGraphQBB``, which executes ``QuantumGraph`` using Quantum-backed butler. + * Abstract classes ``QuantumExecutor`` and ``QuantumGraphExecutor`` do not accept ``Butler`` instance in their ``execute()`` methods. + * ``MPGraphExecutor`` and ``SingleQuantumExecutor`` methods updated to reflect above change and support execution with either full ``Butler`` or ``LimitedButler``. + * New class ``PreExecInitLimited`` which performs pre-exec-init in case of Quantum-backed butler. + The code that it shares with a regular ``PreExecInit`` class is now in their common base class ``PreExecInitBase``. (`DM-33497 `_) +- Added new ``resources`` parameter to ``SingleQuantumExecutor``, ``SimplePipelineExecutor``, and ``SeparablePipelineExecutor`` constructors. + This optional parameter is a `~lsst.pipe.base.ExecutionResources` object and allows the execution context to be passed into the `~lsst.pipe.base.PipelinesTask.runQuantum` method. (`DM-39661 `_) + + +Bug Fixes +--------- + +- Fixed ``SingleQuantumExecutor`` class to correctly handle the case with ``clobberOutputs=True`` and ``skipExistingIn=None``. + Documentation says that complete quantum outputs should be removed in this case, but they were not removed. (`DM-38601 `_) + + +Other Changes and Additions +--------------------------- + +- * ``SingleQuantumExecutor`` has been modified such that it no longer unresolves ``DatasetRef`` when putting the non- ``PipelineTask`` datasets (such as packages and configs). + This has been done so that the refs in the quantum graph are preserved when they are written to a normal Butler. + * Fixed a race condition when ``pipetask run`` creates the graph with a timestamped output run and then executes it. + Previously the graph creation and run execution phases calculated their own timestamped output run and it would be possible for the execution output run to be one second later than the graph run. + Previously this did not matter (the graph run was being ignored) but with the change to always use the ``DatasetRef`` from the graph it becomes critical that they match. (`DM-38779 `_) +- Revive the previously-bitrotted pipeline mocking system. + + Most of the implementation has been moved to `pipe_base`, and the point at which mocking occurs has moved from execution to just before `QuantumGraph` generation, which changes which `pipetask` subcommands the `--mock` option is valid for. (`DM-38952 `_) +- Updated the directed graph color scheme with an aim towards making node text easier to read. + The previous pipeline directed graph nodes used dark gray as their background color. + It had been reported that it is difficult to read the black text on the dark gray background. + In the process of exploring what color schemes would be optimal to satisfy the aim of this ticket, it emerged that making use of the Rubin visual identity colors may be desirable. + This will help to make LSST pipeline graphs more instantly recognizable as Rubin-associated products. + Colors: https://rubin.canto.com/g/RubinVisualIdentity (`DM-39294 `_) +- The ``saveMetadata`` configuration field is now ignored by executors in this package, metadata is assumed to be saved for each task. (`DM-39377 `_) +- Improved logging and removed some obsolete code paths in ``SingleQuantumExecutor``. (`DM-40332 `_) +- Command line help for ``pipetask run`` has been updated to reflect its correct clobbering behavior. + + lsst-ctrl-mpexec v25.0.0 (2023-02-28) =====================================