Merge pull request #266 from lsst/tickets/DM-40883-main

DM-40883 (main): Add v26 release notes
lsst · Sep 26, 2023 · 8e0072f · 8e0072f
2 parents 76ae3ab + e0f94d9
commit 8e0072f
Show file tree

Hide file tree

Showing 18 changed files with 67 additions and 41 deletions.
diff --git a/doc/changes/DM-33497.api.md b/doc/changes/DM-33497.api.md
diff --git a/doc/changes/DM-33497.feature.md b/doc/changes/DM-33497.feature.md
diff --git a/doc/changes/DM-34420.feature.md b/doc/changes/DM-34420.feature.md
diff --git a/doc/changes/DM-36162.feature.md b/doc/changes/DM-36162.feature.md
diff --git a/doc/changes/DM-37582.feature.rst b/doc/changes/DM-37582.feature.rst
diff --git a/doc/changes/DM-38601.bugfix.md b/doc/changes/DM-38601.bugfix.md
diff --git a/doc/changes/DM-38779.misc.rst b/doc/changes/DM-38779.misc.rst
diff --git a/doc/changes/DM-38780.feature.md b/doc/changes/DM-38780.feature.md
diff --git a/doc/changes/DM-38952.misc.md b/doc/changes/DM-38952.misc.md
diff --git a/doc/changes/DM-39122.doc.md b/doc/changes/DM-39122.doc.md
diff --git a/doc/changes/DM-39294.misc.md b/doc/changes/DM-39294.misc.md
diff --git a/doc/changes/DM-39377.misc.md b/doc/changes/DM-39377.misc.md
diff --git a/doc/changes/DM-39661.api.rst b/doc/changes/DM-39661.api.rst
diff --git a/doc/changes/DM-39661.feature.rst b/doc/changes/DM-39661.feature.rst
diff --git a/doc/changes/DM-39672.feature.md b/doc/changes/DM-39672.feature.md
diff --git a/doc/changes/DM-40254.feature.md b/doc/changes/DM-40254.feature.md
diff --git a/doc/changes/DM-40332.misc.md b/doc/changes/DM-40332.misc.md
diff --git a/doc/lsst.ctrl.mpexec/CHANGES.rst b/doc/lsst.ctrl.mpexec/CHANGES.rst
@@ -1,3 +1,70 @@
+lsst-ctrl-mpexec v26.0.0 (2023-09-23)
+=====================================
+
+New Features
+------------
+
+- Added support for executing quantum graph using Quantum-backed butler.
+  ``pipetask`` adds two new commands to support execution with Quantum-backed butler, mostly useful for BPS:
+
+  * ``pre-exec-init-qbb`` which runs ``PreExecInit`` step of the execution to produce InitOutputs.
+  * ``run-qbb`` which executes ``QuantumGraph`` (or individual quanta) using Quantum-backed butler. (`DM-33497 <https://jira.lsstcorp.org/browse/DM-33497>`_)
+- Added ``--coverage`` and ``--cov-packages`` to ``pipetask`` commands to allow for code coverage calculations when running (`DM-34420 <https://jira.lsstcorp.org/browse/DM-34420>`_)
+- Added ``SeparablePipelineExecutor``, a pipeline executor midway in capability between ``SimplePipelineExecutor`` and ``CmdLineFwk``.
+  ``SeparablePipelineExecutor`` is designed to be run from Python, and lets the caller decide when each pipeline processing step is carried out.
+  It also allows certain pipeline steps to be customized by passing alternate implementations of execution strategies (e.g., custom graph builder). (`DM-36162 <https://jira.lsstcorp.org/browse/DM-36162>`_)
+- ``pipetask`` will now produce ``QuantumGraph`` with resolved output references, even with execution butler option. (`DM-37582 <https://jira.lsstcorp.org/browse/DM-37582>`_)
+- Added new command ``update-graph-run`` to ``pipetask``.
+  It updates existing quantum graph with new output run name and re-generates output dataset IDs. (`DM-38780 <https://jira.lsstcorp.org/browse/DM-38780>`_)
+- Added new command line options ``--cores-per-quantum`` and ``--memory-per-quantum``.
+  These can be used to pass some execution context into a quantum, allowing that quantum to change how it executes (maybe by using multiple threads). (`DM-39661 <https://jira.lsstcorp.org/browse/DM-39661>`_)
+- Made it possible to force failures in mocked pipelines from the command-line. (`DM-39672 <https://jira.lsstcorp.org/browse/DM-39672>`_)
+- The output of the ``pipetask ... --show=graph`` now includes extended information about dataset references and their related datastore records. (`DM-40254 <https://jira.lsstcorp.org/browse/DM-40254>`_)
+
+
+API Changes
+-----------
+
+- Several modification to multiple classes to support execution with Quantum-backed butler:
+
+  * ``CmdLineFwk`` class adds two new methods: ``preExecInitQBB``, which only runs ``PreExecInit`` step of the execution to produce InitOutputs; and ``runGraphQBB``, which executes ``QuantumGraph`` using Quantum-backed butler.
+  * Abstract classes ``QuantumExecutor`` and ``QuantumGraphExecutor`` do not accept ``Butler`` instance in their ``execute()`` methods.
+  * ``MPGraphExecutor`` and ``SingleQuantumExecutor`` methods updated to reflect above change and support execution with either full ``Butler`` or ``LimitedButler``.
+  * New class ``PreExecInitLimited`` which performs pre-exec-init in case of Quantum-backed butler.
+    The code that it shares with a regular ``PreExecInit`` class is now in their common base class ``PreExecInitBase``. (`DM-33497 <https://jira.lsstcorp.org/browse/DM-33497>`_)
+- Added new ``resources`` parameter to ``SingleQuantumExecutor``, ``SimplePipelineExecutor``, and ``SeparablePipelineExecutor`` constructors.
+  This optional parameter is a `~lsst.pipe.base.ExecutionResources` object and allows the execution context to be passed into the `~lsst.pipe.base.PipelinesTask.runQuantum` method. (`DM-39661 <https://jira.lsstcorp.org/browse/DM-39661>`_)
+
+
+Bug Fixes
+---------
+
+- Fixed ``SingleQuantumExecutor`` class to correctly handle the case with ``clobberOutputs=True`` and ``skipExistingIn=None``.
+  Documentation says that complete quantum outputs should be removed in this case, but they were not removed. (`DM-38601 <https://jira.lsstcorp.org/browse/DM-38601>`_)
+
+
+Other Changes and Additions
+---------------------------
+
+- * ``SingleQuantumExecutor`` has been modified such that it no longer unresolves ``DatasetRef`` when putting the non- ``PipelineTask`` datasets (such as packages and configs).
+    This has been done so that the refs in the quantum graph are preserved when they are written to a normal Butler.
+  * Fixed a race condition when ``pipetask run`` creates the graph with a timestamped output run and then executes it.
+    Previously the graph creation and run execution phases calculated their own timestamped output run and it would be possible for the execution output run to be one second later than the graph run.
+    Previously this did not matter (the graph run was being ignored) but with the change to always use the ``DatasetRef`` from the graph it becomes critical that they match. (`DM-38779 <https://jira.lsstcorp.org/browse/DM-38779>`_)
+- Revive the previously-bitrotted pipeline mocking system.
+
+  Most of the implementation has been moved to `pipe_base`, and the point at which mocking occurs has moved from execution to just before `QuantumGraph` generation, which changes which `pipetask` subcommands the `--mock` option is valid for. (`DM-38952 <https://jira.lsstcorp.org/browse/DM-38952>`_)
+- Updated the directed graph color scheme with an aim towards making node text easier to read.
+  The previous pipeline directed graph nodes used dark gray as their background color.
+  It had been reported that it is difficult to read the black text on the dark gray background.
+  In the process of exploring what color schemes would be optimal to satisfy the aim of this ticket, it emerged that making use of the Rubin visual identity colors may be desirable.
+  This will help to make LSST pipeline graphs more instantly recognizable as Rubin-associated products.
+  Colors: https://rubin.canto.com/g/RubinVisualIdentity (`DM-39294 <https://jira.lsstcorp.org/browse/DM-39294>`_)
+- The ``saveMetadata`` configuration field is now ignored by executors in this package, metadata is assumed to be saved for each task. (`DM-39377 <https://jira.lsstcorp.org/browse/DM-39377>`_)
+- Improved logging and removed some obsolete code paths in ``SingleQuantumExecutor``. (`DM-40332 <https://jira.lsstcorp.org/browse/DM-40332>`_)
+- Command line help for ``pipetask run`` has been updated to reflect its correct clobbering behavior.
+
+
 lsst-ctrl-mpexec v25.0.0 (2023-02-28)
 =====================================