Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add skeleton for constructor manager documentation #113

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
128 changes: 123 additions & 5 deletions docs/developers/packaging.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,6 @@ The configuration is done through a `construct.yaml` file, documented [here][7].
the fly in the `build_installers.py` script found in `napari/packaging`.
Roughly, we will build this configuration file:


```yaml
# os-agnostic configuration
name: napari
Expand Down Expand Up @@ -209,7 +208,6 @@ On Windows, any Microsoft-blessed certificate will do. Our `constructor` fork al
a path to a PFX certificate and then have the Windows SDK `signtool` add the signature. Note that
`signtool` is not installed by default on Windows (but it is on GitHub Actions).


---

More details about our packaging infrastructure can be found in the [NAP-2 document][nap-2].
Expand All @@ -229,9 +227,9 @@ moving pieces being juggled to make this work. Let's begin by enumerating the st
shortcut creation when the user runs the installer on their machine.
2. `conda-standalone` is a frozen version of `conda`. Among its dependencies, we can find
`menuinst`, which handles the creation of shortcuts and menu entries.
4. `menuinst` was only used on Windows before our work, so we basically rewrote it to handle
3. `menuinst` was only used on Windows before our work, so we basically rewrote it to handle
cross-platform shortcuts.
3. `conda` interfaces with `menuinst` to delegate the shortcut creation. Since this was only enabled
4. `conda` interfaces with `menuinst` to delegate the shortcut creation. Since this was only enabled
on Windows, we needed to unlock the other platforms and rewrite the parts that assumed Windows
only behaviour. Surprise, this involved custom solver behaviour too!

Expand Down Expand Up @@ -263,7 +261,6 @@ Very fun! So where do all these packages live?
| `conda` | [jaimergp/conda @ `cep-menuinst`][10] | [jaimergp-forge/conda-feedstock][14] |
| `menuinst` | [jaimergp/menuinst @ `cep`][11] | [jaimergp-forge/menuinst-feedstock][15] |


Most of the forks live in `jaimergp`'s account, under a non-default branch. They are published
through the `jaimergp-forge` every time a commit to `master` is made. Versions are arbitrary here,
but they are set to be greater than the latest official version, and the `build` number is
Expand Down Expand Up @@ -312,6 +309,125 @@ a high-level list of the main changes introduced in the stack.
* Add signing for Windows
* Add notarization for macOS PKG

#### `constructor-manager` stack for handling updates and recovery

Constructor installers come with a `constructor-manager` stack that allows users to perform
"updates in place", recover from a broken installation, rollback to a previous version or restore
point and finally uninstall the entire application. The constructor manager stack is application
agnostic in that it can be used for any othe application using the updated constructor stack
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
agnostic in that it can be used for any othe application using the updated constructor stack
agnostic in that it can be used for any other application using the updated constructor stack

to build and distribute installers that may or may not handle additional plugins.

![Constructor manager stack](images/constructor_manager_stack.png)

This stack is composed of three main packages [19]:

* `constructor-manager-backend`: Provides a simplified command line interface on top of
`conda`, `mamba` and `conda-lock` to query for updates and perform the actual installation,
reset or uninstallation of the application.
* `constructor-manager-client`: Provides a Python API to interact with the backend and
perform the same operations as the CLI.
Comment on lines +324 to +328
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a little confusing to me. I guess I would consider the Python API the backend and the CLI to be the client? You know more about the implementation though so I defer to you on naming.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @aganders3 yes indeed this might be confusing. Let me explain and from there we can make adjustments

The backend right now is only a CLI, that under the hood calls mamba, conda, conda-lock, and does a bunch of admin work for us (update in place, restore points etc). This could be updated eventually to become something more of a cli, for example an actual server doing things. So in this case instead of pinging a local server, we ping an actual program CLI.

The client is what we use to communicate with this backend (currently a cli but maybe a server in the future). So that is why I went with that naming scheme. The client could be renamed API. I called it like that in the past actually. But in this case is one thing we use to communicate and do thins with the "server" (like a frontend backend thing).

But I am open to suggestions, not super important but I would like to get the naming "right" before publishing packages

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used to have the names

constructor-manager-cli and constructor-manager-api which were pretty accurate in what they are now, the new ones are more generic, maybe I should stick to those old names @aganders3 ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks - this makes sense.

I agree the names are not super important because the descriptions make things clear. I think I do prefer constructor-manager-cli and constructor-manager-api but not by a wide margin, and the new names leave things more open to changes down the line. Maybe split the difference and keep constructor-manager-backend and constructor-manager-api?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will go with constructor-manager and constructor-manager-api, that way we leave the cli to be more open and add a specific entrypoint in there constructor-manager-cli and leave the api one as the api that it is :)

Thanks for the feedback

* `constructor-manager-ui`: Provides a Qt-based UI to interact with the backend via the client
to perform the same operations as the CLI. Additonal shortcuts will be created to access
this application on the different operating systems. It can also be opened from within
`napari` in the `Help` menu.

The constructor based installers will contain 3 main environments when created:

* A `base` environment which will contain:
* `conda`: package and environment manager)
* `mamba`: conda "replacement" for performing faster solves, transactions and downloads.
* `conda-lock`: create locked and reproducible environments.
* A `constructor-manager` environment which will contain:
* `constructor-manager-backend` [napari/packaging#34](https://github.com/napari/packaging/pull/34)
* `constructor-manager-client` [napari/packaging#46](https://github.com/napari/packaging/pull/46)
* `constructor-manager-ui` [napari/packaging#63](https://github.com/napari/packaging/pull/63)
* An application environment, using the conventions `<package-name>-<version>`, for the example image above, `napari-0.4.15`

##### `constructor-manager-backend`

This package is in charge of providing a CLI utility that will work under the hood with `conda`,
`mamba` and `conda-lock` (living in the `base` environment) to provide the following
functionalities in the creation of bundle applications:

The backend can be called using the CLI program `constructor-manager` and the following actions are
available

* `check-updates`: Query for new updates available for the managed application by providing one
or more anaconda.org channels. By default we query the `conda-forge` channel located at [anaconda.org](https://anaconda.org/conda-forge/)
* `update`: update a current application to a new version of it. This process will create a new
conda environment following the convention `<package-name>-<new-version>`, create new menu
shortcuts [20], create a new restore point (using conda-lock), remove the old environment and
the corresponding shortcuts for the old versions of the managed application.
* `revert`: this will revert the current application to the previously installed version on the
computer if a restore point is found. This will follow a similar process of creating a new
conda environment with the convention `<package-name>-<old-version>`.
* `restore`: similar to revert but restore to a previously found state of the current version.
This command could become a single one by providing the specific restore file (which is created
by conda lock)
* `lock-environment`: create a lock file of the current state of the application environment.
This can be called by the client applications (using constructor-manager-client) to check for
changes in the environment. If no changes are detected, no lock is created.
* `uninstall`: to be implemented.
* `get-status`: get information on a currently running update/restore/revert in progress.

Some of these commands can be run in parallel others create a lock to prevent multiple instances
of an update/restore/revert process.

More information on the [README](https://github.com/goanpeca/packaging/blob/constructor-updater/constructor-manager-cli/README.md) of the package.

##### `constructor-manager-client`

This package which in its present form is a `qtpy` (worker and signal) based library providing an
API to call the `constructor-manager-backend`. It provides a python library to run workers and
query information on the different processes managed by the constructor manager.

The library is imported with

```python
from constructor_manager_client import api
```

* `check_updates`: Query for new updates available for the managed application by providing one
or more anaconda.org channels. By default we query the `conda-forge` channel located at [anaconda.org](https://anaconda.org/conda-forge/).
* `update`: update a current application to a new version of it. This process will create a new
conda environment following the convention `<package-name>-<new-version>`, create new menu
shortcuts [20], create a new restore point (using conda-lock), remove the old environment and
the corresponding shortcuts for the old versions of the managed application.
* `revert`: this will revert the current application to the previously installed version on the
computer if a restore point is found. This will follow a similar process of creating a new conda
environment with the convention `<package-name>-<old-version>`.
* `restore`: similar to revert but restore to a previously found state of the current version.
This command could become a single one by providing the specific restore file (which is created
by conda lock).
* `lock_environment`: create a lock file of the current state of the application environment.
This can be called by the client applications (using constructor-manager-client) to check
for changes in the environment. If no changes are detected, no lock is created.
* `uninstall`: to be implemented
* `get_status`: get information on a currently running update/restore/revert in progress.
* `open_application`: open a give application created with `conda` and `menuinst` un a cross
platform way
Comment on lines +390 to +408
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think you need to repeat all this here. Maybe just list any limitations of this API vs. the backend/CLI.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, this was a lot of repetition, will update!


Some of these commands can be run in parallel others create a lock to prevent multiple instances
of an update/restore/revert process.

This library is meant to be used in the application environment by the applications themselves so
they can query the information to know about new updates and either trigger them directly in a
detached process (to be implemented in a following version) or to open the `constructor-manager-ui`
to handle those actions directly.

More information on the [README](https://github.com/goanpeca/packaging/blob/constructor-cli/constructor-manager/README.md) of the package.

##### `constructor-manager-ui`

which in its present form is a pyqt/pyside based application. It provides a graphical interface
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
which in its present form is a pyqt/pyside based application. It provides a graphical interface
In its present form this is a pyqt/pyside based application. It provides a graphical interface

to run the different actions to manage an application built with `constructor`

The ui is called from the command line interface with `constructor-manager-ui <package-name>`.
Since the application will be called using standard application calls created by menuinst,
arguments will be passed with configuration files handled by `constructor-manager-client` calls.

![Constructor manager UI](images/constructor_manager_ui.png)

<!-- hyperlinks -->

[1]: https://github.com/conda-forge/napari-feedstock
Expand All @@ -332,4 +448,6 @@ a high-level list of the main changes introduced in the stack.
[16]: https://github.com/conda-forge/napari-feedstock/pulls?q=is%3Apr+sort%3Aupdated-desc+is%3Aclosed
[17]: https://anaconda.org/napari
[18]: https://github.com/napari/packaging/issues/15
[19]: https://github.com/napari/packaging
[20]: https://github.com/conda/menuinst/tree/cep-devel/menuinst
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be a typo but I'd just link to the root of the tree where the README is.

Suggested change
[20]: https://github.com/conda/menuinst/tree/cep-devel/menuinst
[20]: https://github.com/conda/menuinst/tree/cep-devel

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Yes, I think I copied the wrong link whoops!

[nap-2]: https://napari.org/dev/naps/2-conda-based-packaging.html