Upgrade: Add docs and tests (#436)

This PR adds MicroCloud upgrade tests from `1/stable` to `latest/edge`
as well as for the MicroCeph, MicroOVN and LXD dependencies.

As there might be MicroClouds running on 22.04, the test suite is
modified to allow specifying a `BASE_OS` that will be used when setting
up the testbed.
This allows running the new `upgrade` test suite for both 22.04 and
24.04.

To verify the upgrade we check the cluster status of MicroCloud and its
dependencies after the upgrade. In addition some workload is deployed
before the upgrade whose boot ID and network connectivity is checked
afterwards.

Furthermore dedicated docs are now added for update and upgrade
procedures.

.github/workflows/tests.yml

@@ -86,7 +86,9 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        go: ["1.22.x"]
+        # Test suites that will be combined with the set versions.
+        # Define this first in the matrix so that it's readable
+        # after GitHub as formed the name for the respective check.
         suite:
           - "add"
           - "instances"
@@ -95,6 +97,31 @@ jobs:
           - "interactive"
           - "mismatch"
           - "preseed"
+        # Set of versions to use for the matrix tests.
+        os: ["24.04"]
+        go: ["1.22.x"]
+        microceph: ["latest/edge"]
+        microovn: ["latest/edge"]
+        lxd: ["5.21/edge"]
+        microcloud: ["latest/edge"]
+        # Additional test suites that will get included using a different set of versions.
+        include:
+          # Upgrade MicroCloud from 1 to 2 on 22.04
+          - suite: "upgrade"
+            os: "22.04"
+            go: "1.22.x"
+            microceph: "reef/stable"
+            microovn: "22.03/stable"
+            lxd: "5.21/stable"
+            microcloud: "1/stable"
+          # Upgrade MicroCloud from 1 to 2 on 24.04
+          - suite: "upgrade"
+            os: "24.04"
+            go: "1.22.x"
+            lxd: "5.21/stable"
+            microceph: "reef/stable"
+            microovn: "22.03/stable"
+            microcloud: "1/stable"
 
     steps:
       - name: Performance tuning
@@ -224,16 +251,29 @@ jobs:
         run: |
           set -eux
           chmod +x ~
+
+          export BASE_OS="${{ matrix.os }}"
+          export LXD_SNAP_CHANNEL="${{ matrix.lxd }}"
+          export MICROCEPH_SNAP_CHANNEL="${{ matrix.microceph }}"
+          export MICROOVN_SNAP_CHANNEL="${{ matrix.microovn }}"
+          export MICROCLOUD_SNAP_CHANNEL="${{ matrix.microcloud }}"
+
           cd test
-          sudo --preserve-env=DEBUG,GITHUB_ACTIONS,MICROCLOUD_DEBUG_PATH,MICROCLOUDD_DEBUG_PATH,SKIP_VM_LAUNCH,SNAPSHOT_RESTORE,TEST_STORAGE_SOURCE,TESTBED_READY ./main.sh setup
+          sudo --preserve-env=DEBUG,GITHUB_ACTIONS,MICROCLOUD_DEBUG_PATH,MICROCLOUDD_DEBUG_PATH,SKIP_VM_LAUNCH,SNAPSHOT_RESTORE,TEST_STORAGE_SOURCE,TESTBED_READY,BASE_OS,LXD_SNAP_CHANNEL,MICROCEPH_SNAP_CHANNEL,MICROOVN_SNAP_CHANNEL,MICROCLOUD_SNAP_CHANNEL ./main.sh setup
+
           echo "TESTBED_READY=1" >> "${GITHUB_ENV}"
+          echo "BASE_OS=${BASE_OS}" >> "${GITHUB_ENV}"
+          echo "LXD_SNAP_CHANNEL=${LXD_SNAP_CHANNEL}" >> "${GITHUB_ENV}"
+          echo "MICROCEPH_SNAP_CHANNEL=${MICROCEPH_SNAP_CHANNEL}" >> "${GITHUB_ENV}"
+          echo "MICROOVN_SNAP_CHANNEL=${MICROOVN_SNAP_CHANNEL}" >> "${GITHUB_ENV}"
+          echo "MICROCLOUD_SNAP_CHANNEL=${MICROCLOUD_SNAP_CHANNEL}" >> "${GITHUB_ENV}"
 
-      - name: "Run system tests (${{ matrix.go }}, ${{ matrix.suite }})"
+      - name: "Run system tests (${{ matrix.suite }})"
         run: |
           set -eux
           chmod +x ~
           cd test
-          sudo --preserve-env=DEBUG,GITHUB_ACTIONS,MICROCLOUD_DEBUG_PATH,MICROCLOUDD_DEBUG_PATH,SKIP_VM_LAUNCH,SNAPSHOT_RESTORE,TEST_STORAGE_SOURCE,TESTBED_READY ./main.sh ${{ matrix.suite }}
+          sudo --preserve-env=DEBUG,GITHUB_ACTIONS,MICROCLOUD_DEBUG_PATH,MICROCLOUDD_DEBUG_PATH,SKIP_VM_LAUNCH,SNAPSHOT_RESTORE,TEST_STORAGE_SOURCE,TESTBED_READY,BASE_OS,LXD_SNAP_CHANNEL,MICROCEPH_SNAP_CHANNEL,MICROOVN_SNAP_CHANNEL,MICROCLOUD_SNAP_CHANNEL ./main.sh ${{ matrix.suite }}
 
   documentation-checks:
     uses: canonical/documentation-workflows/.github/workflows/documentation-checks.yaml@main

doc/how-to/index.md

@@ -7,6 +7,7 @@ These how-to guides cover key operations and processes in MicroCloud.
 :maxdepth: 1
 
 Install MicroCloud </how-to/install>
+Update and upgrade </how-to/update_upgrade>
 Manage the snaps </how-to/snaps>
 Initialise MicroCloud </how-to/initialise>
 Configure Ceph networking </how-to/ceph_networking>

doc/how-to/install.md

@@ -6,9 +6,9 @@ To install MicroCloud, install all required {ref}`snaps` on all machines that yo
 To do so, enter the following commands on all machines:
 
     sudo snap install lxd --channel=5.21/stable --cohort="+"
-    sudo snap install microceph --channel=quincy/stable --cohort="+"
-    sudo snap install microovn --channel=22.03/stable --cohort="+"
-    sudo snap install microcloud --channel=latest/stable --cohort="+"
+    sudo snap install microceph --channel=squid/stable --cohort="+"
+    sudo snap install microovn --channel=24.03/stable --cohort="+"
+    sudo snap install microcloud --channel=2/stable --cohort="+"
 
 ```{note}
 Make sure to install the same version of the snaps on all machines.
@@ -17,3 +17,6 @@ See {ref}`howto-snap` for more information.
 If you don't want to use MicroCloud's full functionality, you can install only some of the snaps.
 However, this is not recommended.
 ```
+
+After installing the snaps make sure to hold any automatic updates to keep the used snap versions across MicroCloud in sync.
+See {ref}`howto-snap-hold-updates` for more information.

doc/how-to/recover.md

@@ -1,3 +1,4 @@
+(howto-recover)=
 # How to recover a MicroCloud cluster
 
 ```{note}

doc/how-to/snaps.md

@@ -13,7 +13,7 @@ The installed snap versions must be compatible with one another, and for each of
 Snaps come with different channels that define which release of a snap is installed and tracked for updates.
 See [Channels and tracks](https://snapcraft.io/docs/channels) in the snap documentation for detailed information.
 
-MicroCloud currently provides only the `latest` track.
+MicroCloud currently provides the legacy `1` and the latest `2` track.
 
 ```{tip}
 In general, you should use the default channels for all snaps required to run MicroCloud.
@@ -27,29 +27,22 @@ When installing a snap, specify the channel as follows:
 
 For example:
 
-    sudo snap install microcloud --channel=latest/stable
+    sudo snap install microcloud --channel=2/stable
 
 To see all available channels of a snap, run the following command:
 
     snap info <snap_name>
 
+(howto-snap-control-updates)=
 ## Control updates
 
 By default, snaps are updated automatically.
 In the case of MicroCloud, this can be problematic because the related snaps must always use compatible versions, and because all machines of a cluster must use the same version of each snap.
 
-Therefore, you should schedule your updates and make sure that all cluster members are in sync regarding the snap versions that they use.
+Therefore, you should manually apply your updates and make sure that all cluster members are in sync regarding the snap versions that they use.
 
-### Schedule updates
-
-There are two methods for scheduling when your snaps should be updated:
-
-- You can hold snap updates for a specific time, either for specific snaps or for all snaps on your system.
-  After the duration of the hold, or when you remove the hold, your snaps are automatically refreshed.
-- You can specify a system-wide refresh window, so that snaps are automatically refreshed only within this time frame.
-  Such a refresh window applies to all snaps.
-
-#### Hold updates
+(howto-snap-hold-updates)=
+### Hold updates
 
 You can hold snap updates for a specific time or forever, for all snaps or for specific snaps.
 
@@ -60,48 +53,17 @@ Enter the following command to indefinitely hold all updates to the snaps needed
 
     sudo snap refresh --hold lxd microceph microovn microcloud
 
-When you choose to update your installation, use the following commands to remove the hold, update the snaps, and hold the updates again:
-
-    sudo snap refresh --unhold lxd microceph microovn microcloud
-    sudo snap refresh lxd --cohort="+"
-    sudo snap refresh microceph --cohort="+"
-    sudo snap refresh microovn --cohort="+"
-    sudo snap refresh microcloud --cohort="+"
-    sudo snap refresh --hold lxd microceph microovn microcloud
-
 See [Hold refreshes](https://snapcraft.io/docs/managing-updates#heading--hold) in the snap documentation for detailed information about holding snap updates.
 
-#### Specify a refresh window
-
-Depending on your setup, you might want your snaps to update regularly, but only at specific times that don't disturb normal operation.
-
-You can achieve this by specifying a refresh timer.
-This option defines a refresh window for all snaps that are installed on the system.
-
-For example, to configure your system to update snaps only between 8:00 am and 9:00 am on Mondays, set the following option:
-
-    sudo snap set system refresh.timer=mon,8:00-9:00
-
-You can use a similar mechanism (setting `refresh.hold`) to hold snap updates as well.
-However, in this case the snaps will be refreshed after 90 days, irrespective of the value of `refresh.hold`.
-
-See [Control updates with system options](https://snapcraft.io/docs/managing-updates#heading--refresh-hold) in the snap documentation for detailed information.
-
 (howto-snap-cluster)=
 ### Keep cluster members in sync
 
-The cluster members that are part of the MicroCloud deployment must always run the same version of the snaps.
-This means that when the snaps on one of the cluster members are refreshed, they must also be refreshed on all other cluster members before MicroCloud is operational again.
-
 Snap updates are delivered as [progressive releases](https://snapcraft.io/docs/progressive-releases), which means that updated snap versions are made available to different machines at different times.
 This method can cause a problem for cluster updates if some cluster members are refreshed to a version that is not available to other cluster members yet.
 
 To avoid this problem, use the `--cohort="+"` flag when refreshing your snaps:
 
-    sudo snap refresh lxd --cohort="+"
-    sudo snap refresh microceph --cohort="+"
-    sudo snap refresh microovn --cohort="+"
-    sudo snap refresh microcloud --cohort="+"
+    sudo snap refresh <snap> --cohort="+"
 
 This flag ensures that all machines in a cluster see the same snap revision and are therefore not affected by a progressive rollout.
 

doc/how-to/support.md

@@ -3,13 +3,14 @@
 
 We recommend using the following channels for the snaps required to run MicroCloud:
 
-* For MicroCloud: `latest/[stable|candidate|edge]`
-* For LXD: `5.21/[stable|candidate|edge]`
-* For MicroCeph: `quincy/[stable|candidate|edge]`
-* For MicroOVN: `22.03/[stable|candidate|edge]`
+* For MicroCloud: `2/(stable|candidate|edge)`
+* For LXD: `5.21/(stable|candidate|edge)`
+* For MicroCeph: `squid/(stable|candidate|edge)`
+* For MicroOVN: `24.03/(stable|candidate|edge)`
 
 ```{note}
-Currently, there is no LTS version of MicroCloud.
+The LTS version of MicroCloud is available in the `2` track.
+It's recommended to use the `<track>/stable` channels for production deployments.
 ```
 
 ## Support and community

doc/how-to/update_upgrade.md

@@ -0,0 +1,141 @@
+(howto-update-upgrade)=
+# How to update and upgrade
+
+MicroCloud is made of several snaps that are closely coupled with each other.
+The cluster members that are part of the MicroCloud deployment must always run the same version of the snaps.
+This means that when the snaps on one of the cluster members are refreshed, they must also be refreshed on all other cluster members before MicroCloud is operational again.
+See {ref}`howto-update-upgrade-deps` for the recommended order.
+
+Before performing an update or upgrade make sure to backup your data to prevent any data loss in case of failure.
+See the following backup guides for each of the snaps:
+
+* {doc}`How to backup MicroCeph <microceph:explanation/taking-snapshots>`
+* {ref}`How to backup LXD <lxd:backups>`
+
+In case of error see {ref}`howto-recover` for troubleshooting details.
+
+(howto-update-upgrade-update)=
+## Update MicroCloud
+
+Updating MicroCloud allows getting the latest set of features and fixes in the tracked channels for the various snaps.
+Performing an update requires going through the list of snaps one after another and update each of the individual cluster members.
+
+```{note}
+Depending on which snap gets updated, some services of this snap (e.g. API endpoints) might not be available whilst performing the update.
+Check the respective services documentation on updates for more information.
+```
+
+During an update the snaps channel won't be modified so the snaps get updated to the last available version inside of the current channel.
+This does not introduce breaking changes and can be used on a regular basis to update the MicroCloud.
+
+As MicroCloud consumes the services offered by the dependant snaps (MicroCeph, MicroOVN and LXD), the update procedure starts by updating
+the list of dependencies first.
+
+(howto-update-upgrade-deps)=
+### Update dependency snaps
+
+Updating the dependencies can be done by running `snap refresh` against the respective snap.
+For MicroCloud automatic snap refreshes are put on hold. See {ref}`howto-snap-control-updates` for more information.
+
+To start the update procedure, enter the following command on the first machine:
+
+    sudo snap refresh microceph --cohort="+"
+
+If the command has succeeded, run the same command on the next machine and so on.
+
+```{note}
+Make sure to validate the health of the recently updated dependency before continuing with the next one.
+```
+
+After successfully updating MicroCeph continue with MicroOVN.
+Again enter the following command on the first machine:
+
+    sudo snap refresh microovn --cohort="+"
+
+As before run the command on all the other machines one after another if you don't observe any errors.
+Next we can continue to update LXD.
+
+The refresh will block until each of the LXD cluster members is updated so make sure to perform the following
+command on all machines in parallel:
+
+    sudo snap refresh lxd --cohort="+"
+
+### Update MicroCloud snap
+
+Last but not least we can update MicroCloud.
+As before enter the following command on the first machine:
+
+    sudo snap refresh microcloud --cohort="+"
+
+Continue running the command on the rest of the machines to finish the update.
+You can confirm a healthy state of the MicroCloud after the update by running the following command:
+
+    sudo microcloud status
+
+```{note}
+The status command was introduced in MicroCloud version 2.
+See {ref}`howto-update-upgrade-upgrade` on how to upgrade to another track.
+```
+
+(howto-update-upgrade-upgrade)=
+## Upgrade MicroCloud
+
+Upgrading MicroCloud allows switching to another track with major improvements and enhanced functionality.
+Performing an upgrade requires going through the list of snaps one after another and upgrade each of the individual cluster members.
+
+```{note}
+Depending on which snap gets upgraded, some services of this snap (e.g. API endpoints) might not be available whilst performing the upgrade.
+Check the respective services documentation on upgrades for more information.
+```
+
+During an upgrade the snaps channel will be switched to another track.
+This might introduce breaking changes for MicroCloud and its dependencies and should be done with care.
+See {ref}`howto-update-upgrade-update` for regular non-breaking updates.
+
+As MicroCloud consumes the services offered by the dependant snaps (MicroCeph, MicroOVN and LXD), the update procedure starts by updating
+the list of dependencies first.
+
+### Upgrade dependency snaps
+
+Upgrading the dependencies can be done by running `snap refresh --channel <new track/stable>` against the respective snap.
+
+Make sure to consult the dedicated upgrade guides of each dependency before you perform the actual upgrade:
+
+* {doc}`How to upgrade MicroCeph <microceph:how-to/reef-upgrade>`
+* {doc}`How to upgrade MicroOVN <microovn:how-to/major-upgrades>`
+* {doc}`How to upgrade LXD <lxd:howto/cluster_manage>`
+
+To start the upgrade procedure, enter the following command on the first machine:
+
+    sudo snap refresh microceph --channel "squid/stable" --cohort="+"
+
+If the command has succeeded, run the same command on the next machine and so on.
+
+```{note}
+Make sure to validate the health of the recently upgraded dependency before continuing with the next one.
+```
+
+After successfully upgrading MicroCeph continue with MicroOVN.
+Again enter the following command on the first machine:
+
+    sudo snap refresh microovn --channel "24.03/stable" --cohort="+"
+
+As before run the command on all the other machines one after another if you don't observe any errors.
+Next we can continue to upgrade LXD.
+
+The installer will block until each of the LXD cluster members is upgraded so make sure to perform the following
+command on all machines in parallel:
+
+    sudo snap refresh lxd --channel "5.21/stable" --cohort="+"
+
+### Upgrade MicroCloud snap
+
+Last but not least we can upgrade MicroCloud.
+As before enter the following command on the first machine:
+
+    sudo snap refresh microcloud --channel "2/stable" --cohort="+"
+
+Continue running the command on the rest of the machines to finish the upgrade.
+You can confirm a healthy state of the MicroCloud after the upgrade by running the following command:
+
+    sudo microcloud status

doc/tutorial/get_started.md

@@ -231,9 +231,9 @@ Complete the following steps on each VM (`micro1`, `micro2`, `micro3`, and `micr
 
 1. Install the required snaps:
 
-       snap install microceph --channel=quincy/stable --cohort="+"
-       snap install microovn --channel=22.03/stable --cohort="+"
-       snap install microcloud --channel=latest/stable --cohort="+"
+       snap install microceph --channel=squid/stable --cohort="+"
+       snap install microovn --channel=24.03/stable --cohort="+"
+       snap install microcloud --channel=2/stable --cohort="+"
 
    ```{note}
    The `--cohort="+"` flag in the command ensures that the same version of the snap is installed on all machines.

test/includes/check.sh

@@ -36,10 +36,11 @@ check_empty() {
 }
 
 check_snap_channels() {
-    non_edge=""
-    if [ "${LXD_SNAP_CHANNEL}" != "latest/edge" ]; then
-        non_edge="${non_edge} lxd"
+    if [ "${LXD_SNAP_CHANNEL}" != "5.21/edge" ]; then
+        echo "::warning::lxd channel not set to 5.21/edge, continuing anyway"
     fi
+
+	non_edge=""
     if [ "${MICROCEPH_SNAP_CHANNEL}" != "latest/edge" ]; then
         non_edge="${non_edge} microceph"
     fi