Merge pull request #1931 from OctopusDeploy/veo/upgrade-changes

Document db upgrade improvement

src/pages/docs/administration/upgrading/guide/automate-upgrades.mdx

@@ -1,7 +1,7 @@
 ---
 layout: src/layouts/Default.astro
 pubDate: 2023-01-01
-modDate: 2023-01-01
+modDate: 2023-07-10
 title: How to automate Octopus Deploy upgrades
 description: A how-to guide on how to automate Octopus Deploy upgrades
 navOrder: 4
@@ -393,6 +393,34 @@ $serverExe = "$installPath\Octopus.Server.exe"
 & $serverExe database --instance="OctopusServer" --upgrade
 ```
 
+As of **2023.2.9755**, Octopus will terminate a database upgrade if running nodes are detected. Sometimes the nodes simply haven't exited cleanly.
+And it might take a few seconds for Octopus to recognize this. It's possible to use a simple retry loop to handle this automatically.
+
+```
+$installPath = "${env:ProgramFiles}\Octopus Deploy\Octopus"
+$serverExe = "$installPath\Octopus.Server.exe"
+
+$maxRetry = 3
+$retryInterval = 10
+$attempts = 0
+
+While ($true) {
+    $attempts++
+
+    if ($attempts -gt $maxRetry) {
+        Write-Error "Upgrade failed after $maxRetry retries"
+        exit 1
+    }
+
+    # OctopusServer is the default instance name.  If you have multiple instances, or are not using the default instance name, change the --instance parameter.
+    & $serverExe database --instance="OctopusServer" --upgrade
+    if ($LASTEXITCODE -ne 0) {
+        Write-Warning "Upgrade failed. Retrying in $retryInterval seconds..."
+        Start-Sleep -Seconds $retryInterval
+    }
+}
+```
+
 **8. Restart all nodes (HAServer).**
 ```powershell
 # A server could have multiple instances, as we shut them all down earlier, start them all back up

src/pages/docs/administration/upgrading/index.mdx

@@ -1,7 +1,7 @@
 ---
 layout: src/layouts/Default.astro
 pubDate: 2023-01-01
-modDate: 2023-01-01
+modDate: 2023-07-10
 title: Upgrading Octopus
 description: Everything you need to know about upgrading Octopus to a newer version.
 navOrder: 40
@@ -43,6 +43,10 @@ If you are using the [Service Watchdog](/docs/administration/managing-infrastruc
 
 You are required to install the same MSI on all servers or nodes in your highly available Octopus Deploy instance.  The MSI installs the updated binaries, which include the latest database upgrade scripts.  Unlike the binaries, the database upgrade only needs to happen once.
 
+:::div{.warning}
+As of **2023.2.9755**, a database upgrade will abort if Octopus detects there are nodes still running. Ensure all nodes are properly shutdown and try again.
+:::
+
 :::div{.warning}
 A small outage window will occur when upgrading a highly available Octopus Deploy instance.  The outage window will happen between when you shut down all the nodes and upgrade the first node.  The window duration depends on the number of database changes, the size of the database, and compute resources.  It is highly recommended to [automate your upgrade process](/docs/administration/upgrading/guide/automate-upgrades) to reduce that outage window.
 :::

src/shared-content/upgrade/upgrade-high-availability.include.md

@@ -18,6 +18,10 @@ The process should look something like this:
 1. Test upgraded instance.
 1. Disable maintenance mode.
 
+:::div{.warning}
+As of **2023.2.9755**, a database upgrade will abort if Octopus detects there are nodes still running. Ensure all nodes are properly shutdown and try again.
+:::
+
 :::div{.warning}
 A small outage window will occur when upgrading a highly available Octopus Deploy instance.  The outage window will happen between when you shut down all the nodes and upgrade the first node.  The window duration depends on the number of database changes, the size of the database, and compute resources.  It is highly recommended to [automate your upgrade process](/docs/administration/upgrading/guide/automate-upgrades) to reduce that outage window.
 :::