Document a known issue with MySQL when upgrading to GHES 3.9 (github#39751)

Co-authored-by: Laura Coursen <[email protected]>
Co-authored-by: Jiaqi Liu <[email protected]>
Co-authored-by: Rachael Rose Renk <[email protected]>

Author: 4 people

content/admin/enterprise-management/updating-the-virtual-machine-and-physical-resources/known-issues-with-upgrades-to-your-instance.md

@@ -108,4 +108,78 @@ Finally, if you're willing to help {% data variables.product.company_short %}  u
 
 The data you submit helps {% data variables.product.company_short %} continue to provide a performant product, but {% data variables.product.company_short %} does not guarantee any additional mitigation steps or changes to the product as a result of the data you provide.
 
+## MySQL does not start after upgrade to {% data variables.product.prodname_ghe_server %} 3.9
+
+During an upgrade to {% data variables.product.prodname_ghe_server %} 3.9, if MySQL did not gracefully shut down during the shutdown of the {% data variables.product.prodname_ghe_server %} 3.7 or 3.8 instance, MySQL will attempt to go through crash recovery when the {% data variables.product.prodname_ghe_server %} 3.9 instance starts up. Since {% data variables.product.prodname_ghe_server %} 3.7 and 3.8 uses MySQL 5.7 and {% data variables.product.prodname_ghe_server %} 3.9 has been upgraded to MySQL 8.0, MySQL will not be able to complete crash recovery.
+
+If you experience this problem, the following error will be in the mysql error log (`/var/log/mysql/mysql.err`):
+
+```shell copy
+[ERROR] [MY-012526] [InnoDB] Upgrade after a crash is not supported. This redo log was created with MySQL 5.7.40. Please follow the instructions at http://dev.mysql.com/doc/refman/8.0/en/upgrading.html
+```
+
+### Avoiding this issue
+
+To avoid this issue, update the nomad timeout for MySQL before starting an upgrade to {% data variables.product.prodname_ghe_server %} 3.9
+
+1. Put your instance into maintenance mode:
+
+  ```shell copy
+  ghe-maintenance -s
+  ```
+1. Update consul template for nomad:
+
+  ```shell copy
+  sudo sed -i.bak '/kill_signal/i \      kill_timeout = "10m"' /etc/consul-templates/etc/nomad-jobs/mysql/mysql.hcl.ctmpl
+  ```
+1. Render consul template for nomad:
+
+  ```shell copy
+  sudo consul-template -once -template /etc/consul-templates/etc/nomad-jobs/mysql/mysql.hcl.ctmpl:/etc/nomad-jobs/mysql/mysql.hcl
+  ```
+1. Verify current `kill_timeout` setting:
+
+  ```shell copy
+  nomad job inspect mysql | grep KillTimeout
+  ```
+  
+  Expected response:
+  
+  ```shell copy
+  "KillTimeout": 5000000000
+  ```
+1. Stop MySQL:
+
+  ```shell copy
+  nomad job stop mysql
+  ```
+1. Run new MySQL job:
+
+  ```shell copy
+  nomad job run /etc/nomad-jobs/mysql/mysql.hcl
+  ```
+1. Verify kill_timeout has been updated:
+
+  ```shell copy
+  nomad job inspect mysql | grep KillTimeout
+  ```
+  Expected response:
+  
+  ```shell copy
+  "KillTimeout": 600000000000,
+  ```
+1. Take instance out of maintenance mode:
+
+  ```shell copy
+  ghe-maintenance -u
+  ```
+
+Now that the nomad timeout for MySQL has been updated you can upgrade your {% data variables.product.prodname_ghe_server %} instance to 3.9.
+
+### Mitigating a failed restart of MySQL
+
+If you're affected by this problem, restore your {% data variables.product.prodname_ghe_server %} instance to the state it was in prior to the upgrade attempt, and then follow the steps from the previous section.
+
+For more information about restoring from a failed upgrade, see "[AUTOTITLE](/admin/enterprise-management/updating-the-virtual-machine-and-physical-resources/upgrading-github-enterprise-server#restoring-from-a-failed-upgrade)."
+
 {% endif %}

data/release-notes/enterprise-server/3-7/0.yml

@@ -322,6 +322,8 @@ sections:
     - 'The maximum number of self-hosted runners in a runner group is limited to 10,000. Previously, there was no limit. [Updated: 2023-05-24]'
 
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - On a freshly set up {% data variables.product.prodname_ghe_server %} instance without any users, an attacker could create the first admin user.
@@ -378,4 +380,4 @@ sections:
 
     # https://github.com/github/releases/issues/2042
     - |
-      "[Features](#3.7.0-features)" incorrectly indicated that users of the GitHub Advisory Database can see advisories for Elixir, Erlang's Hex package manager, and more. This feature is unavailable in GitHub Enterprise Server 3.7, and will be available in a future release. [Updated 2023-06-01]
+      "[Features](#3.7.0-features)" incorrectly indicated that users of the GitHub Advisory Database can see advisories for Elixir, Erlang's Hex package manager, and more. This feature is unavailable in GitHub Enterprise Server 3.7, and will be available in a future release. [Updated 2023-06-01]

data/release-notes/enterprise-server/3-7/1.yml

@@ -26,6 +26,8 @@ sections:
     - If a site administrator has not yet configured GitHub Actions for the instance, the UI for setting up code scanning will prompt the user to configure GitHub Actions.
     - To avoid failing domain verification due to the 63-character limit enforced by DNS providers for DNS records, the GitHub-generated `TXT` record to verify domain ownership is now limited to 63 characters.
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - On a freshly set up {% data variables.product.prodname_ghe_server %} instance without any users, an attacker could create the first admin user.

data/release-notes/enterprise-server/3-7/10.yml

@@ -24,6 +24,8 @@ sections:
     - |
       People with administrative SSH access who generate a support bundle using the `ghe-support-bundle` or `ghe-cluster-support-bundle` utilities can specify the period of time to gather data with `-p` or `--period` without using spaces or quotes. For example, in addition to `'-p 5 days'` or `-p '4 days 10 hours'`,  `-p 5days` or `-p 4days10hours` are valid.
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - |

data/release-notes/enterprise-server/3-7/11.yml

@@ -12,6 +12,8 @@ sections:
   changes:
     - People with administrative SSH access to an instance can configure the maximum memory usage in gigabytes for Redis using `ghe-config redis.max-memory-gb VALUE`. 
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - |

data/release-notes/enterprise-server/3-7/12.yml

@@ -13,6 +13,8 @@ sections:
   changes:
     - If a configuration runs fails due to Elasticsearch errors, `ghe-config-apply` displays a more actionable error message. 
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - |

data/release-notes/enterprise-server/3-7/13.yml

@@ -56,6 +56,8 @@ sections:
     - |
       The Management Console displays a warning about unexpected consequences that may result from modification of the instance's hostname after initial configuration. 
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       Custom firewall rules are removed during the upgrade process.
     - |

data/release-notes/enterprise-server/3-7/2.yml

@@ -26,6 +26,8 @@ sections:
     - A user's list of recently accessed repositories no longer includes deleted repositories.
     - '{% data reusables.release-notes.scim-custom-mappings-supported-change %}'
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - On a freshly set up {% data variables.product.prodname_ghe_server %} instance without any users, an attacker could create the first admin user.

data/release-notes/enterprise-server/3-7/3.yml

@@ -22,6 +22,8 @@ sections:
     - The performance of configuration runs started with `ghe-config-apply` has been improved.
     - When exporting account data, backing up a repository, or performing a migration, the link to a repository archive now expires after 1 hour. Previously the archive link expired after 5 minutes.
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - On a freshly set up {% data variables.product.prodname_ghe_server %} instance without any users, an attacker could create the first admin user.

data/release-notes/enterprise-server/3-7/4.yml

@@ -4,6 +4,8 @@ sections:
     - |
       {% data reusables.release-notes.2023-01-git-vulnerabilities %}
   known_issues:
+    - |
+      {% data reusables.release-notes.upgrade-mysql8-cannot-start-up %}
     - |
       {% data reusables.release-notes.enterprise-backup-utils-encryption-keys %}
     - On a freshly set up {% data variables.product.prodname_ghe_server %} instance without any users, an attacker could create the first admin user.
@@ -18,4 +20,4 @@ sections:
     - '{% data reusables.release-notes.babeld-max-threads-performance-issue %}'
     - '{% data reusables.release-notes.git-push-known-issue %}'
     - '{% data reusables.release-notes.replication-commands-in-maintenance-mode-known-issue %}'
-    - '{% data reusables.release-notes.slow-deleted-repos-migration-known-issue %}'
+    - '{% data reusables.release-notes.slow-deleted-repos-migration-known-issue %}'