Upgrading and data directory version checking

MySQL and MariaDB use versions that consist of three numbers X.Y.Z (e.g. 5.6.23).
For version changes in Z part, the server's binary data format stays compatible and thus no
special upgrade procedure is needed. For upgrades from X.Y to X.Y+1, consider doing manual
steps as described at
https://mariadb.com/kb/en/library/upgrading-from-mariadb-101-to-mariadb-102/

Skipping versions like from X.Y to X.Y+2 or downgrading to lower version is not supported;
the only exception is ugrading from MariaDB 5.5 to MariaDB 10.0.

**Important**: Upgrading to a new version is always risky and users are expected to make a full
back-up of all data before.

A safer solution to upgrade is to dump all data using `mysqldump` or `mysqldbexport` and then
load the data using `mysql` or `mysqldbimport` into an empty (freshly initialized) database.

Another way of proceeding with the upgrade is starting the new version of the `mysqld` daemon
and run `mysql_upgrade` right after the start. This so called in-place upgrade is generally
faster for large data directory, but only possible if upgrading from the very previous version,
so skipping versions is not supported.

This container detects whether the data needs to be upgraded using `mysql_upgrade` and
we can control it by setting `MYSQL_UPGRADE` variable, which can have one or more of the following values:

 * `warn` -- If the data version can be determined and the data come from a different version
   of the daemon, a warning is printed but the container starts. This is the default value.
   Since historically the version file `mysql_upgrade_info` was not created, when using this option,
   the version file is created if not exist, but no `mysql_upgrade` will be called.
   However, this automatic creation will be removed after few months, since the version should be
   created on most deployments at that point.
 * `auto` -- `mysql_upgrade` is run at the beginning of the container start, if the data version
   can be determined and the data come with the very previous version. A warning is printed if the data
   come from even older or newer version. This value effectively enables automatic upgrades,
   but it is always risky and users should still back-up all the data before starting the newer container.
   Set this option only if you have very good back-ups at any moment and you are fine to fail-over
   from the back-up.
 * `force` -- `mysql_upgrade` is run right after the daemon has started, no matter what version the data
   come from. This is also the way to create the missing version file `mysql_upgrade_info` if not present
   in the root of the data directory; this file holds information about the version of the data.
 * `optimize` -- runs `mysqlcheck --optimize` before starting the mysqld daemon, no matter what version
   of the data is detected. It optimizes all the tables.
 * `analyze` -- runs `mysqlcheck --analyze` before starting the mysqld daemon, no matter what version
   of the data is detected. It analyzes all the tables.
 * `disable` -- nothing is done regarding data directory version.

Multiple values are separated by comma and run in-order, e.g. `MYSQL_UPGRADE="optimize,analyze"`.

10.0/root/usr/share/container-scripts/mysql/README.md

@@ -260,6 +260,56 @@ Such a directory `sslapp` can then be mounted into the container with -v,
 or a new container image can be built using s2i.
 
 
+Upgrading and data directory version checking
+---------------------------------------------
+
+MySQL and MariaDB use versions that consist of three numbers X.Y.Z (e.g. 5.6.23).
+For version changes in Z part, the server's binary data format stays compatible and thus no
+special upgrade procedure is needed. For upgrades from X.Y to X.Y+1, consider doing manual
+steps as described at
+https://mariadb.com/kb/en/library/upgrading-from-mariadb-55-to-mariadb-100/
+
+Skipping versions like from X.Y to X.Y+2 or downgrading to lower version is not supported;
+the only exception is ugrading from MariaDB 5.5 to MariaDB 10.0.
+
+**Important**: Upgrading to a new version is always risky and users are expected to make a full
+back-up of all data before.
+
+A safer solution to upgrade is to dump all data using `mysqldump` or `mysqldbexport` and then
+load the data using `mysql` or `mysqldbimport` into an empty (freshly initialized) database.
+
+Another way of proceeding with the upgrade is starting the new version of the `mysqld` daemon
+and run `mysql_upgrade` right after the start. This so called in-place upgrade is generally
+faster for large data directory, but only possible if upgrading from the very previous version,
+so skipping versions is not supported.
+
+This container detects whether the data needs to be upgraded using `mysql_upgrade` and
+we can control it by setting `MYSQL_UPGRADE` variable, which can have one or more of the following values:
+
+ * `warn` -- If the data version can be determined and the data come from a different version
+   of the daemon, a warning is printed but the container starts. This is the default value.
+   Since historically the version file `mysql_upgrade_info` was not created, when using this option,
+   the version file is created if not exist, but no `mysql_upgrade` will be called.
+   However, this automatic creation will be removed after few months, since the version should be
+   created on most deployments at that point.
+ * `auto` -- `mysql_upgrade` is run at the beginning of the container start, if the data version
+   can be determined and the data come with the very previous version. A warning is printed if the data
+   come from even older or newer version. This value effectively enables automatic upgrades,
+   but it is always risky and users should still back-up all the data before starting the newer container.
+   Set this option only if you have very good back-ups at any moment and you are fine to fail-over
+   from the back-up.
+ * `force` -- `mysql_upgrade` is run right after the daemon has started, no matter what version the data
+   come from. This is also the way to create the missing version file `mysql_upgrade_info` if not present
+   in the root of the data directory; this file holds information about the version of the data.
+ * `optimize` -- runs `mysqlcheck --optimize` before starting the mysqld daemon, no matter what version
+   of the data is detected. It optimizes all the tables.
+ * `analyze` -- runs `mysqlcheck --analyze` before starting the mysqld daemon, no matter what version
+   of the data is detected. It analyzes all the tables.
+ * `disable` -- nothing is done regarding data directory version.
+
+Multiple values are separated by comma and run in-order, e.g. `MYSQL_UPGRADE="optimize,analyze"`.
+
+
 Changing the replication binlog_format
 --------------------------------------
 Some applications may wish to use `row` binlog_formats (for example, those built

10.1/root/usr/share/container-scripts/mysql/README.md

@@ -260,6 +260,56 @@ Such a directory `sslapp` can then be mounted into the container with -v,
 or a new container image can be built using s2i.
 
 
+Upgrading and data directory version checking
+---------------------------------------------
+
+MySQL and MariaDB use versions that consist of three numbers X.Y.Z (e.g. 5.6.23).
+For version changes in Z part, the server's binary data format stays compatible and thus no
+special upgrade procedure is needed. For upgrades from X.Y to X.Y+1, consider doing manual
+steps as described at
+https://mariadb.com/kb/en/library/upgrading-from-mariadb-100-to-mariadb-101/
+
+Skipping versions like from X.Y to X.Y+2 or downgrading to lower version is not supported;
+the only exception is ugrading from MariaDB 5.5 to MariaDB 10.0.
+
+**Important**: Upgrading to a new version is always risky and users are expected to make a full
+back-up of all data before.
+
+A safer solution to upgrade is to dump all data using `mysqldump` or `mysqldbexport` and then
+load the data using `mysql` or `mysqldbimport` into an empty (freshly initialized) database.
+
+Another way of proceeding with the upgrade is starting the new version of the `mysqld` daemon
+and run `mysql_upgrade` right after the start. This so called in-place upgrade is generally
+faster for large data directory, but only possible if upgrading from the very previous version,
+so skipping versions is not supported.
+
+This container detects whether the data needs to be upgraded using `mysql_upgrade` and
+we can control it by setting `MYSQL_UPGRADE` variable, which can have one or more of the following values:
+
+ * `warn` -- If the data version can be determined and the data come from a different version
+   of the daemon, a warning is printed but the container starts. This is the default value.
+   Since historically the version file `mysql_upgrade_info` was not created, when using this option,
+   the version file is created if not exist, but no `mysql_upgrade` will be called.
+   However, this automatic creation will be removed after few months, since the version should be
+   created on most deployments at that point.
+ * `auto` -- `mysql_upgrade` is run at the beginning of the container start, if the data version
+   can be determined and the data come with the very previous version. A warning is printed if the data
+   come from even older or newer version. This value effectively enables automatic upgrades,
+   but it is always risky and users should still back-up all the data before starting the newer container.
+   Set this option only if you have very good back-ups at any moment and you are fine to fail-over
+   from the back-up.
+ * `force` -- `mysql_upgrade` is run right after the daemon has started, no matter what version the data
+   come from. This is also the way to create the missing version file `mysql_upgrade_info` if not present
+   in the root of the data directory; this file holds information about the version of the data.
+ * `optimize` -- runs `mysqlcheck --optimize` before starting the mysqld daemon, no matter what version
+   of the data is detected. It optimizes all the tables.
+ * `analyze` -- runs `mysqlcheck --analyze` before starting the mysqld daemon, no matter what version
+   of the data is detected. It analyzes all the tables.
+ * `disable` -- nothing is done regarding data directory version.
+
+Multiple values are separated by comma and run in-order, e.g. `MYSQL_UPGRADE="optimize,analyze"`.
+
+
 Changing the replication binlog_format
 --------------------------------------
 Some applications may wish to use `row` binlog_formats (for example, those built

10.2/root/usr/share/container-scripts/mysql/README.md

@@ -260,6 +260,56 @@ Such a directory `sslapp` can then be mounted into the container with -v,
 or a new container image can be built using s2i.
 
 
+Upgrading and data directory version checking
+---------------------------------------------
+
+MySQL and MariaDB use versions that consist of three numbers X.Y.Z (e.g. 5.6.23).
+For version changes in Z part, the server's binary data format stays compatible and thus no
+special upgrade procedure is needed. For upgrades from X.Y to X.Y+1, consider doing manual
+steps as described at
+https://mariadb.com/kb/en/library/upgrading-from-mariadb-101-to-mariadb-102/
+
+Skipping versions like from X.Y to X.Y+2 or downgrading to lower version is not supported;
+the only exception is ugrading from MariaDB 5.5 to MariaDB 10.0.
+
+**Important**: Upgrading to a new version is always risky and users are expected to make a full
+back-up of all data before.
+
+A safer solution to upgrade is to dump all data using `mysqldump` or `mysqldbexport` and then
+load the data using `mysql` or `mysqldbimport` into an empty (freshly initialized) database.
+
+Another way of proceeding with the upgrade is starting the new version of the `mysqld` daemon
+and run `mysql_upgrade` right after the start. This so called in-place upgrade is generally
+faster for large data directory, but only possible if upgrading from the very previous version,
+so skipping versions is not supported.
+
+This container detects whether the data needs to be upgraded using `mysql_upgrade` and
+we can control it by setting `MYSQL_UPGRADE` variable, which can have one or more of the following values:
+
+ * `warn` -- If the data version can be determined and the data come from a different version
+   of the daemon, a warning is printed but the container starts. This is the default value.
+   Since historically the version file `mysql_upgrade_info` was not created, when using this option,
+   the version file is created if not exist, but no `mysql_upgrade` will be called.
+   However, this automatic creation will be removed after few months, since the version should be
+   created on most deployments at that point.
+ * `auto` -- `mysql_upgrade` is run at the beginning of the container start, if the data version
+   can be determined and the data come with the very previous version. A warning is printed if the data
+   come from even older or newer version. This value effectively enables automatic upgrades,
+   but it is always risky and users should still back-up all the data before starting the newer container.
+   Set this option only if you have very good back-ups at any moment and you are fine to fail-over
+   from the back-up.
+ * `force` -- `mysql_upgrade` is run right after the daemon has started, no matter what version the data
+   come from. This is also the way to create the missing version file `mysql_upgrade_info` if not present
+   in the root of the data directory; this file holds information about the version of the data.
+ * `optimize` -- runs `mysqlcheck --optimize` before starting the mysqld daemon, no matter what version
+   of the data is detected. It optimizes all the tables.
+ * `analyze` -- runs `mysqlcheck --analyze` before starting the mysqld daemon, no matter what version
+   of the data is detected. It analyzes all the tables.
+ * `disable` -- nothing is done regarding data directory version.
+
+Multiple values are separated by comma and run in-order, e.g. `MYSQL_UPGRADE="optimize,analyze"`.
+
+
 Changing the replication binlog_format
 --------------------------------------
 Some applications may wish to use `row` binlog_formats (for example, those built

root-common/usr/share/container-scripts/mysql/common.sh

@@ -39,6 +39,7 @@ function export_setting_variables() {
     export MYSQL_INNODB_LOG_FILE_SIZE=${MYSQL_INNODB_LOG_FILE_SIZE:-$((MEMORY_LIMIT_IN_BYTES*15/1024/1024/100))M}
     export MYSQL_INNODB_LOG_BUFFER_SIZE=${MYSQL_INNODB_LOG_BUFFER_SIZE:-$((MEMORY_LIMIT_IN_BYTES*15/1024/1024/100))M}
   fi
+  export MYSQL_UPGRADE=${MYSQL_UPGRADE:-warn}
 }
 
 # this stores whether the database was initialized from empty datadir
@@ -99,6 +100,11 @@ function initialize_database() {
   mysql_install_db --rpm --datadir=$MYSQL_DATADIR --basedir=''
   start_local_mysql "$@"
 
+  # Running mysql_upgrade creates the mysql_upgrade_info file in the data dir,
+  # which is necessary to detect which version of the mysqld daemon created the data.
+  # Checking empty file should not take longer than a second and one extra check should not harm.
+  mysql_upgrade --socket=/tmp/mysql.sock
+
   if [ -v MYSQL_RUNNING_AS_SLAVE ]; then
     log_info 'Initialization finished'
     return 0
@@ -223,3 +229,55 @@ function process_extending_config_files() {
     fi
   done <<<"$(get_matched_files "$custom_dir" "$default_dir" '*.cnf' | sort -u)"
 }
+
+# Converts string version to the integer format (5.5.33 is converted to 505,
+# 10.1.23-MariaDB is converted into 1001, etc.
+function version2number() {
+  local version_major=$(echo "$1" | grep -o -e '^[0-9]*\.[0-9]*')
+  printf %d%02d ${version_major%%.*} ${version_major##*.}
+}
+
+# Converts the version in format of an integer into major.minor
+function number2version() {
+  local numver=${1}
+  echo $((numver / 100)).$((numver % 100))
+}
+
+# Prints version of the mysqld that is currently available (string)
+function mysqld_version() {
+  ${MYSQL_PREFIX}/libexec/mysqld -V | awk '{print $3}'
+}
+
+# Returns version from the daemon in integer format
+function mysqld_compat_version() {
+  version2number $(mysqld_version)
+}
+
+# Returns version from the datadir
+function get_datadir_version() {
+  local datadir="$1"
+  local upgrade_info_file=$(get_mysql_upgrade_info_file "$datadir")
+  [ -r "$upgrade_info_file" ] || return
+  local version_text=$(cat "$upgrade_info_file" | head -n 1)
+  version2number "${version_text}"
+}
+
+# Returns name of the file in the datadir that holds version information about the data
+function get_mysql_upgrade_info_file() {
+  local datadir="$1"
+  echo "$datadir/mysql_upgrade_info"
+}
+
+# Writes version string of the daemon into mysql_upgrade_info file (should be only used when the file is missing and only when
+function write_mysql_upgrade_info_file() {
+  local datadir="$1"
+  local version=$(mysqld_version)
+  local upgrade_info_file=$(get_mysql_upgrade_info_file "$datadir")
+  if [ -f "$datadir/mysql_upgrade_info" ] ; then
+    echo "File ${upgrade_info_file} exists, nothing is done."
+  else
+    log_info "Storing version '${version}' information into the data dir '${upgrade_info_file}'"
+    echo "${version}" > "${upgrade_info_file}"
+    mysqld_version >"$datadir/mysql_upgrade_info"
+  fi
+}

root-common/usr/share/container-scripts/mysql/helpers.sh

@@ -2,6 +2,10 @@ function log_info {
   echo "---> `date +%T`     $@"
 }
 
+function log_warn {
+  echo "---> `date +%T`     Warning: $@"
+}
+
 function log_and_run {
   log_info "Running $@"
   "$@"

root-common/usr/share/container-scripts/mysql/init/40-check-upgrade.sh

@@ -0,0 +1,111 @@
+upstream_upgrade_info() {
+  echo -n "For upstream documentation about upgrading, see: "
+  case ${MYSQL_VERSION} in
+    10.0) echo "https://mariadb.com/kb/en/library/upgrading-from-mariadb-55-to-mariadb-100/" ;;
+    10.1) echo "https://mariadb.com/kb/en/library/upgrading-from-mariadb-100-to-mariadb-101/" ;;
+    10.2) echo "https://mariadb.com/kb/en/library/upgrading-from-mariadb-101-to-mariadb-102/" ;;
+    5.6) echo "https://dev.mysql.com/doc/refman/5.6/en/upgrading-from-previous-series.html" ;;
+    5.7) echo "https://dev.mysql.com/doc/refman/5.7/en/upgrading-from-previous-series.html" ;;
+    *) echo "Non expected version '${MYSQL_VERSION}'" ; return 1 ;;
+  esac
+}
+
+check_datadir_version() {
+  local datadir="$1"
+  local datadir_version=$(get_datadir_version "$datadir")
+  local mysqld_version=$(mysqld_compat_version)
+  local datadir_version_dot=$(number2version "${datadir_version}")
+  local mysqld_version_dot=$(number2version "${mysqld_version}")
+
+  for upgrade_action in ${MYSQL_UPGRADE//,/ } ; do
+    log_info "Running upgrade action: ${upgrade_action}"
+    case ${upgrade_action} in
+      auto|warn)
+        if [ -z "${datadir_version}" ] || [ "${datadir_version}" -eq 0 ] ; then
+          # Writing the info file, since historically it was not written
+          log_warn "Version of the data could not be determined."\
+                   "It is because the file mysql_upgrade_info is missing in the data directory, which"\
+                   "is most probably because it was not created when initialization of data directory."\
+                   "In order to allow seamless updates to the next higher version in the future,"\
+                   "the file mysql_upgrade_info will be created."\
+                   "If the data directory was created with a different version than ${mysqld_version_dot},"\
+                   "it is required to run this container with the MYSQL_UPGRADE environment variable"\
+                   "set to 'force', or run 'mysql_upgrade' utility manually; the mysql_upgrade tool"\
+                   "checks the tables and creates such a file as well. $(upstream_upgrade_info)"
+          write_mysql_upgrade_info_file "${MYSQL_DATADIR}"
+          continue
+          # This is currently a dead-code, but should be enabled after the mysql_upgrade_info
+          # file gets to the deployments (after few monts most of the deployments should already have the file)
+          log_warn "Version of the data could not be determined."\
+                   "Running such a container is risky."\
+                   "The current daemon version is ${mysqld_version_dot}."\
+                   "If you are not sure whether the data directory is compatible with the current"\
+                   "version ${mysqld_version_dot}, restore the data from a back-up."\
+                   "If restoring from a back-up is not possible, create a file 'mysql_upgrade_info'"\
+                   "that includes version information (${mysqld_version_dot} in this case) in the root"\
+                   "of the data directory."\
+                   "In order to create the 'mysql_upgrade_info' file, either run this container with"\
+                   "the MYSQL_UPGRADE environment variable set to 'force', or run 'mysql_upgrade' utility"\
+                   "manually; the mysql_upgrade tool checks the tables and creates such a file as well."\
+                   "That will enable correct upgrade check in the future. $(upstream_upgrade_info)"
+        fi
+
+        if [ "${datadir_version}" -eq "${mysqld_version}" ] ; then
+          log_info "MySQL server version check passed, both server and data directory"\
+                   "are version ${mysqld_version_dot}."
+          continue
+        fi
+
+        if [ $(( ${datadir_version} + 1 )) -eq "${mysqld_version}" -o "${datadir_version}" -eq 505 -a "${mysqld_version}" -eq 1000 ] ; then
+          log_warn "MySQL server is version ${mysqld_version_dot} and datadir is version"\
+                   "${datadir_version_dot}, which is a compatible combination."
+          if [ "${MYSQL_UPGRADE}" == 'auto' ] ; then
+            log_info "The data directory will be upgraded automatically from ${datadir_version_dot}"\
+                     "to version ${mysqld_version_dot}. $(upstream_upgrade_info)"
+            log_and_run mysql_upgrade --socket=/tmp/mysql.sock
+          else
+            log_warn "Automatic upgrade is not turned on, proceed with the upgrade."\
+                     "In order to upgrade the data directory, run this container with the MYSQL_UPGRADE"\
+                     "environment variable set to 'auto' or run running mysql_upgrade manually. $(upstream_upgrade_info)"
+          fi
+        else
+          log_warn "MySQL server is version ${mysqld_version_dot} and datadir is version"\
+                   "${datadir_version_dot}, which are incompatible. Remember, that upgrade is only supported"\
+                   "by upstream from previous version and it is not allowed to skip versions. $(upstream_upgrade_info)"
+          if [ "${datadir_version}" -gt "${mysqld_version}" ] ; then
+            log_warn "Downgrading to the lower version is not supported. Consider"\
+                     "dumping data and load them again into a fresh instance. $(upstream_upgrade_info)"
+          fi
+          log_warn "Consider restoring the database from a back-up. To ignore this"\
+                   "warning, set 'MYSQL_UPGRADE' variable to 'force', but this may result in data corruption. $(upstream_upgrade_info)"
+          return 1
+        fi
+        ;;
+
+      force)
+        log_and_run mysql_upgrade --socket=/tmp/mysql.sock --force
+        ;;
+
+      optimize)
+        log_and_run mysqlcheck --socket=/tmp/mysql.sock --optimize --all-databases --force
+        ;;
+
+      analyze)
+        log_and_run mysqlcheck --socket=/tmp/mysql.sock --analyze --all-databases --force
+        ;;
+
+      disable)
+        log_info "Nothing is done about the data directory version."
+        ;;
+      *)
+        log_warn "Unknown value of MYSQL_UPGRADE variable: '${MYSQL_UPGRADE}', ignoring."
+        ;;
+    esac
+  done
+}
+
+check_datadir_version "${MYSQL_DATADIR}"
+
+unset -f check_datadir_version upstream_upgrade_info
+
+