MySQL Pod fails to become ready on KinD cluster

[kimwnasptd]

We've seen in the past that the Pod with MySQL never becomes ready in a KinD cluster #2273 (comment)

It keeps on biting us, so let's try to fix it once and for all. We also saw it when preparing for the 1.8 release in #2506

Doing a kubectl describe showed the following interesting event:

Warning  Unhealthy         15s (x5 over 75s)  kubelet            Startup probe failed: mysqladmin: [Warning] Using a password on the command line interface can be insecure.
mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)'
Check that mysqld is running and that the socket: '/var/run/mysqld/mysqld.sock' exists!

And I see the following logs

Logs

2022-08-31 15:25:46+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.29-1.el8 started.
2022-08-31 15:25:46+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2022-08-31 15:25:46+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.29-1.el8 started.
2022-08-31 15:25:46+00:00 [Note] [Entrypoint]: Initializing database files
2022-08-31T15:25:46.194799Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.29) initializing of server in progress as process 48
2022-08-31T15:25:46.194840Z 0 [ERROR] [MY-010338] [Server] Can't find error-message file '/usr/share/mysql-8.0/errmsg.sys'. Check error-message file location and 'lc-messages-dir' configuration directive.
2022-08-31T15:25:46.199453Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-31T15:25:46.537790Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-31T15:25:47.566600Z 6 [Warning] [MY-010453] [Server] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
2022-08-31 15:25:49+00:00 [Note] [Entrypoint]: Database files initialized
2022-08-31 15:25:49+00:00 [Note] [Entrypoint]: Starting temporary server
2022-08-31T15:25:49.661301Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.29) starting as process 95
2022-08-31T15:25:49.661360Z 0 [ERROR] [MY-010338] [Server] Can't find error-message file '/usr/share/mysql-8.0/errmsg.sys'. Check error-message file location and 'lc-messages-dir' configuration directive.
2022-08-31T15:25:49.672270Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-31T15:25:50.218488Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-31T15:25:50.420162Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-08-31T15:25:50.420198Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-08-31T15:25:50.420905Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/lib/mysql' in the path is accessible to all OS users. Consider choosing a different directory.
2022-08-31T15:25:50.435929Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: /var/run/mysqld/mysqlx.sock
2022-08-31T15:25:50.435992Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.29'  socket: '/var/lib/mysql/mysql.sock'  port: 0  MySQL Community Server - GPL.
2022-08-31 15:25:50+00:00 [Note] [Entrypoint]: Temporary server started.
Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/leapseconds' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/tzdata.zi' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/zone.tab' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/zone1970.tab' as time zone. Skipping it.
2022-08-31 15:25:52+00:00 [Note] [Entrypoint]: Creating database katib

2022-08-31 15:25:52+00:00 [Note] [Entrypoint]: Stopping temporary server
2022-08-31T15:25:52.353221Z 11 [System] [MY-013172] [Server] Received SHUTDOWN from user root. Shutting down mysqld (Version: 8.0.29).
2022-08-31T15:25:53.878715Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.29)  MySQL Community Server - GPL.
2022-08-31 15:25:54+00:00 [Note] [Entrypoint]: Temporary server stopped

2022-08-31 15:25:54+00:00 [Note] [Entrypoint]: MySQL init process done. Ready for start up.

2022-08-31T15:25:54.585100Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.29) starting as process 1
2022-08-31T15:25:54.585163Z 0 [ERROR] [MY-010338] [Server] Can't find error-message file '/usr/share/mysql-8.0/errmsg.sys'. Check error-message file location and 'lc-messages-dir' configuration directive.
2022-08-31T15:25:54.590410Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-31T15:25:54.773277Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-31T15:25:54.926548Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-08-31T15:25:54.926585Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-08-31T15:25:54.927483Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/lib/mysql' in the path is accessible to all OS users. Consider choosing a different directory.
2022-08-31T15:25:54.943057Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.29'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  MySQL Community Server - GPL.
2022-08-31T15:25:54.943373Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock

[kimwnasptd]

Looking around I see this issue, which is almost identical kubernetes-sigs/kind#3046. They suggest it's apparmor configuration, like an older issue they had faced kubernetes-sigs/kind#1179

I'm also seeing a similar discussion, but it doesn't really apply to us since we are creating a Pod and not starting a service directly.

I'll send a PR to play around with apparmor
cc @DnPlas @annajung