Proxysql user sync generating EOF errors

[dcaputo-harmoni]

Report

I am getting the error messages shown in the attached logs from the cluster operator every time it tries to sync users with proxysql. I have confirmed the operator service account has the necessary permissions to execute the command by downloading kubectl into the operator pod and running the following command:

kubectl exec -n core-mysql mysql-db-proxysql-1 -c proxysql -- proxysql-admin --syncusers --add-query-rule

which successfully completes, as does running the directly on any of the proxysql pods.

More about the problem

2024-08-30T14:38:06.646Z    INFO    Starting Controller    {"controller": "pxcbackup-controller"}
2024-08-30T14:38:06.646Z    INFO    Starting EventSource    {"controller": "pxcrestore-controller", "source": "kind source: *v1.PerconaXtraDBClusterRestore"}
2024-08-30T14:38:06.646Z    INFO    Starting Controller    {"controller": "pxcrestore-controller"}
2024-08-30T14:38:06.753Z    INFO    Starting workers    {"controller": "pxc-controller", "worker count": 1}
2024-08-30T14:38:06.755Z    INFO    Starting workers    {"controller": "pxcrestore-controller", "worker count": 1}
2024-08-30T14:38:06.755Z    INFO    Starting workers    {"controller": "pxcbackup-controller", "worker count": 1}
2024-08-30T14:38:08.833Z    INFO    Creating or updating backup job    {"controller": "pxc-controller", "namespace": "core-mysql", "name": "mysql-db", "reconcileID": "322fff79-75ec-41f7-97bf-0c8a536b6788", "name": "41015-daily-backup", "schedule": "0 0 * * *"}
2024-08-30T14:38:08.833Z    INFO    Creating or updating backup job    {"controller": "pxc-controller", "namespace": "core-mysql", "name": "mysql-db", "reconcileID": "322fff79-75ec-41f7-97bf-0c8a536b6788", "name": "41015-sat-night-backup", "schedule": "0 0 * * 6"}
2024-08-30T14:38:08.834Z    INFO    add new job    {"controller": "pxc-controller", "namespace": "core-mysql", "name": "mysql-db", "reconcileID": "322fff79-75ec-41f7-97bf-0c8a536b6788", "name": "ensure-version/core-mysql/mysql-db", "schedule": "0 4 * * *"}
2024-08-30T14:38:08.834Z    INFO    add new job    {"controller": "pxc-controller", "namespace": "core-mysql", "name": "mysql-db", "reconcileID": "322fff79-75ec-41f7-97bf-0c8a536b6788", "name": "telemetry/core-mysql/mysql-db", "schedule": "10 * * * *"}
2024-08-30T14:50:23.127Z    ERROR    sync users    {"controller": "pxc-controller", "namespace": "core-mysql", "name": "mysql-db", "reconcileID": "f43c9212-9077-4b10-874e-3849722fba04", "error": "exec syncusers: error sending request: Post \"https://10.0.0.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-proxysql-2/exec?command=proxysql-admin&command=--syncusers&command=--add-query-rule&container=proxysql&stderr=true&stdout=true\": unexpected EOF /  / ", "errorVerbose": "exec syncusers: error sending request: Post \"https://10.0.0.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-proxysql-2/exec?command=proxysql-admin&command=--syncusers&command=--add-query-rule&container=proxysql&stderr=true&stdout=true\": unexpected EOF /  / \ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).syncPXCUsersWithProxySQL\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/users.go:922\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).resyncPXCUsersWithProxySQL.func1\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:1233\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).resyncPXCUsersWithProxySQL.func1
    /go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:1235
2024-08-30T15:03:04.987Z    ERROR    sync users    {"controller": "pxc-controller", "namespace": "core-mysql", "name": "mysql-db", "reconcileID": "a014951e-21c1-40bc-b971-d4aa18d7b8dc", "error": "exec syncusers: error sending request: Post \"https://10.0.0.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-proxysql-0/exec?command=proxysql-admin&command=--syncusers&command=--add-query-rule&container=proxysql&stderr=true&stdout=true\": unexpected EOF /  / ", "errorVerbose": "exec syncusers: error sending request: Post \"https://10.0.0.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-proxysql-0/exec?command=proxysql-admin&command=--syncusers&command=--add-query-rule&container=proxysql&stderr=true&stdout=true\": unexpected EOF /  / \ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).syncPXCUsersWithProxySQL\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/users.go:922\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).resyncPXCUsersWithProxySQL.func1\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:1233\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).resyncPXCUsersWithProxySQL.func1
    /go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:1235
... (error continues to repeat)

Steps to reproduce

1. Install percona xtradb operator
2. Install percona xtradb database with proxysql enabled
3. Errors occur

Versions

1. Kubernetes 1.29.4
2. Operator 1.15.0
3. Database 1.15.0

Anything else?

No response

[dcaputo-harmoni]

Dupe with #1837