Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Exec timeouts on PITR reconciler and sync users processes #1837

Open
dcaputo-harmoni opened this issue Oct 6, 2024 · 0 comments · May be fixed by #1838
Open

Exec timeouts on PITR reconciler and sync users processes #1837

dcaputo-harmoni opened this issue Oct 6, 2024 · 0 comments · May be fixed by #1838
Labels
bug

Comments

@dcaputo-harmoni
Copy link
Contributor

Report

I am seeing sporadic errors when the xtradb cluster operator runs the pitr reconciler or sync users commands (see logs below). I verified that both work fine when running the commands directly:

kubectl exec -n core-mysql mysql-db-proxysql-0 -c proxysql -- proxysql-admin --syncusers --add-query-rule

kubectl exec -n core-mysql mysql-db-pitr-6987f698c8-nknx4 -c pitr -- /bin/bash -c "cat /tmp/pitr-timeline"

I noticed that there is a 10 second timeout set here - could this possibly be timing out? The errors appear to be sporadic and the commands execute fine when run directly so I'm wondering if we need to extend the timeout or retry failed commands.

For reference I am running in Azure kubernetes.

More about the problem

    /go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:1235
2024-10-05T22:50:54.326Z    ERROR    Reconciler error    {"controller": "pxc-controller", "namespace": "core-mysql", "name": "mysql-db", "reconcileID": "0e581007-e206-456d-9764-f03a1d3dab93", "error": "check binlog gaps in pod mysql-db-pitr-6987f698c8-nknx4: error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-pitr-6987f698c8-nknx4/exec?command=%2Fbin%2Fbash&command=-c&command=cat+%2Ftmp%2Fgap-detected&container=pitr&stderr=true&stdout=true\": unexpected EOF", "errorVerbose": "error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-pitr-6987f698c8-nknx4/exec?command=%2Fbin%2Fbash&command=-c&command=cat+%2Ftmp%2Fgap-detected&container=pitr&stderr=true&stdout=true\": unexpected EOF\ncheck binlog gaps in pod mysql-db-pitr-6987f698c8-nknx4\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup.CheckPITRErrors\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup/pitr.go:67\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).Reconcile\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:432\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
2024-10-05T23:06:48.687Z    ERROR    Reconciler error    {"controller": "pxc-controller", "namespace": "core-mysql", "name": "mysql-db", "reconcileID": "bc21013c-acad-44e4-890f-8edf5a82cbad", "error": "check binlog gaps in pod mysql-db-pitr-6987f698c8-nknx4: error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-pitr-6987f698c8-nknx4/exec?command=%2Fbin%2Fbash&command=-c&command=cat+%2Ftmp%2Fgap-detected&container=pitr&stderr=true&stdout=true\": unexpected EOF", "errorVerbose": "error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-pitr-6987f698c8-nknx4/exec?command=%2Fbin%2Fbash&command=-c&command=cat+%2Ftmp%2Fgap-detected&container=pitr&stderr=true&stdout=true\": unexpected EOF\ncheck binlog gaps in pod mysql-db-pitr-6987f698c8-nknx4\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup.CheckPITRErrors\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup/pitr.go:67\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).Reconcile\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:432\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
2024-10-05T23:25:00.243Z    ERROR    sync users    {"controller": "pxc-controller", "namespace": "core-mysql", "name": "mysql-db", "reconcileID": "5f7bdf28-a3c7-466d-b15b-833785c11e69", "error": "exec syncusers: error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-proxysql-2/exec?command=proxysql-admin&command=--syncusers&command=--add-query-rule&container=proxysql&stderr=true&stdout=true\": unexpected EOF /  / ", "errorVerbose": "exec syncusers: error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-proxysql-2/exec?command=proxysql-admin&command=--syncusers&command=--add-query-rule&container=proxysql&stderr=true&stdout=true\": unexpected EOF /  / \ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).syncPXCUsersWithProxySQL\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/users.go:922\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).resyncPXCUsersWithProxySQL.func1\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:1233\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).resyncPXCUsersWithProxySQL.func1
    /go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:1235
2024-10-05T23:37:41.927Z    ERROR    Reconciler error    {"controller": "pxc-controller", "namespace": "core-mysql", "name": "mysql-db", "reconcileID": "fe4585d6-95d0-4d7b-beee-3e1656e1967d", "error": "check binlog gaps in pod mysql-db-pitr-6987f698c8-nknx4: error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-pitr-6987f698c8-nknx4/exec?command=%2Fbin%2Fbash&command=-c&command=cat+%2Ftmp%2Fpitr-timeline&container=pitr&stderr=true&stdout=true\": unexpected EOF", "errorVerbose": "error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-pitr-6987f698c8-nknx4/exec?command=%2Fbin%2Fbash&command=-c&command=cat+%2Ftmp%2Fpitr-timeline&container=pitr&stderr=true&stdout=true\": unexpected EOF\ncheck binlog gaps in pod mysql-db-pitr-6987f698c8-nknx4\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup.UpdatePITRTimeline\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup/pitr.go:135\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).Reconcile\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
2024-10-05T23:55:59.867Z    ERROR    Reconciler error    {"controller": "pxc-controller", "namespace": "core-mysql", "name": "mysql-db", "reconcileID": "a95ee408-f7ea-4849-94b5-da7a38d398b6", "error": "check binlog gaps in pod mysql-db-pitr-6987f698c8-nknx4: error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-pitr-6987f698c8-nknx4/exec?command=%2Fbin%2Fbash&command=-c&command=cat+%2Ftmp%2Fpitr-timeline&container=pitr&stderr=true&stdout=true\": unexpected EOF", "errorVerbose": "error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-pitr-6987f698c8-nknx4/exec?command=%2Fbin%2Fbash&command=-c&command=cat+%2Ftmp%2Fpitr-timeline&container=pitr&stderr=true&stdout=true\": unexpected EOF\ncheck binlog gaps in pod mysql-db-pitr-6987f698c8-nknx4\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup.UpdatePITRTimeline\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup/pitr.go:135\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).Reconcile\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
2024-10-05T23:58:12.978Z    ERROR    Reconciler error    {"controller": "pxc-controller", "namespace": "core-mysql", "name": "mysql-db", "reconcileID": "57a7da4f-6046-427d-be8b-8a905e1f2a15", "error": "check binlog gaps in pod mysql-db-pitr-6987f698c8-nknx4: error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-pitr-6987f698c8-nknx4/exec?command=%2Fbin%2Fbash&command=-c&command=cat+%2Ftmp%2Fgap-detected&container=pitr&stderr=true&stdout=true\": unexpected EOF", "errorVerbose": "error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-pitr-6987f698c8-nknx4/exec?command=%2Fbin%2Fbash&command=-c&command=cat+%2Ftmp%2Fgap-detected&container=pitr&stderr=true&stdout=true\": unexpected EOF\ncheck binlog gaps in pod mysql-db-pitr-6987f698c8-nknx4\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup.CheckPITRErrors\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup/pitr.go:67\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).Reconcile\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:432\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
2024-10-06T00:00:57.021Z    INFO    Created a new backup job    {"controller": "pxcbackup-controller", "namespace": "core-mysql", "name": "cron-mysql-db-azure-blob-20241060057-372f8", "reconcileID": "cbaaa485-63eb-4f79-bf65-0e06e9b4e79e", "Namespace": "core-mysql", "Name": "xb-cron-mysql-db-azure-blob-20241060057-372f8"}
2024-10-06T00:24:46.325Z    ERROR    Reconciler error    {"controller": "pxc-controller", "namespace": "core-mysql", "name": "mysql-db", "reconcileID": "224af23c-1276-45b4-ac10-b0d51f67f6fb", "error": "check binlog gaps in pod mysql-db-pitr-6987f698c8-nknx4: error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-pitr-6987f698c8-nknx4/exec?command=%2Fbin%2Fbash&command=-c&command=cat+%2Ftmp%2Fpitr-timeline&container=pitr&stderr=true&stdout=true\": unexpected EOF", "errorVerbose": "error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-pitr-6987f698c8-nknx4/exec?command=%2Fbin%2Fbash&command=-c&command=cat+%2Ftmp%2Fpitr-timeline&container=pitr&stderr=true&stdout=true\": unexpected EOF\ncheck binlog gaps in pod mysql-db-pitr-6987f698c8-nknx4\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup.UpdatePITRTimeline\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup/pitr.go:135\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).Reconcile\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
2024-10-06T00:51:21.719Z    ERROR    Reconciler error    {"controller": "pxc-controller", "namespace": "core-mysql", "name": "mysql-db", "reconcileID": "3a25811b-e28c-4bf2-b359-3fadaecb8049", "error": "check binlog gaps in pod mysql-db-pitr-6987f698c8-nknx4: error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-pitr-6987f698c8-nknx4/exec?command=%2Fbin%2Fbash&command=-c&command=cat+%2Ftmp%2Fpitr-timeline&container=pitr&stderr=true&stdout=true\": unexpected EOF", "errorVerbose": "error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-pitr-6987f698c8-nknx4/exec?command=%2Fbin%2Fbash&command=-c&command=cat+%2Ftmp%2Fpitr-timeline&container=pitr&stderr=true&stdout=true\": unexpected EOF\ncheck binlog gaps in pod mysql-db-pitr-6987f698c8-nknx4\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup.UpdatePITRTimeline\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup/pitr.go:135\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).Reconcile\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
2024-10-06T01:19:14.903Z    ERROR    sync users    {"controller": "pxc-controller", "namespace": "core-mysql", "name": "mysql-db", "reconcileID": "6c4f09eb-7fc3-493d-8164-acbb4258d44b", "error": "exec syncusers: error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-proxysql-2/exec?command=proxysql-admin&command=--syncusers&command=--add-query-rule&container=proxysql&stderr=true&stdout=true\": unexpected EOF /  / ", "errorVerbose": "exec syncusers: error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-proxysql-2/exec?command=proxysql-admin&command=--syncusers&command=--add-query-rule&container=proxysql&stderr=true&stdout=true\": unexpected EOF /  / \ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).syncPXCUsersWithProxySQL\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/users.go:922\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).resyncPXCUsersWithProxySQL.func1\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:1233\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).resyncPXCUsersWithProxySQL.func1
    /go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:1235
2024-10-06T01:20:55.282Z    ERROR    Reconciler error    {"controller": "pxc-controller", "namespace": "core-mysql", "name": "mysql-db", "reconcileID": "2b8d2e22-caf8-482f-88a7-daa246a2e798", "error": "check binlog gaps in pod mysql-db-pitr-6987f698c8-nknx4: error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-pitr-6987f698c8-nknx4/exec?command=%2Fbin%2Fbash&command=-c&command=cat+%2Ftmp%2Fpitr-timeline&container=pitr&stderr=true&stdout=true\": unexpected EOF", "errorVerbose": "error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-pitr-6987f698c8-nknx4/exec?command=%2Fbin%2Fbash&command=-c&command=cat+%2Ftmp%2Fpitr-timeline&container=pitr&stderr=true&stdout=true\": unexpected EOF\ncheck binlog gaps in pod mysql-db-pitr-6987f698c8-nknx4\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup.UpdatePITRTimeline\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup/pitr.go:135\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).Reconcile\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
2024-10-06T01:30:40.983Z    ERROR    sync users    {"controller": "pxc-controller", "namespace": "core-mysql", "name": "mysql-db", "reconcileID": "5faa90eb-6822-49d7-b7b6-62228e32b583", "error": "exec syncusers: error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-proxysql-1/exec?command=proxysql-admin&command=--syncusers&command=--add-query-rule&container=proxysql&stderr=true&stdout=true\": unexpected EOF /  / ", "errorVerbose": "exec syncusers: error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-proxysql-1/exec?command=proxysql-admin&command=--syncusers&command=--add-query-rule&container=proxysql&stderr=true&stdout=true\": unexpected EOF /  / \ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).syncPXCUsersWithProxySQL\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/users.go:922\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).resyncPXCUsersWithProxySQL.func1\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:1233\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).resyncPXCUsersWithProxySQL.func1
    /go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:1235
2024-10-06T01:32:59.073Z    ERROR    Reconciler error    {"controller": "pxc-controller", "namespace": "core-mysql", "name": "mysql-db", "reconcileID": "d1a801b5-3698-4fd1-bc12-427e2381488f", "error": "check binlog gaps in pod mysql-db-pitr-6987f698c8-nknx4: error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-pitr-6987f698c8-nknx4/exec?command=%2Fbin%2Fbash&command=-c&command=cat+%2Ftmp%2Fgap-detected&container=pitr&stderr=true&stdout=true\": unexpected EOF", "errorVerbose": "error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-pitr-6987f698c8-nknx4/exec?command=%2Fbin%2Fbash&command=-c&command=cat+%2Ftmp%2Fgap-detected&container=pitr&stderr=true&stdout=true\": unexpected EOF\ncheck binlog gaps in pod mysql-db-pitr-6987f698c8-nknx4\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup.CheckPITRErrors\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup/pitr.go:67\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).Reconcile\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:432\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
2024-10-06T01:35:01.080Z    ERROR    sync users    {"controller": "pxc-controller", "namespace": "core-mysql", "name": "mysql-db", "reconcileID": "59b00b07-4653-4988-980a-8adde9b68d7d", "error": "exec syncusers: error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-proxysql-2/exec?command=proxysql-admin&command=--syncusers&command=--add-query-rule&container=proxysql&stderr=true&stdout=true\": unexpected EOF /  / ", "errorVerbose": "exec syncusers: error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-proxysql-2/exec?command=proxysql-admin&command=--syncusers&command=--add-query-rule&container=proxysql&stderr=true&stdout=true\": unexpected EOF /  / \ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).syncPXCUsersWithProxySQL\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/users.go:922\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).resyncPXCUsersWithProxySQL.func1\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:1233\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).resyncPXCUsersWithProxySQL.func1
    /go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:1235
2024-10-06T01:36:05.713Z    ERROR    Reconciler error    {"controller": "pxc-controller", "namespace": "core-mysql", "name": "mysql-db", "reconcileID": "4e954cfe-7fb7-4743-aabd-9541930485b4", "error": "check binlog gaps in pod mysql-db-pitr-6987f698c8-nknx4: error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-pitr-6987f698c8-nknx4/exec?command=%2Fbin%2Fbash&command=-c&command=cat+%2Ftmp%2Fpitr-timeline&container=pitr&stderr=true&stdout=true\": unexpected EOF", "errorVerbose": "error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-pitr-6987f698c8-nknx4/exec?command=%2Fbin%2Fbash&command=-c&command=cat+%2Ftmp%2Fpitr-timeline&container=pitr&stderr=true&stdout=true\": unexpected EOF\ncheck binlog gaps in pod mysql-db-pitr-6987f698c8-nknx4\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup.UpdatePITRTimeline\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup/pitr.go:135\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).Reconcile\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
2024-10-06T01:37:06.838Z    ERROR    sync users    {"controller": "pxc-controller", "namespace": "core-mysql", "name": "mysql-db", "reconcileID": "5c24edf2-b04f-4f7e-8239-10b406e784dd", "error": "exec syncusers: error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-proxysql-0/exec?command=proxysql-admin&command=--syncusers&command=--add-query-rule&container=proxysql&stderr=true&stdout=true\": unexpected EOF /  / ", "errorVerbose": "exec syncusers: error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-proxysql-0/exec?command=proxysql-admin&command=--syncusers&command=--add-query-rule&container=proxysql&stderr=true&stdout=true\": unexpected EOF /  / \ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).syncPXCUsersWithProxySQL\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/users.go:922\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).resyncPXCUsersWithProxySQL.func1\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:1233\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).resyncPXCUsersWithProxySQL.func1
    /go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:1235
2024-10-06T02:00:39.408Z    ERROR    sync users    {"controller": "pxc-controller", "namespace": "core-mysql", "name": "mysql-db", "reconcileID": "6a9fc210-be3e-4b68-8aea-ea2cb02886ae", "error": "exec syncusers: error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-proxysql-2/exec?command=proxysql-admin&command=--syncusers&command=--add-query-rule&container=proxysql&stderr=true&stdout=true\": unexpected EOF /  / ", "errorVerbose": "exec syncusers: error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-proxysql-2/exec?command=proxysql-admin&command=--syncusers&command=--add-query-rule&container=proxysql&stderr=true&stdout=true\": unexpected EOF /  / \ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).syncPXCUsersWithProxySQL\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/users.go:922\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).resyncPXCUsersWithProxySQL.func1\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:1233\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).resyncPXCUsersWithProxySQL.func1
    /go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:1235
2024-10-06T02:22:19.637Z    ERROR    Reconciler error    {"controller": "pxc-controller", "namespace": "core-mysql", "name": "mysql-db", "reconcileID": "6674be1e-16cc-4070-a7b2-52eae60fe17f", "error": "check binlog gaps in pod mysql-db-pitr-6987f698c8-nknx4: error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-pitr-6987f698c8-nknx4/exec?command=%2Fbin%2Fbash&command=-c&command=cat+%2Ftmp%2Fpitr-timeline&container=pitr&stderr=true&stdout=true\": unexpected EOF", "errorVerbose": "error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-pitr-6987f698c8-nknx4/exec?command=%2Fbin%2Fbash&command=-c&command=cat+%2Ftmp%2Fpitr-timeline&container=pitr&stderr=true&stdout=true\": unexpected EOF\ncheck binlog gaps in pod mysql-db-pitr-6987f698c8-nknx4\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup.UpdatePITRTimeline\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/pxc/backup/pitr.go:135\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).Reconcile\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:437\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
    /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222
2024-10-06T02:23:16.956Z    ERROR    sync users    {"controller": "pxc-controller", "namespace": "core-mysql", "name": "mysql-db", "reconcileID": "3e33ec67-93bc-407d-abff-9868ce3398a7", "error": "exec syncusers: error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-proxysql-2/exec?command=proxysql-admin&command=--syncusers&command=--add-query-rule&container=proxysql&stderr=true&stdout=true\": unexpected EOF /  / ", "errorVerbose": "exec syncusers: error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-proxysql-2/exec?command=proxysql-admin&command=--syncusers&command=--add-query-rule&container=proxysql&stderr=true&stdout=true\": unexpected EOF /  / \ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).syncPXCUsersWithProxySQL\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/users.go:922\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).resyncPXCUsersWithProxySQL.func1\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:1233\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).resyncPXCUsersWithProxySQL.func1
    /go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:1235
2024-10-06T02:35:52.659Z    ERROR    sync users    {"controller": "pxc-controller", "namespace": "core-mysql", "name": "mysql-db", "reconcileID": "d5e0ecc5-8541-4fb9-8445-3bd89d4466e0", "error": "exec syncusers: error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-proxysql-2/exec?command=proxysql-admin&command=--syncusers&command=--add-query-rule&container=proxysql&stderr=true&stdout=true\": unexpected EOF /  / ", "errorVerbose": "exec syncusers: error sending request: Post \"https://10.0.64.1:443/api/v1/namespaces/core-mysql/pods/mysql-db-proxysql-2/exec?command=proxysql-admin&command=--syncusers&command=--add-query-rule&container=proxysql&stderr=true&stdout=true\": unexpected EOF /  / \ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).syncPXCUsersWithProxySQL\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/users.go:922\ngithub.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).resyncPXCUsersWithProxySQL.func1\n\t/go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:1233\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1695"}
github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc.(*ReconcilePerconaXtraDBCluster).resyncPXCUsersWithProxySQL.func1
    /go/src/github.com/percona/percona-xtradb-cluster-operator/pkg/controller/pxc/controller.go:1235

Steps to reproduce

  1. Install xtradb cluster operator and deploy cluster
  2. Observe errors

Versions

  1. Kubernetes 1.29.4
  2. Operator 1.15.0
  3. Database 1.15.0

Anything else?

No response
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
1 participant
@dcaputo-harmoni