fix: restart minio services must happened at config changed when decommission completed

[jiuker]

restart minio services must happened at config changed when decommisson competed
When decommisson completed, we are facing this for pools pods:

1. stat the config.env changed at 03:19:50
2. minio service restart at 03:19:47
3. others pools pods still log pool-1 that should been removed

test steps:

1. make a dev sidecar image

cd sidecar
make docker

2. make a dev operator image

make operator

3. helm install operator with env OPERATOR_SIDECAR_IMAGE="your sidecar dev image"
4. Deploy a multi pools tenant.`. And wait the tenant get be ready
5. do a decommisson , you can ignore this step if you have a fresh install without data.
6. Remove pool from tenant. You will see sidecar log patched pod annotations and delete the statefulset successful.

[ramondeklein]

If I understand correctly, this PR should ensure that the pool is restarted after decomissioning, but it now waits until all pods have the same generation as the Tenant resource?

[jiuker]

If I understand correctly, this PR should ensure that the pool is restarted after decomissioning, but it now waits until all pods have the same generation as the Tenant resource?

Tenant resource will generate env config, sidecar will watch the Tenant and renew.

Yes. As you can see

1. stat the config.env changed at 03:19:50
2. minio service restart at 03:19:47
3. others pools pods still log pool-1 that should been removed

minio server restart before the env changed. So used a outdate env to restart.
Before decomission:
Volumes: pool1 pool2 pool3
After decomission:
Without the pr, Volumes could be pool1 pool2 pool3
With this pr, Volumes must be pool1 pool3 (if pool2 need been removed) @ramondeklein

[ramondeklein]

Minor suggestion, but not really important.

[pjuarezd]

please rebase this PR @jiuker would like to see how tests do

[ramondeklein]

If it's required to build with Go v1.21.11 and not an earlier version, then I guess we should also ensure that we use at least this version to build the code? If any v1.21 version is fine, then I suggest not to fix the version.

[jiuker]

If it's required to build with Go v1.21.11 and not an earlier version, then I guess we should also ensure that we use at least this version to build the code? If any v1.21 version is fine, then I suggest not to fix the version.

https://github.com/minio/operator/actions/runs/9376441443/job/25816378764 But have an govulncheck failed here @ramondeklein

[cniackz]

Jiuker, can we split this PR in multiple atomic ones, I think having this complex task all at once is maybe not the best idea... what do you think?

[jiuker]

This is a complete fix, the problem now is that pjuarezd test does not pass. But I think that should be his develop step error @cniackz . Could you test this?

[cniackz]

@jiuker, please read Harsha's comment on Slack and decide whether to continue this way or simplify things by restarting the service instead.

[harshavardhana]

@jiuker, please read Harsha's comment on Slack and decide whether to continue this way or simplify things by restarting the service instead.

Adding this flow is important because we shouldn't be restarting services if not all pods have the same tenant configuration.

[harshavardhana]

PTAL @pjuarezd @cniackz

[harshavardhana]

PTAL @pjuarezd @cniackz

ping again on this.