A file deletion is not being synced to minio-pool

[batulziiy]

I use minio-operator as a Github Enterprise Server actions & packages storage. Currently, noticed that the minio-pool is running out of space and immediately took an action by deleting unused container images on the github. But even after deleting hundreds of images, no single MBs freed up on the minio-pool. I suspect file deletion on the github is not being synced to the minio-pool. I can browse the pool, but cannot recognize files easily as the files are named sha256xxxxxxxxxx. Checked operator and pool logs, nothing interesting found. What could be the reason to grow pool size quickly and don't shrink.

Expected Behavior

Expected amount of space freed up on the minio-pool same as amount of image I deleted.

Current Behavior

Minio-pool only grows up and doesn't go down.

Possible Solution

Steps to Reproduce (for bugs)

1. Install github enterprise server
2. Install minio-operator on kubernetes
3. Select minio bucket as a github actions and packages storage
4. Push docker images to the github container registry

Context

Regression

Your Environment

• Version used (minio-operator): 4.5.7
• Environment name and version (e.g. kubernetes v1.17.2): v1.25.4+k3s1
• Server type and version:
• Operating System and version (uname -a): OEL8 Linux 5.15.0-5.76.5.1.el9uek.x86_64
• Link to your deployment file:

[batulziiy]

I found that deleted images are kept for 30 days period due the the github policy. It could be potential reason of the problem. Now I ran into another issue which is sort of chicken egg situation. Let me briefly explain what I did :

1. Normally we have 1 pool under 1 tenant
2. To prevent existing pool gets full, added another pool yesterday.
3. The old pool consists of 3 servers and 4 volumes per server while the new pool consists of 2 servers and 2 volumes per server.

Now when I try to start minio tenant, it never starts and stayed in "provisioning" state due to the below error :
FATAL error : "All current serverPools should have same parity ratio - expected 4, got 2"

I tried to fix the mess I made by updating the new pool with 4 volumes per server. But updating pool settings directly from GUI is not allowed and when I try to change the config or just delete the new pool, cannot connect with mc admin tool due to an error "Server not initialized".

I'm stuck now. If anyone has experience in this situation I would appreciate.

[batulziiy]

Is it possible to delete unused pool while the tenant is in "provisioning" state? Or can we update "volume per server" via GUI? Because since the tenant is not running, I cannot connect with mc tool.

[jiuker]

1. kubectl edit your tenant with secord pool. That should be 1 (servers)*4 (volumes).
2. And I don't think that's not minio or minio-operator issue.
3. We don't support operator with too old version for communty. @batulziiy