Generate correctly the MinIO scrape configs for Multi Tenants scenarios

[JoelRuizRojas]

Description

Next fixes introduced:

1. Prometheus scrape config job names were composed by the tenant name only, which makes it difficult to handle multi tenants with same name different namespace. Namespace was added for the MinIO scrape config job names in the secret minio-prom-additional-scrape-config to fix this issue.
2. In Multi tenant scenarios where one is prometheus enabled and another is not, issues updating and deleting the scrape configs was constantly happening in every default tenant reconciliation loop (5 min).
3. Added the same permissions for prometheusagents resources (as with Helm) when deploying the Operator using kustomize.
4. When a tenant is deleted, allow it to be enqueued for controller reconciliation so that the tenant Prometheus scrape configs can be deleted.

A few unit tests were added to cover points (1) and (2) above.

Note: I need to port this fix to AIStor

Related Issue

Fixing issues:

#2425
#2218

Type of Change

• Bug fix 🐛
• New feature 🚀
• Breaking change 🚨
• Documentation update 📖
• Refactor 🔨
• Other (please describe) ⬇️

Screenshots (if applicable e.g before/after)

Checklist

• I have tested these changes
• I have updated relevant documentation (if applicable)
• I have added necessary unit tests (if applicable)

Test Steps

1. Build operator image, and modify Helm deployment operator values to use the locally built operator image
2. kind load docker-image minio/operator:noop (loads locally built image to kind cluster)
3. helm install minio-operator helm/operator --namespace minio-operator --create-namespace (install Operator)
4. helm install prometheus-operator prometheus-community/kube-prometheus-stack(installs prometheus controllers stack and grafana controller)
5. helm install my-tenant helm/tenant --namespace tenant-ns --create-namespace (installs a tenant in a given namespace)
6. helm install my-tenant helm/tenant --namespace tenant-ns --create-namespace (installs a tenant in another given namespace)
7. Edit the tenants so that prometheusOperator: true
8. Wait for operator to create and update the Prometheus scrape configs at secret minio-prom-additional-scrape-config
9. Play around deleting tenants or setting prometheusOperator: false so that the tenant scrape configs are removed from minio-prom-additional-scrape-config secret. Or add more metric endpoint with tenant config PrometheusOperatorScrapeMetricsPaths and verify the scrape configs are added to the secret.

Additional Notes / Context

[cniackz]

LGTM

[Juoper]

any updates on this when this will be merged?

[ramondeklein]

I don't see any actual deletes being added. It resources need to be removed when the Tenant is removed, then they should use the Tenant as the owner of the resource, so deletion is automatic.

[JoelRuizRojas]

@ramondeklein the prometheus scrape configs are stored by design in same secret for all tenants so it can not have a owner reference.

[ramondeklein]

@JoelRuizRojas Is this still relevant?

[JoelRuizRojas]

@JoelRuizRojas Is this still relevant?

Yes @ramondeklein this is still needed, the same for AIStor (we have a PR there too). Sorry I had missed your comment above.