Ingress Configuration Not Working with MinIO Operator

[adomenech73]

Expected Behavior

The ingress configuration should route traffic correctly to the MinIO console and API endpoints. Users should be able to access the MinIO console and API via the specified ingress hosts (e.g., https://minio-console.localhost and https://minio.localhost).

Current Behavior

Despite configuring the ingress resources, users are unable to access the MinIO console and API endpoints. The ingress controller logs show errors such as:

ingress-nginx-controller-867bbcb78-8slz9 controller 2024/12/22 10:42:29 [error] 13148#13148: *2475267 recv() failed (104: Connection reset by peer) while reading upstream, client: 172.18.0.1, server: minio.localhost, request: "GET / HTTP/2.0", upstream: "http://10.244.1.77:9000/", host: "minio.localhost"
ingress-nginx-controller-867bbcb78-8slz9 controller 172.18.0.1 - - [22/Dec/2024:10:42:29 +0000] "GET / HTTP/2.0" 400 48 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" 427 0.009 [minio-minio-https-minio] [] 10.244.1.77:9000 48 0.009 400 6b7411ea865c119721867aaeec2d0466

Steps to Reproduce (for bugs)

1. Deploy MinIO Operator and Tenant:

• Use the following values.yaml file to deploy the MinIO Operator and a tenant.

operator:
  env:
    - name: OPERATOR_STS_ENABLED
      value: "off"
    - name: MINIO_CONSOLE_TLS_ENABLE
      value: "off"
tenant:
  features:
    bucketDNS: true
  image:
    repository: quay.io/minio/minio
    tag: RELEASE.2024-12-18T13-15-44Z-cpuv1
    pullPolicy: IfNotPresent
  name: minio
  pools:
    - name: pool-0
      servers: 1
      size: 2G
  buckets:
    - name: pyroscope-bucket
      objectLock: false
    - name: tempo-bucket
      objectLock: false

ingress:
  api:
    enabled: true
    ingressClassName: nginx
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
      kubernetes.io/tls-acme: "true"
    host: minio.localhost
    tls:
      - hosts:
        - minio.localhost
        secretName: minio-localhost-tls
  console:
    enabled: true
    ingressClassName: nginx
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
      kubernetes.io/tls-acme: "true"
    host: minio-console.localhost
    tls:
      - hosts:
        - minio-console.localhost
        secretName: minio-console-localhost-tls

2. Apply the Configuration:

helm upgrade --install minio-operator minio-operator/operator \                                           <aws:reviewpro>
    --namespace minio \
    --create-namespace \
    --version 6.0.4 \
    -f values.yaml \
    --kube-context kind-kind

helm upgrade --install minio-tenant minio-operator/tenant \                                               <aws:reviewpro>
    --namespace minio \
    --create-namespace \
    --version 6.0.4 \
    -f values-minio.yaml \
    --kube-context kind-kind

3. Verify Ingress Configuration:

• minio-console

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
    meta.helm.sh/release-name: minio-tenant
    meta.helm.sh/release-namespace: minio
    nginx.ingress.kubernetes.io/backend-protocol: HTTP
  creationTimestamp: "2024-12-22T10:40:15Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
  name: minio-console
  namespace: minio
  resourceVersion: "193936"
  uid: 0aae5725-891f-45e9-a5df-f61f348d96c5
spec:
  ingressClassName: nginx
  rules:
  - host: minio-console.localhost
    http:
      paths:
      - backend:
          service:
            name: minio-console
            port:
              name: https-console
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - minio-console.localhost
    secretName: minio-console-localhost-tls
status:
  loadBalancer:
    ingress:
    - hostname: localhost

• minio

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
    meta.helm.sh/release-name: minio-tenant
    meta.helm.sh/release-namespace: minio
    nginx.ingress.kubernetes.io/backend-protocol: HTTP
  creationTimestamp: "2024-12-22T10:40:15Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
  name: minio
  namespace: minio
  resourceVersion: "193937"
  uid: c204d36f-73f4-4708-84d2-36347ab15c3d
spec:
  ingressClassName: nginx
  rules:
  - host: minio.localhost
    http:
      paths:
      - backend:
          service:
            name: minio
            port:
              name: https-minio
        path: /
        pathType: Prefix
  - host: '*.minio.localhost'
    http:
      paths:
      - backend:
          service:
            name: minio
            port:
              name: https-minio
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - minio.localhost
    secretName: minio-localhost-tls
status:
  loadBalancer:
    ingress:
    - hostname: localhost

4. Access MinIO Console and API Ingreses:
   Attempt to access https://minio-console.localhost and https://minio.localhost in your browser.
5. Validate that the minio-console backend service is working properly

kubectl port-forward -n minio svc/minio-console 9443:9443 --context kind-kind

Attempt to access https://localhost:9443

Context

I am trying to deploy MinIO using the MinIO Operator in a Kind cluster for object storage needs. The ingress configuration is crucial for accessing the MinIO console and API endpoints. However, despite following the documentation and configuring the ingress resources, I am unable to access these endpoints.

Regression

No, this is not a regression. It is a new deployment issue.

Your Environment

• Version used (minio-operator): Latest available version from the Helm chart.
• Environment name and version: Kind cluster with Kubernetes v1.31.2.
• Server type and version: Kind cluster running on Docker.
• Operating System and version: macOS 14.6.1
• Link to your deployment file: please check reproduce step 1

[adomenech73]

As a workaround, if possible, I would like to understand how to fully disable TLS on minIO. In my particular use-case i just want to test some functionalities locally, security is not the most important aspect at this point.

I've tried with the environment variables:

tenant:
  env:
   - name: MINIO_CONSOLE_TLS_ENABLE
      value: "off"
   - name: MINIO_SERVER_TLS_ENABLE
      value: "off"

But seems not to be effective at all and I would say that this could be the root cause of my issue as the certificates used are selfsigned then I got tls: failed to verify certificate: x509: certificate signed by unknown authority errors on the other services side. My guess is that similar issue is arising from the ingress standpoint

[michaelfresco]

A vanilla (simple) docker Minio instance may run with arguments like this: minio server /data --console-address :9001. I noticed that the operator creates pods that have the following arguments: server --certs-dir /tmp/certs --console-address 9443. Maybe the --certs-dir is the flag that turns on TLS?

[adomenech73]

Hi @michaelfresco thank's for your response.

Fortunately after inspecting the helm charts and some try errors I got this config that works for my use-case. Just leaving it here in case it could be useful for someone else

tenant:
  certificate:
    requestAutoCert: false
    externalCertSecret: ""
  env:
    - name: MINIO_SERVER_URL
      value: "http://minio-hl.minio.svc.cluster.local:9000"
    - name: MINIO_BROWSER_REDIRECT_URL
      value: "https://minio-console.localhost"
  features:
    bucketDNS: true
    domains:
      console: https://minio-console.localhost
      minio:
        - https://minio.localhost
  image:
    repository: quay.io/minio/minio
    tag: RELEASE.2024-12-18T13-15-44Z-cpuv1
    pullPolicy: IfNotPresent
  name: minio
  pools:
    - name: pool-0
      servers: 1
      size: 2G
  buckets:
    - name: pyroscope-bucket
      objectLock: false
    - name: tempo-bucket
      objectLock: false
  configSecret:
    accessKey: minio
    secretKey: minio123
ingress:
  api:
    enabled: true
    ingressClassName: nginx
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
      kubernetes.io/tls-acme: "true"
    host: minio.localhost
    tls:
      - hosts:
        - minio.localhost
        secretName: minio-localhost-tls
  console:
    enabled: true
    ingressClassName: nginx
    annotations:
      kubernetes.io/ingress.class: nginx
      nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
      kubernetes.io/tls-acme: "true"
    host: minio-console.localhost
    tls:
      - hosts:
        - minio-console.localhost
        secretName: minio-console-localhost-tls

Some notes about it.

• To disable tls you need tenant.certificate.requestAutoCert: false and tenant.certificate.externalCertSecret: "" because there is a default value tenant.certificate.requestAutoCert: true and conditional exists on the chart template like {{- if or .Values.tenant.certificate.requestAutoCert (not (empty .Values.tenant.certificate.externalCertSecret)) }} to define where to use https or http.
• RELEASE.2024-12-18T13-15-44Z-cpuv1 image is needed to workarround x86_v2 support on Mac Silicon virtualization.
• tenant.configSecret.accessKey and tenant.configSecret.secretKey despite how confusing could seem its referred to the console root user and password.

I have to say that documentation is not very clear about it, but in any case at the end was a matter of diving a bit deeper