Minio-api-ingress does not work with AKS ingress this is related . #2343 #2346

williamsun-hha · 2024-10-19T03:28:23Z

Kubernetes ingress part is configured as following

ingress:
  api:
    enabled: true
    ingressClassName: "webapprouting.kubernetes.azure.com"
    labels: { }
    annotations: { }
    tls:
      - secretName: minio-api-tls-cert  # Reference to the TLS secret for the S3 API
    host: minioapi.xyz.abc.com
    path: /
    pathType: Prefix
  console:
    enabled: true
    ingressClassName: "webapprouting.kubernetes.azure.com"
    labels: { }
    annotations: { }
    tls:
      - secretName: minio-api-tls-cert  # Reference to the TLS secret for the S3 API
    host: minioconsole.xyz.abc.com
    path: /
    pathType: Prefix

Expected Behavior

aws s3 ls --endpoint-url https://minioapi.xyz.abc.com --profile s3
2024-10-18 19:41:45 nextgen-s3

However

Connection was closed before we received a valid response from endpoint URL: "https://minioapi.xyz.abc.com/"

The S3 API is working through the service port.

Current Behavior

Possible Solution

Steps to Reproduce (for bugs)

Context

Regression

Your Environment

Version used (minio-operator):
Environment name and version (e.g. kubernetes v1.17.2):
Server type and version:
Operating System and version (uname -a):
Link to your deployment file:

The text was updated successfully, but these errors were encountered:

ramondeklein · 2024-10-21T12:58:55Z

I do see two issues:

You specify ingressClassName and that seems to be set to a hostname. However, an ingress class name should refer to an ingress class.
It looks like you specify a secret for TLS, so please check if that secret is loaded with a valid TLS certificate.

Most issues should be visible when you describe the ingress resource...

williamsun-hha · 2024-10-22T16:32:42Z

Thank you very much for your help!

webapprouting.kubernetes.azure.com is correct ingress class for Azure web app routing ingress. All of other applications are working in this ingress, include the bitnami minio helm chart.
secret is working as well.
Describe ingress does not have any event ( k describe ingress myminio -n tenant-ns

Name:             myminio
Labels:           app.kubernetes.io/managed-by=Helm
Namespace:        tenant-ns
Address:          17.17.9.11
Ingress Class:    webapprouting.kubernetes.azure.com
Default backend:  <default>
TLS:
  minio-api-tls-cert terminates 
Rules:
  Host                            Path  Backends
  ----                            ----  --------
  minioapi.xyz.abc.com
                                  /   minio:https-minio (10.244.2.198:9000,10.244.3.176:9000,10.244.0.149:9000 + 1 more...)
Annotations:                      meta.helm.sh/release-name: tenant
                                  meta.helm.sh/release-namespace: tenant-ns
Events:                           <none>

). The console ingress has no event either. Since Service Load balancer is working on both api and console, so the backend services are working fine. Bitnami MinIO Console and API ingress is working perfectly on the same AKS cluster as well all of the other cluster.

ramondeklein · 2024-10-22T19:11:15Z

The ingress looks fine at first sight. Can you post the output of curl -v https://minioapi.xyz.abc.com? I see that you are using the https-minio endpoint so the NGINX controller connects to MinIO using TLS too. I can't tell which ingress controller you are using, but you may want to check that too...

williamsun-hha · 2024-10-23T03:50:49Z

Thank you very much for your help, Ramon! I got Client sent an HTTP request to an HTTPS server Response Sent from Yahoo Mail for iPhone On Tuesday, October 22, 2024, 12:11 PM, Ramon de Klein ***@***.***> wrote: The ingress looks fine at first sight. Can you post the output of curl -v https://minioapi.xyz.abc.com? I see that you are using the https-minio endpoint so the NGINX controller connects to MinIO using TLS too. I can't tell which ingress controller you are using, but you may want to check that too... — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

ramondeklein · 2024-10-23T08:45:47Z

You didn't specify the correct arguments to curl. It looks like you used http and https. Also the returned response seems to be incomplete. Please reply via Github instead of e-mail. Please invoke it as curl -v https://<hostname> and post the output.

williamsun-hha · 2024-10-23T21:39:33Z

Thank you very much, Ramon! Here is the full message: * Host minioapi.xyz.abc.com:443 was resolved. * IPv6: (none) * IPv4: 17.9.9.19 * Trying 17.9.9.19:443... * Connected to minioapi.xyz.abc.com(17.9.9.19) port 443 * ALPN: curl offers h2,http/1.1 * (304) (OUT), TLS handshake, Client hello (1): * CAfile: /opt/homebrew/lib/python3.12/site-packages/certifi/cacert.pem * CApath: none * (304) (IN), TLS handshake, Server hello (2): * (304) (IN), TLS handshake, Unknown (8): * (304) (IN), TLS handshake, Certificate (11): * (304) (IN), TLS handshake, CERT verify (15): * (304) (IN), TLS handshake, Finished (20): * (304) (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384 / [blank] / UNDEF * ALPN: server accepted h2 * Server certificate: * subject: CN=*.xyz.abc.com * start date: Oct 2 14:25:33 2024 GMT * expire date: Dec 31 14:25:32 2024 GMT * subjectAltName: host "minioapi.xyz.abc.com" matched cert's "*.xyz.abc.com" * issuer: C=US; O=Let's Encrypt; CN=R10 * SSL certificate verify ok. * using HTTP/2 * [HTTP/2] [1] OPENED stream for https://minioapi.xyz.abc.com/ * [HTTP/2] [1] [:method: GET] * [HTTP/2] [1] [:scheme: https] * [HTTP/2] [1] [:authority: minioapi.xyz.abc.com] * [HTTP/2] [1] [:path: /?] * [HTTP/2] [1] [user-agent: curl/8.7.1] * [HTTP/2] [1] [accept: */*]

GET /? HTTP/2 Host: minioapi.xyz.abc.com User-Agent: curl/8.7.1 Accept: */*

* Request completely sent off < HTTP/2 400 < date: Wed, 23 Oct 2024 21:20:59 GMT < strict-transport-security: max-age=31536000; includeSubDomains < Client sent an HTTP request to an HTTPS server. * Connection #0 to host minioapi.xyz.abc.com left intact ***@***.*** 10-23 % curl -v "https://minioapi.xyz.abc.com?" * Host minioapi.xyz.abc.com:443 was resolved. * IPv6: (none) * IPv4: 17.9.9.19 * Trying 17.9.9.19:443... * Connected to minioapi.xyz.abc.com(17.9.9.19) port 443 * ALPN: curl offers h2,http/1.1 * (304) (OUT), TLS handshake, Client hello (1): * CAfile: /opt/homebrew/lib/python3.12/site-packages/certifi/cacert.pem * CApath: none * (304) (IN), TLS handshake, Server hello (2): * (304) (IN), TLS handshake, Unknown (8): * (304) (IN), TLS handshake, Certificate (11): * (304) (IN), TLS handshake, CERT verify (15): * (304) (IN), TLS handshake, Finished (20): * (304) (OUT), TLS handshake, Finished (20): * SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384 / [blank] / UNDEF * ALPN: server accepted h2 * Server certificate: * subject: CN=*.xyz.abc.com * start date: Oct 2 14:25:33 2024 GMT * expire date: Dec 31 14:25:32 2024 GMT * subjectAltName: host "minioapi.xyz.abc.com" matched cert's "*.xyz.abc.com" * issuer: C=US; O=Let's Encrypt; CN=R10 * SSL certificate verify ok. * using HTTP/2 * [HTTP/2] [1] OPENED stream for https://minioapi.xyz.abc.com/ * [HTTP/2] [1] [:method: GET] * [HTTP/2] [1] [:scheme: https] * [HTTP/2] [1] [:authority: minioapi.xyz.abc.com] * [HTTP/2] [1] [:path: /?] * [HTTP/2] [1] [user-agent: curl/8.7.1] * [HTTP/2] [1] [accept: */*]

GET /? HTTP/2 Host: minioapi.xyz.abc.com User-Agent: curl/8.7.1 Accept: */*

* Request completely sent off < HTTP/2 400 < date: Wed, 23 Oct 2024 21:21:59 GMT < strict-transport-security: max-age=31536000; includeSubDomains < Client sent an HTTP request to an HTTPS server. * Connection #0 to host minioapi.xyz.abc.com left intact

ramondeklein · 2024-10-23T21:58:58Z

It looks like your connection to the Ingress server is fine, but the problem seems to be the connection from the ingress controller to the downstream MinIO server. Your MinIO server seems to be configured to use HTTPS (by default it is), but the ingress controller seems to connect using HTTP. That's probably why it returns Client sent an HTTP request to an HTTPS server..

If you are using NGINX, then you need to add the nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" annotation to your ingress controller to ensure that NGINX connects using HTTPS (docs). I'm not sure if NGINX controller will trust the Kubernetes CA by default (I think it will), so you may run into certificate issues, because NGINX controller doesn't trust the MinIO certificate. If you are using another ingress controller, then please check the documentation of the ingress controller on how to connect to the downstream server using HTTPS.

You can add this annotation in the tenant's Helm value as .ingress.api.annotations, so your values.yaml should probably read something like this:

ingress:
  api:
    enabled: true
    ingressClassName: "webapprouting.kubernetes.azure.com"
    labels: { }
    annotations:
      nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    tls:
      - secretName: minio-api-tls-cert  # Reference to the TLS secret for the S3 API
    host: minioapi.xyz.abc.com
    path: /
    pathType: Prefix
  console:
    enabled: true
    ingressClassName: "webapprouting.kubernetes.azure.com"
    labels: { }
    annotations:
      nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    tls:
      - secretName: minio-api-tls-cert  # Reference to the TLS secret for the S3 API
    host: minioconsole.xyz.abc.com
    path: /
    pathType: Prefix

If you can live without HTTPS within the cluster, then you could also leave it as HTTP and disable HTTPS in MinIO by setting adding the following to the values.yaml:

tenant:
  certificate:
    requestAutoCert: false

This will disable certificates (if you don't specify any other certificates) and should allow NGINX to connect via HTTP.

ramondeklein · 2024-10-23T21:59:39Z

PS: Please reply via Github instead of via email. Replies via email are not formatted and hard to read.

williamsun-hha · 2024-10-24T14:10:04Z

Thank you very much for your help, Ramon! I thought the same thing. As soon as I turn off the Minio TLS, console comes right up. I will check the annotations and test TLS. Really appreciate the help and support! You have a great day!

ramondeklein · 2024-10-24T14:18:31Z

@cesnietor @cniackz We may want to fix this in our Helm script. The default setting is that MinIO will generate certificates and it looks like this break ingress. We may want to add the proper annotations if the MinIO server uses HTTPS.

We also may want to raise warnings instead of generating the annotations. The annotations are ingress controller specific, so we could assume NGINX, but some may use Traefik instead.

williamsun-hha · 2024-10-24T19:01:24Z

Thank you very much for your help, Ramon! Really appreciate the support!

I did exactly as below: It works.

tenant:
  certificate:
    requestAutoCert: false

My goal is to have cert-manager to automatically renew the cert on the ingress. I can try to do that from ingress configuration perspective. Do you have a sample config on helm chart values.yaml to share? If not, I will try to get it implemented from ingress perspective.

Thanks again and have a nice day!

williamsun-hha · 2024-10-30T14:06:57Z

Thank you very much for your help, Ramon!

williamsun-hha added community triage labels Oct 19, 2024

cesnietor closed this as completed Oct 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Minio-api-ingress does not work with AKS ingress this is related . #2343 #2346

Minio-api-ingress does not work with AKS ingress this is related . #2343 #2346

williamsun-hha commented Oct 19, 2024 •

edited by ramondeklein

Loading

ramondeklein commented Oct 21, 2024

williamsun-hha commented Oct 22, 2024 •

edited by ramondeklein

Loading

ramondeklein commented Oct 22, 2024

williamsun-hha commented Oct 23, 2024 via email

ramondeklein commented Oct 23, 2024 •

edited

Loading

williamsun-hha commented Oct 23, 2024 via email •

edited by ramondeklein

Loading

ramondeklein commented Oct 23, 2024

ramondeklein commented Oct 23, 2024

williamsun-hha commented Oct 24, 2024 via email •

edited by ramondeklein

Loading

ramondeklein commented Oct 24, 2024 •

edited

Loading

williamsun-hha commented Oct 24, 2024 •

edited by ramondeklein

Loading

williamsun-hha commented Oct 30, 2024 via email •

edited by ramondeklein

Loading

Minio-api-ingress does not work with AKS ingress this is related . #2343 #2346

Minio-api-ingress does not work with AKS ingress this is related . #2343 #2346

Comments

williamsun-hha commented Oct 19, 2024 • edited by ramondeklein Loading

Expected Behavior

Current Behavior

Possible Solution

Steps to Reproduce (for bugs)

Context

Regression

Your Environment

ramondeklein commented Oct 21, 2024

williamsun-hha commented Oct 22, 2024 • edited by ramondeklein Loading

ramondeklein commented Oct 22, 2024

williamsun-hha commented Oct 23, 2024 via email

ramondeklein commented Oct 23, 2024 • edited Loading

williamsun-hha commented Oct 23, 2024 via email • edited by ramondeklein Loading

ramondeklein commented Oct 23, 2024

ramondeklein commented Oct 23, 2024

williamsun-hha commented Oct 24, 2024 via email • edited by ramondeklein Loading

ramondeklein commented Oct 24, 2024 • edited Loading

williamsun-hha commented Oct 24, 2024 • edited by ramondeklein Loading

williamsun-hha commented Oct 30, 2024 via email • edited by ramondeklein Loading

williamsun-hha commented Oct 19, 2024 •

edited by ramondeklein

Loading

williamsun-hha commented Oct 22, 2024 •

edited by ramondeklein

Loading

ramondeklein commented Oct 23, 2024 •

edited

Loading

williamsun-hha commented Oct 23, 2024 via email •

edited by ramondeklein

Loading

williamsun-hha commented Oct 24, 2024 via email •

edited by ramondeklein

Loading

ramondeklein commented Oct 24, 2024 •

edited

Loading

williamsun-hha commented Oct 24, 2024 •

edited by ramondeklein

Loading

williamsun-hha commented Oct 30, 2024 via email •

edited by ramondeklein

Loading