Single-service ingress not recognizing ssl-passthrough

[kjyo6]

What happened:

We tried using this technique https://kubernetes.io/docs/concepts/services-networking/ingress/#single-service-ingress to send all ingress requests to a single back-end service. The backend service is authenticated using client certificates. We enabled SSL passthrough on the ingress controller via the startup flags. When browsing to the ingress route, we are served the correct certificate from the backend server but we see the client certificate being dropped. This seems like a bug and is not documented anywhere that it should or should not work.

What you expected to happen:

The Ingress Controller's ssl-passthrough is not working

NGINX Ingress controller version (exec into the pod and run nginx-ingress-controller --version.):

NGINX Ingress controller
Release: v1.11.1
Build: 7c44f99
Repository: https://github.com/kubernetes/ingress-nginx
nginx version: nginx/1.25.5

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.6", GitCommit:"11902a838028edef305dfe2f96be929bc4d114d8", GitTreeState:"clean", BuildDate:"2023-06-14T09:56:58Z", GoVersion:"go1.19.10", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"30+", GitVersion:"v1.30.2-eks-db838b0", GitCommit:"04088714581f0ad0a9e2c81c6ecc36bdd30d4b53", GitTreeState:"clean", BuildDate:"2024-06-27T19:09:53Z", GoVersion:"go1.22.4", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.26) and server (1.30) exceeds the supported minor version skew of +/-1

Environment:

AWS EKS, RHEL, AMZN Linux 2
Installed EKS and its node groups using our custom AMI and used our cloudformation templates to create/update node groups

EKS version 1.30, we have 4 worker nodes with latest containerd, EKS and Amazon Linux 2 versions

• How was the ingress-nginx-controller installed:

• I installed ingress-nginx via https://github.com/kubernetes/ingress-nginx/blob/controller-v1.11.1/deploy/static/provider/aws/deploy.yaml

• Current State of the controller:

  • kubectl describe ingressclasses

Name:         ingclass-app
Labels:       app.kubernetes.io/component=controller
              app.kubernetes.io/instance=ingress-nginx
              app.kubernetes.io/name=ingress-app
              app.kubernetes.io/part-of=ingress-nginx
              app.kubernetes.io/version=1.11.1
Annotations:  <none>
Controller:   k8s.io/ingress-app
Events:       <none>

• kubectl -n <ingresscontrollernamespace> get all -A -o wide

• kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>

• kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>

• Current state of ingress object, if applicable:

  • kubectl -n <appnamespace> get all,ing -o wide
    Don't have permissions to show but I want to say that we have running apps and I checked their logs
  • kubectl -n <appnamespace> describe ing <ingressname>
    I have an ingress app that's running with rules (hosts, path, ports, and annotations such as nginx.ingress.kubernetes.io/backend-protocol: https and nginx.ingress.kubernetes.io/ssl-passthrough: true
  • If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

• Others:

  • Any other related information like ;
    • copy/paste of the snippet (if applicable)
    • kubectl describe ... of any custom configmap(s) created and in use
    • Any other related information that may help

Anything else we need to know:

Once I refresh the app more than 10 times, I am getting the error that says: 2024/01/01 00:00:00 [info] 21#21: *1609 peer closed connection in SSL handshake while SSL handshaking, client: ..., server: 0.0.0.0:*port number, then I get several 403 log messages. Everything gets back to normal after a minute

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[longwuyuan]

/remove-kind bug

Edit the issue description and answer all the questions that are asked in the new bug report template because readers need the small tiny details to do any analysis. The details are also critical to being able to reproduce in a minikube or a kind cluster.

Since you are terminating SSL on the backend pod and not in the controller, I am not sure why you would think that the intricate handshake and connection process for SSL has much to do with the controller.

[longwuyuan]

/kind support

[kjyo6]

@longwuyuan I updated my info based on my permission I have. Let me know if you need more.

Also I am investigating nginx conf file and see the location defined in the file.

[longwuyuan]

I see you updated some text. But that text can not be analyzed.

You don't need to provide info from your prod cluster. You can create a cluster a kind cluster or a minikube cluster and then show all the small tiny intricate detailed information of a test from that kind/minikube cluster.

[kjyo6]

Unfortunately I cannot reproduce this on kind or Minikube, but instead I have the /api location that fail in the middle of loading the web page, I am getting 403 errors for several /api/... GET's. I am wondering if there are several failing 403 GETs then is it more likely to be a controller problem or pod problem? I am looking back the previous version, probably before version 1.9.6, since I didn't see this error a few months ago.

/remove-kind bug
/kind support

[k8s-ci-robot]

@kjyo6: Those labels are not set on the issue: kind/bug

In response to this:

Unfortunately I cannot reproduce this on kind or Minikube, but instead I have the /api location that fail in the middle of loading the web page, I am getting 403 errors for several /api/... GET's. I am wondering if there are several failing 403 GETs then is it more likely to be a controller problem or pod problem? I am looking back the previous version, probably before version 1.9.6, since I didn't see this error a few months ago.

/remove-kind bug
/kind support

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[kjyo6]

/remove-kind support

[kjyo6]

Also I am hoping that my issue will be solved with this fix. I am following up with this fix: #11498

[github-actions]

This is stale, but we won't close it automatically, just bare in mind the maintainers may be busy with other tasks and will reach your issue ASAP. If you have any question or request to prioritize this, please reach #ingress-nginx-dev on Kubernetes Slack.