ingress forward both 80 and 443 ports to tenant clusters (#235)

We need to separate HTTP and HTTPS traffic and send them into tenant
clusters.
Currently traffic was sending only on HTTPS port, this PR enables HTTP
traffic forwarding.

Nginx ingress does not support setting correct upstream according to
type of traffic (http or https)
There are set of issues in upstream.

- kubernetes/ingress-nginx#1655
- kubernetes/ingress-nginx#9061
- kubernetes/ingress-nginx#11334

Good to know that we found reliable workaround

fixes:
#209 (comment)

packages/apps/kubernetes/templates/ingress.yaml

@@ -9,13 +9,28 @@ metadata:
   annotations:
     nginx.ingress.kubernetes.io/ssl-redirect: "false"
     nginx.ingress.kubernetes.io/ssl-passthrough: "true"
+    nginx.ingress.kubernetes.io/backend-protocol: "AUTO_HTTP"
+    nginx.ingress.kubernetes.io/configuration-snippet: |
+      set $proxy_upstream_name "{{ .Release.Namespace }}-{{ .Release.Name }}-ingress-nginx-80";
+      if ($scheme = https) {
+        set $proxy_upstream_name "{{ .Release.Namespace }}-{{ .Release.Name }}-ingress-nginx-443";
+        set $service_port 443;
+      }
+      set $proxy_host $proxy_upstream_name;
 spec:
   ingressClassName: "{{ $ingress }}"
   rules:
   {{- range .Values.addons.ingressNginx.hosts }}
   - host: {{ . | quote }}
     http:
       paths:
+      - path: /
+        pathType: ImplementationSpecific
+        backend:
+          service:
+            name: {{ $.Release.Name }}-ingress-nginx
+            port:
+              number: 80
       - path: /
         pathType: ImplementationSpecific
         backend: