Guidance on Self Signed Certificates in TLS

[wf1-brandon-grant]

Hi there,

I am looking to deploy Armada across a few Kubernetes clusters, and am running into issues when connecting Executors to the ingress on the "Server" Cluster.
We are using a self signed TLS certificate and this is seemingly not supported by the Executor, disabling TLS all together on the ingress and the Executor config results in HTTP errors:

cluster_utilisation.go:111 Failed to report cluster usage because rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = "error reading server preface: http2: frame too large"

I could create a custom image for the Executor that includes our CA certificate, and then pull this into the installation, however while the Operator seems to support the use of a custom image for the deployment, there is no way to pass through image-pull-secrets, as such we would need to publish or image to a public repository (with the certificate in it) or rebuild what the Operator deploys for an Executor.

Neither of these are really sustainable or secure approaches and wondered if there was something simple that I was missing to work around this, that I couldn't find digging through the repo?

This is my yaml for some context:

Executor deployment:

---
apiVersion: install.armadaproject.io/v1alpha1
kind: Executor
metadata:
  name: armada-executor
  namespace: armada
spec:
  image:
    repository: gresearch/armada-executor
    tag: 0.3.103
  applicationConfig:
    apiConnection:
      armadaUrl: ${armada_server_url}

Ingress:

---
apiVersion: networking.k8s.io/v1
kind: Ingress
annotations:
  nginx.org/grpc-services: "armada-scheduler"
metadata:
  name: armada-scheduler-ingress
  namespace: armada
spec:
  tls:
    - hosts:
        - armada-scheduler.${dns}
      secretName: tls
  rules:
    - host: armada-scheduler.${dns}
      http:
        paths:
          - pathType: Prefix
            path: /
            backend:
              service:
                name: armada-scheduler
                port:
                  number: 50051

Thank you for any assistance!