Multiregion cluster on EKS: service port type should be UDP

[junaid-ali]

Describe the problem

The doc and manifest suggest to use TCP protocol for port 53 to handle DNS queries. I got i/o timeout errors in CoreDNS logs, and the cross-region DNS resolution kept failing.

Tried to add UDP port alongside the TCP port but mixed ports are only available as part of a feature gate in Kubernetes version 1.20 (kubernetes/kubernetes#23880), and in EKS it doesn't seem to have been enabled yet (kubernetes-sigs/aws-load-balancer-controller#1608)

Ideally, a TCP_UDP protocol would have been the appropriate one, but the aws-loadbalancer-controller needs to have that support: kubernetes-sigs/aws-load-balancer-controller#1608 (comment)

For me, with EKS version 1.20, I was able to resolve the cross-region DB pods only when I recreated the DNS service with the protocol to UDP here:

cockroach/cloud/kubernetes/multiregion/eks/dns-lb-eks.yaml

Line 14 in 44c9505

protocol: TCP

And then we have to remove the force_tcp config under the server blocks:

cockroach/cloud/kubernetes/multiregion/eks/configmap.yaml

Lines 24 to 32 in 44c9505

	<cluster-namespace-2>.svc.cluster.local:53 { # <---- Modify
	log
	errors
	ready
	cache 10
	forward . <ip1> <ip2> <ip3> { # <---- Modify
	force_tcp # <---- Modify
	}
	}

Also, in the docs there should be an example for testing the cross-region DNS resolution (via the NLB), for example:

$ dig @<one-of-the-NLB-IP> cockroachdb.com

$ # Also for pod dns resolution from eu-west-1 to us-west-1, dig via a pod in the eu-west-1 cluster
$ dig @<one of the us-west-1 NLB IP> cockroachdb-0.cockroachdb.<cluster-namespace-name in us-west-1>.svc.cluster.local

To Reproduce
Followed the documentation, NLB with TCP protocol for port 53, CoreDNS will have errors resolving k8s services in the other regions:

$ kubectl -n kube-system logs <coredns pod name> -f
...
[ERROR] plugin/errors: 2 cockroachdb-0.cockroachdb.testdb-us-west-1.svc.cluster.local. AAAA: read tcp 10.13.2.140:40894->44.xx.xx.xx:53: i/o timeout

Expected behavior
Cross-region k8s DNS resolution working seamlessly.

Environment:

• CockroachDB version v21.1.8
• Server OS: Kubernetes EKS v1.20.7-eks-d88609

Jira issue: CRDB-9828

[blathers-crl]

Hello, I am Blathers. I am here to help you get the issue triaged.

Hoot - a bug! Though bugs are the bane of my existence, rest assured the wretched thing will get the best of care here.

I was unable to automatically find someone to ping.

If we have not gotten back to your issue within a few business days, you can try the following:

• Join our community slack channel and ask on #cockroachdb.
• Try find someone from here if you know they worked closely on the area and CC them.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[github-actions]

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!