-
Notifications
You must be signed in to change notification settings - Fork 449
默认VPC暴露CoreDNS服务
oilbeater edited this page Jun 27, 2022
·
2 revisions
Wiki 下的中文文档将不在维护，请访问我们最新的中文文档网站，获取最新的文档更新。
Kube-OVN已经支持了Vpc的实现，具体配置可以参考 Vpc配置。
目前CoreDNS服务只能在默认的Vpc ovn-cluster下访问，自定义Vpc下无法访问到CoreDNS服务，可以通过手动配置来实现自定义Vpc对CoreDNS服务的访问。通过创建vpc-nat-gw实例，可以为默认Vpc ovn-cluster 创建网关Pod，通过网关Pod将CoreDNS服务暴露出去。
首先确认环境上安装了multus-cni和macvlan cni，这是为vpc-nat-gw pod创建附加网卡的前提。其次检查kube-system Namespace下ConfigMap ovn-vpc-nat-gw-config 是否存在，这是创建vpc 网关pod的前提，具体ConfigMap的配置，可以参考Vpc配置。
Kube-OVN安装以后，已经存在默认Vpc ovn-cluster和默认Subnet ovn-default，可以使用默认子网来为网关pod分配地址，也可以创建新子网，分配新子网范围的地址。
使用以下yaml，创建新子网
apiVersion: kubeovn.io/v1
kind: Subnet
metadata:
name: test
spec:
cidrBlock: 192.100.0.0/16
default: false
disableGatewayCheck: false
disableInterConnection: true
gatewayNode: ""
gatewayType: distributed
natOutgoing: false
private: false
protocol: IPv4
provider: ovn
vpc: ovn-cluster
使用以下yaml，创建vpc-nat-gw实例，为默认Vpc创建网关pod
apiVersion: kubeovn.io/v1
kind: VpcNatGateway
metadata:
name: default
spec:
vpc: ovn-cluster # 默认vpc
subnet: test # 给网关pod分配IP的子网，可以使用ovn-default
lanIp: 192.100.10.10 # 网关pod IP
eips:
- eipCIDR: 172.18.0.12/16 # 指定对外暴露的eip地址，根据实际网络情况配置
gateway: 172.18.0.1
- eipCIDR: 172.18.0.22/16
gateway: 172.18.0.1
snatRules:
- eip: 172.18.0.12 # CoreDNS Svc 映射eip地址
internalCIDR: 10.96.0.10/32 # 指定CoreDNS Service IP
dnatRules:
- eip: 172.18.0.12
externalPort: "53"
internalIp: 10.96.0.10 # 指定CoreDNS Service IP
internalPort: "53"
protocol: udp # 指定udp协议
执行yaml创建vpc-nat-gw之后，会在kube-system Namespace下创建对应的网关Pod。
apple@appledeMacBook-Pro ovn-test % kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-f9fd979d6-dcppf 1/1 Running 0 4d18h
coredns-f9fd979d6-fg7rw 1/1 Running 0 4d18h
etcd-kube-ovn-control-plane 1/1 Running 0 4d18h
kube-apiserver-kube-ovn-control-plane 1/1 Running 0 4d18h
kube-controller-manager-kube-ovn-control-plane 1/1 Running 0 4d18h
kube-multus-ds-g782g 1/1 Running 0 22h
kube-multus-ds-knj7m 1/1 Running 0 22h
kube-ovn-cni-2q6b9 1/1 Running 0 4d18h
kube-ovn-cni-6x7jl 1/1 Running 0 4d18h
kube-ovn-controller-7658c87bd-kdwd8 1/1 Running 0 4d18h
kube-ovn-monitor-5dc58b495c-xv5vz 1/1 Running 0 4d18h
kube-ovn-pinger-9mc6l 1/1 Running 0 4d18h
kube-ovn-pinger-xckxs 1/1 Running 0 4d18h
kube-proxy-7xk9j 1/1 Running 0 4d18h
kube-proxy-h9r6x 1/1 Running 0 4d18h
kube-scheduler-kube-ovn-control-plane 1/1 Running 0 4d18h
ovn-central-6b87fcd545-pt8hr 1/1 Running 0 4d18h
ovs-ovn-8nvj8 1/1 Running 0 4d18h
ovs-ovn-wffd2 1/1 Running 0 4d18h
vpc-nat-gw-default-cb7b9677f-q6sbg 1/1 Running 0 17h
apple@appledeMacBook-Pro ovn-test %
在默认Vpc ovn-cluster下，不需要添加到网关pod的自定义路由。ovn-cluster Vpc下的子网之间，在创建时已经添加了路由信息，可以实现网络互通。
在vpc-nat-gw网关Pod中，添加以下两条配置信息
ip route add DNS_SVC_IP via Subnet_Gateway_IP table 100 和 iptables -t nat -I POSTROUTING -d DNS_SVC_IP -j MASQUERADE。Subnet_Gateway_IP 是指vpc-nat-gw网关Pod所在子网的网关地址。
apple@appledeMacBook-Pro ovn-test % kubectl exec -it -n kube-system vpc-nat-gw-default-cb7b9677f-q6sbg -- bash
bash-5.1# ip route add 10.96.0.10 via 192.100.0.1 table 100
bash-5.1# ip route show table 100
default via 172.18.0.1 dev net1
10.16.0.0/16 via 192.100.0.1 dev eth0
10.96.0.10 via 192.100.0.1 dev eth0
100.64.0.0/16 via 192.100.0.1 dev eth0
192.100.0.0/16 via 192.100.0.1 dev eth0
bash-5.1#
bash-5.1# iptables -t nat -I POSTROUTING -d 10.96.0.10 -j MASQUERADE
bash-5.1# iptables -t nat -L -n -v
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
198 13204 DNAT_FILTER all -- * * 0.0.0.0/0 0.0.0.0/0
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
1 60 MASQUERADE all -- * * 0.0.0.0/0 10.96.0.10
2 144 SNAT_FILTER all -- * * 0.0.0.0/0 0.0.0.0/0
Chain DNAT_FILTER (1 references)
pkts bytes target prot opt in out source destination
198 13204 EXCLUSIVE_DNAT all -- * * 0.0.0.0/0 0.0.0.0/0
198 13204 SHARED_DNAT all -- * * 0.0.0.0/0 0.0.0.0/0
Chain EXCLUSIVE_DNAT (1 references)
pkts bytes target prot opt in out source destination
Chain EXCLUSIVE_SNAT (1 references)
pkts bytes target prot opt in out source destination
Chain SHARED_DNAT (1 references)
pkts bytes target prot opt in out source destination
87 6288 DNAT udp -- * * 0.0.0.0/0 172.18.0.12 udp dpt:53 to:10.96.0.10:53
Chain SHARED_SNAT (1 references)
pkts bytes target prot opt in out source destination
0 0 SNAT all -- * * 10.96.0.10 0.0.0.0/0 to:172.18.0.12
Chain SNAT_FILTER (1 references)
pkts bytes target prot opt in out source destination
2 144 EXCLUSIVE_SNAT all -- * * 0.0.0.0/0 0.0.0.0/0
2 144 SHARED_SNAT all -- * * 0.0.0.0/0 0.0.0.0/0
bash-5.1#
自定义Vpc的配置，可以参考Vpc配置。
使用以下yaml，创建自定义Vpc
kind: Vpc
apiVersion: kubeovn.io/v1
metadata:
name: vpc1
spec:
namespaces:
- vpc1
使用以下yaml，创建自定义vpc下的subnet
apiVersion: kubeovn.io/v1
kind: Subnet
metadata:
name: vpc1-subnet1
spec:
cidrBlock: 192.168.0.0/16
default: false
disableGatewayCheck: false
disableInterConnection: true
gatewayNode: ""
gatewayType: distributed
natOutgoing: false
private: false
protocol: IPv4
provider: ovn
vpc: vpc1
namespaces:
- vpc1
使用以下yaml创建vpc-nat-gw实例，为自定义vpc创建网关pod
kind: VpcNatGateway
apiVersion: kubeovn.io/v1
metadata:
name: gw1
spec:
vpc: vpc1
subnet: vpc1-subnet1
lanIp: 192.168.10.10
eips:
- eipCIDR: 172.18.0.30/16
gateway: 172.18.0.1
编辑创建的自定义Vpc vpc1，在spec中添加子网到网关的路由
apiVersion: kubeovn.io/v1
kind: Vpc
metadata:
name: vpc1
spec:
namespaces:
- vpc1
staticRoutes:
- cidr: 0.0.0.0/0
nextHopIP: 192.168.10.10
policy: policyDst
使用以下yaml在自定义Vpc下创建测试Pod和测试Service
---
apiVersion: v1
kind: Service
metadata:
name: dns-test-svc
namespace: vpc1
spec:
selector:
name: busybox
ports:
- name: busybox
protocol: TCP
port: 1000
targetPort: 1000
type: ClusterIP
---
apiVersion: v1
kind: Pod
metadata:
name: dns-test
namespace: vpc1
labels:
name: busybox
spec:
containers:
- image: busybox:1.28.3
command:
- sleep
- "3600"
name: busybox
dnsPolicy: "None" # 测试Pod需要设置dns策略和配置
dnsConfig:
nameservers:
- 172.18.0.12 # nameserver指向默认Vpc下CoreDNS服务的EIP
searches:
- vpc1.svc.cluster.local # 添加search域，namespace取值为测试Pod所在的Namespace
- svc.cluster.local
- cluster.local
options:
- name: ndots
value: "5"
查看Pod和Service信息
apple@appledeMacBook-Pro ovn-test % kubectl get pod -n vpc1 -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
dns-test 1/1 Running 0 56m 192.168.10.12 kube-ovn-worker <none> <none>
apple@appledeMacBook-Pro ovn-test % kubectl get svc -n vpc1 -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
dns-test-svc ClusterIP 10.96.29.246 <none> 1000/TCP 65m name=busybox
apple@appledeMacBook-Pro ovn-test %
修改vpc-nat-gw资源，添加测试pod对应的floatingIP信息
apple@appledeMacBook-Pro ovn-test % kubectl get vpc-nat-gw gw1 -o yaml
apiVersion: kubeovn.io/v1
kind: VpcNatGateway
metadata:
name: gw1
spec:
eips:
- eipCIDR: 172.18.0.30/16
gateway: 172.18.0.1
floatingIpRules:
- eip: 172.18.0.30
internalIp: 192.168.10.12
lanIp: 192.168.10.10
subnet: vpc1-subnet1
vpc: vpc1
apple@appledeMacBook-Pro ovn-test %
在测试pod中，使用nslookup命令查询DNS信息
apple@appledeMacBook-Pro ovn-test % kubectl exec -it -n vpc1 dns-test -- sh
/ # cat /etc/resolv.conf
search vpc1.svc.cluster.local svc.cluster.local cluster.local
nameserver 172.18.0.12
options ndots:5
/ #
/ # nslookup dns-test-svc.vpc1
Server: 172.18.0.12
Address 1: 172.18.0.12
Name: dns-test-svc.vpc1
Address 1: 10.96.29.246 dns-test-svc.vpc1.svc.cluster.local
/ #