feature: Raft cluster mode supports address translation #7069

PeppaO · 2024-12-23T02:44:15Z

I have registered the PR changes.

Ⅰ. Describe what this PR did

现有raft sdk如果在k8s环境下，客户端集群外访问，无法通过域名方式链接raft node，造成无法通信。
支持外部访问的方案：
server: 在元数据Node中添加一个metadata.external数组，在部署时，传入SEATA_REGISTRY_METADATA_EXTERNAL变量，创建节点时解析设置，同步至主节点。
client:
通过api获取集群元数据，然后解析并选择通信的IP及地址。如果客户端配置了preferredNetworks，那么将从元数据Node.metadata.external中正则匹配选择，否则就按照原来方案，使用Node.control及Node.transaction。
客户端配置：

registry {
  # file 、nacos 、eureka、redis、zk、consul、etcd3、sofa
  type = "raft"
  preferredNetworks = "192.168.*",
  raft {
    metadata-max-age-ms = 30000
    serverAddr = "10.10.92.71:30071,10.10.92.71:30072"

  }
 }

部署文件：

---
apiVersion: v1
kind: Service
metadata:
  name: s-0
  namespace: default
  labels:
    k8s-app: s-0
spec:
  type: NodePort
  ports:
    - port: 7091
      nodePort: 30071
      protocol: TCP
      name: http
    - port: 8091
      nodePort: 30091
      protocol: TCP
      name: tcp
  selector:
    statefulset.kubernetes.io/pod-name: v-0
---
apiVersion: v1
kind: Service
metadata:
  name: s-1
  namespace: default
  labels:
    k8s-app: s-1
spec:
  type: NodePort
  ports:
    - port: 7091
      nodePort: 30072
      protocol: TCP
      name: http
    - port: 8091
      nodePort: 30092
      protocol: TCP
      name: tcp
  selector:
    statefulset.kubernetes.io/pod-name: v-1
---
apiVersion: v1
kind: Service
metadata:
  name: s-2
  namespace: default
  annotations:
    my-custom-annotation: "value1"
    another-annotation: "value2"
  labels:
    k8s-app: s-2
    my-label: "30073"
spec:
  type: NodePort
  ports:
    - port: 7091
      nodePort: 30073
      protocol: TCP
      name: http
    - port: 8091
      nodePort: 30093
      protocol: TCP
      name: tcp
  selector:
    statefulset.kubernetes.io/pod-name: v-2
---
apiVersion: v1
kind: Service
metadata:
  name: svc-l
  namespace: default
  labels:
    k8s-app: svc-l
spec:
  type: ClusterIP
  clusterIP: None
  publishNotReadyAddresses: false
  ports:
    - name: http
      port: 7091
      targetPort: 7091
    - name: tcp
      port: 8091
      targetPort: 8091
  selector:
    k8s-app: v

---

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: v
  namespace: default
  labels:
    k8s-app: v
spec:
  replicas: 3
  serviceName: svc-l
  selector:
    matchLabels:
      k8s-app: v
  template:
    metadata:
      labels:
        k8s-app: v
    spec:
      initContainers:
        - name: cluster-init
          image: harbor.local:80/seata/seata-server:4.0
          imagePullPolicy: Always
          command: [
            "sh", "-c", "index=$(hostname | awk -F '-' '{print $NF}'); \
            case $index in \
              0) echo 30071:30091 > /etc/nodeport/port ;; \
              1) echo 30072:30092 > /etc/nodeport/port ;; \
              2) echo 30073:30093 > /etc/nodeport/port ;; \
              *) echo 'Invalid index' > /etc/nodeport/port ;; \
            esac"
          ]
          volumeMounts:
            - name: nodeport-volume
              mountPath: /etc/nodeport
      containers:
        - name: server
          image: harbor.local:80/seata/seata-server:4.0
          imagePullPolicy: Always
          command: ["bash", "-c", "
            export SEATA_REGISTRY_METADATA_EXTERNAL=${NODE_IP}:$(cat /etc/nodeport/port) && \
            echo 'Starting server with SEATA_REGISTRY_METADATA_EXTERNAL='$SEATA_REGISTRY_METADATA_EXTERNAL && \
            /seata-server/bin/seata-server.sh && \
            echo 'Server started successfully!' && \
            tail -f /dev/null"
          ]
          env:
            - name: SEATA_HOST_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: SEATA_IP
              value: "$(SEATA_HOST_NAME).svc-l.default.svc.cluster.local"
            - name: server.raftPort
              value: "9091"
            - name: server.raft.serverAddr
              value: "v-0.svc-l.default.svc.cluster.local:9091,v-1.svc-l.default.svc.cluster.local:9091,v-2.svc-l.default.svc.cluster.local:9091"
            - name: NODE_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
          ports:
            - name: http
              containerPort: 8091
              protocol: TCP
          volumeMounts:
            - name: nodeport-volume
              mountPath: /etc/nodeport
      volumes:
        - name: nodeport-volume
          emptyDir: {}

这里之所以取名这么简单是测试的时候，发现service过长，会出现 #7082 的错误

元数据：

{
	"nodes": [{
		"control": {
			"host": "v-0.svc-l.default.svc.cluster.local",
			"port": 7091
		},
		"transaction": {
			"host": "v-0.svc-l.default.svc.cluster.local",
			"port": 8091
		},
		"internal": {
			"host": "v-0.svc-l.default.svc.cluster.local",
			"port": 9091
		},
		"group": "default",
		"role": "LEADER",
		"version": "2.3.0-SNAPSHOT",
		"metadata": {
			"external": [{
				"host": "192.168.105.7",
				"controlPort": 30071,
				"transactionPort": 30091
			}]
		}
	}, {
		"control": {
			"host": "v-2.svc-l.default.svc.cluster.local",
			"port": 7091
		},
		"transaction": {
			"host": "v-2.svc-l.default.svc.cluster.local",
			"port": 8091
		},
		"internal": {
			"host": "v-2.svc-l.default.svc.cluster.local",
			"port": 9091
		},
		"group": "default",
		"role": "FOLLOWER",
		"version": "2.3.0-SNAPSHOT",
		"metadata": {
			"external": [{
				"host": "192.168.105.7",
				"controlPort": 30073,
				"transactionPort": 30093
			}]
		}
	}, {
		"control": {
			"host": "v-1.svc-l.default.svc.cluster.local",
			"port": 7091
		},
		"transaction": {
			"host": "v-1.svc-l.default.svc.cluster.local",
			"port": 8091
		},
		"internal": {
			"host": "v-1.svc-l.default.svc.cluster.local",
			"port": 9091
		},
		"group": "default",
		"role": "FOLLOWER",
		"version": "2.3.0-SNAPSHOT",
		"metadata": {
			"external": [{
				"host": "192.168.105.7",
				"controlPort": 30072,
				"transactionPort": 30092
			}]
		}
	}],
	"storeMode": "raft",
	"term": 1
}

测试：

Ⅱ. Does this pull request fix one issue?

Ⅲ. Why don't you add test cases (unit test/integration test)?

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

funky-eyes · 2024-12-23T02:55:55Z

...ery-raft/src/main/java/org/apache/seata/discovery/registry/raft/RaftRegistryServiceImpl.java

+
+    private static InetSocketAddress selectEndpoint(String type, Node node) {
+        if (StringUtils.isBlank(PREFERRED_NETWORKS)) {
+            // 采取默认的方式，直接使用node.control  node.transaction


Suggested change

// 采取默认的方式，直接使用node.control node.transaction

// Use the default method, directly using node.control and node.transaction

funky-eyes · 2024-12-23T02:59:41Z

server/src/main/java/org/apache/seata/server/cluster/raft/RaftStateMachine.java

@@ -366,7 +366,7 @@ public RaftClusterMetadata changeOrInitRaftClusterMetadata() {
                    Integer.parseInt(
                        ((Environment)ObjectHolder.INSTANCE.getObject(OBJECT_KEY_SPRING_CONFIGURABLE_ENVIRONMENT))
                            .getProperty("server.port", String.valueOf(7091))),
-                    group, Collections.emptyMap());
+                    group, new HashMap<>());


Why change to new HashMap < > ()

因为Collections.emptyMap()返回的集合使用了final修饰，导致我进行put操作时，报错，unsupoort...

不要直接put,而是构建一个新的map替换它,copyonwrite

不要直接put,而是构建一个新的map替换它,copyonwrite

done

funky-eyes · 2024-12-23T03:01:07Z

server/src/main/java/org/apache/seata/server/cluster/raft/RaftStateMachine.java

@@ -406,7 +406,7 @@ private void syncCurrentNodeInfo(PeerId leaderPeerId) {
                    Integer.parseInt(
                        ((Environment)ObjectHolder.INSTANCE.getObject(OBJECT_KEY_SPRING_CONFIGURABLE_ENVIRONMENT))
                            .getProperty("server.port", String.valueOf(7091))),
-                    group, Collections.emptyMap());
+                    group, new HashMap<>());


funky-eyes · 2024-12-23T03:04:24Z

server/src/main/java/org/apache/seata/server/cluster/raft/sync/msg/dto/RaftClusterMetadata.java

@@ -55,6 +58,10 @@ public Node createNode(String host, int txPort, int internalPort, int controlPor
        node.setGroup(group);
        node.setVersion(Version.getCurrent());
        node.setInternal(node.createEndpoint(host, internalPort, "raft"));
+        String serverRegistryMetadataExternalValue = System.getProperty("SERVER_REGISTRY_METADATA_EXTERNAL_VALUE");


不应该这样读取，如果有用户直接使用properties配置文件去配置怎么办？应该通过下面这样的方式读取
It should not be read like this. What if a user directly uses the properties configuration file to configure it? It should be read in the following way

ConfigurableEnvironment environment=ObjectHolder.INSTANCE.getObject(OBJECT_KEY_SPRING_CONFIGURABLE_ENVIRONMENT); environment.resolvePlaceholders("${seata.registry.metadata.external:${registry.metadata.external:}}")

done

不要直接put,而是构建一个新的map替换它,copyonwrite

...ery-raft/src/main/java/org/apache/seata/discovery/registry/raft/RaftRegistryServiceImpl.java

codecov · 2024-12-28T10:47:08Z

Codecov Report

Attention: Patch coverage is 25.83333% with 89 lines in your changes missing coverage. Please review.

Project coverage is 52.59%. Comparing base (c0a2e92) to head (ff5ff02).
Report is 1 commits behind head on 2.x.

Files with missing lines	Patch %	Lines
...in/java/org/apache/seata/common/metadata/Node.java	0.00%	41 Missing ⚠️
...scovery/registry/raft/RaftRegistryServiceImpl.java	47.05%	20 Missing and 7 partials ⚠️
...seata/common/exception/ParseEndpointException.java	0.00%	10 Missing ⚠️
...cluster/raft/sync/msg/dto/RaftClusterMetadata.java	37.50%	3 Missing and 2 partials ⚠️
...ain/java/org/apache/seata/common/util/NetUtil.java	40.00%	3 Missing ⚠️
...roperties/registry/RegistryMetadataProperties.java	25.00%	3 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##                2.x    #7069      +/-   ##
============================================
- Coverage     52.65%   52.59%   -0.07%     
- Complexity     6666     6680      +14     
============================================
  Files          1129     1131       +2     
  Lines         40168    40277     +109     
  Branches       4708     4721      +13     
============================================
+ Hits          21152    21185      +33     
- Misses        16999    17067      +68     
- Partials       2017     2025       +8

Files with missing lines	Coverage Δ
...ava/org/apache/seata/common/ConfigurationKeys.java	`0.00% <ø> (ø)`
...toconfigure/SeataCoreEnvironmentPostProcessor.java	`100.00% <100.00%> (ø)`
...ta/spring/boot/autoconfigure/StarterConstants.java	`100.00% <ø> (ø)`
...ain/java/org/apache/seata/common/util/NetUtil.java	`52.98% <40.00%> (-0.51%)`	⬇️
...roperties/registry/RegistryMetadataProperties.java	`25.00% <25.00%> (ø)`
...cluster/raft/sync/msg/dto/RaftClusterMetadata.java	`77.14% <37.50%> (-12.15%)`	⬇️
...seata/common/exception/ParseEndpointException.java	`0.00% <0.00%> (ø)`
...scovery/registry/raft/RaftRegistryServiceImpl.java	`23.70% <47.05%> (+6.23%)`	⬆️
...in/java/org/apache/seata/common/metadata/Node.java	`38.09% <0.00%> (-24.41%)`	⬇️

... and 3 files with indirect coverage changes

…aO/seata into support_external_access_2.x

funky-eyes · 2024-12-30T01:36:34Z

...ery-raft/src/main/java/org/apache/seata/discovery/registry/raft/RaftRegistryServiceImpl.java

+            throw new ParseEndpointException("Node metadata is empty.");
+        }
+
+        Object external = metadata.get("external");


这里要判空，高版本客户端对接低版本server是不会有这个字段的

低版本不支持这个功能，不会走到这个逻辑，会直接走这里

funky-eyes · 2024-12-30T01:37:39Z

changes/zh-cn/2.x.md

@@ -4,7 +4,7 @@

 ### feature:

- [[#PR_NO](https://github.com/seata/seata/pull/PR_NO)] 支持XXX
+- [[#7069](https://github.com/apache/incubator-seata/pull/7069)] Raft模式支持集群外访问


应该是 raft集群模式支持地址转换

funky-eyes

LGTM

集群外访问代码基本完成

5d6f777

funky-eyes added this to the 2.4.0 milestone Dec 23, 2024

funky-eyes added type: feature Category issues or prs related to feature request. module/discovery discovery module module/server server module store: raft labels Dec 23, 2024

funky-eyes reviewed Dec 23, 2024

View reviewed changes

PeppaO added 5 commits December 28, 2024 14:15

优化

96ac1ee

update

42da192

update

90683ea

update

82ca589

update

1d631e2

funky-eyes and others added 5 commits December 28, 2024 22:06

Merge branch '2.x' into support_external_access_2.x

263ddb4

update

3e03486

Merge branch '2.x' into support_external_access_2.x

9650fed

Merge branch 'support_external_access_2.x' of https://github.com/Pepp…

fe4ca17

…aO/seata into support_external_access_2.x

登记pr

5c4dd14

funky-eyes reviewed Dec 30, 2024

View reviewed changes

PeppaO changed the title ~~feature: Raft mode supports access outside the cluster~~ feature: Raft cluster mode supports address translation Dec 30, 2024

PeppaO added 2 commits December 30, 2024 10:22

pr修改

f9f9f32

add config

ff5ff02

funky-eyes approved these changes Dec 30, 2024

View reviewed changes

funky-eyes merged commit dc08160 into apache:2.x Dec 30, 2024
7 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature: Raft cluster mode supports address translation #7069

feature: Raft cluster mode supports address translation #7069

PeppaO commented Dec 23, 2024 •

edited

Loading

funky-eyes Dec 23, 2024

PeppaO Dec 28, 2024

funky-eyes Dec 23, 2024

PeppaO Dec 23, 2024

funky-eyes Dec 28, 2024

PeppaO Dec 29, 2024

funky-eyes Dec 23, 2024

PeppaO Dec 23, 2024

funky-eyes Dec 23, 2024

PeppaO Dec 28, 2024

funky-eyes Dec 28, 2024

codecov bot commented Dec 28, 2024 •

edited

Loading

funky-eyes Dec 30, 2024

PeppaO Dec 30, 2024

funky-eyes Dec 30, 2024

PeppaO Dec 30, 2024

funky-eyes left a comment

	// 采取默认的方式，直接使用node.control node.transaction
	// Use the default method, directly using node.control and node.transaction

feature: Raft cluster mode supports address translation #7069

feature: Raft cluster mode supports address translation #7069

Conversation

PeppaO commented Dec 23, 2024 • edited Loading

Ⅰ. Describe what this PR did

Ⅱ. Does this pull request fix one issue?

Ⅲ. Why don't you add test cases (unit test/integration test)?

Ⅳ. Describe how to verify it

Ⅴ. Special notes for reviews

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Dec 28, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

funky-eyes left a comment

Choose a reason for hiding this comment

PeppaO commented Dec 23, 2024 •

edited

Loading

codecov bot commented Dec 28, 2024 •

edited

Loading