Can l force karmada-agent to use HTTP/1.1 ?

[Heylosky]

Please provide an in-depth description of the question you have:
Can I force karmada-agent to connect to karmada apiserver via HTTP/1.1 instead of HTTP/2.0? In some strict production environments or through complex network devices, h2 protocol is not supported. Can karmada-agent support this?

What do you think about this question?:
I think karmada-agent does not provide relevant configuration options, so I modified the code. I added something like "clusterConfig.TLSClientConfig.NextProtos = []string{"http/1.1"}", but it doesn't seem to work. Error log after forced protocol conversion.

I0116 10:12:55.086668       1 execution_controller.go:80] Reconciling Work karmada-es-member1/pyy-f69cf8d6c
I0116 10:12:55.131713       1 work_status_controller.go:229] Reflecting the resource(kind=Namespace, /pyy) status to the Work(karmada-es-member1/pyy-f69cf8d6c).
I0116 10:12:55.131785       1 configurable.go:208] Reflect status of object: /v1, Kind=Namespace /pyy with configurable interpreter.
I0116 10:12:55.131800       1 thirdparty.go:194] Reflect status of object: /v1, Kind=Namespace /pyy with thirdparty configurable interpreter.
I0116 10:12:55.131821       1 default.go:150] Reflect status of object: /v1, Kind=Namespace /pyy with build-in interpreter.
I0116 10:12:55.131878       1 configurable.go:227] Get health status of object: /v1, Kind=Namespace /pyy with configurable interpreter.
I0116 10:12:55.132165       1 thirdparty.go:213] Get health status of object: /v1, Kind=Namespace /pyy with thirdparty configurable interpreter.
I0116 10:12:55.132181       1 default.go:162] Get health status of object: /v1, Kind=Namespace /pyy with build-in interpreter.
I0116 10:12:55.131929       1 recorder.go:104] "Reflect status for object(Namespace//pyy) succeed." logger="events" type="Normal" object={"kind":"Work","namespace":"karmada-es-member1","name":"pyy-f69cf8d6c","uid":"54a103ff-c9a7-40ed-96ca-9d7c745dced5","apiVersion":"work.karmada.io/v1alpha1","resourceVersion":"1103632"} reason="ReflectStatusSucceed"
I0116 10:12:55.132401       1 recorder.go:104] "Interpret health of object(Namespace//pyy) failed, err: default InterpretHealth interpreter for \"/v1, Kind=Namespace\" not found." logger="events" type="Warning" object={"kind":"Work","namespace":"karmada-es-member1","name":"pyy-f69cf8d6c","uid":"54a103ff-c9a7-40ed-96ca-9d7c745dced5","apiVersion":"work.karmada.io/v1alpha1","resourceVersion":"1103632"} reason="InterpretHealthFailed"
E0116 10:12:55.175537       1 execution_controller.go:120] Failed to update work condition type Dispatching. err is Operation cannot be fulfilled on works.work.karmada.io "pyy-f69cf8d6c": the object has been modified; please apply your changes to the latest version and try again
E0116 10:12:55.175604       1 controller.go:324] "Reconciler error" err="Operation cannot be fulfilled on works.work.karmada.io \"pyy-f69cf8d6c\": the object has been modified; please apply your changes to the latest version and try again" controller="work" controllerGroup="work.karmada.io" controllerKind="Work" Work="karmada-es-member1/pyy-f69cf8d6c" namespace="karmada-es-member1" name="pyy-f69cf8d6c" reconcileID="b9f3450f-4423-4ddc-be63-d264efaf60f3"
I0116 10:12:56.304183       1 cluster_status_controller.go:125] Syncing cluster status: member1

Environment:

• Karmada version: v1.11.3
• Kubernetes version:
• Others:

[RainbowMango]

E0116 10:12:55.175537 1 execution_controller.go:120] Failed to update work condition type Dispatching. err is Operation cannot be fulfilled on works.work.karmada.io "pyy-f69cf8d6c": the object has been modified; please apply your changes to the latest version and try again
E0116 10:12:55.175604 1 controller.go:324] "Reconciler error" err="Operation cannot be fulfilled on works.work.karmada.io "pyy-f69cf8d6c": the object has been modified; please apply your changes to the latest version and try again" controller="work" controllerGroup="work.karmada.io" controllerKind="Work" Work="karmada-es-member1/pyy-f69cf8d6c" namespace="karmada-es-member1" name="pyy-f69cf8d6c" reconcileID="b9f3450f-4423-4ddc-be63-d264efaf60f3"

I can see there are two errors saying that the object has been modified, I can't see anything about the the HTTP protocol.

The agent controller will retry the updation until it succeeds. No worries.

[Heylosky]

I can see there are two errors saying that the object has been modified, I can't see anything about the the HTTP protocol.

The agent controller will retry the updation until it succeeds. No worries.

I analyze this problem by checking a network device logs instead of the agent logs. When the request protocol recorded by the network device is HTTP2.0, everything is normal, but if it is HTTP1.1, a large number of PUT request 409 errors will appear (corresponding to the object has been modified log above, but in fact the object has not been modified, and it will fail even if it is retried multiple times).

And the biggest problem is that the agent cannot synchronize changes on the karmada apiserver in time. For example, I created a namespace on karmada, and the corresponding work was successfully created, but it took 5-30 minutes to create the namespace on the member cluster. All the work of this member cluster cannot be obtained by the agent normally, and the work status cannot be updated normally. This problem only not occur in a very short time after the agent is restarted. During the entire problem phase, karmada did not provide any valid errors or actions.

So I think disabling http2 might solve my problem? I think it's a network-caused issue, so I don't know if karmada should support this.

[RainbowMango]

So I think disabling http2 might solve my problem? I think it's a network-caused issue, so I don't know if karmada should support this.

It depends on whether it is based on a real-world use case, if it's a common case, etc.

In some strict production environments or through complex network devices, h2 protocol is not supported.

We need more detailed information.

[Heylosky]

We need more detailed information.

I think the agent should be robust and adaptable to different network requirements. In my use case, network traffic needs to be forwarded by 7-layer nginx and inspected by 7-layer security devices, which leads to the protocol change from http2 to 1.1. The agent may be deployed to various remote environments, otherwise there is no reason to use pull mode.

Anyway, in this case where the agent is almost unavailable, the member cluster status is still READY, and the agent does not automatically restart or report an error. I think this needs to be resolved.

[Heylosky]

I can provide an nginx configuration to simulate my current situation.

server {
    listen 443 ssl http2;
    server_name <nginx>;
    location / {
        proxy_pass https://To-nginx:2443;
        proxy_http_version 1.1;
    }
}

server {
    listen 2443 ssl;
    server_name <nginx>;
    location / {
        proxy_pass https://To-karmda-apiserver;
    }
}