Getting "upstream sent duplicate header line" error (502)

[JanCizmar]

Hey!

We're usingnginx-ingress-0.18.1 as an ingress controller for Spring Boot app.

After some time of running the backend pods, we start to get 502 responses with following error.

2023/10/01 18:15:31 [error] 1646#1646: *2862237 upstream sent duplicate header line: "Transfer-Encoding: chunked", previous value: "Transfer-Encoding: chunked" while reading response header from upstream, client: xxx, server: xxx, request: "GET xxx HTTP/1.1", upstream: "http://10.224.1.48:8080/xxx", host: "xxx", referrer: "xxx"

Not sure whether the issue comes from nginx or the spring-boot backend. But I am not sure, why this actually leads to 502 error while the duplicate header might just be ignored. Is there a way how to tell nginx to ignore it when the header values are just the same?

[brianehlert]

upstream sent duplicate header line and while reading response header from upstream
Is the important thing here. That is your backend application as far as NGINX is concerned.
Unless you have something in-line with the application, like a sidecar that is adding the second header to the response or some other proxy pod, it is the application / framework itself.

Your comment of "After some time of running the backend pods, we start to get"...
Do the responses get continually larger?
If this was NGINX doing anything such as a mis-configured add_header - it should be consistent and not appear after any period of time.

Now, I want to spend a bit of time focusing on which NGINX Ingress controller you are using.
"nginx-ingress" tends to be this project. But the version number "0.18.1" can be the community project or it can be the version of the Helm chart of this project you are using. It is difficult to determine.

[JanCizmar]

It's the helm chart. Sorry.

[JanCizmar]

Do the responses get continually larger?

No, they are the same.

[haywoodsh]

Hi @JanCizmar , without knowing how you deployed the upstream, it is difficult to be sure what the problem is. However, you might be interested at the NGINX directive chunked_transfer_encoding http://nginx.org/en/docs/http/ngx_http_core_module.html#chunked_transfer_encoding.
You may want inject a directive to the nginx config of your upstream with snippets, and toggle the chunked_transfer_encoding on and off and see if there are any effects.

[JanCizmar]

I tried to set it off, with no luck.

[jasonwilliams14]

@JanCizmar can you verify that chunked_transfer_encoding is being set to off in nginx.conf (http,server or location) ?
Do you have .yaml deployment files for reveiw as well?

[JanCizmar]

I've checked the generated nginx.conf by investigating the pod container contents. I've set it for location level.

[JanCizmar]

I've also created this issue in SpringBoot, where I've got the recommendation to log the headers on SpringBoot side, so I'll try to do that and get back with updates, when next 502 wave will occur. 🤠

[vepatel]

@JanCizmar is there an update on this?

[JanCizmar]

Nope. I am sure It's not originated from application code. So it might be a Tomcat issue. I am currently in process of upgrading to Srping Boot v3, with a new Tomcat so we will see after the upgrade.