Spring adding 'chunked' transfer encoding even if this header already exists [SPR-16985]

[spring-projects-issues]

Pradeep Gadi opened SPR-16985 and commented

In our Middleware project we forward rest response from another service as it is to angular UI. The response we are getting to the Middleware already having transfer-encoding →chunked header. while returning the same response from our middleware to angular UI spring adding transfer-encoding again. see below response

content-type →application/json;charset=UTF-8
date →Fri, 29 Jun 2018 07:02:06 GMT
transfer-encoding →chunked, chunked

 

Because of this F5 not able to send the response properly. Spring should not add transfer-encoding If this header already present in the response.

 

---

Affects: 5.0.5

Issue Links:

• HTTP Response should not contain both Transfer-Encoding and Content-Length headers [SPR-15212] #19776 HTTP Response should not contain both Transfer-Encoding and Content-Length headers

[spring-projects-issues]

Juergen Hoeller commented

From a quick review, I can't find any place where we explicitly set the "Transfer-Encoding" header in Spring's processing, in particular not to "chunked". Does this maybe come through the HTTP server or some HTTP client library here?

Brian Clozel, assigning this to you for further consideration...

[spring-projects-issues]

Pradeep Gadi commented

I have added a filter and tested the same, this issue is not because of the Httpserver. I am not sure how to figure it out whether the issue is with HTTP client or Spring. Could you please provide me any suggestion on how to proceed with this issue.

[spring-projects-issues]

Brian Clozel commented

Hi Pradeep Gadi,

Could you provide us with a sample application? I can't seem to reproduce this at the moment.

[BSchwetzel]

We stumbled upon this just yesterday and opened an issue against traefik because of this (traefik/traefik#8060). Basically what we do is using a Spring Boot application as a proxy, as we don't want to deal with CORS and Same-Origin-Policy and all that stuff. Thankfully, our test case for this is rather simple:

package org.my-organization.test;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.ResponseEntity;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.client.RestTemplate;

@Controller
@RequestMapping("/status")
public class StatusController {

    private RestTemplate restTemplate;

    @Autowired
    public StatusController(RestTemplate restTemplate) {
        this.restTemplate = restTemplate;
    }

    @GetMapping(value = "/test")
    public ResponseEntity getStatus() {
        return restTemplate.getForEntity("http://another-spring-boot-service:8080/actuator/health", String.class);
    }
}

Now if we curl that controller:

$  curl -v http://localhost:8080/status/test
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying ::1:8080...
* Connected to localhost (::1) port 8080 (#0)
> GET /status/test HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.75.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Expires: 0
< X-Frame-Options: DENY
< Transfer-Encoding: chunked
< Date: Wed, 14 Apr 2021 08:14:11 GMT
< Keep-Alive: timeout=60
< Connection: keep-alive
< Content-Type: application/json
< Transfer-Encoding: chunked
<
{ [20 bytes data]
100    15    0    15    0     0     78      0 --:--:-- --:--:-- --:--:--    78{"status":"UP"}
* Connection #0 to host localhost left intact

As you can see we have Transfer-Encoding present twice, which breaks traefik (by breaking golang's net/http it seems: https://github.com/golang/go/blob/59bfc18e3441d9cd0b1b2f302935403bbf52ac8b/src/net/http/transfer.go#L634-L637)

We are using Spring Boot 2.4.4 / Spring 5.3.5

[bclozel]

Hi @BSchwetzel

In the meantime, we received out-of-band feedback on issues similar to yours. I'll try to summarize our findings in this comment.

As far as I understand, golang introduced this restriction in the http layer to prevent HTTP smuggling attacks - but this can only apply to requests received by servers/proxies, not responses sent by servers. So while this change is enforcing a new restriction, it's not meant to protect you here. This change not only applies to traefik, but to many proxies/servers using golang.

We've only found that problem in Spring applications with the exact use case you're describing: a controller uses an HTTP client to retrieve a response and returns it as the response of the controller handler method without any modification. We consider this use case as problematic for several reasons:

1. Spring provides a way to set response headers using ResponseEntity, but does not enforce HTTP validation here, since it's the job of the web server. Some server implementations are more lenient than others and might reject or ignore combinations. In this case, servers like Tomcat or Netty might choose to enforce stricter rules as well in the future (see Refuse adding invalid HTTP 2.0 headers apache/tomcat#277 and https://bz.apache.org/bugzilla/show_bug.cgi?id=64144, for example). Some Servlet containers are already "fixing" the response for you if it contains both Transfer-Encoding and Content-Length.

2. This is not specific to Spring, as you could do the same thing with the HTTP Servlet response directly - in that case, Spring would have no way of enforcing rules or cleaning the response.

3. Applications are not in a position to set the Transfer-Encoding response header because it has not the necessary visibility to set that correctly, nor the ability to then send a correctly formatted chunked body.

So while sending duplicate chunked values are illegal anyway, we don't think that Spring is in a position to fix that. It's up to servers to decide what they want to do with this.

Now we've had similar discussions with developers using the same usage pattern (controllers proxying requests). The chunked header value is a symptom of a much bigger problem: copying headers in and out can cause many HTTP and security issues. HTTP headers can carry information about the state of the connection, security restrictions or options negotiated between client and server. This implementation is a typical example of possible HTTP smuggling vulnerabilities and I'd encourage you to consider this in your application.

At a minimum, headers should be properly filtered (in and out). I understand that we're probably looking at a very restricted use case, but implementing a general purpose proxy/gateway is not easy and there are many more problems with that. Maybe Spring Cloud Gateway can be helpful.

[anwarmd]

We are also seeing same issue, "Transfer-Encoding: chunked" in response and using following SPring boot version 2.5.2.

1. How you fixed this issue and what Filter used?
2. Any one tried out Spring Cloud Gateway and is it working without chunked issue?

[bclozel]

@anwarmd this issue is about getting duplicate Transfer-Encoding: chucked response headers. If you're seeing this problem, this is very likely due to your application setting those headers directly in a ResponseEntity returned by a controller: this should not be done by the application, you'll need to fix the problem in your codebase.

[anwarmd]

@bclozel Can you please suggest me on

1. How can we remove one of the header ?
2. Can we override any filter to remove one of the headers ?
3. We might be getting from downstream Services and any period of time we would like to make it one encoding ?
4. What context does spring add encoding ?

[Hollerweger]

@bclozel Is a received ResponseEntity not meant to be directly returned again in a Controller?

For most usecases creating a new ResponseEntity should solve the issue (when original Headers are not required at all):
return ResponseEntity.status(responseEntity.getStatusCode()).body(responseEntity.getBody())

[bclozel]

@Hollerweger Proxies do more than just copying HTTP responses. There are many things to consider, like compression, security headers, caching, content-neogtiation, Forward headers and more. So in general, I don't think that they're meant to be returned directly from a Controller.

[NameFILIP]

Should this issue be reopened? It's still happening for us with Spring 2.7.2

[bclozel]

@NameFILIP we have assessed that this is not a bug in Spring. See #21523 (comment)

If this does not work for you, please create a new issue with a minimal sample application showing the problem.

[xunzhaoderen]

i found prepareResponse() function in tomcat-embed-core package, Http11Processor.java file,
use headers.addValue(Constants.TRANSFERENCODING).setString(Constants.CHUNKED) to add duplicate chunked, not checking if already exists, if it is reasonable ?

[bclozel]

@xunzhaoderen Tomcat is responsible for setting that header because it knows how the body is being written. As said in my previous comment, application components are not in a position to set this header. Guarding against a possible duplicate in Tomcat would not help - maybe the response is not chunked at all and writing this header is invalid anyway?

If your application is adding this header, this should be fixed in your application directly.

[xunzhaoderen]

the response chucked header is passed from another http interface, can I directly filter transfer-encoding header ?

[VariabileAleatoria]

completely wrong

[snowymike]

posting this in a couple spring issues on this topic in the hopes it helps others. the best writeup of this problem i've encountered is https://knowledge.broadcom.com/external/article/298108/too-many-transfer-encodings-502-chunked.html. Trigger number 3 in that article was our problem

In Spring-based microservice applications, when using RestTemplate from a Controller annotated class and returning the result of RestTemplate.exchange in a method that returns a ResponseEntity. Similar to situation 1, this will cause a problem if the microservice to which you are talking returns a transfer-encoding header because RestTemplate.exchange will copy all of the response headers from the client response into the ResponseEntity it creates, including the transfer-encoding header.

@Hollerweger 's comment above in #21523 (comment) provides an alternative that avoided this problem for us.

For most usecases creating a new ResponseEntity should solve the issue (when original Headers are not required at all):
return ResponseEntity.status(responseEntity.getStatusCode()).body(responseEntity.getBody())

[npschoo]

I am also affected by the "Trigger number 3" mentioned above.
This issue seems to be much worse with HTTP/2 headers.
The :status pseudo-header (among others) is duplicated, which causes issues in Chrome. (ERR_HTTP2_PROTOCOL_ERROR)

Hoping for another solution rather than refactoring tons of Controller methods.
Perhaps there's some way to prevent the duplicate headers app-wide with a single Filter?