Add ability to remove HTTP response headers in the HTTP connection manager after HTTP filters have executed

[henrymwang]

Title: Add ability to remove HTTP response headers in the HTTP connection manager after HTTP filters have executed

Description:
The HTTP connection manager offers early header request mutations which run before HTTP filters:

• envoy/api/envoy/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto

  Line 678 in 0038ebb

  repeated config.core.v3.TypedExtensionConfig early_header_mutation_extensions = 52;

  .

• envoy/source/common/http/conn_manager_impl.cc

  Line 1372 in 0038ebb

  const auto mutate_result = ConnectionManagerUtility::mutateRequestHeaders(

We have a similar need for stripping response headers after HTTP filters have executed. This would be useful in situations where HTTP filters might add response headers unintentionally that we'd like to ensure are never leaked.

This could be implemented by adding a new field to the HTTP connection manager config and updating encodeHeaders here,

envoy/source/common/http/conn_manager_impl.cc

Line 1788 in 0038ebb

ConnectionManagerUtility::mutateResponseHeaders(

.

We can prepare a patch if this sounds reasonable.

[optional Relevant Links:]

Any extra documentation required to understand the issue.

[zuercher]

I sort of expected response_headers_to_remove to handle this, but I don't think it does (maybe it used to).

cc @alyssawilk

[henrymwang]

I sort of expected response_headers_to_remove to handle this, but I don't think it does (maybe it used to).

For a test: I added a network listener with 2 http filters: health-check filter in no pass through mode, then the http router filter. The http health-check filter adds some headers and uses localReply if it matches the /healthy (responds with a 200). When I curl the health path, it always returns the header x-envoy-upstream-healthchecked-cluster, which is automatically added by the http healthcheck filter.

Because the http router filter is last, it also modifies response headers first so it's too late then. Another option might be adding a filter in front to remove the response headers but it would require extra care about filter ordering. My proposal is to add this ability to unconditionally remove response headers, even ones added by http filters running before the http router.

[agrawroh]

@henrymwang You can have HeaderMutation as the first filter in the chain and specify any headers you'd like to remove from the outgoing response in response_mutations. On the response path, the filter chain is reversed and this filter would be invoked last. This way you won't be depending on the router filter to do the mutations for you as you have correctly identified that any other filters in the chain could mutate the response after the router filter have already executed.

[henrymwang]

You can have HeaderMutation as the first filter in the chain and specify any headers you'd like to remove from the outgoing response in response_mutations. On the response path, the filter chain is reversed and this filter would be invoked last. This way you won't be depending on the router filter to do the mutations for you as you have correctly identified that any other filters in the chain could mutate the response after the router filter have already executed.

I've tried that as well without any luck - I believe that's because this extension only works on the HTTP connection manager's request path and it doesn't apply currently to response headers.

Code snippets:

• envoy/source/common/http/conn_manager_impl.cc

  Line 1372 in 47f99c5

  const auto mutate_result = ConnectionManagerUtility::mutateRequestHeaders(

• envoy/source/common/http/conn_manager_utility.cc

  Line 94 in 47f99c5

  for (const auto& extension : config.earlyHeaderMutationExtensions()) {

[agrawroh]

@henrymwang Could you link your Envoy config?