reopen: more_set_input_headers/more_clear_input_headers cannot set header "Host"

[cateiru]

Hi!

I am currently suffering from this issue.

• more_set_input_headers/more_clear_input_headers cannot set header "Host" #109

It seems to be closed, but I think it has not been resolved.

Is there a solution or workaround?

[sevmonster]

With no context, this is probably because proxy_set_header defaults to Host: $proxy_host and Connection: close. Try clearing the proxy header array (e.g. proxy_set_header X-dummy '')

[kbolino]

@sevmonster that doesn't seem to make any difference

I think Host is handled specially by nginx, the same as with Connection. The docs imply that the default is equivalent to the following directives:

proxy_set_header Host $proxy_host;
proxy_set_header Connection close;

However, I don't think that this is literally how it works. These two headers cannot be affected by more_set_input_headers no matter what I tried. Curiously enough, you can delete them by doing the following:

proxy_set_header Host '';
proxy_set_header Connection '';

But nevertheless, this still does not allow more_set_input_headers to do its thing with either of these two.

I think the documentation for this module needs to be updated. Neither Host nor Connection can be altered in any way by more_set_input_headers, at least not without modifying nginx itself.

[kbolino]

@cateiru In my experience, the best workaround is:

• Use proxy_set_header exclusively for Host and Connection and if you need to set both, be sure to set them at the same level in the same block
• Use more_set_input_headers for all other headers

This avoids the pitfalls of both directives.

[sevmonster]

I think Host is handled specially by nginx

The Connection header is set by the filter module as mentioned in the project README, while the Host header is first read very early and calculated for the proxy here. The proxy module calculates its headers while processing the proxy_pass directive here.

I am no expert in nginx internals or the execution order of its various components, so take that as you will and rely your own research. But last I more thoroughly investigated, it did work—the proxy headers, as set by the proxy module, were overridden by the headers_more module before sending the request to the proxy.

[kbolino]

I figured some code was worth more than words, so I set up a simple test.

Running this Python server as the proxy_pass target, which simply echoes the headers back:

# http-echo-headers.py

from http.server import BaseHTTPRequestHandler, HTTPServer, HTTPStatus

class EchoHeadersHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(HTTPStatus.OK)
        self.send_header("Content-Type", "text/plain; charset=UTF-8")
        self.end_headers()
        for key, value in self.headers.items():
            self.wfile.write(f"{key}: {value}\n".encode("utf_8"))

if __name__ == "__main__":
    server_address = ("", 8000)
    httpd = HTTPServer(server_address, EchoHeadersHandler)
    httpd.serve_forever()

Using the following configuration as /etc/nginx/conf.d/default.conf, which provides some different config scenarios under different server names:

# conf/default.conf

server {
	listen 8080 default_server;
	server_name _;
	location / {
		return 404;
	}
}

server {
	listen 8080;
	server_name more-set-host.example;
	location / {
		more_set_input_headers "Host: $host";
		proxy_pass http://localhost:8000;
	}
}

server {
	listen 8080;
	server_name proxy-set-host.example;
	location / {
		proxy_set_header Host $host;
		proxy_pass http://localhost:8000;
	}
}

server {
	listen 8080;
	server_name dummy-header.example;
	location / {
		proxy_set_header X-Dummy "";
		more_set_input_headers "Host: $host";
		proxy_pass http://localhost:8000;
	}
}

server {
	listen 8080;
	server_name delete-host.example;
	location / {
		proxy_set_header Host "";
		more_set_input_headers "Host: $host";
		proxy_pass http://localhost:8000;
	}
}

With the openresty docker image running on the host:

$ docker pull openresty/openresty
Using default tag: latest
latest: Pulling from openresty/openresty
1efc276f4ff9: Pull complete 
403035b88bb5: Pull complete 
4e8d5f0d1a02: Pull complete 
aa6116de5cfa: Pull complete 
Digest: sha256:168033f0b908546f874e1ccfcb196f39e7e0ab30562e20e35cf54252c3fa0698
Status: Downloaded newer image for openresty/openresty:latest
docker.io/openresty/openresty:latest
$ docker run docker.io/openresty/openresty:latest openresty -v
nginx version: openresty/1.21.4.1
$ docker run -it --network=host -v $PWD/conf:/etc/nginx/conf.d:ro docker.io/openresty/openresty:latest

And with a shell script to run tests against each of the defined servers, using --resolve to synthesize DNS lookups:

# run-curl-tests.sh

for http_host in \
	more-set-host.example \
	proxy-set-host.example \
	dummy-header.example \
	delete-host.example
do
	echo REQUESTING $http_host
	curl --resolve ${http_host}:8080:127.0.0.1 http://${http_host}:8080/
	echo DONE
	echo
done

Finally, this all yields the following results:

$ bash run-curl-tests.sh 
REQUESTING more-set-host.example
Host: localhost:8000
Connection: close
User-Agent: curl/7.84.0
Accept: */*
DONE

REQUESTING proxy-set-host.example
Host: proxy-set-host.example
Connection: close
User-Agent: curl/7.84.0
Accept: */*
DONE

REQUESTING dummy-header.example
Host: localhost:8000
Connection: close
User-Agent: curl/7.84.0
Accept: */*
DONE

REQUESTING delete-host.example
Connection: close
User-Agent: curl/7.84.0
Accept: */*
DONE

[arianf]

Yea, I am able to reproduce what @kbolino showed above. My entire purpose for using more_set_headers, is so I can inherit the ones set at the server level.

server {
  listen 8080;
  server_name localhost;

  proxy_set_header 'Inherit' 'Me';

  location / {
    proxy_pass http://localhost:8000;
  }

  location /api/1 {
    more_set_headers "Host: override-api-1.host.com";
    proxy_pass http://localhost:8000;
  }

  location /api/2 {
    proxy_set_header Host override-api-2.host.com;
    proxy_pass http://localhost:8000;
  }
}

My goal here is to inherit the server level headers and add my own host header within each location block.

curl localhost:8080/api/1                     
Inherit: Me
Host: localhost:8000
Connection: close
User-Agent: curl/7.84.0
Accept: */*

curl localhost:8080/api/2                     
Host: override-api-2.host.com
Connection: close
User-Agent: curl/7.84.0
Accept: */*

curl localhost:8080/     
Inherit: Me
Host: localhost:8000
Connection: close
User-Agent: curl/7.84.0
Accept: */*