Add option to not lowercase headers

[hissssst]

Current behaviour:
Headers are always translated to lowercase

Desired behaviour:
Have connection option to not translate request and response headers to lowercase in HTTP 1.x

This option would be useful to trick passive http-client fingerprinting
I can implement the PR

[whatyouhide]

I am open to supporting this at the connection or at the request level, I think. @ericmj thoughts?

[ericmj]

If the only use case is avoid fingerprinting I am not sure it is worth it. There are likely plenty of other ways to detect mint and it does not seem feasible to avoid all of them.

[chulkilee]

Related discussion: #172

Things to clarify

• request headers / response headers / both headers
• it only applies to HTTP 1.x (since HTTP 2 requires lowercase header)

[hissssst]

@ericmj
You may be right but as far as I know, HTTP 1.x fingerprinting (in p0f or nmap) is done by

1. User-Agent header
2. Order of headers
3. Case of headers
4. Special headers (e.g DNT)

[ericmj]

We send some headers in the same order as well.

Can you explain your use-case for wanting to avoid finger printing?

[hissssst]

@ericmj
Some big social media websites do fingerprinting to distinguish mobile agents from pc agents. For example https://vk.com blocks some features for mobile agents. I am trying to write crawler for this website and I am always being redirected to the mobile version with crawler on mint. Raw netcat requests work fine and I think the problem is with user-agent fingerprinting

[whatyouhide]

You can customize the user-agent header. Did you try that?

[hissssst]

@whatyouhide
Yes I've tried. I've copy-pasted request from firefox
Headers in code

  [
    {"User-Agent", "Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0"},
    {"Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"},
    {"Accept-Language", "en-US,en;q=0.5"},
    {"Connection", "keep-alive"},
    {"Referer", "https://vk.com/feed"},
    {"Cookie", "cookies"},
    {"Upgrade-Insecure-Requests", "1"},
    {"Sec-Fetch-Dest", "iframe"},
    {"Sec-Fetch-Mode", "navigate"},
    {"Sec-Fetch-Site", "same-origin"},
    {"TE", "Trailers"}
  ]

received request

GET / HTTP/1.1
host: localhost:4000
user-agent: Mozilla/5.0 (Windows NT 10.0; rv:68.0) Gecko/20100101 Firefox/68.0
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
accept-language: en-US,en;q=0.5
connection: keep-alive
referer: https://vk.com/feed
cookie: cookies
upgrade-insecure-requests: 1
sec-fetch-dest: iframe
sec-fetch-mode: navigate
sec-fetch-site: same-origin
te: Trailers

[whatyouhide]

So we don't change the order of headers and set the user-agent correctly, so it seems like the only difference would be the downcasing.

[ericmj]

We have a specific order of headers when mint sets them itself. For example: host, content-length, transfer-encoding etc.

@hissssst I am hesitant to make these kind of changes because it will always be a back and forth between services finding new ways of detecting clients and clients trying to circumvent them. We still don't know if the changes you suggested will actually prevent vk.com from detecting mint.

[hissssst]

@ericmj

specific order of headers

That is right but these headers can be set explicitly

I understand you hesitation but I already have the working fork which prooves my words. I also argee that this option would be useless for 99% of users.
I think that this issue can be closed

[JacquiManzi]

I also agree that this option would be useless for 99% of users.

Just chiming in to add that if someone switches to Mint from a library that previously did not modify the casing of headers, they will inadvertently break backwards compatibility for users that might be relying on headers being case-insensitive. This could have serious implications for anyone switching to Mint that might not be expecting lower-case headers for HTTP1.