Support SNI preread module and routing of TCP traffic with TransportServer

[brianehlert]

SNI based routing of Layer 4 traffic is a way to support customers using DNS names for TCP traffic and support routing based on the SNI header.
With NGINX this is implemented using the stream ssl pre-read module.
https://nginx.org/en/docs/stream/ngx_stream_ssl_preread_module.html

This module is already present in the NGINX Plus binary.

Today, this is possible with heavy use of snippets. The ask is to make this present and first class with the TransportServer resource.

This also historically described here:
https://stackoverflow.com/questions/34741571/nginx-tcp-forwarding-based-on-hostname

There are some additional considerations that need to be included here:

• The possibility for greater flexibility with TransportServer
• Extending this with the concept of a TranspotServerRoute (similar to the VS/VSR relationship)
• Creating theoretical simplicity to role separation and easy to provide controls over the hostname versus the backend definition
  This is conceptually no different than the suggestion to make the VS/VSR relationship easier to implement like Ingress master/minion but with some level of security controls like Gateway API ReferenceGrant.

The overall concept is multiple upstream targets for TCP behind a single listener and to route based on SNI.
This would support both TLS Passthrough as well as advanced programmability that might require TLS decryption and re-encryption.

To bring this all together:

• TransportServer would include the option for a top level domain type SNI match ( foo.com )
• Following a concept similar to an http path or nginx location - the extension of a service specific route that matches to the backend service (service.foo.com)
  this would be available in the TransportServer or possible with a TransportServerRoute 'attaching' to a TransportServer
  This is the same relationship pattern with think about with Ingress master/minion or with VirtualServer/VirtualServerRoute
• This would support TLS passthrough and decrypted TLS (it would use preread)

TLS traffic in -> TransportServer matches TLD of hostheader -> TransportServerRoute matches server.TLD of host header and defines upstream.

[jasonwilliams14]

We will need to use the ngx_stream_ssl_preread_module:
https://nginx.org/en/docs/stream/ngx_stream_ssl_preread_module.html

This allows us to extract the information from the clienthello message and not terminate SSL/TLS. We can grab the server name through SNI.

We would need a flag/toggle of some sort to enable ssl_preread to on:

A simple example of how to configure this in NGINX:

not terminating and doing TLS passthrough:

stream {  

  map $ssl_preread_server_name $dynamicbackend {
    coffee.cafe.com   upstream1.example.com:443;
    tea.cafe.com      upstream2.example.com:443;
  }   
 
  server {
    listen 443;     ### since `ssl` is not present, NGINX does not terminate
        
    proxy_connect_timeout 1s;
    proxy_timeout 3s;
    resolver x.x.x.x;
    
    proxy_pass $dynamicbackend;       
    ssl_preread on;
  }
}

terminating TLS and passing to upstream:

stream {  

  map $ssl_preread_server_name $dynamicbackend {
    coffee.cafe.com   upstream1.example.com:443;
    tea.cafe.com      upstream2.example.com:443;
  }   
 
  server {
    listen 443 ssl; 
     
    ssl_protocols           TLSv1.2;
    ssl_certificate          /path/to/cert;
    ssl_certificate_key   /path/tokey;   
    
    proxy_connect_timeout 1s;
    proxy_timeout 3s;
    resolver x.x.x.x;
    
    proxy_pass $dynamicbackend;       

  }
}

I can do some mockups for the transportServer resource and what that would look, but this is a quick idea of what needs to be brought in.

[brianehlert]

If I am following, does this support TLS passthrough? Since TLS decryption is not required?
So, this could be used both with and without TLS decryption. Did I follow that properly?

[jasonwilliams14]

The first one does not do any decrypt, just passes through. We basically look at the ALPN, map it to a backend and then proxy_pass it.

The second one does do a decrypt.

Just purely examples and things we can think about.

[chrisakker]

You will need 4 changes, between the two configurations:

1. listen directive is different ( with/without ssl parameter ).
   2-3. ssl_cert and ssl_key directives are omitted or declared
2. ssl_preread directive is omitted or declared.

[j1m-ryan]

Can we use server_name for this? https://nginx.org/en/docs/stream/ngx_stream_core_module.html#server_name

[jasonwilliams14]

@j1m-ryan currently, that feature was released in OSS 1.25.5 and not yet available in NGINX Plus.

NGINX Plus is currently on R31 which is based of OSS 1.25.3
https://docs.nginx.com/nginx/releases/#nginxplusrelease-31-r31

Once R32 releases, it should have this directive available to use and we could explore that as an option/addition to the use case requirements above.

[brianehlert]

As I think about this from a resource design perspective I envision something a bit different from the VS/VSR pattern, and nearly its inverse.

A TransportServer resource would link to the listener (following the listener enhancements that have been in the works).
A TransportServerRoute (or TransportServerSNI) would define the individual SNI matches and relating backend service.

Thoughts?

[tkam8]

I agree. This will enable the NetOps persona to manage the TransportServer and listeners, while AppDev can deploy the application specific routing configuration with TransportServerRoute/TransportServerSNI.

Additionally, in cases where NetOps manages both TS and listeners defined in GlobalConfiguration, why not just have the listener be configurable from the TransportServer?

[brianehlert]

Dynamically adding listeners based on TransportServer / VirtualServer / Master Ingress opens those things to frequent change.
Adding / modifying listeners can be reloads or restarts. So, for one thing, it is potentially impactful.
Also, it requires the NIC deployment owner to define the mapped ports. Even if customers define a range, most want to control possible open ports.
I would consider all of that additional work from supporting SNI, which is the scope of this proposal.