Exposing CoAP Service with IP preservation

[StevenCyb]

Hi,

since I didn't really have an idea after a few hours, I just wanted to ask for some advice. I have a CoAP server running in AWS EKS. So far everything works with the transportserver pointing at my service. Now I have the requirement that I need the client IP. By default, the IP of the Nginx pod is passed.

I use helm to deploy the Nginx controller.

One idea I had was to add serverSnippets: proxy_bind $remote_addr transparent; to the transportserver and modifying the daemonset to get more permissions:

        securityContext:
          allowPrivilegeEscalation: true
          runAsUser: 101 #nginx
          runAsNonRoot: true
          privileged: true
          capabilities:
            drop:
            - ALL
            add:
            - NET_BIND_SERVICE
            - NET_ADMIN
            - NET_RAW
            - CAP_NET_ADMIN

But still getting the error:

[alert] 155#155: *13120 setsockopt(IP_TRANSPARENT) failed (1: Operation not permitted) while connecting to upstream, udp client: <IP_I_WANT>, server: 0.0.0.0:8000, upstream: "<IP_I_GET>:8000", bytes from/to client:61/0, bytes from/to upstream:0/0

Another idea was to implement a lua plugin that can add the client IP as CoAP option. The problem ist, I can't find any hints on how to include plugins e.g. by mounting them to a specific directory in combination with the helm chart.

Do someone have a idea or hint on how to achieve it?

[vepatel]

Hi @StevenCyb thanks for opening up a discussion, would you mind taking a look at https://github.com/nginxinc/kubernetes-ingress/tree/v3.0.2/examples/shared-examples/proxy-protocol and see if it matches your requirement?

[StevenCyb]

Thanks @vepatel,
I tried to set it on the Helm values.yaml. Unfortunately there was no change. I forgot to mention CoAP is based on UDP.

controller:
  config:
    entries :
      proxy-protocol : "True"
      real-ip-header : "proxy_protocol"
      set-real-ip-from : "0.0.0.0/0"

[jasonwilliams14]

@StevenCyb Hello there.
Since your application is using transportserver, you will need to use the NGINX layer 4 (stream) to add your specific settings. In your above values.yaml file for helm, the values are being set at the Layer 7 (HTTP) context and will not apply to transportserver.
That being said, you will need to use snippets in your values.yaml file to add the appropriate settings (proxy_protocol and set-realip-from:)
Have a look at these links:

https://nginx.org/en/docs/stream/ngx_stream_realip_module.html
https://nginx.org/en/docs/stream/ngx_stream_proxy_module.html

https://docs.nginx.com/nginx-ingress-controller/configuration/global-configuration/configmap-resource/#snippets-and-custom-templates. (stream-snippets)

[StevenCyb]

Sorry for the late reply. I am currently not working on the project. But I will try it as soon as I have time.
Thanks @jasonwilliams14 and @vepatel :)