Consider using NIO libraries for UDP

[scarytom]

On the dogstatsd fork of this project, some performance improvements were made in this pull request: indeedeng@5d34f15

Consider porting these changes into the java-statsd-client. I'm not actually sure that the NIO UDP client affords any benefit, so worth doing some reading first. The batching of messages so as to fit as many as possible into the MTU will relate to #15

[scarytom]

Also, take a look at finn-no@5b0e893 and evaluate the lmax-disruptor

[scarytom]

See also sgp@fa0b0b1

[michaelsembwever]

Yes, this has been implemented in finn-no@5d34f15#diff-d41d8cd98f00b204e9800998ecf8427e

we've been using this in production at Norway's busiest website for 5 months now.

[scarytom]

I'm not sure I want to introduce a dependency on a 3rd party library like the lmax-disruptor, but moving to NIO, and batching per #15, does seem compelling.

[aklossrbh]

Under heavy stress testing, we noticed this synchronization in java.nio.DatagramChannel which pretty significantly reduced our throughput:

Aug  5 12:26:09 load1-services-0 java: "scheduling-akka.actor.default-dispatcher-92" prio=10 tid=0x00007f0a180e8800 nid=0x7e30 waiting for monitor entry [0x00007f09abf80000]
Aug  5 12:26:09 load1-services-0 java: java.lang.Thread.State: BLOCKED (on object monitor)
Aug  5 12:26:09 load1-services-0 java: at sun.nio.ch.DatagramChannelImpl.send(DatagramChannelImpl.java:427)
Aug  5 12:26:09 load1-services-0 java: - waiting to lock <0x000000078151a408> (a java.lang.Object)
Aug  5 12:26:09 load1-services-0 java: at com.timgroup.statsd.BlockingStatsDClient.send(BlockingStatsDClient.java:527)
Aug  5 12:26:09 load1-services-0 java: at com.timgroup.statsd.BlockingStatsDClient.recordExecutionTime(BlockingStatsDClient.java:358)
Aug  5 12:26:09 load1-services-0 java: at com.redbrickhealth.api.util.metrics.reporter.StatsDMetricReporter$$anonfun$reportEnd$1.apply(StatsDMetricReporter.scala:127)
Aug  5 12:26:09 load1-services-0 java: at com.redbrickhealth.api.util.metrics.reporter.StatsDMetricReporter$$anonfun$reportEnd$1.apply(StatsDMetricReporter.scala:115)
Aug  5 12:26:09 load1-services-0 java: at scala.Option.foreach(Option.scala:257)
Aug  5 12:26:09 load1-services-0 java: at com.redbrickhealth.api.util.metrics.reporter.StatsDMetricReporter.reportEnd(StatsDMetricReporter.scala:115)

We're still evaluating options, but I'm curious if PRs for this are at all likely to be accepted, or should we just enjoy life with a forked driver?