Kafka JVM options for production

[dazza-codes]

See an authoritative comment from Todd Palino in 2014:

• https://mail-archives.apache.org/mod_mbox/kafka-users/201405.mbox/%3CCAOeJiJgnT2c9tZzCRjUaec4V8Lp-a+mcN+RM0Jt1-3oG9HsmEA@mail.gmail.com%3E

To quote Todd, in a thread about Kafka JVM settings for PROD, with Jay Kreps:

On Mon, May 5, 2014 at 10:36 PM, Todd Palino <[email protected]> wrote:

> I apologize for taking a couple days to jump in on this. We¹re currently
> running JDK 1.7 u51, and we¹ve switched over to the G1 collector. If you
> do this (and I highly recommend it), make sure you¹re on u51. We tried out
> u21 in testing, but we had a number of problems with the GC implementation
> in that version.
>
> Our tuning looks like this:
> -Xms4g -Xmx4g -XX:PermSize=48m -XX:MaxPermSize=48m -XX:+UseG1GC
> -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35
>
> For reference, I¹m looking at the stats on one of our busiest clusters (at
> peak):
>         - 15 brokers
>         - 15.5k partitions (replication factor 2)
>
>         - 400k messages/sec in
>         - 70 MB/sec inbound, 400 MB/sec+ outbound
>
> The tuning looks fairly aggressive, but all of our brokers in that cluster
> have a 90% GC pause time of about 21ms, and they¹re doing less than 1
> young GC per second. We haven¹t seen a single full GC on those brokers in
> the last month, and previous to that I think we only saw them when I was
> messing around with the cluster in a very painful way, not under anything
> approaching normal traffic.
>
> -Todd

Also,

Just about all of our systems are actually running 64GB of memory now, and
yes, we want plenty of space for cache with all the disk and network IO
going on.

 -Todd