Upgrade Kafka client to 2.7

[mbarretta]

The current client is 2.4, which is 18mo old. A number of optimizations have been introduced since (especially 2.5), which can improve the performance of Logstash.

[mbarretta]

Expanding this to include support for Static Membership, which was introduced in Kafka 2.3: https://www.confluent.io/blog/kafka-rebalance-protocol-static-membership/#unnecessary-rebalance (<-- closest anchor, scroll down a bit). We will need a way to set the group.instance.id setting.

[Kami]

Related PR I opened in old logstash-kafka-input repo - logstash-plugins/logstash-input-kafka#341.

[andsel]

The static membership concept is proposed and explained in KIP-345.

When a client of a consumer group reconnects to a cluster it triggers a rebalance, to assign partition to consumers and this consumer is assigned an id named member.id which is randomly generated.
If a client is member of a consumer groups, and reconnects frequently is mean it triggers frequent rebalances.
To avoid this a static member.id could be passed by the consumer, to uniquely identify it inside the consumer group and in stored on both sides (broker and consumer). In this way the static group.instance.id avoids rebalances in case of continuous disconnections and reconnections.
Also with Static Membership however, rebalances could happen when a new member joins the group or a leader re-joins, for a full list of cases where rebalances happen also with static membership, please check KIP-345

[andsel]

This issue contains 2 requests:

• kafka client update
• introduction of static membership feature.

The first is resolved in #131 the second is tracked in #134.

So keep this issue just as "client update" and closing it.