update to latest kafka-manager (kafka.common.KafkaException: Unknown offset schema version 3 with kafka 2.1.1)

[blinkeye]

Using kafka version 2.1.1 I see a lot of warn in the kafka-manager after activating it with:

kubectl apply -f yahoo-kafka-manager/

The warnings are:

[warn] k.m.a.c.KafkaManagedOffsetCache - Failed to process a message from offset topic on cluster microk8s!
kafka.common.KafkaException: Unknown offset schema version 3
	at kafka.manager.utils.one10.GroupMetadataManager$.schemaForOffset(GroupMetadataManager.scala:428) ~[kafka-manager.kafka-manager-1.3.3.18-sans-externalized.jar:na]
	at kafka.manager.utils.one10.GroupMetadataManager$.readOffsetMessageValue(GroupMetadataManager.scala:532) ~[kafka-manager.kafka-manager-1.3.3.18-sans-externalized.jar:na]
	at kafka.manager.actor.cluster.KafkaManagedOffsetCache$$anonfun$run$4.apply(KafkaStateActor.scala:332) [kafka-manager.kafka-manager-1.3.3.18-sans-externalized.jar:na]
	at kafka.manager.actor.cluster.KafkaManagedOffsetCache$$anonfun$run$4.apply(KafkaStateActor.scala:308) [kafka-manager.kafka-manager-1.3.3.18-sans-externalized.jar:na]
	at scala.util.Success.foreach(Try.scala:236) [org.scala-lang.scala-library-2.11.12.jar:na]
	at kafka.manager.actor.cluster.KafkaManagedOffsetCache.run(KafkaStateActor.scala:308) [kafka-manager.kafka-manager-1.3.3.18-sans-externalized.jar:na]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144]

I think we need the latest version from https://github.com/yahoo/kafka-manager (or at least a version which contains changes from yahoo/CMAK#620). Unfortunately I could not find a pre-built image: solsson/kafka-manager image. I did try to build it myself but got an error. @solsson: do you plan to update to your latest kafka-manager version?

I see that the latest kafka-manager would also support the latest kafka version (2.2.0), see yahoo/CMAK#626

[solsson]

I get the build error too. They use Scala and I can't find any official-looking image with sbt. Needs more research. I guess we can build from yahoo/CMAK#626 until it's merged.

[blinkeye]

Could the problem from your Dockerfile be the same as reported in the official repo: yahoo/CMAK#565? e.g. Java must not be >8?

[blinkeye]

I could successfully build the latest yahoo/kafka-manager by downgrading to openjdk:8u212, see PR solsson/dockerfiles#24

[solsson]

Yes it looks that way. The build is incredibly heavy though so docker hub will probably fail to build. I'll build locally and push.