Topic creator is too brittle during new cluster construction

[otterley]

I'm building an automated test suite for some broker maintenance automation involving multiple brokers. As part of this test suite, I'm building a three-broker Kafka cluster, and desire to create a topic with a replication factor of three.

Since the brokers are started by a container scheduler, they're not necessarily all started at the same time. However, the topic creator script is run in the background immediately after a broker is started. If there aren't yet enough brokers in the cluster, the topic creator will fail hard with:

creating topics: test:1:3
Error while executing topic command : Replication factor: 3 larger than available brokers: 2.

What I'd like to happen is for the topic-creator to wait for the broker count to reach the desired replication factor before attempting to create the topic, or to retry until successful.

[sscaling]

I'm building an automated test suite for some broker maintenance automation involving multiple brokers.

I would have thought the broker maintenance automation should deal with topic creation / deletion / alteration?

Even so - there is some merit in making this less brittle, but on the other hand, for production use-cases it is likely you would want to manage topics outside of the create topics helper script. There are issues such as not being able to co-ordinate each broker trying to create topics concurrently which creates un-necessary errors.

[sscaling]

Related to (very) old PR #160

[masoncj]

Just came here to say I ran into this recently in a single broker setup. Is there a suggested path forward to avoid this error where topic creation happens before broker has completely started up?