Support automatic DNS lookup for kafka bootstrap servers

[niebayes]

What type of enhancement is this?

User experience

What does the enhancement do?

Amazon MSK users can only obtain bootstrap server addresses in the form of b-1.niebayesmskcluste.vekjkv.c2.kafka.ap-southeast-1.amazonaws.com:9094. However, greptimedb requires addresses in the form of 127.0.0.1:9094. Therefore, we need to support automatic DNS lookup to obtain available bootstrap server addresses.

Implementation challenges

No response

[tisonkun]

@niebayes, since you marked this issue as a "good first issue," could you share a permalink to the related code?

[niebayes]

Currently, users are required to provide their own broker_endpoints for Kafka wal, or the default endpoint 127.0.0.1:9092 would be used. See the provisioned example configuration files, for e.g.

greptimedb/config/metasrv.example.toml

Lines 57 to 58 in 6628c41

	# The broker endpoints of the Kafka cluster. ["127.0.0.1:9092"] by default.
	# broker_endpoints = ["127.0.0.1:9092"]

However, users may not able to obtain broker endpoints in the form of 127.0.0.1:9092, but in the form of b-1.niebayesmskcluste.vekjkv.c2.kafka.ap-southeast-1.amazonaws.com:9094 which requires DNS lookup.

We want to eliminate the need for users to manually perform DNS lookup and instead perform automatic DNS lookup within greptimedb.

There might have two methods to accomplish this task (Any other solutions are welcome):

• Check broker endpoints whenever we use them. If they're not in the form of 127.0.0.1:9092, perform a DNS lookup. You're required to investigate public crates on DNS lookup and choose an appropriate one.
• Modify the deserialization logic for broker_endpoints to perform an automatic DNS lookup when necessary. You may find the serde crate useful.

The definitions of wal configs reside in this file.

[J0HN50N133]

I would like to offer some help. I've found a simple crate dns-lookup and would like to implement it in the second way.

[tisonkun]

@J0HN50N133 Cool. Go ahead and welcome to ping @niebayes or me as a reviewer when you prepare a patch :D