Add Kafka producer configuration #23

Marjori24 · 2024-12-02T20:46:32Z

No description provided.

CLAassistant · 2024-12-02T20:46:40Z

All committers have signed the CLA.

Signed-off-by: Marjori Martinez <[email protected]>

jaogoy · 2025-01-11T00:33:30Z

src/main/java/com/starrocks/plugin/audit/AuditLoaderPlugin.java

@@ -329,6 +341,23 @@ public static class AuditLoaderConf {
        public static final String PROP_SECRET_KEY = "secret_key";
        public String secretKey = "";

+        public static String PROP_KAFKA_ENABLE = "kafka_enableMSK";


Is normal Kafka or Confluent not supported?

Normal Kafka supported.

jaogoy · 2025-01-11T00:48:45Z

src/main/java/com/starrocks/plugin/audit/AuditLoaderPlugin.java

@@ -392,6 +469,9 @@ public void run() {
                    AuditEvent event = auditEventQueue.poll(5, TimeUnit.SECONDS);
                    if (event != null) {
                        assembleAudit(event);
+                        if (kafkaEnable) {
+                            sendToKafka(event);


Will it send a message to kafka when each event arrives? Will it cost FE too resource to process events?

Yes, when each audit event arrives it will be send to the kafka queue. It will not generate extra cost because we are only sending messages to the kafka queue, not doing any extra process to it.

gengjun-git · 2025-01-15T02:35:29Z

@Marjori24 Thank you for your contribution. Could you add the kafka plugin into a sub module in this project.

fe-plugins-auditloader
                                     / sr               // the current plugin 
                                     / kafka         // the new plugin 
                                     / pom.xml   //  common maven dependency.

This will make it easier to maintain

You can refer to starrocks/fe to learn how to build multiple modules.

Marjori24 · 2025-01-20T14:50:36Z

@gengjun-git hi!

Just to confirm, the new structure will be:

/src
/kafka
/lib
pom.xml

Could you share the link to know how to build multiple modules, please?
In this new folder, are you referring to add the whole class or just the "sendToKafka" method? just asking because you quoted the folder as "new plugin"

gengjun-git · 2025-01-22T03:15:00Z

@gengjun-git hi!

Just to confirm, the new structure will be:

/src
/kafka
/lib
pom.xml

Could you share the link to know how to build multiple modules, please?

In this new folder, are you referring to add the whole class or just the "sendToKafka" method? just asking because you quoted the folder as "new plugin"

@Marjori24

The first dir is sr(starrocks), it should include all the files of the current repository
I have not documents to share the maven multi modules, you can search from internet or refer to starrocks/fe, because fe is a multi modules maven project with fe-common, fe-core, hive-udf, plugin-common.
Just the sendToKafka, We should separate the two modules.

Marjori24 added 8 commits December 2, 2024 16:02

Create QueryEventData.java

f59016b

Signed-off-by: Marjori Martinez <[email protected]>

Update AuditLoaderPlugin.java

a4eece0

Signed-off-by: Marjori Martinez <[email protected]>

Update plugin.conf

38da64f

Signed-off-by: Marjori Martinez <[email protected]>

Update pom.xml

a6fb61e

Signed-off-by: Marjori Martinez <[email protected]>

Update plugin.conf

becb033

Signed-off-by: Marjori Martinez <[email protected]>

Update plugin.conf

71106e6

Signed-off-by: Marjori Martinez <[email protected]>

Update AuditLoaderPlugin.java

90a6a26

Signed-off-by: Marjori Martinez <[email protected]>

Update pom.xml

8fee8e5

Signed-off-by: Marjori Martinez <[email protected]>

Marjori24 force-pushed the add-kafka-conf branch from 9c2cdd8 to 8fee8e5 Compare December 2, 2024 21:04

jaogoy reviewed Jan 11, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Kafka producer configuration #23

Add Kafka producer configuration #23

Marjori24 commented Dec 2, 2024

CLAassistant commented Dec 2, 2024 •

edited

Loading

jaogoy Jan 11, 2025

Marjori24 Jan 13, 2025

jaogoy Jan 11, 2025

Marjori24 Jan 13, 2025

gengjun-git commented Jan 15, 2025 •

edited

Loading

Marjori24 commented Jan 20, 2025 •

edited

Loading

gengjun-git commented Jan 22, 2025

Add Kafka producer configuration #23

Are you sure you want to change the base?

Add Kafka producer configuration #23

Conversation

Marjori24 commented Dec 2, 2024

CLAassistant commented Dec 2, 2024 • edited Loading

jaogoy Jan 11, 2025

Choose a reason for hiding this comment

Marjori24 Jan 13, 2025

Choose a reason for hiding this comment

jaogoy Jan 11, 2025

Choose a reason for hiding this comment

Marjori24 Jan 13, 2025

Choose a reason for hiding this comment

gengjun-git commented Jan 15, 2025 • edited Loading

Marjori24 commented Jan 20, 2025 • edited Loading

gengjun-git commented Jan 22, 2025

CLAassistant commented Dec 2, 2024 •

edited

Loading

gengjun-git commented Jan 15, 2025 •

edited

Loading

Marjori24 commented Jan 20, 2025 •

edited

Loading