Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug][Agent] The current MqttSource solution may cause message loss #11175

Closed
2 tasks done
qy-liuhuo opened this issue Sep 22, 2024 · 0 comments · Fixed by #11176
Closed
2 tasks done

[Bug][Agent] The current MqttSource solution may cause message loss #11175

qy-liuhuo opened this issue Sep 22, 2024 · 0 comments · Fixed by #11176
Assignees
Labels
type/bug Something is wrong
Milestone

Comments

@qy-liuhuo
Copy link
Contributor

qy-liuhuo commented Sep 22, 2024

What happened

There is a problem in #10727

The current MqttSource solution may cause message loss

What you expected to happen

@Override
public void messageArrived(String topic, MqttMessage message) throws Exception {
    Map<String, String> headerMap = new HashMap<>();
    headerMap.put("record.topic", topic);
    headerMap.put("record.messageId", String.valueOf(message.getId()));
    headerMap.put("record.qos", String.valueOf(message.getQos()));
    byte[] recordValue = message.getPayload();
    mqttMessagesQueue.offer(new DefaultMessage(recordValue, headerMap), 1, TimeUnit.SECONDS);

}

Due to the timeout mechanism of blocking queue offers(mqttMessagesQueue.offer(new DefaultMessage(recordValue, headerMap), 1, TimeUnit.SECONDS);), messages may be lost when the queue is full

How to reproduce

The mqtt production speed is higher than the consumption speed. When the blocking queue is filled, messages will be lost.

Environment

No response

InLong version

master

InLong Component

InLong Agent

Are you willing to submit PR?

  • Yes, I am willing to submit a PR!

Code of Conduct

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug Something is wrong
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants