Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add initial MQTT support #633

Merged
merged 1 commit into from
May 14, 2024
Merged

Add initial MQTT support #633

merged 1 commit into from
May 14, 2024

Conversation

sergiimk
Copy link
Member

@sergiimk sergiimk commented May 12, 2024

Description

Relates to: #368

This PR allows to define a dataset like this:

kind: DatasetSnapshot
version: 1
content:
  name: temp
  kind: Root
  metadata:
    - kind: SetPollingSource
      fetch:
        kind: mqtt
        host: test.mosquitto.org
        port: 1883
        topics:
          - path: dev.kamu.example.mqtt.temp
            qos: AtMostOnce
      read:
        kind: NdJson
        ...

In this case kamu-cli will act as an MQTT client, subscribing to some topic(s).

Upon kamu pull it will connect to the MQTT broker and either:

  • get nothing and exit after some configurable timeout
  • get "retained" MQTT event - e.g. last temperature measurement that broker stores and sends upon subscription
  • get a history of events it missed if MQTT QoS level is == AtLeastOnce or ExactlyOnce which makes broker buffer events for some time when subscriber is offline (the behavior is highly broker-specific)

Technical debt:

  • MQTT breaks polling/push source separation as it's neither - related to IaC configurability for flow schedules and variables kamu-node#57 we will likely replace these metadata events with standalone Source manifest
  • MQTT client will auto-acknowledge the events, so if ingest fails later those events will be lost - I will convert to manual ACKs in later PR
  • Server / always listening mode for MQTT will be in a separate PR
  • Making kamu-node into an MQTT broker can be done later if needed - for now it will act as a client, so user has to have their own broker and it needs to be accessible to kamu

Checklist before requesting a review

@sergiimk sergiimk force-pushed the feature/mqtt-polling branch from 475b3ca to 5e0dc43 Compare May 12, 2024 16:33
Copy link
Member

@s373r s373r left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A classy first step into the bigger world of small devices!

Added minor comment & question

src/infra/core/Cargo.toml Show resolved Hide resolved
src/infra/core/src/ingest/fetch_service.rs Show resolved Hide resolved
@sergiimk sergiimk force-pushed the feature/mqtt-polling branch from 5e0dc43 to 8073965 Compare May 14, 2024 20:43
@sergiimk sergiimk merged commit 8073965 into master May 14, 2024
6 checks passed
@sergiimk sergiimk deleted the feature/mqtt-polling branch May 14, 2024 20:51
@sergiimk sergiimk mentioned this pull request May 15, 2024
@sergiimk
Copy link
Member Author

Documentation PR: kamu-data/kamu-docs#22

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants