Include SparkplugB Adapter and DataSink

[heinehenning]

SparkplugB is a great standard for SmartFactory use cases, but it can be a pain to build custom SpB gateways by hand. If streampipes would offer SparkplugB connectivity plus data modeling options that would be pretty nice.

[bossenti]

Hey @heinehenning,

thanks a lot for bringing this idea in!
I really like your idea and I assume supporting Sparkplug could be a great addition to StreamPipes.
At least from my point of view, it might take some time to get into Sparkplug and get some ideas how we could support/integrate it with StreamPipes. But @dominikriemer or @tenthe might already have some ideas here?!

Just out of interest: Do you already have a scenario in mind where this would help you?

[heinehenning]

Hi @bossenti,

some scenarios just from the top of my head:

• translate vanilla mqtt into SpB to make all data accessible in the same namespace
• we had to build some custom services to translate e.g. modbus data to SpB. Without knowing streampipes in depth yet, I think w/ streampipes, tasks like this could be done in a matter of hours instead of days without having deeper programming skills
• we are operating a couple services that publish data via SpB. They are becoming more and more important for production. Therefore monitoring is an important topic. Streampipes would be handy to build notification pipelines that monitor the services SpB status payloads

If I have more ideas, Ibwill add them here...

[bossenti]

Thanks a lot. This is really helpful to understand how we the integration could be beneficial.
If I get you corret, there two main scenarios:

• reading data with StreamPipes in SpB format that is provided e.g. via MQTT
• transform data to SpB format that is provided by various sources e.g., Modbus

[heinehenning]

That is basically it. In the second scenario StreamPipes would be especially usefull to add context to the data, but that I think is a nice feature in general and not limited to the above discussed SpB connectivity.

[tenthe]

Hi @heinehenning,
thanks a lot for starting the discussion.
I like the idea to integrate more concepts of SparkplugB. Maybe we can do this step by step.
In the future I could imagine to also adopt the data models for the internal modeling.
For now though, I agree with @bossenti to do a first integration.
What is still missing for this from your point of view?

[heinehenning]

Hi @tenthe!

I am not 100% sure if your last question was directed at me or at @bossenti.
From a user perspective I would say that for the part of reading/receiving SpB StreamPipe offers all the tools necessary. You could map SpB metrics in a similar fashion as JSON payloads during the definition of the connector.
For the transformation of data into SpB we need to define at least the following things at one place or another:

• GroupId
• NodeId
• DeviceId
• MetricName
  The easiest option would be to pack all that stuff into a MQTT-SpB-DataSink and define them together with the rest of the MQTT connection. If a metric should contain more than just one value some kind of mapping needs to happen between DataSource and the Metric.
  The approach from node-red-contrib-mqtt-sparkplug-plus is not to bad.
  It gives some help defining the metric. But it still requires the user to put together the payload by hand. As a user it would be nice to have an option to connect DataSource(s) and one SpB-DataSink. Inside the DataSink I would like to be able to select certain values from the sources and map them to the metrics kind of like in the below screenshot from the mqtt-sparkplug-device of node-red-contrib-mqtt-sparkplug-plus.

[tenthe]

Hi @heinehenning,
thanks a lot for the detailed description.

Internally we represent the data as data streams. Each stream consists of properties and one of those must be a unix timestamp.
E.g.:

{
  "timestamp": 1683297667,
  "Sensor1": 1,
  "Sensor2": 2.2,
  "State": true
}

I like the idea of providing both a SparkplugB adapter and a data sink.
The way I see it, this should not be too hard.
My suggestion for the next step would be to create an issue for this and add more details.
Maybe we can collect some sample events and specify the user input needed to configure them.

What are your thoughts on this?
(We can also convert this discussion into an issue)

[heinehenning]

Hi @tenthe,

awesome that you want to take on this topic. I would say lets go for it. Should I create the issue or should it be someone from the PMC?

One more thing that came to my mind: we would need to include protobuff, I think. SpB is usually protobuff encoded.

If I can help somehow let me know. I dont think I can contribute to much on the code side, as I dont now StreamPipes well enough yet. I could always help with testing, if that would be of any help.

The eclipse/tahu github repo is a good starting point with good examples. And mqtt.fx or ignition can be used as clients that can speak SpB.

Cheers
Henning

[tenthe]

Hi @heinehenning,
if you want you can create the issue with the description.

I'm currently working on the refactoring of the connect API, once it's done I can take a look at the SpB integration.
In the meantime, we can mark the problem as "good first issue", maybe someone is interested to work on it.
I'm happy to support anyone who wants to work on it.

Cheers,
Philipp

[heinehenning]

I cannot create the Issue myself, if one of you would be so kind...

Create SparkplugB Adapter and DataSink

SparkplugB SpB is a MQTT specification aiming primarily at the manufacturing industry to connect network nodes in an IIOT infrastructure. In order to standardize how things communicate in a manufacturing setting via MQTT, SpB defines the following things:

• MQTT topic namespace
• MQTT state management
• MQTT payload structure

The following should be integrated in streampipes:

Adaper:

• reading data with streampipes that is provided via SpB
• mapping the SpB payload structure to the streampipes internal data structure
• acting as a SpB compliant client (e.g. handling NCMD, DCMD, NDEATH payloads)

DataSink:

• building SparkplugB payloads according to the SpB specification
• mapping streampipes data to the SpB payloads
• publishing the SpB payloads

SpB specification: https://sparkplug.eclipse.org/specification/

SpB on github:
https://github.com/eclipse/tahu

[bossenti]

Thanks a lot for preparing the issue that well @heinehenning
Here it is: #1627