Binary Endpoints for IoT Ecosystem #67
Replies: 3 comments 5 replies
-
|
A binary format is crucial to resource constrained devices. JSON is compact, but it's not compact enough for resource constrained devices, on which any character-based format would be bloated. By supporting a binary format for APIs on the node software, we allow (even Class 01) devices to perform actions on the Tangle without hassle. But by simply saying "a binary format" the structure of data is not defined. Fortunately there has been standard ways to binary encode structured data without losing the structure (which makes it convenient to just convert from and to JSON). Because HTTP is also a character-based protocol, we can also take the transfer protocol into consideration (but not necessarily). I can think of a few possible solutions:
We've had MQTT endpoints running, but with a separate Event API. It's possible to further map and provide REST APIs in MQTT topics to enable more IoT devices to communicate with node software without the bloat of HTTP. CBOR(official website, Wikipedia, spec (RFC8949)) stands for "Concise Binary Object Representation". Its data model is based on JSON, so transformative between JSON and CBOR should be straightforward. It can be sent over any transfer protocol - it's just a way to encode data. FlatBuffers (official website, Wikipedia, schema compiler) can be seen as an improvement from Protocol Buffers. It uses schema to define the shape of data, so the serialized binary data has no structure marked in it; in other words, no parsing or unpacking is needed to access the structured data. It's thus more concise than any other serialization format, but it also makes the serialized data non-self-contained. CoAP (official website, Wikipedia, spec (RFC7252)) is a protocol designed to run on resource constrained devices and UDP. It borrows concepts from HTTP, making common features between the two easy to translate between each other. It's usually seen as a competitor of MQTT, who run (mainly) on TCP and emphasize long connection instead. One of the ideal combinations could be CBOR/CoAP. Both are well-defined, widely-used, widely-supported protocol formats designed specially for resource constrained IoT devices. It's also fine to just use HTTP to provide the REST APIs. In this way we do content negotiation, i.e. checking the Footnotes
|
Beta Was this translation helpful? Give feedback.
-
Actually, it is defined in protocol models. I've updated some examples in the description. This proposal is not trying to introduce other serialization tools or network protocols. we can gain the benefits that you mentioned by just changing the content in the HTTP payload. |
Beta Was this translation helpful? Give feedback.
-
You are right, we don't need the |
Beta Was this translation helpful? Give feedback.
-
The |
Beta Was this translation helpful? Give feedback.
-
|
@alexsporn just got my coffee, you are right. a custom MINI type would be a great help for debugging. anyone can write a decoder as needed. |
Beta Was this translation helpful? Give feedback.
-
I second @lmy441900 opinion that we should not roll our own adhoc binary format, and instead use any of the already widely supported ones. As this discussion proposes to extend the binary support to more endpoints of the API, I think it's the right place to open the serialization format debate, if only to discourage moving forward with the proposal as long as it implies further rolling an adhoc binary format.
Just for contextualization, in Rust there are options that make parsing JSON a lesser pain in embedded devices. First,
In my opinion adding another communication protocol that requires its own server sounds overkill, specially assuming the availability of HTTP/2. I would second mapping the REST endpoints over MQTT topics, although replicating the request/response nature of the REST API over MQTT is not a trivial task to do right, and not all constraint devices are suitable for MQTT communication.
I agree with @lmy441900 that CBOR and Flatbuffers are both very well suited options. Considering the coexistence with the JSON format, I would argue in favor of using FlatBuffers instead of CBOR, as the benefits of being a schema-based, zero-copy protocol better set it apart from its JSON counterpart. If schema-less and self-description are considered a hard requirement, FlexBuffers can also be a strong contender. The main argument I would have in favour of CBOR is its standardized nature (which is of course not a minor argument). If it is (arguably) considered not the time to add any of these binary formats, as said I would discourage moving forward with this proposal and instead wait until a sounder binary support can be considered. Rolling an adhoc binary format in a public API entails a high risk not only to have very low adoption, but actually to be harmful to those 3rd-parties that eventually couple to it. |
Beta Was this translation helpful? Give feedback.
-
|
@arnauorriols thanks for your input. this discussion is meant to come out with reasonable support for IoT adoption. imo, FlatBuffers is not a fit for embedded development, it's designed for gaming and streaming for reducing serialization latency. it bloats the binary data, for example, FlatBuffers needs around 400 bytes of memory to send out 200 bytes of data, the memory consumption is similar to JSON which is a big drawback.
agree, binary communication has low adoption but the protocol shouldn't be changed frequently for a production-ready project. |
Beta Was this translation helpful? Give feedback.
-
|
Updated Client response objects in the description, thanks for @alexsporn pointing out, the MIMI type can be a customized one and need to be defined instead of discussed with the protocol team, the conclusion is anyone can build an external plugin to connect IoT devices via their own binary format. We don't know the reqs and what they use. We don't want to optimize for constrained devices in Stardust, at least not before the Shimmer launch. let's move forward and revisit this topic later. |
Beta Was this translation helpful? Give feedback.
-
Introduction
API endpoints are defined in following TIPs:
Most of the data are communicated via JSON objects which is a human-readable text format. For constrained devices, those text data are not packed and data conversions involve more frequently memory operations, especially allocation and free, which leads to memory fragmentation issues and causes potential system instabilities.
To reduce data conversion and serialization during communications, the node can provide endpoints that communicate with binary payload through HTTP. It has been partially implemented in
/api/v2/messages.The client is able to send a message to the node in JSON string, like
or binary data, for example
The payload size in the example for binary is only 810 bytes, JSON is 2863 bytes, the binary is 3 times smaller.
Additionally, the following endpoints can be a customized MIME type, instead of
application/octet-stream, in both requests and responses.Example responses
Excep for data types in Serialization Primitives, two extended data types are introduced for the Binary Client Response layout:
uintXdenotes the length of the list.uintYdenotes the lenght of ByteArrayuintXdenotes the length of the list.The value of
XandYshould be one of 8, 16, 32.Error response:
{ "error": { "code": 400, "message": "invalid data provided" } }Binary:
Restful API responses
/api/v2/info
JSON
{ "name": "HORNET", "version": "0.6.0-alpha", "status": { "isHealthy": true, "latestMilestoneTimestamp": 1617802102, "latestMilestoneIndex": 480, "confirmedMilestoneIndex": 480, "pruningIndex": 0 }, "metrics": { "messagesPerSecond": 17, "referencedMessagesPerSecond": 16.8, "referencedRate": 98.82352941176471 }, "protocol": { "networkName": "iota-testnet", "bech32HRP": "atoi", "minPoWScore": 1000, "rentStructure": { "vByteCost": 500, "vByteFactorData": 1, "vByteFactorKey": 10 } }, "features": [ "PoW" ], "plugins": [ "indexer/v1" ] }Binary
/api/v2/tips
JSON
{ "tipMessageIds": [ "0xf532a53545103276b46876c473846d98648ee418468bce76df4868648dd73e5d", "0x78d546b46aec4557872139a48f66bc567687e8413578a14323548732358914a2" ] }Binary
/api/v2/messages/{messageId}
JSON
{ "protocolVersion": 1, "parentMessageIds": [ "0xc0b728d4c292860b57747c7bfbd989ccc40be8c76229022644c05e1352bf1ea3", "0xe8db39681478cf9826bd2f31d045f16f5e1f478c5abc3030419da7911df9e288" ], "payload": null, "nonce": "1481" }Binary
/api/v2/outputs/{outputId}
JSON
{ "messageId": "0x9cd745ef6800c8e8c80b09174ee4b250b3c43dfa62d7c6a4e61f848febf731a0", "transactionId": "0x1ee46e19f4219ee65afc10227d0ca22753f76ef32d1e922e5cbe3fbc9b5a5298", "outputIndex": 1, "isSpent": false, "milestoneIndexBooked": 1234567, "milestoneTimestampBooked": 1643207146, "ledgerIndex": 946704, "output": { "type": 3, "amount": "1000", "unlockConditions": [ { "type": 0, "address": { "type": 0, "pubKeyHash": "0x8eaf87ac1f52eb05f2c7c0c15502df990a228838dc37bd18de9503d69afd257d" } } ] } }Binary
/api/v2/messages/{messageId}/metadata
JSON
{ "messageId": "0x2b527cfc71ccb87e986f129116b1d59e28c298400f63dd819f9aa04d4057a06a", "parentMessageIds": [ "0x4547f36e54e6b89ce8ac9d238f0f5fb373fa61c734d8d72f976a370bd76ff647", "0x8ca33c5d427a9f2e4850716fdbc48c7653ed27082d64f1637d19dac6d3147072", "0xb09ae4f032b9cd52c39c596b3edb9655eabe64bd6de742d6c870c4a7b2c6b0d8", "0xe46e92df9cfcc00cc48b807dc9e8e8b01afd6fc79c27446f2061e695b258391e" ], "isSolid": true, "referencedByMilestoneIndex": 54472, "ledgerInclusionState": "included" }Binary
Indexer responses
All Indexer response are using the same response structure, for example
{ "ledgerIndex": 67633, "pageSize": 1000, "items": [ "0x971a55b87617db1bf58f3db7878c19280e4293c836a3966584ea77405d168f5e0000", "0x8e9b6a1665db02179e062fdaf8c086f9845c2017b0d9fe70562fc6010d317ece0000" ] }Binary
Pros
Cons
Conclusion
As constrained clients, such as a system under 100MHz CPU and 256KB memory, the communication should be packed and come with a small memory footprint, this proposal provides a friendly communication channel and opens a door for IoT adoption in the early stage.
Beta Was this translation helpful? Give feedback.
All reactions