docs: add spec for status protocol stack, deprecate waku-usage spec #105
Conversation
jm-clius
requested review from
plopezlpz,
kaiserd,
kaichaosun,
chaitanyaprem,
jimstir and
fryorcraken
| The specific set of Waku protocols used depend on desired functionality and resource usage profile for the specific client. | ||
| Resources can be restricted in terms of bandwidth and computing resources. | ||
|
|
||
| Waku protocols that are more appropriate for resource-restricted environments are often termed "light protocols". |
There was a problem hiding this comment.
I have started to write an explanation article about this. Will share draft when ready and we can start to move away from non-relay/light protocol terminology.
There was a problem hiding this comment.
Yes. I'm happy to also adapt to whatever terminology we deem best. In case it helps, reasoning behind my sticking with "light" and "full":
- people seemed to always revert to this, even while we pushed for "resource-restricted" and "adaptive"
- it fits current Status client terminology
- relay vs non-relay wouldn't be accurate as each client will (or should) use a combination of filter/lightpush/relay etc. depending on resources.
| each using the combination of "full" and "light" protocols most appropriate to match its environment and motivations. | ||
|
|
||
| To simplify interaction with the selection of "full" and "light" protocols, | ||
| Status clients MUST define a "full mode" and "light mode" |
There was a problem hiding this comment.
| Status clients MUST define a "full mode" and "light mode" | |
| Status clients MUST define a "full mode" and "light mode" |
Same, I want to potentially propose alternative terminology. Will work on it and share draft.
status/raw/status-app-protocols.md
Outdated
| to retrieve the full contents of historical messages that the client may have missed during offline periods, | ||
| or to populate the local message database when the client starts up for the first time. | ||
|
|
||
| **Store queries for reliability** |
There was a problem hiding this comment.
I think we should clean up reliability specs (my bad, I committed to do it and never got around to it) so we can refer to them here instead of restating logic.
There was a problem hiding this comment.
That will be great.
| in order to provide light subscription and publishing services to other clients | ||
| for each pubsub topic to which they have a relay subscription. | ||
|
|
||
| Status clients MAY mount the store query protocol as service node (see [WAKU2-STORE](https://github.com/waku-org/specs/blob/8fea97c36c7bbdb8ddc284fa32aee8d00a2b4467/standards/core/store.md)) |
There was a problem hiding this comment.
No, but I think the spec should provision for this possibility, but it will remain underspecified until we have a proper design for decentralised store.
| also known as _self-addressed_ messages, | ||
| MUST be published to a distinct pubsub topic or a distinct _set_ of pubsub topics | ||
| used exclusively for messages with local scope (see [Pubsub topics and sharding](#pubsub-topics-and-sharding)). | ||
| Status clients (full or light) MUST use lightpush protocol to publish self-addressed messages (see [Publishing](#publishing)). |
There was a problem hiding this comment.
Shall we set expectations for the service node (eg usage of store sync over relay)?
There was a problem hiding this comment.
I think this is underdefined for now.
|
|
||
| Status clients MAY provide service-side protocols to other clients. | ||
|
|
||
| Full clients SHOULD mount |
There was a problem hiding this comment.
we may want to mention peerExchange and discv5 as well here
There was a problem hiding this comment.
Absolutely. The whole topic of discovery is still (purposely) missing, while I was gathering some thoughts. Will add in a subsequent commit.
| into smaller segments for individual Waku transport. | ||
| The definition of a large message is up to the application. | ||
| However, the maximum size for a [14/WAKU2-MESSAGE](../../waku/standards/core/14/message.md) payload is 150KB. | ||
| Status application payloads that exceed this size MUST be chunked into smaller pieces |
There was a problem hiding this comment.
isn't this statement contradicting in a way.
it states that large payloads must be chunked and also considered for large messages which mean they use a dedicated large-messages pubsubtopic.
I am wondering if pubsubtopics/shards are going to be dedicated for large messages do we need to set the 150KB limit or increase it since clients anyways are not going to subscribe to these topics.
There was a problem hiding this comment.
it states that large payloads must be chunked and also considered for large messages which mean they use a dedicated large-messages pubsubtopic.
I expect large messages to be chunked, in more chunks.
I am wondering if pubsubtopics/shards are going to be dedicated for large messages do we need to set the 150KB limit or increase it since clients anyways are not going to subscribe to these topics.
Fair q.
There was a problem hiding this comment.
I am wondering if pubsubtopics/shards are going to be dedicated for large messages do we need to set the 150KB limit or increase it since clients anyways are not going to subscribe to these topics.
Indeed a fair question to consider in future, but I don't think needs to be defined right now. The point is that the total data transfer for large messages (whether in chunks or not) needs a designed mechanism to (1) prevent them from being broadcast to everyone (2) allow users to download ad-hoc.
status/raw/status-app-protocols.md
Outdated
|
|
||
| The application SHOULD define at least one separate pubsub topic for each separate community's community control and community content messages. | ||
| The application MAY define a set of more than one pubsub topic per community to allow traffic sharding for scalability. | ||
| It is RECOMMENDED that separate pubsub topics be used for global control messages and global content messages. |
There was a problem hiding this comment.
| It is RECOMMENDED that separate pubsub topics be used for global control messages and global content messages. | |
| It is RECOMMENDED that separate pubsub topics be used for community control messages and community content messages. |
There was a problem hiding this comment.
Thanks! Addressed in 9ce3bc5
| To simplify interaction with the selection of "full" and "light" protocols, | ||
| Status clients MUST define a "full mode" and "light mode" | ||
| to allow users to select whether their client would prefer "full protocols" or "light protocols" by default. | ||
| Status Desktop clients are assumed to have more resources available and SHOULD use full mode by default. |
There was a problem hiding this comment.
I think for user facing devices/apps it should be light by default (desktop could have a switch to enable full mode). In most users' experience, the same app in the different devices should mostly behave the same, there may be more features in a desktop environment, but by default we should make it consistent between devices.
Without this, user will likely get surprised when finding the differences, also every app needs to educate user the differences.
I understand that there needs a lot of work to make this a reality, but for specification, the content that are works in long term makes more sense to me.
There was a problem hiding this comment.
I think for user facing devices/apps it should be light by default (desktop could have a switch to enable full mode). In most users' experience, the same app in the different devices should mostly behave the same, there may be more features in a desktop environment, but by default we should make it consistent between devices.
Ultimately the question is for Status product team. But considering the desired properties, it is fair to assume that a desktop app may consume more resources. The same way it's fair to assume that using tor browser is slower, or using BitTorrent over FTP consumes more upload bandwidth.
If the Status product team decides that Desktop and Mobile must behave the same, and both must use light mode by default. Then the question is "where is the infrastructure"?
This can be compensated by either:
- encouraging users to run companion "nodes": a second software on your laptop, rasberry pi, cloud or enable relay mode
- Or having incentivization in place, where users or community owners pay for the resources of the network, with a large enough number of nodes to support this relay network
In this context, it also mean that by "default" privacy and censorship resistance are less than what they could be.
(1) is unlikely to be enough to allow the application to be reliable, as regularly stated.
(2) we are not ready yet, work is in progress and I expect to agressivly ramp up this stream in 2025 H1.
Once (2) is ready, then Status product will have a choice between:
A. Free product to use, but users share resources by default
B. Paying product for users, but they dont share resources by default.
(A) was clearly the selected direction so far by Status.
status/raw/status-app-protocols.md
Outdated
|
|
||
| **Community messages** | ||
|
|
||
| The application SHOULD define at least one separate pubsub topic for each separate community's community control and community content messages. |
There was a problem hiding this comment.
Maybe like this to avoid repeating community thrice?
| The application SHOULD define at least one separate pubsub topic for each separate community's community control and community content messages. | |
| The application SHOULD define at least one separate pubsub topic for each separate community's control and content messages. |
There was a problem hiding this comment.
Yes. Addressed in 9ce3bc5
| Status clients SHOULD use the store query protocol, as specified in [WAKU2-STORE](https://github.com/waku-org/specs/blob/8fea97c36c7bbdb8ddc284fa32aee8d00a2b4467/standards/core/store.md), to retrieve historical messages relevant to the client from store service nodes in the network. | ||
|
|
There was a problem hiding this comment.
| Status clients SHOULD use the store query protocol, as specified in [WAKU2-STORE](https://github.com/waku-org/specs/blob/8fea97c36c7bbdb8ddc284fa32aee8d00a2b4467/standards/core/store.md), to retrieve historical messages relevant to the client from store service nodes in the network. | |
| Status clients SHOULD use the store query protocol, to retrieve historical messages relevant to the client from store service nodes in the network. For more information, see [WAKU2-STORE](https://github.com/waku-org/specs/blob/8fea97c36c7bbdb8ddc284fa32aee8d00a2b4467/standards/core/store.md). | |
Co-authored-by: Jimmy Debe <[email protected]>
Co-authored-by: Jimmy Debe <[email protected]>
Co-authored-by: Jimmy Debe <[email protected]>
Co-authored-by: Jimmy Debe <[email protected]>
Co-authored-by: Jimmy Debe <[email protected]>
|
|
||
| #### Community scope | ||
|
|
||
| 3. _Community control_: messages enabling the basic functioning of the app to control features _only relevant to members of a specific community_. Examples include Community Membership Updates, community Status Updates, etc. |
There was a problem hiding this comment.
Currently, that's not entirely true. Users subscribe to updates for communities they may not be members of. An example is the Discover Communities section.
There was a problem hiding this comment.
Thanks! I imagine this is based on some community description messages that are published "to everyone" - i.e. at least on shards that everyone is subscribed to and is not encrypted by the community key? In this case, these would still be modelled as "global" messages as they affect operations that everyone should be able to participate in. "Community control" refers to those messages that can be distributed only within a single community.
There was a problem hiding this comment.
Yes, that information is derived from CommunityDescription. Please note that CommunityDescription itself is never fully encrypted; only specific sub-elements, like members and channels, are encrypted depending on the context. Even so, these cards will render correctly, with the exception of member counts and active member counts, which remain inaccessible to users without the necessary key.
As for shards, CommunityDescription can reside within its own community shard. The protocol is designed to ensure that everyone can locate the shard a given community uses, enabling them to subscribe to that shard to access the description. Essentially, the community owner publishes CommunityDescription on the community shard and shares shard details on the default shard.
Rationale: status-im/status-go#3961 (comment)
Issue: status-im/status-go#4230
Implementation: status-im/status-go#4499
There was a problem hiding this comment.
Thanks! I think the issue here might be that eventually we'd want Communities to be able to provide their own infrastructure - their own bootstrap nodes, Store nodes, etc. In this case we may not want everyone to be able to retrieve the Community Description (which would require discovering and using that Communities internal service infrastructure). Of course, it will be easy to define a global shard only used for "discoverable" Community descriptions, so I don't think the spec prohibits either design.
| | Status application layer | | ||
| | End-to-end reliability layer | | ||
| | Encryption layer | | ||
| | Transport layer (Waku) | |
There was a problem hiding this comment.
There is also a Segmentation layer just before Transport layer.
There was a problem hiding this comment.
Thanks! This is very helpful for my understanding. I included a section on "chunking large messages", but this should indeed be a separate layer with separate paragraph.
| In other words, the number of content topics defined in the app SHOULD match the number of filter use cases. | ||
| For the sake of illustration, consider the following common content topic and filter use cases: | ||
|
|
||
| - if all messages belonging to the same 1:1 chat are always filtered together, they SHOULD use the same content topic (see [55/STATUS-1TO1-CHAT](../55/1to1-chat.md)) |
There was a problem hiding this comment.
Just a note: 1:1 chats may use different content topics depending on the state of topic negotiation between the two parties.
| and consequently SHOULD also make use of MVDS to achieve reliable data synchronisation between all parties involved in the communication. | ||
| Non-ephemeral 1:1 and private group chat messages MAY make use of of [scalable distributed log reliability](https://forum.vac.dev/t/end-to-end-reliability-for-scalable-distributed-logs/293/16) in future. | ||
| Since MVDS does not scale for large number of participants in the communication, | ||
| non-ephemeral community messages MUST use scalable distributed log reliability as defined in this [original forum post announcement](https://forum.vac.dev/t/end-to-end-reliability-for-scalable-distributed-logs/293/16). |
There was a problem hiding this comment.
In the context of communities, an app-level reliability mechanism called peersyncing was introduced some time ago. It is currently disabled.
There was a problem hiding this comment.
Indeed! I chose to exclude it (for now) as its future is uncertain.
|
|
||
| ## Encryption layer | ||
|
|
||
| The encryption layer wraps the Status App and Reliability layers in an encrypted payload. |
There was a problem hiding this comment.
The payload is not always encrypted. Both public and private messages are wrapped in an encryption layer; however, only private messages are actually encrypted but both carry x3dh bundle information.
| The Status application MAY use a chunking mechanism to break down large payloads | ||
| into smaller segments for individual Waku transport. | ||
| The definition of a large message is up to the application. | ||
| However, the maximum size for a [14/WAKU2-MESSAGE](../../waku/standards/core/14/message.md) payload is 150KB. |
There was a problem hiding this comment.
That's not the case atm: status-im/status-go#4955
|
Thanks for your insights, @osmaczko. Really clarified a couple of things for me.
Well, I wouldn't call the specified protocol the "ideal state" yet, but it certainly aims to specify beyond the current state. I wanted to capture the short and medium-term improvements that we have made and plan to make for the Status protocols to reach a viable state. The next steps would be to ensure that we do all the implementation work to bring the implementation to parity with the spec. Most of the remaining work relates to the introduction of e2e reliability, better content topic usage (advanced stage of implementation done) and different sharding strategy, |
Scope
This adds a new raw specification covering the Status protocol stack, including:
It adds directives on:
It does not define any of the app-level functions itself. These most properly belong in the 1:1 Chat and Community Specifications.
Since this specification also addresses the Waku transport layer, it deprecates the old WAKU2-USAGE spec.
What major items are still missing from this spec?
What are the next steps?
1. Continue revising Status specs
This spec is envisioned to be the "landing page" spec that establishes the high-level strategy and concepts for all Status protocols (similar to the role 10/WAKU2 played for Waku protocols). The next step would be to revise the 1:1 Chat and Community specs according to the concepts established in this specification, e.g. being explicit about content topic and functional scope for each app-level message. In the process, we can probably merge or deprecate a whole bunch of other Status specs.
2. Continue implementing newly specified strategies
The main purpose of writing the Status specs is to carefully establish reasonable strategies to save bandwidth, establish e2e reliability, etc. We should continue the work to bring the application implementation in line with the specifications.