Only feed state events to widgets if we believe they update the state

[robintown]

As part of the effort to make room state more reliable, we're looking at changing how state is communicated to widgets. Rather than passing all state events from the timeline through (and forcing the widget to assume that they belong to the server's view of the room state), let's try passing only those state events through that we believe actually belong to the server's view of the room state. When combined with a matrix-js-sdk that supports MSC4222, this improves the reliability of state tracking for widgets.

We would like to later explore a solution that has a separate 'state update' widget action in addition to the generic 'incoming timeline event' widget action, to make the widget API better reflect the shape of the data sources that drive it (like the sync responses of Simplified Sliding Sync or MSC4222).

[toger5]

should the js-sdk bump to support 4222 also be part of this PR? So that we have a fully supported netlify build?

[robintown]

In theory the build should be working even without MSC4222 (it just won't benefit from that MSC). If we wanted to test with the MSC4222 matrix-js-sdk PR, the easiest way would be to use branch matching, so opening a new draft PR with branch name dbkr/stateafter.

[t3chguy]

If we wanted to test with the MSC4222 matrix-js-sdk PR, the easiest way would be to use branch matching, so opening a new draft PR with branch name dbkr/stateafter.

#28398 is already using that branch matching so please do not interrupt it as it'll mean it'll take longer for the MSC4222 support to land

[toger5]

#28398 is already using that branch matching so please do not interrupt it as it'll mean it'll take longer for the MSC4222 support to land

so we need to update 28398 to have the local store fixes?

[t3chguy]

so we need to update #28398 to have the local store fixes?

Please do not do that, my PR is the minimum required to retain compatibility with the js-sdk changes, any larger than that and it'll increase risk of landing it & also make review take longer than it needs to.

[t3chguy]

AFAICT widgets should be able to receive timeline state which don't update the state too, what if a widget wanted to render a timeline containing state events, omitting them would lead to the timeline inside the widget being desynchronised.

[robintown]

This is true, but as per the comments I think a compromise in which widgets see a slightly different timeline is appropriate here. The current shape of the widget API wants us to choose between reliably communicating state updates and presenting a complete timeline, and the latter is of a lower priority. (cf the fact that we're also dropping events if they were to arrive out of order.) Looking a little beyond the short term we know what can be done in the widget API to avoid this dilemma.

[t3chguy]

Given the widget API is something we control, we should just fix it properly. We didn't put days of effort into MSC4222 for its effort to go in vain. I am not happy enough with this change to approve it, but not hating it enough to block it. Happy for you to pick another reviewer for a 2nd opinion

[toger5]

I see this as a good step into the right direction. The same task is open on the rust sdk as well (for now we also do the same thing, just feed the right state events and basically don't support using state as timeline events in widgets.
This is what one would expect from state events as a widget dev.

But I do very much prefer flagging the state events if they are from timeline or state. And I want this to approached as a full project including an MSC and changes to the rustdriver, the widget api, the embedded js-sdk and the EW driver.

This is only fixing widgets to receive state updates instead of timeline state events.

I added this comment to the MSC: https://github.com/matrix-org/matrix-spec-proposals/pull/2762/files#r1850659461

So that this is tracked where it should be discussed.

[toger5]

This makes it equivalent to what we do in the Matrix Rust SDK as of now.
The widget api MSC is not explicit in what state events are send (timeline or state section)
Its also before SSS times. If it would have been written now it probably would have prioritized giving widgets the chance to compute the actual current room state instead of allowing widgets to display state events as timeline events.

So I think the order here is right:

• widgets getting timeline events (basically only option with sync v2 without state after)
• widgets getting state after events (fixing the issue that widget cannot compute current state)
• widgets getting both (@t3chguy proposal) to decide what they want to use or ignore (needs MSC)

[t3chguy]

The widget api MSC is not explicit in what state events are send (timeline or state section)

I would argue it is fairly clear. A widget can subscribe to m.receive.event:m.room.name and expect to get the state event down it (from /sync timeline), given the MSC does not say this cap excludes state events. With this PR this contract would then be broken.

[dbkr]

What about just adding a feedStateEvent that basically mirrors the state events that come through? This would be more backwards compatible since the feedEvent API wouldn't need to change, but widgets that want to use the new API just treat state events that arrive through feedEvent as timeline-only (the same way web does).

[toger5]

What about just adding a feedStateEvent that basically mirrors the state events that come through?

As a new widget action?

[dbkr]

Yep

[toger5]

That is the other option @t3chguy also preferred. In a slightly different format.
Michael had the idae of adding fromTimeline and fromState (or similar) flags to the widget action and then send all state we receive from state and timeline section. But this is what we would like to do as a follow up. Since it will need an MSC + impls on all repos.

I think as a first step sending the "more useful" (with the thinking: if one wants a timeline like behavior why no use timeline events) state is a good direction. The state events from timeline should be the custom additional feature we can add with another msc.

[dbkr]

OK, I'm not a big fan of this solution either but the widget API is under your purview as the VoIP team so ultimately it's your decision. I'm happy to sign this off as an impl of this in element-web.

[robintown]

#28681 for the full fix involving widget API changes