SSS: misordered messages in room timeline

[ara4n]

Steps to reproduce

1. Opened room
2. Saw completely misordered timeline. It's as if the most recent msg in the room cache got stuck between newer messages and paginated messages. See rageshake for details.
3. This is the 3rd time i've seen a variation of this today (i.e. "wrong room history")

Outcome

What did you expect?

Reliable room history.

What happened instead?

Misordered (or missing) history, which later fixes itself.

Your phone model

No response

Operating system version

No response

Application version

669

Homeserver

No response

Will you send logs?

Yes

[ara4n]

Erik's analysis was:

From what I can tell from jaeger SSS sent down the last message, and then it called /messages and got the rest of the events down in a sane order
I don't see anything abnormal going on in terms of ordering or timestamps for those events
I believe that the misordered message in the screenshot was received via SSS yesterday
Which all looks fine server side, so my guess is its some weird client side ordering of the timeline bug?

[Hywan]

Nothing has changed on the Rust SDK since 2 or 3 weeks, so it looks suspicious.

Could it be related to element-hq/synapse#17510?

[ara4n]

the fact that the timeline doesn't even show date markers for the misordered events is suspicious in terms of there also being a timeline crate bug here too though...

[ara4n]

just had this very badly in build 686. have provided a detailed rageshake. am not sure this is really minor, as it feels very flakey for msgs to disappear and misorder when on slower data

[Hywan]

As long as we don't have proper room subscriptions, I feel it's a bit useless to dig into this :-/. How often does it happen on your side?

[ara4n]

just had this again on 691.

it feels like a variation on #1279

don't think this has anything to do with sss or subscriptions tbh

filed as #3214 given this happened on send, not on open

[manuroe]

it may be related to #3220. @ara4n can you still see this issue happening?