Work on matrix-org/matrix-spec-proposals#2716

Complement MR: matrix-org/complement#68

Also edits for TARDIS visualation: matrix-org/tardis#1

TODO: Is the assumption of anytime we pass in prev_event_ids, we use same depth
good enough? What corner cases are there? I see that we also pass in prev_event_ids this in
synapse/handlers/room_member.py so need to make sure that still work as expected

Related matrix-org/tardis#1

…n prev_event query param is used

Co-authored-by: Patrick Cloke <[email protected]>

Co-authored-by: Patrick Cloke <[email protected]>

Co-authored-by: Patrick Cloke <[email protected]>

…org/synapse into eric/msc2716-backfilling-history

This way we handle the early return case as well

Address https://github.com/matrix-org/synapse/pull/9247/files#r573711617

See #9247 (comment)

Conflicts:
	synapse/events/builder.py

See, #9247 (comment)

Implementing #9247 (comment)

Conflicts:
	synapse/config/experimental.py
	synapse/visibility.py

Spawned from discussion in matrix-org/matrix-spec-proposals#2716 (comment)

Other PR with MSC2716 attempt: #9247

As described from matrix-org/matrix-spec-proposals#2716 (comment)

Conflicts:
	scripts-dev/complement.sh
	synapse/config/experimental.py
	tests/storage/test_redaction.py

…authed)

Copy repo state from #9759

…ervice can use

They do not auth correctly though

```
2021-05-01 02:08:20,888 - synapse.handlers.federation - 2509 - INFO - GET-4 - auth_events refers to events which are not in our calculated auth chain: {'$tLNZXWvubBTZyo8VQwG77xni_lMw67AseQRxQ9hJblM', '$iwYNb_ZF4IHRB-4x1OCjtCT206clBlUjDlTZFtxPbiY', '$79GFmHJKGSpQTYmxL6gpteK7CgaVtAV_KKjTxXKSKtY', '$6U3Dw80m6PwtUqp4hzGdo6LL3Q3CeRyL3CpY2j9NOKo', '$8qUw8T4lslR38TUmeIB2kXwF2eMvOLbAGj-PiIdVAM8', '$bhniSE3Vv71c3Eo1G_YCig3VNflaAbDIyIm51d_6nsU', '$gl7q_qh3siy4Gx_iCwhT8cGJSdnH38_o97kGHR0HJYM'}
2021-05-01 02:08:20,890 - synapse.state - 452 - INFO - GET-4 - Resolving state for !pIRfotsTwFNpPshIIV:hs1 with 2 groups
2021-05-01 02:08:20,893 - synapse.handlers.federation - 2548 - INFO - GET-4 - After state res: updating auth_events with new state {('m.room.create', ''): '$6U3Dw80m6PwtUqp4hzGdo6LL3Q3CeRyL3CpY2j9NOKo', ('m.room.member', '@the-bridge-user:hs1'): '$tLNZXWvubBTZyo8VQwG77xni_lMw67AseQRxQ9hJblM', ('m.room.power_levels', ''): '$iwYNb_ZF4IHRB-4x1OCjtCT206clBlUjDlTZFtxPbiY', ('m.room.join_rules', ''): '$bhniSE3Vv71c3Eo1G_YCig3VNflaAbDIyIm51d_6nsU', ('m.room.history_visibility', ''): '$gl7q_qh3siy4Gx_iCwhT8cGJSdnH38_o97kGHR0HJYM', ('m.room.member', '@alice:hs1'): '$8qUw8T4lslR38TUmeIB2kXwF2eMvOLbAGj-PiIdVAM8'}
2021-05-01 02:08:20,897 - synapse.handlers.federation - 2389 - WARNING - GET-4 - Failed auth resolution for <FrozenEventV3 event_id='$Xow9Ex_IDhUMm8W-nFOkL-NXT4lByajdO-aDcNuScbM', type='m.room.message', state_key=None> because 403: User @maria:hs1 not in room !pIRfotsTwFNpPshIIV:hs1 (None)
```

See #9247 (comment)

Wrong member event coming down /event_auth

See #9247 (comment)

Make sure to use a copy of this list as we add to it and loop here.
Otherwise it will be the same reference and update when we append here later.

For context, see #9247 (comment)

See https://github.com/matrix-org/synapse/pull/9247/files#r634269224

See #9247 (comment)

Start of refactor to order events in each chunk in chronological order
but persist events in reverse-chronological order to get the correct
(topological_ordering, stream_ordering).

No marker events

See #9247 (comment)

Conflicts:
	scripts-dev/complement.sh
	synapse/api/auth.py
	synapse/handlers/message.py

Conflicts:
	synapse/http/servlet.py

See #10049 (comment)

…the event dictionary

See #10049 (comment)

See https://github.com/matrix-org/synapse/pull/9247/files#r636778295

Conflicts:
	changelog.d/10049.misc
	synapse/handlers/message.py

It was a problem with how my Complement tests were only iterating
over events with body fields.

See #9247 (comment)

See #9247 (comment)

… can use it)

See #9247 (comment)

…s well)

See #9247 (comment)

Conflicts:
	synapse/config/experimental.py
	synapse/handlers/room_member.py

See #9247 (comment)

Conflicts:
	synapse/config/experimental.py