Make historical events discoverable from backfill for servers without any scrollback history (MSC2716) (federation)

[MadLittleMods]

Make historical events discoverable from backfill for servers without any scrollback history.

• This does not cover "marker" events or servers which already have all of the scrollback(no backwards extremeties to pull from)
• As the /backfill function iterates up the DAG, we also check for any insertion events which have prev_events pointing to the current backfill event. If we find one, we pull in the insertion event and start of the chunk. Then keep iterating up the chunk of history.

Part of MSC2716: matrix-org/matrix-spec-proposals#2716

Follow-up to #9247

Mermaid live editor playground link

mermaid graph syntax

flowchart BT
    subgraph live
        B -----------------> A
    end
    
    subgraph chunk0
        chunk0-chunk[["chunk"]] --> chunk0-2(("2")) --> chunk0-1((1)) --> chunk0-0((0)) --> chunk0-insertion[/insertion\]
    end

    subgraph chunk1
        chunk1-chunk[["chunk"]] --> chunk1-2(("2")) --> chunk1-1((1)) --> chunk1-0((0)) --> chunk1-insertion[/insertion\]
    end
    
    subgraph chunk2
        chunk2-chunk[["chunk"]] --> chunk2-2(("2")) --> chunk2-1((1)) --> chunk2-0((0)) --> chunk2-insertion[/insertion\]
    end

    
    chunk0-insertionBase[/insertion\] -------------> A
    chunk0-chunk -.-> chunk0-insertionBase[/insertion\]
    chunk1-chunk -.-> chunk0-insertion
    chunk2-chunk -.-> chunk1-insertion

Loading

Dev notes

• https://github.com/matrix-org/synapse/blob/develop/docs/development/database_schema.md
• https://github.com/matrix-org/synapse/blob/develop/docs/postgres.md
• psql synapse: Open the postgres CLI against the synapse database
• \q -> enter: To quit/exit
• \dt: list tables
• \d+ issues: list columns and indexes for issues table
• CREATE TABLE board_labels();: Create a table called board_labels
• DROP TABLE insertion_event_extremeties
• explain SELECT * FROM events WHERE depth = 1;
• explain analyze SELECT * FROM events WHERE depth = 1;

---

Access database in Complement test Docker container:

$ docker exec -it complement_1_hs_with_application_service.hs1_2 /bin/bash

$ apt-get update && apt-get install -y sqlite3

$ sqlite3
> .open /conf/homeserver.db

---

• synapse/storage/schema/__init__.py (synapse.storage.schema.SCHEMA_VERSION and synapse.storage.schema.SCHEMA_COMPAT_VERSION)
• synapse/storage/prepare_database.py ->_setup_new_database and _upgrade_existing_database

  Sets up the physical database by finding a base set of "full schemas" and
  then applying any necessary deltas, including schemas from the given data
  stores.

  The "full_schemas" directory has subdirectories named after versions. This
  function searches for the highest version less than or equal to
  SCHEMA_VERSION and executes all .sql files in that directory.

• asdf

---

• event_backward_extremities
• event_relations
• synapse/handlers/federation.py -> maybe_backfill
• synapse/handlers/federation.py -> on_backfill_request -> synapse/storage/databases/main/event_federation.py -> get_backfill_events

---

Call stacks for backfilling

From the remote server

• RoomMessageListRestServlet.on_GET
• get_messages
• maybe_backfill
• _maybe_backfill_inner
• try_backfill
• backfill

From the origin server

• FederationBackfillServlet.on_GET
• on_backfill_request
• get_backfill_events

---

get_events
get_events_as_list
_get_events_from_cache_or_db
_get_events_from_db
_enqueue_events
_do_fetch
_fetch_event_list
_fetch_event_rows

---

Previous federation attempt at changing backfill looking at successors, https://github.com/matrix-org/synapse/pull/9247/files/13b18a8ec5cf03e3f46cc730464cfbfd10c82664#diff-1f5d8bffadd3271e42c2f6a66474bbf5e3e6694b009aa616b5f5433506217ff1R965

---

go test -v -tags synapse_blacklist,msc2946,msc3083,msc2716,msc2403 -count=1 $EXTRA_COMPLEMENT_ARGS ./tests -run TestBackfillingHistory/parallel/Historical_messages_are_visible_when_joining_on_federated_server

go test -v -tags synapse_blacklist,msc2946,msc3083,msc2716,msc2403 -count=1 $EXTRA_COMPLEMENT_ARGS ./tests -run TestBackfillingHistory/parallel/Historical_messages_are_visible_when_joining_on_federated_server_-_pre-made_insertion_event

Todo

• Wait for Add base starting insertion event when no chunk ID is provided (MSC2716) #10250 to merge
• Remove need for ?user_id query parameter which stops us from importing from multiple users in the chunk -> Fix messages from multiple senders in historical chunk (MSC2716) #10276
• Update /backfill to traverse "markers", "insertion" points, and chunks
• A new "marker" event will allow server already in room to discover the history
  • Covered by Draft: Add support for MSC2716 marker events #10420
• Add test case for base insertion event from the "live" timeline
• Add test for using the generated base insertion event part of the chunk
• Limit who can send insertion events and chunk ID content field on events (reject in auth code)
• Assert (room_id, next_chunk_id) is unique in the app code. Insertion events should have unique next_chunk_id per room.

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
• Pull request includes a sign off
• Code style is correct (run the linters)

[MadLittleMods]

Where is the best place to limit who can add an insertion event or a chunk connection content field?

[erikjohnston]

We probably want to reject such events, so we probably want to add some stuff to the event authorization code.

[MadLittleMods]

I am having trouble fitting this into the existing code.

• I need to make sure that only application services can send insertion, marker, and events with chunk ID's
• Other homeservers need to be able to accept these events over federation. But probably should only trust the originating homeserver where the room is. How does originating homeserver even work when future room ID's don't include the homeserver?
  • Maybe it should be some power level and we can specify the application service user ID with a high enough power level?

---

• synapse/event_auth.py -> check(...):
  • It feels like the checks should go here next to check_redaction
  • That class doesn't have access to the store so I can't use store.get_app_service_by_user_id(...) to make sure the sender is an application service. Plus this code is probably run for events coming over federation which don't know which sender is an application service.
  • It could fit here if it was a new power level since we have the auth events
• synapse/handlers/event_auth.py -> check_from_context(...)
  • I can use store.get_app_service_by_user_id(...) but other code is still only protected by the raw check(...)

---

def _check_if_allowed_to_send_historical_events

    def _check_if_allowed_to_send_historical_events(
        self,
        event: EventBase,
    ):
        """Check whether the event sender is allowed to add insertion, marker,
        or events with a chunk ID

        Raises:
            AuthError if the event sender is not an application service
        """

        if (
            event.type != EventTypes.MSC2716_INSERTION
            and event.type != EventTypes.MSC2716_MARKER
            and event.content.get(EventContentFields.MSC2716_CHUNK_ID) is None
        ):
            pass

        app_service = self._store.get_app_service_by_user_id(event.sender)

        if app_service is None:
            raise AuthError(403, "Only application services can send insertion events")

[MadLittleMods]

Using the m.room.power_levels -> events ("The level required to send specific event types. This is a mapping from event type to power level required.") field seems perfect 🎉

But we would probably want to default the power level for those events in existing rooms which don't have it set yet. Maybe default to only the creator or admins can do it if not explicitly set in power levels. Or keep it simple and not allow the history based events at all unless the power level was set. Is there any precedent for this? There is events_default but it's usually set low to allow any events.

The one problem I see is that there doesn't seem to be precedent for controlling a content field. The content.chunk_id field would also be on normal events from a non-application service user so I don't see a way to differentiate and auth it 🤔

[MadLittleMods]

Added unstable room version org.matrix.msc2716 which adds the historical power level that controls whether you can send insertion, chunk, and marker events ✅

I switched to chunk events so we can easily auth them against the PL level because they are sent by the application service user ID with the proper PL level for the room. See #10432

[MadLittleMods]

I don't think we will use this 🤷

[MadLittleMods]

Switched to backfilling the insertion event in #10420

[erikjohnston]

I think this looks good, modulo adding a get_event_txn function. Have asked in #synapse-dev:matrix.org if its safe to rip the fallback code out

[erikjohnston]

🎉

[MadLittleMods]

Woot! Thanks for all of the review @erikjohnston 🐣