Some users stop getting bridged

[MaximilianGaedig]

I am using mautrix-facebook to bridge some group chats, sometimes when a user sends a message on facebook the message is not getting bridged, and this shows up in the logs:

[timestamp] [ERROR@mau.portal.[portal id] Error handling Facebook message mid.$[message id]
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/mautrix_facebook/portal.py", line 1555, in handle_facebook_message
    await self._handle_facebook_message(source, sender, message, reply_to)
  File "/usr/lib/python3.10/site-packages/mautrix_facebook/portal.py", line 1657, in _handle_facebook_message
    await self._send_message(
  File "/usr/lib/python3.10/site-packages/mautrix/bridge/portal.py", line 432, in _send_message
    event_id = await intent.send_message_event(self.mxid, event_type, content, **kwargs)
  File "/usr/lib/python3.10/site-packages/mautrix/appservice/api/intent.py", line 141, in wrapper
    return await __method(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/mautrix/appservice/api/intent.py", line 427, in send_message_event
    return await super().send_message_event(room_id, event_type, content, **kwargs)
  File "/usr/lib/python3.10/site-packages/mautrix/client/api/events.py", line 450, in send_message_event
    resp = await self.api.request(
  File "/usr/lib/python3.10/site-packages/mautrix/api.py", line 395, in request
    resp_data, resp = await self._send(
  File "/usr/lib/python3.10/site-packages/mautrix/api.py", line 250, in _send
    raise make_request_error(
mautrix.errors.request.MForbidden: eventauth: sender "@facebook_[facebook user id]:[domain]" not in room

Then the user won't get bridged until I ban and unban them in the room.
I couldn't find any pattern on why this happens, it seems like it is random.
I logged into the PostgreSQL database and saw that the affected users are not in the mx_user_profile table.

This is happening for a while already (1-2 months).
Version: dock.mau.dev/mautrix/facebook:latest (Image ID 0180dfa0f970, from 15th January), always getting updated using watchtower.