Synapse 1.111.0 in Worker mode does not register new authenticated media endpoints #17473

AlexanderBabel · 2024-07-23T13:59:23Z

Description

I use a workers configuration for my synapse with a media worker. I already updated my ingress to forward the new authenticated endpoints to the media worker instance. But all of them are returned with 404.

It looks like that the new endpoints are not registered on startup on the media server

Steps to reproduce

Use the 1.111.0 synapse version
Try to upload an image through the new endpoints/Retrieve images through the new endpoint

Homeserver

babel1.eu

Synapse Version

v1.111.0

Installation Method

Docker (matrixdotorg/synapse)

Database

postgres

Workers

Multiple workers

Platform

Ubuntu 22.04, Kubernetes

Configuration

general-config:
  enable_media_repo: false

media-worker-config:
  worker_app: "synapse.app.media_repository"
  
  worker_listeners:
    - port: 9090
      tls: false
      bind_addresses: ["::"]
      type: http
  
      resources:
        - names: [metrics]
          compress: false
  
    - port: 8083
      tls: false
      bind_addresses: ["::"]
      type: http
      x_forwarded: true
  
      resources:
        - names:
            - media
          compress: false
  
  worker_log_config: /synapse/config/log.yaml

Relevant log output

2024-07-23 13:52:49,139 - twisted - 282 - INFO - sentinel- SynapseSite starting on 9090
2024-07-23 13:52:49,139 - synapse.app._base - 439 - INFO - sentinel- Synapse now listening on TCP port 9090
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for POST /_matrix/media/v1/create
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for POST /_matrix/media/(r0|v3|v1)/upload$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for PUT /_matrix/media/v3/upload/(?P<server_name>[^/]*)/(?P<media_id>[^/]*)$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for GET /_matrix/media/(r0|v3|v1)/download/(?P<server_name>[^/]*)/(?P<media_id>[^/]*)(/(?P<file_name>[^/]*))?$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for GET /_matrix/media/(r0|v3|v1)/thumbnail/(?P<server_name>[^/]*)/(?P<media_id>[^/]*)$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for GET /_matrix/media/(r0|v3|v1)/preview_url$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for GET /_matrix/media/(r0|v3|v1)/config$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for POST ^/_synapse/admin/v1/purge_media_cache$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for POST ^/_synapse/admin/v1/room/(?P<room_id>[^/]*)/media/quarantine$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for POST ^/_synapse/admin/v1/quarantine_media/(?P<room_id>[^/]*)$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for POST ^/_synapse/admin/v1/media/quarantine/(?P<server_name>[^/]*)/(?P<media_id>[^/]*)$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for POST ^/_synapse/admin/v1/media/unquarantine/(?P<server_name>[^/]*)/(?P<media_id>[^/]*)$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for POST ^/_synapse/admin/v1/user/(?P<user_id>[^/]*)/media/quarantine$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for POST ^/_synapse/admin/v1/media/protect/(?P<media_id>[^/]*)$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for POST ^/_synapse/admin/v1/media/unprotect/(?P<media_id>[^/]*)$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for GET ^/_synapse/admin/v1/room/(?P<room_id>[^/]*)/media$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for POST ^/_synapse/admin/v1/media/delete$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for POST ^/_synapse/admin/v1/media/(?P<server_name>[^/]*)/delete$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for DELETE ^/_synapse/admin/v1/media/(?P<server_name>[^/]*)/(?P<media_id>[^/]*)$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for GET ^/_synapse/admin/v1/users/(?P<user_id>[^/]*)/media$
2024-07-23 13:52:49,140 - synapse.http.server - 485 - DEBUG - sentinel- Registering for DELETE ^/_synapse/admin/v1/users/(?P<user_id>[^/]*)/media$
2024-07-23 13:52:49,140 - synapse.util.httpresourcetree - 56 - INFO - sentinel- Attaching <synapse.rest.health.HealthResource object at 0xe63816596b90> to path b'/health'
2024-07-23 13:52:49,141 - synapse.util.httpresourcetree - 56 - INFO - sentinel- Attaching <synapse.rest.media.media_repository_resource.MediaRepositoryResource object at 0xe638165c30d0> to path b'/_matrix/media/r0'
2024-07-23 13:52:49,141 - synapse.util.httpresourcetree - 56 - INFO - sentinel- Attaching <synapse.rest.media.media_repository_resource.MediaRepositoryResource object at 0xe638165c30d0> to path b'/_matrix/media/v3'
2024-07-23 13:52:49,141 - synapse.util.httpresourcetree - 56 - INFO - sentinel- Attaching <synapse.rest.media.media_repository_resource.MediaRepositoryResource object at 0xe638165c30d0> to path b'/_matrix/media/v1'
2024-07-23 13:52:49,141 - synapse.util.httpresourcetree - 56 - INFO - sentinel- Attaching <synapse.http.server.JsonResource object at 0xe63816368250> to path b'/_synapse/admin'
2024-07-23 13:52:49,141 - twisted - 282 - INFO - sentinel- SynapseSite starting on 8083
2024-07-23 13:52:49,141 - synapse.app._base - 439 - INFO - sentinel- Synapse now listening on TCP port 8083


2024-07-23 13:57:27,646 - synapse.access.http.8083 - 428 - DEBUG - GET-56- ::ffff: - 8083 - Received request: GET /_matrix/client/v1/media/download/babel1.eu/OHagbmtHsZRLwhrdDCGxRBOQ?allow_redirect=true
2024-07-23 13:57:27,647 - synapse.http.server - 130 - INFO - GET-56- <XForwardedForRequest at 0xe638160dc7d0 method='GET' uri='/_matrix/client/v1/media/download/babel1.eu/OHagbmtHsZRLwhrdDCGxRBOQ?allow_redirect=true' clientproto='HTTP/1.1' site='8083'> SynapseError: 404 - Unrecognized request
2024-07-23 13:57:27,648 - synapse.access.http.8083 - 473 - INFO - GET-56- ::ffff: - 8083 - {None} Processed request: 0.001sec/-0.000sec (0.001sec, 0.000sec) (0.000sec/0.000sec/0) 59B 404 "GET /_matrix/client/v1/media/download/babel1.eu/OHagbmtHsZRLwhrdDCGxRBOQ?allow_redirect=true HTTP/1.1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) ElementNightly/2024072201 Chrome/126.0.6478.127 Electron/31.2.0 Safari/537.36" [0 dbevts]
2024-07-23 13:57:27,655 - synapse.replication.tcp.handler - 592 - DEBUG - process-replication-data-507- Received rdata presence (master) -> 78074332

Anything else that would be useful to know?

No response

tulir · 2024-07-23T14:10:50Z

The issue is that client and federation are not listed in resources, but I'm not sure if requiring them for the new endpoints is intentional or not. If it's intentional, it should probably be highlighted in release notes. If not, then it's a bug.

devonh · 2024-07-23T23:32:52Z

Agreed. Either we should:

update the docs to mention adding client and federation resources during worker configuration
...or
have the media resource also start the relevant client/federation servlets for the new endpoints

devonh · 2024-07-24T15:07:15Z

I think the preferred approach going forward should be to ensure that when specifying the media resource, all appropriate handlers are started in that situation.

…#17483) This is in response to issue #17473. Not all the necessary handlers to deal with media requests are started now when configuring synapse to use a media worker as per the [example config](https://element-hq.github.io/synapse/latest/workers.html#synapseappmedia_repository). The new media endpoints introduced with authenticated media fall under the `client` & `federation` handlers in synapse. This PR starts up handlers for the new media endpoints if a worker has been configured with only the `media` resource type. ### Pull Request Checklist  * [X] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [X] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: Andrew Morgan <[email protected]>

See: element-hq/synapse#17473

4censord · 2024-09-03T10:47:43Z

It might be nice to add a link to this to the upgrade notes as well.
As of right now, this is really confusing because it still does not work, even after modify the reverse proxy config as instructed, if you are deploying with a seperate media worker.
(https://element-hq.github.io/synapse/latest/upgrade)

tulir · 2024-09-03T11:19:41Z

I don't think there's any point in adding upgrade notes for a bug that was already fixed and released 🤔

@devonh the issue can probably be closed now?

4censord · 2024-09-03T14:24:42Z

Ah, then I seem to have overlooked it in the release notes.

4censord · 2024-09-13T13:05:04Z

Just to clarify, in case someone else is confused with this: As of 1.114, this fix is included. 1.113 does not have it yet.

devonh added A-Docs A-Media-Repository O-Frequent S-Minor T-Defect labels Jul 23, 2024

This was referenced Jul 25, 2024

Start handlers for new media endpoints when media resource configured #17483

Merged

Media workers doc #17484

Closed

Chr1s70ph mentioned this issue Jul 30, 2024

Room avatar missing and files not loading element-hq/element-desktop#1808

Closed

archlinux-github pushed a commit to archlinux/infrastructure that referenced this issue Aug 11, 2024

matrix: Work around media worker bug

b3d0687

See: element-hq/synapse#17473

devonh closed this as completed Sep 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Synapse 1.111.0 in Worker mode does not register new authenticated media endpoints #17473

Synapse 1.111.0 in Worker mode does not register new authenticated media endpoints #17473

AlexanderBabel commented Jul 23, 2024 •

edited

Loading

tulir commented Jul 23, 2024

devonh commented Jul 23, 2024

devonh commented Jul 24, 2024

4censord commented Sep 3, 2024 •

edited

Loading

tulir commented Sep 3, 2024

4censord commented Sep 3, 2024

4censord commented Sep 13, 2024 •

edited

Loading

Synapse 1.111.0 in Worker mode does not register new authenticated media endpoints #17473

Synapse 1.111.0 in Worker mode does not register new authenticated media endpoints #17473

Comments

AlexanderBabel commented Jul 23, 2024 • edited Loading

Description

Steps to reproduce

Homeserver

Synapse Version

Installation Method

Database

Workers

Platform

Configuration

Relevant log output

Anything else that would be useful to know?

tulir commented Jul 23, 2024

devonh commented Jul 23, 2024

devonh commented Jul 24, 2024

4censord commented Sep 3, 2024 • edited Loading

tulir commented Sep 3, 2024

4censord commented Sep 3, 2024

4censord commented Sep 13, 2024 • edited Loading

AlexanderBabel commented Jul 23, 2024 •

edited

Loading

4censord commented Sep 3, 2024 •

edited

Loading

4censord commented Sep 13, 2024 •

edited

Loading