MSC4030: Progressive image in Matrix

proposals/MSC4030: Progressive images in Matrix.md

@@ -0,0 +1,82 @@
+# MSC4030: Progressive Images in Matrix
+
+## Introduction
+
+Efficient image handling is essential for delivering a seamless user experience within a communication platform like Matrix.
+This proposal recognizes the potential of modern progressive image formats to enhance image management within Matrix.
+
+While this document will occasionally highlight JPEG XL (referred to as 'jxl') as a notable use case, it is important to note
+that the concepts discussed herein are applicable to both progressive and non-progressive image formats,
+including WebP2 and JPEG itself.
+
+## Clarifications
+
+In Matrix, there are two types of thumbnails:
+
+- **Server-side generated thumbnails** available at the `thumbnail/` endpoint.
+- **Event thumbnails**, uploaded by clients and referenced using a separate MXC.
+
+This distinction exists because server-side generated thumbnails cannot be served in encrypted rooms or for media types other than `m.image`.
+
+### Reasons behind these changes
+
+The Matrix specification currently does not take into account that clients may benefit from receiving content in formats
+that align with their capabilities. By introducing media negotiation mechanisms, Matrix can enhance the client experience
+by serving content in the most efficient way possible.
+
+See [Issue #453](https://github.com/matrix-org/matrix-spec/issues/453) for further details.
+
+## Encrypted Rooms
+
+Due to the inability to re-encode images server-side, it is necessary to maintain the existing structure and functionality of the current API.
+
+To enhance user experience, it would be ideal loosening the requirements for progressive images in `m.image` events
+by eliminating the need for an event thumbnail (`"thumbnail_file"` entry). This change reduces storage costs,
+as progressive images do not need thumbnails. For proper functionality, client implementations should support on-the-fly
+decoding and decryption of media during download.
+
+In summary, if an image is progressive, clients MAY NOT generate the `"thumbnail_file"` entry (and the associated resources)
+in `m.image` messages sent in encrypted rooms.
+
+## Unencrypted Rooms
+
+Since we can re-encode the image server-side it would be optimal to serve the image (`/download` endpoint) and the server-side thumbnails
+(`/thumbnail` endpoint) with the best format that align with the client capabilities.
+
+### Eliminating Unwanted Behavior
+
+Clients do not acknowledge that, in unencrypted rooms, server-side generated thumbnails are available for image resources.
+As a result, they unnecessarily generate client-side thumbnails, upload them to the server, and reference them in the `m.image` event,
+creating redundant and unnecessary resources.
+
+To fix this issue, clients SHOULD NOT generate the `"thumbnail_file"` entry (and the associated resources) in `m.image` messages
+sent in unencrypted rooms.
+
+### Supporting Different Formats
+
+As proposed in [MSC4011](https://github.com/matrix-org/matrix-spec-proposals/pull/4011), media negotiation MUST be performed
+using the [HTTP Content Negotiation mechanism](https://developer.mozilla.org/en-US/docs/Web/HTTP/Content_negotiation).
+This mechanism allows the delivery of images in progressive or more modern formats to clients that support such features.
+
+If the `Accept` header field is unspecified, absent or list unsupported formats, the server MUST provide the image resource
+as `image/png` or `image/jpeg`.
+
+All progressive-capable formats MUST always provide the image as a progressive one, even if the image is uploaded
+as a non-progressive image in a progressive-capable format. When serving the content, the server MUST provide the image
+in the same format (and all others) as a progressive image.
+
+If the negotiation is successful and a different non-progressive format is chosen for the image, the server MAY compare
+the size between the image in the transcoded format and the image in the original format, if the image in the original format
+is smaller than the transcoded one and is supported by the client, the server could chose to send the image in
+the original format instead of the transcoded one. However, this behavior SHOULD NOT be implemented for progressive formats.
+
+Clients SHOULD NOT connect to the `/thumbnail` endpoint if the server can provide images in progressive formats,
+supported by the client, at the `/download` endpoint.
+
+*Enabling this feature allows the server to store images in more efficient formats such as JPEG XL, reducing storage costs.
+These images can be quickly served as JPEG for clients that do not support the jxl format. Additionally,
+JPEG XL can perform lossless conversion of PNG and JPEG, further reducing storage costs and bandwidth usage.*
+
+## Alternatives
+
+[MSC4011 - Thumbnail media negotiation](https://github.com/matrix-org/matrix-spec-proposals/blob/clokep/thumbnail-media-negotiation/proposals/4011-thumbnail-media-negotiation.md).