[Story] EX: Resize media before sending

[manuroe]

• As a user
• I want to send and receive media quickly
• So that I can enjoy my fast EX app even more

It will also improve everyone bandwidth usage.

Acceptance criteria

• The app will upload media with a reduced quality by default.
• The media is automatically resized. There is NO additional step in the sending flow
• Uploaded images are resized with a max edge size of 2048px (and quality of 0.78 for JPEGs)
• Uploaded videos are resized to 480p 720p
• Orientation and aspect ration are both preserved.
• Media with a smaller resolution than the selected quality setting are not resized
• There is a global setting to disable this - when off, media is sent with same processing as we currently use.
• Formats:
  • PNGs are optimised as PNG.
  • GIFs are not optimised to preserve the animations.
  • All other image formats are optimised as JPEG.
  • Videos are optimised as MP4 (H.264/AAC).
• We have analytics to track how many people toggle the setting.

Summary

Image Preset	Resolution	File size	JPEG Quality	Format
Optimised	2048	~200-650KB	0.78	JPEG/GIF/PNG
Optimised off	Original	Original	N/A	Original

Video Preset	Output	Format
Optimised	720p	MP4
Optimised off	1080p	MP4

Leads

• Tech: @pixlwave
• Design:

Time sheeting

ER-97 EX: Persistent Event Cache & Upload Improvements

Size estimate

None

Dependencies

• None

Out of scope

• Nothing

Open questions

Questions

Beta Give feedback

1. Ok for a global setting?
2. What should be the different quality levels and their associated resolutions?
3. Should we resample the audio too? --> No, it is a very specific case

Subtasks

Android

Beta Give feedback

1. [Task] Resize media before sending element-x-android#3651
   T-Task +1
   

iOS

Beta Give feedback

1. [Task] Resize media before sending element-x-ios#3398
   T-Task +1

Rust

Beta Give feedback

No tasks being tracked yet.

Other

Beta Give feedback

No tasks being tracked yet.

Bugs

Beta Give feedback

No tasks being tracked yet.

Sign-off

Android

• Design sign-off on completion
• QA sign-off on completion
• Product sign-off on completion

iOS

• Design sign-off on completion
• QA sign-off on completion
• Product sign-off on completion

[pixlwave]

Dumping the various Image/Video qualities I've found testing apps today. However this isn't quite as simple as these tables make it seem.

Taking signal as an example, some of the apps seem to have a selection of image presents and the user setting maps to a default preset. However, when the default preset exceeds the target file size, resizing is attempted again with a slightly worse preset until the target size is achieved. This seems somewhat overkill without properly validating a bunch of size+quality combinations to me.

Signal

Image Preset	Resolution	File size	JPEG Quality
High	4096	3MB	0.55
Standard	2048	1.5MB	0.6
Standard (share extension)	1600	1MB	0.6

Video Preset	Output
All	480p

iMessage

Image Preset	Resolution	File size	JPEG Quality
Normal	Original	Up to 25MB?	0.93
Low	1280	300KB	0.86

Video Preset	Output
All	720p

Whatsapp

Image Preset	Resolution	File size	JPEG Quality
HD	5120	3MB	0.75
Standard	2048	550KB	0.8

Video Preset	Output
HD 😂	720p
Standard	480p

Slack

Image Preset	Resolution	File size	JPEG Quality
Non-optimised	Passthrough	Passthrough	Passthrough
Optimised	Original	Varies	0.78

Video Preset	Output
Original	Passthrough
Optimised	720p

Discord

Image Preset	Resolution	File size	JPEG Quality
All	Original	Varies	0.63

Video Preset	Output
All	480p

[f35f0ef9d0e827dae86552d3899f78fc]

Instead of targeting a byte size, how about the way the older Element iOS app behaved? I believe it used a constant calculation (simpler) and offered Small, Medium, or Large choices to the user. Each size choice corresponded to a relatively static percentage reduction of the image's dimensions.

I am not a developer, more like a guru, but I think this linked code is related.

[pixlwave]

Instead of targeting a byte size

Yeah don't worry, this seems like an unpredictable "quality" setting to me. The post above is only illustrative of how other apps work as some background research.

[ara4n]

while i'm very happy to see that we may let users choose whether to optimise or not... i'm worried that particularly for video, we're giving the option of 480p (awful) and original (which could well be gigantic 4K@60fps on an iPhone 16).

Can i make a plea to offer both "low", "high" and "original" or something? So 480p, 720p and original in the case of video (similar to what WhatsApp is doing)? Otherwise I can see us being drowned in a 50/50 mix of "why is Element's video quality so crap?" and "why are Element's videos so big?" bugs.

[pixlwave]

Just to clarify, we don't upload 4K video, it has always been processed using Apple's AVAssetExportPreset1920x1080 preset before being handed to the SDK.

[mxandreas]

Otherwise I can see us being drowned in a 50/50 mix of "why is Element's video quality so crap?"

I agree with this; I think these days most users value/expect high-quality more than their bandwidth. I would not add a 3rd option - just use 720 instead of 480. We can always add 3rd option later if this becomes necessary.

@ara4n It is worth mentioning though that 480 was picked because this is what everybody else was doing.

[fibcom-bp]

@pixlwave thanks for the comparrision to other messengers. What is the reason for EX having the optimization turned off by default although the others have it turned on by default? Personally i would like to have optimization turned ON by default. This saves a lot of storage/bandwidth and is what the users are used to from their personal messenger apps.

[ricardo-duarte-av]

A few questions if i may:

Won't it be a bit deceiving to users that, with optimize OFF a video will still be tampered with ?
Worse, It's stated that it will be resized to 1080p MP4.
MP4 is a container, if i record in h265 i'll have my files being converted (silently) to h264.
Worse still, we might not even see that great of file size reduction:

(this is highly subjective to encoding settings and the actual media being encoded)

Having this be a global setting isn't too limiting ? Perhaps a toggle before upload to Disable optimization (if it's enable globally) or Enable optimization (if disabled globally) would be preferable ?

[pixlwave]

@pixlwave thanks for the comparrision to other messengers. What is the reason for EX having the optimization turned off by default although the others have it turned on by default? Personally i would like to have optimization turned ON by default. This saves a lot of storage/bandwidth and is what the users are used to from their personal messenger apps.

It's been decided to default this to on (the initial plan was mine as iMessage (and Slack iirc) defaults to off and it seemed undesirable for a user to be sending reduced quality images without knowing this, but as I was the only person who thinks this, we changed the plan 😢)

Won't it be a bit deceiving to users that, with optimize OFF a video will still be tampered with ? Worse, It's stated that it will be resized to 1080p MP4. MP4 is a container, if i record in h265 i'll have my files being converted (silently) to h264. Worse still, we might not even see that great of file size reduction:

The TL;DR on this is that h.264 is essentially universally compatible, whereas h.265/VP9/AV1 don't have such ubiquitous support. So by always running the video through a 1080p encoding, we ensure reasonable file size (as compared to 4K), along with much wider device compatibility. So yes, we might end up with increased file sizes in some instances, but this comes with the benefit that everybody should be able to play the video. (We have been sending videos like this from EXI since we added media uploads and we've not had any complaints about it. As an aside, it is also an easy way to make sure that any GPS location data is removed from uploaded videos without having to handle all the individual video containers individually).

Having this be a global setting isn't too limiting ? Perhaps a toggle before upload to Disable optimization (if it's enable globally) or Enable optimization (if disabled globally) would be preferable ?

This is very much a first pass at supporting smaller file uploads and we've already discussed internally that we're likely to come back to it to look at such enhancements in the future as there are clearly various tradeoffs being made here.

[daedric7]

It's been decided to default this to on (the initial plan was mine as iMessage (and Slack iirc) defaults to off and it seemed undesirable for a user to be sending reduced quality images without knowing this, but as I was the only person who thinks this, we changed the plan 😢)

You aren't. I'm also not against compression/transcoding, just agains the silent type :)

The TL;DR on this is that h.264 is essentially universally compatible, whereas h.265/VP9/AV1 don't have such ubiquitous support. So by always running the video through a 1080p encoding, we ensure reasonable file size (as compared to 4K), along with much wider device compatibility. So yes, we might end up with increased file sizes in some instances, but this comes with the benefit that everybody should be able to play the video. (We have been sending videos like this from EXI since we added media uploads and we've not had any complaints about it. As an aside, it is also an easy way to make sure that any GPS location data is removed from uploaded videos without having to handle all the individual video containers individually).

Solid facts, h264 has become the jpeg of video. Yes there are better formats, but support is not so great. As for no complains in EXI, i assure you there were at least observations on EXA (i did notice that my screen recordings became... different). The removing of private info from EXIF and others is a must have so no arguments there.

This is very much a first pass at supporting smaller file uploads and we've already discussed internally that we're likely to come back to it to look at such enhancements in the future as there are clearly various tradeoffs being made here.

I think i should say that i didn't feel that this was decided by the flip of a coin. Discussion must have been held, arguments on the pros and the cons must have been put forward and the best decision, at the time, with the info that was available, and the time available to decide was set in code.

As a server owner, i should be preoccupied with the amount of space media will take, and as such compression/transcoding is a must have. The media upload (and download) screens are currently very bland and optionless, i hope in the future we get more info transmited to the user during the preview and before the upload, as well as some options regarding how media should be handled localside (oh please let me draw on images!!) before being uploaded :)