Hi there,

during development of our app that uses Livekit as the video call framework, we discovered that the video quality is very low on some Android devices. I tracked down at least one bug (and a combination of heuristics combined with some bad luck I guess) in WebRTC, so I'm reporting this here and not in https://github.com/livekit/client-sdk-flutter/, the entry point we use.

The issue

We set the defaultCameraCaptureOptions to VideoParametersPresets.h720_43 (i.e. 960x720) with simulcast enabled. On some Android phones, the sent simulcast layers are 144x192 and 288x384, even on local network with a local livekit server. This is ~16% of the requested resolution, and definitely not suitable for high quality video conferencing.

The deep dive into WebRTC

After ruling out network issues, I started a deep dive into client-sdk-flutter and subsequently into WebRTC itself. Here are my findings (on a Fairphone 5 with Android 13):

• Livekit room options:

RoomOptions(
  adaptiveStream: true,
  dynacast: true,
  defaultAudioCaptureOptions:
      AudioCaptureOptions(deviceId: _settings.audioDeviceId),
  defaultCameraCaptureOptions: CameraCaptureOptions(
      deviceId: _settings.videoDeviceId,
      params: VideoParametersPresets.h720_43
)));

• Camera2Session lists following available capture formats: [4080x3060, 4000x3000, 4000x2250, 3960x1760, 3840x2160, 3264x2448, 3072x3072, 3060x3060, 2700x1200, 2592x1944, 2048x1536, 1920x1080, 1600x1200, 1440x1080, 1280x960, 1280x720, 1200x1200, 1080x1080, 1024x768, 800x600, 800x480, 720x480, 640x480, 640x360, 352x288, 320x240, 176x144]
• based on the requested resolution of 960x720, Camera2Session uses the nearest (in terms of total pixels) available format, namely 1024x768: Camera2Session: Using capture format: 1024x768@[7.0:30.0]
• The first major reduction of resolution happens on the boundary between Java and C++:

  webrtc/media/base/video_adapter.cc

  Lines 234 to 236 in 7ddfc43

  	const Fraction scale =
  	FindScale(*cropped_width, *cropped_height, target_pixel_count,
  	max_pixel_count, variable_start_scale_factor_);

  • Using the requested pixel count (960x720=691200) as maximum, the video adapter tries to get the input video (1024x768=786432) below that maximum. It only uses 2/3 and 3/4 as factors for efficiency reasons. 3/4 on each axis reduces the number of overall pixels to 9/16 which is ~56%. The current resolution is now 768x576
• This downscaling causes simulcast to only send 2 layers instead of 3 because it doesn't reach the 960*540 bucket anymore:

  webrtc/video/config/simulcast.cc

  Lines 82 to 106 in 7ddfc43

  	constexpr const SimulcastFormat kSimulcastFormatsVP8[] = {
  	{1920, 1080, 3, webrtc::DataRate::KilobitsPerSec(5000),
  	webrtc::DataRate::KilobitsPerSec(4000),
  	webrtc::DataRate::KilobitsPerSec(800)},
  	{1280, 720, 3, webrtc::DataRate::KilobitsPerSec(2500),
  	webrtc::DataRate::KilobitsPerSec(2500),
  	webrtc::DataRate::KilobitsPerSec(600)},
  	{960, 540, 3, webrtc::DataRate::KilobitsPerSec(1200),
  	webrtc::DataRate::KilobitsPerSec(1200),
  	webrtc::DataRate::KilobitsPerSec(350)},
  	{640, 360, 2, webrtc::DataRate::KilobitsPerSec(700),
  	webrtc::DataRate::KilobitsPerSec(500),
  	webrtc::DataRate::KilobitsPerSec(150)},
  	{480, 270, 2, webrtc::DataRate::KilobitsPerSec(450),
  	webrtc::DataRate::KilobitsPerSec(350),
  	webrtc::DataRate::KilobitsPerSec(150)},
  	{320, 180, 1, webrtc::DataRate::KilobitsPerSec(200),
  	webrtc::DataRate::KilobitsPerSec(150),
  	webrtc::DataRate::KilobitsPerSec(30)},
  	// As the resolution goes down, interpolate the target and max bitrates down
  	// towards zero. The min bitrate is still limited at 30 kbps and the target
  	// and the max will be capped from below accordingly.
  	{0, 0, 1, webrtc::DataRate::KilobitsPerSec(0),
  	webrtc::DataRate::KilobitsPerSec(0),
  	webrtc::DataRate::KilobitsPerSec(30)}};

• The simulcast layers are sorted by ascending resolution and originally have scale factors 4, 2 and 1
• Finally, due to a bug in the EncoderStreamFactory, the two remaining layers (768x576 and 384*288) are further reduced to 384x288 and 192x144, because they are mapped to the scale factors 4 and 2, not 2 and 1:

  webrtc/video/config/encoder_stream_factory.cc

  Line 358 in 7ddfc43

  layers[i].active = encoder_config.simulcast_layers[i].active;

The solution(s)

• one of
  • don't allow Camera2Session to take larger capture format than requested to prevent VideoAdapter to half the video stream later on
  • allow the VideoAdapter a slightly increased maximum resolution, similar to max_roundup_rate in simulcast layer selection
• maybe add 4:3 tables for simulcast layer selection to compensate for the lower resolution of 4:3 videos compared to 16:9 videos for same height. Currently 4:3 videos tend to fall into lower buckets (with lower layer count)
• fix layer mapping bug (PR follows, EDIT here it is: fix: correctly map selected layers to simulcast layers #134)

one more thing

With and without the bug fix, livekit server reports a "Good" connection quality for that phone despite being on the same network. On a different phone, that has 960x720 in its native capture formats, layer selection works and connection quality is "Excellent". This is likely a bug in livekits scorer that I will investigate separately. I still wanted to mention it here because it was the reason for my investigations