Use high resolution local video but limit video size in WebRTC connection

Question

I'm developing a website for a one on one webrtc video call on mobile browsers. I also support the capturing of photos on the local stream. For the photo capture I use ImageCapture api where available, and drawing to canvas elsewhere.

The problem is that I want the video to be HD resolution 1280x720 (to reduce the bandwith needed for the call) while I need to capture the photos at 1920x1080.

What I do now is that before I capture the photo I apply constraints and set the resolution to Full HD and after capturing it I set the resolution back to HD. What happens sometimes is because of the change in resolution, the photos are not focused since the camera has to refocus.

So if I want higher resolution photos I see 2 options, of which I already tried the first one:

1. I tried having 2 separate MediaStreamTracks, one with resolution of 1280x720 that I was sending through the WebRTC connection, and one with resolution of 1920x1080 that I used to display it locally and capture the photos. This worked well on most phones, but there were some phones where one track had the video but the second track didn't work at all (it didn't display any video) (specifically iPhone 6s)
2. If possible I would only use one MediaStreamTrack with the resolution of 1920x1080 and I would limit the video size within the connection itself so it would send the lower resolution through the WebRTC connection.

So my question is, is it possible to locally use higher resolution video and then limit the video size sent through WebRTC connection to reduce bandwith usage?

Answer 1

is it possible to locally use higher resolution video and then limit the video size sent through WebRTC connection to reduce bandwith usage?

Yes, use scaleResolutionDownBy, a parameter of RTCRtpSender that does just this (except in iOS)

This parameter, originally aimed at multi-layer (simulcast) transmissions, works just fine on a single encoding as well. It's a ratio, and you set it with setParameters. There's also a maxBitrate you can control directly:

const sender = pc.addTrack(cameraTrack);

async function setParams(height, bitrate) {
  const ratio = sender.track.getSettings().height / height;

  const params = sender.getParameters();
  params.encodings[0].scaleResolutionDownBy = Math.max(ratio, 1);
  params.encodings[0].maxBitrate = bitrate;
  await sender.setParameters(params);
}

See my blog for a working demo.

Browser bugs and caveats

Unfortunately, browsers have various bugs and different stages of support around this feature:

1. Safari implements maxBitrate but not scaleResolutionDownBy which always shows as 1.
2. Firefox supports both, but still uses an early version of setParameters that needs some tweaks.

We work around this by adding a minor tweak for Firefox, and by falling back to using track.clone and applyConstraints to downscale in Safari:

// With workarounds for Firefox and Safari

const sender = pc.addTrack(cameraTrack.clone()); // Note we clone the track

async function setParams(height, bitrate) {
  const ratio = sender.track.getSettings().height / height;

  const params = sender.getParameters();
  if (!params.encodings) {
    params.encodings = [{}]; // Firefox workaround
  }
  params.encodings[0].scaleResolutionDownBy = Math.max(ratio, 1);
  params.encodings[0].maxBitrate = bitrate;
  await sender.setParameters(params);

  // Safari fallback
  if (sender.getParameters().encodings[0].scaleResolutionDownBy == 1) {
    await sender.track.applyConstraints({height});
  }
}

Here's a tweaked version of the earlier demo that should work in all browsers, including Safari on macOS.

Note: Once Safari adds native support for scaleResolutionDownBy the fiddle should pick up on the new functionality. Until then you'll unfortunately run into the same limitation on iOS you're experiencing now.

Note: Safari 14 supports scaleResolutionDownBy natively (but unsure about iOS).