View Source Simulcast

Simulcast is a technique where a client sends multiple encodings of the same video to the server and the server is responsbile for choosing and forwarding proper encoding to proper receiver (other client). The encoding selection is dynamic (i.e. SFU switches between encodings in time) and it is based on:

  • receiver awailable bandwidth
  • receiver preferences (e.g. explicit request to receive video in HD resolution instead of FHD)
  • UI layaout (e.g. videos being displayed in smaller video tiles will be sent in a lower resolution)

At the moment, Membrane supports only receiver preferences i.e. receiver can chose which encoding it is willing to receive. Additionaly, sender can turn off/on specific encoding. Membrane RTC Engine will detect changes and switch to another available encoding.

turning-simulcast-on-off

Turning simulcast on/off

On the client side simulcast can be enabled while adding a new track e.g.:

    // create MembraneWebRTC class instance
    // ...
    // add simulcasted track
    let trackId = webrtc.addTrack(track, stream, {}, {enabled: true, active_encodings: ["l", "m", "h"]});

This will add a new track that will be sent in three versions:

  • original (identified as h)
  • original scaled down by 2 (identified as m)
  • original scaled down by 4 (identified as l)

You can turn off some of the encodings by excluding them from active_encodings list. Encodings that are turned off might still be enabled using enableTrackEncoding function.

Minimal required resolution

To make all encodings working, original track resolution has to be at least 1280x720. In other case, browser might not be able to scale resolution down. In case of browser, original track resolution can be specified in constraints passed to getUserMedia or getDisplayMedia.

On the server side, simulcast can be configured while adding new WebRTC Endpoint by setting its simulcast_config option.

For example

%WebRTC{
  rtc_engine: rtc_engine,
  # ...
  simulcast_config: %SimulcastConfig{
    enabled: true,
    default_encoding: fn %Track{simulcast_encodings: _simulcast_encodings} -> "m" end
  }
}

Here we turn simulcast on and choose medium encoding for each track to be forwarded to the client.

On the other hand, setting enabled to false will result in rejecting all incoming simulcast tracks i.e. client will not send them to the server.

disabling-and-enabling-specific-track-encoding

Disabling and enabling specific track encoding

After adding simulcasted track, user can disable specific encoding. For example, to disable the highest encoding:

    // create MembraneWebRTC class instance
    // ...
    // add simulcasted track
    let trackId = webrtc.addTrack(track, stream, {}, true);
    webrtc.disableTrackEncoding(trackId, "h");

Disabled encoding can be turned on again using enableTrackEncoding function.

Membrane RTC Engine tracks encoding activity. Therefore, when some encoding is turned off, RTC Engine will detect this and switch to the highest awailable encoding. When disabled encoding becomes active again, RTC Engine will switch back to it.

selecting-encoding-to-receive

Selecting encoding to receive

By default, RTC Engine will forward the highest available encoding. However, this can be changed using selectTrackEncoding function. For example, to receive the lowest resolution:

    webrtc.selectTrackEncoding(peerId, trackId, "l");

where peerId is id of peer this track belongs to.