Is it possible to detect who is speaking with Twilio Video JS SDK?

Question

I'm currently implement video call application, something like Google Hangout using Twilio Video. I want to show the video of person who is speaking at that time in the background.

Is it possible to detect who is speaking with Twilio Video JS SDK ?

Answer 1

Edit

The Twilio Video SDKs now include support for detecting and updating the dominant speaker. Please see the documentation for full details.

Original answer

Twilio developer evangelist here.

There is nothing currently built into the SDK that makes that possible. However, you could try audio analysis using the Web Audio API to determine a moving average of the loudest remote audio track and use that to display the current speaker. This is not something I've tried just yet, but I think it would be interesting to experiment with.

Answer 2

In the end, I managed to get the result I want using getStats() method using Twilio Video JS SDK. I'll put the example code here so it could be useful for people who are looking to implement the same functionality.

Reference: https://media.twiliocdn.com/sdk/js/video/releases/1.10.0/docs/Room.html

  room.getStats().then(statsReportArray => {
    statsReportArray.forEach(report => {
      let maximumAudioLevel = 0;
      let loudestTrackId = null;

      report.remoteAudioTrackStats.forEach(audioStat => {
        if (audioStat.audioLevel > maximumAudioLevel) {
          maximumAudioLevel = audioStat.audioLevel;
          loudestTrackId = audioStat.trackId;
        }
      });