WebRTC onicecandidate: Am getting ICE candidates with sdpMid=audio only but not for video

Question

The browser used is Chrome... I have the caller and receiver code to generate SDP and ICE candidates. I get the caller code to generate proper SDP and ICE candidates with sdpMid=video but for the receiver, I am getting ICE candidates generated only for sdpMid=audio.

UPDATE: Here is the localSessionDescription SDP for the receiver after changes, as suggested:

 v=0
 o=- 7912682607537349212 2 IN IP4 127.0.0.1
 s=-
 t=0 0
 a=group:BUNDLE audio video
 a=msid-semantic: WMS 9f0MAtEwYGWY3pdBDI8ZtTu4dVu92R6IpEFd
 m=audio 9 UDP/TLS/RTP/SAVPF 111 103 104 9 0 8 106 105 13 126
 c=IN IP4 0.0.0.0
 a=rtcp:9 IN IP4 0.0.0.0
 a=ice-ufrag:0D1hLEwxnqReQosQ
 a=ice-pwd:Nsc4EAtefrfgzTetHjJA5lsg
 a=fingerprint:sha-256 6C:85:D8:33:D8:C6:CB:CE:D4:8E:B4:7A:C2:F5:2F:D0:67:04:25:B2:74:F9:C6:3A:2E:96:E6:56:E7:27:B0:F8
 a=setup:active
 a=mid:audio
 a=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level
 a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
 a=sendrecv
 a=rtcp-mux
 a=rtpmap:111 opus/48000/2
 a=fmtp:111 minptime=10; useinbandfec=1
 a=rtpmap:103 ISAC/16000
 a=rtpmap:104 ISAC/32000
 a=rtpmap:9 G722/8000
 a=rtpmap:0 PCMU/8000
 a=rtpmap:8 PCMA/8000
 a=rtpmap:106 CN/32000
 a=rtpmap:105 CN/16000
 a=rtpmap:13 CN/8000
 a=rtpmap:126 telephone-event/8000
 a=maxptime:60
 a=ssrc:2958641119 cname:Iu8s16HLxglPDg9k
 a=ssrc:2958641119 msid:9f0MAtEwYGWY3pdBDI8ZtTu4dVu92R6IpEFd bb63739b-cca2-4aa5-90a6-cf4bbaa199af
 a=ssrc:2958641119 mslabel:9f0MAtEwYGWY3pdBDI8ZtTu4dVu92R6IpEFd
 a=ssrc:2958641119 label:bb63739b-cca2-4aa5-90a6-cf4bbaa199af
 m=video 9 UDP/TLS/RTP/SAVPF 100 101 116 117 96
 c=IN IP4 0.0.0.0
 a=rtcp:9 IN IP4 0.0.0.0
 a=ice-ufrag:0D1hLEwxnqReQosQ
 a=ice-pwd:Nsc4EAtefrfgzTetHjJA5lsg
 a=fingerprint:sha-256 6C:85:D8:33:D8:C6:CB:CE:D4:8E:B4:7A:C2:F5:2F:D0:67:04:25:B2:74:F9:C6:3A:2E:96:E6:56:E7:27:B0:F8
 a=setup:active
 a=mid:video
 a=extmap:2 urn:ietf:params:rtp-hdrext:toffset
 a=extmap:3 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
 a=extmap:4 urn:3gpp:video-orientation
 a=sendrecv
 a=rtcp-mux
 a=rtpmap:100 VP8/90000
 a=rtcp-fb:100 ccm fir
 a=rtcp-fb:100 nack
 a=rtcp-fb:100 nack pli
 a=rtcp-fb:100 goog-remb
 a=rtcp-fb:100 transport-cc
 a=rtpmap:101 VP9/90000
 a=rtcp-fb:101 ccm fir
 a=rtcp-fb:101 nack
 a=rtcp-fb:101 nack pli
 a=rtcp-fb:101 goog-remb
 a=rtcp-fb:101 transport-cc
 a=rtpmap:116 red/90000
 a=rtpmap:117 ulpfec/90000
 a=rtpmap:96 rtx/90000
 a=fmtp:96 apt=100
 a=ssrc-group:FID 3143004909 4248148453
 a=ssrc:3143004909 cname:Iu8s16HLxglPDg9k
 a=ssrc:3143004909 msid:9f0MAtEwYGWY3pdBDI8ZtTu4dVu92R6IpEFd 778ef702-e7fc-47ea-bb3a-477e0b4262ba
 a=ssrc:3143004909 mslabel:9f0MAtEwYGWY3pdBDI8ZtTu4dVu92R6IpEFd
 a=ssrc:3143004909 label:778ef702-e7fc-47ea-bb3a-477e0b4262ba
 a=ssrc:4248148453 cname:Iu8s16HLxglPDg9k
 a=ssrc:4248148453 msid:9f0MAtEwYGWY3pdBDI8ZtTu4dVu92R6IpEFd 778ef702-e7fc-47ea-bb3a-477e0b4262ba
 a=ssrc:4248148453 mslabel:9f0MAtEwYGWY3pdBDI8ZtTu4dVu92R6IpEFd
 a=ssrc:4248148453 label:778ef702-e7fc-47ea-bb3a-477e0b4262ba

This is generated for the corresponding getUserMedia as in:

 navigator.getUserMedia({ audio: true, video: { width: 1280, height: 720 } },...

The ICE candidate generation code is:

pc.onicecandidate = function (event) {
   console.log("Generated Icecandidate:" );
   console.log(event);
   ...
 };

On console.log, I see ICE candidates, as in:

RTCIceCandidate
candidate: "candidate:211156821 1 udp 2122260223 192.168.1.5 41811 typ host generation 0 ufrag kV5Snl0LQhJlYujt"
sdpMLineIndex:0
sdpMid:"audio"

Needless to say, I am not able to get the remote video displayed. I am trying this on a local network so really even STUN is not required.

I would like to know, why I am not getting any ICE candidates for sdpMid=video. Also, of the four ICE candidates generated,three ICE candidates have a sdpMLineIndex:0 and one ICE candidate has candidate property as null!

UPDATE 1: I got the answer to the last issue... Candidate property as Null. "Note: the RTCPeerConnection.onicecandidate will be called once with an empty candidate property to signal the end of trickle ICE event." This is explained here.

On the caller side, I get more than 10 ICE candidates, some with audio and some with video.

Where am I going wrong?

UPDATE 2: Here is the code for the receiver part which does not generate ICE candidates for video. I have stripped authentication and other sections to focus only on the relevant part. I removed caching of ICE candidate and send it as it comes:

$(document).ready(function () {

  var socket = io.connect();
  var pc = new RTCPeerConnection ({
    "iceServers": [{"url": "stun:stun.l.google.com:19302"}]
  });

  pc.onicecandidate = function (event) {
    socket.emit('candidateFromReceiver',event.candidate);
    console.log("Candidate Generated:");
    console.log(event.candidate);
  }; 

  pc.onaddstream = function(ev) {
    stream = ev.stream;        
    var video = $('#vid2'); 
    video.attr('src', URL.createObjectURL(stream));
    video.onloadedmetadata = function(e) {
      video.play();
    }
  };

  socket.on('connect',function() { console.log("Socket connected"); });
  socket.on('candidateFromCaller', function (data) {
      pc.addIceCandidate(new RTCIceCandidate(data));
  });

  navigator.getUserMedia = navigator.getUserMedia || navigator.webkitGetUserMedia ||
                       navigator.mozGetUserMedia;
  if (navigator.getUserMedia) {
    navigator.getUserMedia({ audio: true, video: { width: 1280, height: 720 } },
      function(stream) {
         var video = $('#vid1'); 
         video.attr('src', URL.createObjectURL(stream));
         video.onloadedmetadata = function(e) {
           video.play();
         }
     pc.addStream(stream);
      },error);

    socket.on('sdpOffer', function(data) {
      var sdpOffer = new RTCSessionDescription(data.sdpOffer);
      pc.setRemoteDescription(sdpOffer, function() {
        pc.createAnswer(function(sdpAnswer) {
          localSessionDescription = new RTCSessionDescription(sdpAnswer);
          pc.setLocalDescription(localSessionDescription, function() {
            socket.emit('sdpAnswer',localSessionDescription);
          },error);
        }, error);
      },error);
    });
  }

  function error(err) {
    console.log("ERROR!!!!");
    console.log(err);
  }

}); // End of document.ready function

If I insert code, just after getting user media, to generate an Offer (as I have in the caller code), the ICE candidates generated include ones for video too. Of course, that was only for testing as the rest of the code bombs after that, as expected.

Answer 1

(It sounds from comments that you're caching ICE candidates. Don't do that. I also suspect timing issues may be behind loss of some candidates.)

The whole point of Trickle ICE is to trickle candidates, that is, send candidates as soon as they become available.

With WebRTC, your app is responsible for signaling between peers, which is time-sensitive. So:

1. Send pc.localDescription no later than in the setLocalDescription success callback.
2. Expect pc.onicecandidate to start firing immediately after that callback. Send them.

This is true on both sides (for offer and answer). What you want to see on the wire is:

offer, candidate, candidate, candidate

and the other way:

answer, candidate, candidate, candidate

What not to do:

• Don't cache ICE candidates.
• Don't wait until you have an answer back, that just wastes time.
• Don't delay calling setRemoteDescription when an offer comes in on the receiving end for any reason, or it wont be ready to receive candidates.

Update 2:

Your sdp says a=recvonly and not a=sendrecv, which means the receiver is resigned to receive only, without send anything in return. One of two things can cause this:

1. Caller set createOffer options like offerToReceiveVideo:false and/or offerToReceiveAudio:false.
2. Receiver did not called pc.addStream in time for (before) pc.setLocalDescription.

The second can happen if there's a race between getUserMedia and receiving an offer.

Update 3:

If all else fails, compare to working code. I've shared a cross-tab demo before in other answers, but it only sent video, didn't receive any.

Here's a modified version of that demo that only receives video from the remote camera instead. As usual, open it in two tabs in the same browser.

Note that in Firefox, after you hit Call, you have to physically focus the other tab before it allows access to the camera.

Answer 2

What you are dealing with is called bundeling. Offerer and answerer agree to bundle all ICE transports into a single ICE transport. Therefore you only get a single ICE candidate for the first m-section.

The offerer is still giving you all these ICE candidates for the video m-section, because it does not know if the answerer is going to agree to use bundle.

As AlexD points out in his answer you can influence this behavior on the offerer side via the "bundlePolicy". "maxBundle" as policy will for example result in the offerer assuming the answerer is going to understand bundleling and therefore only create ICE candidates for a single transport.

But as long as the offerer is offering bundle the answerer is going to use it if it supports it.

Answer 3

I was able to solve this by setting:

rtcConfiguration.bundlePolicy = "max-compat"

See: http://w3c.github.io/webrtc-pc/#dom-rtcbundlepolicy-max-compat