1

How does audio and video in a webrtc peerconnection stay in sync? I am using an API which publishes audio and video (I assume as one peer connection) to a media server. The audio can occasionally go out of sync up to 200ms. I am attributing this to the possibility that the audio and video are separate streams and this accounts for the why the sync can be out.

blackirishman
  • 903
  • 1
  • 10
  • 23

2 Answers2

3

In addition to Sean's answer:

WebRTC player in browsers has a very low tolerance for timestamp difference between arriving audio and video samples. Your audio and video streams must be aligned (interleaved) precisely, i.e. the timestamp of last audio sample received from network, should be +- 200ms or so comparing to timestamp of last video frame received from network. Otherwise WebRTC player will stop using NTP Timestamps and will play streams individually. This is because WebRTC player tries to keep latency at a minimum. Not sure it's good decision from WebRTC team. If your bandwidth is not sufficient, or if live encoder provides streams not timestamp-aligned - then you will have out of sync playback. In my opinion, WebRTC player could have a setting - whether to use that tolerance value or always play in sync, using NTP Timestamps, at the expense of latency.

user1390208
  • 1,866
  • 20
  • 20
2

RTP/RTCP (which WebRTC uses) traditionally uses the RTCP Sender Report. That allows each SSRC stream to be synced on a NTP Timestamp. Browsers do use them today, so things should work.

Are you doing any protocol bridging or anything that could be RTP only? What Media Server are you using?

Community
  • 1
  • 1
Sean DuBois
  • 3,972
  • 1
  • 11
  • 22
  • No protocol bridging. I am using LiveSwitch. I am publishing video and getting a desync when my program goes to black and back to program. Its an encoder issue, just wondering why the audio and video would come out of sync. – blackirishman Mar 04 '21 at 21:24