For a few days I have a problem that I do not understand at all. Everything seems to be working well, finally everything should be working well with everything I read right and left and all my tries. I would like to share audio and video on android with Webrtc from client A to client B and vice versa.
The problem is that when onAddStream(MediaStream mediaStream) is called I have mediaStream.videoTracks.get(0).state() = LIVE and mediaStream.videoTracks.size()) = 1 but 0 frames received in the Remoterender. (But on both clients everything is OK in the Localrender).
I think the problem is a network issue in that when client B creates the response to client A's offer it only receives half of the candidate ices and only audio with sdpindex at 0.
I don't understand why! (I tried on different emulator or real phone devices and different networks always the same)
Here is the code I am using. I specify that I know it's not the best, but for my tests I use a somewhat rudimentary signaling server with the Google Volley library and an SQL database instead of node or Firebase database.
If anyone has any idea why I only have audio on client B, or 0 frames in the remoteRender.
public void createPeerConnection() {
if (mPeerConnection != null) return;
final ArrayList<PeerConnection.IceServer> iceServers = new ArrayList<>();
iceServers.add(PeerConnection.IceServer.builder("stun:stun1.l.google.com:19302").createIceServer());
iceServers.add(PeerConnection.IceServer.builder("stun:stun2.l.google.com:19302").createIceServer());
iceServers.add(PeerConnection.IceServer.builder("stun:stun3.l.google.com:19302").createIceServer());
iceServers.add(PeerConnection.IceServer.builder("stun:stun4.l.google.com:19302").createIceServer());
iceServers.add(PeerConnection.IceServer.builder("stun:relay.meterd.ca:80").createIceServer());
iceServers.add(PeerConnection.IceServer.builder("turn:relay.meterd.ca:80").setUsername("877d7df597a8a3c4hf48275566").setPassword("tmprAF+d/lDMEzbfAhtt").createIceServer());
iceServers.add(PeerConnection.IceServer.builder("turn:relay.meterd.ca:443").setUsername("877d7df597a8a3c4hf48275566").setPassword("tmprAF+d/lDMEzbfAhtt").createIceServer());
iceServers.add(PeerConnection.IceServer.builder("turn:relay.metered.ca:443?transport=tcp").setUsername("877d7df597a8a3c4hf48275566").setPassword("tmprAF+d/lDMEzbfAhtt").createIceServer());
mPeerConnection = mFactory.createPeerConnection(rtcConfiguration, this);
Log.d(TAG, "Peer Connection created");
}
Connection(final Context context, ConnectionListener listener) {
final PeerConnectionFactory.InitializationOptions options = PeerConnectionFactory.InitializationOptions.builder(context)
.setEnableInternalTracer(true)
.setFieldTrials("WebRTC-H264HighProfile/Enabled/")
.createInitializationOptions();
final VideoEncoderFactory encoderFactory;
final VideoDecoderFactory decoderFactory;
encoderFactory = new SoftwareVideoEncoderFactory();
decoderFactory = new SoftwareVideoDecoderFactory();
PeerConnectionFactory.initialize(options);
PeerConnectionFactory.Options options1 = new PeerConnectionFactory.Options();
options1.disableEncryption = true;
options1.disableNetworkMonitor = true;
mFactory = PeerConnectionFactory.builder()
.setOptions(options1)
.setVideoEncoderFactory(encoderFactory)
.setVideoDecoderFactory(decoderFactory)
.createPeerConnectionFactory();
mListener = listener;
}
public void initializeMediaDevices(final Context context, SurfaceViewRenderer localRenderer) throws Exception {
MediaStream mMediaStream = mFactory.createLocalMediaStream(MEDIA_STREAM_ID);;
mVideoCapturer = createCameraCapturer(context);
final VideoSource videoSource = mFactory.createVideoSource(false);
// videoSource.adaptOutputFormat(VIDEO_WIDTH, VIDEO_HEIGHT, VIDEO_FPS);
final EglBase.Context eglContext = EglBase.create().getEglBaseContext();
final SurfaceTextureHelper surfaceTextureHelper = SurfaceTextureHelper.create("captureThread", eglContext);
mVideoCapturer.initialize(surfaceTextureHelper, context, videoSource.getCapturerObserver());
localRenderer.init(eglContext, new RendererCommon.RendererEvents() {
@Override
public void onFirstFrameRendered() {
Log.d(TAG, "onFirstFrameRendered");
}
@Override
public void onFrameResolutionChanged(int i, int i1, int i2) {
Log.d(TAG, "Frame resolution changed");
Log.d(TAG, "Frame resolution changed : "+i);
Log.d(TAG, "Frame resolution changed : "+i1);
Log.d(TAG, "Frame resolution changed : "+i2);
}
});
localRenderer.setEnableHardwareScaler(true);
mVideoCapturer.startCapture(VIDEO_WIDTH, VIDEO_HEIGHT, VIDEO_FPS);
// localRenderer.setEnableHardwareScaler(true);
final VideoTrack videoTrack = mFactory.createVideoTrack(VIDEO_TRACK_ID, videoSource);
videoTrack.setEnabled(true);
localRenderer.setMirror(true);
videoTrack.addSink(localRenderer);
final AudioSource audioSource = mFactory.createAudioSource(new MediaConstraints());
final AudioTrack audioTrack = mFactory.createAudioTrack(AUDIO_TRACK_ID, audioSource);
audioTrack.setEnabled(true);
// audioTrack.setEnabled(true);
// audioTrack.setVolume(100);
mMediaStream.addTrack(videoTrack);
mMediaStream.addTrack(audioTrack);
mPeerConnection.addStream(mMediaStream);
public void createOffer() {
final MediaConstraints mediaConstraints = new MediaConstraints();
mediaConstraints.mandatory.add(new MediaConstraints.KeyValuePair("OfferToReceiveVideo", "true"));
mediaConstraints.mandatory.add(new MediaConstraints.KeyValuePair("OfferToReceiveAudio", "true"));
mPeerConnection.createOffer(new SdpObserver() {
@Override
public void onCreateSuccess(SessionDescription sessionDescription) {
mListener.createofferonCreateSuccess(sessionDescription);
mPeerConnection.setLocalDescription(this, sessionDescription);
}
@Override
public void onSetSuccess() {
Log.d(TAG, "Local description set success");
mListener.createofferonSetSuccess();
}
@Override
public void onCreateFailure(String s) {
Log.e(TAG, "Failed to create local offer error:" + s);
}
@Override
public void onSetFailure(String s) {
Log.e(TAG, "Failed to set local description error:" + s);
}
}, mediaConstraints);
}
public static void createAnswerFromRemoteOffer(final String remoteOffer) {
SdpObserver observer = new SdpObserver() {
@Override
public void onCreateSuccess(SessionDescription sessionDescription) {
Log.d(TAG, "Local answer created");
mPeerConnection.setLocalDescription(this,sessionDescription);
mListener.createAnswerFromRemoteOffercreatesuccess(sessionDescription);
}
@Override
public void onSetSuccess() {
mListener.createAnswerFromRemoteOfferSetsuccess();
Log.d(TAG, "Set offer description was successful");
}
@Override
public void onCreateFailure(String s) {
Log.e(TAG, "Failed to create local answer error:" + s);
}
@Override
public void onSetFailure(String s) {
Log.e(TAG, "Failed to set offer description error :" + s);
}
};
final SessionDescription sessionDescription = new SessionDescription(SessionDescription.Type.OFFER, remoteOffer);
final MediaConstraints mediaConstraintss = new MediaConstraints();
mediaConstraintss.mandatory.add(new MediaConstraints.KeyValuePair("OfferToReceiveVideo", "true"));
mediaConstraintss.mandatory.add(new MediaConstraints.KeyValuePair("OfferToReceiveAudio", "true"));
mPeerConnection.setRemoteDescription(observer, sessionDescription) ;
mPeerConnection.createAnswer(observer, new MediaConstraints());
}
public static void createAnswerFromRemoteOffer2(final String remoteOffer) {
SdpObserver observer = new SdpObserver() {
@Override
public void onCreateSuccess(SessionDescription sessionDescription) {
Log.d(TAG, "Local answer created 3");
mListener.createAnswerFromRemoteOffer2onCreateSucess(sessionDescription);
}
@Override
public void onSetSuccess() {
mListener.createAnswerFromRemoteOffer2onSetSuces();
Log.d(TAG, "Set offer description 3 was successful");
}
@Override
public void onCreateFailure(String s) {
Log.e(TAG, "Failed to create local 3 answer error:" + s);
}
@Override
public void onSetFailure(String s) {
Log.e(TAG, "Failed to set offer 3 description error :" + s);
}
};
final SessionDescription sessionDescription = new SessionDescription(SessionDescription.Type.ANSWER, remoteOffer);
mPeerConnection.setRemoteDescription(observer, sessionDescription) ;
}
public static void addRemoteIceCandidate(final JSONObject iceCandidateData) throws JSONException {
String icecandidate = iceCandidateData.getString("ICECANDIDATE");
final String sdpmid = iceCandidateData.getString("SDPMID");
final int sdpMLineIndex = iceCandidateData.getInt("SDPINDEX");
IceCandidate iceCandidate = new IceCandidate(sdpmid, sdpMLineIndex, icecandidate);
Log.d(TAG,"add remote candidate "+iceCandidate.toString());
Log.d(TAG,"add remote sdpmid "+sdpmid);
Log.d(TAG,"add remote sdpindex "+sdpMLineIndex);
if (queuedRemoteCandidates != null) {
queuedRemoteCandidates.add(iceCandidate);
} else {
mPeerConnection.addIceCandidate(iceCandidate);
}
}
CLIENT B only audio candidate
2023-03-19 16:08:17.575 29728-29772 CCC pid-29728 D onIceCandidateReceived : audio:0:candidate:1800307301 1 udp 2122187263 fec0::5054:ff:fe12:3456 59467 typ host generation 0 ufrag iT+A network-id 2::UNKNOWN
2023-03-19 16:08:17.579 29728-29772 CCC pid-29728 D onIceCandidateReceived : audio:0:candidate:841689039 1 udp 2122129151 10.0.2.16 37211 typ host generation 0 ufrag iT+A network-id 5 network-cost 10::UNKNOWN
2023-03-19 16:08:17.583 29728-29772 CCC pid-29728 D onIceCandidateReceived : audio:0:candidate:3776784094 1 udp 2122056191 fec0::9ebc:2b3f:a56d:1ed8 48463 typ host generation 0 ufrag iT+A network-id 6 network-cost 10::UNKNOWN
2023-03-19 16:08:17.585 29728-29772 CCC pid-29728 D onIceCandidateReceived : audio:0:candidate:559267639 1 udp 2122005759 ::1 33214 typ host generation 0 ufrag iT+A network-id 4::UNKNOWN
2023-03-19 16:08:17.591 29728-29772 CCC pid-29728 D onIceCandidateReceived : audio:0:candidate:1510613869 1 udp 2121932543 127.0.0.1 54583 typ host generation 0 ufrag iT+A network-id 3::UNKNOWN
2023-03-19 16:08:17.626 29728-29772 CCC pid-29728 D onIceCandidateReceived : audio:0:candidate:1847424209 1 tcp 1518280447 10.0.2.15 55839 typ host tcptype passive generation 0 ufrag iT+A network-id 1::UNKNOWN
2023-03-19 16:08:17.632 29728-29772 CCC pid-29728 D onIceCandidateReceived : audio:0:candidate:1889982791 1 udp 1685921535 92.134.12.255 63997 typ srflx raddr 10.0.2.16 rport 37211 generation 0 ufrag iT+A network-id 5 network-cost 10:stun:64.233.163.127:19302:UNKNOWN
2023-03-19 16:08:17.636 29728-29772 CCC pid-29728 D onIceCandidateReceived : audio:0:candidate:634224277 1 tcp 1518207487 fec0::5054:ff:fe12:3456 43771 typ host tcptype passive generation 0 ufrag iT+A network-id 2::UNKNOWN
2023-03-19 16:08:17.651 29728-29772 CCC pid-29728 D onIceCandidateReceived : audio:0:candidate:2091440959 1 tcp 1518149375 10.0.2.16 47563 typ host tcptype passive generation 0 ufrag iT+A network-id 5 network-cost 10::UNKNOWN
2023-03-19 16:08:17.690 29728-29772 CCC pid-29728 D onIceCandidateReceived : audio:0:candidate:2946423342 1 tcp 1518076415 fec0::9ebc:2b3f:a56d:1ed8 51819 typ host tcptype passive generation 0 ufrag iT+A network-id 6 network-cost 10::UNKNOWN
2023-03-19 16:08:17.695 29728-29772 CCC pid-29728 D onIceCandidateReceived : audio:0:candidate:1645442729 1 udp 1686052607 92.134.12.255 63999 typ srflx raddr 10.0.2.15 rport 56570 generation 0 ufrag iT+A network-id 1:stun:64.233.163.127:19302:UNKNOWN
2023-03-19 16:08:17.702 29728-29772 CCC pid-29728 D onIceCandidateReceived : audio:0:candidate:1876313031 1 tcp 1518025983 ::1 52341 typ host tcptype passive generation 0 ufrag iT+A network-id 4::UNKNOWN
2023-03-19 16:08:17.707 29728-29772 CCC pid-29728 D onIceCandidateReceived : audio:0:candidate:344579997 1 tcp 1517952767 127.0.0.1 53745 typ host tcptype passive generation 0 ufrag iT+A network-id 3::UNKNOWN
2023-03-19 16:08:17.711 29728-29772 CCC pid-29728 D onIceCandidateReceived : audio:0:candidate:2181291358 1 udp 1685858559 2a01:cb10:8bce:d100:e429:c6d5:f617:76b6 64000 typ srflx raddr fec0::9ebc:2b3f:a56d:1ed8 rport 48463 generation 0 ufrag iT+A network-id 6 network-cost 10:stun:2a00:1450:4010:c06::7f:19302:UNKNOWN
CLIENT A audio and video candidate
2023-03-19 16:08:07.023 9017-9059 CCC pid-9017 D onIceCandidateReceived : audio:0:candidate:2091440959 1 tcp 1518149375 10.0.2.16 59483 typ host tcptype passive generation 0 ufrag 5HN1 network-id 5 network-cost 10::UNKNOWN
2023-03-19 16:08:07.023 9017-9059 CCC pid-9017 D onIceCandidateReceived : audio:0:candidate:1942891970 1 tcp 1518076415 fec0::523e:50b8:47ba:4673 47321 typ host tcptype passive generation 0 ufrag 5HN1 network-id 6 network-cost 10::UNKNOWN
2023-03-19 16:08:07.023 9017-9059 CCC pid-9017 D onIceCandidateReceived : audio:0:candidate:1876313031 1 tcp 1518025983 ::1 52945 typ host tcptype passive generation 0 ufrag 5HN1 network-id 4::UNKNOWN
2023-03-19 16:08:07.078 9017-9059 CCC pid-9017 D onIceCandidateReceived : audio:0:candidate:344579997 1 tcp 1517952767 127.0.0.1 37085 typ host tcptype passive generation 0 ufrag 5HN1 network-id 3::UNKNOWN
2023-03-19 16:08:07.082 9017-9059 CCC pid-9017 D onIceCandidateReceived : video:1:candidate:1847424209 1 tcp 1518280447 10.0.2.15 41627 typ host tcptype passive generation 0 ufrag 5HN1 network-id 1::UNKNOWN
2023-03-19 16:08:07.085 9017-9059 CCC pid-9017 D onIceCandidateReceived : video:1:candidate:634224277 1 tcp 1518207487 fec0::5054:ff:fe12:3456 55563 typ host tcptype passive generation 0 ufrag 5HN1 network-id 2::UNKNOWN
2023-03-19 16:08:07.089 9017-9059 CCC pid-9017 D onIceCandidateReceived : video:1:candidate:2091440959 1 tcp 1518149375 10.0.2.16 51877 typ host tcptype passive generation 0 ufrag 5HN1 network-id 5 network-cost 10::UNKNOWN
2023-03-19 16:08:07.101 9017-9059 CCC pid-9017 D onIceCandidateReceived : video:1:candidate:1942891970 1 tcp 1518076415 fec0::523e:50b8:47ba:4673 46365 typ host tcptype passive generation 0 ufrag 5HN1 network-id 6 network-cost 10::UNKNOWN
2023-03-19 16:08:07.102 9017-9059 CCC pid-9017 D onIceCandidateReceived : video:1:candidate:1876313031 1 tcp 1518025983 ::1 33561 typ host tcptype passive generation 0 ufrag 5HN1 network-id 4::UNKNOWN
2023-03-19 16:08:07.103 9017-9059 CCC pid-9017 D onIceCandidateReceived : video:1:candidate:344579997 1 tcp 1517952767 127.0.0.1 56545 typ host tcptype passive generation 0 ufrag 5HN1 network-id 3::UNKNOWN
2023-03-19 16:08:07.103 9017-9059 CCC pid-9017 D onIceCandidateReceived : audio:0:candidate:1582508722 1 udp 1685858559 2a01:cb10:8bce:d100:e429:c6d5:f617:76b6 63983 typ srflx raddr fec0::523e:50b8:47ba:4673 rport 43345 generation 0 ufrag 5HN1 network-id 6 network-cost 10:stun:2a00:1450:4010:c06::7f:19302:UNKNOWN
2023-03-19 16:08:07.152 9017-9059 CCC pid-9017 D onIceCandidateReceived : audio:0:candidate:1645442729 1 udp 1686052607 92.134.12.255 63981 typ srflx raddr 10.0.2.15 rport 35078 generation 0 ufrag 5HN1 network-id 1:stun:64.233.163.127:19302:UNKNOWN
2023-03-19 16:08:07.154 9017-9059 CCC pid-9017 D onIceCandidateReceived : audio:0:candidate:1889982791 1 udp 1685921535 92.134.12.255 63982 typ srflx raddr 10.0.2.16 rport 45157 generation 0 ufrag 5HN1 network-id 5 network-cost 10:stun:64.233.163.127:19302:UNKNOWN
2023-03-19 16:08:07.159 9017-9059 CCC pid-9017 D onIceCandidateReceived : video:1:candidate:1645442729 1 udp 1686052607 92.134.12.255 63984 typ srflx raddr 10.0.2.15 rport 34609 generation 0 ufrag 5HN1 network-id 1:stun:64.233.163.127:19302:UNKNOWN
2023-03-19 16:08:07.165 9017-9059 CCC pid-9017 D onIceCandidateReceived : video:1:candidate:1582508722 1 udp 1685858559 2a01:cb10:8bce:d100:e429:c6d5:f617:76b6 63988 typ srflx raddr fec0::523e:50b8:47ba:4673 rport 59153 generation 0 ufrag 5HN1 network-id 6 network-cost 10:stun:2a00:1450:4010:c06::7f:19302:UNKNOWN
2023-03-19 16:08:07.168 9017-9059 CCC pid-9017 D onIceCandidateReceived : video:1:candidate:1889982791 1 udp 1685921535 92.134.12.255 63985 typ srflx raddr 10.0.2.16 rport 33873 generation 0 ufrag 5HN1 network-id 5 network-cost 10:stun:64.233.163.127:19302:UNKNOWN
2023-03-19 16:08:07.773 9017-9059 CCC pid-9017 D onIceCandidateReceived : audio:0:candidate:3729096449 1 udp 25108223 216.39.253.20 43389 typ relay raddr 92.134.12.255 rport 57285 generation 0 ufrag 5HN1 network-id 1:turn:216.39.253.20:443?transport=tcp:UNKNOWN
2023-03-19 16:08:07.774 9017-9059 CCC pid-9017 D onIceCandidateReceived : audio:0:candidate:3729096449 1 udp 24977151 216.39.253.20 36391 typ relay raddr 92.134.12.255 rport 57284 generation 0 ufrag 5HN1 network-id 5 network-cost 10:turn:216.39.253.20:443?transport=tcp:UNKNOWN
2023-03-19 16:08:07.783 9017-9059 CCC pid-9017 D onIceCandidateReceived : video:1:candidate:3729096449 1 udp 25108223 216.39.253.20 41892 typ relay raddr 92.134.12.255 rport 57286 generation 0 ufrag 5HN1 network-id 1:turn:216.39.253.20:443?transport=tcp:UNKNOWN
2023-03-19 16:08:07.784 9017-9059 CCC pid-9017 D onIceCandidateReceived : video:1:candidate:3729096449 1 udp 24977151 216.39.253.20 43228 typ relay raddr 92.134.12.255 rport 57287 generation 0 ufrag 5HN1 network-id 5 network-cost 10:turn:216.39.253.20:443?transport=tcp:UNKNOWN`
0 frames on client A and B on remoteRender, but the localRender good and display it
2023-03-19 16:08:35.861 29728-29779 org.webrtc.Logging pid-29728 I CameraStatistics: Camera fps: 11.
2023-03-19 16:08:37.690 29728-29950 org.webrtc.Logging pid-29728 I EglRenderer: remoteRendererDuration: 4003 ms. Frames received: 0. Dropped: 0. Rendered: 0. Render fps: .0. Average render time: NA. Average swapBuffer time: NA.
2023-03-19 16:08:37.862 29728-29779 org.webrtc.Logging pid-29728 I CameraStatistics: Camera fps: 12.
2023-03-19 16:08:39.757 29728-29780 org.webrtc.Logging pid-29728 I EglRenderer: local_viewDuration: 4003 ms. Frames received: 45. Dropped: 0. Rendered: 45. Render fps: 11.2. Average render time: 1100 us. Average swapBuffer time: 987 us.
2023-03-19 16:08:39.863 29728-29779 org.webrtc.Logging pid-29728 I CameraStatistics: Camera fps: 11.
2023-03-19 16:08:41.696 29728-29950 org.webrtc.Logging pid-29728 I EglRenderer: remoteRendererDuration: 4005 ms. Frames received: 0. Dropped: 0. Rendered: 0. Render fps: .0. Average render time: NA. Average swapBuffer time: NA.
2023-03-19 16:08:41.864 29728-29779 org.webrtc.Logging pid-29728 I CameraStatistics: Camera fps: 12.
2023-03-19 16:08:43.758 29728-29780 org.webrtc.Logging pid-29728 I EglRenderer: local_viewDuration: 4001 ms. Frames received: 45. Dropped: 0. Rendered: 45. Render fps: 11.2. Average render time: 1107 us. Average swapBuffer time: 980 us.
2023-03-19 16:08:43.865 29728-29779 org.webrtc.Logging pid-29728 I CameraStatistics: Camera fps: 11.
2023-03-19 16:08:45.702 29728-29950 org.webrtc.Logging pid-29728 I EglRenderer: remoteRendererDuration: 4006 ms. Frames received: 0. Dropped: 0. Rendered: 0. Render fps: .0. Average render time: NA. Average swapBuffer time: NA.
