3

I have some troubles with the WebRTC API (and most particularly RTCPeerConnection).

I have successfully managed to make a video call between two browsers : I know how to get the webcam stream with getUserMedia, I know how to contact a STUN server and react on 'onicecandidate' event and I know how to create the offer (and the answer in the other peer) and send the sdp. I use WebSockets as a signalling channel.

What I need to do is process the video stream with C/C++ algorithms, so I am looking for a way to receive a RTCPeerConnection in C/C++ and receive a call in C/C++.

I have been trying to build and test Google's libjigle library (haven't succeeded yet, though (I'm on Archlinux)). But even when I succeed, I don't see how to re-use the code for my own case.

What I have done so far :

  • I understand how STUN servers work, ICE candidates and SDP sessions and how to create / process them in javascript
  • I managed to make peer-to-peer calls between two browsers (and even between a PC and my Android and this worked perfectly)
  • I managed to use libwebsockets to create a simple signalling server, in which I successfully receive the browser's ICE candidates and sdp messages.

What I am looking for :

  • A way to receive/parse/process in C/C++ what the browser sends i.e. ICE candidates, sdp, offer
  • A way to create the answer (the browser will always be the initiator) and receive/process the webcam stream.

What I have tried :

  • I have tried to have the webcam play in a HTML5 element, and periodically (~ 33ms) draw the frame in a , call getImageData() and send the array of (R,G,B,alpha) with a pure WebSocket connection. But even for a 100x100 px, grayscale frame (hence 10kB), I can only achieve ~7fps with a ~600 kb/s upload stream. This is why I want to use RTCPeerConnection which works on UDP

My constraints :

  • I need to run the native app in C or C++ because I have image/video processing algorithms that are implemented in C++ (I have seen a lot of Node.js-based servers but I can't have that : no way to call my algorithms)
  • I'd like to be able to run at roughly 30 fps so that this is relatively fluid for the user.
  • I can't use Flash or Silverlight : I need to stay HTML5 / javascript for the client

Conclusion :

  • Where I fail short is everything that deals with ICE candidates, SDP sessions and contact STUN server in C/C++ because unlike javascript there are no events ('onicecandidates', 'onaddstream', etc).

Thank you in advance for your help !

nschoe
  • 1,967
  • 1
  • 19
  • 26
  • The Native API or using a breaker (similar to [RTC WebBreaker](http://webrtc2sip.org/#aRTCWebBreaker)) should work for you. – Benjamin Trent May 19 '14 at 21:23
  • I am precisely trying to use the native API, but I have no clue how to begin. I found [this native API](http://www.webrtc.org/reference/native-apis) which seems to refer to libjingle. I have been trying to build libjingle for four days now but I can only build some parts (make all fails, but make peerconnection_server succeeds for instance). Anyway, can somebody explain to me how I can start using the library and the classes to build even a small working program ? I am totally lost here. – nschoe May 21 '14 at 08:46
  • Did you ever work this out? I have a similar issue (actually the reverse problem, streaming series of images in memory from Python/C++ application to a webpage). – griffin2000 Feb 06 '17 at 22:06

0 Answers0