0

I'm trying to establish a real-time audio communication between Pepper's tablet and my PC. I'm using Gstreamer to establish that. The audio from Pepper's mic to PC is working but there seems to be no audio from my PC to Pepper's tablet. What am I doing wrong?

PC side:

audio_pipeline = Gst.Pipeline('audio_pipeline')

audio_udpsrc = Gst.ElementFactory.make('udpsrc', None)
audio_udpsrc.set_property('port', args.audio)

audio_caps = Gst.caps_from_string('application/x-rtp,media=(string)audio, clock-rate=(int)44100, width=16, height=16, encoding-name=(string)L16, encoding-params=(string)1, channels=(int)1, channel-positions=(int)1, payload=(int)96')
audio_filter = Gst.ElementFactory.make('capsfilter', None)
audio_filter.set_property('caps',audio_caps)

audio_depay = Gst.ElementFactory.make('rtpL16depay', None)
audio_convert = Gst.ElementFactory.make('audioconvert', None)

audio_sink = Gst.ElementFactory.make('alsasink', None)
audio_sink.set_property('sync',False)

audio_pipeline.add(audio_udpsrc,audio_filter,audio_depay,audio_convert,audio_sink)
audio_udpsrc.link(audio_filter)
audio_filter.link(audio_depay)
audio_depay.link(audio_convert)
audio_convert.link(audio_sink)

Robot side (Choregraphe):

audio_src = gst.element_factory_make('autoaudiosrc')

audio_convert = gst.element_factory_make('audioconvert')
audio_caps = gst.caps_from_string('audio/x-raw-int,channels=1,depth=16,width=16,rate=44100')
audio_filter = gst.element_factory_make('capsfilter')
audio_filter.set_property('caps',audio_caps)
# audio_enc = gst.element_factory_make('mad')
audio_pay = gst.element_factory_make('rtpL16pay')
audio_udp = gst.element_factory_make('udpsink')
audio_udp.set_property('host',user_ip)
audio_udp.set_property('port',int(user_audio_port))

self.audio_pipeline.add(audio_src,audio_convert,audio_filter,audio_pay,audio_udp)
gst.element_link_many(audio_src,audio_convert,audio_filter,audio_pay,audio_udp)

or

Robot's side (Python SDK):

GObject.threads_init()
Gst.init(None)

audio_pipeline = Gst.Pipeline('audio_pipeline')

audio_src = Gst.ElementFactory.make('autoaudiosrc')

audio_convert = Gst.ElementFactory.make('audioconvert')
audio_caps = Gst.ElementFactory.make('audio/x-raw-int,channels=2,depth=16,width=16,rate=44100')
audio_filter = Gst.ElementFactory.make('capsfilter')
audio_filter.set_property('caps',audio_caps)
audio_pay = Gst.ElementFactory.make('rtpL16pay')
audio_udp = Gst.ElementFactory.make('udpsink')
audio_udp.set_property('host',user_ip)
audio_udp.set_property('port',int(user_audio_port))

audio_pipeline.add(audio_src,audio_convert,audio_filter,audio_pay,audio_udp)
audio_src.link(audio_convert)
audio_convert.link(audio_filter)
audio_filter.link(audio_pay)
audio_pay.link(audio_udp)

audio_pipeline.set_state(Gst.State.PLAYING)

Computer's mic to Pepper:

audio_port = 80

s_audio_pipeline = Gst.Pipeline('s_audio_pipeline')

s_audio_src = Gst.ElementFactory.make('autoaudiosrc')

s_audio_convert = Gst.ElementFactory.make('audioconvert')
s_audio_caps = Gst.ElementFactory.make('audio/x-raw-int,channels=2,depth=16,width=16,rate=44100')
s_audio_filter = Gst.ElementFactory.make('capsfilter')
s_audio_filter.set_property('caps',audio_caps)
s_audio_pay = Gst.ElementFactory.make('rtpL16pay')
s_audio_udp = Gst.ElementFactory.make('udpsink')
s_audio_udp.set_property('host',ip)
s_audio_udp.set_property('port',int(audio_port))

s_audio_pipeline.add(s_audio_src,s_audio_convert,s_audio_filter,s_audio_pay,s_audio_udp)
s_audio_src.link(s_audio_convert)
s_audio_convert.link(s_audio_filter)
s_audio_filter.link(s_audio_pay)
s_audio_pay.link(s_audio_udp)

Pepper receiving:

audio = 80

    r_audio_pipeline = Gst.Pipeline('r_audio_pipeline')

    #defining audio pipeline attributes
    r_audio_udpsrc = Gst.ElementFactory.make('udpsrc', None)
    r_audio_udpsrc.set_property('port', audio)

    r_audio_caps = Gst.caps_from_string('application/x-rtp,media=(string)audio, clock-rate=(int)44100, width=16, height=16, encoding-name=(string)L16, encoding-params=(string)1, channels=(int)2, format=(string)S16LE, channel-positions=(int)1, payload=(int)96')
    r_audio_filter = Gst.ElementFactory.make('capsfilter', None)
    r_audio_filter.set_property('caps',r_audio_caps)

    r_audio_depay = Gst.ElementFactory.make('rtpL16depay', None)
    r_audio_convert = Gst.ElementFactory.make('audioconvert', None)

    r_audio_sink = Gst.ElementFactory.make('alsasink', None)
    r_audio_sink.set_property('sync',False)

    #linking the various attributes
    r_audio_pipeline.add(r_audio_udpsrc,r_audio_filter,r_audio_depay,r_audio_convert,r_audio_sink)
    r_audio_udpsrc.link(r_audio_filter)
    r_audio_filter.link(r_audio_depay)
    r_audio_depay.link(r_audio_convert)
    r_audio_convert.link(r_audio_sink)

    r_audio_pipeline.set_state(Gst.State.PLAYING)

I think there might be a problem with the pepper's receiving port number... I tried different port numbers (including 9559) but nothing seemed to work. Is the source ID wrong?

Is it possible to run the 2-way stream in the same pipeline?

I took a look at other libraries like ffmpeg and PyAudio, but I couldn't any method for live streaming.

androadi
  • 25
  • 7
  • From the pc side, can you confirm you are receiving the data on the port ? (following pipeline should tell you that gst-launch-1.0 -v udpsrc port=xxxx ! fakesink dump=true ) – Prabhakar Lad Mar 07 '18 at 15:16
  • This is my output: gst-launch-1.0 -v udpsrc port=10001 ! fakesink dump=true Setting pipeline to PAUSED ... Pipeline is live and does not need PREROLL ... Setting pipeline to PLAYING ... New clock: GstSystemClock – androadi Mar 08 '18 at 11:41
  • did it print anything further (It should print buffer data) ? If it has not printed the buffer data, look if your network is blocking it, firewall setting on the computer as well, also to avoid your application issues maybe try a pipleine to send from sender gst-launch-1.0 -v audiotestsrc ! audioconvert ! rtpL16pay ! udpsink clients="xxx.xxx.xx.xxx:xxx" – Prabhakar Lad Mar 08 '18 at 11:43
  • If the firewall is blocking it, I shouldn't get video stream also right? I am getting the video with no problems... – androadi Mar 08 '18 at 14:17
  • Yes it should block, is it on the same port and same machines on same network ? did you try sending with the pipleine which I posted my earlier post ? – Prabhakar Lad Mar 08 '18 at 14:18
  • They are on different ports on the same network.... Output: https://pastebin.com/30YHq8st – androadi Mar 08 '18 at 14:39
  • could you also post the output as well on receiver and the pipeline which you used ? – Prabhakar Lad Mar 08 '18 at 15:01
  • the receiver is a tablet on a robot (pepper)... I am not sure if I can access terminal on that... how do I get the output on pipeline? – androadi Mar 08 '18 at 15:10

1 Answers1

0

Make sure you run the Python script on the robot.

Also, did you run the GMainLoop ?

Choregraphe behaviors are run in NAOqi, and NAOqi runs a GMainLoop already in the background. Maybe this is what is missing in your stand-alone script.

Finally, you show no piece of code in your snippets that is meant to take the PC's audio to the network, nor from the network to Pepper's speakers.

Victor Paléologue
  • 2,025
  • 1
  • 17
  • 27
  • I edited in the code for the the other side. Do you think it might be a problem with choosing the port or the source id? – androadi Jun 07 '18 at 14:02
  • Alright. I'm not sure what's wrong, or what is the "source ID" you mention, but you could try having the same capabilities on both sides, and expliciting the host address to listen at – Victor Paléologue Jun 28 '18 at 14:23