7

I know that exoplayer has support for RTSP, but I need C++ code that works on players from lots of OSs, so I need to parse the RTP packet in C++ to NAL units before passing to exoplayer

I found a way to decode RTP packets using live555 and extract its NAL units. According to ExoPlayer's documentation:

Components common to all ExoPlayer implementations are:

A MediaSource that defines the media to be played, loads the media, and from which the loaded media can be read.

A MediaSource is injected via ExoPlayer.prepare at the start of playback. ...

So I need a custom MediaSource that can extract NAL units from my C++ code.

At the class reference for MediaSource we can see that there are already some MediaSources available. I though maybe SmoothStreaming MediaSource could work but there's no description of what it does exactly and in its constructor I have to provide an Uri or an SsManifest (what).

I can see that there exists a NAL unit utility in this library so maybe things are already half done

So how to build or use an already available MediaSource to read NAL units for ExoPlayer to play?

As an additinal, how would you pass the NAL units from C++ to Java? In the code I found it's simply storing them in a C++ buffer. Should I read this buffer in Java somehow?

UPDATE:

I've been researching how this library works. It all begins with a MediaSource which have objects like Extractor and DataSource. Seems that ExtractorMediaSource is a MediaSource where you can provide your own Extractor and DataSource.

As I understood, DataSource is a class that gets the raw bytes from any possible place, be it a file read or a network packet. Based on the available Extractor classes on the library like Mp4Extractor and Mp3Extractor, an Extractor is something that will interpret the data read from DataSource. The two main methods from the Extractor interface are:

void init(ExtractorOutput output)
int read(ExtractorInput input, PositionHolder seekPosition)

I don't know what are ExtractorInput and ExtractorInput for, but they look important.

So somehow Extractor reads from DataSource, parses it and sends to Renderer in a common format?

I need to know how is this common format so I can parse the NAL units that I read from a custom DataSource.

Community
  • 1
  • 1
PPP
  • 1,279
  • 1
  • 28
  • 71

2 Answers2

2

You cannot play NAL units or raw H.264 stream with ExoPlayer.
The picture data must exist within a supported container/format.

It's not clear what your mysterious C++ code is doing, is it an NDK setup? What's its role in Android decoding? Are you saying you're unable to pass [from the C++ function] a data array into some Android function as function parameter? Is it something like this Java to C++ setup? It's not clear what your real problem is...

If you insist on Exoplayer, I can tell you that FLV is the one (on containers list) that might be the best option since it can be built in real-time (re-muxing). You first create an FLV header that holds SPS and PPS data then followed by keyframe (extracted H264 data from the first NAL). You'll have to get familiar with FLV bytes structure, but each frame header is around 13 bytes followed by NAL data, repeat for each frame until end. This woud be realtime transcoding.

As a second option, for Android, you could just use MediaCodec to decode the H264 as extracted from the NAL units. Here is a useful example source. Just use as:

MediaCodec.createDecoderByType("video/avc"); //then later give NAL units

Study also functions of this other source for ideas of how it works to via Android's own decoder.

Other starting points:

VC.One
  • 14,790
  • 4
  • 25
  • 57
  • I'm writing an RTP receiver that must work on Android, iOS and Windows, so it's written in C++. I want to pass the RTP data, which are nal units, to the Android, iOS and Windows players. I'm doing the Android part now. I've found that exoplayer was very used and gave it a try. So yes, it's an NDK setup. – PPP May 18 '18 at 02:56
  • Actually the reason I chose exoplayer was because I'm also using React Native to control these video displays on Android, iOS and Windows. On Android, this React Native library uses exoplayer, so that's why I chose it. – PPP May 18 '18 at 02:56
  • If I were to use mediacodec I'd have to rewrite the android part of https://github.com/react-native-community/react-native-video, so I think the best way would be to create an FLV packet in C++ and pass it to exoplayer and hope iOS and Windows players also have support for this container format – PPP May 18 '18 at 02:58
  • Also, FLV support is likely to be dropped in the long term because no one uses flash anymore. I see a lot of webm these days. Wouldn't Matroska be a good container format? – PPP May 18 '18 at 03:52
  • The easiest way to see expected correct FLV bytes is to use FFmpeg to create a short one from NALUs. There is also a PDF of **FLV specificatons** for reference. Share a link to some few seconds worth of NALU if you need guidance. Yes Flash Player browser plugin is dying but this isn't a web page project, you're just using FLV since it's an **easy to setup** file format that supports h264 codec and in the very same order that h264 frames are packed (eg; can be appended in realtime),... – VC.One May 18 '18 at 12:28
  • Have you ever used live555? I'm using it to receive RTP packets and then I parse with https://github.com/aizvorski/h264bitstream, so I guess everything is half done. I didn't try it yet because `debug_nal` isn't defined for me don't know why. Anyways, if I have the individual nal units, then it's easy to put them into FLV frames, right? – PPP May 20 '18 at 21:30
  • I don't use Live555. I know about a/v formats (not much on server protocols & related). About FLV, this is why I recommend making a test file to study when researching n FLV bytes structure. Take a single small (320x240) image and convert to a 1-frame h264. Now tell FFmpeg to contain same h264 within FLV. Use a hex editor to find (word search) the same NAL data inside the FLV. It will be found after FLV **container header** (holds width, duration, etc), then every a/v frame has 12-byte FLV frame header, followed by some h264 NAL unit or some AAC/MP3 frame.... repeat until end – VC.One May 21 '18 at 20:57
1

I did something very similar, you can check my blog post about some aspects on customizing ExoPlayer (https://medium.com/@mahmoud.mohamed.bahaa/diving-into-exoplayer-getting-more-control-over-the-framework-cac436d1472c)

Poka Yoke
  • 373
  • 3
  • 8
  • 27