0

Request

In Flash/AS3 how do you decode or capture CEA-608 closed-captions embedded in an mp4? I can't seem to get even a hint of the caption data nor can I find any documentation that can point me in the right direction. Any documentation, examples, or ideas would be super helpful.

Specifics

I'm building a video-player in AS3 using OSMF. I can't seem to find /any/ documentation on accessing CEA-608 closed-captions embedded in mp4 chunks in an m3u8 video.

The OSMF CaptioningPlugin requires an external XML file so that won't do; I'm looking for integration details for embedded captioning tracks.

I've tried attaching onTextData, onMetaData, onCaptionData, onTextRR handlers and listeners to the OSMF Netstream with absolutely no luck (like... none of these events or handlers ever fire or return anything).

private function onTraitAdd ($e:MediaElementEvent) : void { var mediaElement: MediaElement = ($e.target as MediaElement);

    switch ($e.traitType) {
        case MediaTraitType.LOAD:
            _netStreamLoadTrait = mediaElement.getTrait(MediaTraitType.LOAD) as NetStreamLoadTrait;
            _netStreamLoadTrait.addEventListener(LoadEvent.LOAD_STATE_CHANGE, onNetStreamLoaded);
            break;
    }
}

private function onNetStreamLoaded ($e:LoadEvent) : void {
    var netStream:NetStream = _netStreamLoadTrait.netStream;
    netStream.client.addHandler("onTextData", onTextData);
    netStream.client.addHandler("onCuePoint", onTextData);
    netStream.client.addHandler("onMetaData", onTextData);
    netStream.client.addHandler("onCaptionData", onTextData);
    netStream.client.addHandler("onTextRR", onTextData);
    netStream.client.addHandler("onCaptionInfo", onTextData);


    netStream.addEventListener("onTextData", onTextData);
    netStream.addEventListener("onCuePoint", onTextData);
    netStream.addEventListener("onMetaData", onTextData);
    netStream.addEventListener("onCaptionData", onTextData);
    netStream.addEventListener("onTextRR", onTextData);
    netStream.addEventListener("onCaptionInfo", onTextData);

    netStream.addEventListener(NetStatusEvent.NET_STATUS, onNetStreamStatus);
    netStream.addEventListener(NetDataEvent.MEDIA_TYPE_DATA, onStreamData);

}

I can't tell if the issue is with my OSMF implementation (maybe I'm listening to the wrong NetStream), or if the issue is that there's no way to get this data out of the video.

Example files:

http://stream.flowplayer.org/big_buck_bunny_with_captions.mp4 http://now.video.nfl.com/i/captiontest/closedcaptiontest_,350k,550k,.mp4.csmil/master.m3u8 (this example file is more complex though because it requires an HLS plugin)

Other

  1. I tried using OSMFCCDecoder.swc (which was rather hard to find, uploaded here). There's very little documentation and no information on the expected result.

  2. Also decompiled JWPlayer to see how they handle captions, they parse-out the byteArray ref.

  3. Steps to see captions playing in JWPlayer

jwplayer("container_wrapper")
    .setup({
        file: "http://now.video.nfl.com/i/captiontest/closedcaptiontest_,350k,550k,.mp4.csmil/master.m3u8"
    });
potench
  • 3,802
  • 1
  • 28
  • 39
  • 1
    I'm out of my depth with OSMF. But I know a perfectly good workflow for encoding video with cue-points that are AS3-readable and can even contain your caption strings in the cue-point data. I prefer do this with a home-made player using either the Video or StageVideo class. My workflow involves After Effects but you can also use Adobe Media Encoder for cue-point insertion, See http://help.adobe.com/en_US/mediaencoder/cs/using/WS2bacbdf8d487e582-30a3408e12f8ee21458-7fe6.html If you're interested in the After Effects way to do it, respond. – Craig May 21 '14 at 06:23
  • Unfortunately, I'm not able to change the encoding process. The HSL streams with embedded CEA-608 captions are currently working in other platforms - just not sure how to decode or capture the data within Flash. – potench May 21 '14 at 18:37
  • 1
    I'm guessing a few things: 1. There are no cue-points readable as data in your streams and 2. Flashplayer and AIR have no way to read 608 captions. I hope that I'm wrong but do you have any evidence about the cue points? One work-around you have is to create your own cue-points programatically with Actionscript, then associate with those cue points your own captions which you can display with Actionscript. In a short video with a reasonable # of captions this wouldn't be too much pain. – Craig May 21 '14 at 22:36
  • Yeah you're right, there's no cue points; I think I have to parse the MP4 byte array and decode or decrypt out the captions. JWPlayer's solution is the nearest working HLS caption-decoding Flash implementation I've seen and it the decoding mechanism is returning garbled text. If I end up decoding the stream, I'll for sure open source it... thanks for the help @Craig – potench May 22 '14 at 18:04

1 Answers1

3

HLS Streams can contain subtitles in a few different formats:

  • CEA608
  • WebVTT
  • ID3 Tags CEA608 works on all Apple iOS devices from 3.0 onwards but is difficult to support because it's a character based format. WebVTT is supported in newer iOS devices but is text based and easer to decode.

CEA608 is embedded in H264 SEI NALU as specified in ANSI/SCTE 128 2010 Section 8. So in order to decode it you would need to partially decode the H.264 bitstream. The link you provided http://help.adobe.com/en_US/adobemediaserver/devguide/WS5262178513756206-55daa065139e25f4596-8000.html seems to describe a library for decoding CEA608 and I believe the expected result will be text rendered over the video.

It's hard work to decode CEA608 yourself as this is a character based format and might include letters, numbers, Spanish, backspace, color and position markers. You can find the CEA608 standard here: http://www.ce.org/Standards/Standard-Listings/R4-3-Television-Data-Systems-Subcommittee/Line-21-Data-Service.aspx You might also need to look at the MPEG2 Transport Stream specifications for the method to decode the H264 NALUs.

Grant
  • 1,608
  • 16
  • 14