3

I am writing a transmuxer to turn MPEG-TS files from an HLS stream into MP4 files, which can then be played in a web browser.

I had expected to be able to just do a simple one-to-one mapping of TS files to MP4 files, but it turns out that some HLS streams are divided up such that the individual TS files do not always start on IDR frames; this results in frozen images and stuttering whenever you cross a segment boundary.

I figure I can fix this by concatenating the data from multiple TS files, ignoring the original file boundaries, identifying the IDR frames in the video stream myself, and starting each new MP4 segment on an IDR frame. If I do this, however, I'm worried about possible corruption at the ends of files; IDR frames guarantee that no later frame can reference any earlier data, but nothing indicates that earlier B frames can't look forward past an IDR frame.

So, how do I know where it is safe to make the cut, such that B frames before the cut won't try to look past it?

Logan R. Kearsley
  • 682
  • 1
  • 5
  • 19

1 Answers1

5

I think you have no problems if you end your cut at an IDR frame. But I am not 100% sure.

From the H.264 spec and for example RFC3984:

IDR picture: A coded picture containing only slices with I or SI slice types that causes a "reset" in the decoding process. After the decoding of an IDR picture, all following coded pictures in decoding order can be decoded without inter prediction from any picture decoded prior to the IDR picture.

Note that frames cannot reference other frames prior the IDR in decoding order.

In order for a B frame to reference any frame after an IDR frame, the referenced frame must appear prior in the file, as it must be decoded first. So that B frame must be after the IDR frame. That would mean that this B-frame cannot reference any frame before the IDR frame. This would not make any sense in my opinion.

For example: assume the following frames in display order:

(other frames) B IDR P

if the P references the IDR, and the B would reference the P, this must be the decoding order:

(other frames) IDR P B

In this example the B cannot reference any of the other frames, it can only reference the IDR and the P.

Note that even if this would happen, you can detect this if you find a frame after the IDR which is in display order before the IDR.

Community
  • 1
  • 1
wimh
  • 15,072
  • 6
  • 47
  • 98
  • Well, this mostly worked, in that the stuttering / frame freezing is gone, but cutting on IDRs is still resulting in weird pixelation artifacts, even when I verify that no frame before the IDR has a presentation time later than any frame after the IDR. Any ideas? – Logan R. Kearsley May 13 '15 at 21:32
  • Can be many reasons, but my first guess would be that you do not properly extract the h.264 from the mpeg-ts. Do you handle the emulation prevention byte? – wimh May 13 '15 at 23:23
  • I don't; I expected the MP4 player should be capable of handling those. If that were the problem, I would expect to see corruption scattered everywhere, not just immediately following segment boundaries. Still, I suppose it can't hurt to check. – Logan R. Kearsley May 14 '15 at 03:05
  • Yup; removing emulation prevention bytes had no effect. – Logan R. Kearsley May 14 '15 at 04:06