1

I have an FLV file with h264 video tags. Each video tag that contains an h264 IDR NAL unit also also contains display orientation SEI, ie my NAL access unit for IDR key frames is [SEI, IDR].

For some reason, converting this FLV file to h264 without the stream copy option strips away all the SEI messages from all IDR frames. In its place, right after the SPS and PPS NAL units, there's a single user unregistered data SEI.

i.e using

ffmpeg -i in.flv out.264

gives [SPS, PPS, SEI, IDR, ...] where the SEI is:

x264 - core 155 r2917 0a84d98 - H.264/MPEG-4 AVC codec - Copyleft 2003-2018 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=11 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=crf mbtree=1 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00

However

ffmpeg -i in.flv -c:v copy out.264

preserves my SEI messages, and pretty much decodes to the raw h264 I put into the FLV muxer to generate the FLV file in the first place. Why does this happen? Is there a flag to perform encoding but preserve SEI messages?

Tunji_D
  • 3,677
  • 3
  • 27
  • 35

1 Answers1

2

Using -c:copy, ffmpeg does not decode the h264, it simply copies the NALUs from the source to destination without even looking at them. Hence every NALU is preserved.

With a transcode, ffmpeg looses the metadata on the decode. It would need to take the SEI, set it as metadata to the AVFrame then write the SEI to the new frame when it is re encoded. But SEI is a per AU object. Meaning if you change the frame rate, ffmpeg would need to know how to split or merge SEIs into the new frame rate. That is a lot of code that nobody has written yet. It would be possible to copy SEI when not changing the frame rate, of writing merge/split handlers for all know SEI types. But somebody would need to sponsor that work.

Ffmpeg is open source, if you were to develop that code, it’s very likely they would accept the patch.

szatmary
  • 29,969
  • 8
  • 44
  • 57
  • I see, so for things like closed captions and any other extra metadata that can be embedded in a bitstream, it's probably easier to add it post trancode rather than pre? – Tunji_D Jun 22 '20 at 00:24
  • Also a quick perusal of libav shows that while transcoding, each AVframe does have a struct associated with it for my particular SEI message, so hopefully it's not too difficult to submit a patch for it myself. – Tunji_D Jun 22 '20 at 00:35
  • szatmary, thank you for this explanation of why SEI doesn't get transcoded. – MAXdB Mar 20 '23 at 16:10