1

We have a stream that is stored in the cloud (Amazon S3) as individual H264 frames. The frames are stored as framexxxxxx.264, the numbering doesn't start from 0 but rather from some larger number, say 1000 (so, frame001000.264)

The goal is to create a mp4 clip which is either timelapse or just faster for inspection and other checking (much faster, compressing around 3 hours of video down to < 20 minutes), this also requires we overlay the frame number (the filename) on the frame itself

At first I was creating a timelapse by pulling from S3 only the keyframes (i-frames? still rather new to codecs & stuff) and overlaying the filename on them and saving as png (which probably isn't needed, but that's what I did) using (this command is used inside a python script)

ffmpeg -y -i {h264_name} -vf \"scale=1920:-1, 
drawtext=fontfile=/usr/share/fonts/truetype/ubuntu-font-family/Ubuntu-B.ttf:fontsize=34:text={txt}:fontcolor=white:x=50:y=50:bordercolor=black:borderw=2\" 
-c:a copy -pix_fmt yuv420p {basename}.png

after this I combined all the frames by using python to convert the lowest numbered frame to 0.png and incrementing (so it would be continuous, because I only used keyframes the numbers originally weren't sequential) and running

ffmpeg -y -f image2 -i %d.png -r {self.params.fps} -vcodec libx264 -crf {self.params.crf} -pix_fmt yuv420p {out_file}

and this worked great, but the difference between keyframes was too long to allow for proper inspection

so now for the question(s)

since I know frames that are not keyframes (p-frames?) can't be used alone by ffmpeg, the method of overlaying the file name and converting it to png (or keep as h264, same thing) won't work, or at least, I couldn't find a way for it to work, maybe there's a way to specify a frame's keyframe?, how can one overlay the filename (and not the frame number as shown here for example)

Also, is it possible to skip some p-frames between the keyframes? (so if a keyframe is every 30 frames, we would take a keyframe, a frame 15 frames later, and next another keyframe)

I thought about using ffmpeg's pipe option to feed it with the files as they're being downloaded, but I'm not sure if I can specify drawtext this way

Also, if there's another alternative that can achieve that (at first I was converting to png, using python and OpenCV to add the filename and then merging the pngs to mp4, but then I found drawtext can do that in a single command so I used it)

La bla bla
  • 8,558
  • 13
  • 60
  • 109
  • You can read the NALUs as an image sequence; then use select filter to pick keyframes and some intermediate frames, then use drawtext to draw the frame number with an offset if you like. – Gyan Sep 24 '20 at 12:33
  • “Also, is it possible to skip some p-frames between the keyframes?” no. P frames can reference other I frames. In fact one h264 frame can reference multiple other frames. Up to 16 – szatmary Sep 24 '20 at 14:53
  • @szatmary, so if I don't take only i-frames, I have to take ALL the frames to avoid possible data corruption? – La bla bla Sep 24 '20 at 15:21
  • @Gyan, any chance you can explain a bit further? or supply some links with explanations? Thanks! – La bla bla Sep 24 '20 at 15:21
  • 1
    "so if I don't take only i-frames, I have to take ALL the frames to avoid possible data corruption?" Not possible corruption.... guaranteed corruption. – szatmary Sep 24 '20 at 18:09
  • Thanks, good to know! That would save me lots of time when I'll find out how to do what I'm trying to with the overlays :) – La bla bla Sep 24 '20 at 18:13

1 Answers1

0

I ended up using https://github.com/DaWelter/h264decoder and not ffmpeg directly.

It has a very simple interface, I just called

decoder = h264decoder.H264Decoder()
# For each file, in order call
framedatas = decoder.decode(data_in)
for framedata in framedatas:
    (frame, w, h, ls) = framedata
    self.process_frame(frame, path) # Using OpenCV to manipulate, resize, add the overlay, etc.

and after that piped the output data into ffmpeg to combine to make a MP4 file

La bla bla
  • 8,558
  • 13
  • 60
  • 109