0

I am having problems with HLS stream creation that seems to be either on FFMPEG or iOS AVFoundation side.

I am currently working on a video app which allows user to record, cut and merge multiple clips together. That is done using AVMutableComposition and code like this:

try videoTrack?.insertTimeRange(clip.range, of: videoAssetTrack, at: accumulatedTime)
try audioTrack?.insertTimeRange(clip.range, of: audioAssetTrack, at: accumulatedTime)

and then it gets exported using 

AVAssetExportSession(asset: composition, presetName: preferredPreset)

Then it gets uploaded to a server which creates a HLS stream out of the video using following code:

ffmpeg -i bunny.mov -y -g 48 -sc_threshold 0 \
-map 0:0 -map 0:1 -s:v:0 1280x720 -c:v:0 h264 -b:v:0 4000k \
-map 0:0 -map 0:1 -s:v:1 854x480 -c:v:1 h264 -b:v:1 2000k \
-map 0:0 -map 0:1 -s:v:2 640x360 -c:v:2 h264 -b:v:2 1000k \
-map 0:0 -map 0:1 -s:v:3 480x270 -c:v:3 h264 -b:v:3 500k \
-c:a copy -var_stream_map "v:0,a:0 v:1,a:1 v:2,a:2 v:3,a:3" \
-master_pl_name bunny.m3u8 -f hls -hls_time 1 -hls_list_size 0 \
-hls_segment_filename ~/Downloads/bunny/bunny.%v.%03d.ts \
-hls_base_url http://tomaskohout.cz/bunny/ ~/Downloads/bunny/bunny.%v.m3u8

Inputting a video that was cut together using two clips (first a bird and second a bunny) will result in following m3u8 file (open in safari to see it play): http://tomaskohout.cz/bunny/bunny.m3u8

The problem is that the audio is overlapping and out of sync after the first cut (notice the sound of nut hitting the bird playing twice) and last second is just a still frame.

The original video for comparison is here: http://tomaskohout.cz/bunny/original.mov

Seems like it is a problem in AVFoundation in general. If you join two clips using QuickTime (with Edit > Add Clip to End) and use the resulting video as the input for ffmpeg it will also have audio and video out of sync. It just seems it generates some strangely timed videos that only quicktime knows how to play. If you take the original video and try to play it using VLC it will be also out of sync and even show part of the clip that were not meant to be in the original cut. It might be related to the Encoder Delay and Synchronization but I am really out of my element here.

I have found a workaround which is to first convert the video to mp4 (in my case it's just converting mp4 to mp4) like this:

ffmpeg -i input.mp4 -c:v:0 h264 input.preexport.mp4

and then use the pre-exported version to generate the m3u8 file. This seems to retime the video correctly.

That is of course not ideal because I have to run through the video twice.

So I would like to ask whether there's either an option which allows me to retime the video at the same time while I am generating the hls or even better if there is some other solution to the problem.

Thank you!

Tomas

kefren
  • 1,042
  • 7
  • 12
Tomáš Kohout
  • 755
  • 6
  • 15
  • The problem here is that there is not one way of doing this. Audio and video are almost never the exact same length in a give file. So when joining, should you concat all audio or attempt to find overlapping audio and trim it? There is no correct answer, ffmpeg chose one method, Apple chose another. If you prefer the other method, you probably have to code it yourself. – szatmary Dec 02 '19 at 17:36
  • Thank you for your reply! I am not sure whether I understand what are you saying. If I run `ffmpeg -i input.mp4 -c:v:0 h264 input.preexport.mp4` before converting to HLS it all works well. So there is a way of doing it. I would only like to save myself one extra pass through the video ... – Tomáš Kohout Dec 02 '19 at 21:06

0 Answers0