6

I'm trying to write an AAC audio stream into an mp4 file using the FFMPEG libraries. I am using a custom IO context that writes directly to a socket so I have to set ioContext->seekable = 0. To make this work I had to add the "movflags" empty_moov and frag_keyframe when writing the header.

After writing the output to a file on the other end of the socket, I can play the file in VLC or Windows Media Player. However, seeking to a specific position in the file is not working properly in both players. WMP also does not show the total duration and VLC only flashes it shortly when reaching the end of the audio.

Is there a way to add more metadata when muxing so the players are able to treat the file as if it was not written as a stream? Transfer via the socket is not interrupted abruptly, so I could write metadata at the end of the file. I also know the total duration in advance, so I could add it to the header of the file if it was possible. I cannot use the faststart flag because this would require output to a seekable file before writing to the socket.

Update: I learned that I can set the duration in AVFormatContext and I can set nb_frames and avg_frame_rate in AVStream. However, it doesn't solve my problem. When I set the codecContext flag AV_CODEC_FLAG_QSCALE, VLC seems to be able to estimate the total time. However, seeking still doesn't work.

lex82
  • 11,173
  • 2
  • 44
  • 69
  • This **[link](http://www.bogotobogo.com/VideoStreaming/ffmpeg_AdaptiveLiveStreaming_SmoothStreaming.php)** might help you understand your issues with MOV flags (word search : `-movflags empty_moov` and read rest of section). So you described the problem but what is context of application? Is receiveing audio and you want to build up an MP4 until X amount then save to file? Why not first buffer all the required AAC and then just encode to final MP4? Anyways consider that : # All audio frames are keyframes, # MPEG decoders can't seek if there is no metadata, # Try M4a output then rename as MP4? – VC.One Apr 15 '16 at 19:51
  • PS: I'm used to FFmpeg as a process (not as directly imported library) but as a running process I would use `ffmpeg -i - -c:a libfdk_aac -b:a 128k output.mp4` here I'd be using `-i -` to let it know there is no input file and instead just repeatedly write bytes of each AAC frame to `process.writeBytes(myAACframe);` until end of AAC where I exit the process and there is a playable file.. In short, a dynamically built-up MP4 file by appending AAC frames as available. Maybe something in that logic can help you? – VC.One Apr 15 '16 at 19:53
  • Why don't you just respond instead of burning away 100 rep points into thin air with no answer? You could add comments that would have helped others to help you. If you want a seekable MP4 file make sure it has metadata as that's **a neccessity for seeking** to MPEG decoders. (This means lose that `-movflags empty_moov & frag_keyframe` since they add to the problem). I suggested the "process" method as that's an easy test, without worrying if your full code is proper ( eg: `nb_frames` etc). It can be done in C++, Java or C# depending on your language... – VC.One Apr 18 '16 at 10:20
  • PS: What duration of the AAC are we talking about here? I still think your best bet is to just collect the input bytes to a buffer and when ready (got 5? 10? 60 mins?) just mux into MP4 and send that down the socket. Remember FFmpeg only adds the metadata after file creation is complete (except here it will be gone down the socket so it can't edit the bytes unlike if it was kept in memory or disk to update the metadata entries first before sending). – VC.One Apr 18 '16 at 10:30
  • @VC.One Sorry for not responding. I haven't been in the office the last 2 days and was not working on the problem. Your hint that all audio frames are keyframes would have probably helped me if I had read it earlier. I just played around with the flags and it works when I drop the empty_moov. When I then exchange frag_keyframe with another fragmentation option (bytes or milliseconds) it also creates smaller fragments and not one huge fragment. – lex82 Apr 18 '16 at 10:32
  • @VC.One Thank you very much for your support. If you copy your comments to an answer I'd be happy to give you the 100 rep. – lex82 Apr 18 '16 at 10:33
  • But to answer your question: I had to write to a socket immediately even for a 3min audio file. The receiver should be able to start playing the file immediately as well as saving it for playing it later (with seeking). – lex82 Apr 18 '16 at 10:35

1 Answers1

3

If you want a seekable MP4 file make sure it definately has metadata as that's a neccessity for MPEG decoders to handle seeking (the MP4 metadata lists the start positions of each AAC frame's bytes.
This means lose that -movflags empty_moov & frag_keyframe since they add to the problem.

Consider:

  • You don't need frag_keyframe since all audio frames are classed as keyframes.
  • Don't force empty_moov since MP4 decoders can't seek if there is no metadata. (FFmpeg wll take care of metadata in fragmentation mode).

A quote from this excellent guide on streaming :

Writing a fragmented file has the advantage that the file is decodable even if the writing is interrupted (while a normal MOV/MP4 is undecodable if it is not properly finished), and it requires less memory when writing very long files (since writing normal MOV/MP4 files stores info about every single packet in memory until the file is closed). The downside is that it is less compatible with other applications.

Options you could try are:

  • frag_duration [num] Create fragments that are num milliseconds long.
  • frag_size [num] Create fragments that contain up to num bytes size of payload.
VC.One
  • 14,790
  • 4
  • 25
  • 57
  • Did any of this actually work? I'm facing a similar problem because my AAC files when streamed over http are not seekable on Android MediaPlayer after 5.0.1 and 5.1. The audio plays but if I try to use seekTo() the audio starts over at 0. The problem is this works in older versions of Android 5.0.1, 4.4.2, etc. I already have the moov element at the head of the file and it IS streaming. It's just not seekable. What mp4 atoms should I look for in my streams to know if the metadata is there? – chubbsondubs May 31 '17 at 19:04
  • I don't think this answer is useful if you want to use fMP4 in browsers. At least under Chrome you need `empty_moov` and `frag_keyframe` (at least from what I can tell). – Stefan Falk Nov 08 '20 at 11:14
  • @chubbsondubs Did you end up with a working solution? :/ – Stefan Falk Nov 08 '20 at 11:14
  • 1
    @StefanFalk Ok I'm trying to remember it all, and yes we did fix it. The AAC streams had originally been encoded as video streams and keyframes weren't encoded in it. The fix was to re-encode all material with ffmpeg AAC as an audio stream. With audio frames every frame is a keyframe. This was a big deal because we were re-encoding something on the order of iTunes. In my case we had the metadata at the front of the stream, but it wouldn't allow seeking until we re-encoded it. – chubbsondubs Nov 09 '20 at 15:23