0

I am trying to export a mutablecomposition with AVAssetExportSession. I am using MTAudioProcessingTap for processing audio data in the mutablecomposition, and based on the audio data, modify some animation in the mutablecomposition with AVVideoCompositionCoreAnimationTool. I want this to happen in real time, since the animation depends on the audio data. This works fine if I use AVPlayer, but not so much with AVAssetExportSession. It seems like the audio data received in MTAudioProcessingTapProcessCallback is out of sync with the video in AVAssetExportSession. So the question here is, does anyone know if they are supposed to be in sync during AVAssetExportSession?

Here's is my MTAudioProcessingTapProcessCallback

var tapProcess: MTAudioProcessingTapProcessCallback = {
    (tap, numberFrames, flags, bufferListInOut, numberFramesOut, flagsOut) in
            let status = MTAudioProcessingTapGetSourceAudio(tap, numberFrames, bufferListInOut, flagsOut, nil, numberFramesOut)

            let viewController = MTAudioProcessingTapGetStorage(tap)

            let viewController = Unmanaged<CanvasViewController>.fromOpaque(viewController).takeUnretainedValue()

            DispatchQueue.main.sync {
                // update my CALayer in viewController based on the audio data
            }
        }

Below is the code for setting up audiomix :

func setupAudioMix(audioTrack: AVAssetTrack) -> AVAudioMix {
        var callbacks = MTAudioProcessingTapCallbacks(
            version: kMTAudioProcessingTapCallbacksVersion_0,
            clientInfo: UnsafeMutableRawPointer(Unmanaged.passUnretained(self).toOpaque()),
            init: tapInit,
            finalize: tapFinalize,
            prepare: tapPrepare,
            unprepare: tapUnprepare,
            process: tapProcess)

        var tap: Unmanaged<MTAudioProcessingTap>?
        let err = MTAudioProcessingTapCreate(kCFAllocatorDefault, &callbacks, kMTAudioProcessingTapCreationFlag_PreEffects, &tap)

        print("err: \(err)\n")
        if err == noErr {
        }

        let inputParams = AVMutableAudioMixInputParameters(track: audioTrack)
        inputParams.audioTapProcessor = tap?.takeUnretainedValue()

        let audioMix = AVMutableAudioMix()
        audioMix.inputParameters = [inputParams]

        return audioMix
    }

And attach audiomix to AVAssetExportSession:

exporter = AVAssetExportSession(asset: mutableComposition!, presetName: AVAssetExportPresetMediumQuality)!
        exporter!.outputURL = exportUrl as URL
        exporter!.videoComposition = videoComposition!
        exporter!.audioMix = audioMix!

        exporter!.outputFileType = AVFileTypeMPEG4
        exporter!.shouldOptimizeForNetworkUse = true
karyboy
  • 317
  • 1
  • 5
  • 22

1 Answers1

0

No, AVAssetExportSession makes no promises that the audio tap will be called before, after or during the processing of similarly timed video frames.

Come to think of it, neither does AVAudioPlayer (AVPlayer?) - so maybe you're just getting lucky there.

Depending how you're generating your frames, it may be relevant that the AVAssetExportSession is probably running faster than 1x normal playback speed.

If you need to export video synched to audio, you'll probably need the lower level AVAssetWriter API.

Rhythmic Fistman
  • 34,352
  • 5
  • 87
  • 159
  • Thanks for the answer. A small correction, like you pointed, I meant AVPlayer, instead of AVAudioPlayer. I see. I will try to use AVAssetWriter. AVAssetWriter should be able to support writing CALayer animations on top the video? Also, I think the AVPlayer works fine coz the audio and video are supposed to be in sync for the viewer, and thats why my animations also look synced. – karyboy May 14 '17 at 08:55
  • It doesn't look like AVAssetWriter is also writing to file in sync. I created two AVAssetWriterInput (one audio and video) and added them to AVAssetWriter. I use AVAssetReaderAudioMixOutput and AVAssetReaderVideoCompositionOutput, to read from the asset with copyNextSampleBuffer. I attach AudioMix to AVAssetReaderAudioMixOutput. MTAudioProcessingTapProcessCallback updates the CALayer. But its not in sync in the exported file. – karyboy May 19 '17 at 06:59
  • I logged presentation time for both AVAssetWriterInput's before appending their respective CMSampleBuffer using CMSampleBufferGetOutputPresentationTimeStamp, it seems like the audio buffer is ahead of the video buffer by approx 6 seconds. Is it possible to keep them in sync somehow? Here's how my code looks https://pastebin.com/kTzpzm1D – karyboy May 19 '17 at 07:02
  • You should be able to do this by processing the audio and creating your `CALayer` animations as part of an `AVMutableComposition` which you then export with `AVAssetExportSession` (along with the audio). e.g. http://stackoverflow.com/q/20391651/22147 and http://stackoverflow.com/a/18798287/22147 This way you won't be mixing realtime/non-realtime APIs & everything should be in sync. – Rhythmic Fistman May 19 '17 at 22:43
  • But that was the original way I was doing it, as you can see in the question. The problem with that is MTAudioProcessingTap isnt in sync while exporting with AVAssetExportSession. Now, in your comment's first sentence, if you mean that I process the complete audio first (as in store audio data in memory with timestamps) and then apply that on CALayer while exporting, I could do that except I dont think AVAssetExportSession has a callback with current time stamp. – karyboy May 19 '17 at 23:37
  • I was thinking on similar terms, except I d use AVAssetWriter and not AVAssetExportSession, since I can get the CMSampleBufferGetOutputPresentationTimeStamp while appending video samples. – karyboy May 19 '17 at 23:37