I am trying to export a mutablecomposition with AVAssetExportSession. I am using MTAudioProcessingTap for processing audio data in the mutablecomposition, and based on the audio data, modify some animation in the mutablecomposition with AVVideoCompositionCoreAnimationTool. I want this to happen in real time, since the animation depends on the audio data. This works fine if I use AVPlayer, but not so much with AVAssetExportSession. It seems like the audio data received in MTAudioProcessingTapProcessCallback
is out of sync with the video in AVAssetExportSession. So the question here is, does anyone know if they are supposed to be in sync during AVAssetExportSession?
Here's is my MTAudioProcessingTapProcessCallback
var tapProcess: MTAudioProcessingTapProcessCallback = {
(tap, numberFrames, flags, bufferListInOut, numberFramesOut, flagsOut) in
let status = MTAudioProcessingTapGetSourceAudio(tap, numberFrames, bufferListInOut, flagsOut, nil, numberFramesOut)
let viewController = MTAudioProcessingTapGetStorage(tap)
let viewController = Unmanaged<CanvasViewController>.fromOpaque(viewController).takeUnretainedValue()
DispatchQueue.main.sync {
// update my CALayer in viewController based on the audio data
}
}
Below is the code for setting up audiomix :
func setupAudioMix(audioTrack: AVAssetTrack) -> AVAudioMix {
var callbacks = MTAudioProcessingTapCallbacks(
version: kMTAudioProcessingTapCallbacksVersion_0,
clientInfo: UnsafeMutableRawPointer(Unmanaged.passUnretained(self).toOpaque()),
init: tapInit,
finalize: tapFinalize,
prepare: tapPrepare,
unprepare: tapUnprepare,
process: tapProcess)
var tap: Unmanaged<MTAudioProcessingTap>?
let err = MTAudioProcessingTapCreate(kCFAllocatorDefault, &callbacks, kMTAudioProcessingTapCreationFlag_PreEffects, &tap)
print("err: \(err)\n")
if err == noErr {
}
let inputParams = AVMutableAudioMixInputParameters(track: audioTrack)
inputParams.audioTapProcessor = tap?.takeUnretainedValue()
let audioMix = AVMutableAudioMix()
audioMix.inputParameters = [inputParams]
return audioMix
}
And attach audiomix to AVAssetExportSession:
exporter = AVAssetExportSession(asset: mutableComposition!, presetName: AVAssetExportPresetMediumQuality)!
exporter!.outputURL = exportUrl as URL
exporter!.videoComposition = videoComposition!
exporter!.audioMix = audioMix!
exporter!.outputFileType = AVFileTypeMPEG4
exporter!.shouldOptimizeForNetworkUse = true