0

I'm trying to convert AVAudioPCMBuffer I got from AVAudioNode's tap block into CMSampleBuffer to append audio input of AVAssetWriter. I'm also creating a new sample buffer with the correct timing - delta of the buffer's time and the time asset writer began writing. The code actually works correctly if I use AVCaptureAudioDataOutput from camera but not in case of pcm data I get from AVAudioEngine.

I'm injecting correct audio settings. Getting it from the output format of the same node I'm installing tap on. I.e., engine.mainMixerNode.outputFormat(forBus: 0).settings

This is the code I'm using for conversion between pcm and sample buffers I found on Github.

public extension AVAudioPCMBuffer {
    /// Converts `AVAudioPCMBuffer` to `CMSampleBuffer`
    /// - Returns: `CMSampleBuffer`
    func asCMSampleBuffer() -> CMSampleBuffer? {
        let audioBufferList = mutableAudioBufferList
        let asbd = format.streamDescription

        var sampleBuffer: CMSampleBuffer? = nil
        var format: CMFormatDescription? = nil
        
        var status = CMAudioFormatDescriptionCreate(
            allocator: kCFAllocatorDefault,
            asbd: asbd,
            layoutSize: 0,
            layout: nil,
            magicCookieSize: 0,
            magicCookie: nil,
            extensions: nil,
            formatDescriptionOut: &format
        )
        guard (status == noErr) else { return nil }
        
        var timing: CMSampleTimingInfo = CMSampleTimingInfo(
            duration: CMTime(value: 1, timescale: Int32(asbd.pointee.mSampleRate)),
            presentationTimeStamp: CMClockGetTime(CMClockGetHostTimeClock()),
            decodeTimeStamp: CMTime.invalid
        )
        status = CMSampleBufferCreate(
            allocator: kCFAllocatorDefault,
            dataBuffer: nil,
            dataReady: false,
            makeDataReadyCallback: nil,
            refcon: nil,
            formatDescription: format,
            sampleCount: CMItemCount(frameLength),
            sampleTimingEntryCount: 1,
            sampleTimingArray: &timing,
            sampleSizeEntryCount: 0,
            sampleSizeArray: nil,
            sampleBufferOut: &sampleBuffer
        )
        guard (status == noErr) else {
            Log.error(
                category: "AVAudioPCMBuffer to CMSampleBuffer",
                message: "CMSampleBufferCreate returned error",
                metadata: ["Error": status]
            )
            return nil
        }
        guard let sampleBuffer else {
            Log.error(
                category: "AVAudioPCMBuffer to CMSampleBuffer",
                message: "Sample buffer not found"
            )
            return nil
        }
        
        status = CMSampleBufferSetDataBufferFromAudioBufferList(
            sampleBuffer,
            blockBufferAllocator: kCFAllocatorDefault,
            blockBufferMemoryAllocator: kCFAllocatorDefault,
            flags: 0,
            bufferList: audioBufferList
        )
        guard (status == noErr) else {
            Log.error(
                category: "AVAudioPCMBuffer to CMSampleBuffer",
                message: "CMSampleBufferSetDataBufferFromAudioBufferList returned error",
                metadata: ["Error": status]
            )
            return nil
        }
        
        return sampleBuffer
    }
}

This is how I create a new sample buffer with a new timing. time is the time AVAssetWriter began writing.

    func getCorrectTimeStampedBuffer(from buffer: CMSampleBuffer) -> CMSampleBuffer? {
        let timeStamp = CMSampleBufferGetPresentationTimeStamp(buffer).seconds
        let newTime: CMTime = .init(seconds: timeStamp - time, preferredTimescale: timeScale)
        var newBuffer: CMSampleBuffer?
        var count: Int = 0
        var info = CMSampleTimingInfo(
            duration: newTime,
            presentationTimeStamp: newTime,
            decodeTimeStamp: newTime
        )
        CMSampleBufferGetSampleTimingInfoArray(
            buffer,
            entryCount: 0,
            arrayToFill: &info,
            entriesNeededOut: &count
        )
        info.decodeTimeStamp = newTime
        info.presentationTimeStamp = newTime
        CMSampleBufferCreateCopyWithNewTiming(
            allocator: kCFAllocatorDefault,
            sampleBuffer: buffer,
            sampleTimingEntryCount: count,
            sampleTimingArray: &info,
            sampleBufferOut: &newBuffer
        )
        return newBuffer
    }

I've double-checked and audio settings is used correctly as I'm setting it before starting writing.

    func getAudioInput() -> AVAssetWriterInput {
        let input = AVAssetWriterInput(mediaType: .audio, outputSettings: audioSettings)
        input.expectsMediaDataInRealTime = true
        return input
    }

I've installed an observer on status.

writer.publisher(for: \.status)
    .filter { $0 == .failed }
        .sink { [writer] value in
            print("failed", writer.error!)
        }
        .store(in: &cancellables)

I'm appending audio like so.

func appendAudio(buffer: CMSampleBuffer) {
        guard audioInput?.isReadyForMoreMediaData == true else {
            Log.debug(
                category: "VideoRecorder",
                message: "Audio input can't be appended as it's not ready for more media data"
            )
            return
        }
        let newBuffer = getCorrectTimeStampedBuffer(from: buffer)
        let hasAppended = audioInput!.append(newBuffer ?? buffer)
        if hasAppended {
            Log.debug(category: "VideoRecorder appendAudio", message: "Buffer appended")
        } else {
            Log.error(
                category: "VideoRecorder appendAudio",
                message: "Could not append audio",
                metadata: [
                    "Asset writer status": String(describing: assetWriter?.status.rawValue),
                    "Asset writer error": String(describing: assetWriter?.error),
                    "Asset writer error description": String(describing: assetWriter?.error?.localizedDescription)
                ]
            )
        }
    }

Error occurs right after calling audioInput!.append(newBuffer ?? buffer). Before that the status is .writing.

Hence, hasAppeared is false.

What's worst is that the error doesn't say anything. I couldn't even find any information using -12780 code. Does AVAssetWriter provide any contextual error at all?

This is the returned error Error Domain=AVFoundationErrorDomain Code=-11800 "The operation could not be completed" UserInfo={NSLocalizedFailureReason=An unknown error occurred (-12780), NSLocalizedDescription=The operation could not be completed, NSUnderlyingError=0x283895a40 {Error Domain=NSOSStatusErrorDomain Code=-12780 "(null)"}}

  • I might have found issues. 1 is that by the time video buffer is received which invokes start method audio settings are not set yet (That's one major issue). 2 is that I'm sending non-interleved signal (hence the settings also provide `AVLinearPCMIsNonInterleaved` as 1 and after checking `canApply(outputSettings:, forMediaType: )` of asset writer it returns false if non-interleaved is true. I'll try to fix this and if it works provide the solution. Fingers crossed it will work. Also, I've removed timestamp conversion and provided timestamp of first buffer as source time to AVAssetWriter. – Kakhi Kiknadze Mar 25 '23 at 11:36

0 Answers0