I'm trying to convert AVAudioPCMBuffer
I got from AVAudioNode
's tap block into CMSampleBuffer
to append audio input of AVAssetWriter
. I'm also creating a new sample buffer with the correct timing - delta of the buffer's time and the time asset writer began writing. The code actually works correctly if I use AVCaptureAudioDataOutput
from camera but not in case of pcm data I get from AVAudioEngine
.
I'm injecting correct audio settings. Getting it from the output format of the same node I'm installing tap on. I.e., engine.mainMixerNode.outputFormat(forBus: 0).settings
This is the code I'm using for conversion between pcm and sample buffers I found on Github.
public extension AVAudioPCMBuffer {
/// Converts `AVAudioPCMBuffer` to `CMSampleBuffer`
/// - Returns: `CMSampleBuffer`
func asCMSampleBuffer() -> CMSampleBuffer? {
let audioBufferList = mutableAudioBufferList
let asbd = format.streamDescription
var sampleBuffer: CMSampleBuffer? = nil
var format: CMFormatDescription? = nil
var status = CMAudioFormatDescriptionCreate(
allocator: kCFAllocatorDefault,
asbd: asbd,
layoutSize: 0,
layout: nil,
magicCookieSize: 0,
magicCookie: nil,
extensions: nil,
formatDescriptionOut: &format
)
guard (status == noErr) else { return nil }
var timing: CMSampleTimingInfo = CMSampleTimingInfo(
duration: CMTime(value: 1, timescale: Int32(asbd.pointee.mSampleRate)),
presentationTimeStamp: CMClockGetTime(CMClockGetHostTimeClock()),
decodeTimeStamp: CMTime.invalid
)
status = CMSampleBufferCreate(
allocator: kCFAllocatorDefault,
dataBuffer: nil,
dataReady: false,
makeDataReadyCallback: nil,
refcon: nil,
formatDescription: format,
sampleCount: CMItemCount(frameLength),
sampleTimingEntryCount: 1,
sampleTimingArray: &timing,
sampleSizeEntryCount: 0,
sampleSizeArray: nil,
sampleBufferOut: &sampleBuffer
)
guard (status == noErr) else {
Log.error(
category: "AVAudioPCMBuffer to CMSampleBuffer",
message: "CMSampleBufferCreate returned error",
metadata: ["Error": status]
)
return nil
}
guard let sampleBuffer else {
Log.error(
category: "AVAudioPCMBuffer to CMSampleBuffer",
message: "Sample buffer not found"
)
return nil
}
status = CMSampleBufferSetDataBufferFromAudioBufferList(
sampleBuffer,
blockBufferAllocator: kCFAllocatorDefault,
blockBufferMemoryAllocator: kCFAllocatorDefault,
flags: 0,
bufferList: audioBufferList
)
guard (status == noErr) else {
Log.error(
category: "AVAudioPCMBuffer to CMSampleBuffer",
message: "CMSampleBufferSetDataBufferFromAudioBufferList returned error",
metadata: ["Error": status]
)
return nil
}
return sampleBuffer
}
}
This is how I create a new sample buffer with a new timing. time
is the time AVAssetWriter
began writing.
func getCorrectTimeStampedBuffer(from buffer: CMSampleBuffer) -> CMSampleBuffer? {
let timeStamp = CMSampleBufferGetPresentationTimeStamp(buffer).seconds
let newTime: CMTime = .init(seconds: timeStamp - time, preferredTimescale: timeScale)
var newBuffer: CMSampleBuffer?
var count: Int = 0
var info = CMSampleTimingInfo(
duration: newTime,
presentationTimeStamp: newTime,
decodeTimeStamp: newTime
)
CMSampleBufferGetSampleTimingInfoArray(
buffer,
entryCount: 0,
arrayToFill: &info,
entriesNeededOut: &count
)
info.decodeTimeStamp = newTime
info.presentationTimeStamp = newTime
CMSampleBufferCreateCopyWithNewTiming(
allocator: kCFAllocatorDefault,
sampleBuffer: buffer,
sampleTimingEntryCount: count,
sampleTimingArray: &info,
sampleBufferOut: &newBuffer
)
return newBuffer
}
I've double-checked and audio settings is used correctly as I'm setting it before starting writing.
func getAudioInput() -> AVAssetWriterInput {
let input = AVAssetWriterInput(mediaType: .audio, outputSettings: audioSettings)
input.expectsMediaDataInRealTime = true
return input
}
I've installed an observer on status.
writer.publisher(for: \.status)
.filter { $0 == .failed }
.sink { [writer] value in
print("failed", writer.error!)
}
.store(in: &cancellables)
I'm appending audio like so.
func appendAudio(buffer: CMSampleBuffer) {
guard audioInput?.isReadyForMoreMediaData == true else {
Log.debug(
category: "VideoRecorder",
message: "Audio input can't be appended as it's not ready for more media data"
)
return
}
let newBuffer = getCorrectTimeStampedBuffer(from: buffer)
let hasAppended = audioInput!.append(newBuffer ?? buffer)
if hasAppended {
Log.debug(category: "VideoRecorder appendAudio", message: "Buffer appended")
} else {
Log.error(
category: "VideoRecorder appendAudio",
message: "Could not append audio",
metadata: [
"Asset writer status": String(describing: assetWriter?.status.rawValue),
"Asset writer error": String(describing: assetWriter?.error),
"Asset writer error description": String(describing: assetWriter?.error?.localizedDescription)
]
)
}
}
Error occurs right after calling audioInput!.append(newBuffer ?? buffer)
. Before that the status is .writing
.
Hence, hasAppeared
is false
.
What's worst is that the error doesn't say anything. I couldn't even find any information using -12780
code. Does AVAssetWriter
provide any contextual error at all?
This is the returned error
Error Domain=AVFoundationErrorDomain Code=-11800 "The operation could not be completed" UserInfo={NSLocalizedFailureReason=An unknown error occurred (-12780), NSLocalizedDescription=The operation could not be completed, NSUnderlyingError=0x283895a40 {Error Domain=NSOSStatusErrorDomain Code=-12780 "(null)"}}