Writing encoded audio CMSampleBuffer not working

Question

I'm using AudioConverter to convert uncompressed CMSampleBuffer being captured via AVCaptureSession to AudioBufferList:

let packetDescriptionsPtr = UnsafeMutablePointer<AudioStreamPacketDescription>.allocate(capacity: 1)

AudioConverterFillComplexBuffer(
                converter,
                inputDataProc,
                Unmanaged.passUnretained(self).toOpaque(),
                &ioOutputDataPacketSize,
                outOutputData.unsafeMutablePointer,
                packetDescriptionsPtr
            )

I am then constructing a CMSampleBuffer containing compressed data using packet descriptions like so:

CMAudioSampleBufferCreateWithPacketDescriptions(
        allocator: kCFAllocatorDefault,
        dataBuffer: nil,
        dataReady: false,
        makeDataReadyCallback: nil,
        refcon: nil,
        formatDescription: formatDescription!,
        sampleCount: Int(data.unsafePointer.pointee.mNumberBuffers),
        presentationTimeStamp: presentationTimeStamp,
        packetDescriptions: &packetDescriptions,
        sampleBufferOut: &sampleBuffer)

When I tried saving the buffer using AVAssetWriter I got the following error: -[AVAssetWriterInput appendSampleBuffer:] Cannot append sample buffer: First input buffer must have an appropriate kCMSampleBufferAttachmentKey_TrimDurationAtStart since the codec has encoder delay'

I decided to prime the first three buffers knowing that each is of consistent length:

if self.receivedAudioBuffers < 2 {
                    let primingDuration = CMTimeMake(value: 1024, timescale: 44100)
                    CMSetAttachment(sampleBuffer,
                                    key: kCMSampleBufferAttachmentKey_TrimDurationAtStart,
                                    value: CMTimeCopyAsDictionary(primingDuration, allocator: kCFAllocatorDefault),
                                    attachmentMode: kCMAttachmentMode_ShouldNotPropagate)
                    self.receivedAudioBuffers += 1
                }
                else if self.receivedAudioBuffers == 2 {
                    let primingDuration = CMTimeMake(value: 64, timescale: 44100)
                    CMSetAttachment(sampleBuffer,
                                    key: kCMSampleBufferAttachmentKey_TrimDurationAtStart,
                                    value: CMTimeCopyAsDictionary(primingDuration, allocator: kCFAllocatorDefault),
                                    attachmentMode: kCMAttachmentMode_ShouldNotPropagate)
                    self.receivedAudioBuffers += 1
                }

Now I no longer get the error and neither do I get any errors when appending the sample but the audio doesn't play in the recording and also messes up the whole video file (seems like the timing info gets corrupted).

Is there anything here that I'm missing? How should I correctly append audio CMSampleBuffer?