Unable to play ACC stream using AVSampleBufferAudioRenderer

Question

I am working on an app that receives a stream of packets with video and audio. I was able to decode the video and play it using AVSampleBufferDisplayLayer. (Code can be found here) But I've been struggling for over a week to decode the audio. At the beginning of the stream, I receive an audio description like this -> sampleRate=48000, channelCount=1, and profileLevel=2.Then I continuously receive packets of ACC data which I try to decode in a similar way as I treat the video packets. In the beginning, I create AudioStreamBasicDescription, CMAudioFormatDescription, and setup the audioRenderer and audioRendererSynchoronizer.

   class AudioDecoderPlayer: NSObject {
  private var audioRenderer = AVSampleBufferAudioRenderer()
  private var audioRendererSynchoronizer = AVSampleBufferRenderSynchronizer()
  private let serializationQueue = DispatchQueue(label: "sample.buffer.player.serialization.queue")

  private var audioStreamBasicDescription: AudioStreamBasicDescription
  private var formatDescription: CMAudioFormatDescription?

  private var outputQueue: AudioQueueRef?

  var sampleBuffers: [CMSampleBuffer] = []
  
  let sampleRate: Double
  let channels: Int

  init(sampleRate: Double = 48000, channels: Int = 1, profileLevel: Int = 2) {

    self.sampleRate = sampleRate
    self.channels = channels
    
    let uChannels = UInt32(channels)
    let channelBytes = UInt32(MemoryLayout<Int16>.size)
    let bytesPerFrame = uChannels * channelBytes


    self.audioStreamBasicDescription = AudioStreamBasicDescription(
      mSampleRate: Float64(sampleRate),
      mFormatID: kAudioFormatMPEG4AAC,
      mFormatFlags: AudioFormatFlags(profileLevel),
      mBytesPerPacket: bytesPerFrame,
      mFramesPerPacket: 1,
      mBytesPerFrame: bytesPerFrame,
      mChannelsPerFrame: uChannels,
      mBitsPerChannel: channelBytes * 8,
      mReserved: 0
    )

    super.init()
    
    let status = CMAudioFormatDescriptionCreate(allocator: kCFAllocatorDefault, asbd: &audioStreamBasicDescription, layoutSize: 0, layout: nil, magicCookieSize: 0, magicCookie: nil, extensions: nil, formatDescriptionOut: &formatDescription)

    if status != noErr {
      fatalError("unable to create audio format description")
    }
    
    
    audioRendererSynchoronizer.addRenderer(audioRenderer)
    subscribeToAudioRenderer()
    startPlayback()
  }
  
  func subscribeToAudioRenderer() {
    audioRenderer.requestMediaDataWhenReady(on: serializationQueue, using: { [weak self] in
      guard let strongSelf = self else {
        return
      }
      
      while strongSelf.audioRenderer.isReadyForMoreMediaData {
        if let sampleBuffer = strongSelf.nextSampleBuffer() {
          strongSelf.audioRenderer.enqueue(sampleBuffer)
        }
      }
    })
  }
  
  func startPlayback() {
    serializationQueue.async {
      if self.audioRendererSynchoronizer.rate != 1 {
        self.audioRendererSynchoronizer.rate = 1
        self.audioRenderer.volume = 1.0
      }
    }
  }
  
  func nextSampleBuffer() -> CMSampleBuffer? {
    guard sampleBuffers.count > 0 else {
      return nil
    }
    
    let sampleBuffer = sampleBuffers.first
    sampleBuffers.remove(at: 0)
    
    return sampleBuffer
  }

And this is how my decode function looks like

  func decodeAudioPacket(data: Data) {
    let headerValue = UInt32(data.count)

    //add data lenght at the beginning
    var sizedData = withUnsafeBytes(of: headerValue.bigEndian) { Data($0) }
    sizedData.append(data)

    let blockBuffer = sizedData.toCMBlockBuffer()

    // Outputs from CMSampleBufferCreate
    var sampleBuffer: CMSampleBuffer?

    let result = CMAudioSampleBufferCreateReadyWithPacketDescriptions(
      allocator: kCFAllocatorDefault,
      dataBuffer: blockBuffer,
      formatDescription: formatDescription!,
      sampleCount: 1,
      presentationTimeStamp: CMTime(value: 1, timescale: Int32(sampleRate)),
      packetDescriptions: nil,
      sampleBufferOut: &sampleBuffer)


    if result != noErr {
      fatalError("CMSampleBufferCreate() failed")
    }

    sampleBuffers.append(sampleBuffer!)
  }

I found out that requestMediaDataWhenReady is only called once which suggests, that the sound is not being played, but I don't understand why. I was able to make it work by editing code from WWDC 2017 session 509 but there is still no sound being played. (Code can be found here) I've also tried to use various solutions with AudioQueue but with no success. (For some reasons the callbacks from AudioFileStreamOpen were never called) (Code can be found here and here). But I would prefer to solve it using AVSampleBufferAudioRenderer as I believe it should be easier and I also want to use AVSampleBufferAudioRendererSynchronizer to synchronize the video with the audio.

Any suggestions on what I do wrong is appreciated.

Thank you