Audio playing but video freezes AVPlayer on slow internet

Question

I load my AVAsset asynchronously through this method:

playerAsset.loadValuesAsynchronously(forKeys: ["duration", "playable", "hasProtectedContent"]) {
  DispatchQueue.main.async {[weak self] in

    guard let strongSelf = self else { return }

    let playerItem = AVPlayerItem(asset: playerAsset)
    self.player?.replaceCurrentItem(with: playerItem)
  }
}

On slow internet, my asset sometimes plays the audio completely but freezes on a certain video frame, or plays the video without audio. On seeking back to the beginning, the video plays properly.

I have KVO observers for playbackBufferEmpty and playbackLikelyToKeepUp, but they are not called in this situation.