How to get a new AVPlayer to play instantly after a different one ends?

Question

I'm using AVPlayer to handle playing a list of remote and/or local audio files. Originally I was using AVQueuePlayer, setting all the items once and letting them play. This results in really seamless playback from one file to the next.

However, I decided to implement my own "queue" system because I need to manipulate that list a lot, or skip over certain items due to certain conditions that are calculated on the fly, etc. As soon as I implemented this, even with caching/preloading, I found that there are gaps in the playback even when they're all local files.

My implementation looks roughly like this, using multiple AVPlayers to preload/cache the audio to avoid delays:

func playFile(audioURL: AudioURL, nextAudioURL: AudioURL?, playImmediately: Bool) {
    let player: AVPlayer
    let playerItem: AVPlayerItem

    // Attempt to use a cached AVPlayer matching this url
    if let cachedAudioPlayer = self.cachedAudioPlayer(forURL: audioURL), let item = cachedAudioPlayer.player.currentItem {
        print("Using cached audio player")
        playerItem = item
        player = cachedAudioPlayer.player
        
        // Remove from cache
        self.cachedAudioPlayers.removeAll(where: { $0.audioURL == audioURL })
    }
    else {
        print("Using brand new audio player")
        playerItem = AVPlayerItem(asset: AVURLAsset(url: audioURL.url))
        
        player = AVPlayer(playerItem: playerItem)
        // NOTE: Explicitly setting `automaticallyWaitsToMinimizeStalling` to `false` seems to have no affect, which makes sense since it will usually be loaded by the time we get here anyway, and the documentation claims it only affects streaming over http.
    }

    let oldPlayer = self.avPlayer
    let oldAudioURL = self.currentAudioURL

    self.avPlayer = player
    self.currentAudioURL = audioURL

    // Listen for when the file ends
    NotificationCenter.default.addObserver(self, selector: #selector(self.handleNotification(_:)), name: .AVPlayerItemDidPlayToEndTime, object: playerItem)
 
    if playImmediately {
        player.play()
    }
    
    // If there was a player already, pause it now
    oldPlayer?.pause()

    // Cache the current (now "old") player if available
    if let oldAudioURL = oldAudioURL, let oldPlayer = oldPlayer {
        // Reset time so it's ready to go
        oldPlayer.seek(to: CMTime(seconds: 0, preferredTimescale: 1000000), completionHandler: { _ in })
        // Add to cache
        self.cachedAudioPlayers.append(.init(player: oldPlayer, audioURL: oldAudioURL))
    }
                
    // Cache the next player so it's ready to go when needed
    if let nextAudioURL = nextAudioURL, self.cachedAudioPlayer(forURL: nextAudioURL) == nil {
        print("Caching next audio url")
        let nextPlayerItem = AVPlayerItem(url: nextAudioURL.url)
        let nextPlayer = AVPlayer(playerItem: nextPlayerItem)
        self.cachedAudioPlayers.append(.init(player: nextPlayer, audioURL: nextAudioURL))
    }

    // Trim cache, keeping the latest added
    self.cachedAudioPlayers = self.cachedAudioPlayers.suffix(5)

}

I then use the AVPlayerItemDidPlayToEndTime notification to play the next file using the same play method, and so on.

This overall works decently well, but it has a significant gap, even for local files, when playing the next file when compared to AVQueuePlayer. I can confirm that the implementation does use a cached player (usually sitting ready to go for >30 seconds) and that with identical files, AVQueuePlayer still has much more seamless playback from one file to another.

Is there any way to more effectively preload an AVPlayer, or have a more timely notification when the current AVPlayerItem ends, so that there can be truly seamless playback?