Playing and caching a remote asset with AVPlayer

Question

I am trying to use AVPlayer to play/cache a remote asset using two tools on Github: CachingPlayerItem with Cache. I found the solution elsewhere(scroll down), which nearly gets me there, My issue now is that I have to tap twice on the remote audio asset (a hyperlink in Firebase) to get it to stream. For some mysterious reason, AVPlayer will not play the remote asset unless it is cached in my case. I am aware that I can directly stream the url using AVPlayerItem(url:) but that is not the solution I am seeking; the sample code for CachingPlayerItem say that should not be necessary.

In my tinkering, I think something is happening with the async operations that are performed when I call playerItem.delegate = self. Maybe I am misunderstanding how this asynchronous delegate operation is working... Any clarity and pointers would be appreciated.

import AVKit
import Cache

class AudioPlayer: AVPlayer, ObservableObject, AVAudioPlayerDelegate {

    let diskConfig = DiskConfig(name: "DiskCache")
    let memoryConfig = MemoryConfig(expiry: .never, countLimit: 10, totalCostLimit: 10)

    lazy var storage: Cache.Storage<String, Data>? = {
        return try? Cache.Storage(diskConfig: diskConfig, memoryConfig: memoryConfig, transformer: TransformerFactory.forData())
    }()

    /// Plays audio either from the network if it's not cached or from the cache.
    func startPlayback(with url: URL) {
        let playerItem: CachingPlayerItem
        do {
            let result = try storage!.entry(forKey: url.absoluteString)
            // The video is cached.
            playerItem = CachingPlayerItem(data: result.object, mimeType: "audio/mp4", fileExtension: "m4a")
        } catch {
            // The video is not cached.
            playerItem = CachingPlayerItem(url: url)
        }

        playerItem.delegate = self // Seems to be the problematic line if the result is not cached.
        self.replaceCurrentItem(with: playerItem) // This line is different from what you do. The behaviour doesnt change whether I have AVPlayer as private var.
        self.automaticallyWaitsToMinimizeStalling = false
        self.play()
    }
}

extension AudioPlayer: CachingPlayerItemDelegate {
    func playerItem(_ playerItem: CachingPlayerItem, didFinishDownloadingData data: Data) {
        // Video is downloaded. Saving it to the cache asynchronously.
        storage?.async.setObject(data, forKey: playerItem.url.absoluteString, completion: { _ in })
        print("Caching done!")
    }
}