If wanting to resume user's background audio when muting a playing video in app, how best to do it with minimal interruptions?

Question

Apologies for the title, a little tricky to explain.

Imagine a user is listening to Spotify or something in the background while using my app, and they tap on a video to play it. The video defaults to muted, so Spotify continues. THEN the user hits unmute, so Spotify will pause, and the now unmuted video audio will start playing.

Now, turns out the video's audio wasn't so interesting and the user would prefer to listen to their music, so the user hits mute this time, so the video's audio ceases and Spotify should resume.

That last bit is the snag, when you set an app's audio session to inactive it seemingly requires the playing video to be paused. I don't want it to be paused, I'd like it to keep playing (just without audio), but it seems like a requirement for some reason.

So in response you can seemingly do a dance of A) pause video, B) set audio session to inactive thus re-enabling Spotify, C) unpause video. But if you do this dance too quickly, iOS gets annoyed and treats it as if you didn't pause it at all, potentially messing with the video's playback and outputting:

Deactivating an audio session that has running I/O. All I/O should be stopped or paused prior to deactivating the audio session

There's no completion handlers or anything though (that I can see) to facilitate when you should be re-enabling any paused video or whatnot, so I've resorted to this gross piece of code where I introduce artificial (guessed) delays:

// Pause it
player.pause()
videoControlsView?.isPaused = true

DispatchQueue.main.asyncAfter(deadline: .now() + .milliseconds(100)) {
    // Wait 0.1 seconds, set audio to inactive
    do {
        try AVAudioSession.sharedInstance().setCategory(.ambient, mode: .default, options: [])
        try AVAudioSession.sharedInstance().setActive(false, options: .notifyOthersOnDeactivation)
    } catch {
        print(error)
    }

    DispatchQueue.main.asyncAfter(deadline: .now() + .milliseconds(100)) {
        // Wait 0.1 seconds, unpause
        player.play()
        player.isMuted = true
        self.videoControlsView?.isMuted = true
    }
}

But despite testing quite a bit, this code is obviously very fragile and not my favorite I've ever written. Is there a proper way to be doing this?

Note: activating (rather than deactivating) an audio session has no such issues.