iOS: Audio and Video tap from Airplay

Question

I have made a video player that is analyzing the realtime audio and video tracks from the video that is currently playing. The videos are stored on the iOS device (in the Apps Documents directory).

This all works fine. I use MTAudioProcessingTap in order to get all the audio samples and do some FFT, and I am analyzing the video by just copy'ing the pixel buffers from the currently played CMTime (the AVPlayer currentTime property). As I said, this works fine.

But now I want to support Airplay. Just the airplay itself is not difficult, but my taps stop working as soon as Airplay is toggled and the video is playing on the ATV. Somehow, the MTAudioProcessingTap won't process and the pixelbuffers are all empty... I can't get to the data.

Is there any way to get to this data ?

In order to get the pixel buffers, I just fire an event every few milli-sec and retrieving the player's currentTime. Then:

CVPixelBufferRef imageBuffer = [videoOutput copyPixelBufferForItemTime:time itemTimeForDisplay:nil];
CVPixelBufferLockBaseAddress(imageBuffer,0);

uint8_t *tempAddress = (uint8_t *) CVPixelBufferGetBaseAddress(imageBuffer);

size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
size_t height = CVPixelBufferGetHeight(imageBuffer);

CVPixelBufferUnlockBaseAddress(imageBuffer,0);

Where tempAddress is my pixelbuffer, and videoOutput is an instance of AVPlayerItemVideoOutput.

For audio, I use:

AVMutableAudioMixInputParameters *inputParams = [AVMutableAudioMixInputParameters audioMixInputParametersWithTrack:audioTrack];

// Create a processing tap for the input parameters
MTAudioProcessingTapCallbacks callbacks;

callbacks.version = kMTAudioProcessingTapCallbacksVersion_0;
callbacks.clientInfo = (__bridge void *)(self);
callbacks.init = init;
callbacks.prepare = prepare;
callbacks.process = process;
callbacks.unprepare = unprepare;
callbacks.finalize = finalize;

MTAudioProcessingTapRef tap;
OSStatus err = MTAudioProcessingTapCreate(kCFAllocatorDefault, &callbacks,
                                          kMTAudioProcessingTapCreationFlag_PostEffects, &tap);
if (err || !tap) {
    NSLog(@"Unable to create the Audio Processing Tap");
    return;
}

inputParams.audioTapProcessor = tap;

// Create a new AVAudioMix and assign it to our AVPlayerItem
AVMutableAudioMix *audioMix = [AVMutableAudioMix audioMix];
audioMix.inputParameters = @[inputParams];
playerItem.audioMix = audioMix;

Regards, Niek

Could you please add the code where you're accessing the pixel buffers? — Dominik Hadl, Jul 23 '13 at 21:03
Hmm... I don't really know. Maybe try checking this article and his approach to getting pixel buffer - http://www.7twenty7.com/blog/2010/11/video-processing-with-av-foundation. — Dominik Hadl, Jul 24 '13 at 11:25
You didn't happen to figure out how to deallocate the MTAudioProcessingTap when you're done with it, did you? I can't get it to fire its unprepare and finalize callbacks. Please take a look at my question: http://stackoverflow.com/q/19202306/506796 — Tenfour04, Oct 11 '13 at 02:14
I just ran into this for audio-only processing. Have you filed a radar I could dupe? — Jeff V, Jun 01 '16 at 04:29

score 0 · Answer 1 · answered Mar 28 '19 at 16:09

Unfortunately, in my experience, it's not possible to get information about the audio/video during Airplay as the playback is being done on the Apple TV so the iOS device doesn't have any of the information.

I had the same issue with getting SMPTE subtitle data out of the timedMetaData, which stops getting reported during Airplay.

score -2 · Answer 2 · answered Jul 22 '13 at 13:02

Here the solution:

this is to implement AirPlay, i use this code only for Audio on my app i don't know if you can improve for the video but you can try ;)

On AppDelegate.m :

- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions {

    [RADStyle applyStyle];
    [radiosound superclass];
    [self downloadZip];

    NSError *sessionError = nil;
    [[AVAudioSession sharedInstance] setDelegate:self];
    [[AVAudioSession sharedInstance] setCategory:AVAudioSessionCategoryPlayAndRecord error:&sessionError];
    [[AVAudioSession sharedInstance] setActive:YES error:nil];

    UInt32 sessionCategory = kAudioSessionCategory_MediaPlayback;
    AudioSessionSetProperty(kAudioSessionProperty_AudioCategory, sizeof(sessionCategory), &sessionCategory);

    UInt32 audioRouteOverride = kAudioSessionOverrideAudioRoute_Speaker;
    AudioSessionSetProperty (kAudioSessionProperty_OverrideAudioRoute,sizeof (audioRouteOverride),&audioRouteOverride);

    [[UIApplication sharedApplication] beginReceivingRemoteControlEvents];
}

An if you use airplay in nice to implement the LockScreen control, ArtWork, Stop/play, Title ecc.

In the DetailViewController of you player use this code:

- (BOOL)canBecomeFirstResponder {

    return YES;
}
- (void)viewDidAppear:(BOOL)animated {

    [super viewDidAppear:animated];
    [[UIApplication sharedApplication] beginReceivingRemoteControlEvents];
    [self becomeFirstResponder];

    NSData* imageData = [[NSData alloc] initWithContentsOfURL:[NSURL URLWithString: (self.saved)[@"image"]]];

    if (imageData == nil){

    MPNowPlayingInfoCenter *infoCenter = [MPNowPlayingInfoCenter defaultCenter];
    MPMediaItemArtwork *albumArt = [[MPMediaItemArtwork alloc] initWithImage:[UIImage imageNamed:@"lockScreen.png"]];

    infoCenter.nowPlayingInfo = @{MPMediaItemPropertyTitle: saved[@"web"],MPMediaItemPropertyArtist: saved[@"title"], MPMediaItemPropertyArtwork:albumArt};

    } else {

        MPNowPlayingInfoCenter *infoCenter = [MPNowPlayingInfoCenter defaultCenter];
        MPMediaItemArtwork *albumArt = [[MPMediaItemArtwork alloc] initWithImage:[UIImage imageWithData:imageData]];

        infoCenter.nowPlayingInfo = @{MPMediaItemPropertyTitle: saved[@"link"],MPMediaItemPropertyArtist: saved[@"title"], MPMediaItemPropertyArtwork:albumArt};

    }

}

Hope this code can help you ;)

Using airplay isn't the problem, but analyzing the realtime audio & video data is. — Niek van der Steen, Jul 23 '13 at 09:49

iOS: Audio and Video tap from Airplay

2 Answers2