CGImageRef faster way to access pixel data?

Question

My current method is:

CGDataProviderRef provider = CGImageGetDataProvider(imageRef);
imageData.rawData = CGDataProviderCopyData(provider);
imageData.imageData = (UInt8 *) CFDataGetBytePtr(imageData.rawData);

I only get about 30 frames per second. I know part of the performance hit is copying the data, it'd be nice if I could just have access to the stream of bytes and not have it automatically create a copy for me.

I'm trying to get it to process CGImageRefs as fast as possible, is there a faster way?

Here's my working solutions snippet:

- (void)applicationDidFinishLaunching:(NSNotification *)aNotification {
    // Insert code here to initialize your application
    //timer = [NSTimer scheduledTimerWithTimeInterval:1.0/60.0 //2000.0
    //                                         target:self
    //                                       selector:@selector(timerLogic)
    //                                       userInfo:nil
    //                                        repeats:YES];
    leagueGameState = [LeagueGameState new];

    [self updateWindowList];
    lastTime = CACurrentMediaTime();






    // Create a capture session
    mSession = [[AVCaptureSession alloc] init];

    // Set the session preset as you wish
    mSession.sessionPreset = AVCaptureSessionPresetMedium;

    // If you're on a multi-display system and you want to capture a secondary display,
    // you can call CGGetActiveDisplayList() to get the list of all active displays.
    // For this example, we just specify the main display.
    // To capture both a main and secondary display at the same time, use two active
    // capture sessions, one for each display. On Mac OS X, AVCaptureMovieFileOutput
    // only supports writing to a single video track.
    CGDirectDisplayID displayId = kCGDirectMainDisplay;

    // Create a ScreenInput with the display and add it to the session
    AVCaptureScreenInput *input = [[AVCaptureScreenInput alloc] initWithDisplayID:displayId];
    input.minFrameDuration = CMTimeMake(1, 60);

    //if (!input) {
    //    [mSession release];
    //    mSession = nil;
    //    return;
    //}
    if ([mSession canAddInput:input]) {
        NSLog(@"Added screen capture input");
        [mSession addInput:input];
    } else {
        NSLog(@"Couldn't add screen capture input");
    }

    //**********************Add output here
    //dispatch_queue_t _videoDataOutputQueue;
    //_videoDataOutputQueue = dispatch_queue_create( "com.apple.sample.capturepipeline.video", DISPATCH_QUEUE_SERIAL );
    //dispatch_set_target_queue( _videoDataOutputQueue, dispatch_get_global_queue( DISPATCH_QUEUE_PRIORITY_HIGH, 0 ) );

    AVCaptureVideoDataOutput *videoOut = [[AVCaptureVideoDataOutput alloc] init];
    videoOut.videoSettings = @{ (id)kCVPixelBufferPixelFormatTypeKey : @(kCVPixelFormatType_32BGRA) };
    [videoOut setSampleBufferDelegate:self queue:dispatch_get_main_queue()];

    // RosyWriter records videos and we prefer not to have any dropped frames in the video recording.
    // By setting alwaysDiscardsLateVideoFrames to NO we ensure that minor fluctuations in system load or in our processing time for a given frame won't cause framedrops.
    // We do however need to ensure that on average we can process frames in realtime.
    // If we were doing preview only we would probably want to set alwaysDiscardsLateVideoFrames to YES.
    videoOut.alwaysDiscardsLateVideoFrames = YES;

    if ( [mSession canAddOutput:videoOut] ) {
        NSLog(@"Added output video");
        [mSession addOutput:videoOut];
    } else {NSLog(@"Couldn't add output video");}


    // Start running the session
    [mSession startRunning];

    NSLog(@"Set up session");
}




- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
    //NSLog(@"Captures output from sample buffer");
    //CMFormatDescriptionRef formatDescription = CMSampleBufferGetFormatDescription( sampleBuffer );
/*
        if ( self.outputVideoFormatDescription == nil ) {
            // Don't render the first sample buffer.
            // This gives us one frame interval (33ms at 30fps) for setupVideoPipelineWithInputFormatDescription: to complete.
            // Ideally this would be done asynchronously to ensure frames don't back up on slower devices.
            [self setupVideoPipelineWithInputFormatDescription:formatDescription];
        }
        else {*/
            [self renderVideoSampleBuffer:sampleBuffer];
        //}
}

- (void)renderVideoSampleBuffer:(CMSampleBufferRef)sampleBuffer
{
    //CVPixelBufferRef renderedPixelBuffer = NULL;
    //CMTime timestamp = CMSampleBufferGetPresentationTimeStamp( sampleBuffer );

    //[self calculateFramerateAtTimestamp:timestamp];

    // We must not use the GPU while running in the background.
    // setRenderingEnabled: takes the same lock so the caller can guarantee no GPU usage once the setter returns.
    //@synchronized( _renderer )
    //{
    //    if ( _renderingEnabled ) {
    CVPixelBufferRef sourcePixelBuffer = CMSampleBufferGetImageBuffer( sampleBuffer );

    const int kBytesPerPixel = 4;

    CVPixelBufferLockBaseAddress( sourcePixelBuffer, 0 );

    int bufferWidth = (int)CVPixelBufferGetWidth( sourcePixelBuffer );
    int bufferHeight = (int)CVPixelBufferGetHeight( sourcePixelBuffer );
    size_t bytesPerRow = CVPixelBufferGetBytesPerRow( sourcePixelBuffer );
    uint8_t *baseAddress = CVPixelBufferGetBaseAddress( sourcePixelBuffer );

    int count = 0;
    for ( int row = 0; row < bufferHeight; row++ )
    {
        uint8_t *pixel = baseAddress + row * bytesPerRow;
        for ( int column = 0; column < bufferWidth; column++ )
        {
            count ++;
            pixel[1] = 0; // De-green (second pixel in BGRA is green)
            pixel += kBytesPerPixel;
        }
    }

    CVPixelBufferUnlockBaseAddress( sourcePixelBuffer, 0 );


    //NSLog(@"Test Looped %d times", count);

    CIImage *ciImage = [CIImage imageWithCVImageBuffer:sourcePixelBuffer];


    /*
    CIContext *temporaryContext = [CIContext contextWithCGContext:
                                             [[NSGraphicsContext currentContext] graphicsPort]
                                                          options: nil];

    CGImageRef videoImage = [temporaryContext
                             createCGImage:ciImage
                             fromRect:CGRectMake(0, 0,
                                                 CVPixelBufferGetWidth(sourcePixelBuffer),
                                                 CVPixelBufferGetHeight(sourcePixelBuffer))];

    */

    //UIImage *uiImage = [UIImage imageWithCGImage:videoImage];

    // Create a bitmap rep from the image...
    NSBitmapImageRep *bitmapRep = [[NSBitmapImageRep alloc] initWithCIImage:ciImage];
    // Create an NSImage and add the bitmap rep to it...
    NSImage *image = [[NSImage alloc] init];
    [image addRepresentation:bitmapRep];
    // Set the output view to the new NSImage.
    [imageView setImage:image];

    //CGImageRelease(videoImage);



    //renderedPixelBuffer = [_renderer copyRenderedPixelBuffer:sourcePixelBuffer];
    //    }
    //    else {
    //        return;
    //    }
    //}

    //Profile code? See how fast it's running?
    if (CACurrentMediaTime() - lastTime > 3) //10 seconds
    {
        float time = CACurrentMediaTime() - lastTime;
        [fpsText setStringValue:[NSString stringWithFormat:@"Elapsed Time: %f ms, %f fps", time * 1000 / loopsTaken, (1000.0)/(time * 1000.0 / loopsTaken)]];
        lastTime = CACurrentMediaTime();
        loopsTaken = 0;
        [self updateWindowList];
        if (leagueGameState.leaguePID == -1) {
            [statusText setStringValue:@"No League Instance Found"];
        }
    }
    else
    {
        loopsTaken++;
    }

}

I get a very nice 60 frames per second even after looping through the data.

It captures the screen, I get the data, I modify the data and I re-show the data.

Answer 1

Which "stream of bytes" do you mean? CGImage represents the final bitmap data, but under the hood it may still be compressed. The bitmap may currently be stored on the GPU, so getting to it might require a GPU->CPU fetch (which is expensive, and should be avoided when you don't need it).

If you're trying to do this at greater than 30fps, you may want to rethink how you're attacking the problem, and use tools designed for that, like Core Image, Core Video, or Metal. Core Graphics is optimized for display, not processing (and definitely not real-time processing). A key difference in tools like Core Image is that you can perform more of your work on the GPU without shuffling data back to the CPU. This is absolutely critical for maintaining fast pipelines. Whenever possible, you want to avoid getting the actual bytes.

If you have a CGImage already, you can convert it to a CIImage with imageWithCGImage: and then use CIImage to process it further. If you really need access to the bytes, your options are the one you're using, or to render it into a bitmap context (which also will require copying) with CGContextDrawImage. There's just no promise that a CGImage has a bunch of bitmap bytes hanging around at any given time that you can look at, and it doesn't provide "lock your buffer" methods like you'll find in real-time frameworks like Core Video.

Some very good introductions to high-speed image processing from WWDC videos:

• WWDC 2013 Session 509 Core Image Effects and Techniques
• WWDC 2014 Session 514 Advances in Core Image
• WWDC 2014 Sessions 603-605 Working with Metal