How to draw cropped bitmap using the Metal API or Accelerate Framework?

Question

I'm implementing a custom video compositor that crops video frames. Currently I use Core Graphics to do this:

-(void)renderImage:(CGImageRef)image inBuffer:(CVPixelBufferRef)destination {
    CGRect cropRect = // some rect ...
    CGImageRef currentRectImage = CGImageCreateWithImageInRect(photoFullImage, cropRect);

    size_t width = CVPixelBufferGetWidth(destination);
    size_t height = CVPixelBufferGetHeight(destination);

    CGContextRef context = CGBitmapContextCreate(CVPixelBufferGetBaseAddress(destination),       // data
                                             width,
                                             height,
                                             8,                                              // bpp
                                             CVPixelBufferGetBytesPerRow(destination),
                                             CGImageGetColorSpace(backImage),
                                             CGImageGetBitmapInfo(backImage));

    CGRect frame = CGRectMake(0, 0, width, height);
    CGContextDrawImage(context, frame, currentRectImage);
    CGContextRelease(context);
}

How can I use the Metal API to do this? It should be much faster, right? What about using the Accelerate Framework (vImage specifically)? Would that be simpler?

score 3 · Answer 1 · answered Apr 27 '15 at 08:01

Ok, I don't know wether this will be useful for you, but still. Check out the following code:

- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
  CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);

  id<MTLTexture> textureY = nil;

  {
    size_t width = CVPixelBufferGetWidth(pixelBuffer);
    size_t height = CVPixelBufferGetHeight(pixelBuffer);

    MTLPixelFormat pixelFormat = MTLPixelFormatBGRA8Unorm;

    CVMetalTextureRef texture = NULL;
    CVReturn status = CVMetalTextureCacheCreateTextureFromImage(NULL, _textureCache, pixelBuffer, NULL, pixelFormat, width, height, 0, &texture);
    if(status == kCVReturnSuccess)
    {
      textureY = CVMetalTextureGetTexture(texture);
      if (self.delegate){
        [self.delegate textureUpdated:textureY];
      }
      CFRelease(texture);
    }
  }
}

I use this code to convert CVPixelBufferRef into MTLTexture. After that you should probably create blitCommandEncoder and use it's

func copyFromTexture(sourceTexture: MTLTexture, sourceSlice: Int, sourceLevel: Int, sourceOrigin: MTLOrigin, sourceSize: MTLSize, toTexture destinationTexture: MTLTexture, destinationSlice: Int, destinationLevel: Int, destinationOrigin: MTLOrigin)

In it, you can select cropped rectangle and copy it to some other texture.

The next step is to convert generated MTLTextures back into CVPixelBufferRefs and then make a video out of that, unfortunately I don't know how to do that.

Would really like to hear what you came up with. Cheers.

You could go from CVMetalTextureRef to CGImageRef and then to CVPixelBufferRef very easily, but if you need real time performance I'm in the same pickle (trying to figure out how to get the CVPixelBufferRef) — Þorvaldur Rúnarsson, Jun 24 '16 at 13:57

score 2 · Answer 2 · answered Jun 08 '16 at 00:15

Because it uses bare pointers and unencapsulated data, vImage would "crop" things by just moving the pointer to the top left corner of the image to point to the new top left corner and reducing the height and width accordingly. You now have a vImage_Buffer that refers to a region in the middle of your image. Of course, you still need to export the content again as a file or copy it to something destined to draw to the screen. See for example vImageCreateCGImageFromBuffer().

CG can do this itself with CGImageCreateWithImageInRect()

Metal would do this either as a simple compute copy kernel, a MTLBlitCommandEncoder blit, or a 3D render application of a texture to a collection of triangles with appropriate coordinate offset.

How to draw cropped bitmap using the Metal API or Accelerate Framework?

2 Answers2