Cropping a captured image exactly to how it looks in AVCaptureVideoPreviewLayer

Question

I have a photo app that is using AV Foundation. I have setup a preview layer using AVCaptureVideoPreviewLayer that takes up the top half of the screen. So when the user is trying to take their photo, all they can see is what the top half of the screen sees.

This works great, but when the user actually takes the photo and I try to set the photo as the layer's contents, the image is distorted. I did research and realized that I would need to crop the image.

All I want to do is crop the full captured image so that all that is left is exactly what the user could originally see in the top half of the screen.

I have been able to sort-of accomplish this but I am doing this by entering in manual CGRect values and it still does not look perfect. There has to be an easier way to do this.

I have literally gone through every post on stack overflow for the past 2 days about cropping images and nothing has worked.

There has to be a way to programmatically crop the captured image so that the final image will be exactly what was originally seen in the preview layer.

Here is my viewDidLoad implementation:

- (void)viewDidLoad
{
    [super viewDidLoad];

    AVCaptureSession *session =[[AVCaptureSession alloc]init];
    [session setSessionPreset:AVCaptureSessionPresetPhoto];

    AVCaptureDevice *inputDevice = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];

    NSError *error = [[NSError alloc]init];
    AVCaptureDeviceInput *deviceInput = [AVCaptureDeviceInput deviceInputWithDevice:inputDevice error:&error];

    if([session canAddInput:deviceInput])
        [session addInput:deviceInput];

    CALayer *rootLayer = [[self view]layer];
    [rootLayer setMasksToBounds:YES];

    _previewLayer = [[AVCaptureVideoPreviewLayer alloc]initWithSession:session];
    [_previewLayer setFrame:CGRectMake(0, 0, rootLayer.bounds.size.width, rootLayer.bounds.size.height/2)];
    [_previewLayer setVideoGravity:AVLayerVideoGravityResizeAspectFill];

    [rootLayer insertSublayer:_previewLayer atIndex:0];

    _stillImageOutput = [[AVCaptureStillImageOutput alloc] init];
    [session addOutput:_stillImageOutput];

    [session startRunning];
    }

And here is the code that runs when the user presses the button to capture a photo:

-(IBAction)stillImageCapture {
    AVCaptureConnection *videoConnection = nil;
    for (AVCaptureConnection *connection in _stillImageOutput.connections){
        for (AVCaptureInputPort *port in [connection inputPorts]){
            if ([[port mediaType] isEqual:AVMediaTypeVideo]){
                videoConnection = connection;
                break;
            }
        }
        if (videoConnection) {
            break;
        }
    }

    NSLog(@"about to request a capture from: %@", _stillImageOutput);

    [_stillImageOutput captureStillImageAsynchronouslyFromConnection:videoConnection completionHandler:^(CMSampleBufferRef imageDataSampleBuffer, NSError *error) {
        if(imageDataSampleBuffer) {
            NSData *imageData = [AVCaptureStillImageOutput jpegStillImageNSDataRepresentation:imageDataSampleBuffer];

            UIImage *image = [[UIImage alloc]initWithData:imageData];
            CALayer *subLayer = [CALayer layer];
            subLayer.frame = _previewLayer.frame;
            image = [self rotate:image andOrientation:image.imageOrientation];

            //Below is the crop that is sort of working for me, but as you can see I am manually entering in values and just guessing and it still does not look perfect.
            CGRect cropRect = CGRectMake(0, 650, 3000, 2000);
            CGImageRef imageRef = CGImageCreateWithImageInRect([image CGImage], cropRect);

            subLayer.contents = (id)[UIImage imageWithCGImage:imageRef].CGImage;
            subLayer.frame = _previewLayer.frame;

            [_previewLayer addSublayer:subLayer];
        }
    }];
}

Answer 1

Have a look at AVCaptureVideoPreviewLayer s

-(CGRect)metadataOutputRectOfInterestForRect:(CGRect)layerRect

This method lets you easily convert the visible CGRect of your layer to the actual camera output.

One caveat: The physical camera is not mounted "top side up", but rather rotated 90 degrees clockwise. (So if you hold your iPhone - Home Button right, the camera is actually top side up).

Keeping this in mind, you have to convert the CGRect the above method gives you, to crop the image to exactly what is on screen.

Example:

CGRect visibleLayerFrame = THE ACTUAL VISIBLE AREA IN THE LAYER FRAME
CGRect metaRect = [self.previewView.layer metadataOutputRectOfInterestForRect:visibleLayerFrame];


CGSize originalSize = [originalImage size];

if (UIInterfaceOrientationIsPortrait(_snapInterfaceOrientation)) {
    // For portrait images, swap the size of the image, because
    // here the output image is actually rotated relative to what you see on screen.

    CGFloat temp = originalSize.width;
    originalSize.width = originalSize.height;
    originalSize.height = temp;
}


// metaRect is fractional, that's why we multiply here

CGRect cropRect;

cropRect.origin.x = metaRect.origin.x * originalSize.width;
cropRect.origin.y = metaRect.origin.y * originalSize.height;
cropRect.size.width = metaRect.size.width * originalSize.width;
cropRect.size.height = metaRect.size.height * originalSize.height;

cropRect = CGRectIntegral(cropRect);

This may be a bit confusing, but what made me really understand it is this:

Hold your device "Home Button right" -> You'll see the x - axis actually lies along the "height" of your iPhone, while the y - axis lies along the "width" of your iPhone. That's why for portrait images, you have to swap the size ;)

Answer 2

@Cabus has a solution that works and you should up-vote his answer. However, I did my own version in Swift with the following:

// The image returned in initialImageData will be larger than what
//  is shown in the AVCaptureVideoPreviewLayer, so we need to crop it.
let image : UIImage = UIImage(data: initialImageData)!

let originalSize : CGSize
let visibleLayerFrame = self.previewView!.bounds // THE ACTUAL VISIBLE AREA IN THE LAYER FRAME

// Calculate the fractional size that is shown in the preview
let metaRect : CGRect = (self.videoPreviewLayer?.metadataOutputRectOfInterestForRect(visibleLayerFrame))!
if (image.imageOrientation == UIImageOrientation.Left || image.imageOrientation == UIImageOrientation.Right) {
    // For these images (which are portrait), swap the size of the
    // image, because here the output image is actually rotated
    // relative to what you see on screen.
    originalSize = CGSize(width: image.size.height, height: image.size.width)
}
else {
    originalSize = image.size
}

// metaRect is fractional, that's why we multiply here.
let cropRect : CGRect = CGRectIntegral(
        CGRect( x: metaRect.origin.x * originalSize.width,
                y: metaRect.origin.y * originalSize.height,
                width: metaRect.size.width * originalSize.width,
                height: metaRect.size.height * originalSize.height))

let finalImage : UIImage = 
    UIImage(CGImage: CGImageCreateWithImageInRect(image.CGImage, cropRect)!, 
        scale:1, 
        orientation: image.imageOrientation )

Answer 3

Here's @Erik Allen's answer in Swift 3:

let originalSize: CGSize
let visibleLayerFrame = self?.photoView.bounds

// Calculate the fractional size that is shown in the preview
let metaRect = (self?.videoPreviewLayer?.metadataOutputRectOfInterest(for: visibleLayerFrame ?? CGRect.zero)) ?? CGRect.zero

if (image.imageOrientation == UIImageOrientation.left || image.imageOrientation == UIImageOrientation.right) {
    // For these images (which are portrait), swap the size of the
    // image, because here the output image is actually rotated
    // relative to what you see on screen.
    originalSize = CGSize(width: image.size.height, height: image.size.width)
} else {
    originalSize = image.size
}

let cropRect: CGRect = CGRect(x: metaRect.origin.x * originalSize.width, y: metaRect.origin.y * originalSize.height, width: metaRect.size.width * originalSize.width, height: metaRect.size.height * originalSize.height).integral

if let finalCgImage = image.cgImage?.cropping(to: cropRect) {
    let finalImage = UIImage(cgImage: finalCgImage, scale: 1.0, orientation: image.imageOrientation)

    // User your image...
}

Answer 4

Image size has different unit with your screen size so you can't use your view's size to crop image.

ex: Captured Image is 1080 * 1920 (pixel) but your full screen is 440 * 800

And you are using videoGravity = .resizeAspectFill so your preview will be scaled by View's Width or Height. So you have to calculate which orientation fits with "captured image" and you need to calculate the other one.

ex: if "captured image" is fit with height => calculate width and versa.

In my case: I have a full screen camera, but I still need to crop when capture images. My solution:

func cropImage(image: UIImage) -> UIImage {
    DispatchQueue.main.sync {
        let originalSize: CGSize = image.size
        var cropFrame = CGRect(x: 0, y: 0, width: view.frame.width, height: view.frame.height)
        if originalSize.width / originalSize.height >= view.frame.width / view.frame.height {
            // base on originalSize.height
            let cropWidth = originalSize.height * view.frame.width / view.frame.height
            cropFrame = CGRect(x: 0, y: 0, width: cropWidth, height: originalSize.height)
        } else {
            // base on originalSize.width
            let cropHeight = originalSize.width * view.frame.height / view.frame.width
            cropFrame = CGRect(x: 0, y: 0, width: originalSize.width, height: cropHeight)
        }
        if let finalCgImage = image.cgImage?.cropping(to: cropFrame) {
            let finalImage = UIImage(cgImage: finalCgImage, scale: 1.0, orientation: image.imageOrientation)
            return finalImage
        }
        return image
    }
}