How to capture a portion of AVCaptureSession?

Question

For my application, I am trying to use OCR Tesseract to convert an image into text. I have learned how to take a screen shot of the whole AVCaptureSession but I only want to capture an image of the green square to make it easier for OCR Tesseract to convert and for a cleaner user experience.

I have read up on article (below) but it only captures the view and not AVCaptureSession behind the view.

ios how to capture a particular portion of screen

Here is the code

@interface OCRScannerViewController ()
@property (strong, nonatomic) IBOutlet UIPinchGestureRecognizer *pinchGestureRecognizer;
@property (weak, nonatomic) IBOutlet UIView *cameraView;
@property (weak, nonatomic) IBOutlet VINCaptureView *captureView;
@property (weak, nonatomic) IBOutlet UIImageView *sampleImageView;



@end

@implementation OCRScannerViewController

- (void)viewDidLoad {
    [super viewDidLoad];


    //Start Session
    //Capture Session
    AVCaptureSession *session = [[AVCaptureSession alloc]init];
    session.sessionPreset = AVCaptureSessionPresetPhoto;

    //Add device
    AVCaptureDevice *device =
    [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];

    //Input
    AVCaptureDeviceInput *input = [AVCaptureDeviceInput deviceInputWithDevice:device error:nil];

    if (!input)
    {
        NSLog(@"No Input");
    }
    [session addInput:input];

    //Output
    AVCaptureVideoDataOutput *output = [[AVCaptureVideoDataOutput alloc] init];
    [session addOutput:output];
    output.videoSettings =
    @{ (NSString *)kCVPixelBufferPixelFormatTypeKey : @(kCVPixelFormatType_32BGRA) };

    //Preview Layer
    AVCaptureVideoPreviewLayer *previewLayer = [[AVCaptureVideoPreviewLayer alloc] initWithSession:session];
    previewLayer.frame = self.cameraView.bounds;
    previewLayer.videoGravity = AVLayerVideoGravityResizeAspectFill;
    [previewLayer.connection setVideoOrientation:AVCaptureVideoOrientationLandscapeLeft];

    //Place Camera View behind all subviews
    [self.view.layer insertSublayer:previewLayer atIndex:0];

    //Start capture session
    [session startRunning];

}

- (UIImage *)takeSnapshotOfView:(UIView *)view
{
    UIGraphicsBeginImageContext(CGSizeMake(view.frame.size.width, view.frame.size.height));
    [view drawViewHierarchyInRect:CGRectMake(0, 0, view.frame.size.width, view.frame.size.height) afterScreenUpdates:NO];
    UIImage *image = UIGraphicsGetImageFromCurrentImageContext();
    UIGraphicsEndImageContext();

    return image;
}

Yun CHEN · Accepted Answer · 2017-08-22T06:20:59.960

You could use CIDetector to detect the card rectangle and crop it from original image, and then crop the green rectangle from card image. Finally, use green rectangle image for OCR. An example for detecting and cropping:

CIImage *ciImage = image.CIImage;
CIDetector *detector = [CIDetector detectorOfType:CIDetectorTypeRectangle
                                          context:nil
                                          options:@{CIDetectorAccuracy:CIDetectorAccuracyHigh,
                                                    CIDetectorTracking:@YES,
                                                    CIDetectorMinFeatureSize:@.5f}];

NSArray<CIRectangleFeature *> *rectangleFeatures = (NSArray<CIRectangleFeature *> *)[detector featuresInImage:ciImage];
for (CIRectangleFeature *rect in rectangleFeatures)
{
    //find a proper rect, like card's width / height = 4:3
    //following procedure is just an example, adjust it to fit your real needs.
    CGFloat width = fabs(rect.topRight.x - rect.topLeft.x);
    CGFloat height = fabs(rect.topLeft.y - rect.bottomLeft.y);
    if ((width / height - 4 / 3) <= 0.1) {
        CIImage *cardImage = [ciImage imageByCroppingToRect:rect.bounds]; //or create a custom rect to crop if it's not good.
        CGRect greenRect = CGRectMake(0, rect.bounds.size.height * 0.8, rect.bounds.size.width, rect.bounds.size.height * 0.2); //in image coordinates
        CIImage *greenRectCIImage = [cardImage imageByCroppingToRect:greenRect];

        UIImage *greenRectImage = [[UIImage alloc] initWithCIImage:greenRectCIImage];
        //use greenRectImage for OCR

        return;
    }
}

The green box is a UIView with a clear background. Should I change it to a UIImage with a clear background? Will it still work? — Mochi, Aug 22 '17 at 17:01
To me, the green box is just indicating and meaning a rectangle area in image, so the view hierarchy doesn't matter. — Yun CHEN, Aug 23 '17 at 01:34

How to capture a portion of AVCaptureSession?

1 Answers1