IOS - Speedup converting CMSampleBufferRef to dlib image

Question

I used the below code to convert iOS CMSampleBufferRef to dlib::array2d:

- (void)inputImageFromSampleBuffer:(CMSampleBufferRef)sampleBuffer
{
    CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
    CVPixelBufferLockBaseAddress(imageBuffer, kCVPixelBufferLock_ReadOnly);

    size_t width = CVPixelBufferGetWidth(imageBuffer);
    size_t height = CVPixelBufferGetHeight(imageBuffer);

    char *baseBuffer = (char *)CVPixelBufferGetBaseAddress(imageBuffer);

    if (inputImage.nr() != height) {
        inputImage.set_size(height, width);
    }

    inputImage.reset();
    long position = 0;
    while (inputImage.move_next()) {
        dlib::bgr_pixel& pixel = inputImage.element();

        long bufferLocation = position * 4;
        char r = baseBuffer[bufferLocation];
        char g = baseBuffer[bufferLocation + 1];
        char b = baseBuffer[bufferLocation + 2];

        dlib::bgr_pixel newpixel(b, g, r);
        pixel = newpixel;

        position++;
    }

    CVPixelBufferUnlockBaseAddress(imageBuffer, kCVPixelBufferLock_ReadOnly);
}

I did this for every frame from delegate - (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection. The time processing of this task was about 10ms (run on Iphone 5S) that is too slow for my need. Have any way to speed up this task?