Getting RGBA values for all pixels of CGImage Swift

Question

I am trying to create a real time video processing app, in which I need to get the RGBA values of all pixels for each frame, and process them using an external library, and show them. I am trying to get the RGBA value for each pixel, but it is too slow the way I am doing it, I was wondering if there is a way to do it faster, using VImage. This is my current code, and the way I get all the pixels, as I get the current frame:

        guard let cgImage = context.makeImage() else {
        return nil
    }
    guard let data = cgImage.dataProvider?.data,
    let bytes = CFDataGetBytePtr(data) else {
    fatalError("Couldn't access image data")
    }
    assert(cgImage.colorSpace?.model == .rgb)
    let bytesPerPixel = cgImage.bitsPerPixel / cgImage.bitsPerComponent
    gp.async {
        for y in 0 ..< cgImage.height {
            for x in 0 ..< cgImage.width {
                let offset = (y * cgImage.bytesPerRow) + (x * bytesPerPixel)
                let components = (r: bytes[offset], g: bytes[offset + 1], b: bytes[offset + 2])
                print("[x:\(x), y:\(y)] \(components)")
            }
            print("---")
        }

    }

This is the version using the VImage, but I there is some memory leak, and I can not access the pixels

        guard
        let format = vImage_CGImageFormat(cgImage: cgImage),
        var buffer = try? vImage_Buffer(cgImage: cgImage,
                                        format: format) else {
            exit(-1)
        }

    let rowStride = buffer.rowBytes / MemoryLayout<Pixel_8>.stride / format.componentCount
    do {
        
        let componentCount = format.componentCount
        var argbSourcePlanarBuffers: [vImage_Buffer] = (0 ..< componentCount).map { _ in
            guard let buffer1 = try? vImage_Buffer(width: Int(buffer.width),
                                                   height: Int(buffer.height),
                                                  bitsPerPixel: format.bitsPerComponent) else {
                                                    fatalError("Error creating source buffers.")
            }
            return buffer1
        }
        vImageConvert_ARGB8888toPlanar8(&buffer,
                                        &argbSourcePlanarBuffers[0],
                                        &argbSourcePlanarBuffers[1],
                                        &argbSourcePlanarBuffers[2],
                                        &argbSourcePlanarBuffers[3],
                                        vImage_Flags(kvImageNoFlags))

        let n = rowStride * Int(argbSourcePlanarBuffers[1].height) * format.componentCount
        let start = buffer.data.assumingMemoryBound(to: Pixel_8.self)
        var ptr = UnsafeBufferPointer(start: start, count: n)

        print(Array(argbSourcePlanarBuffers)[1]) // prints the first 15 interleaved values
        buffer.free()
    }

Answer 1

You can access the underlying pixels in a vImage buffer to do this.

For example, given an image named cgImage, use the following code to populate a vImage buffer:

guard
    let format = vImage_CGImageFormat(cgImage: cgImage),
    let buffer = try? vImage_Buffer(cgImage: cgImage,
                                    format: format) else {
        exit(-1)
    }

let rowStride = buffer.rowBytes / MemoryLayout<Pixel_8>.stride / format.componentCount

Note that a vImage buffer's data may be wider than the image (see: https://developer.apple.com/documentation/accelerate/finding_the_sharpest_image_in_a_sequence_of_captured_images) which is why I've added rowStride.

To access the pixels as a single buffer of interleaved values, use:

do {
    let n = rowStride * Int(buffer.height) * format.componentCount
    let start = buffer.data.assumingMemoryBound(to: Pixel_8.self)
    let ptr = UnsafeBufferPointer(start: start, count: n)
    
    print(Array(ptr)[ 0 ... 15]) // prints the first 15 interleaved values
}

To access the pixels as a buffer of Pixel_8888 values, use (make sure that format.componentCount is 4:

do {
    let n = rowStride * Int(buffer.height)
    let start = buffer.data.assumingMemoryBound(to: Pixel_8888.self)
    let ptr = UnsafeBufferPointer(start: start, count: n)
    
    print(Array(ptr)[ 0 ... 3]) // prints the first 4 pixels
}

Answer 2

This is the slowest way to do it. A faster way is with a custom CoreImage filter.

Faster than that is to write your own OpenGL Shader (or rather, it's equivalent in Metal for current devices)

I've written OpenGL shaders, but have not worked with Metal yet.

Both allow you to write graphics code that runs directly on the GPU.