vImage on the Mac: unable to convert from Core Video

Question

Edit: Fixed. A working sample is at https://github.com/halmueller/vImage-mac-sample.

I'm trying to read the feed of a MacBook Pro's Facetime camera to process it with the vImage framework. I'm following the example in Apple's VideoCaptureSample, which is written for iOS.

I'm getting hung up on creating the vImageConverter, which creates an image buffer that vImage can use. My call to vImageConverter_CreateForCVToCGImageFormat() fails, with the console error "insufficient information in srcCVFormat to decode image. vImageCVImageFormatError = -21601".

The same call works on iOS. But the image formats are different on iOS and macOS. On iOS, the vImageConverter constructor is able to infer the format information, but on macOS, it can't.

Here's my setup code:

func displayEqualizedPixelBuffer(pixelBuffer: CVPixelBuffer) {
    var error = kvImageNoError

    if converter == nil {
        let cvImageFormat = vImageCVImageFormat_CreateWithCVPixelBuffer(pixelBuffer).takeRetainedValue()
        let deviceRGBSpace = CGColorSpaceCreateDeviceRGB()
        let dcip3SolorSpace = CGColorSpace(name: CGColorSpace.dcip3)
        vImageCVImageFormat_SetColorSpace(cvImageFormat,
                                          deviceRGBSpace)
        print(cvImageFormat)

        if let unmanagedConverter = vImageConverter_CreateForCVToCGImageFormat(
            cvImageFormat,
            &cgImageFormat,
            nil,
            vImage_Flags(kvImagePrintDiagnosticsToConsole),
            &error) {

            guard error == kvImageNoError else {
                return
            }

            converter = unmanagedConverter.takeRetainedValue()
        } else {
            return
        }
    }

When I run on iOS, I see in the console:

vImageCVFormatRef 0x101e12210:
type: '420f'
matrix: 
    0.29899999499321 0.58700001239777 0.11400000005960
    -0.16873589158058 -0.33126410841942 0.50000000000000
    0.50000000000000 -0.41868758201599 -0.08131241053343
chroma location: <RGB Base colorspace missing>
RGB base colorspace: =Bo

On macOS, though, the call to vImageConverter_CreateForCVToCGImageFormat returns nil, and I see:

vImageCVFormatRef 0x10133a270:
type: '2vuy'
matrix: 
    0.29899999499321 0.58700001239777 0.11400000005960
    -0.16873589158058 -0.33126410841942 0.50000000000000
    0.50000000000000 -0.41868758201599 -0.08131241053343
chroma location: <RGB Base colorspace missing>
RGB base colorspace: Ð ü

2018-03-13... kvImagePrintDiagnosticsToConsole: vImageConverter_CreateForCVToCGImageFormat error: 
insufficient information in srcCVFormat to decode image. vImageCVImageFormatError = -21601

Note that the image type (4 letter code) is different, as is the RGB base colorspace. I've tried on the Mac using dcip3ColorSpace instead of deviceRGB, and the results are the same.

What am I missing to get this vImageConverter created?

score 3 · Accepted Answer · answered Mar 14 '18 at 10:36

The -21601 error code means that the source CV format is missing chroma siting information (see http://dougkerr.net/Pumpkin/articles/Subsampling.pdf for a nice background of chroma siting). You can fix this by explicitly setting it with vImageCVImageFormat_SetChromaSiting. So, immediately after setting the format's color space, and before creating the converter (i.e. where you have print(cvImageFormat)), add the following:

        vImageCVImageFormat_SetChromaSiting(cvImageFormat,
                                            kCVImageBufferChromaLocation_Center)

Cheers!

simon

That did it! And the chroma siting article was some serious nerd sniping :) — Hal Mueller, Mar 14 '18 at 22:03

score 0 · Answer 2 · answered Dec 11 '18 at 21:43

So, while the answer about calling setting the chorma property on the vImage format does work, there is a better way to do that. Just set the property on the core video pixel buffer and then when you call vImageCVImageFormat_CreateWithCVPixelBuffer() it will just work, like so:

NSDictionary *pbAttachments = @{
  (__bridge NSString*)kCVImageBufferChromaLocationTopFieldKey:(__bridge NSString*)kCVImageBufferChromaLocation_Center,
  (__bridge NSString*)kCVImageBufferAlphaChannelIsOpaque: (id)kCFBooleanTrue,
};

CVBufferRef pixelBuffer = cvPixelBuffer;
CVBufferSetAttachments(pixelBuffer, (__bridge CFDictionaryRef)pbAttachments, kCVAttachmentMode_ShouldPropagate);

For extra points, you can also set the colorspace ref with the kCVImageBufferICCProfileKey and the CGColorSpaceCopyICCData() API.

vImage on the Mac: unable to convert from Core Video

2 Answers2