CIConvolution chain leads to integer overflow

Question

I've been doing some work with Core Image's convolution filters and I've noticed that sufficiently long chains of convolutions lead to unexpected outputs that I suspect are the result of numerical overflow on the underlying integer, float, or half float type being used to hold the pixel data. This is especially unexpected because the documentation says that every convolution's output value is "clamped to the range between 0.0 and 1.0", so ever larger values should not accumulate over successive passes of the filter but that's exactly what seems to be happening.

I've got some sample code here that demonstrates this surprise behavior. You should be able to paste it as is into just about any Xcode project, set a breakpoint at the end of it, run it on the appropriate platform (I'm using an iPhone Xs, not a simulator), and then when the break occurs use Quick Looks to inspect the filter chain.

import CoreImage
import CoreImage.CIFilterBuiltins


// --------------------
// CREATE A WHITE IMAGE
// --------------------

// the desired size of the image
let size = CGSize(width: 300, height: 300)

// create a pixel buffer to use as input; every pixel is bgra(0,0,0,0) by default
var pixelBufferOut: CVPixelBuffer?
CVPixelBufferCreate(kCFAllocatorDefault, Int(size.width), Int(size.height), kCVPixelFormatType_32BGRA, nil, &pixelBufferOut)
let input = pixelBufferOut!

// create an image from the input
let image = CIImage(cvImageBuffer: input)

// create a color matrix filter that will turn every pixel white
// bgra(0,0,0,0) becomes bgra(1,1,1,1)
let matrixFilter = CIFilter.colorMatrix()
matrixFilter.biasVector = CIVector(string: "1 1 1 1")

// turn the image white
matrixFilter.inputImage = image
let whiteImage = matrixFilter.outputImage!

// the matrix filter sets the image's extent to infinity
// crop it back to original size so Quick Looks can display the image
let cropped = whiteImage.cropped(to: CGRect(origin: .zero, size: size))


// ------------------------------
// CONVOLVE THE IMAGE SEVEN TIMES
// ------------------------------

// create a 3x3 convolution filter with every weight set to 1
let convolutionFilter = CIFilter.convolution3X3()
convolutionFilter.weights = CIVector(string: "1 1 1 1 1 1 1 1 1")

// 1
convolutionFilter.inputImage = cropped
let convolved = convolutionFilter.outputImage!

// 2
convolutionFilter.inputImage = convolved
let convolvedx2 = convolutionFilter.outputImage!

// 3
convolutionFilter.inputImage = convolvedx2
let convolvedx3 = convolutionFilter.outputImage!

// 4
convolutionFilter.inputImage = convolvedx3
let convolvedx4 = convolutionFilter.outputImage!

// 5
convolutionFilter.inputImage = convolvedx4
let convolvedx5 = convolutionFilter.outputImage!

// 6
convolutionFilter.inputImage = convolvedx5
let convolvedx6 = convolutionFilter.outputImage!

// 7
convolutionFilter.inputImage = convolvedx6
let convolvedx7 = convolutionFilter.outputImage!

// <-- put a breakpoint here
// when you run the code you can hover over the variables
// to see what the image looks like at various stages through
// the filter chain; you will find that the image is still white
// up until the seventh convolution, at which point it turns black

Further evidence that this is an overflow issue is that if I use a CIContext to render the image to an output pixel buffer, I have the opportunity to set the actual numerical type used during the render via the CIContextOption.workingFormat option. On my platform the default value is CIFormat.RGBAh which means each color channel uses a 16 bit float. If instead I use CIFormat.RGBAf which uses full 32 bit floats this problem goes away because it takes a lot more to overflow 32 bits than it does 16.

Is my insight into what's going on here correct or am I totally off? Is the documentation about clamping wrong or is this a bug with the filters?

score 1 · Accepted Answer · answered Jul 31 '20 at 07:08

It seems the documentation is outdated. Maybe it comes from a time where Core Image used 8-bit unsigned byte texture formates by default on iOS because those are clamped between 0.0 and 1.0.

With the float-typed formates, the values aren't clamped anymore and are stored as returned by the kernel. And since you started with white (1.0) and applied 7 consecutive convolutions with unnormalized weights (1 instead of 1/9), you end up with values of 9^7 = 4,782,969 per channel, which is outside of 16-bit float range.

To avoid something like that, you should normalize your convolution weights so that they sum up to 1.0.

By the way: to create a white image of a certain size, simply do this:

let image = CIImage(color: .white).cropped(to: CGSize(width: 300, height: 300))

I actually used the above technique for creating a white image at first, but to my surprise the overflow didn't happen and the image remained white. My theory is that you have to init your CIImage with a real pixel buffer otherwise the execution of the filters is only done virtually being that there is no actual data to operate on. — bd1170, Aug 01 '20 at 07:35

CIConvolution chain leads to integer overflow

1 Answers1