why the integral-image contains extra row and column of zeros?

Question

I am learning how to use the integral-images using opencv with Java API, and i created a test that displays the grayscale image before using the integral-image and after using it. the grayscale image is 10 x 10, and when i converted it to the integral-image i found it 11 x 11 with extra rows of zeros and extra column of zeros as shown below in the output.

please let me know why the integral-image contains extra row and column of zeros?

Code:

    public static void main(String[] args) {
    MatFactory matFactory = new MatFactory();
    FilePathUtils.addInputPath(path_Obj);
    Mat bgrMat = matFactory.newMat(FilePathUtils.getInputFileFullPathList().get(0));
    Mat gsImg = SysUtils.rgbToGrayScaleMat(bgrMat);

    Log.D(TAG, "MainClas", "gsImg.dump(): " + gsImg.dump());
    Mat integralMat = new Mat();
    Imgproc.integral(gsImg, integralMat, CvType.CV_32F);

    Log.D(TAG, "MainClas", "sumMat.dump(): " + integralMat.dump());
}

OutPut:

    1: Debug: MainClass -> MainClas: gsImg.dump(): [2, 1, 7, 5, 1, 11, 2, 7, 9, 11;
        1, 2, 0, 0, 3, 20, 17, 5, 7, 8;
        4, 8, 0, 2, 6, 30, 31, 5, 2, 2;
        39, 43, 47, 44, 38, 62, 60, 37, 37, 39;
        27, 29, 52, 52, 47, 75, 67, 59, 58, 63;
        25, 21, 49, 51, 51, 78, 64, 66, 76, 80;
        40, 36, 50, 46, 41, 56, 42, 45, 47, 49;
        13, 17, 20, 15, 9, 20, 15, 19, 12, 11;
        17, 13, 8, 5, 4, 7, 13, 20, 17, 17;
        2, 4, 7, 9, 8, 6, 6, 7, 7, 8]
    2: Debug: MainClass -> MainClas: sumMat.dump(): [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0;
        0, 2, 3, 10, 15, 16, 27, 29, 36, 45, 56;
        0, 3, 6, 13, 18, 22, 53, 72, 84, 100, 119;
        0, 7, 18, 25, 32, 42, 103, 153, 170, 188, 209;
        0, 46, 100, 154, 205, 253, 376, 486, 540, 595, 655;
        0, 73, 156, 262, 365, 460, 658, 835, 948, 1061, 1184;
        0, 98, 202, 357, 511, 657, 933, 1174, 1353, 1542, 1745;
        0, 138, 278, 483, 683, 870, 1202, 1485, 1709, 1945, 2197;
        0, 151, 308, 533, 748, 944, 1296, 1594, 1837, 2085, 2348;
        0, 168, 338, 571, 791, 991, 1350, 1661, 1924, 2189, 2469;
        0, 170, 344, 584, 813, 1021, 1386, 1703, 1973, 2245, 2533]

Answer 1

That's the intended behavior. Note that the integral image sum(X,Y) in OpenCV is defined as (see documentation here) the sum of pixels in the original image with indices LESS THAN, sum_(x < X, y < Y), not less than or equal to, those of the integral image. Thus sum(0,0) for example will be the sum of zero pixels, defined to be 0. This is also why the resulting sum image has one more row and column than the original.

The reason for this is that it makes it easier to compute sums etc. over blocks of the image and handle them in a uniform way when they include the top and/or left borders.

Answer 2

There are 2 reasons.

First one is purely mathematical. Say you have a row of 3 numbers (pixels). How many possible cumulative sums it generates? the answer is 4. You can take the sum of 0 first pixels, 1 pixel, 2 pixels or all the 3 pixels. The amount of different sums is 4: (0,1,2,3). 4 is exactly 3+1. Equivalently image of width 10 will yield 11 sums for each row and size 10x10 will yield 11x11 sums.

The second reason is for programming simplicity. Integral image is used to calculate sum of any possible rectangle in the image with just 4 actions (sum of 2 corners minus the 2 other corners). The distance between the corners is exactly equals to the size of the rectangle you want to sum. For example if your rectangle has width of 5 pixels than you access the integral image at indices im[i][j] and im[i][j+5]. However if your rectangle covers the entire image width or height this may produce an index that falls out of array by 1. That is why the integral image is stored in a size that is by 1x1 larger than the image

Note: it is possible to store the integral image in an array of the same size as the image. But then the access to the array will be much slower because one will need to test the indices for out of bound. Accessing the integral image at index [-1] must be detected and produce sum of 0, and accessing at index > width will automatically return sum of the entire width.

OpenCV implemented the larger integral images mainly due to speed reasons. The calculation of the sum of rectangle requires only 4 + or - operation and 4 pointers deference. No need to test that pointers fall inside the image as long as the requested rectangle has legal coordinates inside the image

There are architectures that allow accessing array out of bounds (at illegal indices). For example GPU shaders. On those architectures integral image can be implemented in different fashion (size of NxN instead N+1xN+1 or even as pyramid of sums)

Can you manually remove the extra column from the integral image in openCV?

I strongly do not recommend doing so! openCV has a built in code to access the integral image in a specific way. If you remove the first column you will probably cause unpredictable calculations.

Moreover - as I explained, this additional row and column increase the running time by a factor of up to x10 faster (since addition and subtraction are performed by the CPU much faster than if() conditions)

Please note that integral image is a completely different representation of the original image. Not only it has a different size (N+1)x(N+1) but also a different depth. Your original image can be grayscale (store one byte per pixel) while integral image will need typically 4 bytes per pixel (because summation of many pixels require much larger numbers). So anyway integral image will take ~4 times more memory than the original image. You cannot fit integral image in the original image due to different BPP (bits per pixels) so why be bothered by different width and height