11

This algorithm has been in my mind for a long time, but I cannot find it described anywhere. It's so simple though that I can't be the only one who has thought of it. Here's how it works:

You start with an image. Say, 7x7px:

Algorithm 1

You need to resample it say, to 5x5px:

Algorithm 2

So all you do is take the average color of each new square:

Algorithm 3

This isn't the nearest-neighbor, because that takes the color of just one pixel, not fractional pixels who happen to overlay the source pixel. It's also not bilinear, bicubic, lanczos or anything else interpolating.

So - what is it? It intuitively seems to me that this should be the "mathematically perfect" resampling algorithm, although since I don't have a definition of what "mathematically perfect" is, I cannot prove or disprove that.

Last but not least, "mathematically perfect" isn't always "best looking", so I wonder how it compares to other mainstream image resampling algorithms (bicubic, lanczos) in terms of "quality"? This is a subjective term, of course, so I'm really interested if there are significant differences between this algorithm and others, which most people would agree upon.

P.S. A few things I can already tell about it - it won't be "best looking" for pixel art, as demonstrated here; there are special algorithms for that (2xSAI etc); and also it won't be best for enlarging pictures - interpolation would win out there. But for shrinking pictures...?

Update 1: Hmm, just found out about supersampling. This seems like a variant of it, with a grid-type arrangement of samples, where the number of samples is optimized for the resolution of the source & target images.

Vilx-
  • 104,512
  • 87
  • 279
  • 422
  • Averaging is a very poor low pass filter (i.e. anti-aliassing filter) so I would expect results to be inferior to most other methods, where better filters are used. – Paul R Sep 21 '12 at 08:19
  • @PaulR - even when downsampling? – Vilx- Sep 21 '12 at 08:30
  • Yes, that's the whole point - ideally you need perfect filtering before you decimate, otherwise you get aliasing and/or loss of information. Averaging is effectively a very bad LPF which is not very flat in the passband and has large lobes in the stopband. – Paul R Sep 21 '12 at 09:46
  • 2
    @PaulR - Wow... ok... I guess I'll need to go and read those books about signal & image processing after all. :P – Vilx- Sep 21 '12 at 12:34
  • 1
    You're right, you're not the first - I was using this technique 25 years ago, and I'm sure I wasn't the first either. I didn't have a name for it. It's easy to implement and optimizes very well. To see an example of how it fails vs. a better filter, resize a large image to one pixel smaller - the edges will be nice and crisp but the center will be blurred. – Mark Ransom Sep 21 '12 at 17:01

5 Answers5

3

The algorithm you have stated is called an area-averaging algorithm, it is an algorithm which is seldom applied for shrinking images.

A simpler variant of it is used as an anti-aliasing technique for smoothing rendered images in computer games.

The algorithm for this technique is called Supersampling

enter image description here

Thanks to @Guffa for pointing it out, it is a simplification of the above algorithm, as it takes sample points and it could miss out on certain colors, or choose one color more times than another even though its not most dominant.
The algorithm above is equal to an infinite points sampling of the supersampling algorithm.

Update: Just noticed that even Java appreciates your algorithm :)
AreaAveragingScaleFilter

Yarneo
  • 2,922
  • 22
  • 30
  • No, supersamling is a simplification of the algorithm that the OP is talking about. As you are taking samples at specific positions without considering the distance to the destination pixel, you get a rounding error. – Guffa Sep 21 '12 at 08:48
  • @Guffa: nor did he nor does supersampling take consideration of the "distance" to the destination pixel. Furthermore, supersampling doesn't have a destination pixel as it renders one pixel by sampling points inside of it. – Yarneo Sep 21 '12 at 08:52
  • @Guffa - supersampling _can_ be used to get precisely this effect. You just need to have right sample positions. – Vilx- Sep 21 '12 at 09:12
  • If you would take an infinite number of samples, the result would be correct. The fewer the samples, the bigger the error. An algorithm that consideres where the source pixels are can get away with very few samples if it weighs in the distance between the source pixels and destination pixels, and still get a mathematically perfect result. The destination pixel is simply what you call "one pixel". – Guffa Sep 21 '12 at 09:12
  • 1
    @Vilx-: Yes, if you take samples from a grid based on the image sizes, you can get the correct result. In your example you would need 1225 samples per pixel (35*35) to get the exactly correct result. This would quickly get out of hand... for example resampling an image from 1000x1000 to 999x999 would need 998001000000 samples per pixel. – Guffa Sep 21 '12 at 09:18
  • @Guffa - Hmm, I hadn't thought about that. :P I thought the number of samples would be more...manageable. Still, it can be optimized to perform one weighted sample per overlapping source pixel. – Vilx- Sep 21 '12 at 09:23
3

I will start out by saying I don’t know an official name for your algorithm. I know that Paint Shop Pro called it “Bilinear” early on, but were forced to rename it to “Weighted Average” in version 8 when it was pointed out that the algorithm didn’t match the classic definition of Bilinear.

Most resizing algorithms can be applied in two independent passes, one in the X direction and one in the Y. This is not only more efficient, but it makes it a lot easier to describe and reason about the different algorithms. From this point forward I’m going to work in one dimension and assume you can extrapolate to 2D.

Your input consists of 7 pixels which we will give coordinates of 0, 1, 2, 3, 4, 5, 6. It’s useful to realize that a pixel is not a little square in this context, but is just a single point. To create the output you will want the interpolated values from the points 0.2, 1.6, 3.0, 4.4, 5.8. Why not 0.0, 1.5, 3.0, 4.5, 6.0? Suppose you doubled the size of the input and output to 14x14 and 10x10: the coordinates would now be 0.0, 1.44, 2.89, 4.33, 5.78, 7.22, 8.67, 10.11, 11.56, 13.0. Starting with the second pixel the results would be different, and that’s unacceptable. All the points should be 7/5 apart, giving the coordinates 0.2, 1.6, 3.0, 4.4, 5.8, 7.2, 8.6, 10.0, 11.4, 12.8.

Let’s compare the common resizing algorithms when expressed as a filter, and see how they compare to yours.

Nearest Neighbor filter

This first example in the generic form is called a Box or Averaging filter. But a magical thing happens when the width of the box filter is exactly 1.0: one pixel from the input is going to fall within the box and be given a weight of 1.0, and all the other pixels in the input will be given the weight 0.0. This makes it the equivalent of the Nearest Neighbor algorithm.

Bilinear filter

Our second example is generically called the Tent filter. Again it becomes something special when the width is exactly 2.0, it becomes a Linear interpolation; applied in 2D it’s called Bilinear.

Bicubic filter

The third example is a Cubic filter, which when applied in 2D is called Bicubic. There are different variations of this formula, this example uses the one suggested by Mitchell and Netravali.

Gaussian filter

While the Gaussian filter isn’t often used in resizing applications, I added it here for comparison.

Weighted Average filter

Finally we reach your algorithm. It’s a combination of averaging and bilinear, a tent with a flat top.

Mark Ransom
  • 299,747
  • 42
  • 398
  • 622
  • Wish I could give you more upvotes, both for the detailed analysis and the link. It's a bit outdated, but I see the point. It will take some time to readjust my thinking, I guess. Also, it's clear that I need to brush up my math skills and read up on sampling theorem/image processing. :P – Vilx- Sep 27 '12 at 07:57
  • @Vilx- it took me many years to reach a deep understanding of this stuff, even after I stopped using it on a daily basis. Don't beat yourself up. One of the reasons I wanted to post this answer is that I've never seen the classic algorithms expressed as a filter in such a direct manner and I think it really helps when comparing them. – Mark Ransom Sep 28 '12 at 03:17
1

Contrary to what I read in other answers, this algorithm is actually quite popular for supersampling, at least in the image processing community.

It is implemented in Intel's Performance Primitives library under the name Super Sampling; the (rather uninformative) name is a way to state that there is no alternative algorithm for supersampling in the library. In OpenCV, it goes under the name INTER_AREA; it is listed amongst other interpolation types, which could suggest they are interchangeable, but mention that "it may be a preferred method for image decimation" — a rather conservative statement to my taste.

When you supersample an image by an integer factor, say by a factor of two, taking the mean of the underlying pixels for the resulting image (as done for example by scikit-image's downscale_local_mean) is indeed optimal in a specific sense.

Suppose your image is obtained by some quantitative measurement of a signal by a grid of receptors. For example, photographs or X-rays, counting the number of photons. Pixel values are proportional to the amount of signal received by a given receptor.

If you assume that your camera is perfect — no spread of the signal, 100% coverage of the receiving area — then mean supersampling is optimal because it gives the exact image that would be received by a perfect camera with half the resolution.

Area averaging is a straightforward generalization of this optimal mean supersampling to non-integer ratios, which explains its popularity, although it cannot boast the same property for any supersampling ratio other than integers.

P-Gn
  • 23,115
  • 9
  • 87
  • 104
0

Your description doesn't remind me of an algorithm most likely it's data structure and data type. It reminds me of a kd-tree or a quadtree. A kd-tree or quadtree can help you to solve your problem with nearest-neighbor. However if want a mathematical function for grids you can also take a look at space filling curves especially z order morton curves. This works great for power of 2 and reduce the 2d complexity. The function is f(x,y) = (f(x), f(y)).

Micromega
  • 12,486
  • 7
  • 35
  • 72
-2

It's also not bilinear

Yes, actually it is. Bilinear resampling is just linear interpolation in two dimensions.

You are describing it as getting the average color of the surface of the pixel, but interpolation from the surrounding pixels is just a much simpler way to calculate the same value.

The interpolation is done one dimension at a time, so you just calculate overlapping sides instead of overlapping areas, which is a lot simpler.

Guffa
  • 687,336
  • 108
  • 737
  • 1,005
  • I thought that bilinear didn't calculate how much each source pixel overlapped each destination pixel? – Vilx- Sep 21 '12 at 08:31
  • @Vilx-: It does. You calculate the distance from the source pixels to the destination pixel, and use that to weigh how much to take from each color. – Guffa Sep 21 '12 at 08:40
  • And if the destination pixel overlaps many, many source pixels? If I read Wikipedia correctly, it says that `bilinear interpolation uses only the 4 nearest pixel values which are located in diagonal directions from a given pixel in order to find the appropriate color intensity values of that pixel`. – Vilx- Sep 21 '12 at 08:44
  • @Vilx-: That is a simplified explanation for the case where the image is enlarged or not shrunk more than 50%. You simply calculate the distance to all the source pixels that the destination pixel overlaps. – Guffa Sep 21 '12 at 08:50
  • OK... between you and Yarneo, it seems that there might be several algorithms that reduce to this behavior when the right parameters are given. :P – Vilx- Sep 21 '12 at 08:53
  • Although wait... bilinear considers the distance of each pixel... so, for example, if a target pixel overlaps 5x5 source pixels, won't it be so that the source pixels closer to the center of the target pixel get greater weight than those further off, even though they are all completely inside the target pixel? – Vilx- Sep 21 '12 at 08:55
  • @Vilx-: Yes, you are right. For source pixels that are completely inside the target pixel you don't calculate the distance. They contribute fully to the value. – Guffa Sep 21 '12 at 09:07
  • 1
    @Vilx- This answer is almost but not quite right. Consider the center pixel in your example - it lines up in both images, so by the rules of Bilinear it should take 100% of its value from that center pixel and none from the surroundings. Your method incorporates a small portion of each of the surrounding pixels. – Mark Ransom Sep 21 '12 at 20:47
  • @MarkRansom: Yes, the image is a simplified diagram. The algorithm doesn't sample a single value, it calculates the average of the curve that covers the target pixel. – Guffa Sep 21 '12 at 21:01
  • 1
    @Guffa, a linear interpolation is a spot on the line between two points. It can't be more than two points, otherwise it wouldn't be linear. Your picture is quite accurate and shows exactly what I'm talking about with the center points lining up. – Mark Ransom Sep 21 '12 at 21:18
  • @MarkRansom: I removed the image, as I only seem to have confused you with it. It's only a simple interpolation when enlarging images. When reducing images you will have more than two points from the source in a destination pixel, so it's only the weight of the two outermost source pixels that are calculated using interpolation. – Guffa Sep 21 '12 at 21:36
  • @Guffa what you're describing is *not* bilinear. I suggest you look up the classic definition, your picture illustrated it brilliantly. – Mark Ransom Sep 21 '12 at 21:46
  • @MarkRansom: No, it didn't. Did you just downvote my answer because you misunderstood what bilinear interpolation is? – Guffa Sep 21 '12 at 21:56
  • @Guffa I'll admit that the downvote is mine; I felt it necessary because your answer as it stands is wrong and misleading. Paint Shop Pro called this technique Bilinear for a while and it generated quite a bit of animosity, it was changed in a later version to "Weighted Average" and a true Bilinear was added - see https://groups.google.com/forum/?fromgroups=#!topic/comp.graphics.apps.paint-shop-pro/gvVv0imZWEs%5B1-25%5D – Mark Ransom Sep 24 '12 at 15:56
  • @MarkRansom: You still haven't explained what it is that you think is wrong and misleading with the answer. Even if there was something wrong with the answer, I couldn't correct it just based on you saying that "something" is wrong with it. I'm sorry if you misunderstood the answer, but that doesn't mean that the answer is wrong. – Guffa Sep 24 '12 at 18:26
  • 1
    @Guffa, a linear interpolation is between two points, never more than 2. A bilinear interpolation is between four points, 2x2. The example provided in the question uses parts of 9 pixels to create the center pixel which is definitely not bilinear. Calling it bilinear as you did in your answer is wrong. – Mark Ransom Sep 24 '12 at 20:24
  • @MarkRansom: You are mistaken. Just because it's called an interpolation technique doesn't mean that it *only* does interpolation. You can read about it for example here: http://en.wikipedia.org/wiki/Bilinear_interpolation You will see that the nearest neightbor algorithm is also called an interpolation technique, although it doesn't do any real interpolation at all. – Guffa Sep 24 '12 at 20:30
  • @Guffa, [interpolation](http://en.wikipedia.org/wiki/Interpolation) is a pretty loose term, it just means estimating unknown values based on known values, and nearest neighbor certainly qualifies even if it's a poor example. [Linear interpolation](http://en.wikipedia.org/wiki/Linear_interpolation) has a much stricter definition. Bilinear interpolation is just linear interpolation done in two dimensions. – Mark Ransom Sep 24 '12 at 21:00
  • @MarkRansom: Again, just because the technique uses linear interpolation, doesn't mean that it uses *only* linear interpolation. Read the article that I linked to above, and see if it gets any clearer. – Guffa Sep 24 '12 at 21:07
  • 1
    @Guffa, this is where you're wrong - bilinear interpolation *does* use *only* linear interpolation. Read your own article. – Mark Ransom Sep 24 '12 at 21:18