11

I am looking for an algorithm that I can use to compare two images and determine if there is something significantly different between the two. By "significant", I mean, if you took two photos of a room and a large spider was clearly on the wall in one of them, you'd be able to detect it. I am not really interested in WHAT is detected or even where - just that there is something different. The algorithm would need to ignore brightness. If the room gets brighter or darker during the day, the algorithm should ignore it.

Even if you don't know of an algorithm, any hints in the right direction would help.

Thanks!

Johann
  • 27,536
  • 39
  • 165
  • 279
  • are the photos taken from *exactly* the same viewpoint? Are you trying to make security camera software? – Will Jan 21 '13 at 08:00
  • 1
    How about shadows that change in shape over time? You cannot write them off to brightness. – Alexey Frunze Jan 21 '13 at 08:03
  • Yes, the photos are taken from the exact same location. The camera never moves. This is for a security app. If shadows are extreme that it changes the shape, then yes, that too should be detected. – Johann Jan 21 '13 at 08:05
  • You need to be top notch on your uses cases. A spider can move really slowly, and if designed incorrectly, the spider can go from one side of the screen to another while appearing invisible to your camera. – UmNyobe Jan 21 '13 at 09:27
  • Slow spiders will get caught if they appear between image captures. If they can outrun the capture rate, that's no problem. I'm not out to get superman. – Johann Jan 21 '13 at 09:33
  • @AndroidDev note that the answer your accepted is not really showing how to effectively compare two distinct images. I thought you were after that ? – mmgp Jan 23 '13 at 15:08

4 Answers4

4

I'd try to perform a high-pass filtering of your 2d-data.

According to Fourier, every signal can be transformed to "frequency space" by analyzing which frequencies are in the signal. This also applies to 2d-signals, like images.

By the means of a "high-pass-filter", you remove all low-frequency parts, like constant offsets and slow gradients. If applied to an image it can serve as a simple "edge detection" algorithm. Looking at a sample might make it easier to understand:

High-pass filtering of images

I took an image of a spider on a wall from somewhere on the web (top-left). I then decreased the brightness of this image (lower-left). For both versions, I applied a high-pass filter using GIMP (This plugin). For both input images, the output looks very similar.

My recommendation: First apply a high-pass filter, then look at differences.

Possible problems

As requested, here are some problems that I can imagine.

  • No sharp edges: if the object you want to detect doesn'T have sharp edges you probably will filter it out using HF-pass filtering. But what objects could that be? They must be huge, flat (to not produce shadows) and unstructured.

  • Only color differs, not brightness: If the object only differs in term of its color, but the brightness is the same as the background, the grayscale-conversion might be a problem. But if you run into this problem, just analyse the R, G, B-data separately, then at least one channel should help detecting the object - otherwise, you can't see it anyway.

Edit As reply to ???, if you also adjust the levels of the high-pass filtered image (which of course is all around 0.5*256) by just normalizing it to the range 0, 256 again you get

With adjusted levels

Which probably isn't worse than your result. But, HP-filters are simple and, when using FFT, very fast.

Thorsten Kranz
  • 12,492
  • 2
  • 39
  • 56
  • brilliant use of an FFT! a very nice approach! i would also bump up constrast for better seperation, and perhaps even take a photo in the middle of the day, which i then would use to normalize the brightness of future photos. calculating the cross-correlation could also aid in quantifying the amount of differences in two photos if comparing them visually is not desired. – Fredrik Jan 21 '13 at 09:18
  • I used FFT in audio processing, so I am familiar with it. Never thought of how it applies to images though. This is an interesting approach. If I understand your approach, what you're attempting to do is simply reduce all the colors down to either grey or white, where the white shows up as edges along those structures that have clearly defined edges. Do you forsee any potential problems using this solution under certain situations? – Johann Jan 21 '13 at 10:22
  • Yes, you understood me correctly. The key point is **information reduction**, only keep the information that really is relevant. I'll edit my answer to reflect some possible problems you might encounter. – Thorsten Kranz Jan 21 '13 at 10:36
  • I'm a little confused on a few things. Does the high pass filter cause a color image to convert to grey scale or do I have to first create a grey scale image and then pass that through a high pass filter? I'm not sure of what to even Google for a high pass filter for images. I know what they are for audio. Can you suggest what I kinds of high pass imaging filter algorithms I should be looking for? I need to write code, so a plugin is useless. – Johann Jan 21 '13 at 10:53
  • You said that objects to go undetected would have to be huge and flat and probably without edges. How about thieves carrying a large grey drywall positioned between them and the camera? Would that get detected? – Johann Jan 21 '13 at 11:06
  • Objects as I described would really have to be very special, I guess you won't find real-world objects that are "low-frequent" enough to come along undetected. At least their edges would be visible. On which platform are you working? Your tags don't show it to me, only your name might be a hint. – Thorsten Kranz Jan 21 '13 at 12:20
  • High-pass filtering is here usually applied to 2d-data. So you can either reduce your image to gray-scale or apply the filter to each channel separately, combining the results afterwards or analyzing them separately. – Thorsten Kranz Jan 21 '13 at 12:33
  • Fourier didn't create high-pass filters in any way, but he formulated that any waveform can be constructed by a sum of sine and cosine waves in different frequencies. Furthermore, it is not explicit which high-pass filter is being used here, but if it is something as simple as Sobel, then it is very unlikely that using FFT will provide any speed gain. FFT is useful when you have larger filters, since convolution in the frequency domain is the point-wise multiplication between two Fourier transforms. Lastly, brightness is not being ignored since this answer is working with RGB colorspace. – mmgp Jan 23 '13 at 01:02
  • "Fourier didn't create high-pass filters in any way" - who ever claimed that? I only used Fouriers idea as introduction to the concept of a high pass filter. Especially for images I've seen many colleagues being surprised that there are 'frequencies in an image'. I also never mentioned FFT, which is just one approach to filtering, sure. And finally: yes, brightness is ignored. The first thing every high-pass filter does is (simplified): subtract the average signal, so only deviations from the average remain. Thus, the "baseline" brightness is ignored. – Thorsten Kranz Jan 23 '13 at 05:09
  • @ThorstenKranz FFT is a implementation of the Fourier Transform that is Fast (thus FFT), it does not do any filtering by itself. You start mentioning high-pass filtering, then unnecessarily mentions Fourier, this to me is indicating that Fourier is concerned with your high frequencies -- it is not. A high-pass filter will filter low-frequency components, please don't add more confusions to the list. If your brightness is reduced such that an edge no longer gives a high-frequency, it will be filtered out. So how can you claim the brightness is ignored ? – mmgp Jan 23 '13 at 15:04
  • have you every really used a frequency filter? have you ever compared a filtered and an unfiltered signal? – Thorsten Kranz Jan 23 '13 at 17:42
  • @ThorstenKranz Do you want to know how it is done or do you have some actual question ? Implement any basic high-pass filter, like the one given by Sobel, and apply it yourself without relying on GIMP. – mmgp Jan 23 '13 at 22:27
  • This was no actual question but rather rhetorical, doubting your practical experience in the field of signal processing and analysis. – Thorsten Kranz Jan 24 '13 at 07:47
0

If the camera is completely static and all differences are due to ambient lighting and/or camera exposure settings, then ignoring brightness (and contrast) can be done by normalizing the 2 images.

Subtract the respective image mean (average pixel value) from all pixels of each image and the take the difference. That will take care of brightness.

If you want to handle contrast too, the calculate the variance of each each image (after bringing the mean to 0), and multiply the pixel values by the factor that will bring them both the the same variance. The difference will now be invariant to contrast as well (assuming no over/under exposure regions).

Adi Shavit
  • 16,743
  • 5
  • 67
  • 137
  • This is sort of the approach I already have attempted. However, I was taking the RGB as a single value (ex. 0xaabbcc) and normalizing the image based on subtracting these values. It turns out that the variance is very inconsistent when colors are further away from each other. I think I need to add the RGB values and use that instead. I didn't understand your last paragraph on how to take contrast into account. Can you provide a simple example of a 4x4 pixel image? Thanks! – Johann Jan 21 '13 at 10:39
0

A common approach with such a problem is to average the images taken by your camera over time, and detect any difference above a given threshold.

You need to keep an image in memory that will be the averaged image. Let's call it "avg".

Each time your camera takes a picture (called "pic"), you gonna :

  • Sum up absolute pixels value differences between "avg" and "pic".
    • If above a threshold, something is moving in front of the camera.
    • Else, modify "avg" so it will converge slightly to "pic". Up to you to find the proper formula, avg = avg * 0.95 + pic * 0.05 for instance.

Here, your reference image change over the day to adapt to sun and shadow changes.

Nicolas Repiquet
  • 9,097
  • 2
  • 31
  • 53
  • The problem with this approach is that when you start depending on averages, even a small but valid object in the picture that shouldn't be there could have little to no affect on the overall value because other parts of the image could change due to sunlight (or shadows) and counter the presence of the invalid object. Even if the sunlight only has minor affects, it does add up when averaging. I believe it is necessary to take brightness of each pixel into account or as Thorsten Kranz has suggested, remove the brightness altogether. – Johann Jan 21 '13 at 10:46
  • @AndroidDev Well, shadows will create "moving edges" and be detected as "moving objects" if you do not modify your reference image over time. To rephrase, what you need is some way to detect fast moving objects and ignore slow moving ones. – Nicolas Repiquet Jan 21 '13 at 11:03
0

How about removing the brightness component from the pixels:

Red_ratio = Red / (Red + Blue + Green)
Blue_ratio = Blue / (Red + Blue + Green)
Green_ratio = Green / (Red + Blue + Green)
Sebastien F.
  • 1,613
  • 2
  • 23
  • 40
user5792628
  • 1