30

So I watched the following 1 minute long Youtube clip from CSI New York. In the clip, using what seems to be the recording from a standard bank camera, they zoom in at least 100, and see the image of the culprit in the reflection of the eye of the girl.

Now, I thought this was completely ridiculous, so much so that I thought it was actually really funny.

However, my friend argued that there are very good tricks for image enhancement, such as "super resolution" a procedure where multiple frames of a video to produce a much higher single resolution image. He did think the show bends the truth quite a bit, but how much?

Honestly, I don't actually know anything about these things, so my question is:

How good is modern image resolution enhancement? Also, how far off are the CSI television programs?

Sklivvz
  • 78,578
  • 29
  • 321
  • 428
  • 6
    the part where they got the reflection of the basketball from is less than a *square millimeter* the eye is not that good of a reflector regardless of how good your camera is, the angle of the closeup eye is straight on instead of a bit top down. [check tvtropes on how ridiculous it can get in media](http://tvtropes.org/pmwiki/pmwiki.php/Main/EnhanceButton) (they even reference the scene you linked) – ratchet freak Nov 09 '11 at 21:51
  • I've seen IPv4 addresses shown on at least one CSI episode which included octets that were in the 700s or 800s (each octet can only be in the range of 0 through 255). It's great entertainment, so long as you don't have expertise in the fields they're demonstrating. – Randolf Richardson Nov 09 '11 at 22:42
  • 1
    on a side note the [wiki for super resolution has some examples](http://en.wikipedia.org/wiki/Super-resolution) but those use a dozen frames for double the resolution – ratchet freak Nov 09 '11 at 23:03
  • 1
    I always like the example from Blade Runner, where they not only zoom in but track around. Not just enhancing, but creating :-) – Rory Alsop Nov 10 '11 at 09:24
  • 3
    It is totally real. Check out this video for another example: http://www.youtube.com/watch?v=gF_qQYrCcns ;-) – ESultanik Nov 10 '11 at 13:00
  • @ESultanik: Hahhaha, my favorite parts: "Zoomify." And "Rotate. DeRotate." –  Nov 10 '11 at 13:08
  • 1
    Maybe more appropriate for photography.se – Sklivvz Nov 10 '11 at 13:16
  • @Sklivvz: Ok, I will crosspost then. –  Nov 10 '11 at 13:21
  • 1
    @RandolfRichardson - I loved 24 too but when they would talk about network interfacing and hacking the protocols... – Chad Nov 10 '11 at 14:24
  • @Chad: You should take a look at this 15 second clip from CSI New York. It is pure gold!! http://www.youtube.com/watch?v=hkDD03yeLnU –  Nov 10 '11 at 15:03
  • 1
    @EricNaslund I guess you just proved the saying one man's total utter BS is another mans gold! :) – Chad Nov 10 '11 at 15:13
  • @Rory: Actually there was a presentation at SIGGRAPH a few years ago about rotating arbitrary objects from just a single image - the results were pretty realistic. Of course, they couldn't do full 180-degree turns, and they obviously couldn't read text that was on the back of the object... I wish I could find a reference to it now :( – BlueRaja - Danny Pflughoeft Nov 10 '11 at 22:10
  • Cross posted to photo.se at http://photo.stackexchange.com/q/17098/21 – Rowland Shaw Nov 11 '11 at 12:29
  • Of course you can enhance: http://youtu.be/Vxq9yj2pVWk ;) – Wayne Werner May 19 '12 at 15:25
  • There is one case where image enhancements really works (unless you are very careful): Recovering blurred text. In this case, someone started with unknown black and white text, displayed in a possibly unknown, but common font, and used some algorithm to blur the text to make it unreadable. Someone trying to recover the text can take a font, display each character in the font and apply the blurring algorithm until some pixels exactly match the blurred text, then add the second character and so on. So while there is no way for the human eye to see that the original text was "Top Secret", it may – gnasher729 Dec 31 '14 at 17:11
  • I'm always skeptical of PLOS One, but here's is a [relevant article](http://www.plosone.org/article/info:doi/10.1371/journal.pone.0083325?utm_source=NSNS&utm_medium=SOC&utm_campaign=twitter&cmpid=SOC%7CNSNS%7C2014-GLOBAL-twitter) about detecting faces from reflections in eyes. – Oddthinking Jan 04 '15 at 05:34
  • many of the answers here are now outdated due to huge recent advances in computer vision with machine learning techniques – qwr Sep 19 '22 at 17:57

3 Answers3

57

Short answer: you can obtain some very good results, but only under certain conditions and absolutely not even close to what is shown in the linked video clip.

My company, Amped Software, develops image and video processing software for forensic and intelligence applications, so basically we are the real world counterpart of the CSI software.

With reference to the general problem of quality enhancement, I can tell you that for our market it is a huge problem to live up to the expectations created by TV series and Hollywood movies. You can see on our samples page that sometimes the results we are able to get are really amazing, but it is important to understand that we can obtain them only under some conditions: if there is information that is covered by disturbs, but it is there, we are able to recover it. If there is no information, we can't and we must not recreate it. In this particular application is essential not only getting the results from a visual point of view, but also following a scientific workflow that must be accepted by the court.

In 2010 I presented a research describing issues and results on almost 200 cases I've worked on and the final result was the following:

  • in more than 50% of the cases there is nothing to do (for example recovering a license plate that is 5x2 pixels is completely impossible with any software on the world);
  • in about 30% of the cases we can get some little result (for example restoring some letter of a license plate or improve the overall appearance of a face);
  • in 10% of the cases you get good results (you get most of the license plate, for example).

Please note that all these cases had severe quality issues. If their quality was good, we weren't asked to work on them.

For what regards specifically resolution enhancement:

  • when you zoom on an image you are interpolating missing pixels: from a single image you can improve visually the appearance of the image but you will not add any real detail;
  • super resolution techniques may yield good results under certain conditions: you should have enough frames, shifted by a non integer amount of pixels and preferably with few compression artifacts. In the best case you can expect good results within 2x and 3x zoom.

What is shown in the video clip can be possible only if the original video has been shoot at several megapixels and then you will have the resolution to zoom very close (more or less like you do on Google Maps). Of course, at that point there still would be other problems, like the right focus, low light condition, the fact that the perspective of the eye is different from that of the whole subject in the video, just to mention a few.

Edit 2015-01-01: you can read a more in depth explanation of this on our blog here: http://blog.ampedsoftware.com/2014/12/15/the-untold-secrets-of-forensic-video-enhancement-myth-versus-science/

Community
  • 1
martjno
  • 686
  • 5
  • 3
  • Fantastic answer. Thanks a lot for that. Now I don't have to explain, I can just point to that answer =] – Nicolas de Fontenay Dec 31 '14 at 21:46
  • Your research PDF link is broken. It redirects to Twitter `https://twitter.com/martjnofiles/an%20overview%20of%20forensic%20image%20processing%20issues%20and%20results,%20based%20on%20a%20large%20dataset%20coming%20from%20real%20cases.pdf` – Cole Tobin Jan 07 '15 at 07:24
  • Sorry about that, I just updated the answer with a new (working) link. – martjno Jan 07 '15 at 15:56
  • 1
    Haha, looking through the sample page there is some stuff there that's childplay... And then some stuff where I am like 'how the **** did you guys pull that off :O'. Nice answer! – David Mulder Jan 10 '15 at 13:10
16

There are multiple algorithms to improve image quality. Of course none of them are as spectacular, as in CSI and movies.

As already mentioned, for video multi-frame super-resolution algorithms might be applied. But for generic case it provides very limited improvement. Basically you might expect 2x improvement, but not 100x zoom as in CSI.

However, you might achieve spectacular results if some assumptions about the image being enhanced are made. For example if you know that an image fragment is a face, then specialized algorithm takes in account typical shape of human face. If you assume it's printed text, other specialized algorithm matches to particular alphabet. These algorithms use stochastic methods. Thus basically it's no longer 100% certain information, it's a guess with certain degree of probability.

Also, with these specialized algorithms most typically you don't actually get spectacularly enhanced image, but rather match against database of known faces (for facial recognition) or recognized text (for text/license plates).

TL;DR: given very low resolution image fragment it might be possible to tell if it's the suspect's face. But there is absolutely no way to do it in cool visual way they show it on CSI.

vartec
  • 26,581
  • 5
  • 97
  • 155
  • 1
    Just as a side note: resolution enhancement is better when we're talking about microscopic scale. [Super-resolution microscopy](http://en.wikipedia.org/wiki/Super_resolution_microscopy) allows one to increase resolution over the diffraction limit (that is, the physical resolution limit due to the optics of a microscope). That, of course is something that requires specifically designed machines and a special way of capturing the image, you can't apply those tecniques to a random low-res image. – nico Nov 10 '11 at 11:52
7

This is actually related to another field of my expertise, astrophotography.

There are certainly limitations in what can be digitally enhanced from blurry photographs. It's not uncommon for amateur astronomers to "over-enhance" an image they've taken such that the "sharpen" tool in their image processing software creates details that were not in the original. For example, you can sharpen an image of Saturn enough to make it appear as if you've observed the Encke gap with equipment less than the 24" Ritchie-Cretien-on-top-of-a-mountain-in-perfect-conditions that would be required to actually record the image.

There is software (Registax) that can create a clearer image by selecting the clearest frames of a movie captured by a video camera and adding them together. But the fact of the matter is that the best professional telescopes in the world were not able to get the kind of detail that the tiny telescopes of the Voyager probes got. Which is why the enormous amount of money was spent on such endeavours.

If photographs could be enhanced to anywhere near the detail that TV crime dramas suggest (and produce evidence that could be used in court), then two things would be true: blurry security camera footage would never be shown on TV to catch criminals, and the billions-of-dollars Hubble Space Telescope (and by extension, the much larger James Webb telescope in the works) would be completely unnecessary.

Ernie
  • 1,423
  • 11
  • 11