Template matching - how to ignore pixels

Question

I'm trying to find a digit within an image. To test my code I took an image of the digit and then used AForge's Exhaustive Template Matching algorithm to search for it in another image. But I think there is a problem in that the digit is obviously not rectangular whereas the image that contains it is. That means that there are a lot of pixels participating in the comparison which shouldn't be. Is there any way to make this comparison while ignoring those pixels? If not in AForge then maybe EMGU/OpenCV or Octave?

Here's my code:

Grayscale gray = new GrayscaleRMY();
Bitmap template = (Bitmap)Bitmap.FromFile(@"5template.png");
template = gray.Apply(template);
Bitmap image = (Bitmap)Bitmap.FromFile(filePath);
Bitmap sourceImage = gray.Apply(image);
ExhaustiveTemplateMatching tm = new ExhaustiveTemplateMatching(0.7f);
TemplateMatch[] matchings = tm.ProcessImage(sourceImage, template);

Here's an example image.

If the background is affecting the matching, the first step might be to pre-process both images to remove the background. — Roger Rowland, May 06 '13 at 14:07
How would I do that? A Bitmap object is always rectangular and pixels cannot be set to null AFAIK. — Johnny, May 06 '13 at 14:19
I mean if you can decide if a pixel is foreground or background, set all background pixels in both images to the same colour (e.g. white or black). — Roger Rowland, May 06 '13 at 15:11
Ah, OK. That'll be difficult. I was wondering if there was a way to disregard the superfluous pixels. — Johnny, May 06 '13 at 15:56