opencv feature matching with empty formular template

Question

i've been trying to match a scanned formular with its empty template. The goal is to rotate and scale it to match the template.

Source (left), template (right)

Match (left), Homography warp (right)
The template does not contain any very specific logo, fixation cross or rectangular frame that would conveniently help me with feature or pattern matching. Even worse, the scanned formular can be skewed, altered and contains handwritten signatures and stamps.

My approach, after unsuccessfully testing ORB feature matching, was to concentrate on the shape of the formular (lines and column).

The pictures I provide here are obtained by reconstituting lines after a segment detection (LSD) with a certain minimum size. Most of what remains for source and template is the document layout itself.

In the following script (that should work out of the box along with pictures), I attempt to do ORB feature matching, but fail to make it work because it is concentrating on edges and not on the document layout.

import cv2  # using opencv-python v3.4
import numpy as np
from imutils import resize


# alining image using ORB descriptors, then homography warp
def align_images(im1, im2,MAX_MATCHES=5000,GOOD_MATCH_PERCENT = 0.15):

    # Detect ORB features and compute descriptors.
    orb = cv2.ORB_create(MAX_MATCHES)
    keypoints1, descriptors1 = orb.detectAndCompute(im1, None)
    keypoints2, descriptors2 = orb.detectAndCompute(im2, None)

    # Match features.
    matcher = cv2.DescriptorMatcher_create(cv2.DESCRIPTOR_MATCHER_BRUTEFORCE_HAMMING)
    matches = matcher.match(descriptors1, descriptors2, None)

    # Sort matches by score
    matches.sort(key=lambda x: x.distance, reverse=False)

    # Remove not so good matches
    numGoodMatches = int(len(matches) * GOOD_MATCH_PERCENT)
    matches = matches[:numGoodMatches]

    # Draw top matches
    imMatches = cv2.drawMatches(im1, keypoints1, im2, keypoints2, matches, None)

    # Extract location of good matches
    points1 = np.zeros((len(matches), 2), dtype=np.float32)
    points2 = np.zeros((len(matches), 2), dtype=np.float32)

    for i, match in enumerate(matches):
        points1[i, :] = keypoints1[match.queryIdx].pt
        points2[i, :] = keypoints2[match.trainIdx].pt

    # Find homography
    h, mask = cv2.findHomography(points1, points2, cv2.RANSAC)

    # Use homography
    if len(im2.shape) == 2:
        height, width = im2.shape
    else:
        height, width, channels = im2.shape
    im1Reg = cv2.warpPerspective(im1, h, (width, height))

    return im1Reg, h, imMatches

template_fn = './stack/template.jpg'
image_fn = './stack/image.jpg'

im = cv2.imread(image_fn, cv2.IMREAD_GRAYSCALE)
template = cv2.imread(template_fn, cv2.IMREAD_GRAYSCALE)

# aligh images
imReg, h, matches = align_images(template,im)

# display output
cv2.imshow('im',im)
cv2.imshow('template',template)
cv2.imshow('matches',matches)
cv2.imshow('result',imReg)
cv2.waitKey(0)
cv2.destroyAllWindows()

Is there any way to make the pattern matching algorithm work on the image on the left (source)? (another idea was to leave only lines intersections)

Alternatively, I have been trying to do scale and rotation invariant pattern matching for loops and while keeping max correlation, but it is way too resource consuming and not very reliable.

I'm therefore looking for hints in the right direction using opencv.

Answer 1

SOLUTION

The issue was about reducing the image to what really matters: the layout.
Also, ORB was not appropriate since it is not as robust (rotation and size invariant) as SIFT and AKAZE are.

I proceeded as follows:

• convert the images to black and white
• use line segment detection and filter lines shorter than 1/60th of the width
• reconstruct the image from segments (line width does not have a big impact)
• (optional: resize the pictures to speed up the rest)
• apply a Gaussian transformation on the line reconstruction, 1/25th of the width
• detect and match features using SIFT (patented) or AKAZE (free) algorithm
• find a homography and warp the source picture to match the template

Matches for AKAZE

Matches for SIFT

I noted:

• the layout of the template has to match, otherwise it will only stick to what it recognizes
• line detection is better with higher resolution, then downsizing is possible since Gaussian are applied
• SIFT produces more features and seems more reliable than AKAZE