How can I decide the points which I should give in cv2.getPerspectiveTransform()?

Question

I am trying to warp the following image so that I get a fronto-parallel view of the bigger wall, the one which comes on the left side of the image. However, I am not able to decide which points I should give in the function, cv2.getPerspectiveTransform() so that I get the matrix which if applied would yield the desired results.

The code I am using is:

import cv2
import numpy as np

circles = np.zeros((4,2),np.int)
counter = 0

def mousePoints(event,x,y,flags,params):
    global counter
    if event == cv2.EVENT_LBUTTONDOWN:

        circles[counter] = x,y
        counter = counter + 1
        print(circles)

img = cv2.imread("DSC_0273.JPG")

img = cv2.resize(img,(1500,1000))
q = 0
while True:

    if counter == 4:
        q = q+1

        height1,width1 = 1080,1920
        pts1 = np.float32([circles[0],circles[1],circles[2],circles[3]])
        width = np.sqrt((circles[1][0] - circles[0][0])**2 + (circles[1][1] - circles[0][1])**2)
        height =  np.sqrt((circles[2][1] - circles[0][1])**2 + (circles[2][0] - circles[0][0])**2)
        width = int(np.round(width))
        height = int(np.round(height))
        x1,y1 = circles[0]
    
        pts2 = np.float32([[x1,y1],[(x1+width),y1],[(x1+width),(y1+height)],[x1,(y1+height)]])
        matrix = cv2.getPerspectiveTransform(pts1,pts2)

        if q == 1:
            print(matrix.shape)
            print(matrix)
        imgOutput = cv2.warpPerspective(img,matrix,(width1,height1))
        cv2.imshow("Output Image ", imgOutput)


for x in range (0,4):
    cv2.circle(img,(circles[x][0],circles[x][1]),3,(0,255,0),cv2.FILLED)

cv2.imshow("Original Image ", img)
cv2.setMouseCallback("Original Image ", mousePoints)
cv2.waitKey(1)

So basically, I click 4 points and my code will find the warping matrix so that the area enclosed within those 4 points are mapped to a rectangle, such that the first point I give is mapped to the same pixel location, and the other 4 points are adjusted so that the enclosed becomes a rectangle. In order to extrapolate this, I apply the same matrix over the whole image. A set of points(the 4 points to be given through mouse clicks) I tried out is: [[349, 445], [396, 415],[388, 596], [338, 610]] The result I got is:

Answer 1

I can't get your code to work which is frankly a real pain when code in question isn't a minimal reproducible example, I only persisted wbecause I was interested in the problem - I don't understand how your code is supposed to collect the four mouse clicks then process the perspective but then your indenting is garbage.

One problem with having to click points every run is that it's difficult to get comparable runs - so I collected clicks once and used those clicks to check things out.

One problem I ran into is the sequence of clicks. AFAICT the correct sequence to get a working transform is top-left, top-right, bottom-left, bottom-right. Confusingly the sequence of points you listed in your question isn't in that order. I had to swap the last two values in pts2 to correspond to the correct sequence. Maybe both swaps compensate for each other, I ran out of energy trying to second-guess your non-working code.

Your points are relatively close together - I think you'll get better results with more widely spread points. Note "better" not "perfect" - I don't think there is a "perfect" in what you're trying to do because the perspective distortion is so great and WarpPerspective can't magically reproject for different depths in the image.

It's also difficult to compare the output with the input because the locations of the clicks aren't visible on input or output.

The other thing I'm not sure about is the resizing the source image of the temple I downloaded if 1500 wide by 1000 high. Ignoring this.

Having said all that, I think what you're getting is pretty much what you should be getting. Yes it's very distorted, but warp perspective is a simple 2D operation which doesn't claim to do reprojection allowing for lens distortion.

Here's my simplified version of your code which uses four fixed points - I've overlaid them on the source image before transformation so they're visible in the output image - you could also overlay grid lines on the input image and see what these look like in the output image. Yes the output does contain four green dots, the righthand two are quite tiny due to the transformation. You can check that for example the tips of the arches are pretty much aligned - obviously the 3-d content within them looks weird, but as I said there's no magic wand.

import cv2
import numpy as np

circles = np.zeros((4,2),np.int)

circles = [(349, 473), (903, 158),(336, 713), (918, 758) ]

img = cv2.imread("temple.JPG")

for x in range (0,4):
    cv2.circle(img,(circles[x][0],circles[x][1]),3,(0,255,0),cv2.FILLED)

cv2.imshow("Original Image ", img)

cv2.waitKey(-1)
    
height1,width1 = 1080,1920

pts1 = np.float32([circles[0],circles[1],circles[2],circles[3]])
width = np.sqrt((circles[1][0] - circles[0][0])**2 + (circles[1][1] - circles[0][1])**2)
height =  np.sqrt((circles[2][1] - circles[0][1])**2 + (circles[2][0] - circles[0][0])**2)
width = int(np.round(width))
height = int(np.round(height))
x1,y1 = circles[0]

print( f"{x1=} {y1=} {width=} {height=}")

# NOTE the third and fourth values are swapped from the original code
pts2 = np.float32([[x1,y1],[(x1+width),y1],[x1,(y1+height)],[(x1+width),(y1+height)]])
matrix = cv2.getPerspectiveTransform(pts1,pts2)

print(matrix.shape)
print(matrix)

imgOutput = cv2.warpPerspective(img,matrix,(width1,height1))
cv2.imshow("Output Image ", imgOutput)

cv2.waitKey(-1)

Input image with input points overlaid:

Cropped part of output image: