calculate box around square

Question

I'd like to detect a square in an image like below:

and I'd like to highlight the square by drawing a kind of 3-dimensional box around the squares corners like you can see below in the image:

How exactly can I calculate all of the line coordinates to later on draw the "3-dimensional" box? (What is given are the 4 corner points of the black square)

Note: You can find a video of what I'd like to achieve right here https://www.youtube.com/watch?v=oSq9V2b5AZ8.

If you like to help me I would really be happy if you share some lines of code how to calculate the 4 missing points and how to know which points are matching together to draw a line from startPoint(x,y) to endPoint(x,y). Some lines in for example js would help a lot :)

I didn't try to post an answer and if you are going to have a go at people for trying to help out then I won't bother. As for your question it is way too broad for SO and therefore off topic — Pete, Jun 22 '18 at 15:15
@Pete I seriously don't think it is broad. Besides there is a viable solution posted as well !! — Jeru Luke, Jun 22 '18 at 19:04
Please note that the "square" tag is in regard to APIs for the [*Square POS software*](https://squareup.com), not the geometric figure so you should remove it. ;-) — RobG, Jun 27 '18 at 00:33

kavko · Accepted Answer · 2018-06-22T19:43:38.857

First find your contour and then select the extreme points. Then you appoint your new 3D corners and draw them with cv2.line().

Example:

import cv2
import numpy as np
import imutils

image = cv2.imread('3d2.png')

gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
gray = cv2.GaussianBlur(gray, (5, 5), 0)
thresh = cv2.threshold(gray, 190, 255, cv2.THRESH_BINARY)[1]
cv2.bitwise_not(thresh, thresh)

cnts = cv2.findContours(thresh.copy(), cv2.RETR_EXTERNAL,
        cv2.CHAIN_APPROX_SIMPLE)
cnts = cnts[0] if imutils.is_cv2() else cnts[1]
c = max(cnts, key=cv2.contourArea)

extLeft = tuple(c[c[:, :, 0].argmin()][0])
extRight = tuple(c[c[:, :, 0].argmax()][0])
extTop = tuple(c[c[:, :, 1].argmin()][0])
extBot = tuple(c[c[:, :, 1].argmax()][0])

leftx = int(extLeft[0])
lefty = int(extLeft[1]) - 90

rightx = int(extRight[0])
righty = int(extRight[1]) -90

topx = int(extTop[0])
topy = int(extTop[1]) -90

bottomx = int(extBot[0])
bottomy = int(extBot[1]) -90

leftc = (leftx, lefty)
rightc = (rightx, righty)
topc = (topx, topy)
bottomc = (bottomx, bottomy)

line = cv2.line(image, extLeft, leftc, (0,255,0), 2)
line = cv2.line(image, extRight, rightc, (0,255,0), 2)
line = cv2.line(image, extTop, topc, (0,255,0), 2)
line = cv2.line(image, extBot, bottomc, (0,255,0), 2)
line = cv2.line(image, bottomc, leftc, (0,255,0), 2)
line = cv2.line(image, rightc, topc, (0,255,0), 2)
line = cv2.line(image, leftc, topc, (0,255,0), 2)
line = cv2.line(image, rightc, topc, (0,255,0), 2)
line = cv2.line(image, bottomc, rightc, (0,255,0), 2)
cv2.drawContours(image, [c], -1, (0,255,0), 2)

cv2.imshow("Image", image)
cv2.imwrite('3Dbox1.png', image)
cv2.waitKey(0)

Result:

You can ofcorse make your new points as you wish (like if you want the same as in your picture give x+50 and y-150):

EDIT:

To make the box rotate try to use the angle which you can get from cv2.minAreaRect() function as below:

import cv2
import numpy as np
import imutils

cap = cv2.VideoCapture(0)

while True:

    try:
        ret, image = cap.read()

        gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
        gray = cv2.GaussianBlur(gray, (5, 5), 0)
        thresh = cv2.threshold(gray, 190, 255, cv2.THRESH_BINARY)[1]
        #cv2.bitwise_not(thresh, thresh)

        cnts = cv2.findContours(thresh.copy(), cv2.RETR_EXTERNAL,
                cv2.CHAIN_APPROX_SIMPLE)
        cnts = cnts[0] if imutils.is_cv2() else cnts[1]
        c = max(cnts, key=cv2.contourArea)

        rect = cv2.minAreaRect(c)
        angle = rect[2]


        extLeft = tuple(c[c[:, :, 0].argmin()][0])
        extRight = tuple(c[c[:, :, 0].argmax()][0])
        extTop = tuple(c[c[:, :, 1].argmin()][0])
        extBot = tuple(c[c[:, :, 1].argmax()][0])

        if angle < 0:

            leftx = int(extLeft[0]) - int(angle)
            lefty = int(extLeft[1]) - 50 + int(angle)

            rightx = int(extRight[0]) - int(angle)
            righty = int(extRight[1]) -50 + int(angle)

            topx = int(extTop[0]) - int(angle)
            topy = int(extTop[1]) -50 + int(angle) 

            bottomx = int(extBot[0]) - int(angle)
            bottomy = int(extBot[1]) -50 + int(angle)

            leftc = (leftx, lefty)
            rightc = (rightx, righty)
            topc = (topx, topy)
            bottomc = (bottomx, bottomy)

            line = cv2.line(image, extLeft, leftc, (0,255,0), 2)
            line = cv2.line(image, extRight, rightc, (0,255,0), 2)
            line = cv2.line(image, extTop, topc, (0,255,0), 2)
            line = cv2.line(image, extBot, bottomc, (0,255,0), 2)
            line = cv2.line(image, bottomc, leftc, (0,255,0), 2)
            line = cv2.line(image, rightc, topc, (0,255,0), 2)
            line = cv2.line(image, leftc, topc, (0,255,0), 2)
            line = cv2.line(image, rightc, topc, (0,255,0), 2)
            line = cv2.line(image, bottomc, rightc, (0,255,0), 2)
            cv2.drawContours(image, [c], -1, (0,255,0), 2)

        elif angle > 0:

            leftx = int(extLeft[0]) + int(angle)
            lefty = int(extLeft[1]) + 50 + int(angle)

            rightx = int(extRight[0]) + int(angle)
            righty = int(extRight[1]) +50 + int(angle)

            topx = int(extTop[0]) + int(angle)
            topy = int(extTop[1]) +50 + int(angle) 

            bottomx = int(extBot[0]) + int(angle)
            bottomy = int(extBot[1]) +50 + int(angle)

            leftc = (leftx, lefty)
            rightc = (rightx, righty)
            topc = (topx, topy)
            bottomc = (bottomx, bottomy)

            line = cv2.line(image, extLeft, leftc, (0,255,0), 2)
            line = cv2.line(image, extRight, rightc, (0,255,0), 2)
            line = cv2.line(image, extTop, topc, (0,255,0), 2)
            line = cv2.line(image, extBot, bottomc, (0,255,0), 2)
            line = cv2.line(image, bottomc, leftc, (0,255,0), 2)
            line = cv2.line(image, rightc, topc, (0,255,0), 2)
            line = cv2.line(image, leftc, topc, (0,255,0), 2)
            line = cv2.line(image, rightc, topc, (0,255,0), 2)
            line = cv2.line(image, bottomc, rightc, (0,255,0), 2)
            cv2.drawContours(image, [c], -1, (0,255,0), 2)

    except:
        pass

    cv2.imshow("Image", image)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

This is a nice and tiny solution but tbh it is no really what I want to achieve `:/` Your solution is not really a perspective 3dimensional cube it is a "fake" 2d cube because the value for the x / y translation should be rather calculated then just hard coded. So with a given value (like in your case: `-90`) rotation the square or applying some kind of tilt to the image will kill the right perspective. It is not really the same scenario as you can find in the video I've linked. Thanks anyways. +1 — , Jun 22 '18 at 18:48
I added a few lines...don't know if it will work because I do not have the same video but i think it will maybe work. — kavko, Jun 22 '18 at 19:22
probably have to add if statments for when the angle is negative or poitive to change addition to subtraction and vica versa... — kavko, Jun 22 '18 at 19:24
update: even better if you dont reduce the angle from Y coordinate — kavko, Jun 22 '18 at 19:44
Tbh this is not a 100% working solution and really not that what I've expected (see comment on top) but for some cases it is working quite fine. You'll get the correct mark because it pretty looks like you invested a lot of time an it would be unfair if somebody else or even nobody gets the mark. Thanks a lot. Greetings- jonas — , Jun 24 '18 at 18:58

score 0 · Answer 2 · answered Jun 22 '18 at 19:10

If your photo is an isometric view, this reduces to the following problem:

You want to find the measurements of three vectors corresponding to three orthogonal edges of the cube.

Edge1: (X1, Y1, Z1)
Edge2: (X2, Y2, Z2)
Edge3: (X3, Y3, Z3)

From your image, you're able to measure the X and Y values for two of those vectors, leaving 5 unknown values.

Since all three edges are orthogonal, you also know that their dot products are zero.

Lastly, since you're dealing with a cube, you know that each of the vectors has the same magnitude.

This gives you five equations to solve for five unknown variables, which can uniquely identify a solution.

calculate box around square

2 Answers2