How to make YoloV5 detect by screen grab?

Question

I am using the following in order to grab screenshots from an open application in realtime. How can I run detect.py that only detect input from my grab screen? Thanks.

My grab screen

import cv2 as cv
import numpy as np

import numpy as np
import cv2
from mss import mss
from PIL import Image

bounding_box = {'top': 340, 'left': 800, 'width': 350, 'height': 400}

sct = mss()


while True:
    sct_img = sct.grab(bounding_box)
    scr_img = np.array(sct_img)

    #cv2.imshow('screen', scr_img) # display screen in box
    cv.imshow('Testing', scr_img)

    if (cv2.waitKey(1) & 0xFF) == ord('q'):
        cv2.destroyAllWindows()
        break

YoloV5 detect.py

For now it only detect images instead of my Realtime grab screen

# PyTorch Hub
import torch

# Model
model = torch.hub.load('ultralytics/yolov5', 'yolov5s')

# Images
dir = 'https://ultralytics.com/images/'
imgs = [dir + f for f in ('zidane.jpg', 'bus.jpg')]  # batch of images

# Inference
results = model(imgs)
results.print()  # or .show(), .save()

Expected Result:

score 0 · Accepted Answer · answered Sep 07 '21 at 16:50

0

Jut pass the screen grab to the model:

import cv2 as cv
import numpy as np

import numpy as np
import cv2
from mss import mss
from PIL import Image

# PyTorch Hub
import torch

# Model
model = torch.hub.load('ultralytics/yolov5', 'yolov5s')

bounding_box = {'top': 340, 'left': 800, 'width': 350, 'height': 400}

sct = mss()


while True:
    sct_img = sct.grab(bounding_box)
    scr_img = np.array(sct_img)

    #cv2.imshow('screen', scr_img) # display screen in box
    scr_img = model(scr_img)
    cv.imshow('Testing', scr_img)

    if (cv2.waitKey(1) & 0xFF) == ord('q'):
        cv2.destroyAllWindows()
        break

answered Sep 07 '21 at 16:50

Agustin Rodriguez Cantalapiedr

127
5

Thanks for ur respond, i tried but the code return me `line 26, in cv.imshow('Testing', scr_img) cv2.error: OpenCV(4.5.3) :-1: error: (-5:Bad argument) in function 'imshow' > Overload resolution failed: > - mat is not a numpy array, neither a scalar > - Expected Ptr for argument 'mat' > - Expected Ptr for argument 'mat'` – terry5546 Sep 07 '21 at 16:54
oh, try with `scr_img.show()` maybe the model is returning an object, not just the postporoces picture. – Agustin Rodriguez Cantalapiedr Sep 07 '21 at 17:04
yes, when I use `scr_img.show()`, it returning me as image. Does it mean i need to save the image everytime and read from it ? – terry5546 Sep 07 '21 at 17:08
Not necessarily, if you can access the image some other way you can use cv2.imshow(), something similar to this `img = scr_img.show()`. If this doesn't work try to find the source from the `.show()` method and see what variable are they printing. – Agustin Rodriguez Cantalapiedr Sep 07 '21 at 17:13
The image is still showing up because of `.show()`. I tried `img = scr_img.show() cv.imshow('Testing', img)` – terry5546 Sep 07 '21 at 17:15
I think i found it. Try `cv.imshow('Testing', scr_img.imgs)` . Here is the "result" class, there are many methods line 348. [link](https://github.com/ultralytics/yolov5/blob/2317f86ca4ee140ed6a50fc0cc9857383f755ecd/models/common.py#L399) – Agustin Rodriguez Cantalapiedr Sep 07 '21 at 17:27

How to make YoloV5 detect by screen grab?

1 Answers1