OpenCV / Python - constantly 'read' / 'look at' video game window

Question

I have a video game bot I am working on that looks at the screen of a video game and detects objects within that window.

My current solution is to take a screenshot of that window every x seconds, detect the objects in that screenshot, and take actions accordingly.

I know that open-CV works with web camera inputs and I was wondering if there anything like that I can do for a video game screen?

Please note that this is for purely educational purposes only. I am not going to use this bot for anything more than automation in a single player game - But I don't want to read program memory as I am trying to learn about image classification.

Thanks

You could ask your OS directly. E.g. for Windows, [Fastest way to take a screenshot with python on windows](https://stackoverflow.com/questions/3586046/fastest-way-to-take-a-screenshot-with-python-on-windows/3586280#3586280) has some ideas. Of course, you can then transform the image (if necessary) into an OpenCV-compatible format and continue to process it there. — Amadan, Jun 09 '22 at 05:56
question should be flagged as a duplicate. you linked to an answer from 2010. [this](https://stackoverflow.com/a/72211337/2602877) is much more up-to-date, and compares available python libraries... which use modern APIs that weren't available in 2010. — Christoph Rackwitz, Jul 15 '22 at 13:05

Mike B · Answer 1 · 2022-07-15T13:28:09.913

ImageGrab is slow (as stated here Fastest way to take a screenshot with python on windows) but it suits your needs if you are only taking images every X seconds.

Periodicity based on frames

import numpy as np
import cv2
from PIL import ImageGrab

# do something every 30 frames
n, read= 0, 30
while True:
    # (x, y, w, h), slect portion of the screen to screenshot
    img = ImageGrab.grab(bbox=(0, 1000, 100, 1100)) 
    img_np = np.array(img)
    frame = cv2.cvtColor(img_np, cv2.COLOR_BGR2GRAY)
    if n % read == 0:
        cv2.imshow("frame", frame)
    if cv2.waitKey(1) & 0Xff == ord('q'):
        break
    n+=1
    
cv2.destroyAllWindows()

Periodicity based on time

import numpy as np
import cv2
from PIL import ImageGrab
from time import sleep

# do something every second
while True:
    # (x, y, w, h), slect portion of the screen to screenshot
    img = ImageGrab.grab(bbox=(0, 1000, 100, 1100))
    sleep(1)
    img_np = np.array(img)
    frame = cv2.cvtColor(img_np, cv2.COLOR_BGR2GRAY)
    cv2.imshow("frame", frame)
    if cv2.waitKey(1) & 0Xff == ord('q'):
        break
    
cv2.destroyAllWindows()

OpenCV / Python - constantly 'read' / 'look at' video game window

1 Answers1