1

I am working on a project that involves automating a video game using computer vision. My next task involves separating the game's UI elements from the actual game's field of view. For instance:

We would take a screenshot of the entire client window like so: client screenshot

Then we would locate the various UI elements on screen (this is accomplished with template matching, see: Template matching with transparent image templates using OpenCV Python).

Then, we would need to subtract the matched templates from the client screenshot to end up with something like this: client screenshot with UI elements subtracted

This allows me to perform computer vision functions on the game's field of view without the risk of the UI elements interfering.

Let's say we have the following code:

# Let's assume we have already taken a screenshot of the client window
client_window = cv2.imread('client_window.png')

# Here are the three UI areas I'd like to remove/blackout from the above image
ui_chatbox = {'left': 0, 'top': 746, 'width': 520, 'height': 167}
ui_minimap = {'left': 878, 'top': 31, 'width': 212, 'height': 207}
ui_inventory = {'left': 849, 'top': 576, 'width': 241, 'height': 337}

How can we blackout the pixels of the UI element bounding boxes from the client_window matrix?

Christoph Rackwitz
  • 11,317
  • 4
  • 27
  • 36
ktom
  • 125
  • 8

1 Answers1

1

I ended up solving this by using slicing to set all pixels in a range to black. I thought of this because I knew that cropping an image was a similar process, and both involved selecting a range of pixels:

import cv2
import numpy as np

# Let's assume we have already taken a screenshot of the client window
client_window = cv2.imread('client_window.png')
cv2.imshow('client_window', client_window)
cv2.waitKey(0)

# Here are the three UI areas I'd like to remove/blackout from the above image
ui_chatbox = {'left': 0, 'top': 746, 'width': 520, 'height': 167}
ui_minimap = {'left': 878, 'top': 31, 'width': 212, 'height': 207}
ui_inventory = {'left': 849, 'top': 576, 'width': 241, 'height': 337}

client_window[ui_chatbox['top']:ui_chatbox['top'] + ui_chatbox['height'],
                ui_chatbox['left']:ui_chatbox['left'] + ui_chatbox['width']] = np.array([0,0,0])
client_window[ui_minimap['top']:ui_minimap['top'] + ui_minimap['height'],
                ui_minimap['left']:ui_minimap['left'] + ui_minimap['width']] = np.array([0,0,0])
client_window[ui_inventory['top']:ui_inventory['top'] + ui_inventory['height'],
                ui_inventory['left']:ui_inventory['left'] + ui_inventory['width']] = np.array([0,0,0])
cv2.imshow('result', client_window)
cv2.waitKey(0)
Alexander L. Hayes
  • 3,892
  • 4
  • 13
  • 34
ktom
  • 125
  • 8
  • numpy calls those things "slices", and this uses broadcasting to assign one pixel value to the entire slice. – Christoph Rackwitz Dec 01 '22 at 00:14
  • Can you share the reasoning behing the ```[ui_chatbox['top']:ui_chatbox['top'] + ui_chatbox['height'], ui_chatbox['left']:ui_chatbox['left'] + ui_chatbox['width']]```. This should be set to 0 for blacking them out but how did you arrive at that expression – Pixel_Bear Dec 02 '22 at 10:36