Converting Detectron2 instance segmentation to opencv Mat array

Question

I am trying to get a binary image from the instance segmentation output performed using Detectron2. According to the official documentation the mask's output format is the following:

“pred_masks”: a Tensor of shape (N, H, W), masks for each detected instance.

So i tried converting it to numpy: mask = outputs["instances"].get("pred_masks").numpy() The output was the following:

[[[False False False ... False False False]
  [False False False ... False False False]
  [False False False ... False False False]
  [False False False ... False False False]
  [False False False ... False False False]
  [False False False ... False False False]]

 [[False False False ... False False False]
  [False False False ... False False False]
  [False False False ... False False False]
  ...
  [False False False ... False False False]
  [False False False ... False False False]
  [False False False ... False False False]]

 [[False False False ... False False False]
  [False False False ... False False False]
  [False False False ... False False False]
  ...
  [False False False ... False False False]
  [False False False ... False False False]
  [False False False ... False False False]]

 ...

 [[False False False ... False False False]
  [False False False ... False False False]
  [False False False ... False False False]
  ...
  [False False False ... False False False]
  [False False False ... False False False]
  [False False False ... False False False]]

 [[False False False ... False False False]
  [False False False ... False False False]
  [False False False ... False False False]
  ...
  [False False False ... False False False]
  [False False False ... False False False]
  [False False False ... False False False]]

 [[False False False ... False False False]
  [False False False ... False False False]
  [False False False ... False False False]
  ...
  [False False False ... False False False]
  [False False False ... False False False]
  [False False False ... False False False]]]

However the data type was boolean so i added the following line to get closer to the opencv format: array = (mask > 126) * 255

[[[0 0 0 ... 0 0 0]
  [0 0 0 ... 0 0 0]
  [0 0 0 ... 0 0 0]
  ...
  [0 0 0 ... 0 0 0]
  [0 0 0 ... 0 0 0]
  [0 0 0 ... 0 0 0]]

 [[0 0 0 ... 0 0 0]
  [0 0 0 ... 0 0 0]
  [0 0 0 ... 0 0 0]
  ...
  [0 0 0 ... 0 0 0]
  [0 0 0 ... 0 0 0]
  [0 0 0 ... 0 0 0]]

 [[0 0 0 ... 0 0 0]
  [0 0 0 ... 0 0 0]
  [0 0 0 ... 0 0 0]
  ...
  [0 0 0 ... 0 0 0]
  [0 0 0 ... 0 0 0]
  [0 0 0 ... 0 0 0]]

 ...

 [[0 0 0 ... 0 0 0]
  [0 0 0 ... 0 0 0]
  [0 0 0 ... 0 0 0]
  ...
  [0 0 0 ... 0 0 0]
  [0 0 0 ... 0 0 0]
  [0 0 0 ... 0 0 0]]

 [[0 0 0 ... 0 0 0]
  [0 0 0 ... 0 0 0]
  [0 0 0 ... 0 0 0]
  ...
  [0 0 0 ... 0 0 0]
  [0 0 0 ... 0 0 0]
  [0 0 0 ... 0 0 0]]

 [[0 0 0 ... 0 0 0]
  [0 0 0 ... 0 0 0]
  [0 0 0 ... 0 0 0]
  ...
  [0 0 0 ... 0 0 0]
  [0 0 0 ... 0 0 0]
  [0 0 0 ... 0 0 0]]]

And that is as far as I went to. I would like to be able to visualize each mask individually as an opencv image: cv2.imshow("Mask", mask) , without having to save the image.

What I would like to achieve

Thank you in advance.

score 3 · Answer 1 · answered Jul 30 '21 at 07:28

mask_array = outputs['instances'].pred_masks.to("cpu").numpy()
num_instances = mask_array.shape[0]
scores = outputs['instances'].scores.to("cpu").numpy()
labels = outputs['instances'].pred_classes .to("cpu").numpy()
bbox   = outputs['instances'].pred_boxes.to("cpu").tensor.numpy()

mask_array = np.moveaxis(mask_array, 0, -1)

mask_array_instance = []
#img = np.zeros_like(im) #black
h = im.shape[0]
w = im.shape[1]
img_mask = np.zeros([h, w, 3], np.uint8)
color = (200, 100, 255)
for i in range(num_instances):
    img = np.zeros_like(im)
    mask_array_instance.append(mask_array[:, :, i:(i+1)])
    img = np.where(mask_array_instance[i] == True, 255, img)
    array_img = np.asarray(img)
    img_mask[np.where((array_img==[255,255,255]).all(axis=2))]=color
    
img_mask = np.asarray(img_mask)
output = cv2.addWeighted(im, 0.7, img_mask, 0.3, 0)

How I do to take individual masks? – Curious G. Mar 16 '22 at 12:33 — Curious G., Mar 16 '22 at 12:33

score 2 · Accepted Answer · answered Mar 17 '21 at 05:15

I hope this will give a solution for what you expecting

mask_array = outputs['instances'].pred_masks.numpy()
num_instances = mask_array.shape[0]
mask_array = np.moveaxis(mask_array, 0, -1)
mask_array_instance = []
output = np.zeros_like(im) #black
#print('output',output)
for i in range(num_instances):
    mask_array_instance.append(mask_array[:, :, i:(i+1)])
    output = np.where(mask_array_instance[i] == True, 255, output)
cv2.imwrite(mask_path+'/'+item+'.jpg',output)#mask

It is exactly what I was looking to achieve. Thank you! – Alex Mar 19 '21 at 11:47 — Alex, Mar 19 '21 at 11:47

Converting Detectron2 instance segmentation to opencv Mat array

2 Answers2