2

I am trying to simulate a line follower with Pioneer 3AT in Webots. This is the first step towards my application involving swarm robotics. I have placed camera. However, I am not able to display image processed with OpenCV in the simulation display (in realtime). For now, I am planning to just threshold the lane and display using the display node as mentioned in https://www.cyberbotics.com/doc/reference/display.

However, I am not able to get this up and running. The question is how do I display the processed numpy image array in the webots display?

Code that I currently have:

camera = Camera("camera")
camera.enable(TIME_STEP);
display = Display("display")

while (robot.step(timestep) != -1):
    cameraData = camera.getImage();
    image = np.frombuffer(cameraData, np.uint8).reshape((camera.getHeight(), camera.getWidth(), 4))

Now, how do I display the image in display? I tried imageNew, imagePaste etc. However, I am getting plenty of C/C++ errors via SWIG and I am not able to find any relevant examples too. (The Webots simulator comes with only one example that uses display city.wbt and that uses a static image. I am not sure if that could be used. But, the summarised code of that example which is in C is as below.)

// speedometer
WbDeviceTag display;
int display_width = 0;
int display_height = 0;
WbImageRef speedometer_image = NULL;

// initialize display (speedometer)
if (enable_display) {
    display = wb_robot_get_device("display");
    speedometer_image = wb_display_image_load(display, "speedometer.png");
}

// display background
wb_display_image_paste(display, speedometer_image, 0, 0, false);

P.S.: I have spent more than 4 hours on this and I am not able to find any solution for this. There appears to be a similar question on Qt (I am not using Qt). But, that question has gone unanswered: Ploting an image with 'imshow' of opencv in webots. Any help will be greatly appreciated as I am planning to opensource my entire project.

EDIT 1: I tried the following as per the following note for Java. But, I am getting a blank display screen. No luck after 10 hours of efforts.

Note [Java]: The Display.imageNew function can display the image returned by the Camera.getImage function directly if the pixel format argument is set to ARGB.

display.imageNew(cameraData, display.ARGB, camera.getHeight(), camera.getWidth())

Prasad Raghavendra
  • 198
  • 1
  • 6
  • 15

2 Answers2

2

I figured out an "official way" to get this up and running. However, I do not like this solution as it is not integrated directly into the simulator (and the simulations become too slow when used with this method). For now, I am accepting this answer. However, I will accept any other answer that integrates neatly into Webots.

cv2.startWindowThread()
cv2.namedWindow("preview")

while (robot.step(timestep) != -1):
    cv2.imshow("preview", image)
    cv2.waitKey(TIME_STEP)

Source: https://cyberbotics.com/forum?message=7950

Prasad Raghavendra
  • 198
  • 1
  • 6
  • 15
1

You will find more example of Display device in Webots that might help you (they are in C/C++ but you should be able to apply similar mechanisms for Python):

In python you shoul dbe able to display an OpenCV image into the display with something similar to:

    imageRef = display.imageNew(camera.getImage().tolist(), Display.RGB)
    display.imagePaste(imageRef, 0, 0)
David Mansolino
  • 1,713
  • 7
  • 13