0

I am using Drake to implement a visual servoing scheme. For that I need to process color images and extract features.

I am using the ManipulationStation class, and I have already published the color images of the rgbd_sensor objects to LCM channels (in c++). Now I want to process the images during the simulation. I know that it would be best to process images internally (without using the ImageWriter), and for that, I cannot use the image_array_t on LCM channels or ImageRgbaU8, I have to convert images to an Eigen or OpenCV type.

I will then use feature extraction functions in these libraries. These features will be used in a control law.

Do you have any examples on how to write a system or code in c++ that could convert my Drake images to OpenCV or Eigen types? What is the best way to do it?

Thanks in advance for the help!

Arnaud

Shubham Agrawal
  • 104
  • 10
Arnaud Klipfel
  • 15
  • 2

1 Answers1

0

There's no current example converting the drake::Image to Eigen or OpenCV but it should be pretty straightforward.

What you should do is create a system akin to ImageWriter. It takes an image as an input and does whatever processing you need on the image. You connect its input port to the RgbdSensor's output port (again, like ImageWriter.) The only difference, instead of taking the image and converting it to an lcm data type, you convert it to your OpenCV or Eigen type and apply your logic to that.

If you look, ImageWriter declares a periodic publish event. Change that to (probably) a discrete update event (still periodic, I'd imagine) and then change ImageWriter::WriteImage callback into your "Image-to-OpenCV" callback.

Finally, you could also declare an output port that would make your converted image to other systems.

Sean Curtis
  • 1,425
  • 7
  • 8