display opencv cv::Mat image in gtkmm

Question

I want to display a cv::Mat in a Gui written by gtkmm. So I have done a test.

I have a widget Gtk::Image image, and I want to set the image with the following two methods:

// first method, display from file
void displayImage1()
{
    Glib::RefPtr<Gdk::Pixbuf> pixbuf = Gdk::Pixbuf::create_from_file("gtk.png");
    image.set(pixbuf);
}

// second method, display from cv::Mat
void displayImage2()
{
    cv::Mat outImage = cv::imread("gtk.png");
    cv::cvtColor(outImage, outImage, CV_BGR2RGB);
    Glib::RefPtr<Gdk::Pixbuf> pixbuf = Gdk::Pixbuf::create_from_data(outImage.data, Gdk::COLORSPACE_RGB,false, 8, outImage.cols, outImage.rows, outImage.step);
    image.set(pixbuf);
}

The first method works well.

However, the second method doesn't work well, I got a destroyed image on the screen as shown in the picture.

If I set the has_alpha parameter to true, the result is also strange (see pic. below).

Similar tests were done by using Gtk::DrawingArea. Different IDEs are used (but all g++ compiler under linux). All same results.

Update:

I tested lots of images. Sometimes the images are broken, sometimes the programs crashed with

The program has unexpectedly finished.

Have you checked that the image doesn't actually contain an alpha channel? Seems to me like it does unless the grey background is a part of the image. — KjMag, Oct 12 '17 at 09:05
@KjMag THANKS but the result with has_alpha as "true" is also not correct, I updated the result in the question. — Summer Fang, Oct 12 '17 at 09:17
@Miki also tried, then it displays nothing(no correct image is created). Thank you! — Summer Fang, Oct 12 '17 at 09:44
The conversion code seems correct. I think that this is probably due the data buffer being release as soon as `outImage` goes out of scope at the end of the function. You need a way to actually copy the data inside the gtk image — Miki, Oct 12 '17 at 12:26
@Miki Thank you I tried making the cv::Mat image as a member of my class which is initialized in the constructor and the two methods above are used as member functions. Nothing changed. — Summer Fang, Oct 12 '17 at 14:00
@Miki: `create_from_data` creates a copy of the bytes, and the copy is done before `outImage` goes out of scope... — liberforce, Oct 12 '17 at 15:44
@SummFang - I believe that the problem comes on some particularity about your image. You really got bad luck: I was stuck, I tested your code, and it does work with a Mat from cv::VideoCapture.read, on my Mac Os X with front camera. — jmgonet, Feb 20 '18 at 18:05
@liberforce `create_from_data` **does not** create a copy. AFAIU it just wraps gdk_pixbuf_new_from_data, that does not copy anything: https://gitlab.gnome.org/GNOME/gtkmm/-/blob/master/gdk/src/pixbuf.ccg#L60 — ntd, Nov 18 '21 at 23:39
@ntd: seems you're right, it creates a new GdkPixbuf object that merely references the existing memory. Thanks for the update! — liberforce, Nov 19 '21 at 11:18

score 1 · Answer 1 · answered Jun 07 '22 at 00:54

try add a reference to outimage, like outimage.addref() . all my problems were related to that. the source image being de-referenced before the gdk_pixbuf_new_from_data get a chance to map it. leading to segfaults, corruption and so on. just be sure to release it later, or use the callback provided with gdk_pixbuf_new_from_data

score 1 · Answer 2 · answered Aug 22 '22 at 22:41

For Gtkmm-2.4 and OpenCv 4.6 check https://onthim.blogspot.com/2015/10/using-opencv-in-gtk-applications.html and https://developer-old.gnome.org/gtkmm-tutorial/2.24/sec-draw-images.html.es

    Mat frame;
    frame = imread("gtk.png");    
    cv::cvtColor (frame, frame, COLOR_BGR2RGB);//COLOR_BGR2GRAY
    Glib::RefPtr<Gdk::Pixbuf> image = Gdk::Pixbuf::create_from_data(frame.data,Gdk::COLORSPACE_RGB, false, 8, frame.cols, frame.rows, frame.step);
    image->render_to_drawable(get_window(), get_style()->get_black_gc(),0, 0, 0, 0, image->get_width(), image->get_height(), // draw the whole image (from 0,0 to the full width,height) at 100,80 in the window
Gdk::RGB_DITHER_NONE, 0, 0);

liberforce · Answer 3 · 2017-10-12T13:48:10.373

0

Usually, this kind of "broken" image trigger a warning in my head: "wrong rawstride !". The rawstride in Gdk::Pixbuf in the length in bytes of a line of data. That's because you may have some byte-alignment constraints, so there may be some padding at the end of one line.

I checked what this step argument was, and yes, that's the same in OpenCV as the rawstride in Gdk::Pixbuf. Until I realized outImage.step is a cv:MatStep object, while Gdk::Pixbuf::create_from_data expects an int. I think you're supposed to use outImage.step[0] instead.

Please read https://docs.opencv.org/2.4/modules/core/doc/basic_structures.html#mat

edited Oct 12 '17 at 13:48

answered Oct 12 '17 at 13:40

liberforce

11,189
37
48

1

a `cv::MatStep` is implicitly converted to a `size_t`, so it's ok to use `outImage.step` like this. – Miki Oct 12 '17 at 13:54
Thank you for your kind answer. Yes I agree with @Miki . Nevertheless I tried with outImage.step[0], nothing changled. – Summer Fang Oct 12 '17 at 14:02

score 0 · Answer 4 · answered Oct 12 '17 at 16:11

0

here we go:

auto lenna = imread("Lenna.png");
Image image;

cvtColor(lenna, lenna, COLOR_BGR2RGB);
auto size = lenna.size();
auto img = Gdk::Pixbuf::create_from_data(lenna.data, Gdk::COLORSPACE_RGB, lenna.channels() == 4, 8, size.width, size.height, (int) lenna.step);

image.set(img);

answered Oct 12 '17 at 16:11

The Moisrex

1,857
1
14
16

Thank you for your code, but I think it is essentially the same as mine. I tried it anyhow and the result is the same. If it works on your side, it must be the problem of my compiler or system or the version of my library I don't know :( – Summer Fang Oct 13 '17 at 08:55
I think the problem is that the version of the OpenCV and GTKmm are not the same. try gtkmm 2.4 – The Moisrex Oct 13 '17 at 12:11

score 0 · Answer 5 · answered Nov 21 '19 at 23:53

This is what I did and it show good result on image showing

cvtColor(resize_image, resize_image, COLOR_BGR2RGB);
Pixbuf = Gdk::Pixbuf::create_from_data(resize_image.data, Gdk::COLORSPACE_RGB, false, 8, resize_image.cols, resize_image.rows, resize_image.step);

score 0 · Answer 6 · edited Dec 29 '20 at 11:09

So I tested this (add scale_simple) with success for me:

From: http://orobotp.blogspot.com/2014/01/opencv-with-gtkmm3.html

Version: Gtkmm 3.22.2-2, OpenCV 4.4.0-dev, g++ 7.5.0

void displayImage2()
{
    cv::Mat outImage;
    outImage = cv::imread("gtk.png");
    cv::cvtColor(outImage, outImage, cv::COLOR_RGB2BGR);
    Glib::RefPtr<Gdk::Pixbuf> pixbuf = Gdk::Pixbuf::create_from_data(outImage.data, Gdk::COLORSPACE_RGB,false, 8, outImage.cols, outImage.rows, outImage.step)->scale_simple( outImage.cols, outImage.rows, Gdk::INTERP_BILINEAR );
    image.set(pixbuf);
}

ntd · Answer 7 · 2021-11-19T00:02:20.130

IMO, as already suggested by @Miki in the comments, this is just a lifetime issue.

I had the very same problem with similar code:

{
    cv::Mat rgb;
    cv::cvtColor(src, rgb, cv::COLOR_GRAY2RGB);
    pixbuf = gdk_pixbuf_new_from_data(rgb.data,
                                      GDK_COLORSPACE_RGB, FALSE, 8,
                                      rgb.cols, rgb.rows, rgb.step,
                                      NULL, NULL);
}

The above snippet simply does not work (or works intermittently) because, quoting the gdk_pixbuf_new_from_data documentation, "the data is owned by the caller of the function".

The problem is at the time the image is rendered, rgb has been destroyed. Adding a rgb.addref() just before the pixbuf assignment resolves the issue, although introducing a memory leak.

One solution would be to leverage the destroy callback to unreference the Mat object, e.g.:

static void
unref_mat(guchar *data, gpointer user_data)
{
    ((cv::Mat *) user_data)->release();
}

{
    cv::Mat rgb;
    cv::cvtColor(src, rgb, cv::COLOR_GRAY2RGB);
    rgb.addref()
    pixbuf = gdk_pixbuf_new_from_data(rgb.data,
                                      GDK_COLORSPACE_RGB, FALSE, 8,
                                      rgb.cols, rgb.rows, rgb.step,
                                      unref_mat, &rgb);
}

score 0 · Answer 8 · answered Apr 12 '23 at 10:03

cv::Mat outImage will be distroyed when excution run out of the scope of the function displayImage2(), thus you should declare the variable outImage in your custom class which derived from Gtk::Window. Here is a example worked fine with gtkmm4 and opencv4:

#include <opencv2/opencv.hpp>
#include <gtkmm.h>

class Buttons : public Gtk::Window
{
public:
  Buttons();
  virtual ~Buttons();

protected:
  cv::Mat outImage;
  Gtk::Image image;
};

Buttons::Buttons()
{
  outImage = cv::imread("gtk.jpg");
  cv::cvtColor(outImage, outImage, cv::COLOR_BGR2RGB);
  Glib::RefPtr<Gdk::Pixbuf> pixbuf = Gdk::Pixbuf::create_from_data(outImage.data,
                                                                   Gdk::Colorspace::RGB,
                                                                   false, 
                                                                   8, 
                                                                   outImage.cols, 
                                                                   outImage.rows, 
                                                                   outImage.step);
  image.set(pixbuf);
  set_child(image);
}

Buttons::~Buttons()
{
}

int main(int argc, char *argv[])
{
  auto app = Gtk::Application::create("org.gtkmm.example");

  //Shows the window and returns when it is closed.
  return app->make_window_and_run<Buttons>(argc, argv);
}

display opencv cv::Mat image in gtkmm

8 Answers8