0

I am working with the CMLN-13S2C-CS CCD camera from PointGrey Systems. It uses FlyCapture API to grab images. I would like to grab these images and do some stuff in OpenCV with them using python.

I am aware of the following python binding: pyflycapture2. With this binding I am able to retrieve images. However, I cannot retrieve the images in color, which is what the camera should be able to do.

The videomode and framerate that the camera is able to handle are VIDEOMODE_1280x960Y8, and FRAMERATE_15, respectively. I think it has something to do with the pixel_format, which I think should be raw8.

Is anyone able to retrieve a color image using this or any existing python binding for flycapture? Note that I am working on Linux.

The Dude
  • 3,795
  • 5
  • 29
  • 47

2 Answers2

0

You don't need to use the predefined modes. The Context class has the set_format7_configuration(mode, x_offset, y_offset, width, height, pixel_format) method with which you can use your custom settings. Using this you can at least change the resolution of the grabbed image. Usage example:

c.set_format7_configuration(fc2.MODE_0, 320, 240, 1280, 720, fc2.PIXEL_FORMAT_MONO8)

As for the coloring issue. I've so far managed to get a colored image using PIXEL_FORMAT_RGB8 and modifying the Image class in flycapture2.pyx as follows:

def __array__(self):
    cdef np.ndarray r
    cdef np.npy_intp shape[3]  # From 2 to 3
    cdef np.dtype dtype
    numberofdimensions = 2  # New variable
    if self.img.format == PIXEL_FORMAT_MONO8:
        dtype = np.dtype("uint8")
    elif self.img.format == PIXEL_FORMAT_MONO16:
        dtype = np.dtype("uint16")
    elif self.img.format == PIXEL_FORMAT_RGB8:  # New condition
        dtype = np.dtype("uint8")
        numberofdimensions = 3
        shape[2] = 3
    else:
        dtype = np.dtype("uint8")
    Py_INCREF(dtype)
    shape[0] = self.img.rows
    shape[1] = self.img.cols

    # nd value (numberofdimensions) was always 2; stride set to NULL
    r = PyArray_NewFromDescr(np.ndarray, dtype,
            numberofdimensions, shape, NULL,
            self.img.pData, np.NPY_DEFAULT, None)
    r.base = <PyObject *>self
    Py_INCREF(self)
    return r

This code is most likely not flawless (i.e I removed the stride stuff) for the simple reason that I have pretty much 0 experience with C and Cython but this way I at least managed to get a colored frame (now in the process of trying to get the PIXEL_FORMAT_RAW8 working).

And just as a reminder: the flycapture2.pyx is a Cython file so you need to recompile it before you can use it (I just run the pyflycap2 install script again).

Rene Juuse
  • 575
  • 8
  • 16
  • Thank you Rene. I am able to set custom settings for my camera. Unfortunately, it does not support `RGB8` as a `PIXEL_FORMAT`. Naively I tried to change the `RGB` to `RAW8` in your `array` function. This however does not work and gives weird results. Recompiling is successful via re-installation. Currently I am trying to adjust your code for `RAW8`. Have you had some progress in doing so? – The Dude May 23 '16 at 20:35
  • @TheDude I ended up going with a different route. Didn't go after the `RAW8` format but used `PIXEL_FORMAT_411YUV8` which worked for my purpose. What I ended up doing was connecting to the camera with `pyflycap2` and just used it for applying the settings (pixel format and resolution). After doing that I disconnected from the camera and for frame grabbing opened up a new capture with `OpenCV`. In order to get the camera picked up by `OpenCV`, I first installed the `libusb` and then `libdc1394` with the `libusb` support. You can read more about it here [link](https://www.ptgrey.com/tan/10868) – Rene Juuse May 23 '16 at 20:56
  • @TheDude and I know the link talks about OSX but the process is more or less the same (you use apt-get instead of port) for Linux (running on Ubuntu myself). – Rene Juuse May 23 '16 at 21:05
  • I have installed the following: `sudo apt-get install libusb-1.0-0` and `sudo apt-get install libdc1394-22`. What is next? Am I supposed to follow the rest of the instructions? If so, where is `libdc1394-22` located. If not, what would be my next move? – The Dude May 23 '16 at 21:23
  • @TheDude If you are running on USB2 then this should be enough. If you want to know how to get the frames using OpenCV, take a look here [link](http://docs.opencv.org/3.0-beta/doc/py_tutorials/py_gui/py_video_display/py_video_display.html). – Rene Juuse May 24 '16 at 07:27
  • I am able to get an image. This image is however still in `gray scale`. It turns out that my camera can only handle `mono` and the `raw8/raw16` pixel formats. These raw ones produce the color image if I were to open the camera using `flycap`. I tried to obtain a color image in `opencv` using the option `cv2.COLOR_BAYER_GR2RGB` in `cvtColor`. However, it turns out that my opencv script (a direct copy from the link you provided) sets my camera in a mode which is not the color mode anymore. How and when should I set this? Setting this in flycap and running the script does not work. – The Dude May 24 '16 at 08:26
  • @TheDude what do you get when you print out the shape of the initial (before doing any coloring operations) captured frame (`frame.shape`)? – Rene Juuse May 24 '16 at 21:39
  • the `shape` is `(480, 640)`. Also, running `flycap` after the `opencv` script tells me that the `video mode` is set to `Y8` with a resolution of `640x480`. The image I get in `flycap` is also grayscaled. I need to set the `pixel format` in the custom settings to `RAW8` before I can obtain color images in `flycap` again. Here I can also set the resolution again such that I obtain the full `1280x960`. – The Dude May 25 '16 at 07:40
-1

I'm using the same camera with Matlab and also got an issues with "raw8" format. So, I've chose "rgb8", specifically, "F7_RGB_644x482_Mode1" and all things starts to work (not sure, how it should look at Python).

P.S. At the moment I'm trying to start work with Python and pyflycapture2, let's see, if I would be able to find workaround.

UPD: Okay, now I know the things. :) Your (and mine) issue reasons are buried inside the pyflycapture2 itself, especially "Image" class definition. You can have a look here: https://github.com/jordens/pyflycapture2/blob/eec14acd761e89d8e63a0961174e7f5900180d54/src/flycapture2.pyx 

if self.img.format == PIXEL_FORMAT_MONO8:
            dtype = np.dtype("uint8")
            stride[1] = 1
        elif self.img.format == PIXEL_FORMAT_MONO16:
            dtype = np.dtype("uint16")
            stride[1] = 2
        else:
            dtype = np.dtype("uint8")
            stride[1] = self.img.stride/self.img.cols

ANY image will be converted into grayscale, even if it was RGB initially. So, we need to update that file somehow.

Dmitriy
  • 149
  • 2
  • 5