Stitching Two Webcam Feeds Together in OpenCV

Question

Using the code below:

#include <opencv2/opencv.hpp>
#include <opencv2/stitching/stitcher.hpp>
#include <iostream>
#include <vector>

using namespace std;
using namespace cv;

int main(int argc, char *argv[])
{
    Mat fr1, fr2, pano;
    bool try_use_gpu = false;
    vector<Mat> imgs;
    VideoCapture cap(0), cap2(1);

    while (true)
    {
        cap >> fr1;
        cap2 >> fr2;
        imgs.push_back(fr1.clone());
        imgs.push_back(fr2.clone());

        Stitcher test = Stitcher::createDefault(try_use_gpu);
        Stitcher::Status status = test.stitch(imgs, pano);

        if (status != Stitcher::OK)
        {
            cout << "Error stitching - Code: " <<int(status)<<endl;
            return -1;
        }

        imshow("Frame 1", fr1);
        imshow("Frame 2", fr2);
        imshow("Stitched Image", pano);

        if(waitKey(30) >= 0) 
            break;
    }
    return 0;
}

This code throws a status error of 1 out there. I don't know what that means, nor do I know why this thing is having a hard time with webcam feeds. What's the matter?

-Tony

Have you tried to simply stick together two images from the disk? You should narrow down your problem. Also, debug it with steps into(F11 in MSVC). — LovaBill, Apr 10 '13 at 13:55
Yes, using two images without camera captures works just fine - my problem is specifically with the way OpenCV handles images it grabs using VideoCapture objects and the >> operator. Please see my comment to the answer below. — TonyRo, Apr 10 '13 at 18:29

score 4 · Answer 1 · answered Apr 10 '13 at 07:51

4

The error is somewhere in your capture process, not the stitching part. This code works fine (using these example images):

#include <opencv2/opencv.hpp>
#include <opencv2/core/core.hpp>
#include <opencv2/stitching/stitcher.hpp>
#include <iostream>
#include <vector>

using namespace std;
using namespace cv;

int main()
{
    Mat fr1 = imread("a.jpg");
    Mat fr2 = imread("b.jpg");
    Mat pano;
    vector<Mat> imgs;

    Stitcher stitcher = Stitcher::createDefault(); // The value you entered here is the default

    imgs.push_back(fr1);
    imgs.push_back(fr2);

    Stitcher::Status status = stitcher.stitch(imgs, pano);

    if (status != Stitcher::OK)
    {
        cout << "Error stitching - Code: " <<int(status)<<endl;
        return -1;
    }

    imshow("Frame 1", imgs[0]);
    imshow("Frame 2", imgs[1]);
    imshow("Stitched Image", pano);
    waitKey();

    return 0;
}

The error message Nik Bougalis dug up sounds like the stitcher can't connect the images. Are the images clear enough for the stitcher to find correspondences?

If you're sure they are, split your problem further to find the real error. Can you tweak the stitcher to work with still frames from your cameras? Are your cameras capturing correctly? Which type of image do they return?

On another note, stitching isn't very likely to work in real time, which makes your loop during capturing look a bit out of place. You might want to either capture your frames in advance and do it all in post-processing or expect a lot of manual optimization to get anywhere near a respectable frame rate.

answered Apr 10 '13 at 07:51

FvD

1,286
13
25

Thanks for updating this post. I didn't have the time to research more into this and try to see what the issue might be, but your explanation seems reasonable, as does your point about real-time stitching. – Nik Bougalis Apr 10 '13 at 10:55
Yeah, your code works fine - I tested with your images, and also a few of mine, including still images from the same webcams that I'd be using in real time. The issue is definitely in the capture - I've been fairly certain of this from the beginning. The hang up is that I simply have no idea what the issue is. The >> operator returns type Mat&. It's very, very likely it stems from the problem mentioned here, below VideoCapture::Read(): http://docs.opencv.org/modules/highgui/doc/reading_and_writing_images_and_video.html#videocapture-read That's why I've been cloning, but that doesn't work. – TonyRo Apr 10 '13 at 16:41
I expect that I'll have to downsample the images fairly heavy or change the native camera resolution, as well as investigate the quickest schemes available. It's possible I'll even have to slog through the detailed stitching cpp they provide and remove sections I can do without - I really want to give this simple way a shot first though. – TonyRo Apr 10 '13 at 16:46
Have you tried modifying the Mat containers and the contents of your vector after the capture to see if it really is the problem the doc mentions? Try loading a Mat from an image and getting another Mat from your VideoCapture and comparing their properties. Also make sure that the captured images are suitable for stitching (to eliminate that source of error). If worse comes to worst, work around the problem for now: imwrite and imread the image before stitching it and start working on the optimization. You will probably stumble upon the reason for your error at some point anyway. – FvD Apr 11 '13 at 05:45
Regarding the optimization, if your cameras are stationary you can estimate the transformation in advance and apply it to all your captured images. Distorting an image isn't very fast either, but that will save you the computationally heavy feature point search and matching. – FvD Apr 11 '13 at 05:51
Actually I think the problem is specifically related to the .clone() function - I got the code above to work just fine, simply by changing fr1.clone() to: fr1.copyTo(cpy1); imgs.push_back(cpy1); – TonyRo Apr 11 '13 at 15:37
Also, let's say I wanted to do the feature search (Orb tested faster for me) and point matching outside of the video capture while loop - that doesn't seem possible without using the detailed stitching pipeline, as members like the pairwise matches and features are private - am I reading the documentation correctly? – TonyRo Apr 11 '13 at 15:43
Well, you could call estimateTransform before entering the loop and composePanorama inside the loop. That way you would only do the "calibration" once. If you want to calculate the parameters offline and setup the stitcher with them, you should ask a separate question. I've never used the class before this post. Also you might want to consider accepting an answer or writing your own if your problem is solved. – FvD Apr 12 '13 at 10:39

score 2 · Answer 2 · answered Apr 03 '13 at 23:01

2

Looking through the OpenCV website, we find this:

class CV_EXPORTS Stitcher
{
public:
    enum { ORIG_RESOL = -1 };
    enum Status { OK, ERR_NEED_MORE_IMGS };

    // ... other stuff

Since the returned code is of type Sticher::Status we can be fairly certain that 1 actually is Sticher::Status::ERR_NEED_MORE_IMGS. Which suggests that the sticher needs more images.

Not very informative I'm afraid, but it's a start for you. Have you looked at any of the stitching examples out there?

answered Apr 03 '13 at 23:01

Nik Bougalis

10,495
1
21
37

I have - most of them look identical except that they're not grabbing frames from a webcam feed - they're just opening them or whatever. For instance: http://ramsrigoutham.com/2012/12/21/panorama-app-on-android-using-stitching-module-opencv/ – TonyRo Apr 04 '13 at 03:35

score 1 · Accepted Answer · answered Apr 12 '13 at 14:58

The problem, for whatever reason, lies in the .clone() segment. Changing the code to:

int main(int argc, char *argv[])
{
    Mat fr1, fr2, copy1, copy2, pano;
    bool try_use_gpu = false;
    vector<Mat> imgs;
    VideoCapture cap(0), cap2(1);

    while (true)
    {
        cap >> fr1;
        cap2 >> fr2;
        fr1.copyTo(copy1);
        fr2.copyTo(copy2);        

        imgs.push_back(copy1);
        imgs.push_back(copy2);

        //ETC
     }
     return 0;
}

This worked just fine.

Stitching Two Webcam Feeds Together in OpenCV

3 Answers3