0

I'm trying to use the OpenCV Stitcher class to stitch frames from two cameras. I would just like to do this in the most simple way to start with and then get into the details of the Stitcher class.

The code is really simple and not so many line of code. 

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include "opencv2/imgcodecs.hpp"
#include "opencv2/highgui.hpp"
#include "opencv2/stitching.hpp"
#include <opencv2/opencv.hpp>
#include <opencv2/imgcodecs.hpp>

#include<iostream>
#include <fstream>
#include<conio.h>           // may have to modify this line if not using Windows

using namespace std;
using namespace cv;
//using namespace cv::detail;

bool try_use_gpu = true;
Stitcher::Mode mode = Stitcher::PANORAMA; //Stitcher::SCANS; 
vector<Mat> imgs;

int main()
{

    //initialize and allocate memory to load the video stream from camera 
    cv::VideoCapture camera0(2);
    cv::VideoCapture camera1(3);

    if (!camera0.isOpened()) return 1;
    if (!camera1.isOpened()) return 1;

    //Mat output_frame;
    cv::Mat3b output_frame;
    cv::Stitcher stitcher = cv::Stitcher::createDefault(true);
    //Ptr<Stitcher> stitcher = Stitcher::create(mode, try_use_gpu);
    Mat camera0_frame_resized;
    Mat camera1_frame_resized;

    while (true) {
        //grab and retrieve each frames of the video sequentially 
        cv::Mat3b camera0_frame;
        cv::Mat3b camera1_frame;
        camera0 >> camera0_frame;
        camera1 >> camera1_frame;

        resize(camera0_frame, camera0_frame_resized, Size(640, 480));
        resize(camera1_frame, camera1_frame_resized, Size(640, 480));

        imgs.push_back(camera0_frame_resized);
        imgs.push_back(camera1_frame_resized);
        Stitcher::Status status = stitcher.stitch(imgs, output_frame);
        output_frame.empty();
        if (status != Stitcher::OK) {
            cout << "Can't stitch images, error code = " << int(status) << endl;
            return -1;
        }

        cv::imshow("Camera0", camera0_frame);
        cv::imshow("Camera1", camera1_frame);

        cv::imshow("Stitch", output_frame);

        //wait for 40 milliseconds
        int c = cvWaitKey(5);

        //exit the loop if user press "Esc" key  (ASCII value of "Esc" is 27) 
        if (27 == char(c)) break;
    }

    return 0;
}

I am using OpenCV 3.2 and Visual Studio Community Edition 2017 running on windows 10.

The issue is that it is extremely slow, It seems to stitch the first frame and then it kind of gets stuck and nothing else happens, after seconds/minute maybe next frames appear.

I am running on a extremely fast CPU and top of the line GPU. I was not expecting fast stitching but this is just way to slow and gets me thinking that I am doing something wrong. Any ideas on why it stitches only the first frame and then gets stuck?

theAlse
  • 5,577
  • 11
  • 68
  • 110
  • Can you attach 2 sample input images which you want to stitch ? – ZdaR May 05 '17 at 06:09
  • I don't know how Opencv function is implemented but usually stitching is been done through extracting descriptors, then matching them, then processing the images. This is notoriously expensive. Do you have any reference timing to say that it should be faster ? – kebs May 05 '17 at 07:36
  • @kebs, I know stitching is a heavy process, but this is stitching ~1 frame per minute. This can not be this slow is what I mean. – theAlse May 05 '17 at 07:40
  • Am I correct in assuming that you are stitching two images from stationary free running cameras expecting it to be in real-time? If so I did something similar a long time ago (Pentium III 733) with 3 cameras at low resolution (352x288) and it took several minutes per frame. If this is the case then you need to calculate where these frames overlap once and reuse those calculations. – graham.reeds May 05 '17 at 07:46
  • @graham.reeds I am using 2 cameras, as you can see above I resize images to 640x480 before perform the stitch, I am running on a intel i7-7700 and GeForce GTX 1080 (although I am not sure it is really being used). Should I really be expecting 1 frame per minute? I even lowered the resolution to basically nothing 200x100 and still it was not much faster – theAlse May 05 '17 at 07:52
  • Are you sure you don't have an issue with hardware/OS with acquisition? Did you try the process reading images from disk files? If it's faster, then you have an answer? – kebs May 05 '17 at 07:54
  • Have you tried doing a release build of OpenCV and your project? – Utkarsh Sinha May 05 '17 at 08:02
  • @UtkarshSinha, I tested the release build without debug and still not much of a difference – theAlse May 05 '17 at 08:13
  • The two topics that affect performance is the resolution you choose for the image registration stage and for the composition stage. Can these be lowered and still give you an adequate panorama? – Tiago Cunha May 05 '17 at 09:58
  • @theAlse but it is going to be slow if it is redoing the same calculations over and over. Precalculate if possible! – graham.reeds May 05 '17 at 12:08

0 Answers0