How to get fast frame rate from a camera with a callback api?

Question

I am using a FirstLightImaging C-Blue One camera and an actuator which receives analog voltages. I have built a closed loop system which calculates image shifts and provides feedback to actuator.

The system is working properly, however, always a nice to have feature (live view of images), is killing my main acquisition frame rate. Inspired by the answer by @dan-mašek, I want to ask if I can separate acquisition thread, image processing thread and feedback to actuator thread. The good thing is feedback to actuator thread is already separate. The camera gives images in the dimensions of 768 x 768, we bin it to 256 x 256. The binning function is a fast version of binning but in process destroys original image. Thus, 768 x 768 image cannot be displayed now.

Would it be possible to have a live view of 768x768 image?

Details follow. The code snippets also I have provided and linked my full code as well. I am a physicist and c++ newbie, pardon me for not following a c++ convention. Although I have tried my best the code is readable.

Some Context before I ask:

The camera provides a callback API; that is, we only need to implement a function, and that function has a parameter uint16_t* type. The frame rate is 655 FPS, i.e., the function must finish whatever it should do in 1.52 ms. Otherwise, we will see frame loss. In my function, I calculate an image shift with respect to a prior taken reference image and provide feedback to an actuator to correct the said shift. This is a closed loop system. Already the code is multithreaded, i.e.

1. Main Thread: Admin thread which spawns other threads and takes user inputs if user wants to terminate program or change parameters etc.
2. Image Acquisition Thread: Spawned by the camera API, we only provide our function which this thread call. This function is time critical and must finish before 1.52 ms. The feedback to the actuator is pushed to a global queue.
3. Voltage to Actuator Thread: This threads read from the global queue and apply the voltages (feedback) to the actuator. Voltage to actuator feedback giving method executes in 1.4 ms, round trip time. Thus this functionality is made asynchronous, to achieve frame rate.

Currently, Image acquisition thread's callback function does many things:

1. Bin Image (Executes in 0.34 ms): Answered By @fabian
2. Preprocess Images: Mean Subtraction and Flat Fielding
3. Conputes the Shift (Executes in 0.3-0.4 ms): Computes forward FFT, multiplies by the conjugate of the FFT of the reference, Computes Backward FFT.
4. Computed Subpixel Shift using an interpolation method.
5. Signal to the Voltage to Actuator Thread that feedback is ready.

Overall, the total time to complete steps 1 to 4 is 0.8 ms (which is good I think).

The issue what I face is in the getImageShiftFunction, Currently, I have configured it to display frames at 20 fps, i.e. display every 32th frame out of 655 fps (655/32 = 20.46). But If I try to display every 16th frame (out of 655 fps, 655/16 = 40 frames per second display), The acquisition rate of the camera decreases to 627.

The factory software of the Camera, can display live frames at 50 fps, while simultaneously filling a circular buffer at 655 fps, thus I believe,it is doable.

You can see in the following code, I am only notifying the display thread, no data is being transferred, thus, I believe this code to be very fast.

if (!(curr_count & ((1 << 5) - 1))) {
    unique_lock<mutex> dul(displayMutex);
    displayReady = true;
    dul.unlock();
    displayConditionalVariable.notify_one();
    dul.lock();
    displayConditionalVariable.wait(dul, []() { return !displayReady; });
    dul.unlock();
}

The display thread code follows below. The image pointer is a constant pointer to a non-constant double. The thread when notified, just updates the image using imshow method of OpenCV. The display code is benchmarked separately, showing that it can provide 60 fps, it should be no problem standalone.

Now, If I set the window's width height to 512 x 512, again I see a drop in acquisition frame rate to be about 640 fps.

DWORD WINAPI displayThread(LPVOID lparam) {
    cv::Mat img;
    double *image = (double *)lparam;
    cv::namedWindow("Live!", cv::WINDOW_NORMAL);
    cv::resizeWindow("Live!", 256, 256);
    while (true) {
        unique_lock<mutex> dul(displayMutex);
        displayConditionalVariable.wait(dul, [](){return displayReady;});
        displayReady = false;
        dul.unlock();
        displayConditionalVariable.notify_one();
        const cv::Mat img(cv::Size(NX, NY), CV_64FC1, image);
        cv::imshow("Live!", img);
        cv::waitKey(1);
    }
}

Thus the ultimate question, I ask is, how to get about 40 fps (with hopefully 512x512 window size or non-binned image 768x768) in live display, while not compromising on the raw camera acquisition frame rate.

Supplementary Information:

The codebase is here.

The Loop starts from imageReceived function, workFunction is a function pointer to closedLoopCallBack function.