Improving image processing speed

Question

I am using C++ and OpenCV to process some images taken from a Webcam in realtime and I am looking to get the best speed I can from my system.

Other than changing the processing algorithm (assume, for now, that you can't change it). Is there anything that I should be doing to maximize the speed of processing?

I am thinking maybe Multithreading could help here but I'm ashamed to say I don't really know the ins and outs (although obviously I have used multithreading before but not in C++).

Assuming I have an x-core processor, does splitting the processing into x threads actually speed things up?...or would the management overhead of these threads negate it assuming that I am looking for a throughput of 20fps (I assume that will affect the answer you give as it should give you an indication of how much processing will be done per thread)

Would multithreading help here?

Are there any tips for increasing the speed of OpenCV specifically, or any pitfalls that I might be falling into that reduce speed.

Thanks.

Tossing each incoming frame to a rotating group of threads should allow for dead simple multithreading. — Mark Ransom, Jan 27 '12 at 20:37

score 6 · Accepted Answer · answered Jan 27 '12 at 20:46

The easier way, I think, could be pipelining frame operations.

You could work with a thread pool, allocating sequentially a frame memory buffer to the first available thread, to be released to pool when the algorithm step on the associated frame has completed.

This could leave practically unchanged your current (debugged :) algorithm, but will require substantially more memory for buffering intermediate results.

Of course, without details about your task, it's hard to say if this is appropriate...

score 5 · Answer 2 · answered Jan 27 '12 at 20:29

There is one important thing about increasing speed in OpenCV not related to processor nor algorithm and it is avoiding extra copying when dealing with matrices. I will give you an example taken from the documentation:

"...by constructing a header for a part of another matrix. It can be a single row, single column, several rows, several columns, rectangular region in the matrix (called a minor in algebra) or a diagonal. Such operations are also O(1), because the new header will reference the same data. You can actually modify a part of the matrix using this feature, e.g."

// add 5-th row, multiplied by 3 to the 3rd row
M.row(3) = M.row(3) + M.row(5)*3;

// now copy 7-th column to the 1-st column
// M.col(1) = M.col(7); // this will not work
Mat M1 = M.col(1);
M.col(7).copyTo(M1);

Maybe you already knew this issue but I think it is important to highlight headers in openCV as an important and efficient coding tool.

score 4 · Answer 3 · answered Jan 27 '12 at 20:29

Assuming I have an x-core processor, does splitting the processing into x threads actually speed things up?

Yes, although it very heavily depends on the particular algorithm being used, as well as your skill in writing threaded code to handle things like synchronization. You didn't really provide enough detail to make a better assessment than that.

Some algorithms are extremely easy to parallelize, like ones that have the form:

for (i=0; i < DATA_SIZE; i++)
{
   output[i] = f(input[i]);
}

for some function f. These are known as embarassingly parallelizable; you can simply split the data into N blocks and have N threads process each block individually. Libraries like OpenMP make this kind of threading extremely simple.

kmote · Answer 4 · 2012-01-27T20:32:31.707

3

Unless the particular algorithm you are using is already optimized for a multithreaded/parallel platform, throwing it at an x-core processor will do nothing for you. The algorithm has to be inherently threadable to benefit from multiple threads. But if it wasn't designed with that in mind, it would have to be altered. On the other hand, many image processing algorithms are "embarassingly-parallel", at least in concept. Can you share more details about the algorithm you have in mind?

edited Jan 27 '12 at 20:32

answered Jan 27 '12 at 20:22

kmote

16,095
11
68
91

Well at the moment I am just using standard OpenCV algorithms to "fill in the gap" of processing. (Are they multithreaded/optimised btw?).....I haven't actually decided the algorithm I am going to use, but it will likely be a pixel-by-pixel analysis that I can split into x threads if need be and have them all return a result. – Cheetah Jan 27 '12 at 20:34
Algorithms are implemented in code. Code is inherently thread-safe, so it's down to the data. – Martin James Jan 27 '12 at 20:34
@MartinJames, code is *mostly* thread-safe - you have to avoid mutable static variables in your code. – Mark Ransom Jan 27 '12 at 20:55
All code is thread-safe unless is is self-modifying. Mutable statics are data. – Martin James Jan 28 '12 at 00:38
@MartinJames don't really know what you mean by that. – Cheetah Jan 28 '12 at 12:53

score 2 · Answer 5 · answered Jan 27 '12 at 20:48

If your threads can operate on different data, it would seem reasonable to thread it off, perhaps queueing each frame object to a thread pool. You may have to add sequence numbers to the frame objects to ensure that the processed frames emerging from the pool are delivered in the same order they went in.

score 1 · Answer 6 · answered Jan 16 '14 at 16:57

As example code for multi-threaded image processing with OpenCV, you might want to check out my code:

https://github.com/vmlaker/sherlock-cpp

It's what I came up with wanting to take advantage of x-core CPU to improve performance of object detection. The detect program is basically a parallel algorithm that distributes tasks among multiple threads, a separate pipelined thread for every task:

Allocation of frame memory and video capture.
Object detection (one thread per each Haar classifier.)
Augmenting output with detection result and displaying the frame.
Memory deallocation.

With memory for every captured frame shared between all threads, I got great performance and CPU utilization.

Improving image processing speed

6 Answers6