reduce CPU overhead while proccessing Video Stream

Question

I am developing C# WPF Auto Number Plate Recognition Using an OCR.

The Flow is, i am getting a pictures from a video stream MJPEG and this images should be passed to the OCR to get the plate number and other details.

The problem is : the Video stream is producing about 30 Frame/second and the CPU can't handle this much of processing also it will take around 1 Sec to process 1 frame, Also when i will get many frames on the Queue the CPU will be 70% used (Intel I7 4th G).

Can anyone suggest solution and better implementation.

//This is the queue where it will hold the frames 
        // produced from the video streaming(video_Newfram1)

        private readonly Queue<byte[]> _anpr1Produces = new Queue<byte[]>();


        //I am using AForg.Video to read the MJPEG Streaming
        //this event will be triggered for every frame
        private void video_NewFrame1(object sender, NewFrameEventArgs eventArgs)
        {

            var frameDataAnpr = new Bitmap(eventArgs.Frame);
            AnprCam1.Source = GetBitmapimage(frameDataAnpr);

            //add current fram to the queue 
            _anpr1Produces.Enqueue(imgByteAnpr);

            //this worker is the consumer that will 
            //take the frames from the queue to the OCR processing 
            if (!_workerAnpr1.IsBusy)
            {
                _workerAnpr1.RunWorkerAsync(imgByteAnpr);
            }
        }

        //This is the consumer, it will take the frames from the queue to the OCR

        private void WorkerAnpr1_DoWork(object sender, DoWorkEventArgs e)
        {
            while (true)
            {
                if (_anpr1Produces.Count <= 0) continue;
                BgWorker1(_anpr1Produces.Dequeue());
            }
        }

        //This method will process the frames that sent from the consumer
        private void BgWorker1(byte[] imageByteAnpr)
        {
            var anpr = new cmAnpr("default");
            var objgxImage = new gxImage("default");

            if (imageByteAnpr != null)
            {
                objgxImage.LoadFromMem(imageByteAnpr, 1);
                if (anpr.FindFirst(objgxImage) && anpr.GetConfidence() >= Configs.ConfidanceLevel)
                {
                    var vehicleNumber = anpr.GetText();
                    var vehicleType = anpr.GetType().ToString();
                    if (vehicleType == "0") return;

                    var imagename = string.Format("{0:yyyy_MMM_dd_HHmmssfff}", currentDateTime) + "-1-" +
                                    vehicleNumber + ".png";

                    //this task will run async to do the rest of the process which is saving the vehicle image, getting vehicle color, storing to the database ... etc
                    var tsk = ProcessVehicle("1", vehicleType, vehicleNumber, imageByteAnpr, imagename, currentDateTime, anpr, _anpr1Produces);

                }
                else
                {
                    GC.Collect();
                }
            }
        }

Please dont mess with the GC by yourself. Do use [gcServer](https://learn.microsoft.com/en-us/dotnet/framework/configure-apps/file-schema/runtime/gcserver-element) instead. — Larry, Jun 01 '18 at 19:12

score 1 · Answer 1 · answered Feb 20 '17 at 02:34

What you should do is this:

First, figure out if a frame is worth processing or not. If you're using a compressed video stream, you can usually quickly read the frame's compressed size. It stores the difference between the current frame and the previous one.

When it's small, not much changed (i.e: no car drove by).

That's a low-tech way to do motion detection, without even having to decode a frame, and it should be extremely fast.

That way, you can probably decide to skip 80% of the frames in a couple of milliseconds.

Once and a while you'll get frames that need processing. Make sure that you can buffer enough frames so that you can keep recording while you're doing your slow processing.

The next thing to do is find a region of interest, and focus on those first. You could do that by simply looking at areas where the color changed, or try to find rectangular shapes.

Finally, one second of processing is SLOW if you need to process 30 fps. You need to make things faster, or you'll have to build up a gigantic buffer, and hope that you'll ever catch up if it's busy on the road.

Make sure to make proper use of multiple cores if they are available, but in the end, knowing which pieces of the image are NOT relevant is the key to faster performance here.

I think using the TPL Dataflow library in here would definitely make sense. — Larry, Jun 01 '18 at 19:13
@Larry: yes, you can parallelise the process at different levels. Processing each "region of interest" could be a task, or you can do it on frame level. But remember, paralellism doesn't lead to less work, in fact it adds work to do. You're just spreading out the problem without making it smaller. So it's still important to look in a clever way which tasks can be skipped entirely, or where processing less data yields similar results. For example, processing characters with 30 pixels height leads to almost the same result as 300 pixels, but it's a lot faster. — Wouter van Nifterick, Jun 06 '18 at 07:47
You are right, I missed the point that the question is about a unique stream. Of course TPL usage would better make sense in a multiple streams context, so some controlled amount of resources (threads) could be allocated to the units of work of the pipelined processing you described in your answer. — Larry, Jun 06 '18 at 08:05

reduce CPU overhead while proccessing Video Stream

1 Answers1