4

I had to create a system that can process images in realtime. I have implemented in C++ a pixel format conversion system that can also do some simple transformation (currently: rotation & mirroring).

Input/output format of the system are frame in a the following formats:

  • RGB (24, 32)
  • YUYV420, YUYV 422
  • JPEG
  • Raw greyscale

For instance, one operation can be:

  • YUYV422 -> rotation 90 -> flip Horiz -> RGB24
  • Greyscale -> rotation 270 -> flip Vert -> YUYV420

The goal of the system is to offer best performance for rotation/mirroring and pixel format conversion. My current implementation rely on OpenCV, but I suffer from performance issues when processing data above 2k resolutions.

The current implementation uses cv::Mat and cv::transpose/cv::flip/cv::cvtColor, and I optimized the system to remove transitionnal buffers and copy as much as possible.

Not very happy to reinvent the wheel, I know that using swscale and some filters from FFMpeg, it is possible to achieve the same result. My question are:

  1. The FFMpeg system is rather generic, do you think I might suffer from footprint/performance caveat with this solution?

  2. Format conversion seems somewhat ooptimized in OpenCV, but I have no idea about FFMpeg implementation... (note: I'm on x86_64 intel platform with SSE)

  3. Do you know any library than can handle this kind of simple transformation for real time?

Thank you

lp35
  • 198
  • 3
  • 9
  • 2k is huge. Is there enough room for enhancement in the hardware setup ? (I/O bandwidth, etc.) – 9dan Feb 01 '19 at 14:53
  • OpenCV can use FFMpeg as its background: https://docs.opencv.org/3.4/d0/da7/videoio_overview.html What do you mean by realtime? How many fps do you want to achieve? As @9dan spotted- there may be hardware limitations. – Piotr Siekański Feb 01 '19 at 15:42
  • @9dan what do you suggest? – lp35 Feb 04 '19 at 08:59

1 Answers1

1

OpenCV implementation is optimised for your configuration. Don't expect improvements from ffmpeg. Recently, OpenCV switched to libjpeg-turbo with SSE optimizations, this may improve JPEG conversions.

Alex Cohn
  • 56,089
  • 9
  • 113
  • 307
  • It is `libjpeg-turbo` but without SIMD that is included for now (see [here](https://github.com/opencv/opencv/issues/12115)). Feel free to submit a pull request to add SIMD support. – Catree Feb 02 '19 at 17:43
  • @Catree thanks for this clarification. I understand that this concerns the official release, local build can enable SIMD easily, can't it? – Alex Cohn Feb 02 '19 at 21:20
  • 1
    `libjpeg-turbo` included as a third-party into OpenCV does not have SIMD code. Yes you should be able to install `libjpeg-turbo` (e.g. from a package manager) and OpenCV should be able to automatically detect and link against the host `libjpeg-turbo`. – Catree Feb 03 '19 at 00:49
  • Hi Alex, do you have some resource/numbers/link to give forr this assessment? Is it your personnal experience? Thx – lp35 Feb 04 '19 at 08:58
  • No, I don't have strict measurements at hand. My experience with ffmpeg in similar transformations was rather disappointing. If you find a confirmation or a contradictory evidence, please post it here. – Alex Cohn Feb 04 '19 at 09:27
  • OpenCV is really poor when trying to convert from **other format** to YUV (e.g: impossible to convert RGB(A) to YUV420 for instance) – lp35 Mar 28 '19 at 13:18
  • @lp35 please check https://stackoverflow.com/a/48125183/192373 if you have some troubles with this – Alex Cohn Mar 28 '19 at 13:34