How to decode an MJPEG to raw RGB (or YUV) data

Question

I am using Video4Linux2 to open a connection to the camera connected to my machine. I have the ability to either request YUV or MJPEG data from my camera device. Since increasing the requested resolution from the camera, while also requesting YUV, causes the program to slow past the refresh rate of the camera (presumably because there is too much data to send in that amount of time), I require using the MJPEG data from the camera. I have been stuck for a while, and have found very little resources online on how to decode an MJPEG.

By the way, I have all of the following data:

unsigned char *data; // pointing to the data for the most current mjpeg frame from v4l2
size_t data_size; // the size (in bytes) of the mjpeg frame received from v4l2

unsigned char *r, *g, *b; // three heap allocated arrays in which to store the resulting data
// Can easily be altered to represent an array of structs holding all 3 components,
// as well as using yuv at different rates.

All I need is the ability to convert my mjpeg frame live into raw data, either RGB, or YUV. I have heard of libraries like libjpeg, mjpegtools, nvjpeg, etc, however I have not been able to find much on how to use them to decode an mjpeg from where I am. Any help whatsoever would be greatly appreciated!

Try this https://gist.github.com/PhirePhly/3080633 but instead of reading from a file use your `data` and `data_size` for `jpg_buffer` and `jpg_size`. — jamieguinan, Jan 07 '20 at 18:44
Also, mjpeg cameras usually leave out huffman tables, so see https://github.com/jamieguinan/cti/blob/master/jpeg_misc.c#L234 and https://github.com/jamieguinan/cti/blob/master/jpeghufftables.c — jamieguinan, Jan 07 '20 at 18:48
Thank you @jamieguinan so much! I was able to use the example code that you linked to. — Gabe Rundlett, Jan 10 '20 at 14:29

score 0 · Accepted Answer · answered Jan 14 '20 at 15:51

I figured it out via the sources linked in the comments. My working example is as follows:

// variables:
struct jpeg_decompress_struct cinfo;
struct jpeg_error_mgr jerr;
unsigned int width, height;
// data points to the mjpeg frame received from v4l2.
unsigned char *data;
size_t data_size;
// a *to be allocated* heap array to put the data for
// all the pixels after conversion to RGB.
unsigned char *pixels;

// ... In the initialization of the program:
cinfo.err = jpeg_std_error(&jerr);
jpeg_create_decompress(&cinfo);
pixels = new unsigned char[width * height * sizeof(Pixel)];

// ... Every frame:
if (!(data == nullptr) && data_size > 0) {
    jpeg_mem_src(&cinfo, data, data_size);
    int rc = jpeg_read_header(&cinfo, TRUE);
    jpeg_start_decompress(&cinfo);

    while (cinfo.output_scanline < cinfo.output_height) {
        unsigned char *temp_array[] = {pixels + (cinfo.output_scanline) * width * 3};
        jpeg_read_scanlines(&cinfo, temp_array, 1);
    }

    jpeg_finish_decompress(&cinfo);
}

If this still does not work for anyone who is trying to figure the same thing out, try to incorporate the "Huffman tables", which are needed by some cameras as said in the second comment.

How to decode an MJPEG to raw RGB (or YUV) data

1 Answers1