Camera 2, increase FPS

Question

I'm using Camera 2 API to save JPEG images on disk. I currently have 3-4 fps on my Nexus 5X, I'd like to improve it to 20-30. Is it possible?

Changing the image format to YUV I manage to generate 30 fps. Is it possible to save them at this frame-rate, or should I give up and live with my 3-4 fps?

Obviously I can share code if needed, but if everyone agree that it's not possible, I'll just give up. Using the NDK (with libjpeg for instance) is an option (but obviously I'd prefer to avoid it...).

Thanks

EDIT: here is how I convert the YUV android.media.Image to a single byte[]:

private byte[] toByteArray(Image image, File destination) {

    ByteBuffer buffer0 = image.getPlanes()[0].getBuffer();
    ByteBuffer buffer2 = image.getPlanes()[2].getBuffer();
    int buffer0_size = buffer0.remaining();
    int buffer2_size = buffer2.remaining();

    byte[] bytes = new byte[buffer0_size + buffer2_size];

    buffer0.get(bytes, 0, buffer0_size);
    buffer2.get(bytes, buffer0_size, buffer2_size);

    return bytes;
}

EDIT 2: another method I found to convert the YUV image into a byte[]:

private byte[] toByteArray(Image image, File destination) {

    Image.Plane yPlane = image.getPlanes()[0];
    Image.Plane uPlane = image.getPlanes()[1];
    Image.Plane vPlane = image.getPlanes()[2];

    int ySize = yPlane.getBuffer().remaining();

    // be aware that this size does not include the padding at the end, if there is any
    // (e.g. if pixel stride is 2 the size is ySize / 2 - 1)
    int uSize = uPlane.getBuffer().remaining();
    int vSize = vPlane.getBuffer().remaining();

    byte[] data = new byte[ySize + (ySize/2)];

    yPlane.getBuffer().get(data, 0, ySize);

    ByteBuffer ub = uPlane.getBuffer();
    ByteBuffer vb = vPlane.getBuffer();

    int uvPixelStride = uPlane.getPixelStride(); //stride guaranteed to be the same for u and v planes

    if (uvPixelStride == 1) {

        uPlane.getBuffer().get(data, ySize, uSize);
        vPlane.getBuffer().get(data, ySize + uSize, vSize);
    }
    else {

        // if pixel stride is 2 there is padding between each pixel
        // converting it to NV21 by filling the gaps of the v plane with the u values
        vb.get(data, ySize, vSize);
        for (int i = 0; i < uSize; i += 2) {
            data[ySize + i + 1] = ub.get(i);
        }
    }

    return data;
}

Answer 1

The dedicated JPEG encoder units on mobile phones are efficient, but not generally optimized for throughput. (Historically, users took one photo every second or two). At full resolution, the 5X's camera pipeline will not generate JPEGs at faster than a few FPS.

If you need higher rates, you need to capture in uncompressed YUV. As mentioned by CommonsWare, there's not enough disk bandwidth to stream full-resolution uncompressed YUV to disk, so you can only hold on to some number of frames before you run out of memory.

You can use libjpeg-turbo or some other high-efficiency JPEG encoder and see how many frames per second you can compress yourself - this may be higher than the hardware JPEG unit. The simplest way to maximize the rate is to capture YUV at 30fps, and run some number of JPEG encoding threads in parallel. For maximum speed, you'll want to hand-write the code talking to the JPEG encoder, because your source data is YUV, not RGB, which most JPEG encoding interfaces tend to accept (even though typically the colorspace of an encoded JPEG is actually YUV as well).

Whenever an encoder thread finishes the previous frame, it can grab the next frame that comes from the camera (you can maintain a small circular buffer of the latest YUV Images to make this simpler).