Camera2 Pass Images from ImageReader to MediaRecorder

Question

I'm trying to create a Camera2 CameraCaptureSession that is capable of four outputs:

On-screen preview (SurfaceView, up to 1080p)
Photo capture (ImageReader, up to 8k photos)
Video Capture (MediaRecorder/MediaCodec, up to 4k videos)
Frame Processing (ImageReader, up to 4k video frames)

Unfortunately Camera2 does not support attaching all of those four outputs (Surfaces) at the same time, so I'm going to have to make a compromise.

The compromise that seemed most logical to me was to combine the two video capture pipelines into one, so that the Frame Processing output (#4, ImageReader) redirects the frames into the Video Capture output (#3, MediaRecorder).

How do I write the Images from the ImageReader:

val imageReader = ImageReader.newInstance(4000, 2256, ImageFormat.YUV_420_888, 3)
imageReader.setOnImageAvailableListener({ reader ->
  val image = reader.acquireNextImage() ?: return@setOnImageAvailableListener
  callback.onVideoFrameCaptured(image)
}, queue.handler)

val captureSession = device.createCaptureSession(.., imageReader.surface)

..into the Surface from the MediaRecorder?

val surface = MediaCodec.createPersistentInputSurface()
val recorder = MediaRecorder(context)
..
recorder.setInputSurface(surface)

I'm thinking that I might need an OpenGL pipeline here with a pass-through shader - but I don't know how I get from the ImageReader's Image to an OpenGL texture, so any help here would be appreciated.

What I tried: I looked into the HardwareBuffer APIs, specifically

auto clientBuffer = eglGetNativeClientBufferANDROID(hardwareBuffer);
...
auto image = eglCreateImageKHR(display,
                               EGL_NO_CONTEXT,
                               EGL_NATIVE_BUFFER_ANDROID,
                               clientBuffer,
                               attribs);
...
glEGLImageTargetTexture2DOES(GR_GL_TEXTURE_EXTERNAL, image);

And I think this might work, but it requires API Level 28. So I still need a solution for API Level 23 and above. The image.getPlanes() function returns me three ByteBuffers for the YUV data, not sure how I can create an OpenGL texture from there though..

Did you try to use a `Surface` created from a `SurfaceTexture` with the `CameraCaptureSession`? This should make video frames available as an OpenGL external texture. — dev.bmax, Aug 16 '23 at 19:38
@dev.bmax I did, yes, but I need `Image` instances because I use other ML APIs that only take `Image`s as an argument. — mrousavy, Aug 16 '23 at 19:42
In other words, I need to use an `ImageReader` to begin with, then go through an OpenGL pipeline to also draw to the Video Recording Surface — mrousavy, Aug 16 '23 at 19:43

dev.bmax · Answer 1 · 2023-08-16T21:09:08.787

1

If I understand correctly, you have an android.media.Image that you need to make available to the OpenGL pipeline.

I would suggest the following setup:

Create an OpenGL texture using glGenTextures().
Create a SurfaceTexture using the texture name (which is actually an integer) from the previous step.
Create a Surface from the SurfaceTexture.
Create an ImageWriter passing the Surface from the previous step.
For each video frame call queueInputImage(), using the available Image.

** Don't forget to call setOnFrameAvailableListener() when you prepare the SurfaceTexture and then updateTexImage() for every video frame.

*** If you want to handle image orientation correctly, you can use getTransformMatrix() and transform texture coordinates in the vertex shader.

edited Aug 16 '23 at 21:09

answered Aug 16 '23 at 20:47

dev.bmax

8,998
3
30
41

Hey - thanks for your reply! I was able to skip the OpenGL part and just directly use `ImageWriter` to write Images to the `MediaRecorder` Surface (see my answer above). – mrousavy Aug 17 '23 at 10:01
The problem with this approach is that there are some hickups and I have no idea why, do you think the extra OpenGL route could solve this? – mrousavy Aug 17 '23 at 10:02
I am not familiar with the internals of `MediaRecorder` but I suspect that it ignores the presentation timestamps of the images. This could result in uneven playback. `MediaCodec` + `MediaMuxer` are probably better suited for this task. – dev.bmax Aug 17 '23 at 11:00
Yep, will probably go with MediaCodec + MediaMuxer in a custom OpenGL pipeline soon as I need some more advanced stuff. Then I'll just need to figure out how to write the OpenGL Texture to an Image/ImageWriter – mrousavy Aug 21 '23 at 22:08

score 1 · Accepted Answer · answered Aug 17 '23 at 10:00

I (kinda) figured it out! I found the ImageWriter API, which is exactly what I was about to rebuild from scratch - a pass-through pipeline from an Image to a Surface.

So now I stream Camera Frames into the ImageReader, call the Frame Processor with the Image, then pass the Image through to the MediaRecorder using the ImageWriter as a middle-man :)

val size = config.getOutputSizes(ImageFormat.PRIVATE).max()

// Video Recorder Surface. We need to stream Frames here if we are recording.
val surface = recordingSession.surface
val imageWriter = ImageWriter.newInstance(surface,
                                          VIDEO_OUTPUT_BUFFER_SIZE)

// Image Reader Surface. We stream Frames here for Frame Processor or Recording.
val flags = HardwareBuffer.USAGE_GPU_SAMPLED_IMAGE or HardwareBuffer.USAGE_VIDEO_ENCODE
val imageReader = ImageReader.newInstance(size.width,
                                          size.height,
                                          ImageFormat.PRIVATE,
                                          VIDEO_OUTPUT_BUFFER_SIZE,
                                          flags)

imageReader.setOnImageAvailableListener({ reader ->
  val image = reader.acquireNextImage() ?: return
  image.timestamp = System.nanoTime()

  // Call JS Frame Processor
  frameProcessor?.call(image)
                                         
  // If recording, write to Video File             
  if (isRecording) {
    imageWriter.queueInputImage(image)
  }

  image.close()
}, CameraQueues.videoQueue)

// Camera only streams frames into one single Surface
cameraSession.configure(.., imageReader.surface)

My only problem now is that the resulting video recording sometimes has ~1 second long hickups after around ~3 seconds of recording, I have no idea why. Maybe I should use MediaCodec instead of MediaRecorder. Maybe I should use a different ImageFormat. Maybe I should investigate the resulting .mp4 file to see what's wrong. Maybe I should fix the Image timestamps. I don't know.

Also, logcat gets spammed with this:

2023-08-17 11:38:17.977  3780-3899  GraphicBufferSource     com.mrousavy.camera.example          W  released unpopulated slots: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]
2023-08-17 11:38:18.021  3780-3899  GraphicBufferSource     com.mrousavy.camera.example          W  released unpopulated slots: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]
2023-08-17 11:38:18.050  3780-3899  GraphicBufferSource     com.mrousavy.camera.example          W  released unpopulated slots: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]
2023-08-17 11:38:18.082  3780-3899  GraphicBufferSource     com.mrousavy.camera.example          W  released unpopulated slots: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]
2023-08-17 11:38:18.113  3780-3899  GraphicBufferSource     com.mrousavy.camera.example          W  released unpopulated slots: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]
2023-08-17 11:38:18.146  3780-3899  GraphicBufferSource     com.mrousavy.camera.example          W  released unpopulated slots: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]
2023-08-17 11:38:18.179  3780-3899  GraphicBufferSource     com.mrousavy.camera.example          W  released unpopulated slots: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]

But hey - it records a video. This is a good start.

Camera2 Pass Images from ImageReader to MediaRecorder

2 Answers2

Linked