Access Image Data Buffer in CameraX

Question

I use the new CameraX API for taking a picture like this :

imageButton.setOnClickListener{
            val file = File(externalMediaDirs.first(),
                "${System.currentTimeMillis()}.jpg")

            imageCapture.takePicture(executor, object :
                ImageCapture.OnImageCapturedListener() {
                override fun onCaptureSuccess(
                    image: ImageProxy,
                    rotationDegrees: Int)
                {

                    // DO I NEED TO USE 'image' TO ACCESS THE IMAGE DATA FOR MANIPULATION (e.g. image.planes)

                }

                override fun onError(
                    imageCaptureError: ImageCapture.ImageCaptureError,
                    message: String,
                    cause: Throwable?
                ) {
                    val msg = "Photo capture failed: $message"
                    Log.e("CameraXApp", msg, cause)

                }
            })

            imageCapture.takePicture(file, executor,
                object : ImageCapture.OnImageSavedListener {

                    override fun onError(
                        imageCaptureError: ImageCapture.ImageCaptureError,
                        message: String,
                        exc: Throwable?
                    ) {
                        val msg = "Photo capture failed: $message"
                        Log.e("CameraXApp", msg, exc)

                    }

                    override fun onImageSaved(file: File) {
                        val msg = "Photo capture succeeded: ${file.absolutePath}"
                        Log.d("CameraXApp", msg)



                    }
                }
        }

I want to apply some image processing with Renderscript when the image is captured. But I dont know how to access the pixels of the image that is captured. Can someone provide a solution ? I tried it with image.planes within the onCaptureSuccess() callback (see comment)

I must admit that I am new to this and do not know really what planes are. Before this, I only worked with Bitmaps ( doing some image processing on Bitmaps with Renderscript). Is there a way to turn a frame/image into a Bitmap and apply some sort image processing on it "on the fly" before saving it as a file ? If yes, how ? The official CameraX guidelines are not very helpful when it comes to this.

score 1 · Answer 1 · answered Jun 09 '20 at 13:03

This is probably very late answer but the answer is in ImageCapture callback.

You get an Image with

val image = imageProxy.image

Notice this will only work when you have images in YUV format, in order to configure your camera with that you can do it with

imageCapture = ImageCapture.Builder()
                .setCaptureMode(ImageCapture.CAPTURE_MODE_MINIMIZE_LATENCY)
                .setBufferFormat(ImageFormat.YUV_420_888)
                 ...
                .build()

Now can get image.planes

The default image format is JPEG and you can access the buffer with

buffer: ByteBuffer = imageProxy.image.planes[0].getBuffer()

and if you plan on setting it to YUV

val image = imageProxy.image
val yBuffer = image.planes[0].buffer // Y
val uBuffer = image.planes[1].buffer // U
val vBuffer = image.planes[2].buffer // V

Access Image Data Buffer in CameraX

1 Answers1