How to properly implement camera2 realtime frame processing using RxJava?

Question

I'm making reactive wrapper over camera2, my goal is to get each frame and then pass to face recognition.

So, I created a wrapper method over setOnImageAvailableListener

  fun createOnImageAvailableFlowable(imageReader: ImageReader, handler: Handler): Flowable<ImageReader> {
        return Flowable.create({ subscriber ->
            imageReader.setOnImageAvailableListener({
                if (!subscriber.isCancelled)
                    subscriber.onNext(it)
            }, handler)

            subscriber.setCancellable {
                imageReader.setOnImageAvailableListener(null, null)
            }
        }, BackpressureStrategy.LATEST)
    }

Reactive chain looks as follows:

 createOnImageAvailableFlowable(imageReader!!, null)
 .concatMap {
     it.acquireLatestImage()?.use { image ->
        val rotation = ReactiveCamera.getRotationCompensation(cameraId!!, this, applicationContext)
        val visionImage = FirebaseVisionImage.fromMediaImage(image, rotation)
        firebaseFaceDetector
          .detectInImage(visionImage)
          .toFlowable(BackpressureStrategy.LATEST)
          .map { list ->Optional(list)}
     } ?: Flowable.just(Optional(null))
 }
 ...

This code works, but cause some lags on preview surface because all work performed in the main thread. This needs to be performed in separate thread. My naive solution is to add observeOn operator before concatMap:

createOnImageAvailableFlowable(imageReader!!, null)
.observeOn(Schedulers.io()) // doesn't switch thread
.concatMap {
 // still main thread
}
...

But it doesn't affect, all work still in the main thread. If I specify concatMapEager instead of concatMap, all works as expected in separate thread, but the frames comes with a significant delay.

What I'm doing wrong? How can I instruct the reactive stream to be performed in a separate thread in this case? How can backpressure be handled in case of realtime frame processing?

Upd

I provided my own thread as Kiskae suggested, but now, only first emission happens in scheduler's thread, but the rest emissions remain in the main thread:

createOnImageAvailableFlowable(imageReader!!, null)
.subscribeOn(AndroidSchedulers.from(nonMainThread.looper))
.concatMap {
   val t = Thread.currentThread()
   val name = t.name
   Log.d(TAG, "current thread {$name}")
 ...
}

Output:

D/MainActivity: current thread {Camera2}
D/MainActivity: current thread {main}
D/MainActivity: current thread {main}
D/MainActivity: current thread {main}
D/MainActivity: current thread {main}

score 0 · Answer 1 · answered Nov 06 '18 at 16:46

0

Looking at the documentation of ImageReader.setOnImageAvailableListener:

Handler: The handler on which the listener should be invoked, or null if the listener should be invoked on the calling thread's looper.

Since you're subscribing on the main looper it ends up setting up the callback using the main looper, this causes all the processing before the concatMap to always occur on the application thread.

You can solve this by either providing a handler instead of null or calling subscribeOn and providing a handler-based scheduler like RxAndroid's HandlerScheduler.

answered Nov 06 '18 at 16:46

Kiskae

24,655
2
77
74

The problem remains, the code inside concatMap executed only once in provided thread, and then switching to main. Please take a look at updated question – trueangle Nov 07 '18 at 18:38
That likely means the default `Looper` is tied to the `ImageReader` instance. You will need to replace the second parameter of the `createOnImageAvailableFlowable` call with an appropriate handler to be absolutely sure the listener is invoked on that handler. – Kiskae Nov 07 '18 at 19:32
Thanks fo your reply, the problem here in my method firebaseFaceDetector.detectInImage(visionImage), it is reactive wrapper over firebase's detectInImage(visionImage).addOnSuccessListener{...}. This method doing all computational work in different thread and then invokes addOnSuccessListener on the main thread automatically. That's why concatMap gets invoked in the main thread. Do you have any suggestions about it? – trueangle Nov 09 '18 at 17:14

How to properly implement camera2 realtime frame processing using RxJava?

1 Answers1