1

I recently tried to develop a flutter plugin with cameraX, but I found that there was no way to simply bind Preview to flutter's Texture.

In the past, I only needed use camera.setPreviewTexture(surfaceTexture.surfaceTexture()) to bind camera and texture, now I can't find the api.

camera.setPreviewTexture(surfaceTexture.surfaceTexture())

        val previewConfig = PreviewConfig.Builder().apply {
            setTargetAspectRatio(Rational(1, 1))
            setTargetResolution(Size(640, 640))
        }.build()

        // Build the viewfinder use case
        val preview = Preview(previewConfig).also{

        }

        preview.setOnPreviewOutputUpdateListener {
//            it.surfaceTexture = this.surfaceTexture.surfaceTexture()
        }

   // how to bind the CameraX Preview surfaceTexture and flutter surfaceTexture?
CaiJingLong
  • 129
  • 3
  • 9
  • preview.setOnPreviewOutputUpdateListener Doesn't exist after Aplha6. So no idea how you could bind to a SurfaceTexture now. This is unfortunate since there have been a lot of improvements if you go to Aplha10. – William Steele Feb 13 '20 at 20:24

2 Answers2

4

I think you can bind texture by Preview.SurfaceProvider.

            final CameraSelector cameraSelector = new CameraSelector.Builder().requireLensFacing(CameraSelector.LENS_FACING_BACK).build();
            final ListenableFuture<ProcessCameraProvider> listenableFuture = ProcessCameraProvider.getInstance(appCompatActivity.getBaseContext());
            listenableFuture.addListener(() -> {
                try {
                    ProcessCameraProvider cameraProvider = listenableFuture.get();
                    Preview preview = new Preview.Builder()
                            .setTargetResolution(new Size(720, 1280))
                            .build();
                    cameraProvider.unbindAll();
                    Camera camera = cameraProvider.bindToLifecycle(appCompatActivity, cameraSelector, preview);
                    Preview.SurfaceProvider surfaceProvider = request -> {
                        Size resolution = request.getResolution();
                        surfaceTexture.setDefaultBufferSize(resolution.getWidth(), resolution.getHeight());
                        Surface surface = new Surface(surfaceTexture);
                        request.provideSurface(surface, ContextCompat.getMainExecutor(appCompatActivity.getBaseContext()), result -> {

                        });
                    };
                    preview.setSurfaceProvider(surfaceProvider);
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }, ContextCompat.getMainExecutor(appCompatActivity.getBaseContext()));
Chao Yi
  • 171
  • 1
  • 4
2

Update: CameraX has added functionality which will now allow this since this answer was written, but this might still be useful to someone. See this answer for details.

It seems as though using CameraX is difficult to impossible due to it abstracting the more complicated things away and so not exposing things you need like being able to pass in your own SurfaceTexture (which is normally created by Flutter).

So the simple answer is that you can't use CameraX.

That being said, with some work you may be able to get this to work, but I have no idea if it will work for sure. It's ugly and hacky so I wouldn't recommend it. YMMV.

If we're going to do this, let's first look at how the flutter view creates a texture

    @Override
    public TextureRegistry.SurfaceTextureEntry createSurfaceTexture() {
        final SurfaceTexture surfaceTexture = new SurfaceTexture(0);
        surfaceTexture.detachFromGLContext();
        final SurfaceTextureRegistryEntry entry = new SurfaceTextureRegistryEntry(nextTextureId.getAndIncrement(),
                surfaceTexture);
        mNativeView.getFlutterJNI().registerTexture(entry.id(), surfaceTexture);
        return entry;
    }

Most of that is replicable, so we may be able to do it with the surface texture the camera gives us.

You can get ahold of the texture the camera creates this way:

preview.setOnPreviewOutputUpdateListener { previewOutput ->
    SurfaceTexture texture = previewOutput.surfaceTexture
}

What you're going to have to do now is to pass a reference to your FlutterView into your plugin (I'll leave that for you to figure out). Then call flutterView.getFlutterNativeView() to get ahold of the FlutterNativeView.

Unfortunately, FlutterNativeView's getFlutterJni is package private. So this is where it gets really hacky - you can create a class in that same package that calls that package-private method in a publicly accesible method. It's super ugly, and you may have to fiddle around with Gradle to get the compilation security settings to allow it, but it should be possible.

After that, it should be simple enough to create a SurfaceTextureRegistryEntry and to register the texture with the flutter jni. I don't think you want to detach from the opengl context, and I really have no idea if this will actually work. But if you want to try it out and report back what you find I would be interested in hearing the result!

rmtmckenzie
  • 37,718
  • 9
  • 112
  • 99
  • Although it has not been authenticated in practice, this is a way of thinking. But maybe it's just to use CameraX to do something that shouldn't be done. There's also a security risk due to Flutter. So I think I'll give up the idea of combining CameraX and flutter for a while before CameraX can be better implemented. In addition, writing a lot of code in order to use CameraX may lose the meaning of using CameraX. – CaiJingLong May 18 '19 at 05:21
  • The answer lower down using Preview.SurfaceProvider works and isn't hackish. – David Oldford Jul 04 '20 at 18:01
  • @DavidOldford Glad to hear CameraX was updated with that - I'm pretty sure it didn't exist when I first wrote my answer. I don't want to copy another valid answer but I'll link to it. – rmtmckenzie Jul 05 '20 at 06:37
  • I figured it was something like that. Now hopefully flutter will just add a robust and useful camera widget so people don't have to do any of this :P – David Oldford Jul 06 '20 at 13:21
  • @DavidOldford out of curiosity, what's wrong with the camera package they provide? – rmtmckenzie Jul 06 '20 at 17:10