Switch between different ML models that run on camera in kotlin on runtime

Question

I am trying to make an app that allow user to switch between different ML-Models on fly. Right now i have a simple app that can run camera with no model and camera with an object detection model. There will be five such models in final app.

This is my Home Activity

class HomeActivity : AppCompatActivity(), EasyPermissions.PermissionCallbacks {
private val rqSpeechRec = 102
private var tts: TextToSpeech? = null
private lateinit var binding: ActivityHomeBinding
private lateinit var objectDetector: ObjectDetector
private lateinit var cameraProviderFuture: ListenableFuture<ProcessCameraProvider>

override fun onCreate(savedInstanceState: Bundle?) {
    super.onCreate(savedInstanceState)

    // Binding ViewData
    binding = ActivityHomeBinding.inflate(layoutInflater)
    setContentView(binding.root)

    supportActionBar?.hide() // Hiding App bar

    requestCameraPermission()   // Requesting Camera Permission
}

This is my Request Camera Permission function.

private fun requestCameraPermission() {
    speakOut("Allow Eynetic to access the camera to take photos or videos.")

    EasyPermissions.requestPermissions(
        this,
        "This app can not work without camera.",
        Constants.PERMISSION_CAMERA_REQUEST_CODE,
        permission.CAMERA
    )
}

override fun onRequestPermissionsResult(
    requestCode: Int,
    permissions: Array<out String>,
    grantResults: IntArray
) {
    super.onRequestPermissionsResult(requestCode, permissions, grantResults)

    EasyPermissions.onRequestPermissionsResult(requestCode, permissions, grantResults, this)
}

override fun onPermissionsDenied(requestCode: Int, perms: List<String>) {
    speakOut("Permissions Denied.")

    if (EasyPermissions.somePermissionDenied(this, perms.first()))
        SettingsDialog.Builder(this).build()
            .show()  // If permissions are permanently denied show settings.
    else 
        requestCameraPermission()
}

override fun onPermissionsGranted(requestCode: Int, perms: List<String>) {
    cameraProviderFuture = ProcessCameraProvider.getInstance(this)
    cameraProviderFuture.addListener({
        startCamera(cameraProviderFuture.get())
    }, ContextCompat.getMainExecutor(this))

    // Introduction
    val intro =
        "Welcome to Eyenetic. Please activate any module through voice command.The nodules are obstacle detection, scene recognition, currency detection, human object interaction and human human interaction."
    speakOut(intro)

    allButtons()
}

I only addListener to cameraProviderFuture when permission is granted. And when permission is granted i start the camera with no model running. Note when app is open each time no model will be running.

@SuppressLint("UnsafeOptInUsageError")
private fun startCamera(cameraProvider: ProcessCameraProvider) {
    val preview = Preview.Builder().build()

    val cameraSelector =
        CameraSelector.Builder().requireLensFacing(CameraSelector.LENS_FACING_BACK).build()

    preview.setSurfaceProvider(binding.previewView.surfaceProvider)

    cameraProvider.bindToLifecycle(this as LifecycleOwner, cameraSelector, preview)

}

And code for fetching Object-Detection model.

private fun readyObjectDetectionModel()
{
    val localModel = LocalModel.Builder().setAbsoluteFilePath("object_detection.tflite").build()
    val customObjectDetectionOptions = CustomObjectDetectorOptions.Builder(localModel)
        .setDetectorMode(CustomObjectDetectorOptions.STREAM_MODE)
        .enableClassification()
        .setClassificationConfidenceThreshold(0.5f)
        .build()

    objectDetector = ObjectDetection.getClient(customObjectDetectionOptions)
}

Code for Object-Detection

    @SuppressLint("UnsafeOptInUsageError")
private fun startObjectDetection(cameraProvider: ProcessCameraProvider) {
    val preview = Preview.Builder().build()

    val cameraSelector =
        CameraSelector.Builder().requireLensFacing(CameraSelector.LENS_FACING_BACK).build()

    preview.setSurfaceProvider(binding.previewView.surfaceProvider)
    
    val imageAnalysis = ImageAnalysis.Builder().setTargetResolution(Size(1280, 720))
        .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST).build()

    imageAnalysis.setAnalyzer(ContextCompat.getMainExecutor(this),
        {imageProxy ->
            val rotationDegrees =imageProxy.imageInfo.rotationDegrees
            val image = imageProxy.image

            if (image != null) {
                val processImage = InputImage.fromMediaImage(image, rotationDegrees)

                objectDetector.process(processImage)
                    .addOnSuccessListener {
                        imageProxy.close()
                    }.addOnFailureListener{
                        imageProxy.close()
                    }
            }
    })
    cameraProvider.bindToLifecycle(this as LifecycleOwner, cameraSelector, imageAnalysis, preview)
}

I can run both codes for camera by calling each one at a time and commenting the other. But i have made different buttons to trigger different models. What should i do before calling each method.

Right now the only way to i can think on is unbind the previous Life-cycle and make a new one. But what about the redundant code like

val preview = Preview.Builder().build()

val cameraSelector =
    CameraSelector.Builder().requireLensFacing(CameraSelector.LENS_FACING_BACK).build()

preview.setSurfaceProvider(binding.previewView.surfaceProvider)

Is their way to only change the usecase on current bindedlifecycle?

Should i make camerafragment and switch between different fragments?

Any advice for better approaching this problem.

score 0 · Answer 1 · answered Dec 09 '21 at 00:22

One idea come to my mind is to create an ImageAnalysis.Analyzer subclass that can handle different ML features (in your word - ML models).

In that case, you only setup the imageAnalysis use case once, and the camera will keep feeding images to your Analyzer.

In your Analyzer, you can design some APIs like switchTo(MLFeatureName name). When your UI button is pressed, you can call the switchTo method. After such a switch, the images will be fed to a different detector.

Make sure to close unused detector to release system resources.

Switch between different ML models that run on camera in kotlin on runtime

1 Answers1