How to force an Android camera to display in portrait mode, or for me to do the rotation instead

Question

I'm new to posting so please be nice! I have looked at practically all the other questions related to this and tried most, but am completely frustrated. I simply want to have my Android phone display the camera in portrait mode, not landscape. I am simply modifying the ImageManipulations sample that is provided with opencv.

I'm using opencv 3.01, Android sdk version 23, Android Studio 2.0, and a Nexus 5 phone with Android version 6.0.1. This is 4/28/2016 and this is pretty much the most current stable version of everything.

I have already forced the app into portrait mode in the AndroidManifest.xml file with:

              android:screenOrientation="portrait"

and the entire app is correctly displayed in portrait mode, but the camera image that is displayed is rotated 90 degrees. In other words, if you tilt your head to the left 90 degrees you will see the appropriate image. So I need to rotate the image to the right.

I tried using the opencv Tutorial3 code and setDisplayOrientation(90):

public void setResolution(Size resolution) {
    disconnectCamera();
    mMaxHeight = resolution.height;
    mMaxWidth = resolution.width;
    // mCamera.setDisplayOrientation(90);  //* Crashes if placed here
    connectCamera(getWidth(), getHeight());
    mCamera.setDisplayOrientation(90);  //* Doesn't crash if here, but doesn't work either
}

and that didn't work. And in any case this tutorial uses the "camera" class that was deprecated in Android version 21, and was replaced with the "camera2" class. I didn't go any farther on this because I wanted to use the most current APIs, but maybe something in camera2 could work?

Some people posted about using "transpose" and "flip" functions to rotate the image and I tried doing this many ways in the onCameraFrame method:

Original, works, but rotated:

    public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
    Mat rgba = inputFrame.rgba();
    ... do stuff to the image ...
    return rgba;

Image display goes dark, my FPS calculation goes to zero (weird)::

    public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
    Mat rgba = inputFrame.rgba();

    // Rotate clockwise 90 degrees
    Core.transpose(rgba, rgba);
    Core.flip(rgba, rgba, 1);
    ... do stuff to the image ...
    return rgba;

Rotated 180 degrees and video displays again, now flipped over, but still landscape mode:

    public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
    Mat rgba = inputFrame.rgba();

    // Rotate clockwise 90 degrees, then 90 degrees again
    Core.transpose(rgba, rgba);
    Core.flip(rgba, rgba, 1);
    Core.transpose(rgba, rgba);
    Core.flip(rgba, rgba, 1);
    ... do stuff to the image ...
    return rgba;

So it looks like if onCameraFrame returns a Mat with a different resolution from the original, the display goes blank. The documentation on onCameraFrame is sketchy - so, is this true?

Is there another place where I should be trying to do this image rotation before onCameraFrame is called? Is there another way that I can simply force the camera to provide the frame in portrait mode instead?

Answer 1

setDisplayOrientation works for the camera's standard preview outputs (A SurfaceView or TextureView directly connected to the camear), but OpenCV is in the middle here, and the way it receives its data is unaffected by setDisplayOrientation.

The transpose will change the dimensions of the image array, and unless the rest of the sample knows how to handle this, you'll have problems. You'll have to trace through the code to see where there are assumptions about buffer sizes (the output buffer drawn to the UI, for example) and update them accordingly to be the transposed size past the point where you're inserting the transpose.

Ideally, there'd be some way to set the orientation on the output View that OpenCV is drawing to, but I don't know enough about OpenCV's samples to say if such a control exists.