Google VR how to perform selection with daydream controller

Question

Following is my case

• I have a rectangle with left=0,right=1824,top=0,bottom=989 with aspect ratio 1.84f, with uniform z axis = -3f
• In that rectangle, I have small rectangles with width and height of 344 and 182. I arrange those small rectangles in the 3 rows and 3 columns similar to a grid gallery.
• I did some modification in the orthographic projection without changing the z axis such that the new rectangle will be in range -1,84 to 1.84 in x axis and 1 to -1 in y axis and all the small rectangles lie in those range

• Then to render those rectangles on VR, I simply applied

  view = perspective ( z near 0.1f, far 100 f) * eye.getEyeVew model = Orthographic transformation to come in range ( -1.84f to 1.84 f : X , 1 to 1f :Y ) modelview = view * model

• To emulate the controller, I used a simple circle with ( centerX=0, centerY=0, radius =0.04f, zaxis = -3f)

• Applied following transformation to the circle with controller rotation matrix (https://developers.google.com/vr/reference/android/com/google/vr/sdk/controller/Orientation.html#toRotationMatrix(float[]))

  view = perspective ( z near 0.1f, far 100 f) * eye.getEyeVew controllerMatrix = controller.toRotationMatrix modelView = view * controllerMatrix

My assumption was following,

• Since, I was drawing the 3D planes with uniform Z values, I can consider them as a 2D plane for selection.
• I assumed that the circle 2D coordinate ( left,top,right,bottom) can be used to check with the plane 2D coordinate ( left,top,right,bottom) and see whether the circle falls in the the plane or not.
• But due to the perspective & eye.getEyeView transformation, the z axis of both the circle and plane changes.

So, my question is for this purpose should I go for RayCasting or there is a simple solution for selecting the small rectangles with VR controller. I am using GVR controller ( Google VR version 1.80.0 ) and using Android for app development.

Thank you

Answer 1

The Google VR SDK comes with some useful example projects which demonstrate how to achieve this. If you inspect the Treasure Hunt project, the TreasureHuntActivity class has a method isLookingAtObject():

private boolean isLookingAtObject() {
    // Convert object space to camera space. Use the headView from onNewFrame.
    Matrix.multiplyMM(modelView, 0, headView, 0, modelCube, 0);
    Matrix.multiplyMV(tempPosition, 0, modelView, 0, POS_MATRIX_MULTIPLY_VEC, 0);

    float pitch = (float) Math.atan2(tempPosition[1], -tempPosition[2]);
    float yaw = (float) Math.atan2(tempPosition[0], -tempPosition[2]);

    return Math.abs(pitch) < PITCH_LIMIT && Math.abs(yaw) < YAW_LIMIT;
}

This will calculate the angle between the vector in the direction of the cameras line of sight and modelCube matrix. The modelCube matrix is a 4x4 matrix defining the scale, rotation and translation for the centre point of an object in the 3D scene. In your case this will be the centre point of the rectangle in 3D space.

You must also calculate the angles where the vector that defines the pointing direction of the camera intersects the rectangle. This is depicted in the following diagram:

Here the origin is your camera position, the point C is the centroid of the rectangle rendered in 3D space and the vector OC is the direction vector your camera is pointing in when it is focused directly on the centre of the rectangle. Ꝋ is the angle from the X-axis to the point C in the X-Z plane. Points P1-4 are are vertices of your rectangle in 3D space.

The pitch ɸ is defined in the following diagram:

Here ɸ is the angle from the centre point C to the new axis e.

The PITCH_LIMIT is the angle in the Y-e plane between two vectors. the vector from the camera position to the bottom edge of the bounding box of the object and the vector from the camera position to the upper edge of the bounding box of the object. In this case it can be calculated as either the angle between OP3 and OP4 or the angle between OP2 and OP1. This is depicted in the following 2D cross section:

Assume that the camera is looking directly at the rectangle, this limit defines how much the camera must be tilted up before the centre of the camera is no longer pointing directly at the rectangle.

The YAW_LIMIT is the angle in the X-Z plane between two vectors, the vector from the camera position to the left-most edge of the bounding box of the object, and the vector from the camera position to the right-most edge of the bounding box of the object. In this case it can be calculated as either the angle between OP1 and OP4 or the angle between OP2 and OP3. This is depicted in the following 2D cross section:

Likewise, this limit defines how much the camera can be rotated left or right before the centre of the camera is no longer pointing directly at the rectangle.

Note that the YAW_LIMIT and PITCH_LIMIT values get smaller if you are further away from the object and larger if you are close. You must calculate these limits for the object that you want to check is being looked at based on the current camera position and edge vertices of the object. The coordinate conversions to get from object space to camera space are handled in the isLookingAtObject() method.

Answer 2

First of all, one needs to look upon the above detailed description provided by sparkplug, which contains all the concepts related to the question. And I owe sparkplug for providing such a great concept related to this problem without whom, it would have been impossible to find the answer.

Moving on, if you have trouble understanding what atan2 does, then please follow this link. From trigonometry formula tan(angle) = perpendicular/base and if you have perpendicular and base values atan2 will give the angle. Basically atan2 will give the angle between two points. In our case point 1 being origin 0,0,0 and other point being the center point of rectangle or any vertices of the rectangle. So the difference between origin and point in rectangle will simply be the point values i.e. x-0 = x, y-0 = y, z-0 = z. Further, in our case we are calculating angle with reference to z axis i.e our base.

If you are having some difficulty in understanding the concept of YAW and PITCH, I will make an analogy to head movement. Make your head straight and look in the forward direction, then move your head left and then move your head right. This is YAW i.e. the angle your head is moves in the left and right direction. Again make your head straight and look in the forward direction, then move your head up and down. This is PITCH i.e. by how much angle your head moves up and down.

Making an analogy to head movement, the point between your neck and head is the pivot point. Moving your head upwards and downwards, your eyes will be scanning in the PITCH direction. Likewise, taking the same pivot point, moving your head from left to right, your eyes will be scanning in the YAW direction.

In this solution, we need to calculate the angle in which our head is to be moved to scan the whole rectangle in width and in height.

Moving on, take the above analogy and study the daydream controller sample. You need to study the YAW and PITCH value it gives. To make study simple, put the controller flat in the table with touchpad away from you. Then just move slightly the controller in left and right direction in the manner you spin a bottle in the table. You will notice the range of YAW values. Then gently pick up the controller and just move up and down. This will give the range of PITCH values of the controller. For simplicity, our controller will be set as the origin, or to make the problem even simpler, you can consider the controller as your head situated in the origin and we are observing the values the controller emits when doing YAWing and PITCHing.

Try to put everything in normalized coordinates before calculating the YAW and PITCH values. Let us consider a rectangle with following coordinates

Vertices:  X     Y     Z
      P1: 0.52   0.76  -2.40  //LeftTop
      P2: 0.52   0.21  -2.40  //LeftBottom
      P3: 1.54   0.21  -2.40  //RightBottom
      P4: 1.54   0.76  -2.40  //Right Top
Formula:  
    YAW: Math.atan2(x,-z)
  PITCH: Math.atan2(y,-z)
//Then yaw, pitch values for above vertices will be
         YAW   PITCH
     P1: 0.21  0.31  //LeftTop
     P2: 0.21  0.09  //LeftBottom
     P3: 0.57  0.09  //RightBottom
//We don't require P4 here
      YAW Left: 0.21 ( P2 or P1 YAW)
      YAW Right: 0.57 ( P3 YAW )
    PITCH Top: 0.31 ( P1 PITCH )
    PITCH Bottom: 0.09  ( P2 or P3 PITCH )
// let us find the pitch and yaw values of controllers as well
   angles = controller.orientation.toYawPitchRollRadians
   angles[0]: YAW
   angles[1]: PITCH
   angles[2]: ROLL
 //of controller
 // controller YAW moves positive to negative from left to right, but in 
 // normalized coordinates, negative to positive from left to right, so to 
 // match controller's YAW values
      YAW Left: 0.21 * -1 ( P2 or P1 YAW)
      YAW Right: 0.57 * -1 ( P3 YAW )
    PITCH Top: 0.31 ( P1 PITCH )
    PITCH Bottom: 0.09  ( P2 or P3 PITCH )
 // and to know whether the controller is at the rectangle or not we simply do
    public boolean isOnRectangle(float[] angles) {
    return angles[0] <= yawLeft && angles[0] >= yawRight
            && angles[1] <= pitchTop && angles[1] >= pitchBottom;
}

The above will return true if the controller pointer falls in the rectangle and false otherwise.