3

So here's the problem overview: I render a number of models using OpenGL ES in GvrView using the GvrView.StereoRenderer and I want to determine which exact model I'm looking at.

My idea is to reproject the screen coordinates back to the model space (discarding Z) and check if the point (lets call it Point) is in the range: (ModelsMinX < Point.x < ModelsMaxX) and (ModelsMinY < Point.y < ModelsMaxY).

I was trying to use GLU.gluUnproject to get the initial coordinates. This function requires the current viewport and that's where the problems begin:

GvrView.StereoRenderer has a method .onDrawEye, which is called whenever there's something specific to one eye that should be setup before rendering (aka the view and the projection matrices should be acquired from the Eye instance). An Eye also has a method .getViewport which is supposed to return a viewport for the current eye, however the returned result is completely clear to me. More specifically, I'm developing on Nexus 6 (1440x2560 pixels) and .getViewport returns:

x = 0, y 0, width = 1280, height = 1440 // for the first eye

x = 1280, y 0, width = 1280, height = 1440 // for the second eye.

Now this is interesting.. Somehow I assumed two things about the current viewport:

  1. width = 1440, height = 1280 (we are in the landscape mode after all);

  2. the viewport size for each eye will be half the size of the whole viewport.

Hence, calling .gluUnproject on the middle point of the viewport:

GLU.gluUnProject(viewport.width / 2, viewport.height / 2, 0, mEyeViewMatrix, 0, mEyeProjectionMatrix, 0, new int[] {viewport.x, viewport.y, viewport.width, viewport.height}, 0, center, 0);

does not yield expected results, in fact, it gives me all 0s. I found this question (Determining exact eye view size), but the guy gets even stranger viewport values and it doesn't contain an answer..

So the question is - how to I get from the 'eye'-space coordinates to the model? And what those coordinates even are? Here's the project's github for reference: https://github.com/bowlingforsoap/CardboardDataVisualizationJava.

Some other approaches I'm aware of:

  1. In the treasurehunt demo they use an opposite way of doing things - they go from a model coordinate (0, 0, 0, 1) into the head view space using the HeadTransform to get the headView matrix (look for method .isLookingAtObject in https://github.com/googlevr/gvr-android-sdk/blob/master/samples/sdk-treasurehunt/src/main/java/com/google/vr/sdk/samples/treasurehunt/TreasureHuntActivity.java).

  2. Using raycasting. I'm not sure this is going to help my cause, because after I get an observed object I would like to create an 'floating' activity which is going to contain information about it (I certainly don't wanna render that data through shaders).

Wheh! Boy, that's a lot of text. But yeah, it seems like a generic problem, yet I haven't found an easy/elegant/working solution to that.. Would appreciate any feedback.

Community
  • 1
  • 1
bowlingforsoap
  • 61
  • 6
  • **Quick update**: zero values were a result of me not noticing an exception while in Android Monitor mode. Now I get some values. I haven't checked them yet, so I'll come back with yet another update, which maybe will shine some light on the viewport problem, if there's one. – bowlingforsoap Jun 22 '16 at 15:45
  • Same question. If you finally find how, please, post it here :) – Gonzalo Solera Jun 23 '16 at 11:20
  • What I did once (this is not restricted to vr though) is having two rendering stages, the first rendering all models in an unique color, then getting the pixel color under the cursor, looking up the color and see if it matches any objects color --> hit, second stage: clear color buffer and render models normally. Just be careful about your color depth, this can affect the colors you get back by glReadPixels(). – ByteBit Jun 27 '16 at 15:21
  • That is an interesting approach, ByteBit) As an **update №2**, I ended up with an approach similar to what they do in the TreasureHunt. In my use case, I have a fixed number of models, which are displaced by a certain angle that depends of that number. So what I do is I determine the look at vector and then compute the dot product with the (0, 0, -1), this gives me the angle the head is rotated, which I use to select the model user is looking at currently. – bowlingforsoap Jun 28 '16 at 16:46

0 Answers0