I'm trying to achieve some interaction with 3d models in an a-frame AR-scene when using WebXR and dom-overlay.
So I managed to implement hit-testing and can create and place 3d-objects on detected planes. The dom-overlay works as well with adding normal interactable html/css buttons.
But what does not work correctly for me is the interaction with the created 3d-objects. I tried to activate a raycast going from the position of the camera to a translated 2D-point when touching the mobile screen.
I create a little example with merging different examples together (hit-test, dom-overlay, calculation of raycast-direction): https://glitch.com/edit/#!/famous-outstanding-brother
So, first you need to detect a plane, a reticle will appear and you can place little boxes. When you toggle the switch in the lower left corner, the hit-testing is turned off and you should "touch" the created boxes so that they change their color.
To use the a-frame raycaster I listen to touch-events on the "overlay" (because other events don't fire and I think this is the only way to get a x/y-coordinate of the touchpoint) and set the direction myself. In the center of the screen it does work pretty well.
GIF: Hit-Test and raycast from touchpoint of the center of the screen works well
But the closer you get to the edges of the screen with your touches, the less precise it becomes. And if the 3d-objects are a little bit further away, the correct touchpoint to change the color of the boxes is displaced (you need to touch closer to the center point). So I guess the calculation of the raycast-direction is a little bit wrong?!?
GIF: The direction of the raycast from the edges of the screen is off
component.camera = component.el.sceneEl.camera;
component.camera.parent.updateMatrixWorld();
const screenPosition = {x: ev.touches[0].screenX, y: ev.touches[0].screenY};
const origin = new THREE.Vector3();
const direction = new THREE.Vector3();
const raycasterConfig = {origin: origin, direction: direction};
origin.setFromMatrixPosition(component.camera.matrixWorld);
const normalizedScreenPosition = {x: screenPosition.x/document.body.clientWidth * 2 -1, y: -(screenPosition.y/document.body.clientHeight) * 2 + 1};
const matrix = new THREE.Matrix4();
const matrixInverse = matrix.copy(component.camera.projectionMatrix).invert();
direction.set(normalizedScreenPosition.x, normalizedScreenPosition.y, 0.5).applyMatrix4(matrixInverse).applyMatrix4(component.camera.matrixWorld).sub(origin).normalize();
component.el.setAttribute("raycaster", raycasterConfig);
I searched through the web but couldn't find an example where it actually worked with these settings. And I'm just asking myself, if I'm just blind and there is an easy way or if there is really something which can't be solved (for the moment).
So any help would be really appreciated!
