Using a raycaster with the camera to create the crosshair Google Cardboard effect

Question

An example of what I'm trying to achieve: https://workshop.chromeexperiments.com/examples/guiVR/#1--Basic-Usage

How could I get the Google Cardboard crosshair, (gaze)pointer, reticle, whatever you want to call it effect in three.js? I would like to make a dot, as crosshair, in the center of the screen in my scene. Like that I would like to use a raycaster to identify what I'm looking at in VR. Which way would be best to go here?

Do I fake an X and Y position of my mouse? Because I found other people have answered how to cover this by adding an event listener to the mousemove event. But this works on desktop, and I want to bring this to mobile.

Answer 1

This library works like a charm - https://github.com/skezo/Reticulum

Answer 2

Here are the main parts needed for building your own gaze cursor:

1. An object that serves as the indicator (cursor) for user feedback
2. An array of objects that you want the cursor to interact with
3. A loop to iterate over the array of interactive objects to test if the cursor is pointing at them

Here's an example of how this can be implemented.

Gaze cursor indicator

Using a ring here so it can be animated, as you typically want to give the user some feedback that the cursor is about to select something, instead of instantly triggering the interaction.

const cursor = new THREE.Mesh(
  new THREE.RingBufferGeometry(0.1, 0.15),
  new THREE.MeshBasicMaterial({ color: "white" })
);

Interactive Objects

Keep track of the objects you want to make interactive, and the actions they should execute when they are looked at.

const selectable = [];

const cube = new THREE.Mesh(
  new THREE.BoxBufferGeometry(1, 1, 1),
  new THREE.MeshNormalMaterial()
);

selectable.push({
  object: cube,
  action() {
    console.log("Cube selected");
  },
});

Checking Interactive Objects

Check for interactions on every frame and execute the action.

const raycaster = new THREE.Raycaster();

(function animate() {
  for (let i = 0, length = selectable.length; i < length; i++) {
    const camPosition = camera.position.clone();
    const objectPosition = selectable[i].object.position.clone();

    raycaster.set(camPosition, camera.getWorldDirection(objectPosition));

    const intersects = raycaster.intersectObject(selectable[i].object);
    const selected = intersects.length > 0;

    // Visual feedback to inform the user they have selected an object
    cursor.material.color.set(selected ? "crimson" : "white");
    
    // Execute object action once
    if (selected && !selectable[i].selected) {
      selectable[i].action();
    }
    selectable[i].selected = selected;
  }
})();

Here's a demo of this in action:

* {
  box-sizing: border-box;
  margin: 0;
  padding: 0;
}

canvas {
  display: block;
}

<script type="module">
  import * as THREE from "https://cdn.jsdelivr.net/npm/three@0.121.1/build/three.module.js";
  import { OrbitControls } from "https://cdn.jsdelivr.net/npm/three@0.121.1/examples/jsm/controls/OrbitControls.js";

  const renderer = new THREE.WebGLRenderer();
  renderer.setSize(window.innerWidth, window.innerHeight);
  document.body.appendChild(renderer.domElement);

  const cameraMin = 0.0001;

  const aspect = window.innerWidth / window.innerHeight;
  const camera = new THREE.PerspectiveCamera(75, aspect, cameraMin, 1000);
  const controls = new OrbitControls(camera, renderer.domElement);

  const scene = new THREE.Scene();

  camera.position.z = 5;

  scene.add(camera);

  const cube = new THREE.Mesh(
    new THREE.BoxBufferGeometry(),
    new THREE.MeshNormalMaterial()
  );

  cube.position.x = 1;
  cube.position.y = 0.5;

  scene.add(cube);

  const cursorSize = 1;
  const cursorThickness = 1.5;
  const cursorGeometry = new THREE.RingBufferGeometry(
    cursorSize * cameraMin,
    cursorSize * cameraMin * cursorThickness,
    32,
    0,
    Math.PI * 0.5,
    Math.PI * 2
  );
  const cursorMaterial = new THREE.MeshBasicMaterial({ color: "white" });
  const cursor = new THREE.Mesh(cursorGeometry, cursorMaterial);

  cursor.position.z = -cameraMin * 50;

  camera.add(cursor);

  const selectable = [
    {
      selected: false,
      object: cube,
      action() {
        console.log("Cube selected");
      },
    }
  ];
  const raycaster = new THREE.Raycaster();

  let firstRun = true;
  (function animate() {
    requestAnimationFrame(animate);

    renderer.setSize(window.innerWidth, window.innerHeight);
    camera.aspect = window.innerWidth / window.innerHeight;
    camera.updateProjectionMatrix();

    controls.update();

    if (!firstRun) {
      for (let i = 0, length = selectable.length; i < length; i++) {
        const camPosition = camera.position.clone();
        const objectPosition = selectable[i].object.position.clone();
        raycaster.set(camPosition, camera.getWorldDirection(objectPosition));

        const intersects = raycaster.intersectObject(selectable[i].object);

        const selected = intersects.length > 0;

        cursor.material.color.set(selected ? "crimson" : "white");

        if (selected && !selectable[i].selected) {
          selectable[i].action();
        }
        selectable[i].selected = selected;
      }
    }

    renderer.render(scene, camera);
    firstRun = false;
  })();
</script>