How can I decide what objects should be detected and what should be ignored by raycast and how?

Question

I have two main problems in this case :

1. NAVI is child of the player (FPSController) and therefore the NAVI is moving along with the player. But NAVI is also position a bit in front of the player and the result is that the raycast is hitting first time all the time the NAVI instead hitting the Interactable object.

2. If there is a door between the player and some Interactable object it will detect the object. But if there is a door and other objects between the player and the interactable object/s it should not detect them. The player not see them but yet it's detecting.

This script is attached to the plaher (FPSController) :

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.UI;

public class DetectInteractable : UnityEngine.MonoBehaviour
{
    public Camera cam;
    public float distanceToSee;
    public string objectHit;
    public bool interactableObject = false;
    public Transform parentToSearch;
    public Scaling scaling;
    public int spinX = 0;
    public int spinY = 0;
    public int spinZ = 0;
    public GameObject navi;
    public GameObject itemsDescriptionCanvas;
    public Text itemsDescriptionText;

    private RaycastHit whatObjectHit;
    private Transform[] childrenToSearhc;

    private void Start()
    {
        childrenToSearhc = parentToSearch.GetComponentsInChildren<Transform>();
    }

    private void Update()
    {
        if (cam.enabled == true)
        {
            if (Input.GetMouseButtonDown(0) && !scaling.scaleUp)
            {
                if (whatObjectHit.collider != null)
                    ExecuteActions(whatObjectHit.collider.gameObject);
            }

            Debug.DrawRay(cam.transform.position, cam.transform.forward * distanceToSee, Color.magenta);
            if (Physics.Raycast(cam.transform.position, cam.transform.forward, out whatObjectHit, distanceToSee))
            {
                objectHit = whatObjectHit.collider.gameObject.name;
                interactableObject = true;
                print("Hit ! " + whatObjectHit.collider.gameObject.name);

                if (scaling.objectToScale.transform.localScale == scaling.minSize)
                {
                    scaling.objectToScale.transform.Rotate(spinX, spinY, spinZ);
                }
                ProcessItemsDescripations();
                itemsDescriptionCanvas.SetActive(true);
            }
            else
            {
                if (scaling.objectToScale.transform.localScale == scaling.minSize)
                {
                    navi.transform.rotation = new Quaternion(0, 0, 0, 0);
                }
                itemsDescriptionCanvas.SetActive(false);
                print("Not Hit !");
            }
        }
    }

    private void ExecuteActions(GameObject go)
    {
        var ia = go.GetComponent<ItemAction>();
        if (ia != null)
        {
            ia.ItemMove();
        }
    }

    void ProcessItemsDescripations()
    {
        foreach (Transform child in childrenToSearhc)
        {
            if (child.GetComponent<ItemInformation>() != null)
            {
                ItemInformation iteminformation = child.GetComponent<ItemInformation>();
                if (child.name == objectHit)
                {
                    itemsDescriptionText.text = iteminformation.description;
                }
            }
        }
    }

    public class ViewableObject : UnityEngine.MonoBehaviour
    {
        public string displayText;
        public bool isInteractable;
    }
}

And this small script is attach to each Interactable object :

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class ItemInformation : UnityEngine.MonoBehaviour
{
    [TextArea]
    public string description;
}

This is a screenshot of the player (FPSController) Inspector settings :

This screenshot is of the NAVI that is child of the player : You can see the NAVI at the left game window and scene window :

This is a screenshot example of the window. The window should be Interactable object and should be detected by the raycast :

The window Tag is set to Untagged and the Layer to Default

This is a screenshot while the game is working a situation when the player is standing behind a door in the other side of the door there is a room with the Interactable window. Nothing will prevent from the raycast to pas through the door and detect the window. And that's not logic. The player can't see the window. So the window should not be detected :

On the left the top the magenta color of the raycast pass through the door to the window. On the bottom left the player is standing behind the door and see the window at all but it's detecting the window :

But when using a breakpoint you can see that what was really hit is the NAVI not the door not the window but the NAVI since the NAVI is always standing in front of the player :

Before all that I tried to use raycast mask layer in my script and then changed the window layer to be Interactable and then I saw it's detecting through the door.

At the end the main goal is :

To detect Interactable objects in the game but not to detect the NAVI he is not Interactable object so somehow to make that NAVI will not block the raycast !

And to be able to detect bject only when there is a logic view of the player to the objects. And not to detect objects from behind doors or walls or other objects.

Answer 1

You can use the additional optional parameter layerMask of Physics.Raycast

A Layer mask that is used to selectively ignore Colliders when casting a ray.

I would recommned to use a LayerMask

public LayerMask targetLayer;

configure the wanted layers using the Unity Inspector (select every Layer you want to hit) and finally pass it to the raycast like

if(Physics.Raycast(
    cam.transform.position, 
    cam.transform.forward, 
    out whatObjectHit, 
    distanceToSee, 
    targetLayer.value))
{
    ...
}

---

In the example Unity used a bitmask directly (btw. in a very unefficient way in Update):

// Bit shift the index of the layer (8) to get a bit mask
int layerMask = 1 << 8;

// This would cast rays only against colliders in layer 8.
// But instead we want to collide against everything except layer 8. The ~ operator does this, it inverts a bitmask.
layerMask = ~layerMask;

as said I didn't like this way of hardcoding it. Additionally you now only exclude the layer 8 but hit everything else.

To rather include only exactly one specific layer remove the line

layerMask = ~layerMask;

so you only hit layer 8.

Anyway using my way from before is way more flexible and you rather select those layers you actually want to hit instead of excluding one. And using the LayerMask is way easier than having to code some bitmask operations in order to construct the layermask you want.