Unity Physics.Raycast with LayerMask does not detect object on layer. Used bitshifting, tried inverting layer, still nothing works

Question

novice to intermediate Unity developer here. I've been hitting a pretty significant roadblock the past ~2 days concerning the raycast detection of objects with specific layers. I've been researching this issue quite a lot, and all the solutions I've found don't seem to reflect the strange issue I'm facing.

Basically, the problem follows this sequence of events:

1. My player character has a vision cone shaped trigger mesh called 'InSightBox' which detects all objects with the tag 'Mob' and adds them to a List of colliders called 'MobsInRange'.

    public List<Collider> mobsInRange;

    public List<Collider> GetColliders()
    {
        return mobsInRange;
    }
    // Start is called before the first frame update
    void Start()
    {
        mobsInRange = new List<Collider>();
    }

    // Update is called once per frame
    void Update()
    {
        
    }
    //add enemy with tag 'mob' to list
    private void OnTriggerEnter(Collider other)
    {
        if(!mobsInRange.Contains(other) && other.tag == "Mob")
        {
            mobsInRange.Add(other);
        }
    }
    //remove enemy with tag 'mob' to list
    private void OnTriggerExit(Collider other)
    {
        if (mobsInRange.Contains(other) && other.tag == "Mob")
        {
            mobsInRange.Remove(other);
        }
    }

2. This list is then fed up to the root/parent player game object containing everything relating to the player.

    public Transform closestMob;
    public List<Collider> mobs;
    public Transform GetClosestEnemy()
    {
        Transform tMin = null;
        float minDist = Mathf.Infinity;
        Vector3 currentPos = transform.position;
        foreach(Collider trans in mobs)
        {
            //find enemy with closest distance and set tMin to it. Method returns tMin
            float dist = Vector3.Distance(trans.transform.position, currentPos);
            if(dist < minDist)
            {
                tMin = trans.transform;
                minDist = dist;
                
            }
        }

        //Debug.Log(tMin);
        return tMin;
        
    }

3. The player then uses a 'Look at' method to find the closest of all 'mobs' to the player. The player will set their forward transform to look at the closest mob.

Problem Step ---> 4) When the player raises their gun and attempts to shoot the closest enemy, a ray is cast with a layermask that looks only for objects on the layer 'Enemy', the 8th layer. When the ray detects the enemy, the enemy script should fire its 'TakeDamage' method which decreases the 'curHealth' variable by 8. Only problem is, the cast doesn't seem to detect the enemy object on the 'Enemy' layer.

LayerMask layerMask = 1 << 8;
void Fire()
    {
        //play the audio of the gunshot
        StartCoroutine("SetPlaying");
        RaycastHit hit;
        
        
       //cast a ray from the player forward and check if the hit object is on layer 'Enemy'
       if (Physics.Raycast(transform.position, transform.forward, out hit, Mathf.Infinity, layerMask))
            {
              //if hit object is an enemy, set its 'gotShot' bool to true
              print("hit enemy");
              closestMob.GetComponent<EnemyBase>().gotShot = true;
            }
            //play gunshot sound and stop player from turning
            source.clip = fireSound;
            source.PlayOneShot(fireSound, gunshotVolumeScale);
            turnSpeed = 0;
       
    }

I'll also note that all the solutions I've seen to this issue are not working for me. I declared an int variable called 'layerMask' and initialized it in Awake() by bit shifting layer 8 into it (i.e. int layerMask = 1 << 8), but it still isn't detecting it. The enemy contains all that I belive it should need for this to work, including a rigidbody, a capsule collider, the associated scripts, as well as being on the 'Enemy' layer. This is where it gets weird (at least to my knowledge), when I invert the mask in the cast (~layerMask), it does exactly what I'd expect, and begins firing the code within the raycasts if statement when the player 'shoots' anything that doesn't have the 'Enemy' layer.

Any help would be suuuper appreciated as I'm getting to the point of slamming my face into the desk :/

Side Note: I'm getting to the point where I may just attach a 'fire range' cube trigger to my player and enabled it when the Fire() event is triggered, then have that check for game objects with the tag 'mob' as that kind of detection works most consistent to me.

Answer 1

First, be sure that you didn't confuse layers with tags. They are different.

Second, get rid of any implicit actions and references, i.e. don't use bitshifting, layer indices, or any non-straightforward reference. Instead, create something like this:

[SerializedField] private LayerMask _layerMask;

Use inspector to assign needed layer(s).

This way you will explicitly see, which layer you are using. This is useful not only for you, but for you in future, when you forget layers' indices. Also, for anyone who aren't familiar with the project.

Third is for debug. Be sure that your raycasting works as intended at other aspects:

• Try remove layerMask and see if the ray goes where you want it to
• Use custom gizmos to check if you cast the ray in a right direction
• Try using RaycastAll. Maybe some objects catch (block) your ray earlier than you think

Answer 2

Looks like the issue was that I had my 'Enemy' prefab which contained the necessary components (RigidBody, Collider, scripts, NavmeshAgent) nested inside an empty game object. I thought at first to make something a prefab you needed to have it inside an Empty. I see now that is rather redundant and not necessary (at least in my case).

Physics.RaycastAll solved this issue as it no longer got 'halted' by the parent empty's collider and also hit the child.

I actually got it working just using a regular Physics.Raycast by rebuilding the Enemy prefab as just a single Capsule object (which I'll replace with character meshes later on).

Side Note Before I got this new method working, I also used a different one that achieves the same goal in a pretty lightweight manner.

1. I added a long and thin box trigger to the front of my player so that it has enough distance to collide with any enemies within the shooting range.
2. I then enable the trigger any time the player enters their 'Aiming' state. If the trigger collides with any mesh that has the tag "Mob', it sets a bool on the player that indicates if the enemy is in range.
3. Then if the enemy is in range && the player enters the 'Firing Gun' state, a method in the Enemy Base script is fired that decrements the enemy health by a publicly decided variable called damagetaken.

Both the Raycast method and the box trigger method work equally well for me, but the box trigger one just took less time to figure out and gave me less headache haha.

Hope this helps anyone else in a bind!!!!!