Checking if the Raycast hit, hits a layer from the LayerMask?

Question

Title says it al.

Initially I was using:

hit.transform.gameObject.layer == LayerMask.NameToLayer("Floor")

But eventually I figured out I needed different layers for different floors, I figured I should use a LayerMask, please correct me if I'm wrong but can't get it to work.

At the top of the document I have:

public LayerMask floorLayers;

In the void Update I have:

int floorMask = ~floorLayers.value;

And in the if statement I have:

if (playerHasWetFeet == true && wetSteps < maxWetSteps && controller.isGrounded && hit.transform.gameObject.layer == floorMask)

This however isn't working.

Can anyone point me out what I'm doing wrong?

Edit:

This is within my update function:

int floorMask = ~floorLayers.value;

    currentFrameFootstepLeft = anim.GetFloat("FootstepLeft");

    if (currentFrameFootstepLeft > 0 && lastFrameFootstepLeft < 0)
    {
        if (Physics.Raycast(transform.position, dirDown, out hit, 10f, floorMask))
        {
            if (playerHasWetFeet == true && wetSteps < maxWetSteps && controller.isGrounded && (1 << hit.transform.gameObject.layer & floorMask) != 0)
            {
                LeftWetStepSpawner();
                Debug.Log("Hit");
            }

Answer 1

With this:

if(hit.transform.gameObject.layer == floorMask)

You're just comparing the layer hit in the RaycastHit result. If this is really what you want and you just want to check what the raycast hit by layer then do not do any bitmask operation on the layer.

Remove this:

int floorMask = ~floorLayers.value;

Get the layer Indexes:

int floor1 = LayerMask.NameToLayer("Floor1");
int floor2 = LayerMask.NameToLayer("Floor2");
int floor3 = LayerMask.NameToLayer("Floor2");

then then just compare the Layer index directly:

if (hit.transform.gameObject.layer == floor1)
{

}

if (hit.transform.gameObject.layer == floor2)
{

}

if (hit.transform.gameObject.layer == floor3)
{

}

---

While that should fix your problem, the main reason of using layermask is to filter the raycast. When using layermask, you use it to decide which objects the raycast should hit or ignore and if this is what you actually want to do, do not compare the layer like you did in your code. Simply find the mask then pass the mask to the raycast function. I am not sure if this is what you want to do but both examples should give you idea on what layer mask is used for.

For example, you want to do a raycast to only objects on the "Floor" layer and don't want the raycast to hit any other object on another layer:

int floorLayerIndex = LayerMask.NameToLayer("Floor");

//Check if layer is valid
if (floorLayerIndex == -1)
{
    Debug.LogError("Layer Does not exist");
}
else
{
    //Calculate layermask to Raycast to. (Raycast to "Floor" layer only)
    int layerMask = (1 << floorLayerIndex);

    Vector3 fwd = transform.TransformDirection(Vector3.forward);
    RaycastHit hit;

    //Raycast with that layer mask
    if (Physics.Raycast(transform.position, fwd, out hit, 10, layerMask))
    {

    }
}

You can find many other examples on this post.

Answer 2

Since you want the raycast to ignore anything that isn't a floor, you can use the mask in the raycast so that it only hits floors. To do this, you want to have your LayerMask floorLayers so that only floors are checked, then in your Raycast:

Physics.Raycast(transform.position, dirDown, out hit, 10f, ~floorLayers.value)

If you do this, any hit will be a floor hit, and you don't need to do any bit math below.

If you later decide you can't use the same mask for the raycast (maybe you really need the raycast to be able to collide with numerous possible targets and then see if the hit object happens to be a floor), then read below.

---

First, you want to compare a mask value that has its bits set to true only for floors, which seems to be floorLayers.value.

If you want to compare a GameObject.layer to a LayerMask.value, you will first need to shift a bit by GameObject.layer.

Also, if you want to check for at least one match between gameObject.layer and floorMask, you want to use binary AND (&) and see if it's not 0 (!= 0 is unnecessary but it can help with readability):

... && (1 << hit.transform.gameObject.layer & floorLayers.value) != 0

This is because GameObject.layer is the index of that single layer, but FloorMask.value is a bit mask that is based on true/false values of various bits. It's meaningless to compare them directly, without making a conversion first.