Raycasting with Layermask gives different results in different situations.

Unity Engine
1

So this pretty much makes no sense, or at least I haven’t been able to make any with it for the past hour.

My understanding is that if you pass a layermask with a Raycast, the Raycast will ignore that specific layer, and register hits for all others, correct? And then if I want to reverse this - have collisions register for just this layer and ignore all others, I can just flip the bits using the tilde(~) sign. I mean, even if that is the other way around, I’m getting different behaviors in different situations.

When I do this -
if (Physics.Raycast(transform.position, -1 * transform.up, out hit, 10, ~(1 << LayerMask.NameToLayer(“Travelator”))))

It returns me hits on ONLY Travelator. Which is what I expected.

However when I do this -
if (Physics.Raycast(startOfLine, -1 * transform.up, out hit, 100, ~(1 << LayerMask.NameToLayer(“ProximitySensor”) | 1 << LayerMask.NameToLayer(“BuildingDetector”))))

It gives me collisions on all layers, except these two. Which is the opposite of how ~ was behaving in the first case. Am I doing something wrong in the way I’m pairing the two layers together?

2

Hello, if you pass layermask into a rascast function it will only collide with that layer you passed in.

3

As mate_veres said, the raycast layer mask only collides with the layers you pass. The reason your second example is acting strangely is because you don’t have the parenthesis in the right place. It’s running the bitwise NOT operator on the first layer but not the second.

e.g. Instead of ~(Layer1 | Layer2)

it’s doing

~(Layer1) | Layer2

4

if you’re having a hard time with bit masks, let me try explain it visually

suppose your layers are set up like this
“Travelator” is layer 4
“ProximitySensor” is layer 5
“BuildingDetector” is layer 6

the layer bitmask argument expects you to build up a value that corresponds to a series of switches.
so that the fifth switch corresponds to your “ProximitySensor”, and the sixth switch to your “BuildingDetector” etc.

thus if your switches are set up like this: on off on off off off
that would mean your 4th and your 6th layers are on.

switches themselves are in fact, bits, hence the name, and they can be 0s or 1s.
the above example would be 101000, which can also be represented by an integer value of 40, in this case

this is because:

0 is worth 0*2^0 = 0*1
0 is worth 0*2^1 = 0*2
0 is worth 0*2^2 = 0*4
1 is worth 1*2^3 = 1*8
0 is worth 0*2^4 = 0*16
1 is worth 1*2^5 = 1*32

when you add these up, you get 40

to assemble a bit mask you use bitwise operators like you did.

namely

~  is a bitwise negation operator, it requires one operand and inverts all of its bits
<<  is a bitwise left shift operator, that pushes the bits of the left operand, by the value supplied by the right operand
|  is a bitwise OR operator, it takes two operands, and makes any 'on' bit survive into a result, unless there were none

it also helps to understand that all operators in C# have rules of precedence
in this sense, ~ is more important than << which is more important than |
consider that parentheses will enforce manual precedence

so when you run

~(1 << LayerMask.NameToLayer("Travelator"))

this is equivalent to

~(1 << 4) = ~b1_0000 = b1111_1111_1111_1111_1111_1111_1110_1111

in other words, you’ve specified all layers BUT “Travelator”

in your second example

~(1 << LayerMask.NameToLayer("ProximitySensor") | 1 << LayerMask.NameToLayer("BuildingDetector"))

this is equivalent to

~(1 << 5 | 1 << 6) = ~((1 << 5) | (1 << 6)) = ~(b10_0000 | b100_0000)* = ~(b110_0000)
= b1111_1111_1111_1111_1111_1111_1001_1111

* you'd get the same result here with ^ or even +, but | is arguably cleaner because of intent

in other words, you’ve included all layers BUT “ProximitySensor” and “BuildingDetector”

all in all, your formulas are good, and your precedence is good
but your first example shouldn’t work, while the other works as expected

my guess is that you misspelled the layer name and proceeded to shift by -1

which won’t give you an error, but an anomalous value, that got inverted and gave you a false positive

~(1 << -1) = ~(b1000_0000_0000_0000_0000_0000_0000_0000) = b111_1111_1111_1111_1111_1111_1111_1111

i.e. you got’em all lit except the last one

1 Like