Layermask (raycast) wont work...

Questions & Answers
1

I am using the following code to detect when the player mouses over any active game objects in the layer tilesLayer:

Ray ray = mainCam.ScreenPointToRay(Input.mousePosition);
RaycastHit hit;
	
//if any tileNode collider is moused over...
if(Physics.Raycast(ray, out hit, 500f, tilesLayer)) {

If I remove the tilesLayer parameter, the raycast picks up the specific tiles I’m trying to detect mousing over. However when I use the parameter tilesLayer (which should be a constant value of 8, as that is the layer for all of my tiles), it does not detect the tiles at all… I even added this code when trying without the parameter:

Debug.Log ("Currently moused over: " + hit.collider.gameObject.name + ", on layer: " + hit.collider.gameObject.layer);

And it said “Currently moused over: tile117, on layer: 8”. Yet if I use 8 as the raycast layermask parameter it doesnt detect the tiles… What’s going on???

2

This is a layerMASK, therefore it MASKS the desired layer for raycast.
If you want to only include tilesLayer, use ~tilesLayer.

The ~ operator is a bit-wise NOT. It will turn all the binary 1s into 0s and vice-versa. By doing it, you tell the raycast to ignore every layer, except the one that you specified. In this case, tilesLayer

4

How was this an accepted answer when the documentation says the opposite.

This fixed my issue - Raycast ignores my layer mask? - Unity Answers

3

Or just make LayerMask myLayer; public so you will have nice scroll menu in the inspector and then you can choose what you want.

5

I found an important thing about layermasking, layermasking seems to fail on even number layers.
the time this happenes is when I bitshift int layerMask = gameObject.layer << gameObject.layer;
then I use layerMask = ~layerMask; and use that in the layerMask slot of the raycast function, when tested, I found that the ray would ignore all odd numbered layers 1,3,5,7,9, etc. but the layermask would fail on even layers 2,4,6,8,10 etc