Raycasting, bit masking not behaving as expected, goes against set theory?

Howdy folks,

I’ve tried reading raycast & bitmasking documentation, and I’ve searched around the forums. However, I cannot get my head around this issue.

I’m trying to hit objects in a certain layer (layer 10) using Physics.Raycasts. However, the ignore-certain-layers function of raycasts confuses me. This is my current code:

if (Physics.Raycast(ghostCamera.transform.position, ghostCamera.transform.forward, out hit, nearRadius, ~(1 << 10)))
{
      Debug.Log(hit.transform.name);
 }

As I understand it, “~(1 << 10)” should mean:
Ignore every layer that isn’t layer 10.
However, it does not! It actually hits the player model (In which the camera is positioned) first. So I created a new layer for the player model, layer 11, and tried again. It didn’t work!

I messed around and found this solution:

    if (Physics.Raycast(ghostCamera.transform.position, ghostCamera.transform.forward, out hit, nearRadius, ~(1 << 11 | ~(1 << 10)))
    {
          Debug.Log(hit.transform.name);
     }

Which should work as intended, but bothers me to no end. I interpret ~(1 << 11 | ~(1 << 10)) to be:

Ignore if either: layer 10 or not layer 11.

I’m really confused as to why this works. It seems so illogical to me.

Can anyone help me with this?

Thanks!
Kind regards.

Your understanding of bitwise operations is perfectly fine. However you interpreted the layer mask the wrong way around. It’s not mask to exclude layers, it defines which layers you want to hit. The default value that is used contains all bits except the ignore raycast layer. Your layermask currently includes all layers except layer 10. You even included the ignore raycast layer ^^.

So you have to do (1 << 10) without the “bitwise not”. A layermask value of “0” would mean you can’t hit anything. A value of 0xFFFFFFFF would mean you can hit all 32 layers. If you only want to hit layer 10 you need the value “100 0000 0000” in binary or 1 << 10 or 0x0400

When I got into raycasting I learned it was easier to just do public Layermask myLayermask and assign it in the inspector instead of bitshifting.

Your assumtions on the bitshifting are correct the error must be in what layers you objects are. Its hard to tell. Also its hard to ever debug that code and my advice if you are going to use bitshifting would be to do something more in line with 1:

if (((1 << obstacle.layer) & obstacleLayer) != 0)
{
  // Do code
}