Possible bug (or poor documentation) with Physics.CheckCapsule [C#]

Unity Engine
1

I’m not sure if this is a bug, or me mis-using this function, but either way I wanted to share and try and learn.

I am trying to do some basic hit detection in an RTS-style game, when placing structures. I want to make sure that there is nothing within a certain area around a to-be placed structure. I decided to use Physics.CheckCapsule, but there seems to be either a problem or incorrect documentation, or my understanding of LayerMasks is off.

I wanted to ignore any hits with terrain, and I used the LayerMask of my Terrain (layer 8).

RaycastHit hit;
LayerMask terrainLayerMask = 1 << 8;
Ray ray = Camera.main.ScreenPointToRay(Input.mousePosition); 		
if ( Physics.Raycast(ray, out hit, Mathf.Infinity, terrainLayerMask ) ) {
	Debug.DrawRay(hit.point, Vector3.up * 10, Color.red);

	if ( Physics.CheckCapsule(hit.point, hit.point + (Vector3.up * 20), 10, terrainLayerMask) )
		Debug.DrawRay(hit.point + (Vector3.up * 10), Vector3.up * 10, Color.green);
}

But this code didn’t work. I was always getting a hit, even on terrain. Is this intended, does CheckCapsule not use the LayerMasks in the same way? I eventually was just guessing and managed to make it work by changing this line to specifically use “7”, IE my layer ID - 1:

	if ( Physics.CheckCapsule(hit.point, hit.point + (Vector3.up * 20), 30, 7) )

Can anyone shed some light on this? Thanks!

2

Hello there!

It is slightly unclear on how the layermask is used I suppose.
You see, the layermask tells you what layers you want to check, not the ones you want to ignore.
So if you add the following line below your layermask creation things should work like you want them to:

terrainLayerMask = ~terrainLayerMask;

This reverses the bit mask to it’s opposite, which is everything except the terrain.

3

Sigh… Indeed, you are correct! I swear that I tried that, but I guess I didn’t.

Thanks. :slight_smile: