How to use LayerMask variables with conditionals?

Questions & Answers
1

I have an OnCollisionEnter during which I would like to check whether the colliding object has collided with a specific layer. At the moment I am only using an int to check the layer.

    public LayerMask woodLayer;

	void Start() {
	}

	public void OnCollisionEnter(Collision other) {
		Debug.Log ("We have hit " + other.collider.gameObject.name);
		if (other.gameObject.layer == 11) {
           //Do things
        }
    }

This is very inefficient since every time I reuse this component, I have to go through all of the code and change everywhere the different int values are referenced, to what ever the layer numbers are for that specific project. Also if I want different functionality for the same layer on different gameobjects I have to create a new version of the same script and change the layer specific functionality around.

Instead I would like to use the public LayerMask variable and then simply reference it in the “if” statement. Something like this:

if (other.gameObject.layer == woodLayer) {

This would allow me to change the woodLayer in the inspector for every gameobject this component is attached to. At the moment, this of course will not work because the if statement is looking for an int. Therefore my first question is:

  1. Is there a way to convert a LayerMask variable to an int?

I thought that LayerMask.value was for this purpose, but for my layer 11 it is returning a value of 2048, which I don’t quite understand. If the answer to question 1 is “no”, my second question is:

  1. How can I use a LayerMask variable with a conditional statement?
2

I actually was just taking notes on this! I was quite confused at first, because the instanced property for GameObjects called “layer” returns the layer index, which is in the range [0, 31]. That number alone is not helpful for doing the mask comparisons. But with that number, the layer index, you can bit shift 1, and then use a binary operator and see what you get.

So this is a long explanation if you aren’t familiar with bitmasking. I didn’t want to just give code for you to copy and paste – that’s not creative! If you already know this stuff though, then at least it’ll be helpful for others stumbling upon this :P. The if statement (code) you want is near the bottom.

So say you had a LayerMask variable in your script, and then set it to say “Default, Water, Wood” in the inspector. Default is the first layer, so its layer index is 0. Water is another built-in layer, at layer index 4. And your custom Wood layer you said was at index 11. But when you try to use the LayerMask variable (say woodLayer, as it’s called in your script) as an integer, this converts it to the same integer as it would do if you had done woodLayer.value. This value property on LayerMasks does not give you back the layer index. It also wouldn’t make sense if you had multiple layers selected like “Default, Water, Wood” in the inspector – would it return 0, 4, or 11?

Instead, this value is a “bit masking” value. The thing is that any of the layers can be turned on or off like flags with the LayerMask in the inspector – and many of them could be on at the same time in the mask. This directly translates to binary: On can be represented with a 1, and off with a 0. And because there are up to a total of 32 layers (0 inclusive to 31 inclusive), we can use 32 digits (zeros and ones) to represent the entire LayerMask into a “bit masking” 32-bit integer – just your typical int variable, but this one with special meaning.

So assuming you are familiar with base 2, binary, then your “Default, Water, Wood” LayerMask should look like this as an integer, in base 2:

0000 0000 0000 0000 0000 1000 0001 0001

If you think of this as an array of 32 single digits starting with index 0 at the right side, there’s a 1 at index 0, 4, and 11 – just like the layer indices! In a binary number, these spots (at 0 and 11) have place values, much like the “12th place” in our decimal, base 10 number system means one hundred billion – 100,000,000,000. (if you starting counting the ones place as place 0, the tens place as place 1, hundreds as place 2, etc., then place 11, which is the 12th spot, gets you 100,000,000,000)

So in binary, the…

0000 0000 0000 0000 0000 1000 0000 0000 means 2048,

0000 0000 0000 0000 0000 0000 0001 0000 means 16,

0000 0000 0000 0000 0000 0000 0000 0001 means 1.

Together, they would add to 2065, which is equal to 0000 0000 0000 0000 0000 1000 0001 0001 in binary. With bitmasking values like this, the 2065 isn’t visibly that significant to us – I don’t immediately see that there’s three “flags” turned on in that number – the binary is more visually “with it” I suppose haha. Meaning it’s more obvious for us to look at and see what’s “on” and “off”.

Anyway, so how does this help us with the gameObject.layer, which returns us the single layer index of the GameObject in question?

The goal is to take gameObject.layer and convert it from a layer index into its bitmask value. With both bitmask values (being significant binary values), then we can use a handy bitwise operator on the two binary numbers. And finally, the bitwise operator will let us efficiently use binary math to check whether this important question is true or not:

If gameObject.layer tells us that the GameObject’s layer index is some number n, which lies in between 0 and 31 (inclusive), then is that same place n in the LayerMask’s bitmask also on?

To do this, we can take the two 32-bit numbers (of 32 digits each) multiply each corresponding digit with each other, keeping in mind 0 * 0, 0 * 1, and 1 * 0 are all 0. The only possibility of getting a 1 with multiplication here is if it’s both 1 and 1 – because 1 * 1 =1. The operator for this is called bitwise AND, using the symbol &, not to be confused with the conditional and for if statements and booleans &&. It’s probably called bitwise AND because you can only get a resulting 1 if you were doing it with 1 and 1 – both of them would have to be 1.

So the if statement you want to use is:

if (((1 << other.gameObject.layer) & woodLayer.value) != 0) {
	//other.gameObject IS in the woodLayer mask!
}

The 1 << other.gameObject.layer part is how we convert the layer index into the actual bitmask value, and the bitmask value is what we need to compare with the other bitmask value. << is the left bitshift operator, another binary operator. Here, it’s saying take the number:

0000 0000 0000 0000 0000 0000 0000 0001 (which is just the 1)

and shift this left by other.gameObject.layerplaces. So if the GameObject’s layer index was 0, then it (1 << 0) means move 1 left 0 places. So you haven’t moved it anywhere, and you’re left with 1.

1 << 1 means move 1 left 1 place, so you get:

0000 0000 0000 0000 0000 0000 0000 0010 (which is the place value for 2, so that is 2)

1 << 2 means move 1 left 2 places, so you get:

0000 0000 0000 0000 0000 0000 0000 0100 (which is the place value for 4, so that is 4)

Each time, the result goes up by a power of 2 (because we’re in base 2). So it’d go:

1 << 3 = 8

1 << 4 = 16

1 << 5 = 32

1 << n = pow(2, n) or 2^n

But luckily with the bitshift, we don’t need to worry about what the bitmask value actually converts to (the 2, 4, 8, 16, 32 numbers) – whatever it comes out to, we just know it’s an int with that “special meaning” – and we bitwise AND it with the other bitmask value. If the bitwise AND (digit by digit multiplication) ended up with 0, then they had no flags in common. Each case ended up with a 0 and 0, 1 and 0, or 0 and 1. If the result is NOT 0, as in != 0, then there was at least one flag that they had in common. The GameObject can only be on 1 layer anyway, so it doesn’t matter which – the important fact about the result being nonzero is that it tells us the LayerMask had a 1 in one of the same spots as the GameObject’s layer did!