This is an attempt to simplify Stelimars insightful post.
LayerMask, or any "Mask" in Computer Science, is an integer which uses its 32 bits to represent different flags. While it is an integer, the value doesn't mean anything useful numerically. What matters is which flags are turned on or off.
In the case of LayerMask, each flag (each bit) represents one layer. If you select any game object from the editor you'll notice the Layer field in the top right corner of the inspector. Clicking this will bring up a list of different layers. There are some predefined Layers, but you can also add layers. Notice how you have 32 slots for your layers. This is because Unity uses an integer type to store the flags in each of the 32 bits that makes up an integer. As we can see, Layer 8 to 31 is available for us to define for our own game.
Since Masks use a single bit to represent if some option is true or false, a Mask can contain many different options. The same goes when thinking about Layers. A LayerMask specifies which Layers are enabled for any given mechanism. This enables us to tell the game that our GameObject "Don", for example, is thought to belong to the user defined layer "NPC" and "Bob" to belong to the user defined layer "SpellCaster".
Then, we make us of this information for example in a raycast call, specifying that the raycast should only return objects that belong to the "NPC" layer. An example when this could be used is when you want to target NPCs only with the click of the mouse. Then, imagine you have a game where you want to cast a spell, and that spell can only target Spell Casters. Then you'd only include the game objects that has the SpellCaster layer enabled. A third option would be to include both NPC and SpellCaster in the search. That way we would test both Don and Bob.
Finally some technical mumbo.
Technical
A LayerMask is represented as a 32bit integer. It also has a name table for naming the different bits so it is more readily understood by us game developers. Just to give some perspective, here's how a 32 bit integer would look like if it was written.
0000 0000 0000 0000 0000 0000 0000 0000
I put a space between every 4th bit to make it easier to read.
The first bit is actually located on the right side. Think of this as being the first layer. Here's how it would look if the first bit (or layer) would be set:
0000 0000 0000 0000 0000 0000 0000 0001
Since Unity predefines 8 layers, we can display the Mask for including all of those layers as such:
0000 0000 0000 0000 0000 0000 1111 1111
And if we like to search for NPC as from the example above, given we've put it in User Layer 8 (starting from 0, this is the 9th layer), we'd mask it as:
0000 0000 0000 0000 0000 0001 0000 0000 // NPC Layer
Just for completeness, this is how you should think a mask that include NPC OR SpellCaster should look like, if we assume the SpellCaster was set to User Layer 9:
0000 0000 0000 0000 0000 0011 0000 0000 // NPC | SpellCaster Layer
Example
Allright, now that you see we're setting bits to enable the different layers, let's look at how we generate the bit field from scratch using C#. In my example, I will make three masks that can be used to query NPCs, SpellCasters, and "NPCs and SpellCasters". We'll use Physics.Raycast in our example.
Example 1: Raycast vs NPCs
int NPCLayer = 8;
// This is used to illustrate which Layer we want to use.
int NPCMask = 1 << NPCLayer;
// This is used to generate the Mask derived from NPC Layer.
if Physics.Raycast(fromPosition, direction, Mathf.Infinity, NPCMask)
{
// We cast a ray onto a NPC!
}
Example 2: Raycast vs SpellCasters
int SpellCasterLayer = 9;
// This is used to illustrate which Layer we want to use.
int SpellCasterMask = 1 << SpellCasterLayer;
// This is used to generate the Mask derived from SpellCaster Layer.
if Physics.Raycast(fromPosition, direction, Mathf.Infinity, SpellCasterMask)
{
// We cast a ray onto a spell caster!
}
Example 3: Raycast vs NPCs and SpellCasters
int NPCLayer = 8;
int SpellCasterLayer = 9;
// This is used to illustrate which Layers we want to use.
int NPCMask = 1 << NPCLayer;
int SpellCasterMask = 1 << SpellCasterLayer;
// This is used to generate the Mask derived from the layers.
int CombinedMask = NPCMask | SpellCasterMask;
// We use the binary OR operator to combine masks.
if Physics.Raycast(fromPosition, direction, Mathf.Infinity, CombinedMask)
{
// We cast a ray onto a NPC or a SpellCaster!
}
Stelimar also have excellent information in his posts that you should read.