Why is Physics.Raycast returning colliders in a layer not included in the layermask?

[camomatic]

I’ve been through several similar questions and I still cannot get my code to work, so I’m offering it up in the hope that someone can point out my mistake.

---

I have a city object that generates building objects. To do this it does a Raycast on objects in the “Planet” layer to find the location for the buildings and then checks there are no buildings already in that location by using a SphereCast in the “Buildings” layer. The city object has its own collider, but is marked as being in the “City” layer. However, instead of generating buildings on the Raycast contact with the planet, buildings are being generated on the city object, as if it is not being ignored by the raycast. All I can imagine is that I have misunderstood how the raycast method works and what it is returning. Can anyone point out a mistake?

---

In summary: Physics.Raycast is returning a collider in the “City” layer, even though I have specified the layermask as including “Planet” only.

The code:

while (i < houses) {
	Vector2 rand = radius * Random.insideUnitCircle;
	Vector3 pos = transform.position + transform.TransformDirection (rand.x, rand.y, 100f);

	Ray ray = new Ray (pos, 110f * -pos);
	RaycastHit hit;

    //Draw ray to collide with planet layer. THIS IS THE PROBLEM LINE
    if (Physics.Raycast (ray, out hit, LayerMask.GetMask ("Planet"))) { 
		//...and not touching any other houses
		if (!Physics.CheckSphere (hit.point, 2f, LayerMask.GetMask ("Building"))) {
			//Spawn	
			Transform spawned = Object.Instantiate (housePrefab, hit.point, transform.rotation);
			spawned.Rotate (0f, 0f, Random.value * 360f);
			i++;
		}
	}

	n++;
	if (n > maxAttempts) {
		Debug.Log ("Time out error: Unable to place " + (houses - i) + " houses.");
		return;
	}
}

Example of current wrong behaviour. (City object is red for illustration)

Correct behaviour by turning off the city collider.

[Bunny83]

You pass your mask as distance, not as mask. As you can see in the signature after the RaycastHit parameter comes the distance followed by the layermask.

public static bool Raycast(Ray ray, out RaycastHit hitInfo, float maxDistance = Mathf.Infinity, int layerMask = DefaultRaycastLayers, QueryTriggerInteraction queryTriggerInteraction = QueryTriggerInteraction.UseGlobal);

If you want to do an infinite raycast you have to pass Mathf.Infinity (or float.PositiveInfinity) as third parameter and pass your layermask as fourth parameter.

if (Physics.Raycast (ray, out hit, Mathf.Infinity, LayerMask.GetMask ("Planet"))) { 

Note that instead of casting a Physics.Raycast against your planet you can also use Collider.Raycast. This method will only cast against this one collider and will ignore anything else. It’s actually more efficient to use Collider.Raycast if you only want to hit one specific collider.

[camomatic]

Problem solved. I had entered the LayerMask in the Raycast distance parameter. The method should be Physics.Raycast (Ray, RaycastHit, maxDistance, layerMask). The offending line now reads:

if (Physics.Raycast (ray, out hit, 100f, LayerMask.GetMask ("Planet"))) {

[camomatic]

Problem solved. I had entered the LayerMask in the Raycast distance parameter. The method should be Physics.Raycast (Ray, RaycastHit, maxDistance, layerMask). The offending line now reads:

if (Physics.Raycast (ray, out hit, 100f, LayerMask.GetMask ("Planet"))) {