NavMesh.SamplePosition custom area bug?

Unity Engine
,
1

The following will sample all areas

NavMesh.SamplePosition(myQueryPos, out navHit, myRangeDistance, NavMesh.AllAreas);

Now, when we look into NavMesh.AllAreas, under the hood it has a value of -1

so if I have a nav layer with index 22' I had to supply -22` for the nav mesh to pick it up:

//had to introduce a 'minus sign' to make it work:
NavMesh.SamplePosition(myQueryPos, out navHit, myRangeDistance, -22);

If we a retrospective into Physics.raycast, there we would have to do a bit-shifting operation, like 1<<22
Here, not only do we simply supply the index of nav layer, but we also add a minus sign

bug or a feature?

2

Update: just checked and negative values just make it work like NavMesh.AllAreas :confused:

So how can we get the sample only on certain type NavMesh area?

Another Update: hm… I’ve just built a bare, empty project with two cubes, and 1 << 22 actually works fine… That’s really weird

3

Pfff… without even closing the previos project, after Alt-tabbing, the code was working fine… That’s so weird.

If anyone has the same issue, I am sorry - couldn’t find the cause.

word of advice, tripple-check:

  1. that you are using a correct, existing index of your nav layer (debug.log it if you are using NavMesh.GetAreaFromName(), because you might have supplied a non-existing layer name (with a typo)

  2. that distance to that layer in scene from your query coordinate is less than myRangeDistance in my code above

  3. that the query coordinate is in the location you expect it to be, especially if you are doing raycasts from screen

  4. that your layer’s index (in my case 22) is bitshifted like this 1 << 22, and use the resulting integer as the mask.

mandelbug probably due to indescretion. Case closed.

4

-22 just happens to “work” for you because its bitmask representation is quite dense - only bits at positions 0,2,4 are zero - the rest are ones (specifically bit 22).

The correct mask is - as you mention - 1<<22

1 Like