Bitmask/LayerMask used in raycast not working

Questions & Answers
1

Hey guys, long story short I’m working on a topdown shooter thing and one of my enemy types is a snail like creature that leaves a trail on the ground. I want to make it so that when I shoot I hit the creature itself and not its trail. I want to be able to shoot through the trail essentially. I should mention that the trail is made up of smaller subsections which are triggers. I have basic raycast-based shooting implemented and at first I just put the trail piece prefabs on the “Ignore Raycast” layer which worked. But then I realized it’d be better to limit the creation of trail objects so I added a check in the relevant code to not place a trail if there is already one in the position it would be placed.

The problem is that I use raycasts to do this and it wasn’t working with the trail pieces on Ignore Raycast. So I made a new layer in the editor and passed in a bitmask to the raycasts for the trail detecting code to find only that layer. That worked. But then I wanted to do the same with my shooting raycast so I used the same bitcast but negated it so that it would hit all the layers except the trail layer. I thought it would work but it actually made my gun not able to hit ANYTHING, even the default layer. I printed out the binary and all the bits are 1 except for the slime layer which is bit 8 and theoretically should work. I even read that I shouldn’t use Layer 31 in a bitmask so I bitwise &'d the negation to also exclude that, but it was the same result. I’m very confused at this point and any help would be appreciated.

The bitmask I’m using to check overlaps:

    int bitmask = 1 << 8;

The bitmask I’m using for my shooting raycast:

int bitmask = ~(1 << 8) & ~(1 << 31);

My trail checking code: (passing in the bitmask for layer 8)

bool checkSlimeOverlap() 
    {
        RaycastHit2D ray = Physics2D.Raycast(transform.position, -direction, .2f, bitmask);
        Debug.DrawRay(transform.position, -direction * .2f, Color.blue);
        return ray.collider != null;
    }

My raycast-based shooting code:

IEnumerator Shoot()
    {
        var playerControl = GetComponentInParent<PlayerController>();
        if (playerControl != null)
        {
            if (playerControl.AmmoCounts[1] > 0)
            {
                if (!IsAkimbo || playerControl.AmmoCounts[1] == 1) playerControl.AmmoCounts[1]--;
                else playerControl.AmmoCounts[1] -= 2;
                RaycastHit2D hit;
                transform.parent.GetComponent<SpriteRenderer>().sprite = shooting;
                source.PlayOneShot(pistol_shot, audioScale);
                yield return new WaitForSeconds(.1f);
                transform.parent.GetComponent<SpriteRenderer>().sprite = oldSprite;
                if (hit = Physics2D.Raycast(transform.parent.position, transform.parent.GetComponent<PlayerMovement>().position, Mathf.Infinity, bitmask))
                {
                    var enemy = hit.transform.GetComponent<EnemyController>();
                    if (enemy != null)
                    {
                        if (!IsAkimbo)
                            enemy.Damage(10);
                        else
                            enemy.Damage(20);
                        if (enemy.Health <= 0)
                            enemy.Die(Weapon.Pistol);
                    }
                }
            }
            else
            {
                yield return new WaitForSeconds(0f); //play empty sound;
                source.PlayOneShot(empty_shot);
            }
        }
    }

Wasn’t sure how much of my code to post, that may be overkill but hopefully there’s enough context. Thanks!

Edit: I got it working. For anyone with a similar problem, I’ll put what worked here. It turned out that it WAS hitting stuff, just it was hitting Edge Colliders in my scene which I didn’t even put myself. I determined this by putting in
Debug.Log(hit.collider.name);
after my if(hit = ...)

My solution was to get rid of the bitmask and just pass in the number 1 to hit only the Default layer. I was confused initially because I thought calling Physics2D.Raycast with no LayerMask argument would make it only the Default layer but that didn’t work. Passing in 1 did work however.

2

the code for your bitmask creation looks fine to me. Are you sure that the issue originates from that?

asking because the “direction” of your raycast here does seem to be set a bit weird as it is the position of your scripts parent:
Physics2D.Raycast(transform.parent.position, transform.parent.GetComponent<PlayerMovement>().position, Mathf.Infinity, bitmask))

which by the way is a bit redundant in itself, can also be written as this:
Physics2D.Raycast(transform.parent.position, transform.parent.position, Mathf.Infinity, bitmask))

3

So we are on the same page, bitmask is layermask, correct?

To create the mask so that it will hit only the desired layers, you write

//script defined Layermask
//Ground on layer 8
//StickyWalls layer 9

Layermask navigableMask = 1<<8 | 1<<9;

//Using raycast

if (Physics.Raycast (ray, out hit, navigableMask))
       Debug.Log ($"Hit the {LayerMask.LayerToName(8)} and {LayerMask.LayerToName(9)} layers in the navigableMask");