Problems with Raycast [Collider2D version of Collider.Raycast()?]

Unity 2018.4 (I have an old computer)

Short version:

Collider.Raycast() does a raycast check that ignores all colliders but this collider. Collider2D.Raycast() does something different. I’m making a 2D game, but I want the functionality that comes from the 3D collider’s raycast.

Do I need to use Collider objects in my 2D game, or is there some way to use Collider2D to do this?

Long version:

I have many Character objects. These objects have polygon colliders with irregular shapes. Part of the game mechanics require them to navigate by finding the closest point between their collider and a target position.

I want to use raycasts (or better: circlecasts) to find this point.

My problem is that if these character objects use the Physics2D, or Collider2D, version of Raycast(), they might hit another character’s collider and mistake it for their own.

I’ve attached some example code and screenshots of my setup that (I hope) demonstrate the problem.

Is there a better way of achieving this? Should I just use Collider objects instead of Collider2D?

Thanks.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class Example1 : MonoBehaviour
{
  public GameObject target;
  private Collider2D myCollider;
  private Vector2 mousePosition, direction;
  private RaycastHit2D hitInfo;

    // Start is called before the first frame update
    void Start()
    {
      myCollider = GetComponent<Collider2D>();
    }

    // Update is called once per frame
    void Update()
    {
      mousePosition = Camera.main.ScreenToWorldPoint(Input.mousePosition);
      hitInfo = Physics2D.Raycast(mousePosition, (Vector2)gameObject.transform.position);
      if (hitInfo.collider != null) target.transform.position = hitInfo.point;
    }
}


Fig 1. System before code execution
(Each “Character” object has it’s own target and copy of Example1 script to execute.)


Fig 2. System after code execution.
(Notice how some of the targets have jumped over to other characters.)

You should make use of Layers and LayerMasks in Raycast methods. When you create layers, you can edit their collision matrix in:
Edit > Project Settings > Physics2D

From there, you can define which layers can collide with which other layers.
When you provide a LayerMask value in a Raycast method, any layers not included in the mask will be ignored.

So for instance, you can create a “Character” layer, and in the collision matrix settings, disallow this “Character” layer from colliding with itself. Assign this layer onto your character GameObjects, and include a LayerMask field in your script that you can assign in the inspector (or through code, whichever you prefer).
Finally, use the LayerMask with the Raycast method:

//Assume that the GameObject this script is attached to has the "Character" layer assigned to it.
public class Example : MonoBehaviour
{
   //Assume this is set in the inspector to collide with all layers except for the "Character" layer.
   public LayerMask layerMask;

   void DoRaycast()
   {
      //The raycast will ignore anything on the "Character" layer.
      RaycastHit2D hit = Physics2D.raycast(mousePosition, transform.position, layerMask = layerMask);
   }
}

See:

As above you should be using layer-masks to determine what you hit. Also, you can disable the following option to start line/ray casts from detecting colliders they start inside.

Physics2D.queriesStartInColliders

I wanted to add that all the Collider2D.Raycast does it calculate the position of the collider for you and then performs the standard raycast.

(Reminder: I’m using Unity 2018.4. I’m going to try upgrading tonight.)

So I tried some things out, and here’s where I’ve gotten:

1) I was using Physics2D.Raycast() incorrectly.
You don’t plug two points in, you plug in an origin and a direction. Having corrected this, my raycasts are less buggy, but the targets still jump from collider to collider.

2) I’m still using layerMasks incorrectly.
This is a trip. Let me take you with me.

FIRST I set up my code as Vryken suggested above, e.g. I made a public LayerMask object and set it to everything BUT the “Character” layer in the UI.

This didn’t work, because for some reason it zeroed out bit 3 instead of bit 8 (Character being layer 8).

NOW I’ve set up a bitwise operation to set my layermask instead. This gives me even weirder behavior in that it ignores all raycasts unless they’re between two colliders. Which sure is wild.

But none of this addresses my primary concern that:

3) I don’t think layermasks will do what I want.
Please correct me if I’m wrong, but my current understanding of layermasks goes thusly:

Set objects to a layer.
Create a mask that ignores that layer.
Run a raycast with that mask.
That raycast will ignore all objects in that layer.

What I want is a raycast that ignores EVERY collider but ONE.
I have multiple objects that need to run a raycast that ignore EVERY collider but THEIR collider.

If I wanted to do this with raycasts I would have to (and please correct me if I’m wrong here) either:
Set each object to their own layer and run raycasts with masks that ignore all layers but one,
or
Create a special layer and MOVE an object to that layer each time I wanted to raycast to that object specifically. (I recall reading that this second option is very resource intensive and kinda jank, not to mention that there would still be collisions if multiple objects wanted to use it at the same time.)

Am I missing some fundamental understanding of raycasts? Please help.

For what it’s worth, here is my current example code:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class Example1 : MonoBehaviour
{
  public GameObject target;
  private LayerMask layerMask;
  private Collider2D myCollider;
  private Vector2 mousePosition, direction;
  private RaycastHit2D hitInfo;

    // Start is called before the first frame update
    void Start()
    {
      myCollider = GetComponent<Collider2D>();
      layerMask = ~(1 << 8);
    }

    // Update is called once per frame
    void Update()
    {
      mousePosition = Camera.main.ScreenToWorldPoint(Input.mousePosition);
      direction = (Vector2)transform.position - mousePosition;
      hitInfo = Physics2D.Raycast(mousePosition, direction, layerMask);
      Debug.DrawLine(mousePosition, (Vector2)transform.position, Color.yellow);
      if (hitInfo.collider != null) target.transform.position = hitInfo.point;
    }
}

So I’ve continued poking for a bit and learned the following things:

  1. For whatever reason, when I use the IDE to set my layermask to everything BUT one layer, it gives me garbage. To get an inverted layermask I have to invert the layermask. No idea why.

  2. When fed a properly inverted layermask, my raycast is acting like the direction vector is NEGATIVE.
    I’ve attached a picture, and it’s hard to see, but if you extend the yellow lines out past the cursor, they point to where the targets end up. The purple target belongs to the star, the red target to the circle, and the green target to the cross. The green target isn’t moving anywhere because its raycast isn’t hitting anything.

Again, I have no idea why it’s doing this.

Yes, if you have two world points then use Physics2D.Linecast (line segment query rather than ray query).

Well yes, the code you submitted above is passing LayerMask to the Distance argument as shown in the docs here.

Yeah that’d do it. -__-

Thanks for the correction MelvMay!

I’ve switched to linecast, but it continues to exhibit the behavior I don’t want (allowing targets to jump between characters). I’m going to try using Collider.Raycast() later. If that doesn’t work, I’ll just use Physics2D.LinecastAll() and sift through the results until I get the collider I want.

Z DEPTH!

I got it!

I don’t know if anybody else will benefit from this, but I’ve gotten my answer and it’s the min and max depth parameter for Physics3D.Linecast! If I assign each character a unique Z depth, I can single each one out using that! It’s a touch janky, and it means I’ll have to set limits to how many characters I have, but that limit can be in the hundreds if I want so no trouble there.

Thanks to MelvMay and Vryken for your help!