Colliders attached to objects on separate layers are colliding?

I have objects that I have on a layer that I use to be collided with when raycasting, and then I have objects another layer that is used for physics collisions. The objects on the physics layer are colliding with the objects on the raycast layer instead of going through them. How do I get the colliders for the objects on the physics layer to ignore collisions with objects on the raycast layer?

Go to Edit → Project Settings → Physics. Use the collision matrix to set up collision between layers.