How to make raycast only collide with two layers?

I’m trying to implement a way for the raycast to only collide with two types of objects, obstacles and players. I made the appropriate layers, set the player to the Player layer and the obstacle to the Obstacle layer. I also created the sight layer which is used by the game object running the script below, and made sure that the sight layer only collides with Player and Obstacle layers. Then I tried doing this in my code:

if (Physics.Raycast (from, dirToTarget, out hit, (1 << LayerMask.NameToLayer("Sight")))) {
   Debug.Log ("HIT : " + hit.transform.root.gameObject);
}

Most of the time the Debug either prints that it hits a player or an obstacle but sometimes it will print that it hits an Enemy, which is the game object that is running the script, and from which the ray originates. The ray is drawn from the center of the Enemy to the center of the Target (the player). I am really confused as to why the raycast collides with the enemy, since I instructed it not to in the layer… How can I fix it so that the raycast only collides with either objects of the Player layer or the Obstacle layer?

Edit / Project Settings / Physics
Unity already offers a way to allow different settings for different layers:

@jellythedonut this is the way to do it…

            if (Physics.Raycast(from, dirToTarget, out hit, (1 << LayerMask.NameToLayer("Sight") | (1 << LayerMask.NameToLayer("OtherLayerMaskName"))))
            {
                Debug.Log("HIT : " + hit.transform.root.gameObject);
            }

//First declare variable

[SerializeField]
LayerMask lm;

**//assign maxdistance and lm value

if (Physics.Raycast(from, dirToTarget, out hit, maxdistance,  lm.value))
      {
                     Debug.Log("HIT : " + hit.transform.root.gameObject);
      }

//Save the script and then select lm layer which objects do want to collide with from unity editor

string layerNames = { “Sight”, “OtherLayerName”, “OneMoreLayerName” };
if (Physics.Raycast (from, dirToTarget, out hit, LayerMask.GetMask(layerNames))) {
Debug.Log ("HIT : " + hit.transform.root.gameObject);
}