What exactly makes you think it collides with all layers? You don’t seem to check what you actually hit. Keep in mind that your raycast is infinite. Also your * 1.5f is pointless as the direction vector is just a directionm the length of the vector doesn’t matter. The distance of the raycast is specified in the 3rd parameter. So if you want a raycast that is 1.5 units long you may want to do:
RaycastHit2D hit = Physics2D.Raycast(transform.position, -Vector2.up, 1.5f, LayerMask.GetMask("Ground"));