Problem with colliders overlapping

Hey! Im working on my first Unity game in 2D and Im experiencing some problems with the box colliders. In the game the karakter is moving back and forth between building depending on which you click on right after selecting the character. Everything is working fine and is able to move between the buildings except for when the character is right in front of (overlapping) a particular house (the house is just a sprite with an image, no different from the other buildings in any way). It seems to be totally random if its the character or the house that is getting selected, and I cant see why! I’ll click on the character ten times but the house behind it gets selected eight of them.

Here is the relevant part of the code: (clicks is an array in which Im storing the mouse clicks’ order)

void Update () {
if (Input.GetMouseButtonDown (0)) {
RaycastHit2D hit = Physics2D.Raycast(Camera.main.ScreenToWorldPoint(Input.mousePosition), Vector2.zero);
if (hit.collider != null) {
if (hit.collider.gameObject.name.Equals (“Karakter”)) {
if (clicks.Count > 0) {
if (clicks [0].Equals (“Karakter”)) {
clicks.Clear ();
}
} else {
clicks.Add (“Karakter”);
}
}
else if (hit.collider.gameObject.CompareTag (“Building”)) {
if (!hit.collider.gameObject.name.Equals (currentPos)) {
if (clicks.Count > 0) {
if (clicks [0].Equals (“Karakter”)) {
clicks.Add (hit.collider.gameObject.name);
}
}
}
}
}
}

It seems to work now that I put the rotation-z to a negative number, but i dont see why that would do anything when changing the position-z didnt?