Hidden Objects scoring logic problem

Creating a hidden objects game using the New UI 4.6 as a learning project. I have a object prefab set to an AddListener calling the ButtonTest function. It increases score when an object is clicked as it should. Then I added a bool BeenClicked to check if indeed an object was clicked or not still no problem. When I combine it with the Input.GetMouseButton (0) it should fire when the mouse button is clicked but not on an object and thus detract score. However this is not the case it is firing everytime the button is clicked. I have tried several different options it appears for some reason there is a basic incompatibility with these two things although I do not understand why.
Immediate question is how to get a decrease in score when something other than a button is clicked. Any additional information would be greatly appreciated. Code is posted below.

using UnityEngine;

using System.Collections;

using UnityEngine.UI;

public class ScoreKeeper : MonoBehaviour

{

public Text ScoreText;
public int scoreChange = 100;
public static int  startScore= 1000;
public bool BeenClicked = false;

public void Awake()
{
	
	ScoreText = gameObject.GetComponent<Text> ();

	if (ScoreText == null) 
	{
		ScoreText = gameObject.AddComponent<Text> ();
	}
	if (Input.GetMouseButtonDown (1)) 
	{
		startScore -= scoreChange;
	}
}

public void Update() 
{	
	if (BeenClicked) 
	{
		ScoreText.text = startScore.ToString ();
	} 
	else if (Input.GetMouseButtonDown (0))
	{
		DownScore();
	}
	BeenClicked = false;
	ScoreText.text = startScore.ToString();
}

public void ButtonTest()
{
	BeenClicked = true;
	startScore += scoreChange;
	
}
public void DownScore()
{
	startScore -= scoreChange;
	print ("not a button");
}

}

Update. Think I have determined this is just the wrong way to attack the problem. Seemed like a good idea at the time. Further research shows I may need to utilize raycast instead. Got some more reading to do.

Yep that was the problem. I got tunnel vision thinking about the score changing and not the simpler problem. Just in case anyone else is looking for a solution to the problem of 2D object detections with multiple object outcomes I will post the link to the answer that seemed explain it best. C# mouse Raycast question - Unity Answers Hope this is helpful to someone. Sorry to have wasted anybodys time. Really am happier to have discovered the answer on my own.