Destroy one prefab clone of set of clones.

To be more specific, is there a way to destroy a certain clone from a list of clones?
My problem is that I have 5 HousePrefab(Clone)s in my scene, I placed them using raycast, say I only wanted 4 and I built too many. Using this code deletes the first once built which is not what I want, I’d like to delete the one my mouse is over.

		if (Input.GetButtonDown ("Fire1") && collider.Raycast (ray, out hitInfo, Mathf.Infinity) && KeepingScore1.Selected == 4) {
			Destroy(GameObject.Find("House(Clone)"));
		}
// KeepingScore.Selected == 4 is a reference to my 'remove' button being active.

I’m a bit stuck because I’m not sure if I can use Raycast to solve my issue. Or whether I need to make an array of gameObjects or if attaching a script to the HousePrefab would do it. Any advice on the way forward would be greatly helpful.

It seems what you’re saying is that, if they have placed 4 houses down, then when they place the fifth, destroy that fifth. The solution to this: Before the code that involves placing the house, have an if statement that checks the number of houses and only executes if there are only <4 houses; i.e.

if(houses <4)
{
    if(Mouse1)
    {
        if(raycast)
        {
        }  
    }
}

public Transform HousePreFab = (Transform)Resources.Load(“House”, typeof(Transform));

public class Tile : MonoBehaviour {

void Update () {
		ClicktoChangeWeight ();
	}	
void ClicktoChangeWeight(){ 
		int a = Weight;
		Ray ray = Camera.main.ScreenPointToRay (Input.mousePosition);
		RaycastHit hitInfo;

		if (collider.Raycast (ray, out hitInfo, Mathf.Infinity)) {
			if (Physics.Raycast (ray, out hitInfo)){

				if (Input.GetButtonDown("Fire1") && hitInfo.transform.gameObject.tag == "House" && KeepingScore1.Selected == 4) {
					//Destroy(GameObject.Find("House(Clone)")); // Destroys House(clone) at the top of the list.
					//Destroy(GameObject.FindWithTag ("House")); // Destroys House(clone) at the top of the list.
					Debug.Log("House Hit"); // Works fine

				}
              }
    }  

}}

So this is my code.
Sorry for the syntax.
I have tried verious versions of Destroy(); as shown with the comments. Each time the House(Clone) that gets destroyed is the first one created. I am trying to identify the House(Clone) prefab that is at the cursor position.