How to collide with everything but trigger script whenever certain object hits

I have a cannon and a wall, i have a setup cannon which shoots canonball at a wall and i have a simple script that works by deleting old wall and replacing it with new fractured one.
But OnCollisionEnter doesn’t work very good, because wall(or cubes) very often collides with ground or each other i tried ignoring ground but then cube falls through ground. Is there any other way? Maybe to still collide with everything but execute script whenever cannonball hits wall, but i’m not sure how to do that… PS: I tried with tags or layers but that didn’t work either.

public class Destructible : MonoBehaviour {
	RaycastHit hit;
	public GameObject destroyedVersion;
	void Start() {
	}
	void OnCollisionEnter(Collision collider)
	{
		if (hit.collider.gameObject.tag == "cannonball") {
			Instantiate (destroyedVersion, transform.position, transform.rotation);
			Destroy (gameObject);
		}

	}

}

what is the raycast variable for?

if you want a collision you dont need a raycast variable. it should just look like this:

 public class Destructible : MonoBehaviour {
    
     public GameObject destroyedVersion;
     void Start() {
     }
     void OnCollisionEnter(Collision collider)
     {
         if (collider.gameObject.tag == "cannonball") {
             Instantiate (destroyedVersion, transform.position, transform.rotation);
             Destroy (gameObject);
         }
 
     }
 
 }