onMouse Events stop working when parent is changed, how to resolve without adding a Rigidbody?

I have a number of game objects that need to move / group together. I started to connect them with a fixed joint. As the number of objects connected increase we get wobble and end up in a mess. I have tried playing around with mass and other settings, and get it working for a while, then the problem comes back and is really unpredictable.

I have now switched approach.
I am reparenting the gameobjects together to group the colliders.
Adding the rigidbody mass to the parent object and removing the child rigidbodys.
My gameObjects all now behave 100% within the physics world.

The problem now is the onMouse events has moved to the parent. The OnCollisionEnter still works on the children ok.

How can I get the onMouse events working on the child objects?

Failed to find a fix for this so did a workaround.
Triggered my own mouse events around a Raycast using the hit.collider

    {
        // ### Ray cast out from main camera to mouse pointer to get object hit.
        ray = Camera.main.ScreenPointToRay(Input.mousePosition);
        if (Physics.Raycast(ray, out hit))
        {
            if  (hit.collider != previousHitCollider)
            {
                // Debug.Log("DoRayCast: Collider that is hit has changed");
                blockMouseIsNowOver = hit.collider.GetComponent<BlockControllerV3>();
                previousHitCollider = hit.collider;
            }
            return true;
        }
        else
        {
            return false;
        }
    }