How to glue objects together and still getting individual collision detection?

I am just starting developing a game in VR and the effect I am trying to achieve here is similar to minicraft. I would have a bunch of cubes where I can randomly stick cubes together and take away a cube from a group of cubes. I want to be able to use the Vive controller to do this.

My current idea is that I have a parent game object which has many children objects. I want this group of objects to always move together, behaving like the parent has a rigidbody. The only problem is, if I only put the rigidbody on the parent, than I would not get individual collision detection for the children. On the other hand, I can’t let the children have rigidbodies of their own as well because that breaks the physics engine. I have considered using joints as well, but I am not getting very good results. I am wondering if there is anyway of achieving what I am describing here. The number of children the parent object has would be dynamically changing in the game, so any sort of hard coding solution would not be possible. I am a new unity developer and I was really surprised that moving objects together and getting individual collision events would turn out to be so difficult.

If the Child Cubes have their own Colliders, You can access the specific collisions whit the hitpoints :

foreach (ContactPoint contact in collision.contacts)