Having lots of small cubes simulating dirt appears to me to maybe not be a good plan. As Dwair pointed out, having many small colliders can be quite a nightmare for the physics engine.
You could consider having a mesh for the excavation site and procedurally modifying the mesh when the player digs. It will probably take a lot of tweaking to get right though.
Why do they need to collide with each other? Can't you just make the shovel tool follow the mouse cursor and use OnMouseDown() of the soil-cube to determine which cube to scale?
Did you put rigidbody component to the shovel too?
It is very important both phys colliders have mass, if you don't want the shovel to bounce or make funny movements, just use isKinematic and there you go!
The other part about "Small colliders" worries me most, small colliders are a headache for any physics engine, the only thing I can think of is modifying physics settings: You'll probably need to reduce min penetration for penalty and increment solver iteration count...
if you need help with your build, let me know. the picking things was not the hard part, the rigging of the arms was for a totally predictable, stable and reliable rig.
basically the big trick is to increase the solver Iteration count and adapt the time settings. Otherwise, joints are not tight and it becomes utterly unstable. My demo has quite large objects to pick up, but I tried with smallers during build phase and it can handle it fine.