The best solution is to backoff the drag setting and instead keep your objects from moving around by assigning the right physic materials to the colliders.
You might get what you want just by upping the setting for gravity to compensate for the friction. Go to Edit > Project Settings > Physics and set the ‘Y’ value for gravity to a larger negative number.
Alternately you could play with doing your own drag. Here is a simple version. Set the drag in the Rigidbody back to ‘0’ and then place the following two lines in FixedUpdate():