VR non delayed object movement with collision

I am starting development in VR and have problem with movement and collision of objects.
There are three types of movement and collision behaviours available and I have three identical cubes with each of them but I found that each of them has issues when I am holding the cube in hand.
Kinematic - no matter the settings, movement is delayed and cube doesnt collide with objects without non-kinematic rigidbody ( non kinematic rigidbody objects react to it and move as normal, cube just goes through wherever hand goes).
Velocity tracking - does have collision with everything but movement is delayed no matter the settings.
Instantaneous - movement is exact with no delay but cube does not have collision with kinematic rigidbody objects or objects without rigidbody.
I understand why these types of physics do what they do, for instance two kinematic rigidbodies do not react cause they do not use physics, but what I want to know, is there a way to make it so that object has instantaneous movement with hand when player holds it but when object touches another object which has no (or kinematic) rigidbody, it reacts to it like velocity tracking?

I think it to be a crucial thing, for example player holding weapon or some item needs it to have instant response to movement of hand and aiming but weapon also needs to react when against wall or ground and other non moving objects.