Vr objects not colliding correctly when collided with controller

I’m making a vr demo game currently and the movement, hmd positioning, character controller height scaling, controller tracking is all working correctly but when i try to hit an object that has a rigidbody with my controller that has a collider, the object doesn’t react as if it was hit at all, instead it just jitters awkwardly. But even with simple stuff like balancing an object on my controller will send it flying in a random direction. With games like boneworks, blade and sorcery, and the lab existing, which were all made with unity, I know there has to be some sort of solution.

I have a similar problem as well.
All colliders work and behave normally if they get hit with a non-controller game object, but when I hit a game object with an item that is tracking a controller…then the game object will just move out of the way. It will not have any inertia or physics behavior at all.

it’s good to know I’m not the only one then