Rotate an object using VR controller while changing player's position

Hello, I’m currently working on implementing a feature where an object can be rotated while maintaining its position in a virtual environment.

The interaction involves using the Oculus Quest 2 right controller, specifically utilizing the grab button to initiate the rotation. The goal is for the object to retain its last orientation when the grab button is released.

However, I’m facing a challenge in achieving a realistic rotation experience. I aim to allow the object to be rotated from various player positions and orientations, mimicking the behavior of an object in the real world.

It’s essential for the rotation not to be tied exclusively to a fixed reference axis of the controller. I want to avoid a scenario where, for instance, rotating the controller around the z-axis results in the object rotating around its z-axis as well.

In summary, I’m seeking a solution that enables dynamic and intuitive object rotation from different player perspectives while maintaining a natural and realistic feel. Any insights or guidance on how to achieve this would be greatly appreciated.