Creating a VR flightstick - Issues with Physics

I’m working on a small space flight simulator and have the project so close to functioning and need just a little help with the controller aspect. I have the flight controller script functioning perfectly so my ship moves just the way I wanted (inertia based). I am now working on the input scripts. I was able to slap together a working slider inside the ship to control the throttle, but I am absolutely stuck on building a virtual flight stick. Where I’m stuck is on the physical model of the flight stick and how to make it function with the HTC Vive controllers to control the pitch and roll of the craft. I’m currently forced to use the keyboard (ASWD) until I solve this problem.

For simplicity, I’ve dumb down the model to a cylinder (as the flight stick) and two sphere to act as the X and Y axis independently connected at the base of the cylinder with two hinge joints. Sphere #1 is a parent of the Cylinder and is labeled as the X-Axis with a Hinge Joint (in the X direction). Works almost perfectly by itself. But when I create the second sphere (The Z-Axis) as a parent of the X-Axis and attach a Hinge joint, things start to act a little funny.

When I run simulations with the hinge motor, things look really good. The stick moves to 30 degrees is any direction and springs back to center. But when I power up the headset and operate the control stick with the Vive controller, the control stick and cylinders begin to stretch from their positions and don’t operate the stick in a realistic way. I feel like it must have something to do with how my controllers are attaching to the cylinder. Even when I freeze the X, Y, and Z positions of the spheres, they still pull out of their origin when moved with the Vive controllers. Any ideas how to fix this? I feel like this should be so frickin’ simple, but no matter what I try, something does not act correctly. Is the rigidbody/physics approach the wrong way to do this? Is there a different or easier way? I have to imagine several games have implemented a virtual flight stick. I seriously thought this would be the most simple part of the game, but it has had me stumped for several days now.

I’d try an approach that is not physics-based. For example, read the angles from the controller when it’s actually close enough to the virtual stick (and maybe with the trigger pressed), then apply those angles to control the stick and the plane.