When using a controller on the Xbox One is just acts kind of like a mouse and allows me to close the app. Is there some trick to have the controller control my game the same way it does when i test on PC?
The Input Manager button and axis mappings are often input device and platform depended.
For example, if you add input mappings for an Xbox One Controller for Windows 10, the same mapping are unlikely to work on OSX and Linux. I imagine the same behavior occurs if you deploy to an Xbox One, which is a different platform than Windows.
In this case, I assume you’d need to add input mappings that work with the given controller on Xbox One.
Here is the documentation how to add button and axis mappings to the Input Manager: