What are the equation/equations used in PhysX for adding torque to a rigidbody? and what mathematical representation does is use to store/add torque? Euler angles? Body rates?

Most likely not the equations you would expect. PhysX uses a simplified dynamics model for efficiency rather than accuracy:

Anyways, the source code of PhysX is available here:

The files you may want to look at are NpRigidDynamic.cpp/h (=Rigidbody) and NpScene.cpp.