AddRelativeTorque causing wild spining

Hey all, I am trying to make simple flight controls. Using the mouse for roll and pitch and the horizontal axis for yaw.

void FixedUpdate () {

	// Get Input
	float rollInput = Input.GetAxis("Mouse X");
	float pitchInput = Input.GetAxis("Mouse Y");
	float yawInput = Input.GetAxis("Horizontal");

	// Calculate Change
	float targetYawVelocityChange = yawInput - rigidbody.angularVelocity.y;
	float targetRollVelocityChange = rollInput - rigidbody.angularVelocity.z;
	float targetPitchVelocityChange = pitchInput - rigidbody.angularVelocity.x;

	// Apply Torque
	rigidbody.AddRelativeTorque(targetPitchVelocityChange,
	                            targetYawVelocityChange,
	                            targetRollVelocityChange);
}

I am trying to use a similar idea to the RigidbodyFPSController on the wiki, using the velocity change so that when all input stops the craft will slow to a stop (rotationally)

The problem is this, whenever I turn it starts madly spinning and I cant really see why. Can anyone point out my error?

Doing some reading I found the best way to achieve what I am looking for is to use a PID controller. Im implementing one now. Thanks for all the input. Cheers