Apply torque to align rigidbody to a specific angle in 3D

I have a rigidbody ObjectToAttract and an empty gameobject with a script attached to it. I would like to apply torque to the rigidbody in order to align its rotation with the empty gameobject. Here’s the basic code for the script;

Quaternion AngDifference = Quaternion.Inverse(ObjectToAttract.transform.rotation) * (transform.rotation);
Vector3 AngDiffMovement = RectifyAngleDifference(AngDifference.eulerAngles);
ObjectToAttract.AddTorque(AngDiffMovement - ObjectToAttract.angularVelocity, ForceMode.VelocityChange);

And here’s the RectifyAngleDifference function;

private Vector3 RectifyAngleDifference(Vector3 angdiff)
{
    if (angdiff.x > 180) angdiff.x -= 360;
    if (angdiff.y > 180) angdiff.y -= 360;
    if (angdiff.z > 180) angdiff.z -= 360;
    return angdiff;
}

For some angles of the empty object the script works perfectly fine. But for others the rigidbody wobbles around in seemingly random directions. How can I fix that?

I have a separate revision of the above code;

Quaternion AngDifference = Quaternion.FromToRotation(ObjectToAttract.transform.up, transform.up);
 Vector3 AngDiffMovement = RectifyAngleDifference(AngDifference.eulerAngles);
 ObjectToAttract.AddTorque(AngDiffMovement - ObjectToAttract.angularVelocity, ForceMode.VelocityChange);

Which is significantly more stable, however it ignores an entire axis of rotation.

I solved my problem by using the second algorithm and applying a secondary corrective vector in addition to the main angular vector. Here is what the full code looks like;

Quaternion AngleDifference = Quaternion.FromToRotation(ObjectToAttract.transform.up, transform.up);

float AngleToCorrect = Quaternion.Angle(transform.rotation, ObjectToAttract.transform.rotation);
Vector3 Perpendicular = Vector3.Cross(transform.up, transform.forward);
if (Vector3.Dot(ObjectToAttract.transform.forward, Perpendicular) < 0)
    AngleToCorrect *= -1;
Quaternion Correction = Quaternion.AngleAxis(AngleToCorrect, transform.up);

Vector3 MainRotation = RectifyAngleDifference((AngleDifference).eulerAngles);
Vector3 CorrectiveRotation = RectifyAngleDifference((Correction).eulerAngles);
ObjectToAttract.AddTorque((MainRotation - CorrectiveRotation/2) - ObjectToAttract.angularVelocity, ForceMode.VelocityChange);

I found this solution to be super elegant and much more easy to wrap my head around when using a Vector 3 to define the desired rotation.

var rot = Quaternion.FromToRotation(transform.up, Vector3.up);
 rb.AddTorque(new Vector3(rot.x, rot.y, rot.z)*uprightTorque);

Source unity forum post for the code above