I want to know how to rotate D about A so that A, B, and D are collinear.

Then I want to know how to rotate E about A so that A, C, and E are collinear.

And then I want to know how to take those two rotations, average them together, and instead rotate D and E about A with that average rotation, so that A, B, and D are sort-of collinear and A, C, and E are sort-of collinear… it’s supposed to be a best attempt at satisfying both constraints only rotating D and E about A with a single rotation.

I think this will involve Quaternions, but beyond that, I’m not really sure how to do this.

This is a direct translation of your problem above in Javascript/Unityscript. A, B, C, D, and E are assumed to be Vector3 positions in world space. Without a better description of how this is going to be used, I cannot be sure this is the ‘real’ solution to your probelm.

var q1 = Quaternion.FromToRotation(D - A, B - A);
var q2 = Quaternion.FromToRotation(E - A, C - A);
var avg = Quaternion.Slerp(q1, q2, 0.5);
D = A + avg * (D - A);
E = A + avg * (E - A);