Transform.TransformDirection() only rotates the vector, it ignores translation. The method simply converts the transform’s world-space quaternion to a 3x3 matrix and then multiplies the given vector to that matrix. The algorithm can be demonstrated thus…
(Note: While this is functionally identical to Transform.TransformDirection(), it is almost certainly less optimized.)
Vector3 TransformDirection(Transform transform, Vector3 v)
{
Matrix4x4 m = Matrix4x4.TRS(Vector3.zero, transform.rotation, Vector3.one);
return m.MultiplyPoint3x4(v);
}
I’m not sure what you mean by “based around the direction of VectorB.” You need more than just VectorB to define the rotation. At the very least, you need a rotation angle around VectorB, from which we can infer the other two axes of the coordinate system.
You always need three vectors to define orientation in 3D space: the x-axis, y-axis, and z-axis vectors. It’s common to write functions that take in only two of those axis vectors and computes the third axis vector automatically. For example, you’ll notice that the Quaternion.LookRotation() method only takes a forward vector (z-axis) and an up vector (y-axis) as input, it calculates the right axis (x-axis) vector from the cross product of the forward and up vectors and then builds an orthogonal basis for the rotation.
You can use Quaternion.LookRotation() to write a function that transforms a point into a coordinate system defined by a forward vector (z-axis), an up vector (y-axis), and a position vector. Maybe this is closer to what you had in mind.
// transform vector v into the coordinate system defined by forward, up, and position.
Vector3 TransformToForwardUpPosition(Vector3 forward, Vector3 up, Vector3 position, Vector3 v)
{
Quaternion rotation = Quaternion.LookRotation(forward, up);
Matrix4x4 m = Matrix4x4.TRS(position, rotation, Vector3.one);
return m.MultiplyPoint3x4(v);
}