as rotation’s magnitude becomes zero, the rotation of such a vector is undefined and will also become unstable due to floating point imprecision.
You may be able to simply drive transform.forward (or transform.up when in 2D) equal to your movement vector, never when it falls below a certain magnitude threshold.
Otherwise, use Mathf.Atan2() to derive heading from cartesian coordinates:
Sine/Cosine/Atan2 (sin/cos/atan2) and rotational familiarity and conventions