Quaternion.slerp oscillates and wont face correct direction

I have a character that is supposed to walk forward then look at the player. This was working before but then I made a bunch of changes at once and it started oscillating instead of turning towards the player. Here is a video of the issue:

Here is the relevant code:

if (navAgent.remainingDistance < .2)
{
  lookPos = playerTarget.transform.position - agent.transform.position;
  lookPos.y = 0;
  rotation = Quaternion.LookRotation(lookPos);
  Debug.Log(rotation);
  agent.transform.rotation = Quaternion.Slerp(agent.transform.rotation, rotation, 0.01f);
  //do alignment stuff
  anim.SetBool("align", true);
  Align();
}

You seem to not understand what lerping (and slerping) is for. If you want to rotate something towards something else, you want to use FromToRotation, LookRotation, and RotateTowards, not Slerp.

https://www.youtube.com/results?search_query=what+is+lerp

Using any of those methods alone would just snap the character to that rotation. By using slerp it rotates slowly.

Of course, that’s because Unity won’t magically animate mathematical functions.
Your slerp doesn’t animate either, you’re just using it wrong and it appears like it’s doing something.
If you think my suggestion is not suitable, feel free to continue using slerp.

I mean RotateTowards would only do that if you passed in a huge delta…

I usually achieve a smooth ‘rotate towards’ with, well, RotateTowards, but measuring the angle to the target, and using an AnimationCurve or any easing function to reduce the delta as you get closer to your target.

In any case, as above, not a correct use of Slerp, and not really a situation where it should be used either.

You would definitely want to use RotateTowards because that’s the whole purpose of this function, to actually help smoothly animate rotating towards something, but you need to supply animation values.

FromToRotation and LookRotation can be used as well, it’s just this would work less optimal because they’re better used in scenarios where you work with directions, not angles. In your case you don’t need the angles either, however you want to animate the angle linearly, and therefore need to use something that helps with that.

In theory you can use lerping for this, to lerp the angle between start and end, right? But in practice this is a hassle, because you have access to better methods, instead of having to extract the angle (and angles are a poor choice when you have access to a quaternion already), lerping that, and then building a rotation quaternion out of that.

All you need to do is the following

void Update() {
  this.transform.rotation = Quaternion.RotateTowards(this.transform.rotation, _targetRotation, _rotationSpeed * Time.deltaTime);
}

Of course to establish _targetRotation you’ll have to do something of the above. Say

_targetRotation = Quaternion.LookRotation(dir(player.transform.position, enemy.transform.position), Vector3.up);

and then

Vector3 dir(Vector3 a, Vector3 b) => (b - a).normalized;

Full thing

using UnityEngine;

public class RotationDemo : MonoBehaviour {

  [SerializeField] [Min(0f)] float _rotationSpeed; // in degrees per second
  [SerializeField] GameObject _target;

  void Update() {
    if(_target == null) return;
    var currentRot = this.transform.localRotation;
    var degPerFrame = _rotationSpeed * Time.deltaTime; // degrees per frame
    var dir = direction(this.transform.position, _target.transform.position);
    currentRot = rotateTowards(currentRot, dir, degPerFrame);
  }

  static Vector3 direction(Vector3 a, Vector3 b) => (b - a).normalized;
  static Quaternion lookAt(Vector3 dir) => Quaternion.LookRotation(dir, Vector3.up);
  static Quaternion rotateTowards(Quaternion from, Quaternion to, float maxDegrees)
    => Quaternion.RotateTowards(from, to, maxDegrees);

}

It is preferrable to rebuild direction only when the target actually moves, and you can track this instead.

using UnityEngine;

public class RotationDemo : MonoBehaviour {

  [SerializeField] [Min(0f)] float _rotationSpeed; // in degrees per second
  [SerializeField] GameObject _target;

  Transform _xf;
  Vector3 _targetPos, _targetDir;

  void Awake() => _xf = this.transform;

  void Update() {
    if(_target == null) return;

    if(_targetPos != _target.transform.position) {
      _targetPos = _target.transform.position;
      _targetDir = direction(_xf.position, _targetPos);
    }

    var degPerFrame = _rotationSpeed * Time.deltaTime;
    _xf.localRotation = rotateTowards(_xf.localRotation, _targetDir, degPerFrame);
  }

  // ...

}