Changing distance to aim target cause LookAt jump/jitter

I have an archer in a third-person camera view. The camera rotates (aims) towards where the bow is pointing + added distance, which works well. I use Raycast to spot targets and shorten the aim distance to that target.
Problem: when I change distance to target; the cameras rotation jumps towards the new target position. Is there a way to make the transition smooth using LookAt on the camera?

Bow code for updating aimTarget:

void FixedUpdate() {
    float targetDistance = AIM_DISTANCE;
    RaycastHit hit;
    if (Physics.Raycast(transform.position, transform.TransformDirection(Vector3.forward), out hit, AIM_DISTANCE))
        targetDistance = Vector3.Distance(hit.transform.position, transform.position);
    aimAtPosition = transform.position+transform.TransformDirection(Vector3.forward) * targetDistance;

public Vector3 getBowTarget()
    return aimAtPosition;

Cameracode for looking at the target (positioning code omitted):


SmoothDamp or Lerp