It could be partly due to different frame durations. You can test that by adding a random-length delay to your Update function and observing how well your game deals with it.
To compensate for that, for the last parameter to either Lerp function, pass:
1 - Mathf.Exp(-k * Time.deltaTime)
Here ‘k’ is a constant that controls the lerp rate.
Some people use a multiple of Time.deltaTime as the third parameter here, but that is wrong for Lerp - you should use an exponential of the form I stated above, to properly correct for varying frame times.
Beyond that, if the target object is moving with a fairly constant speed then you’ll still get some jitter, and can do even better with an alternate lerp function. See this thread for more details:
The third parameter should be a number between 0 and 1.
If you imaging a line going from the first parameter to the second parameter, the third parameter represents percentage of progress along than line that is used for the return value.
Here is a table that may help visualise the concept.
As you can see the rate of movement (delta) starts high but quickly tends towards zero.
For this use of lerp the trick to to find a “t” value that is not too high and not too low, it involves manual tweaking and a judjment call.