The problem (a very annoying one, I might add!) is that Vector3.Angle takes into account ALL rotation axes, not just the y axis. That’s why you get those high angle values unless looking directly at the target. There are basically two workarounds:
a) Make sure both positions are on the same y position
b) Use an object taht only rotates around the y axis: for example, a Character Controller that only rotates left and right, and where the camera is a child of it and only the camera rotates up and down (head rotation).