Camera stuff is pretty tricky… you may wish to consider using Cinemachine from the Unity Package Manager.
There’s even a dedicated forum: Unity Engine - Unity Discussions
Otherwise perhaps some type of Lerp-based low-pass filtering on the position / rotation might work.
Lerping to smooth things out: