If you don’t want the character to rotate then turn-on the Z rotation constraint on the Rigidbody2D.
You say your terrain is smooth but if it’s made-up of multiple (Box?) colliders then no, to the physics system it’s not a continuous surface so occasionally you’ll get your collider catching the corner of them which causes a counter-rotation.
Try using a single EdgeCollider2D which does produce a single, continuous surface which won’t suffer from this problem.