Able to collide with objects without effects of collision on player not using gravity

I am making a game where you fly around in 1st person view and I managed to have collision working with objects and player but when I hit into something the camera knocks and the player rolls.

How do I fix this

In your camera controller, you can smoothly follow your player instead of directly setting the position. Look up Vector3.SmoothDamp() and examples for it. That should take away the knocking effect when you hit something. As for your player rolling you can lock rotation for an axis for a RigidBody in the Inspector.