Weirdest Collision EVER

I have set up a collision between my player and a door. But the thing is, when my player collides with the door, he bounces back and starts rotating away from the world. Is that got something to do with gravity, or mass? Can anyone help me figure out what could be triggering this odd behavior? Many thanks in advance!

Make sure that Freeze Rotation x,y,z are all checked in the Rigidbody of the character.