Best way to do collision for walls or buildings?

When I put a mesh render on the wall my character just goes right through, and obviously I cant put Rigidbody on because then my wall will float away when you hit it. My Character has both a character controller and a rigidbody.

Add a mesh collider as well as the mesh renderer.