Box mesh exhibiting odd collision with 1st person controller - how to fix?

My issue essentially boils down to this:

I am importing a "level" for my game from Blender. The limits of the level are brick walls, which are 4 box meshes. When I test the game, the collision for the 1st person controller registers with the outer side of the box, not the inside that is visible from the level. If you can imagine what this looks like, it means that the camera gets caught on the inside of the box, and is subsequently stuck. I've tried various ideas to edit both the collider and the mesh itself, to no avail.

How can I fix this issue?

Place extra box colliders as invisible walls around areas you don't want the player to be

Make sure that the box has inward facing polygons (and not double faced polygons which really points outwards).

Mesh colliders only work in one direction.

If that doesn't work then yeah, primitive colliders are a lot less breakable as spinalajck suggested :)