Specific NavMesh Obstacles

I have procedurally randomly generated houses in my game on a terrain with a pre-baked Navmesh. I want the agents to be able to walk into the house but not through the walls and everything, so I tried using Navmesh links and creating a Navmesh surface to the house itself. Still, I have the problem that when the agent walks from the house surface to the terrain surface the house doesn’t take into account. The agent can walk through it, I want something like the house being an Obstacle but only to the terrain surface.
If anyone can help me I will be glad, thanks in advance!

Have you tried to make the houses static? And re-bake the map afterwards?