How do I exclude objects in a building from wind zone outside?

I’ve got a world with trees and commotion outside the building. Inside the building, I have a few potted plants that should sway with the AC. Outside the building, the trees should be rustling for different parameters of wind.

How do I affect these two objects differently? Multiple directional wind zones just compound the amount of wind. I’m looking for a way to exclude the building and everything in it from the outside wind zone. Inside and outside wind zones, anyone?

Disable the Tree component in the trees that you don’t want to be affected by wind.