Occlusion Culling stops working as camera moves away from occluder walls

I'm having a very strange problem with occlusion culling where as the camera gets farther away from a giant wall that should be occluding everything behind it, more and more things behind it stop being occluded. I have no idea what's going on or how to fix this and was hoping someone could help me. I absolutely need this to work properly to get performance to be acceptable.

Here's a little video showing what's happening in the editor: hqkzkv

So after some more testing I found that I could stop this particular problem from happening by increasing the "Smallest Occluder" setting from 0.5 to 6. However a setting of 6 is much too large to have any meaningful occlusion happen inside that building you see on the right side of the map, where the walls are made up of much smaller sections. This really seems like a bug to me.

6265008--691494--upload_2020-9-1_0-6-3.png

So this is my workaround... That's how big of fake occluder wall and floor I had to put in to be able to hide that building while still keeping my 0.5 smallest occluder setting. Then I just disable the fake wall and floor after baking occlusion.