Baked Lighting Broken?

I’m trying to bake the lighting in my scene, and it comes out looking horrible. I looked online and people said that it could be because my walls are transparent on one side (I did this for optimization) but I cant find a way to fix it without making my walls double-sided and dropping my framerate.
Here is what my walls look like without baked lighting. The normal maps are working fine and such.


and here are the walls when I bake lighting:

why does this happen?

Here are my lighting settings:

It may also be worth mentioning that this problem gets worse if I edit my UV maps in blender (so that the bricks face the right way)