Here is a Google Drive link to some images:
https://drive.google.com/folderview?id=0BxBuzdJ2dyguNE5LaXd2X1VQb0E&usp=sharing
I’ve said it many times before and I’ll say it one more time for you 
Don’t use SketchUp Free for Unity. The Meshes and artifacts are HORRIBLE. Learn Blender or invest in some better Professional software. Blender is Pro enough, though.
With SketchUp you are required to buy the Pro version before you can make a mesh which will properly work in Unity (last time I checked).
The trouble doesn’t come from Sketchup.
I’ve got exactly the same weird behavior using blender.
I thinks it can comes from realtime light and shadows.
Set your directional light to baked shadows. bake the scene.
Check if the size of your imported mesh is good by adding a cube next to it
@MrSecretMansion I had a similar issue, and the problem ended up being that I had internal faces in my geometry. Have you tried looking inside of you walls for extra geometry? If you find any extra faces where two walls meet, for example, delete them and rebake.