Question relating to incorrect shadows when using directional lightmaps

When using directional lightmaps with specularity in the original shader, the shadows appear strange from many angles. I think this is because there is specularity visible in the shadows, and there shouldn’t be. As you can see from the image attached, when specularity is turned off the lightmap looks great, but when it is on, the shadows appear very strange indeed.

This happens even with only one light, and I’m using directional lights.

Are my settings wrong, or is there a way to disable specularity just in the shadows of the lightmap? Or is this just a necessary evil of directional lightmaps in Unity? Thanks in advance.

Normals are lost when you bake a scene, that issue will be address by Unity 4.

Take a look on the “Normal maps considered when light mapping” topic in this post