Ambient light and lightmapping

I understand the basics of lightmapping my scene and my meshs but my problem is that to obtain shadows on the terrain I must use light size 0. This means that my directional light doesn't determine where the shadows are being placed, so it seems there is a global light coming at a 45 degree angle onto my terrain that I cant get rid of.

So how can I remove that global light?

If you're looking to get rid of the ambient light, you can find that in the panel that is displayed when you click on Edit > Render Settings.