Basically, how do i remove the terrain’s reflection without disabling or changing my directional light?
There’s not too much you can do about this without changing the actual material of the terrain. In the terrain settings tab, you could change the shader or material to something like legacy diffuse, however most of the time all you should really do is add a base texture, and set the ‘smoothness’ to something low. It can be annoying if you are just testing stuff and don’t want to add a texture, but that’s just how the standard shader is.
Looks like in the new versions of Unity you can click the Material, then edit the Smoothness slider or untick Specular Lights to solve this issue.