Directional Light + Deffered = Framerate Killer?

I am using Deffered Light to use HDR Lighting. I’ve got one test hall with simple geometry lit by point light and one directional light with shadows.

When I turn the Directional Light with shadows on, I’ve got about 15 frames per second less. Which is a lot, considering that without these shadows, the scene looks bad. What can I do to improve it or optimize it?

Namey5 was right. It’s just how deffered and forward are working in Unity - I read a lot on the internet about it. :confused: