NavmeshManager MASSIVE spikes at unloading scene WITHOUT navmesh.

Hello,

After switching from Unity 2017.1 into 2017.3 and later 2017.4 we are occuring a issue which is giving a very hard time to our users as from time to time they are having a massive lag spike while travelling trough game.
We are not using navmesh on client side and scenes players are using doesnt have navmesh.asset at all.

I have attached a profiler to show how bad issue is, its not happening on every scene unload but could any of Unity devs give us any tip where should we start looking for a issue if its on our side?

This issue is very painfull ;(