Hi guys, trying to shed some light on how things work 
Packages are enforced to use asmdef’s, so the package’s code do not get included into user compiled assemblies and by result should minimize the compile time when the packages are already compiled.
During a domain reload, JIT is only a cost when the code is executed, but if packages is using InitializeOnLoad, or other’s that are invoked after a reload, and they are doing something time consuming, it will slow down the whole flow, and will do so even if you are referencing the package or not in your own asmdef.
We are working on some tooling that will help Users understand what is happening, and what is taking up time under compilations/domain reloads.