When building or updating from asset server using the new command-line switch -nographics is there any drawback? Like is it slower/faster, not building some stuff...
No, it's meant to be used when there's no gfx-card available ("true headless servers"). So, the only "disadvantage" (or advantage, depending from which side you're looking at it) is that you have no GUI. See also Unity 2.6 release notes, section Windows Specific Improvements. Quote:
-nographics option on Windows Editor allows you to use editor batch mode on a truly headless servers without any graphics card support.