I found this thread related to the issue, explains kind of why it happens, but I still find this extremely annoying when working with lots of prefabs. I guess the solution is just to limit the amount prefabs you have childed or create some sort of custom workflow to manage? Seems like this should be much easier regardless.