Restructuring Project without Breaking Everything?

We have a problem where we are trying to adapt a new structure to the project in the asset folder. When I move stuff around, it breaks everything that is in the scenes. I get a bunch of errors that say Unity has no idea where the files are. Is this because the meta file must stay with the original file or is there is something else?

Before 3.5, the only possibility was to restructure everything from within Unity, as @Fattie noted.

Since 3.5 and the introduction of .meta files, however (need to be enabled in ProjectSettings->Editor->VersionControl, or the Asser Server needs to be in use), you can now also move/rename assets outside Unity, provided that you also move/rename its associated .meta file(s) along with it.